#small-batch decoding

Accelerating LLM Inference via Vector Index Based Output Embeddings⭐10

Vector index-based output embeddings accelerate LLM decoding by replacing dense projections with HNSW retrieval, boosting throughput up to 82% while preserving ...