Stanford and UC Berkeley Researchers Unveil TokenGate: Token-Gated Attention Accelerates Long-Context Prefill Up to 11.3x

Stanford and UC Berkeley researchers have released TokenGate, a module designed to accelerate long-context prefill in large language models. TokenGate uses learned token pruning that lets an existing LLM "skim" its input, so that only the tokens that genuinely matter receive full processing. The research team reports 1.4x to 11.3x faster time-to-first-token (TTFT) with no accuracy loss, alongside a 1.5x to 3.4x reduction in peak prefill memory. Code and checkpoints are available on GitHub under the Apache-2.0 license.

TL;DR

• MLP gate: A small MLP gate and LoRA (r=16) are added to Qwen3-1.7B or Qwen3.5-2B and tested on contexts up to 128k tokens.
• Hardware: Paper timings use a single A100 at batch size 1; training used 4 A100s.
• Performance: Pareto-optimal on 5 tasks against 6 baselines.
• Best case: NIAH 16k at 11.3x faster TTFT and 3.42x lower memory, with TTFT gains reaching 13.0x at 128k.
• Worst case: Speedup drops to 1.4x on some tasks.

What Is TokenGate?

TokenGate is a drop-in component that adds trainable token pruning to pretrained LLMs. Standard transformers push every token through every layer, and attention cost grows quadratically with sequence length. Training-free pruning is inexpensive but loses accuracy at high pruning rates. As the 2026 landscape increasingly shifts toward agentic workflows and million-token context windows, inference efficiency during prefill has become a critical bottleneck for production deployments—making learned, accuracy-preserving pruning approaches like TokenGate especially relevant.

Arc...

via MarkTechPost