Speculative decoding addresses the fundamental challenge of slow, sequential token generation in language model inference. DFlash speculative decoding support for CPUs was recently enabled in vLLM v0.25.0. In our testing with Qwen3.5-9B on an r8i AWS instance, powered by Intel® Xeon® 6 processors with Performance-cores, DFlash increased average token generation throughput to 3.92x that of the autoregressive baseline at concurrency 1—a 74% cost reduction per generated token.
This post explains what speculative decoding is, and how DFlash in particular allows you to speed up your AI workloads on CPU. Our configurations are detailed at the end of this post if you’d like to follow along and reproduce the results. (Full disclosure: the author is affiliated with Intel.)
What is Speculative Decoding?
A typical autoregressive decoder produces one token, appends it to the context, and runs again to predict the next token. No kernel optimization can parallelize across a dependency that has not been resolved yet, so a 500-token response requires about 500 dependent cycles. Each token generation step requires the transfer of the entire model’s parameters from memory to the processor’s compute units. This makes the operation memory bound and leaves those compute units mostly idle at low concurrency. Speculative decoding works around that serial dependency without changing the model’s output distribution. A lightweight draft model proposes several future tokens. The larger target model checks all of them in a single pass, accepts the longest valid prefix, and corrects the first miss, and—if every token is accepted—generates one bonus token. When the draft proposals are high-quality and fast, each expensive target pass commits several tokens instead of one. Unlike lossy optimization techniques such as quantization, speculative decoding is a lossless acceleration since rejection sampling recovers the target distribution.
This technique is especially relevant to inference on Xeon. At small batch sizes, decode often spends much of its time moving model weights from memory for very little work per weight. Verification turns per-token matrix-vector operations that lean on Intel® Advanced Vector Extensions 512 (Intel® AVX-512) into matrix-matrix operations that Intel® Advanced Matrix Extensions (Intel® AMX) accelerates: the target weights, once loaded, are reused across several candidate positions. The catch is that speculation adds a drafter and widens verification. It only pays off when the accepted work exceeds that overhead: we’re trading spare compute for saved memory bandwidth; when there is no spare compute, the trade becomes a loss.
DFlash: Block Diffusion for Drafting Plus Target KV Injection
DFlash is a novel speculative decoding method developed by Z Lab. Instead of generating draft tokens serially as in some older algorithms, the DFlash speculator predicts a block in one pass with a small block-diffusion drafter. It also injects hidden features from the target model into the draft model’s KV cache, which improves draft quality without requiring the drafter to reconstruct the full context by itself.
Why DFlash on CPUs?
The combination of DFlash’s block-level drafting and CPU-specific optimizations makes it particularly effective on Intel Xeon processors. The block-diffusion drafter reduces the number of sequential steps, while the verification phase leverages Intel AMX to process multiple candidate positions in parallel, maximizing compute utilization. In 2026, with the growing demand for cost-efficient inference, CPU-based deployment is becoming a key alternative to expensive GPU clusters, and DFlash narrows the performance gap significantly.
Benchmark Results
We benchmarked DFlash on vLLM v0.25.0 with Qwen3.5-9B on an r8i instance (Intel Xeon 6 with Performance-cores). At concurrency 1, the average token generation throughput improved to 3.92x the autoregressive baseline. This translates to a 74% cost reduction per token, making it a compelling option for production workloads where latency and cost matter.
How to Reproduce
To reproduce our results, use the following configuration:
- Model: Qwen3.5-9B
- Hardware: AWS r8i instance (Intel Xeon 6 with Performance-cores)
- Software: vLLM v0.25.0 with DFlash speculative decoding enabled
- Concurrency: 1
Enable DFlash by setting the speculative decoding flag in vLLM (e.g., --speculative-model dflash). Full steps are available in the vLLM documentation.
Conclusion
Speculative decoding, and DFlash in particular, offers a lossless way to accelerate token generation on CPUs, addressing the memory-bound bottleneck of autoregressive inference. With nearly 4x throughput improvements and significant cost savings, DFlash on Intel Xeon makes CPU-based inference a practical choice for AI workloads in 2026.