Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention
By Asif Razzaq β November 2, 2026
Zyphra has released Zyphra Hierarchical Attention (ZHA), a sparse attention method built around a tree-structured indexer. ZHA targets a known gap in sparse attention: the indexer that selects tokens still scans the entire context, which keeps overall cost effectively quadratic. Zyphra's technical paper shows that ZHA can be distilled into frozen pretrained models using roughly 4B tokens. On Qwen3.6-35B-A3B, it stays close to dense attention on reasoning, agentic coding, and 256K-context retrieval benchmarks.
TL;DR
- Size: ZHA adds 81,920 trainable parameters to Qwen3.6-35B-A3B (about 0.00024%) and was tested up to a 256K context window.
- Runs on: It retrofits frozen pretrained models after a 4B-token KL warmup at 32K context. Reported speedups come from a modeled H100 BF16 roofline; no optimized kernels were released. An FP4 router variant was also tested.
- Performance: Within a few points of dense attention across seven reasoning, coding, and retrieval benchmarks.
- Best result: LongBench v2 reaches 63.4% with the deep tree variant versus 58.0% for dense attention, while attending to about 1.1% of keys.
- Worst result: Degradation appears on NoPE-based Nem...
via MarkTechPost