#sparse attention
Sparse Attention: 3 AI articles covering sparse attention news, analysis, and research
Articles
Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path⭐8
Zyphra's Hierarchical Attention (ZHA) adds a tree-structured indexer for true subquadratic sparse attention, matching dense performance while attending to ~1% o...
Build and Train a 25M Parameter LLM From Scratch on Your CPU⭐8
Build and train a 25M parameter LLM from scratch on your CPU. This freeCodeCamp tutorial covers pre-training, RL fine-tuning, MoE, and modern AI research workfl...
RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context⭐8
RBS-Attention is a training-free sparse prefill method that fixes mean dilution in long-context LLM inference, delivering up to 5.97× faster time-to-first-token...