Alibaba's Qwen team has released Qwen3.8-Flash-Next, an open-weight multimodal Mixture-of-Experts (MoE) model optimized for cost per token. The checkpoint pairs a 125B backbone with a 51B N-gram embedding table and a 4B multi-token prediction module, all while activating only 6B parameters per token. The team positions this as an early preview of the architecture that will underpin Qwen4, mirroring the role Qwen3-Next played for Qwen3.5. This release introduces four key innovations: a hybrid Gated DeltaNet and Qwen Sparse Attention mechanism, Gated Residual connections, N-gram Embedding, and the Muon optimizer. According to the team, training costs are roughly one-ninth that of Qwen3.7-Plus.
Deployment Considerations
Yes, but not on a workstation. The FP8 checkpoint is 172.78 GiB, while the BF16 variant is 335.28 GiB. Per vLLM recipes, TP2 is the minimum validated FP8 configuration on GB300, with TP4 recommended. On an 8×H200 node, TEP8 is required; plain TP8 is incompatible with the checkpoint's 128-wide quantization blocks. Sparse activation reduces compute but not storage needs.
via MarkTechPost