Together Compiler: Continual Compilation for AI Inference
Authors: Reyna Abhyankar, Dylan Lim, Drew Wadsworth, Venkata Hasith Vattikuti, YuTong Julia Zhang, Jung Hwan Heo, Taeksang Kim, Kesavan Ramakrishnan, Evangelene Stanley, Xinyu Fang, Daniel Y. Fu, Simran Arora
Subject: Computer Science > Artificial Intelligence (cs.AI)
arXiv: 2611.14681 — Submitted 10 November 2026
Abstract
Today's AI workloads are changing rapidly: new models are released more frequently, model architectures are increasingly diverse, and traffic distributions shift constantly. At Together AI, the core challenge is continual optimization for these evolving workloads. Compiling a deployment into an optimized forward pass is a task well-suited to agent hill-climbing—an iterative, automated optimization loop driven by AI agents.
Existing systems struggle with continual compilation because of a tradeoff between measurement throughput and fidelity. Many systems use agents to optimize individual kernels within the inference forward pass using offline benchmarks, but the resulting timings are low-fidelity due to data-dependent behavior in primitives such as mixture-of-experts (MoE). Conversely, systems that use agents to optimize end-to-end engines directly suffer from slow engine launch times.
To address these challenges, we introduce Together Compiler (TKCC), the first truly continual agent loop designed specifically for AI inference optimization. To resolve the throughput–fidelity tradeoff, we propose two key components:
- A Tiered Persistent Engine that decouples engine launch overhead from forward pass computation.
- A cost model that selects the parallelism configurations used to seed the optimization process.
We also conduct extensive ablations on agent context management and share insights from deploying TKCC in production.
Results
TKCC outperforms highly optimized hand-written Megakernels on the Llama 1B workhorse workload by up to 1.7×. On the latest frontier models—Kimi K3, GLM 5.3, MiniMax M3.1 Flash, and DeepSeek v4.1 Flash—TKCC achieves up to 2.6× higher tokens per second (TPS) than the best open-source engines across hardware platforms on the InferenceX benchmark.
TKCC also sets state-of-the-art performance at batch size 1, delivering 1111 TPS on M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash.
Finally, TKCC generalizes beyond frontier language model inference. It currently serves trillions of tokens across Together AI's production fleet.
via ArXiv AI