Together AI's Together Compiler: Agent-Driven Continual Tuning Delivers Up to 2.6x Faster Inference
Overview
Together AI has introduced Together Compiler (TKCC), an agent-driven system that continuously re-optimizes inference as production traffic patterns shift. The compiler already runs on several of Together's serverless endpoints. The core argument for developers is simple: an inference engine tuned once goes stale, so tuning should never stop.
TL;DR
- System: An optimization system tested on Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash, and Llama 1B.
- Runs on: NVIDIA B200, B300, GB300, and H100 GPUs, plus AMD MI355X and AWS Trainium2. Each run used at most two nodes per model.
- Performance: Up to 2.6x higher tokens per second (TPS) than the best published InferenceX configurations, achieved in under 24 hours per setting.
- Best result: 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1), a 1.8x gain over the 633 TPS baseline.
- Worst result: GLM 5.3 on MI355X improved only from 463 to 480 TPS (roughly 4%).
What Is Together Compiler?
Together Compiler is a continual compilation loop from Together AI that uses coding agents to rewrite a model's forward pass. Given a server launch command, a traffic profile, and a hardware configuration, the agents propose faster forward-pass implementations, while a persistent engine measures each candidate. The winning code ships directly into the serving stack.
By treating inference optimization as an ongoing process rather than a one-time tuning exercise, Together Compiler adapts to changing traffic rather than assuming a fixed workload. This is particularly relevant in 2026, as agentic and multi-turn workloads make request patterns increasingly dynamic, and as the hardware landscape grows more heterogeneous across NVIDIA, AMD, and custom silicon such as AWS Trainium2.
Why It Matters
As production inference increasingly spans mixed hardware and volatile traffic profiles, static kernels and hand-tuned configurations decay quickly. Together Compiler's continual loop aims to keep endpoints aligned with the actual workload and silicon they run on, with results achieved in under a day per configuration rather than through months of manual kernel engineering.
via MarkTechPost