Together AI unveiled Together Compiler on Wednesday, a system that uses AI coding agents to generate, test and improve GPU kernels for its inference infrastructure without an engineer directing each experiment, the company said. The agents can do overnight what might otherwise take engineers weeks of manual tuning, but human review remains required before any code is deployed to production.
Together AI says 80% of OpenRouter traffic now comes from models released within the past three months, making it difficult to keep inference infrastructure optimised as each model has different requirements. The compiler uses multiple coding agents running experiments simultaneously, writing and testing GPU kernels while sharing results to guide further improvements.
The company has also accelerated testing by keeping the inference engine running and loading new code rather than restarting it with each change. According to Together AI, this cuts initialisation time from roughly 467 seconds to nine seconds.
Together AI measures the system's advantage by the number of optimisation experiments it can run. Using Apple's RSI Inference Bench harness as its baseline, the company says Together Compiler can run 252 times as many experiments as Apple's system on the same workload and hardware. It separately reports inference throughput improvements of up to 2.6 times compared with published benchmark configurations, though these results have not been independently verified and depend on the model and hardware.
The company says the performance gains will translate into better performance for customers at the same dollar cost rather than lower prices. "Together Compiler improves Together AI's infrastructure efficiency. For the same dollar cost, customers will get better performance," the company told The New Stack.
Together AI says the compiler is already serving trillions of tokens daily. The development is another sign that coding agents are moving beyond application code into GPU performance optimisation, with engineers reviewing their work.
Distilling most of our expertise in kernel and inference engine optimization into an agent loop that powers our production. A 2-year project that we're finally able to share x.com/simran_s_arora…
QuotingIntroducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar@dylan__lim@realDanFu@togethercompute!!View post
The system automates a task that previously required engineers weeks of manual work, potentially lowering the cost and time needed to optimise inference for new AI models.
It demonstrates AI agents taking on infrastructure-level performance tuning, a domain traditionally handled by human specialists.
The approach keeps humans on the deploy button, suggesting a practical middle ground between full autonomy and manual coding.
Background
Together AI is a cloud platform that provides access to open-source and custom AI models for inference. The company competes with providers like Groq, Fireworks AI and others in optimising inference speed and cost. GPU kernel tuning is a specialised engineering task that adjusts low-level code to better use a GPU's hardware for a specific workload. The company previously offered inference infrastructure but not an automated compiler for kernel optimisation.
What to Watch
Independent verification of Together Compiler's throughput claims, which have not been peer-reviewed or audited.
Whether the system's reliance on human sign-off for deployment slows adoption or becomes a competitive advantage.
How other inference providers respond with similar automated kernel optimisation tools.
12 claims in this story checked against 1 source: 12 supported.
Show each claim
SupportedTogether AI unveiled an autonomous compiler to optimise GPU code for inference.The New Stack: “The company unveiled Together Compiler on Wednesday, a system that generates, tests, and improves GPU kernels without an engineer directing each experiment”
SupportedTogether Compiler can run 252 times as many experiments as existing agent-based systems.The New Stack: “Together AI says its compiler can run 252 times as many optimization experiments as existing agent-based systems.”
SupportedThe agents can do overnight what might otherwise take engineers weeks of manual tuning.The New Stack: “the agents can do overnight what might otherwise take engineers weeks of manual tuning.”
SupportedTogether AI unveiled Together Compiler on Wednesday.The New Stack: “The company unveiled Together Compiler on Wednesday”
SupportedThe system generates, tests and improves GPU kernels without an engineer directing each experiment.The New Stack: “a system that generates, tests, and improves GPU kernels without an engineer directing each experiment”
SupportedA human must review changes before they are deployed to production.The New Stack: “the final evaluations and deployment process are automatic but must be kicked off by a human, who reviews the changes before deploying to production.”
SupportedTogether AI said: “Together Compiler improves Together AI's infrastructure efficiency. For the same dollar cost, customers will get better performance.”The New Stack: ““Together Compiler improves Together AI’s infrastructure efficiency. For the same dollar cost, customers will get better performance.””
SupportedTogether AI says 80% of OpenRouter traffic comes from models released within the past three months.The New Stack: “Together AI says 80% of OpenRouter traffic now comes from models released within the past three months.”
SupportedMultiple coding agents run experiments simultaneously, writing and testing GPU kernels and sharing results to guide improvements.The New Stack: “Multiple coding agents run experiments simultaneously, writing and testing GPU kernels while sharing results to help guide the next round of improvements.”
SupportedKeeping the inference engine running and loading new code cuts initialization time from roughly 467 seconds to nine seconds.The New Stack: “According to Together AI, this cuts initialization time from roughly 467 seconds to just nine seconds.”
SupportedThe 252-times comparison uses Apple's RSI Inference Bench harness, with both systems tested on the same workload and hardware.The New Stack: “The company tells The New Stack that it used Apple’s RSI Inference Bench harness as its baseline, comparing both systems on the same inference workload and hardware, with only the optimization approach changing.”
SupportedTogether AI reports inference throughput improvements of up to 2.6 times versus published benchmark configurations; results are unverified and depend on model and hardware.The New Stack: “The company separately reports inference throughput improvements of up to 2.6x compared with published benchmark configurations. Those gains depend on the model and hardware, and the results haven’t been independently verified.”
