Models and the way people use them keep changing, so the best way to run them can change too. Testing isolated code is quick but can miss real request behavior; testing the whole serving system is realistic but slow to restart.
Together Compiler keeps the serving engine running while automated agents replace and test the model’s computation. It uses quick tests to explore and realistic tests to verify promising changes, while a cost model, an estimate of which setups are worth trying, narrows the search.
On selected newer models, it reports up to 2.6 times the output rate of the open-source engines it compared against. This suggests continually retuning inference software for real workloads can beat relying on fixed optimizations as models and traffic change.