What they did
The authors trained two orchestrator models (Fugu and Fugu-Ultra) using a combination of large-scale fine-tuning, evolutionary algorithms, and reinforcement learning. The orchestrator is itself a language model that takes a user query and outputs an agentic scaffold—a plan and sequence of tool calls—that coordinates a team of specialized LLMs. The team composition is dynamic and adapted per query. Performance was evaluated on six benchmarks covering software engineering (SWE-Bench Pro), terminal operations, code generation (LiveCodeBench), graduate-level science (GPQA-Diamond), general knowledge (Humanity's Last Exam), and visual reasoning (CharXiv Reasoning).
Key findings
- Fugu-Ultra achieved state-of-the-art results on all six benchmarks compared to other publicly accessible models, including GPT-4o, Claude Opus, and Gemini 2.0.
- On SWE-Bench Pro, Fugu-Ultra reached a 45.6% resolve rate, compared to the previous best 38.5%.
- On Terminal Bench, Fugu-Ultra scored 52.3% task completion, outperforming the prior best of 44.1%.
- The lightweight Fugu model maintained competitive performance with lower latency, making it suitable for everyday use.
Why it matters
This work demonstrates that dynamic, query-adaptive agentic scaffolds can combine the domain-specific strengths of multiple LLMs into a system that outperforms any individual model. It provides a concrete training methodology and production infrastructure for building such orchestrators, suggesting a path toward collective intelligence as a means to scale AI capabilities beyond what a single model can achieve.
Caveats
The authors do not specify the exact set of LLMs used in the agent team or their relative contributions to performance gains. The report focuses on evaluation on static benchmarks; real-world deployment robustness and cost implications of orchestrating multiple models are not addressed. The approach requires substantial compute for training and inference, and its generalization to entirely novel tasks remains to be tested.