AgentPerfBench is a suite for inference performance of agentic LLMs. It builds workloads from real traces of agentic benchmarks such as SWE-Bench and TerminalBench, plus chat baselines, and samples input length, output length, and turn count distributions from those traces to create synthetic profiles. It also collects kernel-level Nsight Compute traces and uses a multi-dimensional roofline model to identify whether hardware is limited by memory bandwidth or memory capacity.
The authors report that on LLaMA-3.1-70B with H100, switching from chat to coding-agent workloads increases TTFT by 4.8x (239 ms to 1139 ms) and TPOT by 1.5x (19 ms to 28 ms) at saturation. For SWE-Bench agents over 88 turns, context lengths exceed 32K tokens, and median TTFT rises to 433 ms versus 62 ms for chat, a 7.0x increase, while TPOT stays around 16-18 ms.
This matters because existing serving benchmarks mostly use fixed-length single-turn chat workloads and report a single operating point. AgentPerfBench argues that those methods miss context growth and do not measure hardware at saturation, so system optimizations tuned on them may not reflect real agentic workloads.