DolphinBench is a new benchmark for that moves beyond traditional question-answering formats. It defines three knowledge-work personas (startup CEO, infrastructure engineer, product manager) with roughly 500,000 tokens of user messages each, and evaluates agents on 200 tasks per persona that depend on information from that history. Crucially, tasks require actions (e.g., posting an update) rather than answering a direct query, removing the retrieval signal that QA benchmarks provide.
The benchmark reports accuracy alongside total cost and median latency, enabling a holistic view of memory system performance. Each test is verified by running an agent with the relevant history (must succeed) and without it (must fail). In evaluations, the highest-scoring agent achieved 70.67% of tasks. Memory systems ranked differently across agents, and the most expensive system was neither the most accurate nor the fastest.
This is interesting because existing memory benchmarks omit cost and latency, rely on QA formats that hint at what to retrieve, and lack per-test . DolphinBench addresses these gaps, offering a more realistic and rigorous evaluation for agent memory systems in production-like settings.