ChurnBench is an open-source that generates a changing four-source enterprise data fabric as a timeline, not a snapshot. Every change is recorded in an append-only ground-truth ledger, and gold answers are computed from that ledger. This allows freshness errors to be detected separately from reasoning errors by comparing ground truth at retrieval and evaluation timestamps. The benchmark reveals that across cache ages of 1, 14, and 28 days, freshness errors were only 7, 4, and 4, because scheduled refresh bounds staleness by time-to-live (TTL). A controlled ablation that disabled tiered refresh caused freshness errors to jump from 4 to 45 at 28 days. This shows that refresh scheduling and TTL configuration, not drift-window length, govern answer staleness. Existing retrieval benchmarks freeze data, so they cannot detect whether an answer remains true. ChurnBench is significant because it separates freshness from reasoning and identifies the actual driver of staleness in production agentic systems.
Refresh schedules, not cache age, determine agent answer staleness
ChurnBench separates freshness failures from reasoning errors by evaluating answers against timestamped changes across four enterprise data sources.
Independent
Vivek Kumar Singh · Preeti Priyam
Research Digest··2 min read
In one line
Refresh scheduling, not cache age, governs answer staleness in agentic AI systems.
Before you read this
Two ideas this paper builds onRead it as
Singh and Priyam introduce ChurnBench, an open-source benchmark that models changing enterprise data as a timeline rather than a fixed snapshot.
The authors generated a four-source enterprise data fabric in which events such as user offboarding, license reassignment, price changes, and contract renewals occur over time.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.