What they did
The authors generated a four-source enterprise data fabric in which events such as user offboarding, license reassignment, price changes, and contract renewals occur over time. Every update is recorded in an append-only ledger, and gold answers are derived from that ledger rather than from the mutable live stores.
For each evaluated answer, ChurnBench resolves ground truth at both retrieval time and evaluation time. This distinguishes a freshness error—an answer that became outdated after retrieval—from a reasoning error. The authors tested cache ages of 1, 14, and 28 days, then ablated the system's tiered scheduled-refresh mechanism.
Key findings
- With scheduled refresh enabled, the benchmark recorded 7, 4, and 4 freshness errors at cache ages of 1, 14, and 28 days, respectively; error rates did not rise with cache age.
- No TTL lapse occurred in the reported evaluation windows, supporting the proposed mechanism: scheduled refresh bounded how stale cached data could become.
- Disabling tiered refresh increased freshness errors at 28 days from 4 to 45, while producing no change at one day.
- The authors argue that drift evaluations should vary TTL settings relative to each entity's rate of change, rather than treating the total drift-window length as the main independent variable.
Why it matters
Static retrieval benchmarks cannot determine whether an answer remains true as underlying data changes. ChurnBench provides a way to isolate temporal freshness failures and evaluate cache-refresh policies separately from retrieval and reasoning quality, which is directly relevant to agents operating over mutable enterprise systems.
Caveats
The abstract reports results from a synthetic four-source enterprise environment, so it remains unclear how well the observed relationship generalizes to real workloads with irregular updates, failed refreshes, delayed synchronization, or more complex cache dependencies. The reported cache-age comparison is also based on a small set of age settings and no observed TTL lapses under scheduled refresh.