The comprises seven temporal graph datasets with node and edge anomaly tasks, eleven detector architectures, and eight chronological deployment windows per dataset. In each window, a controller must choose a detector before any training occurs, using only historical aggregate context and delayed feedback from earlier windows. A sandboxed executor trains the selected detector on mature data and evaluates on a hidden window, releasing results after a one window delay. The benchmark measures detection utility via average precision and capture at fixed review depth, and characterizes adaptation through model switches and compute cost.
The authors validated the benchmark with two language model based controllers and a no memory reference on four datasets, plus three additional controllers on three datasets. The results show that the benchmark can distinguish useful, costly, and ineffective adaptation behaviors under delayed evidence, without relying on an unrealistic current window oracle. This is interesting because it simulates real world deployment where labels arrive late and controllers must make decisions under uncertainty, providing a more realistic evaluation than fixed pipeline benchmarks.