Forecast-Dojo turns resolved Polymarket questions into replayable forecasting episodes. Each event comes with dated news so an agent can search and read only information available at a given historical date, then revisit and update its forecast as the event progresses. The environment supports both benchmarking and training because past dates can be reset and rerun, and outcomes are known for scoring without waiting. The dataset includes 1,568 events and 18.8M news articles, split by time into training and evaluation periods.
Across 12 models, giving agents research tools lowered Brier score for all 12 compared with forecasting without tools. Forecasts with tools also improved as events unfolded, with the largest gains at steps where more newly dated evidence appeared, while forecasts without tools stayed flat. A belief notebook carried between dates cut research cost but did not consistently improve forecast quality. Every model still trailed historical market forecasts on both Brier score and accuracy.
This matters because current live forecasting run on wall-clock time and historical benchmarks typically score only a single snapshot. Forecast-Dojo's replayability lets models be compared under identical conditions, scored immediately, and trained on past interactions, making it a reusable testbed for forecasting agents.