AI models answer financial facts they claim not to know

Financial backtests check whether a model can predict past events, but large language models (LLMs, AI models trained on vast text) may have already seen those outcomes in training. Practitioners try to block this leak by trusting the vendor's stated knowledge cutoff, asking the model itself, or telling it to pretend it's in the past. This paper tests those three safeguards. The authors created FinDendro, a benchmark of 2,226 dated U.S. financial facts from 1999 to 2026. For each fact, they asked frontier LLMs (the most capable models) to answer, to say whether they knew, and to answer with a past date. Results show the safeguards are unreliable. GPT-5 answers 76% of facts correctly but claims to know only 38%. It still answers 75% of the facts it says it does not know. A past-date instruction made knowledge claims drop to 1% but accuracy did not change. Stated cutoffs also misalign with actual knowledge. This matters because backtests using these safeguards can wrongly appear successful, making financial decisions based on leaked information rather than true prediction. The paper provides a procedure to measure a model's true knowledge boundary and restrict backtests to periods after that boundary.