When AI systems answer questions with citations, they often point to sources they didn't actually use, they relied on their own memory instead. This misleads readers into thinking the evidence backs up the claim when it doesn't.
To measure this, researchers replaced a fact in the provided passages with a false version and re-asked the question. If the model repeated the original fact, it had used its memory; if it adopted the false version, it had used the passage. They tested this across different model sizes and families.
Across models, from 1.2% to 15.7% of citable claims came from memory, not the cited source. Standard checks could not detect this failure, and no current detection method is accurate enough to flag it in practice. The problem remains an open challenge.