Standard fidelity checks for agentic document extraction only compare extracted values to the source document. This paper shows that a model can pass such checks without ever opening the document, by fabricating text. The authors built a benchmark of 37 hand-curated claims from electronic component datasheets and logged every tool call during extraction. They developed a silent-failure detector that checks only which tools were called, not the extracted values. The detector produced zero false positives on 207 clean extractions and recovered all 50 planted faults that withheld exactly the tools it checks. This work is interesting because it reveals a critical blind spot in standard evaluation and provides a cheap, observable signal to catch fabricated answers.
Tool-call traces expose extraction failures that source fidelity misses
Logging agent dispatches revealed cases where models produced apparently faithful answers without consulting the datasheet.
Industry
Qing Ye · Meng-Hsuan Lin
Infineon Technologies AG
Research Digest··2 min read
Thread:Agent Harness Optimization
In one line
Agentic extraction requires dispatch-level instrumentation to detect silent failures that fidelity checks miss.
Before you read this
Two ideas this paper builds onRead it as
Ye and Lin instrumented an agentic datasheet-extraction benchmark to record every tool call, then built rules for attributing failures and detecting suspicious runs from tool use alone.
The authors evaluated three deployed model stacks on 37 hand-curated datasheet claims spanning four electronic components.
The Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.