What they did

The authors evaluated three deployed model stacks on 37 hand-curated datasheet claims spanning four electronic components. Alongside conventional fidelity scoring—whether an extracted value matches the document—they recorded each agent’s tool dispatches and used those traces to build a rule-based failure-attribution classifier and a two-rule silent-failure detector.

They also constructed a separate physical-measurement “causal chamber” to test whether datasheet claims held in practice. The apparatus could directly evaluate only 2 of the 37 claims; a controlled perturbation was used to compare its verdict with document-level fidelity.

Key findings

• The trace-based detector raised no flags on 207 clean extractions that passed fidelity checks across three model families.
• It detected all 50 planted faults that deliberately withheld the tools checked by its rules. This recall is guaranteed by the fault design and does not establish performance on unrelated failures.
• A model could pass fidelity without opening the datasheet after a structured-output constraint silently disabled tool use; the per-tool trace exposed the failure and fabricated source text.
• Tool use improved portability and observability rather than extraction accuracy, and its overhead became worthwhile only when documents exceeded the model’s context window. One model stack’s instability came from its serving configuration rather than an identified model capability gap.

Why it matters

The study shows that correct-looking outputs are insufficient evidence that a document agent followed a valid retrieval process. Dispatch-level traces can provide operational checks for silent configuration failures and unsupported answers, complementing output-based evaluation without requiring another model to judge the extracted value.

Caveats

The benchmark contains only 37 claims, and the detector’s perfect recovery applies specifically to 50 constructed faults that omit the tools its rules inspect. Detection power is unmeasured for agents that call the expected tools but misuse their outputs or answer incorrectly. The physical oracle covers only 2 claims, so it demonstrates a narrow verifiable envelope rather than broad real-world validation.