Study A compared idealized and stressed trade execution across three synthetic settings sharing a 24-day upward market phase.
Fixed controls expose limits in financial LLM agent evaluation
Holding model responses constant isolated execution effects, while matched audit tasks showed that target recall can conceal severe false-positive behavior.
Industry
Weicheng Xue
Virginia Tech
Research Digest··2 min read
Xue tests whether common performance and auditing scores support the conclusions drawn from them in a financial agent harness.
Why this paper
From Virginia Tech
In one line
Fixed-response replay and matched defect controls show execution comparisons confound model behavior, while target recall alone misrepresents audit quality.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§