This paper introduces a to evaluate how well LLM agents investigate advanced persistent threats under different telemetry settings, such as full logs or endpoint-only logs. The benchmark includes 370 cases from 56 attack reconstructions with over 16 million log records. Eleven LLMs and four frameworks were tested. On average, agents acquired sufficient evidence for 44.3% of recoverable attack actions but only 25.0% of those actions received formal citations. When telemetry changed from full to endpoint-only, aggregate coverage dropped only 1.6 percentage points, yet 35.5% of previously supported actions lost their citation support despite being still recoverable. This instability persisted across all frameworks even when supporting records remained unchanged. These results show that aggregate metrics hide serious gaps in agent robustness, which is important for security operations centers operating under varied telemetry constraints.
Security agents lose evidence support when available telemetry changes
APTInvestBench shows that autonomous investigations can appear stable in aggregate while losing support for many individual attack actions.
Academic
Yu Wang · Shuhao Li · Tao Yin · Ziyang Li · Xueying Zhao · Peishuai Sun · +1 more
Zhongguancun Laboratory
Research Digest··2 min read
Thread:Agent Security & Attacks
In one line
When telemetry changes, LLM agents lose citation support for 35.5% of previously covered attack actions even though the actions remain recoverable.
Before you read this
Three ideas this paper builds onbenchmarkA standard set of tasks or data used to evaluate and compare the performance of AI systems.366 papers →agentic evaluationMeasuring AI agents' performance in complex tasks, highlighting failures in planning, monitoring, and real304 papers →evidence retrievalPulling specific, relevant information from a large corpus to support an answer, decision, or reasoning step.115 papers →
Read it as
The authors introduce APTInvestBench, a benchmark that tests whether LLM agents can investigate advanced persistent threats when the available security logs change.
The contains 370 investigation cases spanning seven SOC-inspired telemetry conditions, including full, endpoint-only and network-only views.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.