The authors built AgentCIScope with an automated pipeline that turns public and synthetic privacy cases into executable tasks.
Agent benchmark shows LLM agents frequently violate contextual integrity when handling user data.
Across 936 tasks in 11 applications, single-run leakage ranged from 32.5% to 94.4% and increased with repeated execution.
Big Tech
Chengquan Guo · Zhaorun Chen · Zinan Lin · Xiaodong Liu · Bo Li
University of Chicago · Microsoft Research · University of Illinois Urbana-Champaign
Research Digest··2 min read
The authors introduce AgentCIScope, a benchmark that converts privacy cases into executable multi-step tasks in realistic application environments.
Why this paper
From Microsoft Research and 2 others
In one line
LLM agents leak sensitive user information across applications at high rates, with leakage varying by attribute, environment, and privacy policy.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§