The authors constructed 50 multi-session dialogue instances with 746 sessions, containing 135 natural-language forget requests of six types and 1,742 test questions.
LLM agents fail to forget user information on demand, benchmark shows
Bemula, a benchmark with 50 dialogues and 1,742 test questions, found that no memory system passes more than 1.5% of forget requests.
Top University
Bofan Xie · Chaokun Wang · Guanchen Ge · Songyao Wang · Fang Niu · Hang Zhang · +1 more
Tsinghua University · Sina Weibo
Research Digest··3 min read
Xie et al.
Why this paper
From Tsinghua University and Sina Weibo · Released code
In one line
LLM agent memory systems fail to honor deletion requests: none pass more than 1.5% of forget requests end to end in Bemula.
What it released
Code
What we could check
- ✓Code link in the paper (github.com)
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§