The authors constructed 9,045 questions requiring list-formatted answers of 3-10 elements, spanning eight categories and covering factual knowledge from over 60 countries.
New benchmark reveals LLMs' limitations in retrieving multiple facts from memory
ListQA evaluates models on composing structured lists of facts, with the best model scoring under 37% and strong improvement when facts are provided
Big Tech
Mayank Kulkarni · Rakesh Vaideeswaran
Amazon AGI
Research Digest··2 min read
The authors introduce ListQA, a benchmark of 9,045 human-curated questions that require large language models to recall multiple facts and present them as structured lists.
Why this paper
From Amazon AGI
In one line
Multi-element factual retrieval is harder than single-answer QA; the best LLM scores below 37% F1 on the ListQA benchmark.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ·No benchmark numbers found
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§