The authors developed an automated pipeline that uses a paper's arXiv ID to produce draft annotations, which authors then verified.
Benchmark reveals AI systems struggle to retrieve papers that inspire new research
ScholarCatalyst collects author-identified catalyst papers and finds current retrieval systems recover less than half of them.
Top University
Sohyeon Kim · Yoonho Lee · Bo Liu · Dayoon Ko · Rulin Shao · Seungone Kim · +8 more
Stanford University · Seoul National University · University of Washington · Carnegie Mellon University · Allen Institute for AI
Research Digest··2 min read
Sohyeon Kim and colleagues built ScholarCatalyst, a benchmark where 184 lead authors of 207 computer science papers labeled which prior papers could have advanced their projects.
Why this paper
From Allen Institute for AI and 5 others
In one line
Current retrieval systems recover at most 48% of catalyst papers that authors credit.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§