The evaluated model, called the generator, repeatedly poses multiple-choice questions and assigns probabilities to the possible answers.
Priced guidance turns research idea generation into a compression test
The framework measures how many bits of targeted guidance a language model needs to reconstruct the central idea of a future paper.
Top University
Kaiyue Wen · Tengyu Ma · Percy Liang
Stanford University
Research Digest··3 min read
Wen, Ma and Liang evaluate research ideation by asking models to recover the essence of 87 recent deep-learning papers while charging for every hint.
Why this paper
From Stanford University
In one line
A model guided by K bits of information can generate a future research idea without guidance with probability at least 2^{-K}.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors (2 noted)
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§