The authors created Intent-Eval, comprising 414 tasks and 3,312 evaluation instances across tool actions, code, databases and mathematics.
Models confuse rejected proposals with active instructions in dialogue
Intent-Eval shows that language models often act on requirements that users rejected or replaced, while decision-conditioned self-distillation reduces the resulting errors.
Chinese Tech
Junle Chen · Wei Chen · Zhengjun Huang · Zhoujin Tian · Yuxuan Liu · Kai Wang · +2 more
HKUST · Tencent Hy
Research Digest··2 min read
Chen et al.
Why this paper
From Tencent Hy and HKUST
In one line
Models treat mentioned but rejected changes as still active, degrading accuracy in multi-turn tasks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§