The authors performed a retrospective evaluation of a plugin-upgrade skill designed for coding agents.
Agent skill for plugin migration shows reward gains but hides grading errors
A retrospective study of a shipped plugin-upgrade skill found that while mean reward increased from 93.83 to 98.75, detailed contract inspection revealed grading errors including a containment predicate that accepted the parent directory.
Chinese Tech
Tsinghua University · Fudan University · PetroChina Southwest Oil & Gasfield Company · Jilin University · The Frederick Gunn School
Research Digest··3 min read
The authors studied 64 reports on 16 static migration tasks, with two attempts per condition and 328 criterion decisions, evaluating a shipped plugin-upgrade skill.
Why this paper
From University of Chinese Academy of Sciences and 18 others
In one line
Version-specific agent skills raised plugin-migration scores, but concentrated gains and grading errors made aggregate reward unreliable evidence of contract-correct advice.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§