A shipped plugin-upgrade skill was evaluated retrospectively using 64 reports across 16 static migration tasks. Mean recorded reward increased from 93.83 to 98.75, but the improvement was concentrated in one task, while eight task pairs already achieved ceiling scores.
Contract-level review found grading mistakes favoring both conditions. One fully credited containment check allowed access to the parent directory, a defect confirmed by an executable probe. Alternative model judges produced different gain estimates, showing that high aggregate scores can conceal contract violations and depend materially on the grader.