The authors designed a three-stage pipeline: Resource Preparation curates a domain-balanced set of agent skills and retrieves relevant artifacts and user queries from the web; Skill-driven Task Synthesis uses skills to partition artifacts into editable targets and reference contexts, then composes naturalistic instructions and packages each scenario into a sandboxed environment; Holistic Evaluation defines fine-grained completion rubrics (checking final deliverables) and process rubrics (checking execution trajectories), with both strict (all constraints met) and loose scoring.
Skill-based benchmark reveals stark agent reliability gaps hidden by lenient scoring
SkillForgeBench tests 8 LLM agents on 842 sandboxed tasks across 5 domains, finding that strict constraint satisfaction drops to 12.6–40.9% while loose pass rates exceed 70%.
Chinese Tech
Zhengxi Li · Junyi Zhou · Qiyuan Zhang · Chuhan Wu · Fuyuan Lyu · Le Tian · +3 more
MBZUAI · City University of Hong Kong · WeChat AI, Tencent · McGill – Mila & Quebec AI Institute
Research Digest··3 min read
The authors introduce SkillForgeBench, a benchmark that uses human-authored agent skills as procedural blueprints to automatically generate 842 sandboxed tasks spanning five professional domains.
Why this paper
From WeChat AI, Tencent and 3 others
In one line
AI agents achieve high completion rates but fail to meet all constraints, especially procedural ones, in realistic professional tasks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ·No stated limitations found
- ✓Reports numbers on named benchmarks
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§