The authors constructed SCCGBench by extracting RapidAPI call functions from real GitHub repositories, then filtering out template or auto-generated code and applying decontamination via redacted-intent generation (masking hosts, endpoints, parameter keys before generating natural-language intents).
Executability-oriented benchmark exposes LLM service-call grounding failures
The authors show that models produce syntactically correct but non-executable HTTP requests, with Mock success rates below 3.8% in intent-only settings, while gold documentation raises scores to about 90%.
Academic
Shuo Wang · Guodong Fan · Ming Zhu · Jing Li · Chun Wang
Shandong University of Technology · Shandong Agriculture and Engineering University · Concordia University
Research Digest··3 min read
Wang et al.
§