This paper identifies and measures "schema bias" in LLM agents: the phenomenon where agent performance varies significantly depending on how the same set of executable actions is represented as tool schemas. The authors introduce a transformation framework with nine operators that create different but functionally equivalent schema variants, such as merging tools, splitting them, or renaming functions. They evaluate 11 LLMs on up to 32 schema variants and find that success rates range from complete failure to 97% depending solely on the schema representation.
Key findings include that this bias is substantial even for the newest models, and that the most difficult schema variant differs across model families. Training only reduces bias for schema variants that appear in the training data, and reliably predicting schema difficulty requires running a small sample of target queries rather than using offline metrics.