The paper investigates how local serving stacks (Ollama, llama.cpp, vLLM, SGLang) affect measurement of tool-call success in . Using a harness that sends the 'tools=' request, they found that different stacks handle requests differently: some reject models (Phi-3, Gemma-3) before inference, others return text or native calls. The harness does not preserve rejection as structured metadata, causing downstream analysis to misclassify it as a model non-call, reporting 0% fidelity for capable models.
Adding a text tool list while keeping the native channel recovers most of the measured fidelity for accepted models, but a uniform text protocol reduces fidelity for models with native tool-call support (e.g., Llama-3.2). Cross-stack probes show the same weights handled differently by each stack. Constrained decoding removes parse failures but induces non-termination, and turn-pooled versus per-instance estimates can differ by up to 55 points.
This is significant because it reveals a validity threat: standard evaluations may measure the serving stack's behavior rather than the model's inherent capability. The paper provides a checklist for incorporating serving behavior into evaluation protocols.