An agent powered by a large language model can be improved by changing its harness: the prompts, tools, and control rules around the model. But judging each change immediately by the final quality-versus-running-cost preference can discard a seemingly costly change that enables later gains; repeated runs can also produce different results.
ParetoSaddler keeps a change when repeated tests suggest it credibly improves either answer quality or cost, rather than demanding it already meet the final trade-off. It uses a Bayesian bootstrap, a resampling method that checks whether an apparent gain is dependable, then applies the deployment preference only at the end.
Across both benchmarks, quality improved while cost fell. On ClawMark, its quality-focused choice beat quality-only optimization by 7.49 percentage points and cost less. This suggests preserving promising intermediate changes can lead to a stronger, cheaper final agent.