Code & software

Code generation, coding agents, software engineering with AI, program synthesis

77 articles · page 2 of 2

Code & software

Benchmark shows coding agents struggle to synthesize static-analysis checkers

The authors introduce CheckerBench, an executable benchmark of 300 real-world tasks for long-horizon static-analysis checker synthesis, and CheckerLab, a unified evaluation framework. Across 21 model-harness configurations, the mean Pass@1 is 32.30%, with the best configuration reaching 45.33%. The results indicate that reliable, reusable checker development remains difficult for current coding agents.

Code & software

Training coding agents to locate, vary, and verify boosts repair reliability.

The authors diagnose three teachable behaviors—location diversity, edit diversity, and verification—that determine the success of autonomous coding agents. They propose a method combining differential verification with two-stage training (weighted SFT and RL) that resolves 52.8% of SWE-bench Verified issues using 48.1% of the agent-steps of an eight-sample baseline, and raises greedy pass@1 from 31.9% to 43.0% on held-out issues.

Code & software

Training language models to ignore constraints that will be applied at inference improves efficiency

Jinwoo Kim proposes a constraint-aware (CA) training objective that externalizes prefix-definable analyses from the language model, allowing the model to ignore tokens that will later be filtered by a constrained decoder. The paper proves three theorems about reduced transformer width, depth, and data requirements when externalizing analyses, and validates these benefits through controlled synthetic experiments: CA-trained transformers consistently achieve lower loss than standard cross-entropy models, even when deployed under the same constrained decoding scheme.

Code & software

Program-graph evidence and blinded LLM surrogates detect code equivalence gaps

The authors present FEAgent, a system that assesses functional equivalence of code by combining typed program-graph analysis with differential surrogate execution. Instead of comparing programs directly, FEAgent uses a branch-aware generator to propose discriminative inputs and two blinded LLM surrogates to independently predict observable behavior. On EquiBench and SWE-bench Verified, execution-grounded adjudication confirmed that 18% of benchmark labels and 28% of unit-test-passing patches conceal behavioral divergences.

Code & software

Expert verification failures become reusable guidance for coding agents

The authors developed a verification kit that captures expert diagnosis of verification failures into reusable procedures, combined with executable language definitions in the K framework. On the HumanEval benchmark, a human-guided campaign achieved a perfect 164/164 success rate after two targeted repairs, as confirmed by final AI audit. Additionally, audits of 12 clean/defective packages successfully identified all defects while accepting all clean packages, demonstrating the kit's ability to detect problems that proofs alone miss.

Code & software

Training models on runtime program-state reasoning boosts automated software engineering

The authors introduce two complementary program-state reasoning tasks—buggy input-output reasoning and precondition-postcondition reasoning—and incorporate them into a staged post-training pipeline to build Comet-9B, a 9B-parameter language model. They find that adding both tasks to supervised fine-tuning on issue resolution improves success rates by 7.25 percentage points on SWE-bench Pro and 9.70 points on SWT-Bench Verified, with further gains from sequential reinforcement learning. Despite its small size, Comet-9B achieves scores comparable to reported results from GPT-5.2 and GPT-4o-based agents.

Code & software

New benchmark tests AI's ability to spot invalid code reviews

The authors introduce CRJudgeBench, a benchmark of 1199 instances built from real pull requests and expert-verified perturbations, assessing whether AI agents can distinguish technically valid code review comments from plausible but invalid ones. They also present Sentinel, a repository-grounded agentic judge trained via iterative action-level learning, which achieves 76.60% accuracy on the test set, substantially outperforming base models and showing that general-purpose LLMs struggle with this task.

Code & software

Code-linked evidence lets AI software claims expire when dependencies change

Tiwari, Vass and Singh present Assay, a Python system that combines repository indexing with an evidence ledger for AI-assisted software delivery. Each claim about code is tied to a Merkle hash of the relevant dependency cone, meaning the module and everything it depends on, so changes invalidate precisely the claims that may no longer hold. Across five public repositories, this approach reduced unnecessary re-verification while a model-free merge gate blocked all nine scripted adversarial behaviors.