Coding assistants that write code for a project first gather relevant context from the codebase, like similar functions or imports. This context helps the model produce working code, but it also determines whether the model sees the project's security safeguards or outdated insecure examples.
The researchers created a benchmark of 145 real-world security tasks, each with a functional test and a verified exploit. They tested how different ways of gathering context affected both the correctness and security of the generated code.
They found that context tuned for function raised correct code by up to 8.2 percentage points but secure code by at most 3.7 points. Removing a safeguard hurt security by up to 8.1 points, while adding one insecure example increased exploitable code by 6.3 points. saw the relevant safeguard in only about a third of attempts.