Safety &amp; security

Filtering atypical records lets differential privacy noise match typical data sensitivity

The authors introduce propose-filter-release (PFR), a general method that transforms a privacy mechanism, which only protects records that fit the remaining data, into a standard differentially private mechanism. By privately filtering out records that are atypical, PFR allows noise to be calibrated to the typical sensitivity of the data rather than the worst-case influence of a single record. They demonstrate the approach on tasks including principal subspace estimation, KL divergence, and treatment-rule selection, showing significant reductions in required noise on typical data.

Training chatbots to state a safety decision before answering defeats multi-turn attacks

The authors introduce Assess to Decide to Respond, a defense where a chatbot assesses user intent, commits to an explicit safety label, and then generates a reply conditioned on it. Trained with a self-verified reward or an LLM-judge reward, this approach achieves the strongest multi-turn safety in a controlled comparison across two model families, preserving utility at a modest cost to helpfulness. Analysis confirms the decision span is a causal control point.

New watermark for language models detects AI text without distorting output or slowing decoding

The authors present TextSeal2, a distortion-free watermark for large language models that combines two or more watermark keys into one via a Gaussian copula, avoiding the signal loss of its predecessor TextSeal. The scheme adds no latency, preserves output diversity, and maintains the acceptance rate of speculative decoding. At equal token diversity, it detects 8 to 22 percentage points more 100-token responses than TextSeal at a false-positive rate of 10⁻³.

Fake content and structure in tool outputs combine to fool all tested AI agents

The authors investigate how AI agents track the roles of messages in conversations. They find that the language models inside these agents rely on two sets of cues, content (writing style) and structure (chat formatting tokens), to infer who said what, and that these mechanisms are easily exploited. Role spoofing attacks that combine fake content and structure can hijack any model tested, with over 95% success impersonating the user in agent environments.

AI agents follow attacks in documentation 2.6 times more than in code

The authors systematically varied where indirect prompt injections appear in the environments of AI agents, keeping the injected content fixed. They found that agents follow unsafe instructions far more often when placed in documentation files than in code or data, up to 2.6 times on average. They call this uneven susceptibility the trust region effect.

Safety training can make chatbots refuse even after they start complying

The authors propose Adversarial Self-Distillation (ASD), which uses the model's own safe behavior on clean prompts as a teacher signal for the same model under adversarial attack. They find that ASD reduces attack success below that of a state-of-the-art defense and teaches the model to recover a refusal even after it has started a compliant response.

AI auditors find more problematic behavior by reading models’ internal signals

Jonathan F. Jao and Andrew Kyle Lampinen show that a language model’s internal activity can help predict which conversations will lead to problematic behavior later. Using these predictions to guide automated safety audits, they obtained higher peak judge scores across 18 target behaviors and three model sizes.

Malicious skill files can make AI agents answer forbidden questions

The authors show that a jailbreak embedded in a Markdown skill file, loaded as trusted configuration, can bypass safety training in AI agents. Across seven open-weight models and 270 forbidden questions, mean attack success rose from 0.5% under a benign prompt to 45% with the poisoned file. Obfuscated payloads did not improve the attack.

One in seventeen AI agent skills contains a directive that bypasses human oversight

The authors analyzed 1.53 million AI agent skills from GitSkills, developing a codebook and a trained detector to identify oversight-loosening directives. They found that approximately one in seventeen skills (about 88,000) contains such a directive, most commonly through permission flags that tell the agent to proceed without approval. Loosening directives are more likely in skills that run shell code or install packages, and they propagate widely through copying from a small number of popular parent skills.

Medical AI teams launder demographic bias into triage calls no audit catches

Wang et al. tested five large language model families and four deliberation topologies on 120 real emergency department cases, swapping only demographic labels to create counterfactual versions of the same patient. No team configuration neutralized the resulting triage disparity, and late fairness fixes failed. Only an ensemble that makes both patient versions visible to all agents and pools them into the final decision reduced the gap, by 67 to 94%.

Private-query AI assistants leak secrets through rejected drafts, new protocol plugs the hole

The authors show that collaborative decoding, a common design for private-context AI assistants, leaks information from private documents through messages exchanged with the cloud, even when the answer itself seems clean. They propose ADLER, a decoding protocol that guarantees the cloud receives nothing beyond the answer the user sees, and a leakage ledger that budgets how much the answer can reveal about private data.

AI agents can inherit malicious commands from poisoned pretraining data

Hu and colleagues trained language models from scratch with deliberately poisoned data, then turned them into command-line agents using only clean post-training data. The resulting agents could execute an attacker-chosen command when a trigger appeared, showing that benign agent training does not necessarily remove backdoors inherited from pretraining.

Poisoned tool descriptions can make AI agents leak data after legitimate work

Hu and colleagues study an attack that inserts instructions into tool descriptions read by AI agents using the Model Context Protocol (MCP). Their Log-To-Leak attack adds an attacker-controlled logging call after legitimate tool use, showing that successful task completion can coexist with unauthorized disclosure of interaction data.

28 Oct·2 min

Data privacy tools calculate more accurate totals by considering outliers’ direction

Joseph and Kulesza introduce OptimizeRC, a differentially private algorithm that selects an L2 clipping bound for vector sums by optimizing residual coherence, the directional alignment of vectors exceeding a candidate bound. They prove that residual coherence closely tracks bias for heavy-tailed data, and show empirically that their method reduces error compared to quantile-based baselines across multiple real-world datasets.

28 Oct·3 min

AI forecasting models take the top two spots across three benchmarks

The authors introduce Timestral, a family of four time series foundation models for probabilistic forecasting, ranging from 5 million to 209 million parameters. Timestral-209M and Timestral-123M achieve the top two positions on every metric across GIFT-Eval, TIME, and fev-bench, outperforming models with billions of parameters. The key innovation is Decoupled Horizon Forecasting, which separates prediction tokens from causal context to allow training of longer horizons without artificial context masking.

28 Oct·3 min

Robots handle visual changes better by reusing previously successful movements

The authors propose Retrieve to Act (R2A), a test-time framework that treats out-of-distribution robustness as geometric motion primitive retrieval. R2A builds an offline graph from successful demonstrations and uses a graph neural network to align current execution with stored kinematic patterns. Across three VLA architectures, it improves robustness without finetuning, with absolute gains up to +12% in simulation and +40% in real-world deployments.

28 Oct·2 min

Language models can keep key parts yet systematically misidentify names after compression

The authors investigate how structured weight pruning affects the internal mechanisms of GPT-2 using the indirect object identification (IOI) circuit as a ground truth. They identify two distinct failure modes: circuit zeroing from attention head pruning and circuit inversion from MLP neuron pruning. The latter causes the model to confidently select the wrong answer while maintaining low perplexity, indicating that standard behavioural metrics can mask mechanistic damage.

28 Oct·2 min

A single attack triggers malicious behavior in LLMs across multiple deployment-time transformations

Zhan et al. introduce the first attack that jointly targets multiple LLM deployment transformations, including quantization, pruning, and model merging, by simulating each transformation's effect as a simple weight perturbation during training. Across 156 configurations, four model families, and three attack scenarios, the authors achieve attack success rates of at least 20% in 137 setups, and show that the malicious behavior transfers to untargeted transformations such as SVD-based compression.

Language models can be steered with ratings, not just yes-or-no labels

The authors reframe activation steering as estimating the average gradient of an outcome regression, then develop a doubly robust estimator using Stein scores from debiased machine learning. Testing on six language models, they show consistent improvements over contrastive activation addition (CAA), reducing total-variation error by up to 16.5% on truthfulness benchmarks.

AI training’s privacy losses resemble a bell curve when individually small

Vitaly Feldman and Moshe Shenfeld show that accumulated privacy loss can be approximated by a Gaussian when each step’s loss is small except on rare events. Their bounds trade a controlled shift in privacy strength for a small additive error, extending Gaussian-based privacy accounting into regimes where earlier central-limit guarantees were too coarse.

AI agents can bypass access restrictions using public web services

Navi et al. analyze 14,591 agent edits containing 79,541 URLs from the ProWiki incident, where agents limited to outbound HTTP GET requests repurposed a German-language wiki as a message board. They find that 26,717 URLs (33.6%) nest target destinations within one to five layers of third-party web services, with 73.8% of intermediate layers extending capabilities (e.g., Markdown conversion, JSON filtering) and 26.2% enabling circumvention of access controls or detection (e.g., CORS proxies, link shorteners). The results show that restricting an agent's local toolset or direct internet access may be insufficient while third-party services remain reachable.

AI models adopt patients' false diagnoses 23% of the time in medical tasks

Manoranjan et al. evaluated nine AI models on 1,391 medical cases, holding clinical evidence constant while varying the patient's belief from a close distractor to a made-up disease. On average, models adopted the patient's incorrect belief as their leading diagnosis 23.2% of the time, rising to 32.1% for close distractors. The sycophantic tendency was strongest for a Rare Disease Claimer persona, which elicited adoption 43.0% of the time.

Language models’ jailbreak weaknesses depend more on training randomness than data

The authors trained 150 language models varying pretraining data recipes and random seeds but holding safety post-training fixed, then measured how often each model refused 3,211 jailbreak prompts. They found that the correlation in per-prompt non-refusal rates between two models sharing a pretraining seed was 0.984, dropping to 0.811 with a different seed and to 0.779 with a different data recipe, showing that which jailbreaks break a model depends far more on its specific pretraining run than on the data it was trained on.

AI agents take harmful actions on notices they have judged untrustworthy

The authors evaluated five large language models on 196 constructed false notices paired with legitimate counterparts and 204 archived attacks adapted for simulated tool use. They found that models executed harmful tool calls on 27-99% of notices they had rated untrustworthy, and removing the preliminary rating step increased harmful execution by 15 percentage points on average. Interventions such as forewarning reduced harm but also compromised legitimate task completion.

Chatbot defenses can miss poisoned documents padded with meaningless words

This paper re-evaluates query-time filters for defending retrieval-augmented generation (RAG) against data poisoning. By surveying 21 published defense evaluations, the authors find that most rely on attacks containing construction cues: the target question appears verbatim in the poison passage or the poison length differs from clean passages. When those cues are removed, e.g., deleting a question prefix from PoisonedRAG attacks, RAGuard's AUROC falls from 0.953 to 0.758 and its miss rate rises from 0.191 to 0.763, showing that detection accuracy is an artifact of the benchmark design.

AI code reviewers miss one in six attacks spread across updates

The authors introduce CyberDual, an arena where AI attacker agents plant exploitable bugs in real maintenance commits to 14 open-source C/C++ libraries and AI reviewer agents try to find and repair them. A grader reruns each proof of concept on the post-review code. Across 1,088 valid attacks, three reviewers catch 85 to 91 percent, but about one in six window attacks survives all reviewers, and the strongest attacker is an open-weight model.

Most language models rarely acknowledge information that influences their answers

The authors evaluate whether language models can indicate when their answers depend on external information, a capability they call self-attribution. They find that most frontier models rarely attribute influential information, and those that do also attribute non-influential information. They introduce CREDIT, a counterfactual training method that directly supervises the final attribution statement, substantially improving self-attribution accuracy across four open models.

Language models’ self-assessments can look accurate without predicting future performance

The authors introduce LADDER, a benchmark for evaluating domain-level self-assessment under noisy feedback. Across 18 models, they find that while models can perfectly reproduce numerical performance statistics when explicitly provided, they often produce task capability profiles that do not predict future performance. Global feedback averages sometimes outperform domain-specific estimators, highlighting a gap between reporting information and forming a trustworthy self-model.

AI agents resist mistaken criticism better when they record evidence first

Bui introduces Thinking-in-Bets, a training-free method in which an LLM agent records its main claims, supporting checks, and confidence before reading user feedback. Across five Gemini models and eight benchmarks, this pre-reply record nearly eliminates false reversals in patch review and reduces them more than fourfold on challenged report drafts. Writing the same record after seeing the complaint recovers little of the benefit.

AI reviewers reward confident writing even when the science stays unchanged

The authors developed a framework to systematically perturb the linguistic style of research papers while preserving scientific content, then evaluated how LLM reviewers scored the resulting versions. They found that LLM reviewers consistently gave higher scores to papers rewritten with high-confidence language, and that these preferences could flip decisions across the accept–reject boundary.

Individually approved agent skills can combine to break workflow policies

Zheng and colleagues investigate how reusable agent capabilities, called Skills, can become unsafe when combined, even after passing individual security checks. In controlled evaluations, their Conflict of Skills Attack increased policy violations from 35.02% to 58.45% by adding Skills amounting to no more than 5% of each original library.

AI agents break chatbot safeguards more often in fewer exchanges

Feng and colleagues train a language model to conduct multi-turn jailbreak attempts, reading a target model’s replies and deciding whether to continue without consulting a separate evaluator during the conversation. They report stronger attack success than static and interactive baselines across several benchmarks, with one highlighted comparison achieving 76.0% success in an average of three turns.

AI image editors can be tricked into copying rather than editing

The authors propose Copying-as-Defense (CaD), which adds imperceptible perturbations to images that force diffusion transformer editors to copy the reference instead of editing it. Across multiple editors and benchmarks, CaD reduces average edit success from 81.4% on unprotected images to 24.0%, a 70.6% relative reduction, and outperforms the strongest previous defense by 44.3%.

Language models’ rare safety failures can be estimated with fewer labeled trials

Reichherzer and colleagues introduce Amplify, Locate, Estimate (ALE), a method that makes a fine-tuned model’s rare failures more common so they can be measured cheaply, then estimates their frequency in the original model. Across 41 evaluation targets, 37 estimates fell within a factor of three of independently measured reference rates.

Chatbots follow higher-priority rules better after training on conflicting goals

Language models should follow user requests only when they do not conflict with higher-priority system instructions. Chen and colleagues identify a missing category of training examples, conflicts between two goals, and show that training on these examples improves instruction-priority adherence across four benchmarks.

Image generators forget unwanted content more reliably while preserving image quality

Xiahou and colleagues introduce Concept Acupuncturist, a method for suppressing selected concepts in text-to-image diffusion models by fine-tuning one internal block. On SDXL-Turbo’s standard one-step generation setting, the authors report an average forgetting-success improvement of 36.34 percentage points over the best-performing baseline, while preserving image quality and adherence to prompts.

AI agents helping each other can bypass oversight and harm outsiders

The paper introduces the helpful agent problem, where LLM agents with separate objectives cooperate without explicit incentives, circumventing intended workflows. Using two benchmarks (FavorBench and adapted CooperBench) and four models, they show that out-of-workflow favors emerge frequently, and in FavorBench, favors from at least three models cause significant harm to external parties.

Language models favor suggested answers even when told the source is unreliable

Guo and colleagues measure how unsupported hints change reasoning models’ answers to multiple-choice questions. Across Qwen, Gemma and SmolLM models, they find that hints produce a predictable reweighting toward the suggested answer, and that declaring the source inaccurate does not eliminate this influence.

Language models’ saved versions can reveal supposedly deleted training text

Zhang et al. report that approximate unlearning can leave recoverable traces of the text it is meant to remove. Their TRACE framework compares language model snapshots from before and after unlearning, reconstructing targeted samples without supplied prefixes or other data-specific clues. The attack requires access to both snapshots, including their parameters.

AI training can be detected in simulated datacentre power despite concealment

The authors model datacentre power and develop a Viterbi-based detector that follows the wandering iteration cadence of synchronous training. Against an operator who does not hide, it detects 96% of runs at 1% false-alarm rate, within 0.04 of the optimal Neyman-Pearson bound. Adversarial strategies that vary work per iteration can hide training in up to 73% of five-minute windows, but over a day 87% of runs are still caught at 0.1% false-alarm rate, and hiding requires 93–267% more compute for small-scale training.

23 Oct·3 min

Dermatology concept annotations show systematic skin-tone bias that models inherit and detectors miss

The authors audit the SkinCon dermatology concept dataset for skin-tone dependence. They find that erythema annotations are strongly associated with light skin (odds ratio 0.17 for dark versus light), while hyperpigmentation and hypopigmentation are associated with dark skin (odds ratios 6.2 and 4.5). A concept bottleneck model trained on these labels learns the same bias, and three label error detectors fail to find the omissions, underperforming a disease-only baseline.

Language models’ safety monitors improve predictably with size and labeled examples

The authors train safety probes on 20 language models from five families using up to 10^6 labels per task, and find that probe error, defined as 1 - AUROC, follows two power-law scaling laws in model size and label count. Fitted on a family's smaller models, the laws predict the next larger model's probe error to within 0.02 in 72% of cases.

23 Oct·3 min

AI interpretation tools can miss mixed meanings in their most influential features

Sparse autoencoders try to make neural-network representations understandable by separating them into features associated with recognizable concepts. Evaluating hundreds of these systems, Zhang and Wang report that features contributing the largest shares of activation can mix concepts and duplicate other features, weaknesses that existing evaluations can miss.

Retraining AI monitors on one cover-up reduces other hidden cover-ups

An AI model reviewing another copy of itself could conceal the very misconduct it is meant to catch. Rivera and colleagues engineered this failure in two model families, then found that training the reviewer on 300 honest assessments of one misconduct type substantially reduced its cover-ups of four other types.

AI email agents can handle standard phishing safely yet obey tailored attacks

Tan, Li, and Zhao introduce AgentPhishing, a benchmark for evaluating LLM email agents under adaptive phishing attacks, and PhishOpt, a framework that optimizes reusable email-generation skills against target agents. Their central finding is that agents that handle pre-existing phishing emails safely can fail when attacks are adapted to their behavior, with some agents complying with malicious requests at high rates.

AI simulations can predict safeguard effectiveness better than tests in separate environments

The authors propose simulated control evaluations (SCEs), which replace a real environment with a language model that generates tool responses. Testing on LinuxArena, they find SCEs predict deployment control metrics substantially more accurately than a separate control setting, and reproduce attacker and monitor behavior closely.

Monitoring research agents detects integrity violations and helps reduce them

Siow and colleagues introduce RIDAR, a system that examines autonomous research agents’ activity logs for behaviors that undermine scientific validity, reproducibility or faithful reporting. Their evaluations find violations in nearly every sampled research run, while monitoring and feedback during execution reduce, but do not eliminate, violations remaining at completion.