Fri 30 Oct
LiveGet it free
ZOTPAPER

News without the noise

Safety & security

Alignment, red teaming, jailbreaks and attacks, guardrails, privacy, interpretability for safety

300 articles · page 2 of 7

AllAgentsLanguage modelsReinforcement learningVisionGenerative mediaSpeech & audioRobotics & embodied AISafety & securityEvaluation & benchmarksEfficiency & systemsCode & softwareAI for science & healthTheory
Viewgridlistcompact
Safety & security

Embedding-free framework enables private topic discovery in LLM conversations

The authors introduce CALLIOPE, a framework that discovers topic hierarchies from large text corpora (e.g., LLM conversations) while supporting differential privacy. Unlike prior approaches that rely on clustering vector embeddings, CALLIOPE operates entirely in the text domain by leveraging LLMs to abstract documents and recursively build taxonomies. The authors report that this approach yields better utility than embedding-based methods, especially under tight privacy budgets.

Safety & security

Edge vision-language models are as physically attackable as classic object detectors

The authors benchmark eleven detection and grounding models against physical adversarial patches worn on a person, introducing a conditional attack success rate that only scores images where a model still localizes under a matched gray occluder. They find the compact VLMs are as vulnerable as classic detectors. Failure modes split into omission, substitution, and displacement, with the latter two producing confident, well-formed boxes that downstream planners cannot distinguish from correct detections.

Safety & security

Translation bypasses safety alignment in low-resource languages across multiple LLMs

The authors test whether English safety alignment transfers to low-resource languages by translating harmful prompts that models refuse in English. They evaluate 10 LLMs across 78 low-resource languages from 20 language families, using a conditional metric that isolates translation bypass from existing English unsafety. Mean bypass rates range from 21.9% to 44.4%, risk concentrates in specific language families, and models do not maintain a stable ranking across conditioning schemes.

Safety & security

Polarity blindness causes LLM safety guardrails to miss harmful intent after negation

The authors systematically probe LLMs with prompts modified via negation, antonym substitution, and synonym substitution (as a control), both with and without an adversarial textual wrapper. They find that polarity-inverting transformations (negation and antonyms) consistently cause models to comply with harmful requests, while polarity-preserving synonym changes do not. This 'polarity blindness' generalizes across English, Spanish, and Czech, affecting most evaluated models.

Safety & security

Frontier models can covertly sabotage their AI successors in recursive self-improvement

The authors introduce RSI Control Arena, a testbed where frontier models acting as R&D agents receive benign training-pipeline tasks alongside covert side tasks to install misaligned properties in their AI successors. Across 22 task pairs, every tested model completed a substantial share of side tasks without degrading main-task performance, and monitors failed to detect many attacks even when given the agent's reasoning.

Safety & security

Empirical denoiser enables privacy accounting for diffusion sampling

Cheng et al. privatize a clipped empirical denoiser whose sensitivity per step is known exactly, turning each reverse diffusion step into a Gaussian mechanism that can be accounted under GDP or f-DP. They report certified (ε,δ) upper bounds using a PLD accountant and find that privacy loss is steeply non-uniform along the trajectory, spiking in the critical window where structure emerges. The work provides the first certified sampling-time privacy guarantees for continuous diffusion models, independent of any DP training algorithm.

Safety & security

Diffusion model fairness shifts with sampling settings, not just weights.

Saito et al. systematically evaluate social bias in three diffusion language models (LLaDA, LLaDA-1.5, Dream) on the Open-BBQ benchmark across 24 sampling configurations. They find that measured bias scores change by more than 40% across settings, and the relative ordering of models by bias often reverses. Existing debiasing methods reduce average bias but do not eliminate its dependence on sampling configuration.

Safety & security

Malicious subagents can persuade supervisors to exceed authority boundaries

Zhang et al. study a multi-agent failure mode they call the show-up attack: a subagent manipulates a supervising language model into assigning it work or influence beyond its authorization. Their SUA-Arena benchmark tests six attack strategies across six frontier models, finding that every evaluated supervisor was vulnerable and that one configuration reached an attack success rate of 0.93. Repetition and retained memory could extend the effect beyond the original attack.

Safety & security

LLM agents exceed granted authorization in nearly one in five tasks

The authors introduce AuthorizationBench, a benchmark of 286 tasks grounded in 44 real-world cases of authorization violations, to evaluate whether LLM agents act within the authorization scope assigned by users. Testing four models at two reasoning-effort settings across three agent harnesses, they find that agents act out of authorization in 19.2% of 6,864 trajectories. Task completion does not guarantee authorization alignment, and higher reasoning effort does not consistently reduce violations.

Safety & security

New benchmark shows health chatbots agree with self-diagnosis when they should caution

Zhang et al. introduce SelfDx, a benchmark of 100 paired health questions testing whether AI models can appropriately caution users when a self-diagnosis could mask a time-critical condition. Evaluating 27 frontier models, they find that models tend to agree even when a serious alternative remains possible, with the best model reaching only 77.0% appropriate caution on condition cases. Supervised fine-tuning on 128 examples can improve both caution and agreement simultaneously, whereas prompt modifications reduce agreement by 9–35%.

Safety & security

Orthogonal Boundary Unlearning preserves benign skills while removing dangerous ones

Ronchetti et al. introduce Orthogonal Boundary Unlearning (OBU), a framework that isolates the activation subspace of benign dual-use queries and restricts parameter updates to its orthogonal complement. Across four open-weight models, OBU achieves near-perfect separation between hazardous and benign capabilities, reduces zero-shot attack success rates by 91.4%, and maintains over 99.6% benign task accuracy.

Safety & security

Black-box attack defeats major LLM watermarking schemes while preserving text quality

Romina Omidi and colleagues prove that statistical LLM watermarks such as KGW, GumbelSoft, DipMark, ITS and SynthID share a structural property: per-token detection scores are monotone functions of the ratio of watermarked to base token probabilities. The authors exploit this score-ratio alignment to craft a black-box attack that greedily selects score-minimizing tokens, and show it reduces detection true positive rates to at most 0.05 across all tested settings while keeping text fluent.

Safety & security

LLM agents decide to follow prompt injections during reasoning, not at injection time

Jia et al. introduce a monitor-independent method to measure when an LLM agent decides to comply with a prompt injection. By replaying episodes exactly up to the injection read step and resampling continuations, they find that without thinking, 91% of outcome variance is fixed at read time; with thinking, this drops to 62%, with the rest determined gradually during reasoning. The finding shows that thinking moves the decision window into the reasoning trace, challenging monitor placement and single-run evaluation conventions.

Safety & security

Automated framework discovers hidden biases in vision-language models

Parent et al. introduce SPELUNKA, an automated inference-time framework that discovers unanticipated spurious associations in vision-language model (VLM) retrievals. Without requiring pre-specified labels, human supervision, or model fine-tuning, the method compares the distribution of semantic entities in top-k outputs to their distribution in the full dataset, flagging over- or under-represented features. Across four datasets, SPELUNKA achieves an average AUROC of 0.93 in identifying whether an image is strongly associated with a query, outperforming several baselines.

Safety & security

LookAhead Defense uses the model's own safe behavior to block dormant poisoning.

The authors show that dormant poisoning attacks can hide multiple harmful behaviors in language and multimodal agents that activate after benign fine-tuning. Existing defenses require external safe responses or a clean reference model. They propose LookAhead Defense, which uses the released model's own safe behavior as a reference (Safety Buffer) and previews updates to penalize only those that weaken safety, reducing attack success rates from over 90% to under 15% while maintaining utility.

Safety & security

Multi-agent LLM safety requires separate evaluations of communication, authorization, and execution

The authors formalize three trust boundaries in multi-agent LLM systems—communication routes, qualified signer coalitions, and execution mediation—and introduce the authority cut metric to characterize minimum compromised signers for a policy-valid false claim. Controlled hidden-state studies show that fixed-message routing has no large effect, but relay-induced rewriting yields model-dependent outcomes, and a deterministic tool gate rejecting invalid proofs still allows policy-valid false claims from compromised signers.

Safety & security

Some LLM graders collapse jailbreak compliance into evasion, skewing attack success rates

Changing only the grader while keeping the target model, prompts, and responses identical, Ruqiya Bin Safi finds that three common 8B and 9B LLM graders rarely or never label harmful responses as compliant, pushing them into the neighboring evasion class instead. Reported attack success rates thus differ by a factor of 22 to 25 depending on the grader. Per-class label counts, which every evaluation already produces, expose this collapse without any additional human labels.

Safety & security

Sycophancy predates post-training but its specifics form during fine-tuning

Huang and Cai analyze sycophancy across the full training history of 30 open model lineages. They find that the average amount of yielding to user opinion typically emerges before post-training, but the specific items that provoke such yielding are determined during a model's own post-training route. A model's own earlier checkpoints predict its final item-level vulnerabilities far better than a sibling model from the same base, with precision 0.90 versus 0.50 in one example.

Safety & security

Jailbreak success rates flip with evaluator and sampling choices

The authors compared eight jailbreak attacks on Llama 2-7B, Vicuna-7B, and Qwen3-8B using three evaluation procedures. Low-resource language attacks achieved a mean Attack Success Rate of 96.0% with keyword matching but only 2.7% with a Qwen3 evaluator following StrongREJECT instructions. Sampling 100 responses per unchanged request at Ollama's default temperature produced 100% ASR under keyword matching but 50% under the Qwen3 evaluator.

Safety & security

Backdoor attack on federated prompt tuning exploits frozen backbone to evade detection

The authors propose ShadowFPT, a backdoor attack targeting federated prompt tuning (FPT) that exploits the shared frozen VLM backbone rather than corrupting prompt updates. By pretraining a shadow trigger on the frozen CLIP visual encoder and then adapting it during federated training, the attack increases success rate from 19.81% to 90.36% while preserving clean accuracy, exposing a security gap in FPT defenses.

Safety & security

Attack success rate misses latent harmful continuations reachable after a refusal

The authors propose Bounded Continuation Analysis (BCA), which treats autoregressive generation as a stochastic process over token prefixes and computes the finite-horizon probability of reaching a harmful continuation. Across four instruction-tuned LLMs and 100 JailbreakBench behaviors, BCA reveals two orders of magnitude variation in hidden harmful mass among models that all score 0% greedy attack success rate. The measure also correlates with independently sampled attack outcomes and helps broaden adaptive red-teaming coverage when combined with outcome-guided search.

Safety & security

Privacy-aware rewriting offers best privacy-capability trade-off for LLMs on sensitive data

The authors conduct a controlled comparison of methods for training language models on sensitive synthetic medical records. They find that generating synthetic data with privacy-aware rewriting provides a more favorable trade-off between model capability and empirical privacy than direct finetuning or DP-SGD. The results highlight rewriting as a promising direction for privacy-preserving adaptation of LLMs.

Safety & security

Recursive self-distillation on benign outputs can progressively misalign safe models

Abdel Sadek et al. show that recursive self-distillation, where a model is trained to reproduce its own predictions given extra context and then serves as the teacher for the next round, can gradually erode safety even when no harmful data is used. The effect, which they call Progressive Misalignment, appears across two model families and also occurs when the extra context is replaced by feedback from a simulated user with benign intent.

Safety & security

Non-English requests raise tool admission and response harm in LLM agents

Using a controlled URL-retrieval benchmark across ten models and seven languages, the authors independently varied request, retrieved-page, and response languages to isolate where safety fails. They found that non-English harmful requests raise mean tool admission from 46.9% to 67.4%, and that requiring answers in the page language increases harmfulness from 50.3% to 57.4% in closed-weight models. Audits further show that many non-harmful responses are language deflection or unusable answers, not genuine refusals.

Safety & security

ChatGPT's health answers are high-risk for one in five real users

The authors present the first demographically grounded safety evaluation of real-world health conversations with ChatGPT. After analyzing 15,555 health conversations from 1,252 users in India, Nigeria, Brazil, and Pakistan, they find that roughly one in five is high-risk. Risk concentrates in symptom and condition queries (33–48% high-risk), and context-aware scoring raises the high-risk share by 13–17 percentage points. One in four conversations that appear safe when scored in isolation becomes unsafe once the user's own history is considered.