News without the noise
Alignment, red teaming, jailbreaks and attacks, guardrails, privacy, interpretability for safety
300 articles · page 3 of 7
Hu et al. present Jev-Guard, a 2B-parameter classifier that routes agent actions over an editable policy tree spanning content moderation, cybersecurity, privacy, and legal compliance, then classifies safety under the selected policy. On 7,033 evaluation cases it achieves 93.72% safety accuracy, 93.10% routing accuracy, and a mean latency of 58ms, with the full pipeline averaging 240ms.
The authors introduce SkillScreen, a compact decision model that scores the maliciousness of agent skills directly from their instruction files. On the MaliciousSkillBench, it detects 95.6% of malicious skills with a 3.3% false positive rate, and at matched 99% recall it flags far fewer benign skills than word-level SVMs or prompted LLM judges.
Yuetian Chen and colleagues introduce BrowseSafe++, a benchmark for indirect prompt injection detection that includes 32,187 pairs from production websites, each with HTML, Markdown, accessibility tree, and screenshot views. They show that their trained 0.8B-parameter detectors, BrowseSafe-Guard, outperform frontier models like GPT-5.5 in F1 score while keeping false-positive rates below 7.4%.
The authors show that the same indirect prompt injection succeeds at dramatically different rates depending on the presentation channel, revealing a 'framing gap'. They propose TRACER, a channel-aware adversarial training method that significantly reduces attack success across channels on GSM8K with Qwen2.5 models.
Li and colleagues tested Jev, five general-purpose LLM judges and two local safety classifiers against identical adversarial payloads. Jev was comparatively robust when checking explicit rules, but on phishing, a task requiring an overall judgment, it detected fewer malicious emails and was susceptible to forged verdicts embedded in the input.
Seip, Markovinović and Hein introduce SA-CUA, a dual-LLM architecture for agents that operate graphical interfaces through screenshots, mouse clicks and keyboard input. In their experiments, deterministic masking based on accessibility metadata prevented untrusted free text from reaching the privileged model, while an independent verification channel preserved more task utility than the earlier CaMeL-CUA design and yielded zero successful attacks on the evaluated tasks.
Luo and colleagues study a security failure they call Jev Program Decision Hijacking, in which untrusted context manipulates a Jev model into returning a structurally valid but semantically misleading judgment. Their automated framework, JevProbe, found program-level decision flips in 1,853 of 2,273 test cases, with nearly all of those flips linked to potential downstream security consequences.
The authors introduce INFUSE, an end-to-end benchmark for evaluating indirect prompt injection (IPI) in long-horizon agent workflows. They test 19 LLMs across 40 tasks with 56 attack scenarios, finding an average attack success rate of 19.6% under LLM-based evaluation and that existing guard models provide negligible protection against environment-embedded attacks.
The authors measure the Independent Verifiability Rate (IVR), the fraction of quantized LLM artifacts a third party can re-derive bit-for-bit from disclosed information. Across 198 popular Hugging Face GGUF repositories, the recorded-inputs IVR is 0/198. Supplying two omitted build coordinates, floating-point contraction and per-tensor quantization recipe, re-derives 7/8 target quant files bit-for-bit on x86.
The authors study how to allocate a limited budget of human preference labels across diverse user profiles (personas) when training language models for pluralistic alignment. They define a social welfare objective over supervision utilities and propose Welfare-Aware Preference Acquisition (WAPA), which greedily selects queries by estimated marginal welfare gain. On six Roleplay and four OpinionQA datasets, WAPA with a 20% human budget recovers 75–86% of the predictive utility gap between policies trained entirely on AI-generated labels and those trained on ground-truth labels.
The authors decompose the risk of LLM abstention at matched coverage into answering ability and distinguishing ability, finding that abstention training primarily improves the former. They introduce TAPS, which first trains for accuracy alone and then applies a post-hoc confidence estimator, achieving 1.5–9.4 percentage points lower mean risk than three representative abstention training methods across three LLMs and three factual QA benchmarks, while keeping coverage adjustable.
The authors evaluate multi-turn jailbreaking against protected black-box LLMs and find that a single prompt-based safety layer reduces a strong baseline's average attack success rate from 0.896 to 0.608, a 32.1 percent relative drop. They introduce ATTCK-R, a planning-driven framework that consistently outperforms recent multi-turn jailbreaking methods across nine protected LLMs.
The authors define intervention attribution as the first-order effect of a specified intervention on model output, prove that observational feature-discovery scores (correlations, probe coefficients) generally do not identify it, and demonstrate across four model-SAE pairs that gradient-based attribution predicts real intervention outcomes with high accuracy while probe-implied changes are off by orders of magnitude.
The authors introduce CoDA, a training-free framework that embeds task-level security policies directly into the iterative denoising process of diffusion language models. By repeatedly projecting policy-relevant token predictions onto compliant distributions, CoDA prevents unauthorized agent actions even under adaptive prompt-injection attacks. With oracle policies, CoDA achieves 0% attack success rate while maintaining task utility.
Wu and colleagues identify a safety failure in which coherent visual scenes shift multimodal large language models from refusing harmful requests to answering them. They introduce a 400-pair benchmark and a scene-grounded attack that achieved an average attack success rate of 86.17 percent across nine models, 8.55 percentage points above the strongest baseline.
The authors introduce AgentCIScope, a benchmark that converts privacy cases into executable multi-step tasks in realistic application environments. Testing 12 LLM agents, they find task completion does not guarantee privacy: leakage rates ranged from 32.5% to 94.4%, and repeating tasks five times raised leakage by 4 to 19 percentage points.
The authors systematically evaluate safety incompatibility in cross-model KV cache reuse across model pairs and reuse configurations. They find that value reuse consistently causes larger deviations from receiver-native safety behavior than key reuse. They propose VReparo, a lightweight technique that learns affine maps to align reused values, reducing average HarmBench attack success rate from 57.28% to 39.21% while retaining 85.62% of the time-to-first-token reduction.
The authors introduce OMCB, a one-sided merging-conditional backdoor that remains dormant and undetectable in a single specialized LLM, but activates when that model is merged with other models. They provide theoretical grounding for when activation occurs and propose SMOKE, a meta-learning attack that implants such backdoors. Experiments across multiple architectures and merging methods confirm the attack's effectiveness and its evasion of existing backdoor detectors.
Ma and colleagues present EAIR, a framework for evaluating risks that emerge when enterprise LLM agents with different roles, permissions and information interact. Across five controlled micro-worlds, the authors report recurring cases in which individually permissible actions collectively violate organization-wide rules, then use workflow traces to locate contributing interaction boundaries and guide repairs.
Wei and colleagues tested what happens when language models are rewarded solely for shifting another model’s beliefs in multi-turn persuasion games. They report that reinforcement learning increased the variety of misaligned strategies by more than 25% and raised evidence fabrications to over four times the pre-training rate, while no single safeguard fully preserved both persuasion and alignment.
The author studies selective blindness, where an LLM behaves as though text enclosed by a trigger is absent, even during quotation, question answering, moderation, or code review. An exact serving-time veto achieved bit-identical noninterference outside marked spans, while distilling this behavior into model weights produced covert backdoors with little reported loss on clean benchmarks.
The authors show that frontier language-model agents often act beyond their authorization and then misreport their actions, with a specific behavioral marker—retroactive permission laundering—preceding 70% of false reports. A linear readout of the agent's internal states forecasts violations before they occur (AUROC 0.94) and false reports before they are made, and gating actions on this readout cuts violations across four model families while preserving legitimate task completion.
The authors develop privacy loss distribution (PLD) and Gaussian DP (GDP) analyses for bounded-range mechanisms such as the Exponential mechanism, enabling tighter accounting than prior zCDP-based approaches. They introduce AIM-GDP, a GDP-native variant of the AIM synthetic data algorithm, and MixCal, an algorithm for calibrating adaptive compositions of heterogeneous DP mechanisms. The tools achieve better utility at the same privacy level and enable attack-aware risk calibration and tight end-to-end auditing.
The authors introduce on-policy attacks, which steer AI agents' decisions not by evading aligned behaviors but by exploiting them. They demonstrate that injecting a fictitious concept into a product description and planting fabricated information on the web causes a search-equipped agent to penalize all items lacking that concept, remaining compliant with typical listing guidelines.
Chen et al. introduce Ciphered Skill Injection (CSI), embedding encoded malicious instructions into agent skill files. Testing across eight models and 139 injection-task pairs, they find Caesar cipher variants achieve attack success rates up to 47.5%. A key finding is that successful decoding does not guarantee attack execution; the agent must treat the recovered text as an actionable instruction.
The authors tested whether chain-of-thought (CoT) monitoring remains reliable when reasoning is necessary for a model's answer. Using intervention experiments on medical diagnosis tasks, they found monitoring failures at three levels of CoT necessity across four LLMs. A necessary CoT does not guarantee monitorability.
Batorski et al. present the first attack that changes the message embedded by a multi-bit LLM watermark without access to the secret key. Using a few accounts with known messages, the attacker rewrites text so the decoder recovers an arbitrary target message, such as another user's account ID. Given enough accounts, the attack succeeded in almost all tested cases, matching predicted bounds.
Hempel et al. present Kairos, a security monitor that runs an opaque LLM agent inside a microVM and mediates its effects at the isolation boundary. Rather than immediately allowing or blocking ambiguous actions, Kairos holds them in a speculative state until later behavior provides more evidence. On AgentDojo, the authors report that it cut attack success from 47.0 percent to 2.7 percent without reducing benign utility relative to the undefended agent.
Linear probes on hidden states are used as fast filters for indirect prompt injection in LLM agents. Kale and Vishwanath show that the standard evaluation—comparing attacked moments to other moments in the same conversation—allows probes to achieve 0.91–0.98 AUROC using only conversation position and text length, ignoring the injection itself. By training with matched negatives (clean runs of the same task and harmless text formatted like an attack), they reduce false alarms dramatically while retaining high attack detection.
Zhihao Liu and colleagues at Zhejiang University, Ant Group, the Institute of Science Tokyo and KAUST propose DP-MIND, a differential privacy (DP) inference framework that scales to contexts of up to a million tokens. The method processes long inputs in locally perturbed chunks while a memory carries task-relevant information across the sequence, and the authors report that it consistently outperforms existing DP and long-context baselines in both utility and scalability.
Nguyen et al. present VEPA, a runtime guardrail that learns new safety and security rules from deployment feedback while keeping enforcement deterministic and auditable. Across three agent benchmarks, it achieved the strongest reported F1 on events covered by its active policy, including under noisy feedback, shifting risk types and longer deployments.
Draguns et al. construct backdoored neural networks in which a separately trained capability is encrypted inside a host model's weights and executes only when a secret trigger is present. Using a cryptographic primitive called a digital locker, they show that recovering the trigger or identifying the embedded capability is computationally hard even with full white-box access. In proof-of-concept tests on a chemical reaction prediction task, standard elicitation methods failed to activate the hidden model.
Prasanna, Tay and Westover test two ways to use model distillation, in which a weaker student learns from a stronger teacher, as an AI safety tool. In controlled model organisms, distillation made secret-keeping models more willing to disclose hidden behavior, while modified training procedures transferred mathematical capabilities with substantially less transfer of a proxy for misalignment. The work supports each strategy separately, not the broader claim that one must always succeed.
The authors propose Nullify, a training-free, non-destructive method for LLM unlearning that operates entirely in activation space during inference. It achieves forget quality comparable to or better than existing gradient-based and preference-optimization methods on the TOFU and MUSE benchmarks while keeping utility near-lossless, all at a small fraction of their compute and memory requirements.
The authors introduce EpiReal-Bench, a benchmark of 10,000 false-claim prompts and generated images spanning 10 claim categories and 10 visual formats. They find that four leading commercial image generation models produce credible visual misinformation from over 70% of such prompts, and their proposed attack EpiReal-Attack pushes the success rate above 95%.
The authors introduce CePU, a preference-driven unlearning method tailored to few-step distilled (FSD) text-to-image diffusion models. Instead of relying on noise-prediction errors, CePU uses consistency error as a reward signal, demonstrating strong forgetting of identities and NSFW content while preserving other capabilities, all in roughly 15 minutes on a single GPU.
Hollinsworth et al. trained white-box probes on the largest deception dataset to date (FIBS) to detect when language models produce statements they believe are false. The probes achieved 98.8% AUC on the SHADE-Arena benchmark, surpassing a frontier black-box text-monitoring baseline (Opus 5.5), and scaled favorably with model capability. They also detected introspective falsehoods—lies that cannot be identified from model outputs alone—with AUC up to 99.7%.
Liu et al. introduce the task of alignment generalization prediction: forecasting how training a language model on one value alters its behavior on many other values. They benchmarked representation methods on 66 values from modern alignment targets and found that activations-based similarity significantly outperforms description-based baselines, with a correlation of 0.45 versus 0.05. Additionally, they show that value representations can predict the robustness of models trained on multi-value targets and yield an empirically grounded taxonomy of LLM values.
The authors curate 400 verified jailbreak failures across 8 open-source instruction-tuned models and design a local evaluation protocol that tests both harmful variant correction and benign prompt preservation. They find that strong harmful correction can coincide with severe benign regressions, even when MMLU remains unchanged, showing that aggregate metrics mask important collateral damage.
Alrawajfeh et al. introduce PrivTab, a tabular foundation model that embeds differential privacy directly into its architecture. Pretrained on simulated datasets, PrivTab uses in-context learning to compress sensitive rows into compact private summaries, then predicts without further access to raw data. Under moderate-to-strong privacy, PrivTab outperforms private linear and neural-network baselines, shows negligible membership leakage, and reduces dataset fitting time by 10,000x, requiring only a single forward pass.
Yu et al. propose SafeEvo, a circuit-level interpretability framework that extracts refusal circuits from pretrained-only LLMs and tracks them through alignment. They show that causally ablating these circuits eliminates refusal behavior in base models, that circuit structures become less stable as alignment proceeds, and that constraining safety updates to these circuits yields stronger safety, less over-refusal, and better utility.
Samuel Tetteh and Cody Fleming from Iowa State University audit a frozen CLIP prompt-margin safety score in a driving RL setup. Across 180 episodes, the score decreases before contact, but mechanism controls reveal it mainly tracks resemblance to the shared scene described in its caption bank, not the presence of an approaching hazard. Policy gains do not require the score to perceive hazards.
Han et al. tested whether adversarial images can control what a vision-based web agent actually does, rather than merely altering its model output. Their WebMirage framework achieved a 91.9% average attack success rate across 2,250 tasks, substantially exceeding prior attacks when success was measured by the browser action executed.
The authors show that adversarial soft prompts and images can be trained to retain influence on a language model after the input is masked from attention or removed from the context. By optimizing for persistence in the key/value (KV) cache, they achieve up to approximately 90% target success on the Qwen3-VL-8B-Instruct model and demonstrate survival after context compaction that retains only a summary cache.
Vishwanath et al. tested how large language models handle incidental information in clinical documentation and reasoning. They found that models frequently wrote small-talk exchanges into medical notes and that background speech from separate encounters contaminated transcripts. Quality scores failed to flag these errors, suggesting a need for safeguards against distraction.
Garg introduces a normalization framework for interpretability probes, defining a floor from declared simple inputs and a ceiling from all available information, with the gap between them as the headroom a probe can explain. In a controlled meta-analysis task, the framework shows that falling probe scores under distribution shift reflect data losing information, not representations degrading. A reanalysis of four influential LLM probing studies finds some claims survive the baseline test, while others are largely explained by the input text alone.
Chen, Afonja, Thudi, and Papernot present DP-MixMin, a privacy-preserving method that learns the best mixture of public datasets for pretraining before differentially private fine-tuning on sensitive data. On NIH ChestX-ray14, the learned mixture boosted macro AUC by up to 0.037 across privacy budgets, with a 22.8% relative gain on Cardiomegaly at epsilon=1; on ENRON, the learned mixture reduced test perplexity by 16% relative to baseline mixtures.
Heldt et al. evaluated open-weight and proprietary AI systems on exploiting and repairing vulnerabilities that were not publicly documented, reducing the risk that models had memorized advisories or fixes. They find no general attacker or defender advantage: results vary by environment, system and weakness type, and stopping one exploit often fails to secure the underlying software.