The paper analyzes why training-free safeguards for text-to-image generation often fail. It shows that assuming all unsafe concepts share a common global subspace is flawed, leading to a trade-off: narrow subspaces miss many unsafe prompts, while broad ones distort benign prompts. The authors propose CALM, a method that instead uses local counterfactual corrections for each prompt, reducing unsafe content more selectively.
Local counterfactual edits outperform global unsafe subspace removal
Training-free safeguard CALM selectively suppresses unsafe image generations by routing prompts to category-specific safe directions.
Big Tech
NaHyeon Park · Minhyun Lee · Hyunjung Shim
KAIST · Samsung Electronics
Research Digest··2 min read
Thread:VLM Hallucination Circuits
In one line
Text-to-image safety fails when using a single global unsafe subspace due to a coverage-selectivity trade-off.
Before you read this
Three ideas this paper builds oncounterfactual reasoningComparing actual outcomes with imagined alternatives to estimate the effects of different actions or conditions.116 papers →model routingDirecting each query or subtask to the most suitable model to optimize cost, accuracy, or quality.91 papers →activation steeringActivation steering modifies a model's internal vector representations during inference to shift behavior, such as editing outputs, removing capabilities, or controlling trust.83 papers →
Read it as
Park, Lee, and Shim analyze why global safety signals fail in text-to-image models and propose CALM, which uses prompt-local counterfactual pairs to edit token representations.
The authors first tested the assumption behind training-free safety measures: that unsafe content can be modeled as a single global subspace in the text encoder.
How it works
before and afterThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.