What they did

CLAMP uses an initial visual observation to limit object references to entities supported by the scene. A supplied symbolic action model defines valid state transitions, action preconditions, and goals, while the underlying vision-language model remains frozen.

During generation, hard masks remove invalid next-token candidates. An HMM-based world-state lookahead module then reweights feasible candidates according to preconditions and goal reachability; for unseen tasks and environments, the HMM is adapted at test time using unlabeled continuations sampled from the frozen model. The authors evaluate CLAMP on VLABench, SafeAgentBench, and TaPA.

Key findings

• Scene-grounded decoding reduced references to objects unsupported by the visual observation.
• Enforcing action and state constraints improved plan safety and feasibility across the evaluated benchmarks.
• CLAMP retained the frozen model’s language prior while preventing candidates ruled out by visual evidence or the symbolic action model.
• The authors attribute most remaining failures to perception errors or mismatches between the constraints and the actual task.

Why it matters

The results suggest that embodied planners can become more reliable without retraining their underlying vision-language models. Decoding-time constraints offer a way to combine flexible language generation with explicit guarantees about available objects, valid actions, and progress toward a goal.

Caveats

The abstract provides no quantitative effect sizes or model-by-model comparisons. CLAMP also depends on accurate initial perception and a correctly aligned symbolic action model; hard constraints may exclude valid plans when either source is wrong, and the reported evidence is limited to three benchmarks.