Predicting a spread of returns helps most when training data is scarce

Enforcing consistency between predicted return distributions at every noise level of a flow model stops the distribution collapsing to a single number and improves policy learning across 26 long-horizon tasks.

Top University

Chongyi Zheng · Zhiyuan Zhou · Qiyang Li · Benjamin Eysenbach · Sergey Levine

Princeton University · University of California, Berkeley

Research Digest·3 November 2026·3 min read

Illustration: Predicting a spread of returns helps most when training data is scarce

In one line

Supervising a flow-matching return model at every noise level, not just the endpoint, yields more accurate value estimates in offline reinforcement learning, especially with limited data.

Read it as

Reinforcement learning trains an agent by trial and error, judging situations with a value function: a guess at total future reward. No direct teacher for that guess, only small immediate rewards, so errors compound when data is scarce.

Rather than one number, the method predicts the full spread of possible future rewards using flow matching, which turns random noise into a distribution over many denoising steps. It forces agreement with the next state's prediction at every noise level, not only the last.

Earlier methods often let that spread collapse to a single value; this one kept it and estimated returns more accurately. Across 26 long-horizon tasks it beat earlier flow-based value learners, with the biggest gains on scarce data.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

flow matching distributional reinforcement learning bellman consistency value function estimation offline reinforcement learning implicit value learning