New system halves compute waste during AI response generation training

FlexFloor releases idle expert workers early, improving rollout throughput by over half while preserving output reward.

Independent

Jia Wei · Ziyi Qiu · Yi Xu · Xuehai Qian

Research Digest·6 November 2026·2 min read

Illustration: New system halves compute waste during AI response generation training

In one line

FlexFloor increases mixture-of-experts reinforcement-learning response throughput by 52.74% and occupied-compute efficiency by about 65.40% by releasing workers as their responses finish.

Before you read this

Two ideas this paper builds on

mixture of expertsA neural architecture with many specialized sub-networks, only a few activated per input via a router, balancing capacity and efficiency.77 papers →reinforcement learningA machine learning method where an agent learns to make sequences of decisions by receiving rewards or penalties for its actions.272 papers →

Read it as

During reinforcement-learning post-training, a model generates answers that are scored so it can improve. In a mixture-of-experts model, each answer uses selected specialist parts spread across computing workers. A worker may finish its answers but stay allocated because slower workers still need those specialists.

FlexFloor predicts answer lengths and schedules prompts and workers around them. As answers finish, it releases workers while keeping the specialist parts available to workers still generating answers and respecting memory limits.

Against the baseline, it generated 52.74% more answer text per second and about 65.40% more per occupied worker-time, with little observed change in reward scores. This can speed training and free computing capacity.

How it works

before and after

RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Concepts

mixture of experts expert parallelism reinforcement learning elastic execution kv cache admission tail-aware batching