The paper introduces in-situ representation refinement, where support labels guide changes to the episode's representations during a forward pass, without parameter updates. This is implemented in RefineICL, an attention-gated architecture without a full FFN. RefineICL achieves higher accuracy than TabPFN-3 on multiple benchmarks and uses less memory.
Tabular foundation models refine representations in-context with attention-gated updates
The authors introduce in-situ representation refinement and the RefineICL model, which outperforms TabPFN-3 while using no feedforward layers.
Ant Group · Independent Researcher
In one line
Tabular foundation models refine in-context representations via attention-gated updates, improving accuracy and memory efficiency over TabPFN.
Before you read this
An idea this paper builds onThe authors derive a regularized leave-one-out objective that separates attention-based reading from state-dependent scaling, leading to the RefineICL architecture: an attention-gated contextual stack without feedforward networks, with selected low-rank feature interaction and typed memory.
How it works
stage by stageThe Daily Edition
The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.