The authors trained encoders with different parameter counts, token budgets and masking ratios, then evaluated them on classification and information-retrieval tasks.
UltraBERT balances masking strategies to improve classification and retrieval
Separate training heads for two masking rates let one bidirectional encoder perform strongly across tasks that previously favored different pretraining recipes.
Big Tech
Jungwoo Park · Taewhoo Lee · Jaewoo Kang
Korea University · AIGEN Sciences · Amazon
Research Digest··3 min read
8 trillion tokens, focusing on masked language modeling, where models reconstruct hidden tokens.
Why this paper
From Amazon and 2 others
In one line
UltraBERT is the first encoder to achieve state-of-the-art on both classification and retrieval tasks.
What we could check
- ·No code link found
- ·No weights link found
- ·No dataset link found
- ·No compute details found
- ✓Limitations stated by the authors
- ✓Reports numbers on named benchmarks (2 benchmarks)
Observed from the paper text and links we have. Absence here means we did not find it, not that it does not exist.
§