Tuesday 15 September 2026
Evening EditionRSS

ZOTPAPER

News without the noise

ZOTPAPER
AI & ML

A 2.8-trillion-parameter open MoE model approaches frontier performance.

Kimi K3, with 104B activated parameters and million-token context, outperforms most open and proprietary models but trails Claude Fable 5 and GPT-5.6 Sol.

PaperIndependentcs.CLarXiv:2607.24653v2 ↗
Kimi Team · Tongtong Bai · Yifan Bai · Yiping Bao · M. C. · Jianfeng Cai · +396 more
Research Digest··2 min read
Read the paper →
Kimi Team introduces Kimi K3, a 2.8T parameter Mixture-of-Experts model with 104B activated parameters, native vision, and a 1M-token context window. The authors demonstrate that innovations in attention (Kimi Delta Attention, Attention Residuals) and routing (Stable LatentMoE) yield roughly 2.5× scaling efficiency improvement over Kimi K2, enabling frontier-level performance across long-horizon coding, agentic, reasoning, and vision tasks.

what_they_did

The authors trained Kimi K3, a 2.8T parameter MoE model with 896 experts, activating 16 per token. They introduced Kimi Delta Attention (KDA) and Attention Residuals to improve information flow across long sequences and deep layers, and Stable LatentMoE for balanced expert utilization. Post-training involved reinforcement learning across general, agentic, and coding domains with multiple reasoning-effort levels. The model supports up to 1M token contexts and includes vision capabilities. Infrastructure advances included algorithm-system co-design for KDA, balanced expert-parallel training, and persistent rollout for million-token agentic RL.

key_findings

  • Kimi K3 achieves approximately 2.5× improvement in overall scaling efficiency over Kimi K2.
  • On long-horizon coding and agentic tasks, Kimi K3 outperforms all other open models and most proprietary models evaluated, trailing only Claude Fable 5 and GPT-5.6 Sol.
  • The model demonstrates strong compositional generalization and robust long-horizon execution capabilities enabled by multi-domain RL post-training.
  • Kimi K3 achieves competitive results on vision, knowledge, and reasoning benchmarks, establishing a new high-water mark for open frontier models.

why_it_matters

This paper represents a significant step toward open-weight frontier models. By releasing the full 2.8T model weights, the authors enable independent research on large-scale MoEs, attention mechanisms, and RL post-training. The demonstrated scaling efficiency and multi-domain performance show that open models can nearly match the capabilities of proprietary systems, potentially accelerating broader adoption and democratizing access to frontier intelligence.

caveats

Kimi K3 still lags behind the two most powerful proprietary models (Claude Fable 5, GPT-5.6 Sol) across the full evaluation suite. The model requires massive computational resources (2.8T total parameters), making it inaccessible for many practitioners. The paper does not provide detailed ablation studies for each architectural innovation, and the evaluation suite may not cover all real-world use cases.

§

Analysis

Kimi K3 continues the trend of open models rapidly closing the gap with proprietary frontier models. The specific innovations in attention and routing offer new architectural patterns that could be adapted by other researchers. Combined with the release of model weights, this work provides a concrete baseline for future exploration of scaling efficiency and multi-stage RL training. It also highlights the growing importance of infrastructure innovations to support million-token agentic training and deployment, which may become standard for future models.

newspaper

Research Digest

Articles published under the Zotpaper byline are synthesized from multiple source publications by our AI editor and reviewed by our editorial process. Each story combines reporting from credible outlets to give readers a balanced, comprehensive view.

Editorial ProcessSubscribe