mixture of experts

The papers

Most central to this idea first, not most recent.

• Academiccs.LG

  Feedback control theory yields simpler MoE load balancer that adapts to shifting data

  A pure integral feedback controller achieves the best load balance for MoE training under shifting data distributions.

  Umeå University · Nov 2026

• Chinese Techcs.AI

  Language models spread subject expertise across neurons inside multiple experts

  Neurons, not experts, are the unit of domain specialization in Mixture-of-Experts language models.

  Nanyang Technological University, Tencent · Nov 2026

• Independentcs.AI

  Flexible framework speeds AI model inference across diverse hardware setups

  AdaOffload adaptively coordinates CPU and GPU execution across multiple GPUs for MoE inference, achieving 1.9× to 15.5× throughput speedups over prior systems.

  Nov 2026

• Independentcs.DC

  AI models exchange expert data up to three times faster

  NVLinkOneSided accelerates MoE inference by using one-sided communication to push tokens directly into rank-major buffers without metadata exchange.

  Nov 2026

• Independentcs.AI

  New system halves compute waste during AI response generation training

  FlexFloor increases mixture-of-experts reinforcement-learning response throughput by 52.74% and occupied-compute efficiency by about 65.40% by releasing workers as their responses finish.

  Nov 2026

• Top Universitycs.DC

  MoE model disaggregation yields modest speedup on rack-scale NVLink, model predicts payoff

  On rack-scale NVLink, attention-FFN disaggregation can deliver up to 1.17x more throughput than wide expert parallelism for some MoE models.

  University of California, Berkeley · Nov 2026

• AI Startupcs.LG

  Language models can accelerate more when handling one request alone

  Speculative decoding on sparse MoE models remains fast at low batch size because routing overlap and GPU execution effects make verification cheaper than activated-expert bytes suggest.

  Cohere · Oct 2026

• Big Techcs.CL

  AI coding model beats predecessor using roughly one-fifth the active parameters

  Nemotron 3.5 Super trains a 12B-active multimodal model that exceeds a 4.6x larger model on SWE-bench Pro.

  NVIDIA · Oct 2026

• Big Techcs.AI

  Fewer draft tokens verified per round makes hybrid models a third faster

  HybridSpec speeds up hybrid GDN-MoE language model inference by adaptively selecting verification length and using parallel GDN kernels.

  Meta AI, New York University · Nov 2026

• Chinese Techcs.CL

  Several routers pick experts in language models, boosting accuracy without extra compute

  Multiple routers with differentiated preferences improve expert selection and computation in mixture-of-experts models.

  Baidu · Oct 2026

• Chinese Techcs.LG

  Low-precision training of AI expert models needs matching fixes on both sides

  Symmetric quantization recovery closes the accuracy gap in FP4 reinforcement learning for mixture-of-experts LLMs.

  Huawei · Nov 2026

• Industrycs.CL

  Open-weights German-English model claims Pareto frontier for quality and serving cost

  Kolibri is an open-weight mixture-of-experts model on the Pareto frontier of quality and serving cost for English and German.

  Aleph Alpha · Oct 2026

• Big Techcs.LG

  Joint expert selection preserves MoE performance under aggressive pruning

  Joint reconstruction-based expert selection and weight adaptation preserve MoE accuracy better than independent importance rankings, especially under aggressive pruning.

  LinkedIn, MIT · Oct 2026

• Big Techcs.AR

  Generating MoE expert weights on chip reduces inference memory pressure

  FORGE regenerates MoE expert weights on-chip from compact rotor parameters, replacing HBM weight loads and improving throughput 1.3-2.8x with some accuracy loss.

  University of Wisconsin-Madison, Korea University · Oct 2026

• Chinese Techcs.LG

  Two-rate precision allocation cuts MoE memory footprint and decode traffic

  DuoRate stores MoE weights once and allocates per-expert read precision per workload, matching 4-bit formats at 2.9 bpw on Qwen3-30B-A3B.

  Huawei Technologies Ltd. · Oct 2026

• Chinese Techcs.LG

  Asymmetric compute allocation improves looped Mixture-of-Experts models

  Compressing the first loop visit and spending the saved compute on hidden width lowers validation loss in looped mixture-of-experts models across budget sizes.

  SJTU, Shanghai AI Lab · Oct 2026

• Chinese Techcs.AI

  Near-square experts and depth-aware allocation improve MoE models

  For latent mixture-of-experts, near square experts maximize per parameter rank, and layerwise intrinsic dimension guides capacity allocation.

  Xiamen University, Tencent Hunyuan · Oct 2026

• Big Techcs.LG

  Soft routing among process experts helps PDE forecasting, but gains prove seed-dependent

  Dense soft routing yields 2.2 times lower rollout error than hard top-1 routing on a compositional PDE benchmark.

  Indian Institute of Technology Bombay, IBM Research · Oct 2026

• Chinese Techcs.LG

  Pruning experts by replacement damage better preserves MoE reasoning performance

  RAZOR prunes replaceable experts in MoE models using consensus residuals, surpassing baselines on reasoning tasks.

  Tencent, Code Models · Oct 2026

• Chinese Techcs.CV

  Specializing latent tokens as visual experts improves vision-language reasoning

  In latent visual reasoning, specializing latent experts to complementary evidence improves performance more than increasing the number of latent tokens.

  Tongji University, Hong Kong Polytechnic University · Oct 2026

• Top Universitycs.CV

  Two denoising experts make video diffusion distillation far more efficient

  Glance Forcing distills bidirectional video diffusion into a few-step autoregressive model using one PF-ODE trajectory, outperforming a fully trained baseline at far lower collection cost.

  Wuhan University, University of Electronic Science and Technology of China · Oct 2026

• Academiccs.LG

  Predicting expert removal error makes batched MoE decoding more efficient

  BASE selects experts by predicting removal error, improving MoE decoding accuracy by 29.5 points under tight batch budgets.

  Vanderbilt University · Oct 2026

• Chinese Techcs.DC

  Host-centric routing makes MoE inference practical on consumer GPUs

  CoMoE uses host-centric token routing to make multi-GPU MoE inference on consumer GPUs up to 1.46x faster and substantially cheaper than A800 deployments.

  Tsinghua University, Alibaba Cloud Computing · Oct 2026

• Big Techcs.LG

  Looping and sparse experts provide complementary routes to efficient scaling

  Looped MoE scaling follows a bounded, sparsity-conditional recurrence law that subsumes dense and MoE scaling laws and predicts held-out loss.

  Meta AI · Oct 2026

• Top Universitycs.LG

  Expert pruning in MoE models can be tailored to decode phase, preserving prefill quality while boosting throughput

  SlimWise improves MoE decode throughput by pruning experts only during decode while keeping full prefill.

  a2sys, KAIST · Oct 2026

• Top Universitycs.LG

  Low-rank router enables scaling mixture of experts to many more experts

  Low-rank routing in mixture-of-experts layers enables scaling to many more experts without increasing active compute.

  University of Pennsylvania · Oct 2026

• Top Universitycs.AR

  Joint sparsity and quantization enable efficient trillion-parameter MoE serving

  A hardware-software co-design framework jointly sparsifies and quantizes MoE weights for acceleration on Sparse Tensor Cores.

  POSTECH, ISTA · Oct 2026

• Chinese Techcs.CV

  Splitting experts by role improves video diffusion model scaling

  SplitMoE improves video diffusion by separating experts into semantic and generic groups to avoid spatiotemporal fragmentation.

  University of Chinese Academy of Sciences, ByteDance · Oct 2026

• Chinese Techcs.LG

  Feedback control stabilizes expert loads in extremely sparse language models

  ID Balancing reduces expert load imbalance in extremely sparse MoE training using an integral-derivative controller.

  Qwen Team, Alibaba Token Hub · Oct 2026

• Top Universitycs.RO

  Federated robot learning preserves local expertise while cutting communication costs

  A federated method assembling local LoRA adapters as server MoE experts outperforms centralized PEFT by 12.23% with 86.81% less communication.

  Nanjing University, Peking University · Sept 2026

• Industrycs.LG

  Exact quantiles and gradient corrections improve expert load balancing

  Exact Quantile Balancing and Load-Error Injection improve MoE load balance and performance over prior methods.

  Aleph Alpha · Oct 2026

• Big Techcs.AI

  Supervising token-level loss improves routing in sparse mixture-of-experts models

  Sparse MoE routing improves when token-level cross-entropy loss directly supervises expert selection.

  Bar-Ilan University, NVIDIA · Oct 2026

• Big Techcs.LG

  Mixture of experts unifies time-series forecasting and reasoning in one model

  A single time-series language model matches top specialized systems in forecasting, contextual prediction, and reasoning by mixing independently trained experts.

  Columbia University, Stanford University · Oct 2026

• Big Techcs.LG

  Masked co-adaptive fine-tuning cuts expert fetches by over 23% for MoE models.

  MaskCoFT cuts expert fetches per token by 23.7% on Mixtral-8x7B and 10.1% on DeepSeek-V2-Lite while staying above base model accuracy.

  Rensselaer Polytechnic Institute, IBM Research · Oct 2026

• Big Techcs.LG

  Segment-level expert routing gives one model four inference budgets

  Stepped MoE combines configurable active parameter budgets with segment-level expert routing, outperforming comparable dense models while retaining similar latency on memory-constrained devices.

  Apple · Oct 2026

• Chinese Techcs.CL

  Reusing Earlier Expert Choices Improves Sparse Model Training Efficiency

  HERO-MoE reuses preceding routing distributions to improve MoE router quality, reducing training loss with minimal overhead.

  University of Science and Technology of China, Huawei Inc. · Oct 2026

• Chinese Techcs.CL

  Pruned language models retain problem-solving coverage despite lower one-shot accuracy

  Pruning can preserve or expand the set of solvable problems despite lower single-attempt accuracy, but at increased sampling cost.

  Tencent · Nov 2026

• Top Universitycs.CV

  Joint video and 3D motion prediction reduces trajectory errors by over half

  WorldFlow jointly predicts future video and queried three-dimensional object motion, achieving the best displacement errors among evaluated learning-based methods on three test sets.

  RoboScience, National University of Singapore · Oct 2026

• Industrycs.AI

  Language models can learn nearly as well with four-bit numbers

  MXFP4 training closes validation loss gap to BF16 to 0.007 using 2D quantization, RHT, SR, and weight de-oscillation.

  Advanced Micro Devices, Inc. · Oct 2026

• Big Techcs.LG

  Tree-based routing balances expert use without auxiliary loss in MoE models

  BRANCH-MoE uses a binary decision tree with EMA anchors to balance expert utilization without auxiliary losses.

  Google Research, University of Southern California · Oct 2026

• Academiccs.CV

  Splitting diffusion distillation into two phase-specific half-sized experts beats a full-size student.

  Phase-wise Velocity Distillation uses two half-sized phase-specific experts to generate high-quality images in one full-backbone forward pass.

  The Hong Kong Polytechnic University, OPPO Research Institute · Oct 2026

• Top Universitycs.CV

  Better video predictions improve robots’ performance on unfamiliar manipulation tasks

  Video Prediction Policy 2 achieves strong zero-shot generalization in video prediction and action generation for robot manipulation.

  Robotera, Tsinghua University · Oct 2026

• Top Universitycs.CL

  Routing attention heads by token enables joint scaling of parameters and context

  NAMOH activates K of H attention heads per token, so scaling heads shortens routed contexts and reduces KV access without increasing storage.

  Peking University · Oct 2026

• Top Universitycs.LG

  Sparse mixture-of-experts models overfit repeated training data faster

  MoE models degrade more rapidly than dense models under data repetition, with overfitting increasing with sparsity.

  Stanford University, Paul G. Allen School of Computer Science, University of Washington · Sept 2026

• Independentcs.CLcode

  A 2.8-trillion-parameter open MoE model approaches frontier performance.

  Kimi K3 is an open, 2.8T parameter Mixture-of-Experts model with 104B activated parameters that achieves frontier-level performance across multiple domains.

  July 2026

• Top Universitycs.CL

  Multimodal training needs different scaling recipes for text, vision and audio

  Scaling laws for multimodal pretraining reveal modality-specific compute optima, asymmetric interference, and saturation of repeated non-text data.

  MILA - Quebec AI Institute, McGill University · Oct 2026

• Chinese Techcs.CV

  Latent trajectory guidance improves video generation reasoning

  VR-JEPA uses latent trajectory prediction to guide video generation for improved visual reasoning.

  Sun Yat-sen University, Shenzhen Loop Area Institute · Oct 2026

• Independentcs.DC

  Optical circuits cut LLM inference latency by 65% and energy use by 119%

  Static optical circuits configured per deployment improve LLM decode efficiency and latency over electronic switching.

  Nov 2026

• Big Techcs.CL

  Selective high-precision layers close the accuracy gap in four-bit training

  A recipe combining selective high-precision layers, unbiased gradient estimation, and improved scale selection achieves the lowest FP4 training loss gap to BF16 at 8B-30B scale.

  NVIDIA, Institute of Science and Technology Austria · Oct 2026

• Academiccs.RO

  Predictive visual models improve action generation for generalist robot manipulation

  InternW0-Δ jointly models visual dynamics and action generation for robot manipulation, outperforming prior methods across benchmarks.

  Shanghai AI Laboratory · Sept 2026

• Big Techcs.CV

  World state registers enable consistent multi-agent video generation across views

  WorldWeaver uses learnable world state registers to improve logical consistency in multi-agent video generation.

  University of California, Los Angeles, Adobe Research · July 2026

• Industrycs.CRcode

  Language model proofs take 41 to 50 times serving time

  Sable proves, on a single GPU, that a committed integer-quantized language model produced a specified token sequence exactly.

  MoonMath.ai · Nov 2026

• Big Techcs.RO

  AI driving models go farther when action feedback updates perception

  The benefit of VLM gradient insulation depends on action-token supervision and evaluation, questioning its default use in VLA training.

  NVIDIA, Princeton University · Nov 2026

• Top Universitycs.LGcode

  4-bit language models run as integer math on Apple's M5 chips without extra accuracy loss

  Standard MXFP4 weight rows fold into one exact INT8 vector times a power-of-two scale, a condition real weights already meet, so 4-bit models run on INT8 matrix units.

  UT Southwestern Medical Center, Emory University · Nov 2026

• Top Universityeess.AS

  YuE2 unifies symbolic and audio music generation via explicit score planning

  YuE2 unifies symbolic and audio music generation by first writing a readable score then producing full song audio.

  HKUST, M-A-P · Oct 2026

• Chinese Techcs.CV

  One-step action generation and asynchronous pipeline achieve real-time world action models

  RealtimeWAM enables near-lossless real-time action generation with 25x speedup via one-step denoising and asynchronous pipelining.

  Nanyang Technological University, Beihang University · Oct 2026

• Big Techcs.LG

  Language models can train up to 24% faster on Google chips

  Systolic Linear Attention runs 1.6x to 2.5x faster on TPUs than existing gated delta attention by widening keys and grouping decay gates.

  Google · Nov 2026

• Chinese Techcs.RO

  Future 3D motion prediction boosts robot learning with fewer demonstrations

  Augmenting robot action learning with future 3D motion prediction improves manipulation performance and generalization.

  The Hong Kong Polytechnic University, Alibaba Group · Oct 2026

• Independentcs.CL

  Cortex folds context benefits into model weights without degrading other skills

  Cortex makes context benefits persist after removal by consolidating them into parametric model updates via prospective queries and low-rank changes.

  Oct 2026

• Industrycs.CL

  One random probe can guide data-free mixed-precision model quantization

  Round-to-nearest quantization error is spectrally flat, letting one Gaussian probe estimate tensor sensitivity and allocate mixed precision without calibration data.

  Blacksheep ai. · Oct 2026

• Independentcs.LG

  Method converts LLMs to efficient attention while handling QKNorm and MoE

  MLAfication converts existing attention architectures to MLA, enabling up to 5.35x throughput and 94.73% KV-cache reduction.

  Oct 2026

• Chinese Techcs.CV

  Encoder-free multimodal models could close the gap with enough compute

  Encoder-free multimodal LLMs catch up to encoder-based ones in multimodal loss at roughly 10^22 FLOPs, making them promising at scale.

  CASIA, UCAS · Oct 2026

• Chinese Techcs.LG

  Aligning FP4 training and rollout stabilizes reinforcement learning

  TRACE aligns FP4 training and rollout quantization for MoE RL, matching BF16 rollout performance while accelerating rollout by up to 5.4x.

  Alibaba Token Hub, Alibaba Group, Ohio State University · Oct 2026

• Chinese Techcs.AI

  Search framework finds better layer-wise settings for efficient LLM inference

  InferOpt replaces mechanism-specific heuristics with a unified search framework for LLM inference configurations.

  Huawei Technologies Co., Ltd. · Oct 2026

• Chinese Techcs.AI

  Routing-signature diversity prevents expert collapse and boosts MoE specialization

  Distributional Orthogonalization loss and Replica Expert Mechanism prevent expert collapse and balance load, outperforming existing routing algorithms on MoE downstream tasks.

  Peking University, Tencent Hunyuan · Oct 2026

• Independentcs.CV

  Encoder-free model unifies image and video understanding and generation in pixel space

  PixelUMM unifies image and video understanding and generation in pixel space, delivering comparable performance without pretrained visual encoders or VAE tokenizers.

  Oct 2026

• Chinese Techcs.CV

  Explicit weather transitions improve control in generated future-scene videos

  MeteoVerse generates future videos from a single image with explicit control over weather transitions including introduction, preservation, and removal.

  Harbin Institute of Technology, Huawei · Oct 2026

• Chinese Techcs.LG

  Shared-prefix training accelerates reinforcement learning for hybrid-attention agents

  HARTS accelerates agentic RL training on hybrid-attention models by compressing shared rollout-tree prefixes and planning compact packed executions.

  Ant Group · Sept 2026

• Big Techcs.AR

  AI chips can hide most of flash memory’s read delays

  SlipStream prefetcher achieves 92.4% of ideal HBF performance with a 40 MB staging buffer for LLM inference.

  Georgia Tech, NVIDIA Research · Nov 2026

• Big Techcs.LG

  LLM serving becomes batch-invariant with near-stock speed

  Detent achieves batch-invariant LLM serving and RL rollouts at near-stock speed by pinning only the batch-dependent values that affect kernel selection.

  Boston University, Google · Nov 2026

• Top Universitycs.CV

  Larger vision-language models can be worse on specific subtasks

  In vision-language model families, larger models can underperform smaller ones on specific subtasks, including spatial reasoning, visual perception, and STEM questions.

  University of Edinburgh, Imperial College London · Oct 2026

• Independentcs.CL

  Safety checks inside a model can flag harmful output as it streams

  A guardrail built into a large language model's own internal signals detects unsafe text as it is generated, with little added serving cost.

  Nov 2026

• Big Techcs.AI

  Many learned transcoder features closely reproduce individual transformer neurons

  Many transcoder features are just MLP neurons in disguise, providing little new structure for interpretability.

  Anthropic Fellows, Georgia Institute of Technology · Oct 2026

• Top Universitycs.LG

  When specialized training data helps depends on the learning rate

  Optimal training requires designing data and learning rate schedules jointly, as their interaction affects final loss.

  Tsinghua University, Peng Cheng Laboratory · Oct 2026

• Big Techcs.AR

  Weight mapping algorithm for 3D AI chips boosts throughput 2x

  CAMP improves compute utilization on 3D-IMC arrays from 42.4% to 60.3%, achieving up to 2.0x throughput and 1.74x end-to-end gains over prior mapping.

  IBM Research · Nov 2026

• Top Universitycs.LG

  Specialized GFlowNet mixtures find more modes without extra oracle calls

  A population of GFlowNets with a learned classifier divides state space to improve mode coverage beyond single-policy training.

  Mila – Québec AI Institute, Intel · Oct 2026

• Big Techcs.LG

  In-distribution accuracy does not predict abstract reasoning transfer in small models

  Small language models can score well on abstract reasoning benchmarks without acquiring transferable rules; performance degrades sharply under shifts such as grid-scale changes.

  Leibniz University of Hanover, TIB – Leibniz Information Centre for Science and Technology and University Library · Oct 2026

• Chinese Techcs.LG

  Upcycling pretrained LLMs with layer-wise loops breaks the two-loop saturation limit

  UpLoop converts a pretrained non-looped LLM into a recurrent model, raising 20-benchmark average accuracy from 51.90 to 58.07 with three local executions.

  Inclusion AI, Ant Group, Shanghai Jiao Tong University · Oct 2026

• Top Universitycs.AI

  Lower reconstruction loss can degrade quantized LLM performance; robust refinement fixes it

  Distributionally robust quantization (DRQ) minimizes worst-case reconstruction loss under constrained activation shifts, avoiding the calibration-loss trap.

  University of Science and Technology of China, University College London · Oct 2026

• Chinese Techcs.SD

  Audio-language model suite adds speaker-aware transcription and understanding

  Qwen-Audio-3.1-ASR improves multilingual speech recognition and adds speaker-labeled, timestamped transcription and broader audio understanding through variant-specific capabilities.

  Alibaba Group · Oct 2026

• Industrycs.LG

  Reinforcement learning helps coding agents finish tasks across benchmarks

  RL on 1,700 expert coding tasks improves a 1T-parameter model on six benchmarks and shortens agent steps.

  Surge AI · Oct 2026

• AI Startupcs.LG

  Training models for deployment preserves quality under four-bit quantization

  QATFactory enables quantization-aware training that improves deployed LLM quality over post-training quantization for NVFP4, MXFP4, and Q4_K formats.

  Together AI, University of Illinois Urbana-Champaign · Oct 2026

• Big Techcs.LG

  Sparse feed-forward activations cut language-model inference costs substantially

  Model casting and LoPA gating sparsify FFN activations, achieving up to 3.31x GPU speedup with minimal quality loss.

  Meta FAIR, École Normale Supérieure – PSL · Oct 2026

• Big Techcs.LG

  A lightweight PyTorch-native framework matches Megatron-based agentic RL training performance.

  Molt provides scalable PyTorch-native agentic RL training in 9.2K lines, with trillion-parameter support, asynchronous execution, and distributed experience storage.

  NVIDIA · July 2026