Microsoft will expand its Azure Maia family to multi-gigawatt deployments starting in 2027, making Maia the primary inference system for Microsoft products and services, the company announced. The architecture behind that commitment is unusually specific about data movement, but the efficiency claim leading the announcement has drawn questions from practitioners.
The Maia 200 is already in production, serving Microsoft Copilot and the MAI team, with GitHub Copilot and Microsoft Foundry to follow by the end of the year. It is deployed across the United States, Europe, and Australia, and Microsoft says it delivers up to 1.7 times more tokens per watt for built-in AI in Teams and Copilot than leading GPU infrastructure. Maia 300 begins production deployments in 2027.
Maia 300 is inference-first, aimed at long-context, memory-intensive, and decode-heavy work rather than training, which neither announcement mentions. Each accelerator carries eight HBM4 stacks, providing 288 GB and 18 TB/s within a 950-watt TDP. A rack holds 96 accelerators in 167 kW across 24 four-accelerator trays, reaching 27.6 TB of HBM and 1.7 PB/s aggregate. Compute chiplets are built on TSMC N2P, with memory-controller and I/O functions on separate dies, which Microsoft says raises compute FLOPs by 36% and contributes to 2.5 times the memory bandwidth and roughly twice the HBM bandwidth per watt of Maia 200. Memory organization moves from uniform access to a NUMA layout across chiplets so software can place data near the compute using it.
The networking is where the design is most opinionated. Maia's AI network controller implements a custom AI transport built on Ethernet, and Microsoft describes the result as 'one fabric rather than separate scale-up and scale-out domains.' Accelerators initiate communication to any peer in a superpod 'without involving the host or a discrete scale-out NIC.' Two accelerators are directly linked within a tray, a switched fabric joins all 96 in a rack, and optical second-tier switches connect up to 128 racks into a superpod of 12,288 accelerators. The post names three latency domains: a direct pair, one hop within the pod, and three hops across pods, and expects software to keep tensor-parallel traffic in the lowest. The split between in-rack and cross-rack bandwidth is 'not fixed at the system design time' but set by job packing, which is how an accelerator can drive up to 750 GB/s to any peer when only a subset cross racks.
That fabric carries a specific inference pattern. Prefill and decode workers are sized and placed independently, and after prefill completes, KV state 'moves directly to the selected decode workers over the same scale-up fabric rather than leaving the domain through a separate host-mediated scale-out path.' The same fabric lets a spare tray be used for additional capacity.