Microsoft’s email and office software went dark from shortly before 5am AEDT on Friday after an upgrade intended to add capacity. A day later, US authorities grounded all departing flights for five hours after a contractor installed a router at a remote site in El Dorado, Arkansas, as part of a project to replace ageing copper lines with fibre.
The two failures were unrelated and Microsoft says it has no indication its services played any part in the grounding. What failed was containment: one fault in one place was allowed to course through an entire network.
For Microsoft, the outage “was introduced by an issue during an earlier network capacity upgrade that surfaced” after a later network change, the company said. Microsoft’s preliminary notices say traffic moved around during that maintenance and exposed a hidden bug in the systems connecting users to their mailboxes. Requests banked up until the servers choked, and work spilled onto healthy servers, which then buckled. Even servers that engineers revived “can become unresponsive again once they begin receiving traffic”, Microsoft warned. Restoration took nearly two days.
Australians felt the impact within hours. ASIO and the federal police lost email, and so did councils, courts and schools. Australia Post pulled its phone payment system, while suppliers could not tell supermarket distribution centres what was on the way. In Germany, 15 hotel guests were reportedly injured by chlorine gas after a pool safety system that sent its warnings by email went silent.
The flight grounding began with a single router in El Dorado, Arkansas, which contractor L3Harris was installing. According to the Federal Aviation Administration, a flood of connection attempts scrambled radio and radar feeds to 17 of the 22 US centres that guide planes between airports, more than three-quarters of the system. US Transportation Secretary Sean Duffy said the routers “got overloaded with data”. Investigators have not yet established whether faulty equipment or a slip during installation was to blame. Flight tracker FlightAware counted about 9800 delays and more than 4700 cancellations, and US-bound flights from Sydney were held on the ground.
Dr Mohammad Goudarzi, who researches cloud computing at Monash University, said the chaos stemmed from misaligned priorities. “Investment in reliability needs to keep pace with investment in new features,” he said.
The immediate question for Australian businesses is who pays. Microsoft customers may be able to claim service credits, the small refunds written into their contracts. Prins Ralston, chief executive of the Australian Computer Society, said those credits “won’t make up for a lost morning of trade for a small business”. A litigation funder, Victoria Peak, is already inviting affected Australian businesses to register for a possible class action, though no case has been filed.
Ralston said there was a clear difference between who coped and who did not. “ASIO and the AFP had workarounds. Most small businesses and many agencies don’t,” he said.
Microsoft has promised a full post-incident report.