Google Cloud has disclosed the embarrassing root cause of a partial outage at its us-central1-b region this week: an engineer unplugged it. The incident, which occurred on September 1 between 07:41 and 11:52 PT, saw a portion of the G-Cloud region experience "severe network degradation and resource isolation."

According to Google's service health report, "Traffic flow drop rates for resources hosted in the affected area reached 100% during the peak of the incident, resulting in unreachable virtual machines and elevated packet loss."

The preliminary root cause explains that Google designs its datacenters with redundancy across multiple routing devices. "The system is designed to be resilient to all single device or fiber-path failures, and most double-or triple-failures do not affect customer traffic. To ensure this, the devices and fiber paths are physically separated in each datacenter, with diverse power sources," the document states.

However, all that redundancy was defeated when the engineer pulled out every cable. "The immediate technical trigger for this event was the inadvertent physical disconnection of network fiber-optic cables during a routine hardware maintenance procedure," Google explained. "A procedural error meant that the physical maintenance action sequentially unplugged 100% of fiber paths across all devices within 13 minutes. The nature of the error, combined with the speed of the action, prevented warnings of incorrect action reaching the engineer before complete disconnection."

Once the routers were disconnected, virtual machines in one zone of the us-central1-b region lost contact with the outside world. They could still communicate with each other, but users could not access their VMs to diagnose the problem. Google responded by moving traffic away from the affected routers, allowing its resilience provisions to kick in and automatically shift traffic to healthy capacity elsewhere in the region. Technicians then "identified the disconnected optical links and physically reseated the fibers. Once the physical links were fully restored, traffic flow rates normalized and the traffic was redirected back to return the capacity to service."

In summary, a Google Cloud zone went offline because an engineer pulled out all the cables connecting it to the internet. Once the cables were reconnected, the system resumed normal operation.

The Register notes that while companies like Google create detailed procedures for safely maintaining infrastructure, this incident appears to have resulted from a failure to follow what is arguably one of the most basic rules of technology: RTFM (Read The Friendly Manual).