stackery.co

The Meta Blackout: How One Command Erased Facebook From the Internet for Six Hours

On 4 October 2021, a configuration change during routine backbone maintenance withdrew Facebook's BGP routes, making every Meta service unreachable for more than six hours. The real problem was not the initial failure but the architectural decision that made recovery tools unreachable alongside the product.

Last updated 2026-09-05

The background

By October 2021, Meta's collection of platforms had become something closer to infrastructure than optional service. Facebook, Instagram, WhatsApp, and Messenger collectively served over 3.5 billion users. In countries across the global south, WhatsApp was the primary means of business communication. Instagram had become the shopfront for millions of small enterprises. Facebook Messenger handled personal and commercial conversations that would once have travelled through telecom networks. Oculus connected the virtual reality community. Mapillary provided crowdsourced street imagery.

This was not simply a social media company. It was a communications backbone that much of the world's population had come to treat as reliable as electricity. Businesses scheduled product launches around it. Customer service departments routed through it. Family members coordinated care for elderly relatives over it. The assumption of availability was so deeply embedded that alternative channels had atrophied.

Meta's engineering operation matched that scale. The infrastructure team managed one of the world's largest private networks, with data centres distributed globally and backbone connectivity stitched together through careful routing protocols. Their monitoring systems tracked millions of metrics. Their tooling allowed engineers to diagnose and repair problems from anywhere. The entire operation reflected sophisticated understanding of how to build resilient systems at planetary scale.

Routine maintenance was part of that sophistication. Large networks require constant capacity planning, and backbone audits ensure sufficient headroom for growth. These audits involve examining routing configurations, identifying bottlenecks, and occasionally adjusting how traffic flows between facilities. Standard procedure. Low-risk work performed by experienced teams with established processes. The sort of task that happens dozens of times without incident before it happens once with consequences.

What actually happened

Monday morning, 4 October 2021, began with a routine audit of spare capacity on Facebook's backbone network. This was maintenance work, the kind that keeps infrastructure healthy but rarely makes headlines. Engineers were reviewing how the backbone routers managed traffic between data centres, looking for opportunities to optimise capacity allocation.

Sometime during that review, a configuration change was prepared and applied to backbone routers. The change went live between 15:39 and 15:45 UTC—mid-morning on the American east coast, afternoon in Europe, evening in parts of Asia. Within that six-minute window, Facebook's Border Gateway Protocol routes were withdrawn.

BGP is the routing protocol that holds the internet together. It is how networks tell each other where to send traffic. When you type a domain name, your request travels through multiple networks, each one consulting BGP to determine the next hop. Withdrawing BGP routes is the digital equivalent of removing every road sign pointing to your destination. The destination still exists. The buildings are still standing. But nobody can find the way there.

Facebook's backbone network disappeared from the internet's routing tables. Every other network on the planet stopped being able to direct traffic toward Meta's services. The effect was immediate and absolute. Facebook, Instagram, WhatsApp, Messenger, Oculus, and Mapillary became unreachable simultaneously across the globe.

Within minutes, the DNS failure cascaded. Facebook's DNS records carried a five-minute time-to-live value, meaning other servers would cache the records for only five minutes before checking again. As those caches expired, servers worldwide tried to refresh their knowledge of where Facebook's services lived. They found nothing. The DNS servers that would normally answer those queries were themselves behind the vanished BGP routes, equally unreachable. The services were running, the servers were powered on, the data was intact—but the internet had lost the map.

The spike on Downdetector was one of the largest in that service's history. Users across the globe reported every Meta property down simultaneously. The correlation was immediate and obvious: this was not a gradual degradation or a regional problem. Everything had stopped at once.

Back at Meta's facilities, engineers quickly identified the BGP withdrawal as the proximate cause. The solution seemed straightforward: reverse the configuration change, restore the routes, wait for propagation. Except they could not reach the systems they needed to reach. The internal monitoring tools were unreachable. The remote access infrastructure was unreachable. The recovery systems built to handle exactly this sort of incident were all running on the same infrastructure that had just vanished from the internet.

Hours passed. The only option remaining was physical access. Engineers had to travel to data centres, present themselves to security, gain physical access to the equipment, and restore connectivity manually. This was not a five-minute fix deployed from a laptop. This was getting in cars, driving to buildings, and touching hardware. Service was gradually restored at approximately 22:45 UTC, more than six hours after the initial withdrawal.

The people in the room

Nobody sets out to disconnect 3.5 billion people from their primary communication platform. The engineers conducting that backbone audit were doing exactly what they were supposed to do: maintaining infrastructure proactively, reviewing capacity during a quiet operational window, applying changes designed to improve performance and reliability. These were not inexperienced operators experimenting with unfamiliar systems. This was routine work performed by people who understood the architecture deeply.

The configuration change that triggered the cascade was almost certainly reviewed. Meta's engineering culture emphasised process and peer review. Someone prepared the change, someone else likely examined it, and multiple people would have considered the risks before deployment. The failure was not individual carelessness but collective blind spot: nobody in that chain anticipated that the change would withdraw all BGP routes simultaneously, and nobody had recently tested whether the recovery tooling would survive that particular failure mode.

Routine maintenance creates its own psychological risks. When you have performed similar work dozens or hundreds of times without incident, the vigilance erodes slightly each time. The checklist becomes familiar enough that you stop truly questioning each item. The review becomes a ritual rather than a genuine interrogation of assumptions. This was not negligence. This was the normal human response to repetitive tasks in complex systems, the same dynamic that causes incidents across the technology industry with weary regularity.

The damage

What actually went wrong

The technical failure had two distinct layers, and both were necessary for the outage to reach its eventual duration and severity.

The first layer was the BGP withdrawal itself. Border Gateway Protocol is the internet's routing fabric, the system by which networks advertise their presence and learn routes to other networks. When Facebook's backbone routers withdrew their BGP advertisements, they effectively told every other network on the internet to stop sending traffic their direction. This was not a subtle degradation or a partial failure. Facebook's autonomous system numbers vanished from routing tables globally. As far as the internet's routing infrastructure was concerned, Facebook simply ceased to exist.

The DNS failure followed inevitably from the routing failure. Domain Name System servers translate human-readable names into machine-readable addresses, but those DNS servers are themselves addressed through the routing system. Facebook's DNS records had a five-minute TTL, meaning cached copies expired quickly. As caches expired across the internet, queries for facebook.com and instagram.com and whatsapp.com reached out to Meta's authoritative DNS servers and found nothing. You cannot look up an address from a server you cannot reach.

Many organisations have suffered BGP incidents. Most recover within minutes or tens of minutes, because their monitoring systems detect the problem immediately and their engineers can access recovery tools remotely. This brings us to the second layer: architectural coupling that transformed a fixable problem into an hours-long ordeal.

Meta's internal tooling—the monitoring systems, the remote access infrastructure, the diagnostic interfaces, the recovery automation—all ran on the infrastructure that had just become unreachable. This was not an oversight in the traditional sense. Running internal tools on your own infrastructure is efficient, reduces external dependencies, and simplifies authentication and security. But it creates a dependency loop that fails catastrophically when the infrastructure itself becomes unreachable from the outside.

The engineers who could fix the problem could not reach the systems they needed to fix. They could not log in remotely because the remote access systems were behind the vanished routes. They could not check monitoring because the monitoring ran on the same unreachable network. They could not deploy automated recovery because the automation lived in the same dark space. The company connecting 3.5 billion people could not connect its own engineers to its own servers, and ended up sending them to the building with physical keys.

Facebook published a detailed post-incident report confirming this sequence: the backbone network went down before DNS, and the internal recovery infrastructure failed alongside the product. This was not speculation or external analysis. This was Meta's own engineering team documenting exactly how their recovery architecture failed them when they needed it most.

What small businesses can learn

Sources

Get the shortlist, not the noise

One email a week. The tool we would actually buy, and why.

Join the newsletter