Facebook Follies
What took down Facebook, Instagram, and WhatsApp
Facebook inadvertently broke Facebook, Instagram, and WhatsApp for more than five hours on Monday, the company said Monday night.
"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication," Facebook's Santosh Janardhan posted. "This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt." He said they "have no evidence that user data was compromised as a result of this downtime."
Doug Madory at network monitoring company Kentik said someone at Facebook — probably accidentally, but possibly maliciously — sent an update to the company's Border Gateway Protocol (BGP) records, the mechanism internet service providers use to route traffic to domain names.
"In simpler terms," internet security journalist Brian Krebs explained, "sometime this morning Facebook took away the map telling the world's computers how to find its various online properties. As a result, when one types Facebook.com into a web browser, the browser has no idea where to find Facebook.com." After the routine BGP update went wrong, the same blackout prevented employees from undoing the change remotely or at Facebook's physical facilities, because everything from Facebook's internal communications to its electronic door locks are tied to the same stranded domains, Krebs added.
"Facebook eventually restored service after a team got access to its server computers at a data center in Santa Clara, California," and were able to reset them, The New York Times reports, citing three people with knowledge of the matter. In an internal memo, Facebook's global security operations center deemed the outage "a HIGH risk to the People, MODERATE risk to Assets, and a HIGH risk to the Reputation of Facebook."
Facebook was already facing a whistleblower, Frances Haugen, who is sharing company secrets on TV, with news organizations, and with Congress. And "the outage came the same day Facebook asked a federal judge that a revised antitrust complaint against it by the Federal Trade Commission be dismissed because it faces vigorous competition from other services," The Associated Press reports. In fact, the company inadvertently proved Monday that "despite the presence of Twitter, Telegram, Signal, TikTok, Snapchat, and a bevy of other platforms, nothing can easily replace the social network that over the past 17 years has effectively evolved into critical infrastructure," especially in the developing world.