Facebook blames a server configuration change for yesterday’s outage
Facebook has finally released its assessment of what was responsible for yesterday’s massive outage of its Instagram, Messenger, and core Facebook service.
Contrary to the thesis circulating yesterday that it was a route leak, the company said that a server configuration change was to blame.
Yesterday, as a result of a server configuration change, many people had trouble accessing our apps and services. We’ve now resolved the issues and our systems are recovering. We’re very sorry for the inconvenience and appreciate everyone’s patience.
— Facebook (@facebook) March 14, 2019
Millions of people were impacted by the outage yesterday, which had users taking to Twitter (and our tips inbox) to vent their frustration or share theories about what was happening inside the company’s networks.
The Facebook outage was the largest ever recorded by the service Downdectector.
“By duration, this is by far the largest outage we have seen since the launch of Downdetector in 2012,” said Tom Sanders, co-founder of Downdetector. “Our systems processed about 7.5 million problem reports from end users over the course of this incident. Never before have we such a large scale outage.”
Network monitoring firms first began specualting about the cause of the outage yesterday.
“The cause would appear to be internal rather than a network or Internet delivery issue–for example we saw ‘500 internal server errors’ from Facebook,” a ThousandEyes spokesperson wrote in an email. “Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.”
Netscout, which as we reported initially claimed the outage was due to a BGP route leak, later retracted that explanation.
Roland Dobbins, principle engineer at Netscout’s Assert team, says he has no data whatsoever to support the claim that a BGP leak is the cause of today’s Facebook or Instagram outages. “There was an internal miscommunication here,” he says of the email PR people sent to reporters
— Dan Goodin (@dangoodin001) March 13, 2019
Update: Facebook, Instagram and Messenger were down for many users