Facebook Outage Reveals Critical DevOps Lessons … Again
June 25, 2014

Mehdi Daoudi

Share this

If you live in the United States, there’s a good chance you had no idea that the Internet turned into a wide-ranging traffic accident last week when Facebook went down for half an hour. This is because the outage occurred on Thursday morning at around 3:50 am EDT, meaning that West Coast night owls were the only group on the continent that was really affected.

Elsewhere, however, it was a different story. Due to the time difference, Europe experienced the outage during early business hours, and much of Asia saw it happen in the late afternoon, resulting in widespread website failures during critical points in the day.

Now you may be asking why a social media site experiencing problems would be that big of an issue for business. Heck, given how much people procrastinate on Facebook, one might even wonder if the outage led to an increase in productivity.

The answer lies with the massive Internet footprint that Facebook carries. Many sites rely on the social media giant for third party services like login, commenting, and sharing platforms, so when Facebook is completely unavailable, it can wreak havoc on thousands of other sites as well. We saw plenty of examples of this last week during the outage.

Every one of those red dots in the above graphic represents a document complete from a specific location that took 30+ seconds. Because Facebook was blocking the document complete, the user experience was dramatically impacted, resulting in many infuriating pinwheels and hourglasses spinning over and over.

This is a perfect example of what is known in the DevOps world as a Single Point of Failure (SPOF). When one component of a website can render the entire thing completely unavailable if not functioning properly, it becomes a weak link that compromises the strength of the entire chain.

From a DevOps perspective, what is needed is a detailed plan in place to serve as a backup in case the third party service goes down. In the case of this latest Facebook outage, the problem lay with the fact that many sites, rather than using the asynchronous tags that Facebook provides, were using outdated ones that block document complete. These new tags, had they been applied to the affected sites, would have prevented any bad user experience and allowed the rest of the site to continue to function normally even if the Facebook components weren’t working.

This risk is not exclusive to Facebook, however; it’s one that is an inherent aspect of all third party services. Facebook may be one of the largest providers of these services, but they’re hardly alone.

The lesson learned from this experience – and one that most European or Asian sites are likely more aware of since the outage had a much greater effect on their businesses – is to build processes that ensure that you stay up to date with vendor changes. For example, Facebook began offering their asynchronous tags in late 2012, but nearly a year and a half later, many sites were clearly not yet using them due to the widespread performance issues that we saw during that half-hour window on Thursday morning.

Facebook’s login API, however, is a separate matter altogether. While asynchronous tags will prevent an entire page from being slowed down by a single non-critical element like sharing or commenting, if your site is inaccessible without a properly functioning login system, you’re facing a much greater problem. The solution here, therefore, is to have an alternative in-house login system in place so that your site is not relying on a single third party component that is ultimately outside of your control.

Identifying a SPOF is only the first step. Once located, implementing asynchronous tags or alternative solutions will prevent the SPOF from existing, thus proving a reliable and fast website.

Mehdi Daoudi is CEO and Co-Founder of Catchpoint
Share this

The Latest

March 16, 2018

The State of the Mainframe report from Syncsort revealed an increased focus on traditional data infrastructure optimization to control costs and help fund strategic organizational projects like AI, machine learning and predictive analytics in addition to widespread concern about meeting security and compliance requirements ...

March 15, 2018

The 2018 Software Fail Watch report from Tricentis investigated 606 failures that affected over 3.6 billion people and caused $1.7 trillion in lost revenue ...

March 14, 2018

Gartner predicts there will be nearly 21 billion connected “things” in use worldwide by 2020 – impressive numbers that should catch the attention of every CIO. IT leaders in nearly every vertical market will soon be inundated with the management of both the data from these devices as well as the management of the devices themselves, each of which require the same lifecycle management as any other IT equipment. This can be an overwhelming realization for CIOs who don’t have an adequate configuration management strategy for their current IT environments, the foundation upon which all future digital strategies – Internet-connected or otherwise – will be built ...

March 13, 2018

Many network operations teams question if they need to TAP their networks; perhaps they aren't familiar with test access points (TAPs), or they think there isn't an application that makes sense for them. Over the past decade, industry best-practice revealed that all network infrastructure should utilize a network TAP as the foundation for complete visibility. The following are the seven most popular applications for TAPs ...

March 12, 2018

Organizations are eager to adopt cloud based architectures in an effort to support their digital transformation efforts, drive efficiencies and strengthen customer satisfaction, according to a new online cloud usage survey conducted by Denodo ...

March 09, 2018

Globally, cloud data center traffic will represent 95 percent of total data center traffic by 2021, compared to 88 percent in 2016, according to the Cisco Global Cloud Index (2016-2021) ...

March 08, 2018

Enterprise cloud spending will grow rapidly over the next year, and yet 35 percent of cloud spend is wasted, according to The RightScale 2018 State of the Cloud Survey ...

March 07, 2018

What often goes overlooked in our always-on digital culture are the people at the other end of each of these services tasked with their 24/7 management. If something goes wrong, users are quick to complain or switch to a competitor as IT practitioners on the backend race to rectify the situation. A recent PagerDuty State of IT Work-Life Balance Report revealed that IT professionals are struggling with the pressures associated with the management of these digital offerings ...

March 06, 2018

Businesses everywhere continually strive for greater efficiency. By way of illustration, more than a third of IT professionals cite "moving faster" as their top goal for 2018, and improving the efficiency of operations was one of the top three stated business objectives for organizations considering digital transformation initiatives ...

March 05, 2018

One of the current challenges for IT teams is the movement of the network to the cloud, and the lack of visibility that comes with that shift. While there has been a lot of hype around the benefits of cloud computing, very little is being said about the inherent drawbacks ...