What the Facebook Outage Teaches Us About Error Monitoring
May 28, 2020

James Smith
Bugsnag

Share this

On Wednesday, May 6th, iOS users all over the world experienced an app crash when they tried to open popular apps such as TikTok, GroupMe, Spotify, and Pinterest.

How did simultaneous crashes occur across so many independent apps? What's the common thread that would cause widespread app crashes?

Turns out, it was a change in behavior in the Facebook API. Some may wonder what Facebook has to do with logging into your favorite music source or social media app. And the answer is: Quite a lot, actually.

Most major consumer apps connect to Facebook to utilize functions such as seamless login, share functionality, and advertising insights. When the API change impacted Facebook's iOS SDK, consumer apps were the first in line to discover the issue, thanks to deep integrations with Facebook that occur at application launch.

What Facebook's outage demonstrates is how important it is to invest in an error monitoring solution that is built to handle widespread and concurrent crash reports without dropping them, as well as the importance of practicing defensive programming.


Why Your Selection of an Error Monitoring Solution Matters

When the Facebook issue occurred, every error monitoring provider saw an immediate and direct impact on the iOS apps they support. A huge volume of crash reports occurred simultaneously within minutes of the issue being reported on Github.

Naturally, this type of error spike dramatically increases the volume of crash reports ingested by error monitoring services. With a sudden deluge of crash reports, how should an error monitoring solution handle it?

The correct response is to buffer increases in crash reports without dropping any and to scale systems to handle the additional load. A robust error monitoring tool should be able to manage the spike and continue to process events, even if this means the worst-case scenario of a slight delay in processing time. Once the bug is rolled back or fixed (as was the case with Facebook within about three hours), any backlog of events should be addressed quickly.

This response demonstrates what you want from an error monitoring solution, which can be summed up in three points.

1. Acknowledge there's a problem: Error monitoring providers should be first in line to provide information about the source of a problem. Because they have more insight into a widespread issue than any single app provider will have, error monitoring organizations should provide information and updates to customers to keep them abreast of the larger situation and how the problem is being handled.

2. Provide error events and data: When an app starts to see a huge spike in user crashes, developers need to know why. The first thing they do is turn to their error monitoring solution to see where the error is originating. A solid error monitoring solution should be able to process a stream of error data within a reasonable amount of time and provide some clues about what's happening.

3. Deliver, not disable, continuous processing: Delays in processing errors are one thing; suspending processing of errors is another. Disabling error processing is completely unacceptable from any error monitoring provider. The main function of these tools is to provide continuous monitoring so that developers can view, dissect, and measure all errors — in real time or, at worst, in hindsight. Anything less means the tool isn't doing its job.

As the Facebook issue demonstrated for many unsuspecting organizations, challenges arise when you rely on free error monitoring tools. First and foremost, free services are more likely to take the "easy path" when things get tough. Rather than manage an onslaught of errors, these tools may simply shut down.

That's right: Free error monitoring tools are often turned off for the duration (or longer) of a widespread problem, based on a very simple cost/benefit analysis. Since organizations aren't paying directly for the service (collection of data is the "payment"), there's no accountability from these providers, no incentive to manage the situation correctly, and no real customer support.

After all, disabling the service is much easier than handling the crisis. The only loss for the free error monitoring provider? The opportunity to collect more data.

As a result, organizations that rely on free tools don't have the benefit of hindsight. They can't use their own error data to understand what happened, nor do they see any subsequent errors that should've been captured, starting at the time of the original occurrence and extending to the moment the service is turned back on.

This situation perfectly sums up that old adage, You get what you pay for. Sadly, it ain't a lot.

Practice Defensive Programming and Error Monitoring

The SDK issue isn't unique to Facebook. A few weeks ago, almost the exact same thing happened to Doordash, Uber Eats, and other apps that rely on maps when the Google Maps iOS SDK experienced an issue.

The two most important takeaways from these widespread app disasters: 

1. Good SDK design tenets dictate that SDKs should never crash an application. What was missing in both the Facebook and Google Maps cases — and what every app company must have — are defensive programming measures that ensure better handling of malformed data from outside servers. 

2. Error monitoring solutions matter a great deal. In order to understand how outages and errors occur, you must have error processing in real time in order to address a challenge and pinpoint bad code. If you select an error monitoring provider that simply disables its service when things get tough, then you'll remain in the dark. And, with free services, you can pretty much bet on that outcome.

What these outages also demonstrate is the absolute need for good software design and error monitoring processes. Developers must know exactly what app features are controlled remotely and why, where everything is documented, and how to turn off third-party apps when things go sideways without impacting the user experience.  
 
In an interconnected app world, errors are going to happen. The real question is, can you trust your error monitoring system to always have your crash reports? 

James Smith is CEO and Co-Founder of Bugsnag
Share this

The Latest

June 25, 2020

I've had the opportunity to work with a number of organizations embarking on their AIOps journey. I always advise them to start by evaluating their needs and the possibilities AIOps can bring to them through five different levels of AIOps maturity. This is a strategic approach that allows enterprises to achieve complete automation for long-term success ...

June 24, 2020

Sumo Logic recently commissioned an independent market research study to understand the industry momentum behind continuous intelligence — and the necessity for digital organizations to embrace a cloud-native, real-time continuous intelligence platform to support the speed and agility of business for faster decision-making, optimizing security, driving new innovation and delivering world-class customer experiences. Some of the key findings include ...

June 23, 2020

When it comes to viruses, it's typically those of the computer/digital variety that IT is concerned about. But with the ongoing pandemic, IT operations teams are on the hook to maintain business functions in the midst of rapid and massive change. One of the biggest challenges for businesses is the shift to remote work at scale. Ensuring that they can continue to provide products and services — and satisfy their customers — against this backdrop is challenging for many ...

June 22, 2020

Teams tasked with developing and delivering software are under pressure to balance the business imperative for speed with high customer expectations for quality. In the course of trying to achieve this balance, engineering organizations rely on a variety of tools, techniques and processes. The 2020 State of Software Quality report provides a snapshot of the key challenges organizations encounter when it comes to delivering quality software at speed, as well as how they are approaching these hurdles. This blog introduces its key findings ...

June 18, 2020

For IT teams, run-the-business, commodity areas such as employee help desks, device support and communication platforms are regularly placed in the crosshairs for cost takeout, but these areas are also highly visible to employees. Organizations can improve employee satisfaction and business performance by building unified functions that are measured by employee experience rather than price. This approach will ultimately fund transformation, as well as increase productivity and innovation ...