What the Facebook Outage Teaches Us About Error Monitoring
May 28, 2020

James Smith
SmartBear

Share this

On Wednesday, May 6th, iOS users all over the world experienced an app crash when they tried to open popular apps such as TikTok, GroupMe, Spotify, and Pinterest.

How did simultaneous crashes occur across so many independent apps? What's the common thread that would cause widespread app crashes?

Turns out, it was a change in behavior in the Facebook API. Some may wonder what Facebook has to do with logging into your favorite music source or social media app. And the answer is: Quite a lot, actually.

Most major consumer apps connect to Facebook to utilize functions such as seamless login, share functionality, and advertising insights. When the API change impacted Facebook's iOS SDK, consumer apps were the first in line to discover the issue, thanks to deep integrations with Facebook that occur at application launch.

What Facebook's outage demonstrates is how important it is to invest in an error monitoring solution that is built to handle widespread and concurrent crash reports without dropping them, as well as the importance of practicing defensive programming.


Why Your Selection of an Error Monitoring Solution Matters

When the Facebook issue occurred, every error monitoring provider saw an immediate and direct impact on the iOS apps they support. A huge volume of crash reports occurred simultaneously within minutes of the issue being reported on Github.

Naturally, this type of error spike dramatically increases the volume of crash reports ingested by error monitoring services. With a sudden deluge of crash reports, how should an error monitoring solution handle it?

The correct response is to buffer increases in crash reports without dropping any and to scale systems to handle the additional load. A robust error monitoring tool should be able to manage the spike and continue to process events, even if this means the worst-case scenario of a slight delay in processing time. Once the bug is rolled back or fixed (as was the case with Facebook within about three hours), any backlog of events should be addressed quickly.

This response demonstrates what you want from an error monitoring solution, which can be summed up in three points.

1. Acknowledge there's a problem: Error monitoring providers should be first in line to provide information about the source of a problem. Because they have more insight into a widespread issue than any single app provider will have, error monitoring organizations should provide information and updates to customers to keep them abreast of the larger situation and how the problem is being handled.

2. Provide error events and data: When an app starts to see a huge spike in user crashes, developers need to know why. The first thing they do is turn to their error monitoring solution to see where the error is originating. A solid error monitoring solution should be able to process a stream of error data within a reasonable amount of time and provide some clues about what's happening.

3. Deliver, not disable, continuous processing: Delays in processing errors are one thing; suspending processing of errors is another. Disabling error processing is completely unacceptable from any error monitoring provider. The main function of these tools is to provide continuous monitoring so that developers can view, dissect, and measure all errors — in real time or, at worst, in hindsight. Anything less means the tool isn't doing its job.

As the Facebook issue demonstrated for many unsuspecting organizations, challenges arise when you rely on free error monitoring tools. First and foremost, free services are more likely to take the "easy path" when things get tough. Rather than manage an onslaught of errors, these tools may simply shut down.

That's right: Free error monitoring tools are often turned off for the duration (or longer) of a widespread problem, based on a very simple cost/benefit analysis. Since organizations aren't paying directly for the service (collection of data is the "payment"), there's no accountability from these providers, no incentive to manage the situation correctly, and no real customer support.

After all, disabling the service is much easier than handling the crisis. The only loss for the free error monitoring provider? The opportunity to collect more data.

As a result, organizations that rely on free tools don't have the benefit of hindsight. They can't use their own error data to understand what happened, nor do they see any subsequent errors that should've been captured, starting at the time of the original occurrence and extending to the moment the service is turned back on.

This situation perfectly sums up that old adage, You get what you pay for. Sadly, it ain't a lot.

Practice Defensive Programming and Error Monitoring

The SDK issue isn't unique to Facebook. A few weeks ago, almost the exact same thing happened to Doordash, Uber Eats, and other apps that rely on maps when the Google Maps iOS SDK experienced an issue.

The two most important takeaways from these widespread app disasters: 

1. Good SDK design tenets dictate that SDKs should never crash an application. What was missing in both the Facebook and Google Maps cases — and what every app company must have — are defensive programming measures that ensure better handling of malformed data from outside servers. 

2. Error monitoring solutions matter a great deal. In order to understand how outages and errors occur, you must have error processing in real time in order to address a challenge and pinpoint bad code. If you select an error monitoring provider that simply disables its service when things get tough, then you'll remain in the dark. And, with free services, you can pretty much bet on that outcome.

What these outages also demonstrate is the absolute need for good software design and error monitoring processes. Developers must know exactly what app features are controlled remotely and why, where everything is documented, and how to turn off third-party apps when things go sideways without impacting the user experience.  
 
In an interconnected app world, errors are going to happen. The real question is, can you trust your error monitoring system to always have your crash reports? 

James Smith is SVP of the Bugsnag Product Group at SmartBear
Share this

The Latest

December 07, 2021

The Holiday Season means it is time for APMdigest's annual list of Application Performance Management (APM) predictions, the most popular content on APMdigest, viewed by tens of thousands of people in the IT community around the world for more than a decade. Industry experts offer thoughtful, insightful, and often controversial predictions on how APM, AIOps, Observability, Open Telemetry, and related technologies will evolve and impact business in 2022 ...

December 06, 2021

As organizations strive to advance digital acceleration efforts, outpace competitors, and better service customers, the path to better, more secure software lies in AIOps. As DevOps teams continue to adopt progressive delivery models and the volume of production deployments and configuration changes sees even more growth, here are a few of the things that your DevOps teams should keep in mind, as they look to make the most of their IT toolkits via AIOps ...

December 02, 2021

In the old days of monolithic architectures, IT operations teams could manage service-disrupting incidents themselves. But these architectures have evolved, and the systems our digital economy relies on today are too complex and produce too much data for human operators to monitor, let alone fix. Artificial Intelligence for IT Operations (AIOps) solutions automate system monitoring and remediation strategies to help DevOps and SRE teams ensure that services and apps are continuously available ...

December 01, 2021

As global and emerging technology trends continue to drive the network to evolve at an accelerated pace, we wanted to better understand the current trends and challenges these teams face. As a result, LiveAction conducted a survey of networking professionals that on average manage more than 500 networking devices at organizations with more than 600 employees. Let's dive into four of the key insights revealed in this report ...

November 30, 2021

Thanks to pandemic-related work-from-home (WFH) and digital/mobile customer experience initiatives, employees and users are more distributed than ever. At the same time, organizations everywhere are adopting a cloud-first or cloud-smart architecture, distributing their business applications across private and public cloud infrastructures. Private data centers continue to be consolidated, while more and more branch offices are connecting to data centers and the public cloud simultaneously. Maintaining application performance for distributed users in this increasingly hybrid environment is a significant challenge for IT teams ...