Fastly Outage Illustrates Importance of Testing in Production
August 30, 2021

James Smith

Share this

The Fastly outage in June 2021 showed how one inconspicuous coding error can cause worldwide chaos. A single Fastly customer making a legitimate configuration change, triggered a hidden bug that sent half of the internet offline, including web giants like Amazon and Reddit. Ultimately, this incident illustrates why organizations must test their software in production.

Businesses have increasingly adopted continuous integration and delivery tools and practices to support modern Quality Engineering efforts. However, CI/CD tools live almost entirely on the left-hand side of the software development life cycle, providing testing and monitoring only during pre-production. But testing is just as important on the right-hand side — the production side — where customers are actually using software. It's simply impossible to catch all bugs in pre-production. If orgs don't continue to test production apps, they're dramatically reducing their chances of detecting hidden bugs before they impact customers.

With shortening software development cycles, it's getting even harder to catch bugs in pre-production. Today, customers expect app updates — complete with cool new features and other upgrades — on a more frequent basis. As a result, software engineering teams are under increasing pressure to develop new app releases quicker and quicker. In the past, when new app versions only came out every few months or so, the pre-production phase lasted longer, giving engineers more time to test and look for bugs before production. Now, new app versions are coming out every week or two, leaving engineers less time to find coding errors in pre-production.

Testing and monitoring in production doesn't just give organizations more time to find quality issues, it also provides them more information that makes identifying errors much easier in the future. Once apps are being used by customers, enterprises are constantly collecting important data and feedback from those customers (i.e. crash rates, bounce rates, conversion rates, etc.). This live data provides critical insights — which are unavailable during pre-production — that indicate how a new app release is performing.

This production data gives clues about where a bug may reside. For example, if conversion rates drop in a new app release, it might indicate that there's an error in the code for a "sign up" or "buy now" button that's preventing users from making the desired conversion. Or, if crash rates are higher for a new version of an iOS app, it could mean there's a bug causing fatal iOS app hangs. By closely monitoring this data and using it to help guide testing on production apps, engineering teams can find bugs in production easier, identifying these errors when they're only affecting a few customers and fixing them before they impact all users.

Although testing in production is gradually gaining ground, many mid-sized and large organizations have yet to incorporate comprehensive testing on production apps to achieve rapid iteration. Even major enterprises like Fastly tend to fly blind once apps are in production, lacking the proper tools or best practices to test and monitor these apps for coding errors and stability problems.

This is incredibly risky, as even a seemingly minor coding error can cause apps to crash. Consider what happened last year when a hidden bug in Facebook's iOS software development kit (SDK) caused Spotify, Pinterest, TikTok, Venmo, Tinder, DoorDash and many other top iOS apps to crash upon opening.

With shortened software development lifecycles, these inconspicuous bugs are harder than ever to find during pre-production. Organizations must extend testing to production to have more opportunity to find these errors, understand their potential impact and fix them before they wreak havoc. Fundamentally, this requires a shift in philosophy: Software engineering teams must change how they approach testing. Testing isn't something that's just done rigorously before an app is shipped to production, it's an ongoing process that must be continued throughout the entire life of an app. No app will ever be released completely free of bugs — it's just not possible. Organizations must recognize this and adapt accordingly.

James Smith is SVP of the Bugsnag Product Group at SmartBear
Share this

The Latest

June 29, 2022

When it comes to AIOps predictions, there's no question of AI's value in predictive intelligence and faster problem resolution for IT teams. In fact, Gartner has reported that there is no future for IT Operations without AIOps. So, where is AIOps headed in five years? Here's what the vendors and thought leaders in the AIOps space had to share ...

June 27, 2022

A new study by OpsRamp on the state of the Managed Service Providers (MSP) market concludes that MSPs face a market of bountiful opportunities but must prepare for this growth by embracing complex technologies like hybrid cloud management, root cause analysis and automation ...

June 27, 2022

Hybrid work adoption and the accelerated pace of digital transformation are driving an increasing need for automation and site reliability engineering (SRE) practices, according to new research. In a new survey almost half of respondents (48.2%) said automation is a way to decrease Mean Time to Resolution/Repair (MTTR) and improve service management ...

June 23, 2022

Digital businesses don't invest in monitoring for monitoring's sake. They do it to make the business run better. Every dollar spent on observability — every hour your team spends using monitoring tools or responding to what they reveal — should tie back directly to business outcomes: conversions, revenues, brand equity. If they don't? You might be missing the forest for the trees ...

June 22, 2022

Every day, companies are missing customer experience (CX) "red flags" because they don't have the tools to observe CX processes or metrics. Even basic errors or defects in automated customer interactions are left undetected for days, weeks or months, leading to widespread customer dissatisfaction. In fact, poor CX and digital technology investments are costing enterprises billions of dollars in lost potential revenue ...

June 21, 2022

Organizations are moving to microservices and cloud native architectures at an increasing pace. The primary incentive for these transformation projects is typically to increase the agility and velocity of software release and product innovation. These dynamic systems, however, are far more complex to manage and monitor, and they generate far higher data volumes ...

June 16, 2022

Global IT teams adapted to remote work in 2021, resolving employee tickets 23% faster than the year before as overall resolution time for IT tickets went down by 7 hours, according to the Freshservice Service Management Benchmark Report from Freshworks ...

June 15, 2022

Once upon a time data lived in the data center. Now data lives everywhere. All this signals the need for a new approach to data management, a next-gen solution ...

June 14, 2022

Findings from the 2022 State of Edge Messaging Report from Ably and Coleman Parkes Research show that most organizations (65%) that have built edge messaging capabilities in house have experienced an outage or significant downtime in the last 12-18 months. Most of the current in-house real-time messaging services aren't cutting it ...

June 13, 2022
Today's users want a complete digital experience when dealing with a software product or system. They are not content with the page load speeds or features alone but want the software to perform optimally in an omnichannel environment comprising multiple platforms, browsers, devices, and networks. This calls into question the role of load testing services to check whether the given software under testing can perform optimally when subjected to peak load ...