The Fastly outage in June 2021 showed how one inconspicuous coding error can cause worldwide chaos. A single Fastly customer making a legitimate configuration change, triggered a hidden bug that sent half of the internet offline, including web giants like Amazon and Reddit. Ultimately, this incident illustrates why organizations must test their software in production.

Businesses have increasingly adopted continuous integration and delivery tools and practices to support modern Quality Engineering efforts. However, CI/CD tools live almost entirely on the left-hand side of the software development life cycle, providing testing and monitoring only during pre-production. But testing is just as important on the right-hand side — the production side — where customers are actually using software. It's simply impossible to catch all bugs in pre-production. If orgs don't continue to test production apps, they're dramatically reducing their chances of detecting hidden bugs before they impact customers.

With shortening software development cycles, it's getting even harder to catch bugs in pre-production. Today, customers expect app updates — complete with cool new features and other upgrades — on a more frequent basis. As a result, software engineering teams are under increasing pressure to develop new app releases quicker and quicker. In the past, when new app versions only came out every few months or so, the pre-production phase lasted longer, giving engineers more time to test and look for bugs before production. Now, new app versions are coming out every week or two, leaving engineers less time to find coding errors in pre-production.

Testing and monitoring in production doesn't just give organizations more time to find quality issues, it also provides them more information that makes identifying errors much easier in the future. Once apps are being used by customers, enterprises are constantly collecting important data and feedback from those customers (i.e. crash rates, bounce rates, conversion rates, etc.). This live data provides critical insights — which are unavailable during pre-production — that indicate how a new app release is performing.

This production data gives clues about where a bug may reside. For example, if conversion rates drop in a new app release, it might indicate that there's an error in the code for a "sign up" or "buy now" button that's preventing users from making the desired conversion. Or, if crash rates are higher for a new version of an iOS app, it could mean there's a bug causing fatal iOS app hangs. By closely monitoring this data and using it to help guide testing on production apps, engineering teams can find bugs in production easier, identifying these errors when they're only affecting a few customers and fixing them before they impact all users.

Although testing in production is gradually gaining ground, many mid-sized and large organizations have yet to incorporate comprehensive testing on production apps to achieve rapid iteration. Even major enterprises like Fastly tend to fly blind once apps are in production, lacking the proper tools or best practices to test and monitor these apps for coding errors and stability problems.

This is incredibly risky, as even a seemingly minor coding error can cause apps to crash. Consider what happened last year when a hidden bug in Facebook's iOS software development kit (SDK) caused Spotify, Pinterest, TikTok, Venmo, Tinder, DoorDash and many other top iOS apps to crash upon opening.

With shortened software development lifecycles, these inconspicuous bugs are harder than ever to find during pre-production. Organizations must extend testing to production to have more opportunity to find these errors, understand their potential impact and fix them before they wreak havoc. Fundamentally, this requires a shift in philosophy: Software engineering teams must change how they approach testing. Testing isn't something that's just done rigorously before an app is shipped to production, it's an ongoing process that must be continued throughout the entire life of an app. No app will ever be released completely free of bugs — it's just not possible. Organizations must recognize this and adapt accordingly.