The Fastly outage in June 2021 showed how one inconspicuous coding error can cause worldwide chaos. A single Fastly customer making a legitimate configuration change, triggered a hidden bug that sent half of the internet offline, including web giants like Amazon and Reddit. Ultimately, this incident illustrates why organizations must test their software in production.
Businesses have increasingly adopted continuous integration and delivery tools and practices to support modern Quality Engineering efforts. However, CI/CD tools live almost entirely on the left-hand side of the software development life cycle, providing testing and monitoring only during pre-production. But testing is just as important on the right-hand side — the production side — where customers are actually using software. It's simply impossible to catch all bugs in pre-production. If orgs don't continue to test production apps, they're dramatically reducing their chances of detecting hidden bugs before they impact customers.
With shortening software development cycles, it's getting even harder to catch bugs in pre-production. Today, customers expect app updates — complete with cool new features and other upgrades — on a more frequent basis. As a result, software engineering teams are under increasing pressure to develop new app releases quicker and quicker. In the past, when new app versions only came out every few months or so, the pre-production phase lasted longer, giving engineers more time to test and look for bugs before production. Now, new app versions are coming out every week or two, leaving engineers less time to find coding errors in pre-production.
Testing and monitoring in production doesn't just give organizations more time to find quality issues, it also provides them more information that makes identifying errors much easier in the future. Once apps are being used by customers, enterprises are constantly collecting important data and feedback from those customers (i.e. crash rates, bounce rates, conversion rates, etc.). This live data provides critical insights — which are unavailable during pre-production — that indicate how a new app release is performing.
This production data gives clues about where a bug may reside. For example, if conversion rates drop in a new app release, it might indicate that there's an error in the code for a "sign up" or "buy now" button that's preventing users from making the desired conversion. Or, if crash rates are higher for a new version of an iOS app, it could mean there's a bug causing fatal iOS app hangs. By closely monitoring this data and using it to help guide testing on production apps, engineering teams can find bugs in production easier, identifying these errors when they're only affecting a few customers and fixing them before they impact all users.
Although testing in production is gradually gaining ground, many mid-sized and large organizations have yet to incorporate comprehensive testing on production apps to achieve rapid iteration. Even major enterprises like Fastly tend to fly blind once apps are in production, lacking the proper tools or best practices to test and monitor these apps for coding errors and stability problems.
This is incredibly risky, as even a seemingly minor coding error can cause apps to crash. Consider what happened last year when a hidden bug in Facebook's iOS software development kit (SDK) caused Spotify, Pinterest, TikTok, Venmo, Tinder, DoorDash and many other top iOS apps to crash upon opening.
With shortened software development lifecycles, these inconspicuous bugs are harder than ever to find during pre-production. Organizations must extend testing to production to have more opportunity to find these errors, understand their potential impact and fix them before they wreak havoc. Fundamentally, this requires a shift in philosophy: Software engineering teams must change how they approach testing. Testing isn't something that's just done rigorously before an app is shipped to production, it's an ongoing process that must be continued throughout the entire life of an app. No app will ever be released completely free of bugs — it's just not possible. Organizations must recognize this and adapt accordingly.
The Latest
APMdigest and leading IT research firm Enterprise Management Associates (EMA) are partnering to bring you the EMA-APMdigest Podcast, a new podcast focused on the latest technologies impacting IT Operations. In Episode 2 - Part 2 Pete Goldin, Editor and Publisher of APMdigest, discusses Network Observability with Shamus McGillicuddy, Vice President of Research, Network Infrastructure and Operations, at EMA ...
Most organizations suffer from some form of alert noise. Alert noise is only going to increase as organizations support cloud-native applications spanning multiple public and private clouds, including ephemeral deployments and more. It's not going to get easier for organizations to understand the signal from all those alerts being sent. So what can be done about it? ...
This blog presents the case for a radical new approach to basic information technology (IT) education. This conclusion is based on a study of courses and other forms of IT education which purport to cover IT "fundamentals" ...
To achieve maximum availability, IT leaders must employ domain-agnostic solutions that identify and escalate issues across all telemetry points. These technologies, which we refer to as Artificial Intelligence for IT Operations, create convergence — in other words, they provide IT and DevOps teams with the full picture of event management and downtime ...
APMdigest and leading IT research firm Enterprise Management Associates (EMA) are partnering to bring you the EMA-APMdigest Podcast, a new podcast focused on the latest technologies impacting IT Operations. In Episode 2 - Part 1 Pete Goldin, Editor and Publisher of APMdigest, discusses Network Observability with Shamus McGillicuddy, Vice President of Research, Network Infrastructure and Operations, at EMA ...
CIOs have stepped into the role of digital leader and strategic advisor, according to the 2023 Global CIO Survey from Logicalis ...
Synthetic monitoring is crucial to deploy code with confidence as catching bugs with E2E tests on staging is becoming increasingly difficult. It isn't trivial to provide realistic staging systems, especially because today's apps are intertwined with many third-party APIs ...
Recent EMA field research found that ServiceOps is either an active effort or a formal initiative in 78% of the organizations represented by a global panel of 400+ IT leaders. It is relatively early but gaining momentum across industries and organizations of all sizes globally ...
Managing availability and performance within SAP environments has long been a challenge for IT teams. But as IT environments grow more complex and dynamic, and the speed of innovation in almost every industry continues to accelerate, this situation is becoming a whole lot worse ...
Harnessing the power of network-derived intelligence and insights is critical in detecting today's increasingly sophisticated security threats across hybrid and multi-cloud infrastructure, according to a new research study from IDC ...