The Fastly outage in June 2021 showed how one inconspicuous coding error can cause worldwide chaos. A single Fastly customer making a legitimate configuration change, triggered a hidden bug that sent half of the internet offline, including web giants like Amazon and Reddit. Ultimately, this incident illustrates why organizations must test their software in production.
Businesses have increasingly adopted continuous integration and delivery tools and practices to support modern Quality Engineering efforts. However, CI/CD tools live almost entirely on the left-hand side of the software development life cycle, providing testing and monitoring only during pre-production. But testing is just as important on the right-hand side — the production side — where customers are actually using software. It's simply impossible to catch all bugs in pre-production. If orgs don't continue to test production apps, they're dramatically reducing their chances of detecting hidden bugs before they impact customers.
With shortening software development cycles, it's getting even harder to catch bugs in pre-production. Today, customers expect app updates — complete with cool new features and other upgrades — on a more frequent basis. As a result, software engineering teams are under increasing pressure to develop new app releases quicker and quicker. In the past, when new app versions only came out every few months or so, the pre-production phase lasted longer, giving engineers more time to test and look for bugs before production. Now, new app versions are coming out every week or two, leaving engineers less time to find coding errors in pre-production.
Testing and monitoring in production doesn't just give organizations more time to find quality issues, it also provides them more information that makes identifying errors much easier in the future. Once apps are being used by customers, enterprises are constantly collecting important data and feedback from those customers (i.e. crash rates, bounce rates, conversion rates, etc.). This live data provides critical insights — which are unavailable during pre-production — that indicate how a new app release is performing.
This production data gives clues about where a bug may reside. For example, if conversion rates drop in a new app release, it might indicate that there's an error in the code for a "sign up" or "buy now" button that's preventing users from making the desired conversion. Or, if crash rates are higher for a new version of an iOS app, it could mean there's a bug causing fatal iOS app hangs. By closely monitoring this data and using it to help guide testing on production apps, engineering teams can find bugs in production easier, identifying these errors when they're only affecting a few customers and fixing them before they impact all users.
Although testing in production is gradually gaining ground, many mid-sized and large organizations have yet to incorporate comprehensive testing on production apps to achieve rapid iteration. Even major enterprises like Fastly tend to fly blind once apps are in production, lacking the proper tools or best practices to test and monitor these apps for coding errors and stability problems.
This is incredibly risky, as even a seemingly minor coding error can cause apps to crash. Consider what happened last year when a hidden bug in Facebook's iOS software development kit (SDK) caused Spotify, Pinterest, TikTok, Venmo, Tinder, DoorDash and many other top iOS apps to crash upon opening.
With shortened software development lifecycles, these inconspicuous bugs are harder than ever to find during pre-production. Organizations must extend testing to production to have more opportunity to find these errors, understand their potential impact and fix them before they wreak havoc. Fundamentally, this requires a shift in philosophy: Software engineering teams must change how they approach testing. Testing isn't something that's just done rigorously before an app is shipped to production, it's an ongoing process that must be continued throughout the entire life of an app. No app will ever be released completely free of bugs — it's just not possible. Organizations must recognize this and adapt accordingly.
The Latest
Generative AI may be a great tool for the enterprise to help drive further innovation and meaningful work, but it also runs the risk of generating massive amounts of spam that will counteract its intended benefits. From increased AI spam bots to data maintenance due to large volumes of outputs, enterprise AI applications can create a cascade of issues that end up detracting from productivity gains ...
A long-running study of DevOps practices ... suggests that any historical gains in MTTR reduction have now plateaued. For years now, the time it takes to restore services has stayed about the same: less than a day for high performers but up to a week for middle-tier teams and up to a month for laggards. The fact that progress is flat despite big investments in people, tools and automation is a cause for concern ...
Companies implementing observability benefit from increased operational efficiency, faster innovation, and better business outcomes overall, according to 2023 IT Trends Report: Lessons From Observability Leaders, a report from SolarWinds ...
IT leaders are driving an increasing number of automation initiatives as a way to stay competitive, reduce costs and scale as they navigate an unpredictable social and economic environment, according to the 2023 State of Automation in IT survey conducted by Jitterbit ...
Customer loyalty is changing as retailers get increasingly competitive. More than 75% of consumers say they would end business with a company after a single bad customer experience. This means that just one price discrepancy, inventory mishap or checkout issue in a physical or digital store, could have customers running out to the next store that can provide them with better service. Retailers must be able to predict business outages in advance, and act proactively before an incident occurs, impacting customer experience ...
Earlier this year, New Relic conducted a study on observability ... The 2023 Observability Forecast reveals observability's impact on the lives of technical professionals and businesses' bottom lines. Here are 10 key takeaways from the forecast ...
Only 33% of executives are "very confident" in their ability to operate in a public cloud environment, according to the 2023 State of CloudOps report from NetApp. This represents an increase from 2022 when only 21% reported feeling very confident ...
The majority of organizations across Australia and New Zealand (A/NZ) breached over the last year had personally identifiable information (PII) compromised, but most have not yet modified their data management policies, according to the Cybersecurity and PII Report from ManageEngine ...
