If you were lucky, you found out about the massive CrowdStrike/Microsoft outage last July by reading about it over coffee. Those less fortunate were awoken hours earlier by frantic calls from work. The unluckiest learned about the problem firsthand, finding a sea of "blue screens of death" across their organization's Windows systems, with no way to restart them and no immediate fix. Many had to shut down business operations for hours.

By now, we know what happened. One of the world's most prominent cybersecurity vendors inadvertently released a bad software update to its widely deployed endpoint agent, causing Windows systems to crash and prevented them from recovering naturally from a reboot. In this case, it was a "rapid response" patch developed to address an emerging threat, which was erroneously cleared for delivery.

The incident made headlines due to its scope: 8.5 million devices affected. Thousands of businesses ground to a halt, with losses among Fortune 500 companies alone totaling more than $5 billion. The broader problem illustrated, however, goes far beyond any single vendor or outage. Not for the first time, and likely not the last, a seemingly minor change to one arcane component of one element of the enterprise IT and security stack ended up wreaking havoc.

Whether you were directly affected or not, there's an important lesson: all organizations should be conducting in-depth reviews of testing and change management. As the IT landscape grows more complex, with new partners and services and cyberthreats emerging daily, businesses can expect more ongoing software changes from more sources — and higher risk of bad updates. If you haven't taken steps to automate continuous testing, it's time to get started.

Automating Testing

The DevOps revolution transformed the way organizations develop and maintain software, yielding countless benefits. Continuous integration/continuous delivery (CI/CD) frameworks in particular, and the toolchains that automate them, give organizations far more agility to keep up with a constantly changing technology landscape, while stabilizing operations across the software lifecycle.

At the same time, any software change carries risk of introducing unexpected problems. So, pushing updates continually — for internal products as well as third-party software — inevitably increases exposure to that risk. Ideally, robust continuous testing (CT) should play a central role in CI/CD toolchains to mitigate this issue. In practice though, the way organizations implement DevOps tooling matters a great deal. Too many focus on automating integration and delivery elements, without giving automated testing the consideration it deserves. After all, few things are more dangerous than automating updates if you're not confident that each update is safe.

So, what should effective CT look like?

In mature DevOps frameworks, automated testing is fully integrated into the software delivery pipeline, so that any new patch or version release automatically invokes testing to quickly obtain feedback and identify risks. This testing is "continuous" not just in the sense that it's repeated for every change, but it's also baked into each phase of the software lifecycle from early development through release. It establishes pass/fail data points aligned with predefined requirements, executing a much larger pool of tests, much more frequently than traditional QA testing. Ideally, test automation is directly integrated into an orchestrated CI/CD pipeline, running from an on-demand infrastructure that can elastically scale as needed.

Implemented properly, CT enables:

■ Earlier issue detection: As the CrowdStrike incident illustrates, major outages result not only from malicious acts. Often, they're preventable errors resulting from misconfigurations or policy changes. When automated testing is fully integrated into change management procedures, however, organizations can identify problems sooner, before they get pushed to production.

■ Improved stability and security: With automated CT, organizations can quickly identify changes that would adversely affect the stability, performance, or security of IT systems. They can maintain baseline KPIs of security posture and network performance over time, and more readily detect when they're drifting off target.

■ Increased efficiency and speed: Automated testing, especially combined with automated testbed and lab management solutions, reduces the time it takes to validate software and network updates. Ultimately, organizations can keep pace with the evolving IT and security landscape, better manage compliance, and avoid costly disruptions.

Do CT right, and you can expect heightened productivity, improved quality, faster time to market, and significant cost savings.

Implementing Continuous Testing

CI/CD toolchains tend to be as varied as the organizations using them. The most effective CT implementations, however, share some commonalities. A mature testing framework should be:

■ Comprehensive: CT tooling should address all potential changes to the environment. New product releases, network upgrades, third-party patches, and version updates should all be rigorously tested before deployment to identify any potential issues. That should include automatically spinning up different OSes that a patch may be designed for to validate its quality and impact in every environment where it might be deployed.

■ Tightly controlled: All updates should be deployed under sufficient control to ensure that any changes are authorized and intentional.

■ Continuously monitored: Organizations should use active testing to monitor IT networks under lifelike conditions, so they can collect feedback and resolve issues earlier, without having to wait for users to be impacted.

■ Independent: Even trusted partners can inadvertently release unsafe software. Ultimately, it's an organization's own responsibility to thoroughly test all updates, wherever they come from, to minimize supply chain risks.

■ Fully automated: The most effective testing frameworks are fully integrated and automated within an end-to-end CI/CT/CD toolchain. Not only do such frameworks execute testing as part of any change, they often automate test tools themselves. Many organizations now use on-demand lab-as-a-service (LaaS) and test-as-a-service (TaaS) solutions as a nimbler, more scalable alternative to repeatedly building and rebuilding traditional testbeds.

Guarding Against the Next Outage

DevOps success stories rarely make the headlines. Tales of CEOs getting called to testify before Congress will always get more attention than those of a business successfully executing yet another software update without issue. Behind the scenes though, few developers dispute how revolutionary DevOps has been, or just how much CI/CD frameworks contribute to the success of modern businesses.

With contemporary approaches to automated testing, we need not fear the risk exposure that comes with continuous change. As long as organizations treat robust CT as a core enabler of DevOps automation, they can benefit from ongoing improvements to the stability, security, and performance of their environments, without breaking them.