6 Valuable Takeaways from the Internet Fails of 2021
Avoiding Internet failures requires a strategic prevention and preparation plan, solid change management process and a holistic monitoring and observability solution
February 01, 2022

Mehdi Daoudi
Catchpoint

Share this

The adjective "disruptive" applied to the year 2021 is an apt descriptor of the tumultuous change we have all experienced due to the many transformations wrought by the pandemic. Digital transformation has delivered a two-edged sword of salvation and chaotic complexity. We have also seen Internet disruptions and outages become more frequent, a worrying trend with an increasingly remote workforce heavily reliant upon distributed cloud-based apps.

Our growing dependence on the cloud and Internet for business means we must take time to prepare for downtime and latency issues. There are valuable lessons found in most failures, and the Internet outages of 2021 certainly provide ample motivation to revamp processes for mitigating system disruptions. Here are six take-aways from 2021's Internet fails that can be used to increase efficiencies in managing the system infrastructure of any enterprise, no matter its size or sector.

1. Even a small change can lead to a major fail

Even the most technically sophisticated business can experience a serious system glitch. Almost all outages are the result of a manual or automated change to code or configuration.

Developing a rigorous change management approach and putting solid protocols in place will help to manage change and its possible consequences. It's important to establish clear policies and procedures around every change, with rollback steps in place for quick restoration when needed.

Your approach should involve tracking every change, testing every change before deployment, and monitoring all services, transactions and outputs that may be impacted if things do go wrong.

2. Monitor beyond your own areas of control

IT typically monitors those areas in which they are most active, like VMs, hardware and code. But they must look beyond an assumption that code bugs or infrastructure load issues are the primary causes of failure. It is equally crucial to observe what is actually delivered to consumers or users.

For an end-to-end view, IT needs visibility into areas outside of their control, such as third-party CDNs, managed DNS, and backbone ISPs. This will allow IT teams to act quickly in the event of a failure, whether that's dropping a third-party, switching to a backup solution, and of course, clearly communicating with users while teams work to resolve the situation.

3. Know the foundations of your network

Many of us live by the old adage, "If it a ain't broke, don't fix it." In the context of system infrastructure, this is often applied as, "If it ain't fixed, it won't break." If there have been no changes or modifications, we often make the mistake of assuming everything is stable.

Unfortunately, this mindset may lead you to miss those single points of failure in your system infrastructure that are rarely changed, such as DNS, BGP, and TCP configurations. All system components need continuous monitoring. Equally, teams must be prepared with a solid plan of action and having regularly practiced their response.

4. I trust you, but let's double check

When another team or vendor is making a change, it is easy to simply trust they have initiated the proper planning and analysis to make sure it's a success, but it's essential you take your own measures to verify this since the outcomes are so crucial.

Using a "Trust and Verify" approach ensures that all the checks and balances are in place when determining the impact of a change. It is essential to have a crisis call plan in place that outlines who is on call, what to do, and who to notify about the specific issue.

Other essentials are a mitigation plan for the failure, which has been pre-tested, and a communication process with templates that include need-to-know info for users and customers. Moreover, developing a monitoring and observability plan is crucial for covering all aspects of system analysis and awareness.

5. An experience monitoring and observability platform solution is your fail-safe

Deploying a holistic monitoring and observability solution platform, that enables deep visibility into all internal system components and the entire delivery chain, should be an enterprise essential. This ensures independent monitoring of every potential point of failure, which ensures you can detect outages and issues from anywhere in real time.

By establishing a baseline for how things look before a change is made, you can understand the impact of the change in regards to areas such as latency, dropped connections, slower DNS servers, and so on. As opposed to simply looking at code tracings and logs, there should be continual testing and evaluation of the output of IT services from the perspective of the end user. Monitoring must be conducted both inside the product environment and outside for 360 degree visibility into the experience.

It is also important not to rely on a cloud-only monitoring and observability solution, which can leave dangerous blind spots across the service delivery chain, and inaccurately report the end user experience.

DNS observability is essential, for instance, since a DNS problem can cause havoc and lengthy outages, like the one experienced on October 1, 2021 at Slack, one of the world's largest collaboration and messaging apps where the core problem was due to a DNS misconfiguration. Users in need of Slack's services were unable to access the app, nor or did they know why since the Slack status page was also down. The outage lasted over 15 hours, as the teams at Slack tried to discern the root cause.

If an enterprise has a monitoring plan and solution that includes a combination of observation across backbone and last mile networks, they would be able to collect data on the availability and performance of real end users trying to access digital services on their home or office networks, including pinpointing DNS as the root cause.

6. Ultimately, communication is key

A communication plan that lays out responsibilities for every role and clear contact channels can alleviate confusion during an outage. It is important to take control of media communications with clearly laid-out protocols, including the potential involvement of a PR firm in the case of a substantial Internet outage, to prevent your enterprise from being a victim of speculation and brand erosion.

Take a lesson from the November 16 Google Cloud outage and establish a process where you are able to change the DNS or CDN configuration to point users to a clearly designed error page that acknowledges the failure and assures resolution with honesty and transparency. Practice your communication plan regularly, so that teams are always prepared, and your end users know what is going on.

Mehdi Daoudi is CEO and Co-Founder of Catchpoint
Share this

The Latest

September 30, 2022

For businesses with vast and distributed computing infrastructures, one of the main objectives of IT and network operations is to locate the cause of a service condition that is having an impact. The more human resources are put into the task of gathering, processing, and finally visual monitoring the massive volumes of event and log data that serve as the main source of symptomatic indications for emerging crises, the closer the service is to the company's source of revenue ...

September 29, 2022

Our digital economy is intolerant of downtime. But consumers haven't just come to expect always-on digital apps and services. They also expect continuous innovation, new functionality and lightening fast response times. Organizations have taken note, investing heavily in teams and tools that supposedly increase uptime and free resources for innovation. But leaders have not realized this "throw money at the problem" approach to monitoring is burning through resources without much improvement in availability outcomes ...

September 28, 2022

Although 83% of businesses are concerned about a recession in 2023, B2B tech marketers can look forward to growth — 51% of organizations plan to increase IT budgets in 2023 vs. a narrow 6% that plan to reduce their spend, according to the 2023 State of IT report from Spiceworks Ziff Davis ...

September 27, 2022

Users have high expectations around applications — quick loading times, look and feel visually advanced, with feature-rich content, video streaming, and multimedia capabilities — all of these devour network bandwidth. With millions of users accessing applications and mobile apps from multiple devices, most companies today generate seemingly unmanageable volumes of data and traffic on their networks ...

September 26, 2022

In Italy, it is customary to treat wine as part of the meal ... Too often, testing is treated with the same reverence as the post-meal task of loading the dishwasher, when it should be treated like an elegant wine pairing ...

September 23, 2022

In order to properly sort through all monitoring noise and identify true problems, their causes, and to prioritize them for response by the IT team, they have created and built a revolutionary new system using a meta-cognitive model ...

September 22, 2022

As we shift further into a digital-first world, where having a reliable online experience becomes more essential, Site Reliability Engineers remain in-demand among organizations of all sizes ... This diverse set of skills and values can be difficult to interview for. In this blog, we'll get you started with some example questions and processes to find your ideal SRE ...

September 21, 2022

US government agencies are bringing more of their employees back into the office and implementing hybrid work schedules, but federal workers are worried that their agencies' IT architectures aren't built to handle the "new normal." They fear that the reactive, manual methods used by the current systems in dealing with user, IT architecture and application problems will degrade the user experience and negatively affect productivity. In fact, according to a recent survey, many federal employees are concerned that they won't work as effectively back in the office as they did at home ...

September 20, 2022

Users today expect a seamless, uninterrupted experience when interacting with their web and mobile apps. Their expectations have continued to grow in tandem with their appetite for new features and consistent updates. Mobile apps have responded by increasing their release cadence by up to 40%, releasing a new full version of their app every 4-5 days, as determined in this year's SmartBear State of Software Quality | Application Stability Index report ...

September 19, 2022

In this second part of the blog series, we look at how adopting AIOps capabilities can drive business value for an organization ...