Fault Domain Isolation Key to Avoiding Network Blame Game - Part 2
April 14, 2015

Jeff Brown
Emulex

Share this

Start with Part 1 of this Blog

What’s the Hold Up?

It always reduces costs and decreases time-to-resolution when root cause analysis is being done in earnest, with confidence (and perhaps a bit of guilt) that the problem simply cannot lay elsewhere. RCA works best when the people working on the problem have the expertise to properly evaluate the cause and resolve the problem.

In Part 1 of this Blog, I explained how a packet-driven FDI process is an effective way to accelerate incident investigations and reduce the number of people involved. Further, to achieve its primary goal of getting only the right people involved in the incident investigation, we know it doesn’t take a lot of taps and equipment to isolate the major technology tiers. So why do team-of-expert meetings still persist in so many major incident investigations?

The problem might be that some simply do not believe that complex incidents can be fully resolved with just a few taps and some network recorders. And you know what, they’re right! But that isn’t the goal of the FDI stage of the incident investigation process. The goal is fault isolation, and that can be done simply and reliably. All you need is the underlying packets and a process to analyze them.

Divide and Conquer

The primary or first-layer FDI process isolates the incident to a single technology tier as defined by the organization’s internal structure and outsourcing arrangement.

Primary FDI is best achieved by:

1. Using network recording tools to monitor and store the network traffic occurring between technology tiers

2. Applying application transaction analysis to perform fault isolation.

Packet storage (rather than just averages or summaries) is key to enabling the back-in-time analysis upon which efficient FDI depends.

As you’ve probably guessed, FDI is a divide and conquer process that can be deployed in layers. FDI can also be used within each tier to further isolate the problem until highly efficient RCA can be done. This can be called intra-tier FDI, or perhaps secondary FDI.

Not surprisingly, network incident investigations are particularly amenable to a secondary FDI workflow, and once again, this is best achieved by monitoring and storing the actual packet flows between key network components for efficient back-in-time analysis.

It is valid to ask where the network tap points and network recording tools should be deployed when intra-network FDI is the goal. The main difference between primary FDI and intra-network FDI is that the location of the observation points is less an organizational issue, and more about physical location, technology, staff expertise, and of course, level of outsourcing and external suppliers. But the FDI process is similar: use packet-based analysis to provide irrefutable evidence as to which technology or service provider is at fault, and which are not.

Always-On or Always-Available?

You do not want to wait for a major incident to occur before you start deploying the tap points and monitoring tools needed for performing FDI -- that would defeat its purpose. So it seems pretty clear that the tap points and network recording tools needed for primary or first-level FDI should be deployed and running all the time. Those are your always-on appliances.

But what about secondary or intra-technology FDI? What about remote sites, regional data centers, and non-critical applications? You can’t tap everywhere, nor can you store everything.

Fortunately many network recording tools have been built to satisfy the needs of the always-on recording required between primary technology tiers, and the “always-available” recording connected via Network Packet Brokers to a plethora of secondary tap points. Always-available appliances do not necessarily give you long-term back-in-time visibility, but they can be quickly configured to begin monitoring where needed, on demand, tuned to the specific visibility needs of the incident investigation underway.

How Simple Is It?

So, is FDI truly as simple as we’ve described? Well, yes and no. Obviously there are plenty of unusual, complex, and just plain bizarre problems that can appear in a system as complex and dynamic as a modern organization’s networked business application infrastructure. And these types of problems will always require deep investigation, and the skills and knowledge of specialists and experts to resolve. But that doesn’t render FDI irrelevant or ineffective for these complex issues. Indeed it makes the need for a rigorous, repeatable, data-driven FDI process all the more important. Put another way, for complex problems why wouldn’t you use a proven divide and conquer approach like FDI?

Jeff Brown is Global Director of Training, NVP at Emulex.

Share this

The Latest

July 09, 2020

Enterprises that halted their cloud migration journey during the current global pandemic are two and a half times more likely than those that continued their move to the cloud to have experienced IT outages that negatively impacted their SLAs, according to Virtana's latest survey report The Current State of Hybrid Cloud and IT ...

July 08, 2020

Every business has the responsibility to do their part against climate change by reducing their carbon footprint while increasing sustainability and efficiency. Harnessing optimization of IT infrastructure is one method companies can use to reduce carbon footprint, improve sustainability and increase business efficiency, while also keeping costs down ...

July 07, 2020

While the adoption of continuous integration (CI) is on the rise, software engineering teams are unable to take a zero-tolerance approach to software failures, costing enterprise organizations billions annually, according to a quantitative study conducted by Undo and a Cambridge Judge Business School MBA project ...

June 25, 2020

I've had the opportunity to work with a number of organizations embarking on their AIOps journey. I always advise them to start by evaluating their needs and the possibilities AIOps can bring to them through five different levels of AIOps maturity. This is a strategic approach that allows enterprises to achieve complete automation for long-term success ...

June 24, 2020

Sumo Logic recently commissioned an independent market research study to understand the industry momentum behind continuous intelligence — and the necessity for digital organizations to embrace a cloud-native, real-time continuous intelligence platform to support the speed and agility of business for faster decision-making, optimizing security, driving new innovation and delivering world-class customer experiences. Some of the key findings include ...

June 23, 2020

When it comes to viruses, it's typically those of the computer/digital variety that IT is concerned about. But with the ongoing pandemic, IT operations teams are on the hook to maintain business functions in the midst of rapid and massive change. One of the biggest challenges for businesses is the shift to remote work at scale. Ensuring that they can continue to provide products and services — and satisfy their customers — against this backdrop is challenging for many ...

June 22, 2020

Teams tasked with developing and delivering software are under pressure to balance the business imperative for speed with high customer expectations for quality. In the course of trying to achieve this balance, engineering organizations rely on a variety of tools, techniques and processes. The 2020 State of Software Quality report provides a snapshot of the key challenges organizations encounter when it comes to delivering quality software at speed, as well as how they are approaching these hurdles. This blog introduces its key findings ...

June 18, 2020

For IT teams, run-the-business, commodity areas such as employee help desks, device support and communication platforms are regularly placed in the crosshairs for cost takeout, but these areas are also highly visible to employees. Organizations can improve employee satisfaction and business performance by building unified functions that are measured by employee experience rather than price. This approach will ultimately fund transformation, as well as increase productivity and innovation ...

June 17, 2020

In the agile DevOps framework, there is a vital piece missing; something that previous approaches to application development did well, but has since fallen by the wayside. That is, the post-delivery portion of the toolchain. Without continuous cloud optimization, the CI/CD toolchain still produces massive inefficiencies and overspend ...

June 16, 2020

The COVID-19 pandemic has exponentially accelerated digital transformation projects. To better understand where IT professionals are turning for help, we analyzed the online behaviors of IT decision-makers. Our research found an increase in demand for resources related to APM, microservices and dependence on cloud services ...