Fault Domain Isolation Key to Avoiding Network Blame Game - Part 2
April 14, 2015

Jeff Brown
Emulex

Share this

Start with Part 1 of this Blog

What’s the Hold Up?

It always reduces costs and decreases time-to-resolution when root cause analysis is being done in earnest, with confidence (and perhaps a bit of guilt) that the problem simply cannot lay elsewhere. RCA works best when the people working on the problem have the expertise to properly evaluate the cause and resolve the problem.

In Part 1 of this Blog, I explained how a packet-driven FDI process is an effective way to accelerate incident investigations and reduce the number of people involved. Further, to achieve its primary goal of getting only the right people involved in the incident investigation, we know it doesn’t take a lot of taps and equipment to isolate the major technology tiers. So why do team-of-expert meetings still persist in so many major incident investigations?

The problem might be that some simply do not believe that complex incidents can be fully resolved with just a few taps and some network recorders. And you know what, they’re right! But that isn’t the goal of the FDI stage of the incident investigation process. The goal is fault isolation, and that can be done simply and reliably. All you need is the underlying packets and a process to analyze them.

Divide and Conquer

The primary or first-layer FDI process isolates the incident to a single technology tier as defined by the organization’s internal structure and outsourcing arrangement.

Primary FDI is best achieved by:

1. Using network recording tools to monitor and store the network traffic occurring between technology tiers

2. Applying application transaction analysis to perform fault isolation.

Packet storage (rather than just averages or summaries) is key to enabling the back-in-time analysis upon which efficient FDI depends.

As you’ve probably guessed, FDI is a divide and conquer process that can be deployed in layers. FDI can also be used within each tier to further isolate the problem until highly efficient RCA can be done. This can be called intra-tier FDI, or perhaps secondary FDI.

Not surprisingly, network incident investigations are particularly amenable to a secondary FDI workflow, and once again, this is best achieved by monitoring and storing the actual packet flows between key network components for efficient back-in-time analysis.

It is valid to ask where the network tap points and network recording tools should be deployed when intra-network FDI is the goal. The main difference between primary FDI and intra-network FDI is that the location of the observation points is less an organizational issue, and more about physical location, technology, staff expertise, and of course, level of outsourcing and external suppliers. But the FDI process is similar: use packet-based analysis to provide irrefutable evidence as to which technology or service provider is at fault, and which are not.

Always-On or Always-Available?

You do not want to wait for a major incident to occur before you start deploying the tap points and monitoring tools needed for performing FDI -- that would defeat its purpose. So it seems pretty clear that the tap points and network recording tools needed for primary or first-level FDI should be deployed and running all the time. Those are your always-on appliances.

But what about secondary or intra-technology FDI? What about remote sites, regional data centers, and non-critical applications? You can’t tap everywhere, nor can you store everything.

Fortunately many network recording tools have been built to satisfy the needs of the always-on recording required between primary technology tiers, and the “always-available” recording connected via Network Packet Brokers to a plethora of secondary tap points. Always-available appliances do not necessarily give you long-term back-in-time visibility, but they can be quickly configured to begin monitoring where needed, on demand, tuned to the specific visibility needs of the incident investigation underway.

How Simple Is It?

So, is FDI truly as simple as we’ve described? Well, yes and no. Obviously there are plenty of unusual, complex, and just plain bizarre problems that can appear in a system as complex and dynamic as a modern organization’s networked business application infrastructure. And these types of problems will always require deep investigation, and the skills and knowledge of specialists and experts to resolve. But that doesn’t render FDI irrelevant or ineffective for these complex issues. Indeed it makes the need for a rigorous, repeatable, data-driven FDI process all the more important. Put another way, for complex problems why wouldn’t you use a proven divide and conquer approach like FDI?

Jeff Brown is Global Director of Training, NVP at Emulex.

Share this

The Latest

December 05, 2023

Industry experts offer thoughtful, insightful, and often controversial predictions on how APM, AIOps, Observability, OpenTelemetry and related technologies will evolve and impact business in 2024. Part 2 covers more on Observability ...

December 04, 2023

The Holiday Season means it is time for APMdigest's annual list of Application Performance Management (APM) predictions, covering IT performance topics. Industry experts — from analysts and consultants to the top vendors — offer thoughtful, insightful, and often controversial predictions on how APM, observability, AIOps and related technologies will evolve and impact business in 2024. Part 1 covers APM and Observability ...

November 30, 2023

To help you stay on top of the ever-evolving tech scene, Automox IT experts shake the proverbial magic eight ball and share their predictions about tech trends in the coming year. From M&A frenzies to sustainable tech and automation, these forecasts paint an exciting picture of the future ...

November 29, 2023
The past few years have presented numerous challenges for businesses: a pandemic, rising interest rates, supply chain disruptions, and geopolitical conflict that sent shockwaves across the global economy. But change may finally be on the horizon. According to a recent report by Endava ... a majority of executives confirmed they are feeling optimistic about the current business climate, and as a result, are forecasting larger IT budgets, increased technology funding and rollout, and prioritized innovation in the coming year ...
November 28, 2023

Incident management processes are not keeping pace with the demands of modern operations teams, failing to meet the needs of SREs as well as platform and ops teams. Results from the State of DevOps Automation and AI Survey, commissioned by Transposit, point to an incident management paradox. Despite nearly 60% of ITOps and DevOps professionals reporting they have a defined incident management process that's fully documented in one place and over 70% saying they have a level of automation that meets their needs, teams are unable to quickly resolve incidents ...

November 27, 2023

Today, in the world of enterprise technology, the challenges posed by legacy Virtual Desktop Infrastructure (VDI) systems have long been a source of concern for IT departments. In many instances, this promising solution has become an organizational burden, hindering progress, depleting resources, and taking a psychological and operational toll on employees ...

November 22, 2023

Within retail organizations across the world, IT teams will be bracing themselves for a hectic holiday season ... While this is an exciting opportunity for retailers to boost sales, it also intensifies severe risk. Any application performance slipup will cause consumers to turn their back on brands, possibly forever. Online shoppers will be completely unforgiving to any retailer who doesn't deliver a seamless digital experience ...

November 21, 2023

Black Friday is a time when consumers can cash in on some of the biggest deals retailers offer all year long ... Nearly two-thirds of consumers utilize a retailer's web and mobile app for holiday shopping, raising the stakes for competitors to provide the best online experience to retain customer loyalty. Perforce's 2023 Black Friday survey sheds light on consumers' expectations this time of year and how developers can properly prepare their applications for increased online traffic ...

November 20, 2023

This holiday shopping season, the stakes for online retailers couldn't be higher ... Even an hour or two of downtime for a digital storefront during this critical period can cost millions in lost revenue and has the potential to damage brand credibility. Savvy retailers are increasingly investing in observability to help ensure a seamless, omnichannel customer experience. Just ahead of the holiday season, New Relic released its State of Observability for Retail report, which offers insight and analysis on the adoption and business value of observability for the global retail/consumer industry ...

November 16, 2023

As organizations struggle to find and retain the talent they need to manage complex cloud implementations, many are leaning toward hybrid cloud as a solution ... While it's true that using the cloud is not a "one size fits all" proposition, it is clear that both large and small companies prefer a hybrid cloud model ...