4 Broken AA NPM Promises
August 18, 2015

Gary Kaiser
Dynatrace

Share this

Application-Aware Network Performance Management (AA NPM) solutions tout benefits from capabilities embedded in such themes as "User Experience," "Application Performance," or "Business Impact" – with enticing dashboards and lots of metrics and graphs to grab attention. In this blog, I'll outline four of the more significant broken AA NPM promises, both implied and explicit.

Broken Promise #1: You can measure end-user experience

The Promise: By measuring "response time" for end users, you will gain important insight into problems before they become pervasive, and can apply this context to infrastructure metrics to focus fault domain isolation and problem resolution efforts.

The Reality: The definition of "end-user experience" is often diluted from its performance-centric meaning – which should be end-user response time or "click to glass." In fact, few AA NPM solutions measure real end-user response time, instead redefining the term to suit available measurements. Many AA NPM vendors simply group a few key network metrics to suggest the quality of the end-user's experience – even though these metrics may have little or nothing to do with end-user response time. In a way, this approach supports the classic "defend the network" position; if the network is clean, it must be someone else's problem.

Lacking an understanding of true end-user experience, IT faces at least two significant challenges:

■ You don't know that there's a problem until users complain

■ You are inundated with false alerts, chasing after problems that don't exist or ignoring problems that warrant attention

Example: Application availability from a remote site is 100%, latency is normal at 30 milliseconds, and link quality (e.g., re-transmission rate) is perfect. Does that mean the end user experience is good? What if latency changes from 30 milliseconds to 50 milliseconds? What impact does that have on end-user experience? (The answer is "It depends." Some applications would exhibit minimal impact from such a change in latency, while others might slow dramatically.)

Summary: End-user experience provides the business-centric context for interpreting the importance and impact of the underlying device and application metrics; it's the single metric sitting at the intersection between business and IT metrics. But it must be defined as end-user response time to deliver on its implied value.

Broken Promise #2: End-to-end Visibility Means End-through-end Visibility

The Promise: End-to-end visibility suggests the ability to understand application performance through the entire application delivery chain – from the client, across the network and through firewalls and load balancers, and also through backend application and database tiers.

The Reality: AA NPM solutions may provide network-level metrics at each tier, and often also include generic session-layer response time measurements. There might also be some ability to group servers together – graphically and/or logically – to form an end-to-end picture of the application's architecture, reporting network performance at each tier. The solutions often apply heuristics to detect changes in response time at each tier; if a change exceeds a threshold, a fault is assumed and the offending tier is highlighted as a likely suspect. There are quite a few problems inherent in this approach; here are two of the most significant:

■ Without an understanding of end-user experience, how do you know you have a problem? When do you start to care about changes in performance reported by the solution?

■ What thresholds are used to trigger alerts or anomalies?

Example: If first-tier session-layer response time changes from 50 milliseconds to 60 milliseconds, should you take action? If response time at the database tier changes from 4 milliseconds to 6 milliseconds, how does this impact end users?

Summary: An understanding of transaction-level performance at each tier of a complex application environment is critical to effective fault domain isolation, but it must be correlated to the measurement of end-user experience.

Broken Promise #3: Response Time Measurements are Actionable

The Promise: By measuring response time and comparing this to baselines, IT can take informed actions to remediate problems as they occur, or prevent problems from affecting users.

The Reality: While response time has always been integral to performance monitoring, its definition changes according to the discipline. Response time can refer to isolated singular request/response exchanges such as a memory fetch, a disk write, or a database query; these metrics are internal to the technology discipline. Response time can also be external, referring to much more complex exchanges such as loading a web page, requesting a document download, or running a complex report. With today's hyper-focus on the user, the ability to monitor response time is quite compelling – if response time is defined in the user's terms. But not all response times are created equal. In fact most AA NPM vendors apply an internal network-centric definition, not an external user-centric definition.

Example: Consider a web page comprised of a JSP, some JavaScripts, some style sheets, and a bunch of images. Let's say there are 40 page components, meaning there will be 40 session-layer response time measurements for the page. Now let's say they range from 1 millisecond (for static content) to 10 seconds (for the JSP). The simple application association (e.g., NBAR2 classification) inherent in session-layer response time means that these measurements are all included in the same reporting bucket. They may average 0.30 seconds – for a single page. Can you tell if the user is having a problem? Compound this with hundreds of users loading many pages, and you end up with statistical insignificance. Now let's add labels to the page components to arrive at hit-level performance insight; no longer are the requests for images mixed with the long-running JSPs. But the performance of individual page components doesn't directly correspond with the end-user's experience, limiting the value to generally reactive interaction with development teams (see the next broken promise).

Summary: Response time measurements benefit from correlation to end-user experience; without this context, only the more catastrophic problems are clearly visible and actionable.

Broken Promise #4: AA NPM Facilitates Collaboration with Development Teams

The Promise: Application awareness provides the foundation for effective collaboration with development teams. For problem and incident management, the solution should be able to provide the diagnostic information to IT operations that allows the development team to:

■ Accept that there is a problem related to an understandable application/code function

■ Begin to investigate the application logic invoked by the transaction, along with underlying dependencies

The Reality: This may sometimes be partially true, with a few important limitations. Some AA NPM solutions provide hit-level response time measurements (for some application types) that can be considered the lowest common denominator. But there remains a critical missing link, one that significantly impairs your ability to leverage this application transaction insight; if you haven't guessed by now, that missing link is end-user experience.

Without measuring end-user experience, you won't know users are having a problem until they call you (unless something catastrophic happens). And you will chase after problems that don't affect users (because you're monitoring hundreds, or thousands, of application components).

Example: Let's say you're monitoring a handful of web applications, with hit-level measurements for tier 1 (HTTP), tier 2 (SOAP) and tier 3 (SQL). How will you know when you have a problem? If the performance of the OpenSession JSP changes from 3 to 4 seconds, is that a problem that needs to be shared with the development team? What happens if JavaScript file download performance degrades from 1.0 to 1.5 seconds? Does the network team need to respond to this? What if the performance of a specific SQL query changes from 4 milliseconds to 6 milliseconds? How do you go about setting appropriate performance thresholds for thousands of application components?

These four promises are all interrelated; they break primarily due to the lack of visibility into end-user experience.

Gary Kaiser is a Subject Matter Expert in Network Performance Analysis at Dynatrace.

Share this

The Latest

November 30, 2023

To help you stay on top of the ever-evolving tech scene, Automox IT experts shake the proverbial magic eight ball and share their predictions about tech trends in the coming year. From M&A frenzies to sustainable tech and automation, these forecasts paint an exciting picture of the future ...

November 29, 2023
The past few years have presented numerous challenges for businesses: a pandemic, rising interest rates, supply chain disruptions, and geopolitical conflict that sent shockwaves across the global economy. But change may finally be on the horizon. According to a recent report by Endava ... a majority of executives confirmed they are feeling optimistic about the current business climate, and as a result, are forecasting larger IT budgets, increased technology funding and rollout, and prioritized innovation in the coming year ...
November 28, 2023

Incident management processes are not keeping pace with the demands of modern operations teams, failing to meet the needs of SREs as well as platform and ops teams. Results from the State of DevOps Automation and AI Survey, commissioned by Transposit, point to an incident management paradox. Despite nearly 60% of ITOps and DevOps professionals reporting they have a defined incident management process that's fully documented in one place and over 70% saying they have a level of automation that meets their needs, teams are unable to quickly resolve incidents ...

November 27, 2023

Today, in the world of enterprise technology, the challenges posed by legacy Virtual Desktop Infrastructure (VDI) systems have long been a source of concern for IT departments. In many instances, this promising solution has become an organizational burden, hindering progress, depleting resources, and taking a psychological and operational toll on employees ...

November 22, 2023

Within retail organizations across the world, IT teams will be bracing themselves for a hectic holiday season ... While this is an exciting opportunity for retailers to boost sales, it also intensifies severe risk. Any application performance slipup will cause consumers to turn their back on brands, possibly forever. Online shoppers will be completely unforgiving to any retailer who doesn't deliver a seamless digital experience ...

November 21, 2023

Black Friday is a time when consumers can cash in on some of the biggest deals retailers offer all year long ... Nearly two-thirds of consumers utilize a retailer's web and mobile app for holiday shopping, raising the stakes for competitors to provide the best online experience to retain customer loyalty. Perforce's 2023 Black Friday survey sheds light on consumers' expectations this time of year and how developers can properly prepare their applications for increased online traffic ...

November 20, 2023

This holiday shopping season, the stakes for online retailers couldn't be higher ... Even an hour or two of downtime for a digital storefront during this critical period can cost millions in lost revenue and has the potential to damage brand credibility. Savvy retailers are increasingly investing in observability to help ensure a seamless, omnichannel customer experience. Just ahead of the holiday season, New Relic released its State of Observability for Retail report, which offers insight and analysis on the adoption and business value of observability for the global retail/consumer industry ...

November 16, 2023

As organizations struggle to find and retain the talent they need to manage complex cloud implementations, many are leaning toward hybrid cloud as a solution ... While it's true that using the cloud is not a "one size fits all" proposition, it is clear that both large and small companies prefer a hybrid cloud model ...

November 15, 2023

In the same way a city is a sum of its districts and neighborhoods, complex IT systems are made of many components that continually interact. Observability requires a comprehensive and connected view of all aspects of the system, including even some that don't directly relate to its technological innards ...

November 14, 2023

Multicasting in this context refers to the process of directing data streams to two or more destinations. This might look like sending the same telemetry data to both an on-premises storage system and a cloud-based observability platform concurrently. The two principal benefits of this strategy are cost savings and service redundancy ...