Application-Aware Network Performance Management (AA NPM) solutions tout benefits from capabilities embedded in such themes as "User Experience," "Application Performance," or "Business Impact" – with enticing dashboards and lots of metrics and graphs to grab attention. In this blog, I'll outline four of the more significant broken AA NPM promises, both implied and explicit.

Broken Promise #1: You can measure end-user experience

The Promise: By measuring "response time" for end users, you will gain important insight into problems before they become pervasive, and can apply this context to infrastructure metrics to focus fault domain isolation and problem resolution efforts.

The Reality: The definition of "end-user experience" is often diluted from its performance-centric meaning – which should be end-user response time or "click to glass." In fact, few AA NPM solutions measure real end-user response time, instead redefining the term to suit available measurements. Many AA NPM vendors simply group a few key network metrics to suggest the quality of the end-user's experience – even though these metrics may have little or nothing to do with end-user response time. In a way, this approach supports the classic "defend the network" position; if the network is clean, it must be someone else's problem.

Lacking an understanding of true end-user experience, IT faces at least two significant challenges:

■ You don't know that there's a problem until users complain

■ You are inundated with false alerts, chasing after problems that don't exist or ignoring problems that warrant attention

Example: Application availability from a remote site is 100%, latency is normal at 30 milliseconds, and link quality (e.g., re-transmission rate) is perfect. Does that mean the end user experience is good? What if latency changes from 30 milliseconds to 50 milliseconds? What impact does that have on end-user experience? (The answer is "It depends." Some applications would exhibit minimal impact from such a change in latency, while others might slow dramatically.)

Summary: End-user experience provides the business-centric context for interpreting the importance and impact of the underlying device and application metrics; it's the single metric sitting at the intersection between business and IT metrics. But it must be defined as end-user response time to deliver on its implied value.

Broken Promise #2: End-to-end Visibility Means End-through-end Visibility

The Promise: End-to-end visibility suggests the ability to understand application performance through the entire application delivery chain – from the client, across the network and through firewalls and load balancers, and also through backend application and database tiers.

The Reality: AA NPM solutions may provide network-level metrics at each tier, and often also include generic session-layer response time measurements. There might also be some ability to group servers together – graphically and/or logically – to form an end-to-end picture of the application's architecture, reporting network performance at each tier. The solutions often apply heuristics to detect changes in response time at each tier; if a change exceeds a threshold, a fault is assumed and the offending tier is highlighted as a likely suspect. There are quite a few problems inherent in this approach; here are two of the most significant:

■ Without an understanding of end-user experience, how do you know you have a problem? When do you start to care about changes in performance reported by the solution?

■ What thresholds are used to trigger alerts or anomalies?

Example: If first-tier session-layer response time changes from 50 milliseconds to 60 milliseconds, should you take action? If response time at the database tier changes from 4 milliseconds to 6 milliseconds, how does this impact end users?

Summary: An understanding of transaction-level performance at each tier of a complex application environment is critical to effective fault domain isolation, but it must be correlated to the measurement of end-user experience.

Broken Promise #3: Response Time Measurements are Actionable

The Promise: By measuring response time and comparing this to baselines, IT can take informed actions to remediate problems as they occur, or prevent problems from affecting users.

The Reality: While response time has always been integral to performance monitoring, its definition changes according to the discipline. Response time can refer to isolated singular request/response exchanges such as a memory fetch, a disk write, or a database query; these metrics are internal to the technology discipline. Response time can also be external, referring to much more complex exchanges such as loading a web page, requesting a document download, or running a complex report. With today's hyper-focus on the user, the ability to monitor response time is quite compelling – if response time is defined in the user's terms. But not all response times are created equal. In fact most AA NPM vendors apply an internal network-centric definition, not an external user-centric definition.

Example: Consider a web page comprised of a JSP, some JavaScripts, some style sheets, and a bunch of images. Let's say there are 40 page components, meaning there will be 40 session-layer response time measurements for the page. Now let's say they range from 1 millisecond (for static content) to 10 seconds (for the JSP). The simple application association (e.g., NBAR2 classification) inherent in session-layer response time means that these measurements are all included in the same reporting bucket. They may average 0.30 seconds – for a single page. Can you tell if the user is having a problem? Compound this with hundreds of users loading many pages, and you end up with statistical insignificance. Now let's add labels to the page components to arrive at hit-level performance insight; no longer are the requests for images mixed with the long-running JSPs. But the performance of individual page components doesn't directly correspond with the end-user's experience, limiting the value to generally reactive interaction with development teams (see the next broken promise).

Summary: Response time measurements benefit from correlation to end-user experience; without this context, only the more catastrophic problems are clearly visible and actionable.

Broken Promise #4: AA NPM Facilitates Collaboration with Development Teams

The Promise: Application awareness provides the foundation for effective collaboration with development teams. For problem and incident management, the solution should be able to provide the diagnostic information to IT operations that allows the development team to:

■ Accept that there is a problem related to an understandable application/code function

■ Begin to investigate the application logic invoked by the transaction, along with underlying dependencies

The Reality: This may sometimes be partially true, with a few important limitations. Some AA NPM solutions provide hit-level response time measurements (for some application types) that can be considered the lowest common denominator. But there remains a critical missing link, one that significantly impairs your ability to leverage this application transaction insight; if you haven't guessed by now, that missing link is end-user experience.

Without measuring end-user experience, you won't know users are having a problem until they call you (unless something catastrophic happens). And you will chase after problems that don't affect users (because you're monitoring hundreds, or thousands, of application components).

Example: Let's say you're monitoring a handful of web applications, with hit-level measurements for tier 1 (HTTP), tier 2 (SOAP) and tier 3 (SQL). How will you know when you have a problem? If the performance of the OpenSession JSP changes from 3 to 4 seconds, is that a problem that needs to be shared with the development team? What happens if JavaScript file download performance degrades from 1.0 to 1.5 seconds? Does the network team need to respond to this? What if the performance of a specific SQL query changes from 4 milliseconds to 6 milliseconds? How do you go about setting appropriate performance thresholds for thousands of application components?

These four promises are all interrelated; they break primarily due to the lack of visibility into end-user experience.

Gary Kaiser is a Subject Matter Expert in Network Performance Analysis at Dynatrace.