In the world of Application Performance Management (APM) it is always better to enlist more than one entity to help solve the mystery of performance problems.
It's kind of like arriving at the scene of the crime on foreign soil, being blindfolded, shoved out the door, and then asked to help solve the injustice without any insight. All you can do is begin by asking people in the vicinity, providing you speak their language, for information on what they have seen (i.e. end-user-experience).
Gathering facts related to a crime is essential, and can be likened to utilizing an APM solution for solving application performance problems. The more information about an application’s behavior that you can obtain, along with understanding its idiosyncrasies within the environment, the more likely you will be able to pinpoint root causes of performance issues.
The Three People You Need
Wouldn't it be helpful if there was an eye witness you could interview, a watchman who was on duty during the time of the incident, and an agent you could hire to translate the native tongue and provide insight into the culture?
In much the same way, a smart APM strategy enlists the help from these three entities: the Witness, the Watchman, and the Agent. You start by listening to the testimony from the eye witness (aka. wire data), collecting the observations from the watchman (aka. web robots), and analyzing details from the agent (aka. code level instrumentation).
The Witness
Passive monitoring, wire-data analytics
The Witness reports what they see within their field of vision, (aka. passive monitoring, wire-data analytics). The Witness is watching everything in their purview and sees things as they happen, which corresponds to what is coming across "the wire" in front of them.
The Witness will tell you how many people were involved, if anyone was injured, and what time the event occurred, (e.g. user names, packet loss, timelines, etc.). She can tell you what doors the people went through, how wide the aisles were, and how fast people were traveling, (e.g. network port listeners, realized bandwidth, round-trip-time, etc.).
The Watchman
Active monitoring - synthetic transactions
The Watchman (aka. web robot) is actively checking and is always on patrol, methodically taking the same path every time. He will tell you what doors are locked and monitor the ones that are open, collecting measurements along the way on how long it takes to complete his rounds, (i.e. synthetic transactions).
The Watchman will report the status of the rooms and buildings on his patrol and will note if anything happens to him along the way, (e.g. application availability, transaction errors, timeouts, etc.).
The Agent
Application code instrumentation
The Agent you hire is critical for solving the crime within the territory you're operating in. The Agent will watch activity from specific vantage points throughout the environment and report back his findings. It's crucial he speaks the local language, (e.g. Java, .Net, PHP) and can easily translate for you.
His approach will be to deploy probes on rooftops and inside the buildings for monitoring all conversations and actions in the environment, (aka. application code instrumentation). He will also tap the communication systems, (i.e. script injection) when appropriate and capture specific measurements from each conversation and record them.
Going from Red to Green
Identifying an application that has gone catatonic is one thing, but assessing the insidious slow performance of a complex multi-tiered application and fixing it, can be very time consuming and costly. Enlisting all three entities described above to assist is a thoughtful strategy for any IT Leader to consider.
Based on eye witness testimony, the forensics collected, and the conversations recorded, you will be well on your way to providing an accurate account of what has transpired and why, (i.e. root cause analysis).
Conclusion
Remember, the end-user is the supreme judge in this case and if performance is chronically slow, your sentence could be harsh. Either directly by inundating you with complaints creating bad press or indirectly by abandoning your site in favor of one that is much faster and more intuitive to use.
Embracing a smart but simple APM Methodology within your environment may be the only thing that exonerates you when the verdict for your slow application is "guilty as charged."