APM for Enterprise: How Does It Scale?
May 18, 2015

Larry Haig

Share this

It is easy to feel that so called "second generation" Application Performance Management (APM) tooling rules the world.

And for good reason, many would argue – certainly the positive disruptive effects of support for highly distributed / Service Orientated architectures, and the requirements of many fast moving businesses to support a plethora of different technologies are a powerful dynamic. That leaves aside the undoubted advantages of comprehensive traffic screening (as opposed to "hard" sampling), ease of installation and commissioning (relative in some cases), user accessibility, flexible reporting and tighter productive association between IT and business – in short, empowering the DevOps and PerfOps revolution.

So, modern APM is certainly well attuned to the requirements of current business. What's not to like?

Could these technologies have an Achilles heel? Certainly, they are generally strong on lists of customer logos, but tight lipped when it comes to detailed high volume case studies.

Hundreds or thousands of JVMs and moderately high transaction volumes are all very well (and well attested), but how do these technologies stack up for the high end enterprise? What other options might exist?

It could be argued that an organization with tens of thousands of JVMs and millions of metrics has a fundamentally different issue than those closer to the base of the pyramid. Certainly these organizations are fewer in number, but that is scant comfort for those with the responsibility of managing their application delivery. Whether in banking/financial trading, FMCG or elsewhere, the issue of effectively analyzing daily transaction flows at high scale is real. The situation is exacerbated at peak – one large UK gaming company generates 20-30,000 events per second during a normal daily peak. During the popular Grand National race meeting, traffic increases 5-10 times – creating the need to transfer several terabytes a day into an APM data store.

The question is: which if any of the APM tools can even come close to these sorts of volumes?

It is certainly possible to instrument these organizations with second generation APM – but what snares lie in wait for the unwary, and what compromises will have to be made?

To some extent, the answer depends upon the particular technology deployed. All will have their own weaknesses, but those architected around collector/analysis servers are likely to be particularly vulnerable to the effects of extreme data volume unless high scale technology/architectural interventions have been made "under the covers". Cloud based solutions may duck this bullet (although they are not guaranteed to do so), but come with their own security concerns, at least in theory.

So, you are a high volume Enterprise, and have plumped for second generation APM. What issues may arise? Essentially, software agent based APM is likely to evidence stress in one or more of three principal areas:

■ Length of data storage/"live" access

■ Data granularity

■ Production system performance overhead

Compromises essentially hinge around reducing the data flows processed by the APM to reduce the amount of data written to disk, or improving the inherent efficiency of such data handling. Traditionally, this involves sampling rather than screening all transactions; and this is an option for some. However, sampling has no value for businesses needing to identify and analyze a particular single customer session.

Other approaches are to increase the hardware capacity of collector/server components, or reducing the application server to collector ratio. Either way, these compromises run the risk of eroding the underlying value proposition supporting much of second generation tool philosophy. In addition they will push the architecture of these solutions to their limit and potentially expose fundamental issues in how they scale.

Open Source approaches to extreme scale have evolved using NoSQL – creating products such as Hadoop and ElasticSearch. The pedigree of these is generally good, in that they have been developed as strategies within companies such as Google and Facebook to deal with the problems of ultra-high volume environments.

Certainly, integration of these technologies into their tooling by APM vendors can be a potential solution, providing that they have been architected/implemented appropriately – and tested with extreme scale in mind.

Given that most if not all major volume Enterprises have de facto constraints on their flexibility and speed of adoption of extension technologies (not to mention change generally), perhaps there is a case for revisiting "traditional" APM tooling models. These certainly had (and have) a track record of delivering value in large enterprise deployments, albeit without some of the bells and whistles offered by later entrants. Any high scale developments made by these vendors would certainly have the advantage of leveraging the often considerable sunk investment made in them.

Provided that any constraints are well understood, and appropriate investment is made in initial commissioning and ongoing support, then this option would in our view be worth adding to the mix – for consideration, at least.

Alternatively, perhaps a "dual tool" approach may have validity – second generation APM pre-production, and traditional high volume solutions in the live environment.

For Enterprises with extremely strong nerves, and appropriate skills, "building your own" using Open Source technologies is a possibility, although it is likely to be both extremely high risk and costly. Such an approach comes with its own ongoing maintenance challenges as well.

We would like to see more open sourcing of the key components of APM, for example the agents that instrument Java and .Net applications. These, conforming to open standards, enable a flexible approach to open-APM. Choose your agents, your transport method (Apache Flume, FluentD etc.), and your data storage and analysis methods (Elastic Kibana) that are appropriate for your scale and company skillset.

Either way, we would strongly suggest that major enterprises face these issues squarely, and certainly not make significant investments in APM without appropriate high volume (production scale) Proof of Concept preliminary trialling.

Above all, put little trust in marketing. Prove it in your environment – ideally in production.

Larry Haig is Senior Consultant at Intechnica.

This blog was written with contributions by James Billingham, Performance Architect at Intechnica.

Share this

The Latest

June 26, 2019

It is inevitable that employee productivity and the quality of customer experiences suffer as a consequence of the poor performance of O365. The quick detection and rapid resolution of problems associated with O365 are top of mind for any organization to keep its business humming ...

June 25, 2019

Employees at British businesses rate computer downtime as the most significant irritant at their current workplace (41 percent) when asked to pick their top three ...

June 24, 2019

The modern enterprise network is an entirely different beast today than the network environments IT and ops teams were tasked with managing just a few years ago. With the rise of SaaS, widespread cloud migration across industries and the trend of enterprise decentralization all playing a part, the challenges IT faces in adapting their management and monitoring techniques continue to mount ...

June 20, 2019

Almost two-thirds (63%) of organizations now allow technology to be managed outside the IT department, a shift that brings both significant business advantages and increased privacy and security risks, according to the 2019 Harvey Nash/KPMG CIO Survey ...

June 19, 2019

In a post-apocalyptic world, shopping carts filled with items sit motionless in aisles, left abandoned by the humans who have mysteriously disappeared. At least that’s the cliche scene depicted by sci-fi filmmakers over the past two decades. The audience is left to wonder what happened to force people to stop what they were doing and leave everything behind. If this past weekend was any indication, Armageddon begins when Target's cash registers shut down ...

June 18, 2019

Three-quarters of organizations surveyed by Gartner increased customer experience (CX) technology investments in 2018 ...

June 17, 2019

Users today expect a more consumer-like experience and many self-service web sites are too focused on automating the submission of tickets and presenting long, technically written knowledge articles with little to no focus on UX. Understanding the need for a more modern experience, a newer concept called "self-help" now dominates the conversation in its ability to provide a more deliberate knowledge experience approach that better engages the user and dramatically improves the odds of them finding an answer ...

June 13, 2019

Establishing a digital business is top-of-mind, even more so than last year, as 91% of organizations have adopted or have plans to adopt a digital-first strategy, according to IDG Communications Digital Business Research ...

June 12, 2019

If digital transformation is to succeed at the pace enterprises demand, IT teams, the CIOs who lead them, and the boardroom must forge a far greater alignment than presently exists. That is the over-arching sentiment expressed by IT professionals in a recent survey on the state of IT infrastructure and roadblocks to digital success ...

June 11, 2019

Given the incredible amount of traffic traversing corporate WANs, it's not surprising that businesses are seeing performance issues. If anything, it's amazing applications work as well as they do ...