APM Monitoring Elasticsearch
August 21, 2018

Chris Klug
CA Technologies

Seemingly everything we do online now — from social media to website browsing — creates data that is collected and analyzed. When it comes to enterprise IT software applications, we already know that they also can generate a large amount of data, often through the applications' and systems' logging features. It has been quickly understood that the more data you can collect, the more information is available for you to gain insight and make decisions with. Some data is good, more is better. Elasticsearch has become very popular because it is flexible in that it allows different data sources to be stored while also keeping it manageable, searchable and fast. Because of this, it is quickly becoming a staple in most corporate IT environments.

As noted, Elasticsearch provides the ability to store and to quickly search large volumes of data. This offers the benefit of providing near real-time search capabilities. Elasticsearch's full-text search capabilities make it extremely useful for IT environments because of its ability to search patterns or specific strings and error messages in log files. Entire log files can be quickly stored and searched, which is not easy to do or efficient in a traditional relational database. Elasticsearch does this by pulling in raw data, such as log files, and storing them as documents in a json type structure. Elasticsearch can then search those documents, no matter the data set size, at speeds that a relational database cannot match.

Elasticsearch also has the added benefit of being scalable, in that there doesn't seem to be a limit to the amount of data that can be stored. If more storage is needed, it is as simple as just adding more nodes.

Another useful feature of Elasticsearch is that it leverages a distributed architecture, which also helps to deliver the fast search results. Elasticsearch's distributed architecture means that the collected data is not stored within a number of database tables, but rather it's stored across the different nodes of an Elasticsearch cluster. This eliminates the need for complex queries and allows for the cumulative processing power of each of the Elasticsearch nodes to be applied to the job resulting in a reduction of query times from seconds or minutes to milliseconds.

Since Enterprise IT applications today have begun to rely on Elasticsearch more frequently, it's important to monitor the Elasticsearch cluster to help ensure that it is performing optimally at all times.

Elasticsearch has a REST API that not only provides the ability to manage your data, it provides APIs to monitor your cluster as well. There are literally hundreds of metrics available through the different REST APIs from a very high cluster level down to detailed information on each node in the cluster.

The best API to use to get a high-level view of the status of your Elasticsearch cluster is the Cluster Health API. Since it can quickly let you know if everything is ok or if there is a problem that needs to be investigated further, take a look at this API when starting out on your Elasticsearch monitoring journey.

The other main REST APIs that you can use to monitor Elasticsearch are:

1. Node Stats API - The Node Stats API provides detailed information for each node in your cluster. In addition to index metrics, it also can provide metrics such as CPU utilization, memory usage, disk usage and jvm heap.

2. Cluster Stats API - The Cluster Stats API essentially rolls up all of the metrics from each node into a single metric. 

3. Index Stats API - The Index Stats API allows you to look at your cluster from an index point of view.

4. Pending Tasks API - The Pending Tasks API shows you any cluster level metadata tasks that a master node might be performing.

While Elasticsearch works hard for you, there's now an ever-growing, vast sea of data that you need to monitor. That's why we're leveraging Elasticsearch as the foundation of CA's Digital Experience Insights monitoring and analytics platform. We've created a highly performant, scalable, real-time data ingestion and processing analytics platform leveraging all the power of Elasticsearch to help you deliver flawless customer experiences.

Chris Klug is Sr. Engineering Services Architect at CA Technologies
Share this