Log Management for IT Ops: 5 Best Practices
February 23, 2015

Jim Frey
Kentik

Share this

Log data may be many things, but one thing is for sure – it isn't sexy. In fact, in most cases, it's downright ugly, because there are really no standards out there for how log data should be structured. For decades, this fact has kept log data from being a practical source of information for anything beyond a few specific use cases, such as watching for important events (like system reboots or config changes), security monitoring (like firewall blockages), or deep troubleshooting.

Times have changed, and the most recent crop of log management vendors have taken advantage of the steady growth in processor capacity to overcome the complexity and scale challenges of harvesting and analyzing all of the log data that an IT infrastructure continuously throws off. Now there are practical ways for taking advantage of the unique perspective and insights that log data can provide on a much broader basis.

In my last post, I shared some key findings from an EMA research report published last fall that dove into the ways in which log analytics is being used to support network operations. Building on that, following are a couple of the recommendations that EMA is making on how best to think about log data as part of an integrated management architecture and strategy:

1. Think twice before planning to store all log data

While most organizations are gathering log data for analysis on a continuous, ongoing basis, only a third are storing all log entries all the time. Interesting, those organizations considering log data to be "strategic" are actually much less likely to be storing all log entries all the time than those who consider log data to be "tactical". Strategic log users prefer instead to be more surgical, looking for specific types of logs or storing all log data only when certain trigger situations occur.

2. Consolidate your log analysis tools

We find that an overwhelming majority of organizations are either currently using one centralized log analysis system or are planning to consolidate the multiple tools that they have into a single system. This makes tremendous sense if you are trying to get the most out of your log data either in support of integrated operations or simply for better collaboration and cross-team sharing.

3. Focus on fast and intuitive search capabilities

The number one challenge voiced with respect to analyzing log data is knowing what to look for. It's not surprising then that the most popular feature that IT pros look for in a log data analysis solution is fast search. The latest generation of tools have made quick and effective search a high priority, and if you don't have such capabilities in your current system, you should consider an upgrade or alternative.

4. Don't implement log data analysis as an island

Consistently, we find that organizations are getting the most value when log data collection and analysis is integrated with other data sets and analysis systems. This can be done either via log data collection/analysis tools incorporating non-log data themselves or by openly sharing log data with other management aggregation systems. Some of the strongest values are being achieved by connecting the insights available from streaming log data with other performance monitoring measures, to proactively recognize performance degradations and related root causes.

5. Log data is relevant for BSM/ITSM

EMA has found a very high usage rate of network log data for higher level BSM and ITSM type initiatives, such as service quality monitoring, unified IT operations, and CMDB. Such usages were particularly high among those who consider log data to be strategic rather than tactical. So even though log data may be ugly, don't overlook its importance in supporting your highest level management objectives.

There were a couple of surprising dichotomies uncovered in the research study as well. For instance, the top reason people value log data is that they consider it to be cost-effective, however the second greatest challenge was identified as cost of tools. Another example involves just how effective log data is. The second highest perceived value was faster time to resolution than other data sources, however the number one challenge was knowing what to look for.

Clearly there is great and growing value in collecting and analyzing log data for IT planning, operations, and security. And while there are still challenges to be faced, best practices are emerging to help everyone understand what to expect and how to get the most returns on investments into log data collection and analysis tools.

Jim Frey is VP of Strategic Alliances at Kentik
Share this

The Latest

August 19, 2019

One common infrastructure challenge arises with virtual private networks (VPNs). VPNs have long been relied upon to deliver the network connectivity and security enterprises required at a price they could afford. Organizations still routinely turn to them to provide internal and trusted third-parties with "secure" remote access to isolated networks. However, with the rise in mobile, IoT, multi- and hybrid-cloud, as well as edge computing, traditional enterprise perimeters are extending and becoming blurred ...

August 15, 2019

The configuration management database (CMDB), along with its more federated companion, the configuration management system (CMS), has been bathed in a deluge of negative opinions from all fronts — industry experts, vendors, and IT professionals. But from what recent EMA research on analytics, ITSM performance and other areas is indicating, those negative views seem to be missing out on a real undercurrent of truth — that CMDB/CMS alignments, whatever their defects, strongly skew to success in terms of overall IT progressiveness and effectiveness ...

August 14, 2019

The on-demand economy has transformed the way we move around, eat, learn, travel and connect at a massive scale. However, with disruption and big aspirations comes big, complex challenges. To take these challenges head-on, on-demand economy companies are finding new ways to deliver their services and products to an audience with ever-increasing expectations, and that's what we'll look at in this blog ...

August 13, 2019

To thrive in today's highly competitive digital business landscape, organizations must harness their "digital DNA." In other words, they need to connect all of their systems and databases — including various business applications, devices, big data and any instances of IoT and hybrid cloud environments — so they're accessible and actionable. By integrating all existing components and new technologies, organizations can gain a comprehensive, trusted view of their business functions, thereby enabling more agile deployment processes and ensuring scalable growth and relevance over the long-term ...

August 12, 2019

Advancements in technology innovation are happening so quickly, the decision of where and when to transform can be a moving target for businesses. When done well, digital transformation improves the customer experience while optimizing operational efficiency. To get there, enterprises must encourage experimentation to overcome organizational obstacles. In other words ...

August 08, 2019

IoT adoption is growing rapidly, and respondents believe 30% of their company’s revenue two years from now will be due to IoT, according to the new IoT Signals report from Microsoft Corp ...

August 07, 2019

It's been all over the news the last few months. After two fatal crashes, Boeing was forced to ground its 737. The doomed model is now undergoing extensive testing to get it back into service and production. Large organizations often tell stakeholders that even though all software goes through extensive testing, this type of thing “just happens.” But that is exactly the problem. While the human component of application development and testing won't go away, it can be eased and supplemented by far more efficient and automated methods to proactively determine software health and identify flaws ...

August 06, 2019

Despite significant investment in AI, many companies are still struggling to stabilize and scale their AI initiatives, according to State of Development and Operations of AI Applications 2019 from Dotscience ...

August 05, 2019

IT has two principal functions: create a network that satisfies the business needs of the company and then optimize that network. Unfortunately, the modern enterprise is buried under a myriad of network improvement projects that often do not get deployed ...

August 01, 2019

Even large companies are not yet realizing the potential of digital transformation, according to a new study from Cherwell Software, The Power of Process Integration in the Information Age ...