Log Management for IT Ops: 5 Best Practices
February 23, 2015

Jim Frey
Kentik

Share this

Log data may be many things, but one thing is for sure – it isn't sexy. In fact, in most cases, it's downright ugly, because there are really no standards out there for how log data should be structured. For decades, this fact has kept log data from being a practical source of information for anything beyond a few specific use cases, such as watching for important events (like system reboots or config changes), security monitoring (like firewall blockages), or deep troubleshooting.

Times have changed, and the most recent crop of log management vendors have taken advantage of the steady growth in processor capacity to overcome the complexity and scale challenges of harvesting and analyzing all of the log data that an IT infrastructure continuously throws off. Now there are practical ways for taking advantage of the unique perspective and insights that log data can provide on a much broader basis.

In my last post, I shared some key findings from an EMA research report published last fall that dove into the ways in which log analytics is being used to support network operations. Building on that, following are a couple of the recommendations that EMA is making on how best to think about log data as part of an integrated management architecture and strategy:

1. Think twice before planning to store all log data

While most organizations are gathering log data for analysis on a continuous, ongoing basis, only a third are storing all log entries all the time. Interesting, those organizations considering log data to be "strategic" are actually much less likely to be storing all log entries all the time than those who consider log data to be "tactical". Strategic log users prefer instead to be more surgical, looking for specific types of logs or storing all log data only when certain trigger situations occur.

2. Consolidate your log analysis tools

We find that an overwhelming majority of organizations are either currently using one centralized log analysis system or are planning to consolidate the multiple tools that they have into a single system. This makes tremendous sense if you are trying to get the most out of your log data either in support of integrated operations or simply for better collaboration and cross-team sharing.

3. Focus on fast and intuitive search capabilities

The number one challenge voiced with respect to analyzing log data is knowing what to look for. It's not surprising then that the most popular feature that IT pros look for in a log data analysis solution is fast search. The latest generation of tools have made quick and effective search a high priority, and if you don't have such capabilities in your current system, you should consider an upgrade or alternative.

4. Don't implement log data analysis as an island

Consistently, we find that organizations are getting the most value when log data collection and analysis is integrated with other data sets and analysis systems. This can be done either via log data collection/analysis tools incorporating non-log data themselves or by openly sharing log data with other management aggregation systems. Some of the strongest values are being achieved by connecting the insights available from streaming log data with other performance monitoring measures, to proactively recognize performance degradations and related root causes.

5. Log data is relevant for BSM/ITSM

EMA has found a very high usage rate of network log data for higher level BSM and ITSM type initiatives, such as service quality monitoring, unified IT operations, and CMDB. Such usages were particularly high among those who consider log data to be strategic rather than tactical. So even though log data may be ugly, don't overlook its importance in supporting your highest level management objectives.

There were a couple of surprising dichotomies uncovered in the research study as well. For instance, the top reason people value log data is that they consider it to be cost-effective, however the second greatest challenge was identified as cost of tools. Another example involves just how effective log data is. The second highest perceived value was faster time to resolution than other data sources, however the number one challenge was knowing what to look for.

Clearly there is great and growing value in collecting and analyzing log data for IT planning, operations, and security. And while there are still challenges to be faced, best practices are emerging to help everyone understand what to expect and how to get the most returns on investments into log data collection and analysis tools.

Jim Frey is VP of Strategic Alliances at Kentik
Share this

The Latest

December 15, 2017

CIOs around the globe are more determined than ever to achieve digital transformation within their organizations despite setbacks, according to a survey by Logicalis ...

December 14, 2017

The Spiceworks 2018 IT Career Outlook found that 32 percent of IT professionals plan to search for or take an IT job with a new employer in the next 12 months ...

December 12, 2017

Downtime and security risks were present in each cloud environment tested, according to 2016 Private Cloud Resiliency Benchmarks, a report from Continuity Software ...

December 11, 2017

Companies that empower employees with the applications they want and need, and make them readily accessible — anytime, anywhere, on any device — can benefit from measurable gains at the individual and organizational level, according to a survey, The Impact of the Digital Workforce: A New Equilibrium of the Digitally Transformed Enterprise, conducted by VMware ...

December 08, 2017

Metrics-oriented thinking is key to continuous improvement – and a core tenant of any agile or DevOps philosophy. Metrics are factual and once agreed upon, these facts are used to drive discussions and methods. They also allow for a collaborative effort to execute decisions that contribute towards business outcomes ...

December 06, 2017

The recent outage of the University of Cambridge website hosting Stephen Hawking's doctoral thesis is a prime example of what happens when niche websites become exposed to mainstream levels of traffic ...

December 05, 2017

Even as many organizations continue to adopt multi-cloud technologies as part of their dramatic transformation, the mainframe remains a relevant and growing data center hub for many, according to BMC's 12th annual Mainframe Research Report ...

December 04, 2017

Banks are laying the foundation for the digitization of their businesses and anticipate emerging technologies -- from IoT to biometric authentications and blockchain -- to make a substantial imprint on the industry within five years, according to a recent survey of banking professionals commissioned by VMware ...

December 01, 2017

A recent blog on APMdigest — Protecting Network Performance is as Essential as Securing the Network — mentions that performance issues and outages are possible when security tools (like an IPS, WAF, etc.) are inserted inline. However, one easy way to mitigate this concern is to deploy a bypass switch before the inline tool ...

November 30, 2017

While self-service and self-help IT are in common practice, about half of organizations surveyed are still struggling with full deployment and realizing its value, according to a new report by Ivanti and the Service Desk Institute ...