Log data may be many things, but one thing is for sure – it isn't sexy. In fact, in most cases, it's downright ugly, because there are really no standards out there for how log data should be structured. For decades, this fact has kept log data from being a practical source of information for anything beyond a few specific use cases, such as watching for important events (like system reboots or config changes), security monitoring (like firewall blockages), or deep troubleshooting.

Times have changed, and the most recent crop of log management vendors have taken advantage of the steady growth in processor capacity to overcome the complexity and scale challenges of harvesting and analyzing all of the log data that an IT infrastructure continuously throws off. Now there are practical ways for taking advantage of the unique perspective and insights that log data can provide on a much broader basis.

In my last post, I shared some key findings from an EMA research report published last fall that dove into the ways in which log analytics is being used to support network operations. Building on that, following are a couple of the recommendations that EMA is making on how best to think about log data as part of an integrated management architecture and strategy:

1. Think twice before planning to store all log data

While most organizations are gathering log data for analysis on a continuous, ongoing basis, only a third are storing all log entries all the time. Interesting, those organizations considering log data to be "strategic" are actually much less likely to be storing all log entries all the time than those who consider log data to be "tactical". Strategic log users prefer instead to be more surgical, looking for specific types of logs or storing all log data only when certain trigger situations occur.

2. Consolidate your log analysis tools

We find that an overwhelming majority of organizations are either currently using one centralized log analysis system or are planning to consolidate the multiple tools that they have into a single system. This makes tremendous sense if you are trying to get the most out of your log data either in support of integrated operations or simply for better collaboration and cross-team sharing.

3. Focus on fast and intuitive search capabilities

The number one challenge voiced with respect to analyzing log data is knowing what to look for. It's not surprising then that the most popular feature that IT pros look for in a log data analysis solution is fast search. The latest generation of tools have made quick and effective search a high priority, and if you don't have such capabilities in your current system, you should consider an upgrade or alternative.

4. Don't implement log data analysis as an island

Consistently, we find that organizations are getting the most value when log data collection and analysis is integrated with other data sets and analysis systems. This can be done either via log data collection/analysis tools incorporating non-log data themselves or by openly sharing log data with other management aggregation systems. Some of the strongest values are being achieved by connecting the insights available from streaming log data with other performance monitoring measures, to proactively recognize performance degradations and related root causes.

5. Log data is relevant for BSM/ITSM

EMA has found a very high usage rate of network log data for higher level BSM and ITSM type initiatives, such as service quality monitoring, unified IT operations, and CMDB. Such usages were particularly high among those who consider log data to be strategic rather than tactical. So even though log data may be ugly, don't overlook its importance in supporting your highest level management objectives.

There were a couple of surprising dichotomies uncovered in the research study as well. For instance, the top reason people value log data is that they consider it to be cost-effective, however the second greatest challenge was identified as cost of tools. Another example involves just how effective log data is. The second highest perceived value was faster time to resolution than other data sources, however the number one challenge was knowing what to look for.

Clearly there is great and growing value in collecting and analyzing log data for IT planning, operations, and security. And while there are still challenges to be faced, best practices are emerging to help everyone understand what to expect and how to get the most returns on investments into log data collection and analysis tools.