Log Management for IT Ops: 5 Best Practices
February 23, 2015

Jim Frey
Kentik

Share this

Log data may be many things, but one thing is for sure – it isn't sexy. In fact, in most cases, it's downright ugly, because there are really no standards out there for how log data should be structured. For decades, this fact has kept log data from being a practical source of information for anything beyond a few specific use cases, such as watching for important events (like system reboots or config changes), security monitoring (like firewall blockages), or deep troubleshooting.

Times have changed, and the most recent crop of log management vendors have taken advantage of the steady growth in processor capacity to overcome the complexity and scale challenges of harvesting and analyzing all of the log data that an IT infrastructure continuously throws off. Now there are practical ways for taking advantage of the unique perspective and insights that log data can provide on a much broader basis.

In my last post, I shared some key findings from an EMA research report published last fall that dove into the ways in which log analytics is being used to support network operations. Building on that, following are a couple of the recommendations that EMA is making on how best to think about log data as part of an integrated management architecture and strategy:

1. Think twice before planning to store all log data

While most organizations are gathering log data for analysis on a continuous, ongoing basis, only a third are storing all log entries all the time. Interesting, those organizations considering log data to be "strategic" are actually much less likely to be storing all log entries all the time than those who consider log data to be "tactical". Strategic log users prefer instead to be more surgical, looking for specific types of logs or storing all log data only when certain trigger situations occur.

2. Consolidate your log analysis tools

We find that an overwhelming majority of organizations are either currently using one centralized log analysis system or are planning to consolidate the multiple tools that they have into a single system. This makes tremendous sense if you are trying to get the most out of your log data either in support of integrated operations or simply for better collaboration and cross-team sharing.

3. Focus on fast and intuitive search capabilities

The number one challenge voiced with respect to analyzing log data is knowing what to look for. It's not surprising then that the most popular feature that IT pros look for in a log data analysis solution is fast search. The latest generation of tools have made quick and effective search a high priority, and if you don't have such capabilities in your current system, you should consider an upgrade or alternative.

4. Don't implement log data analysis as an island

Consistently, we find that organizations are getting the most value when log data collection and analysis is integrated with other data sets and analysis systems. This can be done either via log data collection/analysis tools incorporating non-log data themselves or by openly sharing log data with other management aggregation systems. Some of the strongest values are being achieved by connecting the insights available from streaming log data with other performance monitoring measures, to proactively recognize performance degradations and related root causes.

5. Log data is relevant for BSM/ITSM

EMA has found a very high usage rate of network log data for higher level BSM and ITSM type initiatives, such as service quality monitoring, unified IT operations, and CMDB. Such usages were particularly high among those who consider log data to be strategic rather than tactical. So even though log data may be ugly, don't overlook its importance in supporting your highest level management objectives.

There were a couple of surprising dichotomies uncovered in the research study as well. For instance, the top reason people value log data is that they consider it to be cost-effective, however the second greatest challenge was identified as cost of tools. Another example involves just how effective log data is. The second highest perceived value was faster time to resolution than other data sources, however the number one challenge was knowing what to look for.

Clearly there is great and growing value in collecting and analyzing log data for IT planning, operations, and security. And while there are still challenges to be faced, best practices are emerging to help everyone understand what to expect and how to get the most returns on investments into log data collection and analysis tools.

Jim Frey is VP of Strategic Alliances at Kentik
Share this

The Latest

May 21, 2019

Findings of the Digital Employee Experience survey from VMware show correlation between enabling employees with a positive digital experience (i.e., device choice/flexibility, seamless access to apps, remote work capabilities) and an organization's competitive position, revenue growth and employee sentiment ...

May 20, 2019

In today's competitive landscape, businesses must have the ability and process in place to face new challenges and find ways to successfully tackle them in a proactive manner. For years, this has been placed on the shoulders of DevOps teams within IT departments. But, as automation takes over manual intervention to increase speed and efficiency, these teams are facing what we know as IT digitization. How has this changed the way companies function over the years, and what do we have to look forward to in the coming years? ...

May 16, 2019

Although the vast majority of IT organizations have implemented a broad variety of systems and tools to modernize, simplify and streamline data center operations, many are still burdened by inefficiencies, security risks and performance gaps in their IT infrastructure as well as the excessive time it takes to manage legacy infrastructure, according to the State of IT Transformation, a report from Datrium ...

May 15, 2019

When it comes to network visibility, there are a lot of discussions about packet broker technology and the various features these solutions provide to network architects and IT managers. Packet brokers allow organizations to aggregate the data required for a variety of monitoring solutions including network performance monitoring and diagnostic (NPMD) platforms and unified threat management (UTM) appliances. But, when it comes to ensuring these solutions provide the insights required by NetOps and security teams, IT can spend an exorbitant amount of time dealing with issues around adds, moves and changes. This can have a dramatic impact on budgets and tool availability. Why does this happen? ...

May 14, 2019

Data may be pouring into enterprises but IT professionals still find most of it stuck in siloed departments and weeks away from being able to drive any valued action. Coupled with the ongoing concerns over security responsiveness, IT teams have to push aside other important performance-oriented data in order to ensure security data, at least, gets prominent attention. A new survey by Ivanti shows the disconnect between enterprise departments struggling to improve operations like automation while being challenged with a siloed structure and a data onslaught ...

May 13, 2019

A subtle, deliberate shift has occurred within the software industry which, at present, only the most innovative organizations have seized upon for competitive advantage. Although primarily driven by Artificial Intelligence (AI), this transformation strikes at the core of the most pervasive IT resources including cloud computing and predictive analytics ...

May 09, 2019

When asked who is mandated with developing and delivering their organization's digital competencies, 51% of respondents say their IT departments have a leadership role. The critical question is whether IT departments are prepared to take on a leadership role in which collaborating with other functions and disseminating knowledge and digital performance data are requirements ...

May 08, 2019

The Economist Intelligence Unit just released a new study commissioned by Riverbed that explores nine digital competencies that help organizations improve their digital performance and, ultimately, achieve their objectives. Here's a brief summary of 7 key research findings you'll find covered in detail in the report ...

May 07, 2019

Today, the overall customer scenario has digitally transformed and practically there is no limitation to the ways in which the target customers can be reached. These opportunities are throwing multiple challenges for brands and enterprises, and one of the prominent ones is to ensure Omni Channel experience for customers ...

May 06, 2019

Most businesses (92 percent of respondents) see the potential value of data and 36 percent are already monetizing their data, according to the Global Data Protection Index from Dell EMC. While this acknowledgement is positive, however, most respondents are struggling to properly protect their data ...