Skip to main content

5 Tips for Getting the Most Value from Logs

Ishan Mukherjee
New Relic

Logs are one of the most useful tools for observability and application performance monitoring. However, getting the most mileage from logs requires paying careful attention to planning what data to collect, the best way to display it, and the proper context for log entries.

Logs provide a comprehensive view of events and errors that occur while software is running or when a failure occurs. A log monitoring solution ingests activity records generated by applications, services, and components of the operating systems stack and writes them in the form of text files so issues can be detected and resolved before they slow down the system or impact user experience.

Configuring logs for an entire infrastructure and application stack can be overwhelming because of the sheer amount of data that is generated. Nearly every event that takes place in a system can generate a log entry, which means that modern applications stacks may throw off millions or billions of events each day.

Collecting too much irrelevant information can cause log files to swell to huge proportions and make it difficult for humans or automated solutions to spot anomalies. Conversely, capturing too little information can cause important events to be missed.

Here are five best practices that will ensure you get the greatest value from log analytics.

1. Choose carefully what to log

Decide what information is most critical to understanding system performance and configure the logging solution accordingly. Collecting too many messages can drive up storage costs and make it difficult to identify relevant information when a problem occurs.

The data you gather should be relevant and useful. Some messages may not need to be captured at all. For example, success and redirect entries, which indicate that an operation was completed as planned, are usually not very useful in troubleshooting.

Seek input from everyone on the team to ensure that their needs are considered. Log information should provide the necessary details to understand issues and make decisions at every level of the operating and application stack. Capturing metadata is crucial to pinpointing events and root causes. For example, a message stating that an operation failed is less useful than one that states what operation was attempted and why it failed.

Pay careful attention to sensitive information such as passwords, personal data, and business secrets. If you must capture this data, be sure your logging solution supports encryption. In many cases, you don't need to log this information at all.

Be sure to include timestamp information for all log messages. The level of detail should be customized to the application as some tasks require extremely precise time information while others may need no more than an hourly mark. It's best to apply whatever standard metric you choose across the entire stack so logs can be correlated with other telemetry data types like metrics and events.

2. Establish a baseline for comparison

Logs can help you understand your stack better, which is important for performance tuning as well as distinguishing between real problems and false alerts.

Your first step when adopting a log monitoring solution should be to establish a foundation that can be used to identify anomalies. Choose common scenarios that will help you determine which data points to monitor and use as a baseline. For example, application monitoring can detect if parts of an application are increasing their use of memory over time, which is a symptom of a memory leak, but only if you know what constitutes normal memory usage.

3. Choose messages that support decisions

Infrastructure tends to generate a large amount of log data, only some of which are likely to be useful to you. If your monitoring is confined to applications, you should determine which details relate most directly to the conditions you are looking for, such as slow performance or restarts, and focus on those metrics.

Log messages should provide specific information about errors. For example, a failed transaction should generate a message that includes a detailed description of the problem, the timestamp, the name of the file where the problem occurred, and the line number of the failed code.

Timestamp: 2023-04-11 14:37:05

Error: Exception caught in processOrder() method

Error Message: NullPointerException: Order object is null

Stack Trace:

at com.example.OrderProcessor.processOrder(OrderProcessor.java:36)

at com.example.Application.main(Application.java:22)

The example above tells us that the application encountered a NullPointerException while processing an order. The Order object is null, which caused the processOrder() method to throw an exception. This error occurred in the processOrder() method at line 36 of the OrderProcessor.java file. The Application.java file is the entry point to the application and the main() method called the processOrder() method.

This message will make it easier to discover why the transaction failed and where in the code the problem occurred.

4. Keep log messages concise and relevant

While verbose messages may be helpful in diagnosis, they also drive up storage needs, make log searches more difficult, and increase debugging complexity.

When formatting logs, specify that only the information needed to debug an error should be collected. Chances are you don't need every detail about the operating environment. For example, a message regarding an application program interface failure probably doesn't need information about memory usage.

5. Make sure log messages are clear

You have a variety of logging formats to choose from, including JSON, Common Event Format, the NCSA Common Log Format, the W3C Extended Log File Format, and others. Each has its strengths and weaknesses, so make your selection based on your specific needs.

Whichever option you choose, avoid arcane or overly technical message formats that will only be decipherable by a few people. Emphasize consistency and clarity to ensure that logs are accessible to everyone who needs to see them now and in the future. Some log managers make it easy to customize log parsing rules but only if the underlying data is readable.

An example of an easily parsed format is:

2023-04-12 09:27:55 INFO [server] User "John" logged in from IP address 192.168.0.1.

This format is structured and consistent with a standard date and time format, and each piece of information is separated by a specific delimiter such as a space or a comma. This makes it easy for log monitoring software to read and process.

Following these five guidelines saves money, speeds error diagnosis, and makes logs an even more valuable asset in your observability toolkit.

Ishan Mukherjee is SVP of Growth at New Relic

Hot Topics

The Latest

Businesses that face downtime or outages risk financial and reputational damage, as well as reducing partner, shareholder, and customer trust. One of the major challenges that enterprises face is implementing a robust business continuity plan. What's the solution? The answer may lie in disaster recovery tactics such as truly immutable storage and regular disaster recovery testing ...

IT spending is expected to jump nearly 10% in 2025, and organizations are now facing pressure to manage costs without slowing down critical functions like observability. To meet the challenge, leaders are turning to smarter, more cost effective business strategies. Enter stage right: OpenTelemetry, the missing piece of the puzzle that is no longer just an option but rather a strategic advantage ...

Amidst the threat of cyberhacks and data breaches, companies install several security measures to keep their business safely afloat. These measures aim to protect businesses, employees, and crucial data. Yet, employees perceive them as burdensome. Frustrated with complex logins, slow access, and constant security checks, workers decide to completely bypass all security set-ups ...

Image
Cloudbrink's Personal SASE services provide last-mile acceleration and reduction in latency

In MEAN TIME TO INSIGHT Episode 13, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at EMA discusses hybrid multi-cloud networking strategy ... 

In high-traffic environments, the sheer volume and unpredictable nature of network incidents can quickly overwhelm even the most skilled teams, hindering their ability to react swiftly and effectively, potentially impacting service availability and overall business performance. This is where closed-loop remediation comes into the picture: an IT management concept designed to address the escalating complexity of modern networks ...

In 2025, enterprise workflows are undergoing a seismic shift. Propelled by breakthroughs in generative AI (GenAI), large language models (LLMs), and natural language processing (NLP), a new paradigm is emerging — agentic AI. This technology is not just automating tasks; it's reimagining how organizations make decisions, engage customers, and operate at scale ...

In the early days of the cloud revolution, business leaders perceived cloud services as a means of sidelining IT organizations. IT was too slow, too expensive, or incapable of supporting new technologies. With a team of developers, line of business managers could deploy new applications and services in the cloud. IT has been fighting to retake control ever since. Today, IT is back in the driver's seat, according to new research by Enterprise Management Associates (EMA) ...

In today's fast-paced and increasingly complex network environments, Network Operations Centers (NOCs) are the backbone of ensuring continuous uptime, smooth service delivery, and rapid issue resolution. However, the challenges faced by NOC teams are only growing. In a recent study, 78% state network complexity has grown significantly over the last few years while 84% regularly learn about network issues from users. It is imperative we adopt a new approach to managing today's network experiences ...

Image
Broadcom

From growing reliance on FinOps teams to the increasing attention on artificial intelligence (AI), and software licensing, the Flexera 2025 State of the Cloud Report digs into how organizations are improving cloud spend efficiency, while tackling the complexities of emerging technologies ...

Today, organizations are generating and processing more data than ever before. From training AI models to running complex analytics, massive datasets have become the backbone of innovation. However, as businesses embrace the cloud for its scalability and flexibility, a new challenge arises: managing the soaring costs of storing and processing this data ...

5 Tips for Getting the Most Value from Logs

Ishan Mukherjee
New Relic

Logs are one of the most useful tools for observability and application performance monitoring. However, getting the most mileage from logs requires paying careful attention to planning what data to collect, the best way to display it, and the proper context for log entries.

Logs provide a comprehensive view of events and errors that occur while software is running or when a failure occurs. A log monitoring solution ingests activity records generated by applications, services, and components of the operating systems stack and writes them in the form of text files so issues can be detected and resolved before they slow down the system or impact user experience.

Configuring logs for an entire infrastructure and application stack can be overwhelming because of the sheer amount of data that is generated. Nearly every event that takes place in a system can generate a log entry, which means that modern applications stacks may throw off millions or billions of events each day.

Collecting too much irrelevant information can cause log files to swell to huge proportions and make it difficult for humans or automated solutions to spot anomalies. Conversely, capturing too little information can cause important events to be missed.

Here are five best practices that will ensure you get the greatest value from log analytics.

1. Choose carefully what to log

Decide what information is most critical to understanding system performance and configure the logging solution accordingly. Collecting too many messages can drive up storage costs and make it difficult to identify relevant information when a problem occurs.

The data you gather should be relevant and useful. Some messages may not need to be captured at all. For example, success and redirect entries, which indicate that an operation was completed as planned, are usually not very useful in troubleshooting.

Seek input from everyone on the team to ensure that their needs are considered. Log information should provide the necessary details to understand issues and make decisions at every level of the operating and application stack. Capturing metadata is crucial to pinpointing events and root causes. For example, a message stating that an operation failed is less useful than one that states what operation was attempted and why it failed.

Pay careful attention to sensitive information such as passwords, personal data, and business secrets. If you must capture this data, be sure your logging solution supports encryption. In many cases, you don't need to log this information at all.

Be sure to include timestamp information for all log messages. The level of detail should be customized to the application as some tasks require extremely precise time information while others may need no more than an hourly mark. It's best to apply whatever standard metric you choose across the entire stack so logs can be correlated with other telemetry data types like metrics and events.

2. Establish a baseline for comparison

Logs can help you understand your stack better, which is important for performance tuning as well as distinguishing between real problems and false alerts.

Your first step when adopting a log monitoring solution should be to establish a foundation that can be used to identify anomalies. Choose common scenarios that will help you determine which data points to monitor and use as a baseline. For example, application monitoring can detect if parts of an application are increasing their use of memory over time, which is a symptom of a memory leak, but only if you know what constitutes normal memory usage.

3. Choose messages that support decisions

Infrastructure tends to generate a large amount of log data, only some of which are likely to be useful to you. If your monitoring is confined to applications, you should determine which details relate most directly to the conditions you are looking for, such as slow performance or restarts, and focus on those metrics.

Log messages should provide specific information about errors. For example, a failed transaction should generate a message that includes a detailed description of the problem, the timestamp, the name of the file where the problem occurred, and the line number of the failed code.

Timestamp: 2023-04-11 14:37:05

Error: Exception caught in processOrder() method

Error Message: NullPointerException: Order object is null

Stack Trace:

at com.example.OrderProcessor.processOrder(OrderProcessor.java:36)

at com.example.Application.main(Application.java:22)

The example above tells us that the application encountered a NullPointerException while processing an order. The Order object is null, which caused the processOrder() method to throw an exception. This error occurred in the processOrder() method at line 36 of the OrderProcessor.java file. The Application.java file is the entry point to the application and the main() method called the processOrder() method.

This message will make it easier to discover why the transaction failed and where in the code the problem occurred.

4. Keep log messages concise and relevant

While verbose messages may be helpful in diagnosis, they also drive up storage needs, make log searches more difficult, and increase debugging complexity.

When formatting logs, specify that only the information needed to debug an error should be collected. Chances are you don't need every detail about the operating environment. For example, a message regarding an application program interface failure probably doesn't need information about memory usage.

5. Make sure log messages are clear

You have a variety of logging formats to choose from, including JSON, Common Event Format, the NCSA Common Log Format, the W3C Extended Log File Format, and others. Each has its strengths and weaknesses, so make your selection based on your specific needs.

Whichever option you choose, avoid arcane or overly technical message formats that will only be decipherable by a few people. Emphasize consistency and clarity to ensure that logs are accessible to everyone who needs to see them now and in the future. Some log managers make it easy to customize log parsing rules but only if the underlying data is readable.

An example of an easily parsed format is:

2023-04-12 09:27:55 INFO [server] User "John" logged in from IP address 192.168.0.1.

This format is structured and consistent with a standard date and time format, and each piece of information is separated by a specific delimiter such as a space or a comma. This makes it easy for log monitoring software to read and process.

Following these five guidelines saves money, speeds error diagnosis, and makes logs an even more valuable asset in your observability toolkit.

Ishan Mukherjee is SVP of Growth at New Relic

Hot Topics

The Latest

Businesses that face downtime or outages risk financial and reputational damage, as well as reducing partner, shareholder, and customer trust. One of the major challenges that enterprises face is implementing a robust business continuity plan. What's the solution? The answer may lie in disaster recovery tactics such as truly immutable storage and regular disaster recovery testing ...

IT spending is expected to jump nearly 10% in 2025, and organizations are now facing pressure to manage costs without slowing down critical functions like observability. To meet the challenge, leaders are turning to smarter, more cost effective business strategies. Enter stage right: OpenTelemetry, the missing piece of the puzzle that is no longer just an option but rather a strategic advantage ...

Amidst the threat of cyberhacks and data breaches, companies install several security measures to keep their business safely afloat. These measures aim to protect businesses, employees, and crucial data. Yet, employees perceive them as burdensome. Frustrated with complex logins, slow access, and constant security checks, workers decide to completely bypass all security set-ups ...

Image
Cloudbrink's Personal SASE services provide last-mile acceleration and reduction in latency

In MEAN TIME TO INSIGHT Episode 13, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at EMA discusses hybrid multi-cloud networking strategy ... 

In high-traffic environments, the sheer volume and unpredictable nature of network incidents can quickly overwhelm even the most skilled teams, hindering their ability to react swiftly and effectively, potentially impacting service availability and overall business performance. This is where closed-loop remediation comes into the picture: an IT management concept designed to address the escalating complexity of modern networks ...

In 2025, enterprise workflows are undergoing a seismic shift. Propelled by breakthroughs in generative AI (GenAI), large language models (LLMs), and natural language processing (NLP), a new paradigm is emerging — agentic AI. This technology is not just automating tasks; it's reimagining how organizations make decisions, engage customers, and operate at scale ...

In the early days of the cloud revolution, business leaders perceived cloud services as a means of sidelining IT organizations. IT was too slow, too expensive, or incapable of supporting new technologies. With a team of developers, line of business managers could deploy new applications and services in the cloud. IT has been fighting to retake control ever since. Today, IT is back in the driver's seat, according to new research by Enterprise Management Associates (EMA) ...

In today's fast-paced and increasingly complex network environments, Network Operations Centers (NOCs) are the backbone of ensuring continuous uptime, smooth service delivery, and rapid issue resolution. However, the challenges faced by NOC teams are only growing. In a recent study, 78% state network complexity has grown significantly over the last few years while 84% regularly learn about network issues from users. It is imperative we adopt a new approach to managing today's network experiences ...

Image
Broadcom

From growing reliance on FinOps teams to the increasing attention on artificial intelligence (AI), and software licensing, the Flexera 2025 State of the Cloud Report digs into how organizations are improving cloud spend efficiency, while tackling the complexities of emerging technologies ...

Today, organizations are generating and processing more data than ever before. From training AI models to running complex analytics, massive datasets have become the backbone of innovation. However, as businesses embrace the cloud for its scalability and flexibility, a new challenge arises: managing the soaring costs of storing and processing this data ...