Distributed Tracing - The Next Step of APM
September 26, 2016

Gergely Nemeth
RisingStack

Share this

Transforming your monolithic application into a microservices-based one is not as easy as many think. When you are breaking a software down into smaller pieces, you're moving the communication to the network layer and the complexity of your architecture is heavily increasing. Other issues arise as well since performance monitoring and finding the root source of an error becomes extremely challenging.

With the rise of microservices, developers need proper Application Performance Management (APM) tools to develop and operate their applications successfully. This blog examines the particular difficulties of monitoring microservices and what APM should be able to do to alleviate the major pain-points of monitoring and debugging them.

Figuring Out What Breaks in a Microservices Application

In a monolithic application, specific code pieces are communicating in the applications memory. It means that when something breaks, the log files will probably be useful to find the cause of an error and you can start debugging right away.

When something goes wrong in a microservices call-chain – called distributed transactions – all of the services participating in that request will throw back an error. It means that you need an excellent logging system, and if you have one, you'll still experience problems since you have to manually correlate the log files to find out what caused the trouble in the first place.

What's the solution to this problem? Distributed Tracing.

For microservices applications, there is a much more sophisticated application performance monitoring method available, called Distributed Tracing.

For distributed tracing, you have to attach a correlation ID to your requests which you can use to track what services are communicating with each other. With these IDs, you can to reverse engineer what happened during an error since all of the services involved in a request will be there for you to see instantly.

Next-gen APM solutions can already attach correlation IDs to requests and can also group the services taking part in a transaction and visualize the exact dataflow on a simple tree-graph. A tool like this enables you to see the distributed call stacks, the root cause of an error, and the dependencies between your microservices.


A distributed tracing timeline shows all of the services taking part in a certain transaction and the source of the error that later propagated back to all of them



There are only a few Distributed Tracing solutions available right now, but you can find open source solutions for Java monitoring and a SaaS solution focusing on Node.js – the technology primarily used for building microservices.

The concept of Distributed Tracing is based on Google’s Dapper whitepaper, which is publicly available here.

Increasing Architecture Complexity and Slow Response Times

As I mentioned above, increasing architecture complexity comes by the definition with microservices.

In a microservices application, the services will usually use a transport layer, like the HTTP protocol, RabbitMQ or Kafka. It will add delays to the internal communication of your application, and when you put services into a call chain, your response times will be higher. A modern APM solution must be prepared for this, and support message queue communication to map out a distributed system. If you have one, you'll be able to figure out what makes it slow.

Companies that build microservices should be able to deal with slow response times by using a distributed tracing APM tool. Correlation IDs let you visualize whole call chains and look for slow response times, whether it's caused by a slow service or the slow network.

If the transaction timeline graph shows that your services are fine, but your network is slow, you can to speed up your application by investigating that. One time, we could figure out that our PaaS provider was using external routing, so every request between our services went outside the public network and back, it reached more than 30 network hops, which caused the bad response time. The next step, in this case, was to choose another without external routing.

If your network times are fine, you have to investigate what slows your services down. It's quite easy if you have an APM with a built-in CPU profiler, or you have some profiling solution enabled. Requesting a CPU profile in the right time (presumably when the response time of a service gets high) will allow you to look for the slow functions and find the location of them. Thankfully, Chrome's Developer Tools support loading and analyzing javascript CPU profiles which solves this problem.

Conclusion

Application performance monitoring solutions have been around for a while, offering the same functionalities for years without major breakthroughs. This has to change. The way how we develop and deploy software is not the same than it was three years ago, and legacy APM tools are not helping as much as they used to. We need solutions that are treating microservices as first class citizens, and the developers who are building them too.

Gergely Nemeth is Co-Founder and CEO of RisingStack.

Share this

The Latest

February 29, 2024

Despite the growth in popularity of artificial intelligence (AI) and ML across a number of industries, there is still a huge amount of unrealized potential, with many businesses playing catch-up and still planning how ML solutions can best facilitate processes. Further progression could be limited without investment in specialized technical teams to drive development and integration ...

February 28, 2024

With over 200 streaming services to choose from, including multiple platforms featuring similar types of entertainment, users have little incentive to remain loyal to any given platform if it exhibits performance issues. Big names in streaming like Hulu, Amazon Prime and HBO Max invest thousands of hours into engineering observability and closed-loop monitoring to combat infrastructure and application issues, but smaller platforms struggle to remain competitive without access to the same resources ...

February 27, 2024

Generative AI has recently experienced unprecedented dramatic growth, making it one of the most exciting transformations the tech industry has seen in some time. However, this growth also poses a challenge for tech leaders who will be expected to deliver on the promise of new technology. In 2024, delivering tangible outcomes that meet the potential of AI, and setting up incubator projects for the future will be key tasks ...

February 26, 2024

SAP is a tool for automating business processes. Managing SAP solutions, especially with the shift to the cloud-based S/4HANA platform, can be intricate. To explore the concerns of SAP users during operational transformations and automation, a survey was conducted in mid-2023 by Digitate and Americas' SAP Users' Group ...

February 22, 2024

Some companies are just starting to dip their toes into developing AI capabilities, while (few) others can claim they have built a truly AI-first product. Regardless of where a company is on the AI journey, leaders must understand what it means to build every aspect of their product with AI in mind ...

February 21, 2024

Generative AI will usher in advantages within various industries. However, the technology is still nascent, and according to the recent Dynatrace survey there are many challenges and risks that organizations need to overcome to use this technology effectively ...

February 20, 2024

In today's digital era, monitoring and observability are indispensable in software and application development. Their efficacy lies in empowering developers to swiftly identify and address issues, enhance performance, and deliver flawless user experiences. Achieving these objectives requires meticulous planning, strategic implementation, and consistent ongoing maintenance. In this blog, we're sharing our five best practices to fortify your approach to application performance monitoring (APM) and observability ...

February 16, 2024

In MEAN TIME TO INSIGHT Episode 3, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses network security with Chris Steffen, VP of Research Covering Information Security, Risk, and Compliance Management at EMA ...

February 15, 2024

In a time where we're constantly bombarded with new buzzwords and technological advancements, it can be challenging for businesses to determine what is real, what is useful, and what they truly need. Over the years, we've witnessed the rise and fall of various tech trends, such as the promises (and fears) of AI becoming sentient and replacing humans to the declaration that data is the new oil. At the end of the day, one fundamental question remains: How can companies navigate through the tech buzz and make informed decisions for their future? ...

February 14, 2024

We increasingly see companies using their observability data to support security use cases. It's not entirely surprising given the challenges that organizations have with legacy SIEMs. We wanted to dig into this evolving intersection of security and observability, so we surveyed 500 security professionals — 40% of whom were either CISOs or CSOs — for our inaugural State of Security Observability report ...