SRE Revisited: SLO in the Age of Microservices
February 16, 2022

Dotan Horovits
Logz.io

Share this

Site Reliability Engineering (SRE) practice was established by Google nearly 20 years ago and was popularized with Google's monumental SRE Book. Everyone's been attempting to follow that iconic path ever since.

What Are SLI, SLO and SLA Anyway?

SRE practice revolves around the concepts of SLO, SLI, SLA, and the related "error budget." The Google SRE Book, which laid the foundations for the practice, defines them as follows:

Service Level Indicator (SLI) is a carefully defined quantitative measure of some aspect of the level of service that is provided. Common examples include latency, error rate, request throughput and availability.

Service Level Objective (SLO) is a target value or range of values for a service level that is measured by an SLI. For example, an SLO can state that the average latency per request should be under 120 milliseconds. Many companies set SLO targets as a number of nines, for example "five nines" uptime means 99.999% uptime, which means a maximum of 5.26 minutes downtime per year.

Service Level Agreement (SLA) is an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain. Put simply, if you've got a penalty attached to breaching an SLO — you're talking SLA.

SRE typically doesn't deal with SLA directly, as it's more commercial in nature. Rather, the SRE is concerned with meeting the SLOs that underlie the SLA.

Error budget is an important related concept, which determines the rate at which the SLOs can be missed. Error budget is an important indicator for meeting the SLO goals consistently, and SRE teams track it weekly and even daily. Error budget is also important, as it enables downtime windows that can be used for maintenance, experimenting with new features, and other innovation.

So How Is SRE Changing in the Age of Microservices?

The SRE concepts remain valid these days as they've been before. But then again, their system of reference has drastically changed.

SLOs have traditionally referred to assets such as the application's binary or other deployment artifacts. That made perfect sense when the binary encapsulated the entire application, or a major piece of it such as the business logic tier.

In the world of microservices, however, the deployed artifacts are, well — microservices. And as the name suggests, each individual micro-service executes only a tiny piece of functionality.
In such systems it makes little sense to track SLO on the individual artifacts. SLO should refer to the functionality that is meaningful to the users.

Therefore, the practice of monitoring SLO on the service binary is no longer relevant.

In microservice architecture, rather than the individual service, it is the mesh of services that needs to play well together to execute. In these systems, tens and even hundreds of services may be traversed in order to satisfy a single user request. In these systems, rather than monitoring individual services, SLI and SLO should be defined for flows and user journeys.

The complex request flows brought about the rise of Distributed Tracing discipline. It enables tracing a single request through the call chain between the different interacting microservices, by propagating context along the request path. It's typically used to pinpoint where failures occur and what causes poor performance. However, tracing can also be used to trace FinOps and business workflows.

Google Dapper is in fact one of the forefathers of modern distributed tracing, which served as inspiration for tools such as Jaeger and Zipkin. According to Ramón Medrano Llamas, a staff site reliability engineer at Google who leads the SRE team in charge of Google's identity services, "it's the closest tool we have right now for a definite observability tool."

To make things more interesting, these user journeys not only span several services, but may actually involve a few requests. Let's look for an example at the Google sign-in process: as Ramón from Google shares, what used to be a matter of three calls in the pre-microservices era, is today a full business flow, consisting of multiple steps: loading page, then putting username & password, then second factor auth. and so on. Each individual step is a request, triggering a chain of microservices to satisfy. And the overall multi-step flow isn't stateful, it depends on the end user to carry it over from step to step. Monitoring these business flows is less trivial, but that's what we need to measure to ensure the functionality we deliver to the user.

How should you go about defining SLO then?

■ Start by identifying the main flows and user journeys in your system. These should be well defined as part of the system and product specifications.

■ Then for each flow, clarify the expected end result from the user perspective, to define the SLI. This should also be a direct derivative of the product flow definition.

■ If needed, add missing instrumentation to track, collect and aggregate the relevant metrics to measure the defined SLI.

■ Make sure that the relevant flow context is propagated throughout your system. This will serve to associate a metric with the respective tenant, account or similar entities.

■ Carefully define the SLO around each SLI, to meet technical and business goals. This often involves additional stakeholders, to factor in the tradeoff between stability and velocity of innovation.

■ Start monitoring the SLI and set relevant alerting according to the threshold defined by the SLO. Track your error budget regularly to see how close you get to the SLO bar.

Dotan Horovits is Principal Developer Advocate at Logz.io
Share this

The Latest

June 23, 2022

Digital businesses don't invest in monitoring for monitoring's sake. They do it to make the business run better. Every dollar spent on observability — every hour your team spends using monitoring tools or responding to what they reveal — should tie back directly to business outcomes: conversions, revenues, brand equity. If they don't? You might be missing the forest for the trees ...

June 22, 2022

Every day, companies are missing customer experience (CX) "red flags" because they don't have the tools to observe CX processes or metrics. Even basic errors or defects in automated customer interactions are left undetected for days, weeks or months, leading to widespread customer dissatisfaction. In fact, poor CX and digital technology investments are costing enterprises billions of dollars in lost potential revenue ...

June 21, 2022

Organizations are moving to microservices and cloud native architectures at an increasing pace. The primary incentive for these transformation projects is typically to increase the agility and velocity of software release and product innovation. These dynamic systems, however, are far more complex to manage and monitor, and they generate far higher data volumes ...

June 16, 2022

Global IT teams adapted to remote work in 2021, resolving employee tickets 23% faster than the year before as overall resolution time for IT tickets went down by 7 hours, according to the Freshservice Service Management Benchmark Report from Freshworks ...

June 15, 2022

Once upon a time data lived in the data center. Now data lives everywhere. All this signals the need for a new approach to data management, a next-gen solution ...

June 14, 2022

Findings from the 2022 State of Edge Messaging Report from Ably and Coleman Parkes Research show that most organizations (65%) that have built edge messaging capabilities in house have experienced an outage or significant downtime in the last 12-18 months. Most of the current in-house real-time messaging services aren't cutting it ...

June 13, 2022
Today's users want a complete digital experience when dealing with a software product or system. They are not content with the page load speeds or features alone but want the software to perform optimally in an omnichannel environment comprising multiple platforms, browsers, devices, and networks. This calls into question the role of load testing services to check whether the given software under testing can perform optimally when subjected to peak load ...
June 09, 2022

Networks need to be up and running for businesses to continue operating and sustaining customer-facing services. Streamlining and automating network administration tasks enable routine business processes to continue without disruption, eliminating any network downtime caused by human error or other system flaws ...

June 08, 2022

Enterprises have had access to various Project and Portfolio Management (PPM) tools for quite a few years, to guide in their project selection and execution lifecycle. Yet, in spite of the digital evolution of management software, many organizations still fail to construct an effective PPM plan or utilize cutting-edge management tools ...

June 07, 2022

It has become increasingly difficult for DevOps and SRE teams to minimize the impact of issues and ensure high-quality end-user experiences. In this blog, I'm going to propose a new approach to support real-time use cases — edge observability — that enables you to detect issues as they occur and resolve them in minutes ...