Skip to main content

How to Create Programmatic Service Level Indicators and Service Level Objectives

Ishan Mukherjee
New Relic

Programmatically tracked service level indicators (SLIs) are foundational to every site reliability engineering practice. When engineering teams have programmatic SLIs in place, they lessen the need to manually track performance and incident data. They're also able to reduce manual toil because our DevOps teams define the capabilities and metrics that define their SLI data, which they collect automatically — hence "programmatic."

Programmatic SLIs have three key characteristics: they're current (they reflect the state of a system right now), they're automated (they're reported by instrumentation, not by humans), and they're useful (they're selected based on what a system's user cares about). In this post, I'll explain how site reliability engineers (SREs) can help their teams develop and create programmatic SLIs.

SLIs — Identifying Capabilities

An important part of creating programmatic SLIs is identifying the capability of the system or service for which you're creating the SLI. Here are a few definitions:

■ A system is a group of services and infrastructure components that exposes one or more capabilities to external customers (either end users or other internal teams).

■ A service is a runtime process (or a horizontally-scaled tier of processes) that makes up a portion of a system.

■ A capability is a particular aspect of functionality exposed by a service to its users, phrased in plain-language terms.

SLIs and SLOs — Indicators and Objectives

But first, we need some more definitions. An indicator is something you can measure about a system that acts as a proxy for the customer experience. An objective is a goal for a specific indicator that you're committed to achieving.

Configuring indicators and objectives is the easy part. The hard part is thinking through what measurable system behavior serves as a proxy for customer experience. When setting system-level SLIs, think about the key performance indicators (KPIs) for those systems, for example:

■ User-facing system KPIs most often include availability, latency, and throughput.

■ Storage system KPIs often emphasize latency, availability, and durability.

■ Big data systems, such as data processing pipelines, typically use KPIs such as throughput and end-to-end latency.

Your indicators and objectives should provide an accurate snapshot of the impact of your system on your customers.

A more precise description of the indicator and objective relationship is to say that SLIs are expressed in relation to service level objectives (SLOs). When you think about the availability of a system, for example, SLIs are the key measurements of the availability of the system while SLOs are the goals you set for how much availability you expect out of that system. And service level agreements (SLAs) explain the results of breaking the SLO commitments.

Create Programmatic SLIs

You should write your programmatic SLIs in collaboration with your product managers, engineering managers, and individual contributors who work on a system. To define your programmatic SLIs (and SLOs), apply these steps:

1. Identify the system and its services.

2. Identify the customer-facing capabilities of the system or services.

3. Articulate a plain-language definition of what it means for each capability to be available.

4. Define one or more SLIs for that definition.

5. Measure the system to get a baseline.

6. Define an SLO for each capability, and track how you perform against it.

7. Iterate and refine our system, and fine-tune the SLOs over time.

Example capabilities and definitions

Here are two example capabilities and definitions for an imaginary team that manages an imaginary dashboard service:

Capability: Dashboards overview.

Availability Definition: Customers are able to select the dashboard launcher, and see a list of all dashboards available to them.

Capability: Dashboards detail view.

Availability Definition: Customers can view a dashboard, and widgets render accurately and timely manner.

To express these availability definitions as programmatic SLIs (with SLOs to measure them), you'd state these service capabilities as:

■ Requests for the full list of available dashboards returns within 100 milliseconds 99.9% of the time.

■ Requests to open the dashboard launcher complete without error 99.9% of the time.

■ Requests for an individual dashboard return within 100 milliseconds 99.9% of the time.

■ Requests to open an individual dashboard complete without error 99.9% of the time.

After you've settled on your SLIs, they should be reasonably stable, but systems evolve, and you'll need to revisit them regularly. It's a good idea to revisit them quarterly, or whenever you make changes to your services, traffic volume, and upstream and downstream dependencies.

Ishan Mukherjee is SVP of Growth at New Relic

Hot Topics

The Latest

Cloud migration is a highly strategic decision that involves leadership sponsorship, business justifications for moving to the cloud, and a clear understanding of expected value. Lack of this alignment can be the reigning cause of cost and budget overruns and why almost half of the migration efforts underway today will fail in the next three years ...

One of the most misunderstood culprits of poor application performance is packet loss. Even minimal packet loss can cripple the throughput of a high-speed connection, making enterprise applications sluggish and frustrating for remote employee ... So, what's going wrong? And why does adding more bandwidth fail to fix the issue? ...

Image
Cloudbrink

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 5 covers the infrastructure and hardware supporting AI ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 4 covers advancements in AI technology ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 3 covers AI's impact on employees and their roles ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 2 covers the challenges presented by AI, as well as solutions to those problems ...

In the final part of APMdigest's 2025 Predictions Series, industry experts offer predictions on how AI will evolve and impact technology and business in 2025 ...

E-commerce is set to skyrocket with a 9% rise over the next few years ... To thrive in this competitive environment, retailers must identify digital resilience as their top priority. In a world where savvy shoppers expect 24/7 access to online deals and experiences, any unexpected downtime to digital services can lead to significant financial losses, damage to brand reputation, abandoned carts with designer shoes, and additional issues ...

Efficiency is a highly-desirable objective in business ... We're seeing this scenario play out in enterprises around the world as they continue to struggle with infrastructures and remote work models with an eye toward operational efficiencies. In contrast to that goal, a recent Broadcom survey of global IT and network professionals found widespread adoption of these strategies is making the network more complex and hampering observability, leading to uptime, performance and security issues. Let's look more closely at these challenges ...

Image
Broadcom

The 2025 Catchpoint SRE Report dives into the forces transforming the SRE landscape, exploring both the challenges and opportunities ahead. Let's break down the key findings and what they mean for SRE professionals and the businesses relying on them ...

Image
Catchpoint

How to Create Programmatic Service Level Indicators and Service Level Objectives

Ishan Mukherjee
New Relic

Programmatically tracked service level indicators (SLIs) are foundational to every site reliability engineering practice. When engineering teams have programmatic SLIs in place, they lessen the need to manually track performance and incident data. They're also able to reduce manual toil because our DevOps teams define the capabilities and metrics that define their SLI data, which they collect automatically — hence "programmatic."

Programmatic SLIs have three key characteristics: they're current (they reflect the state of a system right now), they're automated (they're reported by instrumentation, not by humans), and they're useful (they're selected based on what a system's user cares about). In this post, I'll explain how site reliability engineers (SREs) can help their teams develop and create programmatic SLIs.

SLIs — Identifying Capabilities

An important part of creating programmatic SLIs is identifying the capability of the system or service for which you're creating the SLI. Here are a few definitions:

■ A system is a group of services and infrastructure components that exposes one or more capabilities to external customers (either end users or other internal teams).

■ A service is a runtime process (or a horizontally-scaled tier of processes) that makes up a portion of a system.

■ A capability is a particular aspect of functionality exposed by a service to its users, phrased in plain-language terms.

SLIs and SLOs — Indicators and Objectives

But first, we need some more definitions. An indicator is something you can measure about a system that acts as a proxy for the customer experience. An objective is a goal for a specific indicator that you're committed to achieving.

Configuring indicators and objectives is the easy part. The hard part is thinking through what measurable system behavior serves as a proxy for customer experience. When setting system-level SLIs, think about the key performance indicators (KPIs) for those systems, for example:

■ User-facing system KPIs most often include availability, latency, and throughput.

■ Storage system KPIs often emphasize latency, availability, and durability.

■ Big data systems, such as data processing pipelines, typically use KPIs such as throughput and end-to-end latency.

Your indicators and objectives should provide an accurate snapshot of the impact of your system on your customers.

A more precise description of the indicator and objective relationship is to say that SLIs are expressed in relation to service level objectives (SLOs). When you think about the availability of a system, for example, SLIs are the key measurements of the availability of the system while SLOs are the goals you set for how much availability you expect out of that system. And service level agreements (SLAs) explain the results of breaking the SLO commitments.

Create Programmatic SLIs

You should write your programmatic SLIs in collaboration with your product managers, engineering managers, and individual contributors who work on a system. To define your programmatic SLIs (and SLOs), apply these steps:

1. Identify the system and its services.

2. Identify the customer-facing capabilities of the system or services.

3. Articulate a plain-language definition of what it means for each capability to be available.

4. Define one or more SLIs for that definition.

5. Measure the system to get a baseline.

6. Define an SLO for each capability, and track how you perform against it.

7. Iterate and refine our system, and fine-tune the SLOs over time.

Example capabilities and definitions

Here are two example capabilities and definitions for an imaginary team that manages an imaginary dashboard service:

Capability: Dashboards overview.

Availability Definition: Customers are able to select the dashboard launcher, and see a list of all dashboards available to them.

Capability: Dashboards detail view.

Availability Definition: Customers can view a dashboard, and widgets render accurately and timely manner.

To express these availability definitions as programmatic SLIs (with SLOs to measure them), you'd state these service capabilities as:

■ Requests for the full list of available dashboards returns within 100 milliseconds 99.9% of the time.

■ Requests to open the dashboard launcher complete without error 99.9% of the time.

■ Requests for an individual dashboard return within 100 milliseconds 99.9% of the time.

■ Requests to open an individual dashboard complete without error 99.9% of the time.

After you've settled on your SLIs, they should be reasonably stable, but systems evolve, and you'll need to revisit them regularly. It's a good idea to revisit them quarterly, or whenever you make changes to your services, traffic volume, and upstream and downstream dependencies.

Ishan Mukherjee is SVP of Growth at New Relic

Hot Topics

The Latest

Cloud migration is a highly strategic decision that involves leadership sponsorship, business justifications for moving to the cloud, and a clear understanding of expected value. Lack of this alignment can be the reigning cause of cost and budget overruns and why almost half of the migration efforts underway today will fail in the next three years ...

One of the most misunderstood culprits of poor application performance is packet loss. Even minimal packet loss can cripple the throughput of a high-speed connection, making enterprise applications sluggish and frustrating for remote employee ... So, what's going wrong? And why does adding more bandwidth fail to fix the issue? ...

Image
Cloudbrink

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 5 covers the infrastructure and hardware supporting AI ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 4 covers advancements in AI technology ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 3 covers AI's impact on employees and their roles ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 2 covers the challenges presented by AI, as well as solutions to those problems ...

In the final part of APMdigest's 2025 Predictions Series, industry experts offer predictions on how AI will evolve and impact technology and business in 2025 ...

E-commerce is set to skyrocket with a 9% rise over the next few years ... To thrive in this competitive environment, retailers must identify digital resilience as their top priority. In a world where savvy shoppers expect 24/7 access to online deals and experiences, any unexpected downtime to digital services can lead to significant financial losses, damage to brand reputation, abandoned carts with designer shoes, and additional issues ...

Efficiency is a highly-desirable objective in business ... We're seeing this scenario play out in enterprises around the world as they continue to struggle with infrastructures and remote work models with an eye toward operational efficiencies. In contrast to that goal, a recent Broadcom survey of global IT and network professionals found widespread adoption of these strategies is making the network more complex and hampering observability, leading to uptime, performance and security issues. Let's look more closely at these challenges ...

Image
Broadcom

The 2025 Catchpoint SRE Report dives into the forces transforming the SRE landscape, exploring both the challenges and opportunities ahead. Let's break down the key findings and what they mean for SRE professionals and the businesses relying on them ...

Image
Catchpoint