How to Create Programmatic Service Level Indicators and Service Level Objectives
May 23, 2022

Ishan Mukherjee
New Relic

Share this

Programmatically tracked service level indicators (SLIs) are foundational to every site reliability engineering practice. When engineering teams have programmatic SLIs in place, they lessen the need to manually track performance and incident data. They're also able to reduce manual toil because our DevOps teams define the capabilities and metrics that define their SLI data, which they collect automatically — hence "programmatic."

Programmatic SLIs have three key characteristics: they're current (they reflect the state of a system right now), they're automated (they're reported by instrumentation, not by humans), and they're useful (they're selected based on what a system's user cares about). In this post, I'll explain how site reliability engineers (SREs) can help their teams develop and create programmatic SLIs.

SLIs — Identifying Capabilities

An important part of creating programmatic SLIs is identifying the capability of the system or service for which you're creating the SLI. Here are a few definitions:

■ A system is a group of services and infrastructure components that exposes one or more capabilities to external customers (either end users or other internal teams).

■ A service is a runtime process (or a horizontally-scaled tier of processes) that makes up a portion of a system.

■ A capability is a particular aspect of functionality exposed by a service to its users, phrased in plain-language terms.

SLIs and SLOs — Indicators and Objectives

But first, we need some more definitions. An indicator is something you can measure about a system that acts as a proxy for the customer experience. An objective is a goal for a specific indicator that you're committed to achieving.

Configuring indicators and objectives is the easy part. The hard part is thinking through what measurable system behavior serves as a proxy for customer experience. When setting system-level SLIs, think about the key performance indicators (KPIs) for those systems, for example:

■ User-facing system KPIs most often include availability, latency, and throughput.

■ Storage system KPIs often emphasize latency, availability, and durability.

■ Big data systems, such as data processing pipelines, typically use KPIs such as throughput and end-to-end latency.

Your indicators and objectives should provide an accurate snapshot of the impact of your system on your customers.

A more precise description of the indicator and objective relationship is to say that SLIs are expressed in relation to service level objectives (SLOs). When you think about the availability of a system, for example, SLIs are the key measurements of the availability of the system while SLOs are the goals you set for how much availability you expect out of that system. And service level agreements (SLAs) explain the results of breaking the SLO commitments.

Create Programmatic SLIs

You should write your programmatic SLIs in collaboration with your product managers, engineering managers, and individual contributors who work on a system. To define your programmatic SLIs (and SLOs), apply these steps:

1. Identify the system and its services.

2. Identify the customer-facing capabilities of the system or services.

3. Articulate a plain-language definition of what it means for each capability to be available.

4. Define one or more SLIs for that definition.

5. Measure the system to get a baseline.

6. Define an SLO for each capability, and track how you perform against it.

7. Iterate and refine our system, and fine-tune the SLOs over time.

Example capabilities and definitions

Here are two example capabilities and definitions for an imaginary team that manages an imaginary dashboard service:

Capability: Dashboards overview.

Availability Definition: Customers are able to select the dashboard launcher, and see a list of all dashboards available to them.

Capability: Dashboards detail view.

Availability Definition: Customers can view a dashboard, and widgets render accurately and timely manner.

To express these availability definitions as programmatic SLIs (with SLOs to measure them), you'd state these service capabilities as:

■ Requests for the full list of available dashboards returns within 100 milliseconds 99.9% of the time.

■ Requests to open the dashboard launcher complete without error 99.9% of the time.

■ Requests for an individual dashboard return within 100 milliseconds 99.9% of the time.

■ Requests to open an individual dashboard complete without error 99.9% of the time.

After you've settled on your SLIs, they should be reasonably stable, but systems evolve, and you'll need to revisit them regularly. It's a good idea to revisit them quarterly, or whenever you make changes to your services, traffic volume, and upstream and downstream dependencies.

Ishan Mukherjee is SVP of Growth at New Relic
Share this

The Latest

March 18, 2024

Gartner has highlighted the top trends that will impact technology providers in 2024: Generative AI (GenAI) is dominating the technical and product agenda of nearly every tech provider ...

March 15, 2024

In MEAN TIME TO INSIGHT Episode 4 - Part 1, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses artificial intelligence and network management ...

March 14, 2024

The integration and maintenance of AI-enabled Software as a Service (SaaS) applications have emerged as pivotal points in enterprise AI implementation strategies, offering both significant challenges and promising benefits. Despite the enthusiasm surrounding AI's potential impact, the reality of its implementation presents hurdles. Currently, over 90% of enterprises are grappling with limitations in integrating AI into their tech stack ...

March 13, 2024

In the intricate landscape of IT infrastructure, one critical component often relegated to the back burner is Active Directory (AD) forest recovery — an oversight with costly consequences ...

March 12, 2024

eBPF is a technology that allows users to run custom programs inside the Linux kernel, which changes the behavior of the kernel and makes execution up to 10x faster(link is external) and more efficient for key parts of what makes our computing lives work. That includes observability, networking and security ...

March 11, 2024

Data mesh, an increasingly important decentralized approach to data architecture and organizational design, focuses on treating data as a product, emphasizing domain-oriented data ownership, self-service tools and federated governance. The 2024 State of the Data Lakehouse report from Dremio presents evidence of the growing adoption of data mesh architectures in enterprises ... The report highlights that the drive towards data mesh is increasingly becoming a business strategy to enhance agility and speed in problem-solving and innovation ...

March 07, 2024
In this digital era, consumers prefer a seamless user experience, and here, the significance of performance testing cannot be overstated. Application performance testing is essential in ensuring that your software products, websites, or other related systems operate seamlessly under varying conditions. However, the cost of poor performance extends beyond technical glitches and slow load times; it can directly affect customer satisfaction and brand reputation. Understand the tangible and intangible consequences of poor application performance and how it can affect your business ...
March 06, 2024

Too much traffic can crash a website ... That stampede of traffic is even more horrifying when it's part of a malicious denial of service attack ... These attacks are becoming more common, more sophisticated and increasingly tied to ransomware-style demands. So it's no wonder that the threat of DDoS remains one of the many things that keep IT and marketing leaders up at night ...

March 05, 2024

Today, applications serve as the backbone of businesses, and therefore, ensuring optimal performance has never been more critical. This is where application performance monitoring (APM) emerges as an indispensable tool, empowering organizations to safeguard their applications proactively, match user expectations, and drive growth. But APM is not without its challenges. Choosing to implement APM is a path that's not easily realized, even if it offers great benefits. This blog deals with the potential hurdles that may manifest when you actualize your APM strategy in your IT application environment ...

March 04, 2024

This year's Super Bowl drew in viewership of nearly 124 million viewers and made history as the most-watched live broadcast event since the 1969 moon landing. To support this spike in viewership, streaming companies like YouTube TV, Hulu and Paramount+ began preparing their IT infrastructure months in advance to ensure an exceptional viewer experience without outages or major interruptions. New Relic conducted a survey to understand the importance of a seamless viewing experience and the impact of outages during major streaming events such as the Super Bowl ...