Complexity and Scale of Kubernetes Highlight Need for Observability and Optimization
May 11, 2021
Share this

Kubernetes is rapidly becoming the standard for cloud and on-premises clusters, according to the 2021 Kubernetes & Big Data Report from Pepperdata based on a survey of 800 IT professionals.


However, Kubernetes is a complex technology, and companies are struggling to properly and effectively implement and manage these environments. The complexity of big data applications makes resource optimization a real challenge. Unsurprisingly, when IT doesn't have granular visibility into big data Kubernetes performance, optimized performance and spend are hard to achieve.

"Kubernetes is increasingly being adopted by our customers for big data applications. As a result, we see customers experiencing performance challenges," said Ash Munshi, CEO, Pepperdata. "This survey clearly indicates that these problems are universal and there is a need to better optimize these big data workloads."

The report states:"Kubernetes is extremely complicated. Manual monitoring cannot keep up, and proprietary solutions are unlikely to be up to the task. It’s always tempting, faced with a new tool like Kubernetes, to use a homegrown solution that already exists. But with Kubernetes, more custom solutions are required. General-purpose APM won’t cut it; companies need tools purpose built for big data workloads on Kubernetes."

The survey reveals a number of insights into how businesses are adopting Kubernetes for big data applications:

■ When asked what their goals were for adopting Kubernetes for big data workloads, 30% said to "improve resource utilization for reduced cloud costs." 23% want to enable their migration to the cloud; 18% said to shorten deployment cycles; 15% wanted to make their platforms and applications cloud-agnostic; and 14% said to containerize monolithic apps.

■ Porting hundreds or thousands of apps over to Kubernetes can be challenging, and the biggest hurdles for survey respondents included initial deployment, followed by migration, monitoring and alerting, complexity and increased cost, and reliability, in that order.

■ The kinds of applications and workloads respondents are running, in order of most to least, include Spark, 30%; Kafka, 25%; Presto 23%; AI/deep learning workloads using PyTorch or Tensorflow at 18%; and "other" at 5%.

■ Surprisingly, and despite how much the media writes about the move to public cloud, this survey found that 47% of respondents are using Kubernetes in private cloud environments. On-premises use made up 35%, and just 18% of respondents said they were using Kubernetes containers in public cloud environments.

■ 45% of Kubernetes workloads are in development and testing environments, as users move production workloads into a new resource management framework. 30% are doing proof-of-concept work.

■ 66% of respondents said 75–100% of their big data workloads will be on Kubernetes by the end of 2021.

■ IT operations was the clear leader — at 80% — in deploying Spark and other big data apps built on Kubernetes; Engineering followed with 11%; with business unit developers at just 9%.

Methodology: The survey was conducted in March 2021, among 800 participants from a range of industries, 72% of which worked at companies with between 500 and 5000 employees. 

Share this

The Latest

June 01, 2023

The journey of maturing observability practices for users entails navigating peaks and valleys. Users have clearly witnessed the maturation of their monitoring capabilities, embraced DevOps practices, and adopted cloud and cloud-native technologies. Notwithstanding that, we witness the gradual increase of the Mean Time To Recovery (MTTR) for production issues year over year ...

May 31, 2023

Optimizing existing use of cloud is the top initiative — for the seventh year in a row, reported by 62% of respondents in the Flexera 2023 State of the Cloud Report ...

May 30, 2023

Gartner highlighted four trends impacting cloud, data center and edge infrastructure in 2023, as infrastructure and operations teams pivot to support new technologies and ways of working during a year of economic uncertainty ...

May 25, 2023

Developers need a tool that can be portable and vendor agnostic, given the advent of microservices. It may be clear an issue is occurring; what may not be clear is if it's part of a distributed system or the app itself. Enter OpenTelemetry, commonly referred to as OTel, an open-source framework that provides a standardized way of collecting and exporting telemetry data (logs, metrics, and traces) from cloud-native software ...

May 24, 2023

As SLOs grow in popularity their usage is becoming more mature. For example, 82% of respondents intend to increase their use of SLOs, and 96% have mapped SLOs directly to their business operations or already have a plan to, according to The State of Service Level Objectives 2023 from Nobl9 ...

May 23, 2023

Observability has matured beyond its early adopter position and is now foundational for modern enterprises to achieve full visibility into today's complex technology environments, according to The State of Observability 2023, a report released by Splunk in collaboration with Enterprise Strategy Group ...

May 22, 2023

Before network engineers even begin the automation process, they tend to start with preconceived notions that oftentimes, if acted upon, can hinder the process. To prevent that from happening, it's important to identify and dispel a few common misconceptions currently out there and how networking teams can overcome them. So, let's address the three most common network automation myths ...

May 18, 2023

Many IT organizations apply AI/ML and AIOps technology across domains, correlating insights from the various layers of IT infrastructure and operations. However, Enterprise Management Associates (EMA) has observed significant interest in applying these AI technologies narrowly to network management, according to a new research report, titled AI-Driven Networks: Leveling Up Network Management with AI/ML and AIOps ...

May 17, 2023

When it comes to system outages, AIOps solutions with the right foundation can help reduce the blame game so the right teams can spend valuable time restoring the impacted services rather than improving their MTTI score (mean time to innocence). In fact, much of today's innovation around ChatGPT-style algorithms can be used to significantly improve the triage process and user experience ...

May 16, 2023

Gartner identified the top 10 data and analytics (D&A) trends for 2023 that can guide D&A leaders to create new sources of value by anticipating change and transforming extreme uncertainty into new business opportunities ...