Complexity and Scale of Kubernetes Highlight Need for Observability and Optimization
May 11, 2021
Share this

Kubernetes is rapidly becoming the standard for cloud and on-premises clusters, according to the 2021 Kubernetes & Big Data Report from Pepperdata based on a survey of 800 IT professionals.


However, Kubernetes is a complex technology, and companies are struggling to properly and effectively implement and manage these environments. The complexity of big data applications makes resource optimization a real challenge. Unsurprisingly, when IT doesn't have granular visibility into big data Kubernetes performance, optimized performance and spend are hard to achieve.

"Kubernetes is increasingly being adopted by our customers for big data applications. As a result, we see customers experiencing performance challenges," said Ash Munshi, CEO, Pepperdata. "This survey clearly indicates that these problems are universal and there is a need to better optimize these big data workloads."

The report states:"Kubernetes is extremely complicated. Manual monitoring cannot keep up, and proprietary solutions are unlikely to be up to the task. It’s always tempting, faced with a new tool like Kubernetes, to use a homegrown solution that already exists. But with Kubernetes, more custom solutions are required. General-purpose APM won’t cut it; companies need tools purpose built for big data workloads on Kubernetes."

The survey reveals a number of insights into how businesses are adopting Kubernetes for big data applications:

■ When asked what their goals were for adopting Kubernetes for big data workloads, 30% said to "improve resource utilization for reduced cloud costs." 23% want to enable their migration to the cloud; 18% said to shorten deployment cycles; 15% wanted to make their platforms and applications cloud-agnostic; and 14% said to containerize monolithic apps.

■ Porting hundreds or thousands of apps over to Kubernetes can be challenging, and the biggest hurdles for survey respondents included initial deployment, followed by migration, monitoring and alerting, complexity and increased cost, and reliability, in that order.

■ The kinds of applications and workloads respondents are running, in order of most to least, include Spark, 30%; Kafka, 25%; Presto 23%; AI/deep learning workloads using PyTorch or Tensorflow at 18%; and "other" at 5%.

■ Surprisingly, and despite how much the media writes about the move to public cloud, this survey found that 47% of respondents are using Kubernetes in private cloud environments. On-premises use made up 35%, and just 18% of respondents said they were using Kubernetes containers in public cloud environments.

■ 45% of Kubernetes workloads are in development and testing environments, as users move production workloads into a new resource management framework. 30% are doing proof-of-concept work.

■ 66% of respondents said 75–100% of their big data workloads will be on Kubernetes by the end of 2021.

■ IT operations was the clear leader — at 80% — in deploying Spark and other big data apps built on Kubernetes; Engineering followed with 11%; with business unit developers at just 9%.

Methodology: The survey was conducted in March 2021, among 800 participants from a range of industries, 72% of which worked at companies with between 500 and 5000 employees. 

Share this

The Latest

February 06, 2023

This year 2023, at a macro level we are moving from an inflation economy to a recession and uncertain economy and the general theme is certainly going to be "Doing More with Less" and "Customer Experience is the King." Let us examine what trends and technologies will play a lending hand in these circumstances ...

February 02, 2023

As organizations continue to adapt to a post-pandemic surge in cloud-based productivity, the 2023 State of the Network report from Viavi Solutions details how end-user awareness remains critical and explores the benefits — and challenges — of cloud and off-premises network modernization initiatives ...

February 01, 2023

In the network engineering world, many teams have yet to realize the immense benefit real-time collaboration tools can bring to a successful automation strategy. By integrating a collaboration platform into a network automation strategy — and taking advantage of being able to share responses, files, videos and even links to applications and device statuses — network teams can leverage these tools to manage, monitor and update their networks in real time, and improve the ways in which they manage their networks ...

January 31, 2023

A recent study revealed only an alarming 5% of IT decision makers who report having complete visibility into employee adoption and usage of company-issued applications, demonstrating they are often unknowingly careless when it comes to software investments that can ultimately be costly in terms of time and resources ...

January 30, 2023

Everyone has visibility into their multi-cloud networking environment, but only some are happy with what they see. Unfortunately, this continues a trend. According to EMA's latest research, most network teams have some end-to-end visibility across their multi-cloud networks. Still, only 23.6% are fully satisfied with their multi-cloud network monitoring and troubleshooting capabilities ...

January 26, 2023

As enterprises work to implement or improve their observability practices, tool sprawl is a very real phenomenon ... Tool sprawl can and does happen all across the organization. In this post, though, we'll focus specifically on how and why observability efforts often result in tool sprawl, some of the possible negative consequences of that sprawl, and we'll offer some advice on how to reduce or even avoid sprawl ...

January 25, 2023

As companies generate more data across their network footprints, they need network observability tools to help find meaning in that data for better decision-making and problem solving. It seems many companies believe that adding more tools leads to better and faster insights ... And yet, observability tools aren't meeting many companies' needs. In fact, adding more tools introduces new challenges ...

January 24, 2023

Driven by the need to create scalable, faster, and more agile systems, businesses are adopting cloud native approaches. But cloud native environments also come with an explosion of data and complexity that makes it harder for businesses to detect and remediate issues before everything comes to a screeching halt. Observability, if done right, can make it easier to mitigate these challenges and remediate incidents before they become major customer-impacting problems ...

January 23, 2023

The spiraling cost of energy is forcing public cloud providers to raise their prices significantly. A recent report by Canalys predicted that public cloud prices will jump by around 20% in the US and more than 30% in Europe in 2023. These steep price increases will test the conventional wisdom that moving to the cloud is a cheap computing alternative ...

January 19, 2023

Despite strong interest over the past decade, the actual investment in DX has been recent. While 100% of enterprises are now engaged with DX in some way, most (77%) have begun their DX journey within the past two years. And most are early stage, with a fourth (24%) at the discussion stage and half (49%) currently transforming. Only 27% say they have finished their DX efforts ...