Complexity and Scale of Kubernetes Highlight Need for Observability and Optimization
May 11, 2021
Share this

Kubernetes is rapidly becoming the standard for cloud and on-premises clusters, according to the 2021 Kubernetes & Big Data Report from Pepperdata based on a survey of 800 IT professionals.


However, Kubernetes is a complex technology, and companies are struggling to properly and effectively implement and manage these environments. The complexity of big data applications makes resource optimization a real challenge. Unsurprisingly, when IT doesn't have granular visibility into big data Kubernetes performance, optimized performance and spend are hard to achieve.

"Kubernetes is increasingly being adopted by our customers for big data applications. As a result, we see customers experiencing performance challenges," said Ash Munshi, CEO, Pepperdata. "This survey clearly indicates that these problems are universal and there is a need to better optimize these big data workloads."

The report states:"Kubernetes is extremely complicated. Manual monitoring cannot keep up, and proprietary solutions are unlikely to be up to the task. It’s always tempting, faced with a new tool like Kubernetes, to use a homegrown solution that already exists. But with Kubernetes, more custom solutions are required. General-purpose APM won’t cut it; companies need tools purpose built for big data workloads on Kubernetes."

The survey reveals a number of insights into how businesses are adopting Kubernetes for big data applications:

■ When asked what their goals were for adopting Kubernetes for big data workloads, 30% said to "improve resource utilization for reduced cloud costs." 23% want to enable their migration to the cloud; 18% said to shorten deployment cycles; 15% wanted to make their platforms and applications cloud-agnostic; and 14% said to containerize monolithic apps.

■ Porting hundreds or thousands of apps over to Kubernetes can be challenging, and the biggest hurdles for survey respondents included initial deployment, followed by migration, monitoring and alerting, complexity and increased cost, and reliability, in that order.

■ The kinds of applications and workloads respondents are running, in order of most to least, include Spark, 30%; Kafka, 25%; Presto 23%; AI/deep learning workloads using PyTorch or Tensorflow at 18%; and "other" at 5%.

■ Surprisingly, and despite how much the media writes about the move to public cloud, this survey found that 47% of respondents are using Kubernetes in private cloud environments. On-premises use made up 35%, and just 18% of respondents said they were using Kubernetes containers in public cloud environments.

■ 45% of Kubernetes workloads are in development and testing environments, as users move production workloads into a new resource management framework. 30% are doing proof-of-concept work.

■ 66% of respondents said 75–100% of their big data workloads will be on Kubernetes by the end of 2021.

■ IT operations was the clear leader — at 80% — in deploying Spark and other big data apps built on Kubernetes; Engineering followed with 11%; with business unit developers at just 9%.

Methodology: The survey was conducted in March 2021, among 800 participants from a range of industries, 72% of which worked at companies with between 500 and 5000 employees. 

Share this

The Latest

June 20, 2024

The total cost of downtime for Global 2000 companies is $400 billion annually — or 9% of profits — when digital environments fail unexpectedly, according to The Hidden Costs of Downtime, a new report from Splunk ...

June 18, 2024

With the rise of digital transformation and the increasing reliance on applications for business operations, the need for application performance management (APM) has become more critical ... This blog explains what APM is all about, its significance and key features ...

June 17, 2024

Generative AI (GenAI) has captured significant attention by redefining content creation and automation processes. Despite this surge in GenAI's popularity, it's crucial to highlight the continuous, vital role of machine learning (ML) in underpinning crucial business functions. This era is not about GenAI replacing ML; rather, it's about these technologies collaborating to supercharge intelligent automation across industries ...

June 13, 2024

As organizations continue to navigate their digital transformation journeys, the need for efficient, secure, and scalable data movement strategies has never been more critical ... In an era when enterprise IT landscapes are continually evolving, the strategic movement of data has become a cornerstone of maintaining agility, competitive edge, and operational efficiency ...

June 12, 2024

In May, New Relic published the State of Observability for IT and Telecommunications Report to share insights, statistics, and analysis on the adoption and business value of observability for the IT and telecommunications industries. Here are five key takeaways from the report ...

June 11, 2024
Over the past decade, the pace of technological progress has reached unprecedented levels, where fads both quickly rise and shrink in popularity. From AI and composability to augmented reality and quantum computing, the toolkit of emerging technologies is continuing to expand, creating a complex set of opportunities and challenges for businesses to address. In order to keep pace with competitors, avoiding new models and ideas is not an option. It's critical for organizations to determine whether an idea has transformative properties or is just a flash in the pan — a challenge tackled in Endava's new 2024 Emerging Tech Unpacked Report ...
June 10, 2024

The rapidly evolving nature of the industry, particularly with the recent surge in generative AI, can catch firms off-guard, leaving them scrambling to adapt to new trends without the necessary funds ... This blog will discuss effective strategies for optimizing cloud expenses to free up funds for emerging AI technologies, ensuring companies can adapt and thrive without financial strain ...

June 06, 2024

Software developers are spending more than 57% of their time being dragged into "war rooms" to solve application performance issues, rather than investing their time developing new, cutting-edge software applications as part of their organization's innovation strategy, according to a new report from Cisco ...

June 05, 2024

Generative Artificial Intelligence (GenAI) is continuing to see massive adoption and expanding use cases, despite some ongoing concerns related to bias and performance. This is clear from the results of Applause's 2024 GenAI Survey, which examined how digital quality professionals use and experience GenAI technology ... Here's what we found ...

June 04, 2024

Many times customers want to know why their measured performance doesn't match the speed advertised (by the platform vendor, software vendor, network vendor, etc). Assuming the advertised speeds are (a) within the realm of physical possibility and obeys the laws of physics, and (b) are real achievable speeds and not "click-bait," there are at least ten reasons for being unable to achieve advertised speeds. In situations where customer expectations and measured performance don't align, use the following checklist to help determine the reason(s) why ...