Kubernetes is rapidly becoming the standard for cloud and on-premises clusters, according to the 2021 Kubernetes & Big Data Report from Pepperdata based on a survey of 800 IT professionals.
However, Kubernetes is a complex technology, and companies are struggling to properly and effectively implement and manage these environments. The complexity of big data applications makes resource optimization a real challenge. Unsurprisingly, when IT doesn't have granular visibility into big data Kubernetes performance, optimized performance and spend are hard to achieve.
"Kubernetes is increasingly being adopted by our customers for big data applications. As a result, we see customers experiencing performance challenges," said Ash Munshi, CEO, Pepperdata. "This survey clearly indicates that these problems are universal and there is a need to better optimize these big data workloads."
The report states:"Kubernetes is extremely complicated. Manual monitoring cannot keep up, and proprietary solutions are unlikely to be up to the task. It’s always tempting, faced with a new tool like Kubernetes, to use a homegrown solution that already exists. But with Kubernetes, more custom solutions are required. General-purpose APM won’t cut it; companies need tools purpose built for big data workloads on Kubernetes."
The survey reveals a number of insights into how businesses are adopting Kubernetes for big data applications:
■ When asked what their goals were for adopting Kubernetes for big data workloads, 30% said to "improve resource utilization for reduced cloud costs." 23% want to enable their migration to the cloud; 18% said to shorten deployment cycles; 15% wanted to make their platforms and applications cloud-agnostic; and 14% said to containerize monolithic apps.
■ Porting hundreds or thousands of apps over to Kubernetes can be challenging, and the biggest hurdles for survey respondents included initial deployment, followed by migration, monitoring and alerting, complexity and increased cost, and reliability, in that order.
■ The kinds of applications and workloads respondents are running, in order of most to least, include Spark, 30%; Kafka, 25%; Presto 23%; AI/deep learning workloads using PyTorch or Tensorflow at 18%; and "other" at 5%.
■ Surprisingly, and despite how much the media writes about the move to public cloud, this survey found that 47% of respondents are using Kubernetes in private cloud environments. On-premises use made up 35%, and just 18% of respondents said they were using Kubernetes containers in public cloud environments.
■ 45% of Kubernetes workloads are in development and testing environments, as users move production workloads into a new resource management framework. 30% are doing proof-of-concept work.
■ 66% of respondents said 75–100% of their big data workloads will be on Kubernetes by the end of 2021.
■ IT operations was the clear leader — at 80% — in deploying Spark and other big data apps built on Kubernetes; Engineering followed with 11%; with business unit developers at just 9%.
Methodology: The survey was conducted in March 2021, among 800 participants from a range of industries, 72% of which worked at companies with between 500 and 5000 employees.
The Latest
The use of hybrid multicloud models is forecasted to double over the next one to three years as IT decision makers are facing new pressures to modernize IT infrastructures because of drivers like AI, security, and sustainability, according to the Enterprise Cloud Index (ECI) report from Nutanix ...
Over the last 20 years Digital Employee Experience has become a necessity for companies committed to digital transformation and improving IT experiences. In fact, by 2025, more than 50% of IT organizations will use digital employee experience to prioritize and measure digital initiative success ...
While most companies are now deploying cloud-based technologies, the 2024 Secure Cloud Networking Field Report from Aviatrix found that there is a silent struggle to maximize value from those investments. Many of the challenges organizations have faced over the past several years have evolved, but continue today ...
In our latest research, Cisco's The App Attention Index 2023: Beware the Application Generation, 62% of consumers report their expectations for digital experiences are far higher than they were two years ago, and 64% state they are less forgiving of poor digital services than they were just 12 months ago ...
In MEAN TIME TO INSIGHT Episode 5, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at EMA discusses the network source of truth ...
A vast majority (89%) of organizations have rapidly expanded their technology in the past few years and three quarters (76%) say it's brought with it increased "chaos" that they have to manage, according to Situation Report 2024: Managing Technology Chaos from Software AG ...
In 2024 the number one challenge facing IT teams is a lack of skilled workers, and many are turning to automation as an answer, according to IT Trends: 2024 Industry Report ...
Organizations are continuing to embrace multicloud environments and cloud-native architectures to enable rapid transformation and deliver secure innovation. However, despite the speed, scale, and agility enabled by these modern cloud ecosystems, organizations are struggling to manage the explosion of data they create, according to The state of observability 2024: Overcoming complexity through AI-driven analytics and automation strategies, a report from Dynatrace ...
Organizations recognize the value of observability, but only 10% of them are actually practicing full observability of their applications and infrastructure. This is among the key findings from the recently completed Logz.io 2024 Observability Pulse Survey and Report ...
Businesses must adopt a comprehensive Internet Performance Monitoring (IPM) strategy, says Enterprise Management Associates (EMA), a leading IT analyst research firm. This strategy is crucial to bridge the significant observability gap within today's complex IT infrastructures. The recommendation is particularly timely, given that 99% of enterprises are expanding their use of the Internet as a primary connectivity conduit while facing challenges due to the inefficiency of multiple, disjointed monitoring tools, according to Modern Enterprises Must Boost Observability with Internet Performance Monitoring, a new report from EMA and Catchpoint ...
