Pepperdata announced that the Pepperdata product portfolio now provides autonomous optimization and observability for Spark applications running on Kubernetes.
Kubernetes is a key part of the modern hybrid, multi-cloud architecture in today’s enterprises. Spark is the #1 big data application running on Kubernetes, according to a recent survey of enterprise users. As big data applications move from Spark on legacy systems to Spark on Kubernetes, the performance of these applications can change dramatically.
Pepperdata offers full-stack observability for Spark on Kubernetes, allowing developers to manually tune their applications, while autonomously optimizing resources at run time. The combination of manual and autonomous tuning is necessary to deliver the best price and performance for these applications. Pepperdata uses machine learning across clusters, containers, pods, nodes, users and workflows to give you a complete understanding of your environment.
Pepperdata will automatically optimize Kubernetes resources while providing a correlated and granular understanding of the applications and infrastructure. Observability provides actionable information to debug and understand complex applications, and autonomous optimization ensures that the compute resources are used efficiently.
Features include:
- Autonomous optimization of resources and workloads on Amazon EKS, HPE Bluedata and Red Hat OpenShift.
- Application and infrastructure observability for Spark on EKS, Bluedata and OpenShift as well as YARN.
- A self-service dashboard so developers can manually tune using recommendations for speed or resource utilization.
- Detailed usage attribution for chargeback.
“Kubernetes is becoming increasingly important for a unified IT infrastructure, both in the cloud and the data center. Spark is the number one big data application moving to the cloud, but Spark applications tend to be quite inefficient. Optimization is key to successful implementations,” said Ash Munshi, CEO, Pepperdata. “We saw this early on with our customers, which is why we invested in the development of Spark on Kubernetes, together with Red Hat, Palantir and Google.”
The Latest
Gartner identified the top data and analytics (D&A) trends for 2025 that are driving the emergence of a wide range of challenges, including organizational and human issues ...
Traditional network monitoring, while valuable, often falls short in providing the context needed to truly understand network behavior. This is where observability shines. In this blog, we'll compare and contrast traditional network monitoring and observability — highlighting the benefits of this evolving approach ...
A recent Rocket Software and Foundry study found that just 28% of organizations fully leverage their mainframe data, a concerning statistic given its critical role in powering AI models, predictive analytics, and informed decision-making ...
What kind of ROI is your organization seeing on its technology investments? If your answer is "it's complicated," you're not alone. According to a recent study conducted by Apptio ... there is a disconnect between enterprise technology spending and organizations' ability to measure the results ...
In today’s data and AI driven world, enterprises across industries are utilizing AI to invent new business models, reimagine business and achieve efficiency in operations. However, enterprises may face challenges like flawed or biased AI decisions, sensitive data breaches and rising regulatory risks ...
In MEAN TIME TO INSIGHT Episode 12, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at EMA discusses purchasing new network observability solutions....
There's an image problem with mobile app security. While it's critical for highly regulated industries like financial services, it is often overlooked in others. This usually comes down to development priorities, which typically fall into three categories: user experience, app performance, and app security. When dealing with finite resources such as time, shifting priorities, and team skill sets, engineering teams often have to prioritize one over the others. Usually, security is the odd man out ...
IT outages, caused by poor-quality software updates, are no longer rare incidents but rather frequent occurrences, directly impacting over half of US consumers. According to the 2024 Software Failure Sentiment Report from Harness, many now equate these failures to critical public health crises ...
In just a few months, Google will again head to Washington DC and meet with the government for a two-week remedy trial to cement the fate of what happens to Chrome and its search business in the face of ongoing antitrust court case(s). Or, Google may proactively decide to make changes, putting the power in its hands to outline a suitable remedy. Regardless of the outcome, one thing is sure: there will be far more implications for AI than just a shift in Google's Search business ...
In today's fast-paced digital world, Application Performance Monitoring (APM) is crucial for maintaining the health of an organization's digital ecosystem. However, the complexities of modern IT environments, including distributed architectures, hybrid clouds, and dynamic workloads, present significant challenges ... This blog explores the challenges of implementing application performance monitoring (APM) and offers strategies for overcoming them ...