A colleague of mine recently embarked on a journey to explore the capabilities of a well-known legacy observability platform within his Kubernetes environment. He dedicated a week to familiarize himself with the platform, primarily testing out the different features for traces, logs, and infrastructure monitoring. However, his focus shifted when a critical feature needed an early release, diverting his attention away from the observability tool. Unfortunately, without any prior notification or warning, there was no rate limitation to the platform logs collection mechanism. One line of YAML configuration file meant all logs were collected, ingested and stored — with no mention of the projected cost.
Fast forward to the following week, a member of the billing department barged into his office, demanding an explanation for an astronomical observability bill totaling $33,000 for a single month, a staggering contrast to the anticipated $1,700.
This series of events left my work buddy struggling with the size of his mistake, and me questioning whether it really was entirely his fault.
The Complex Landscape of Observability Pricing
Navigating observability pricing models can be compared to solving a perplexing puzzle which includes financial variables and contractual intricacies. Predicting all potential costs in advance becomes an elusive endeavor, exemplified by a recent eye-popping $65 million observability bill.
Avoiding miscalculations as the one that happened to my friend requires continuous monitoring of the monitoring solution. This practice slows down day-to-day operations and long-term growth efforts.
The Challenge of Affordability in Observability
The escalating costs associated with observability represent a vast challenge which is confronting many organizations currently. Particularly in the age of cloud computing, IT leaders and even top executives have come to realize the imperative of reining in their infrastructure budgets, which often spiral out of control.
The proliferation of microservices and distributed architectures has ushered in a flood of data that demands observability. Traditionally, more data translates into higher expenses, accompanied by substantial resource consumption, leading not only to increased costs but also inefficiencies.
Regrettably, most observability tools employ pricing models that defy prediction. While applications generate large amounts of log data, instead of an advantage, this abundance has become a cause for concern. In response, best practices now advocate monitoring "only what you need" or limiting the retention period for collected data to a minimum. This raises two questions: how can you know in advance what you will need, and will limiting the retention period to a minimum make it impossible to correlate with out-of-range historical data.
Enter eBPF: A Game-Changer
eBPF (extended Berkeley Packet Filter) has recently emerged as a revolutionary technology that has significantly impacted the Linux kernell. eBPF operates at specific hook points within the kernel, extracting data with minimal overhead, safeguarding the application's resources from excessive consumption. It observes every packet entering or exiting the host, mapping them to processes or containers running on the host, thereby offering granular insights into network traffic.
Moreover, eBPF-powered agents operate independently of the primary application being monitored, ensuring minimal impact on microservice resources.
The combination of visibility depth and stability has made eBPF a groundbreaking technology for cybersecurity companies, and is predicted to have the same effect on observability, for exactly the same reasons.
Hassle-Free Observability
Observability should empower engineers, not bury them in a load of unexpected overheads, data volume surges, and huge subscription bills. The goal of observability platforms should be to guarantee complete protection against such surprises, offering immunity against sudden spikes in data volume and shielding engineers from unfortunate encounters with the billing department.
In conclusion, the journey to achieving efficient and cost-effective observability is full of challenges, but with the right tools and strategies, IT and DevOps leaders can help their organizations emerge from financial uncertainty and empower their engineers to become true observability heroes.
The Latest
Incident management processes are not keeping pace with the demands of modern operations teams, failing to meet the needs of SREs as well as platform and ops teams. Results from the State of DevOps Automation and AI Survey, commissioned by Transposit, point to an incident management paradox. Despite nearly 60% of ITOps and DevOps professionals reporting they have a defined incident management process that's fully documented in one place and over 70% saying they have a level of automation that meets their needs, teams are unable to quickly resolve incidents ...
Today, in the world of enterprise technology, the challenges posed by legacy Virtual Desktop Infrastructure (VDI) systems have long been a source of concern for IT departments. In many instances, this promising solution has become an organizational burden, hindering progress, depleting resources, and taking a psychological and operational toll on employees ...
Within retail organizations across the world, IT teams will be bracing themselves for a hectic holiday season ... While this is an exciting opportunity for retailers to boost sales, it also intensifies severe risk. Any application performance slipup will cause consumers to turn their back on brands, possibly forever. Online shoppers will be completely unforgiving to any retailer who doesn't deliver a seamless digital experience ...
Black Friday is a time when consumers can cash in on some of the biggest deals retailers offer all year long ... Nearly two-thirds of consumers utilize a retailer's web and mobile app for holiday shopping, raising the stakes for competitors to provide the best online experience to retain customer loyalty. Perforce's 2023 Black Friday survey sheds light on consumers' expectations this time of year and how developers can properly prepare their applications for increased online traffic ...
This holiday shopping season, the stakes for online retailers couldn't be higher ... Even an hour or two of downtime for a digital storefront during this critical period can cost millions in lost revenue and has the potential to damage brand credibility. Savvy retailers are increasingly investing in observability to help ensure a seamless, omnichannel customer experience. Just ahead of the holiday season, New Relic released its State of Observability for Retail report, which offers insight and analysis on the adoption and business value of observability for the global retail/consumer industry ...
As organizations struggle to find and retain the talent they need to manage complex cloud implementations, many are leaning toward hybrid cloud as a solution ... While it's true that using the cloud is not a "one size fits all" proposition, it is clear that both large and small companies prefer a hybrid cloud model ...
In the same way a city is a sum of its districts and neighborhoods, complex IT systems are made of many components that continually interact. Observability requires a comprehensive and connected view of all aspects of the system, including even some that don't directly relate to its technological innards ...
Multicasting in this context refers to the process of directing data streams to two or more destinations. This might look like sending the same telemetry data to both an on-premises storage system and a cloud-based observability platform concurrently. The two principal benefits of this strategy are cost savings and service redundancy ...
In today's rapidly evolving business environment, Chief Information Officers (CIOs) and Chief Technology Officers (CTOs) are grappling with the challenge of regaining control over their IT roadmap. The constant evolution and introduction of new technology releases, combined with the pressure to deliver innovation on shrinking budgets, has added layers of complexity for executives who must transform the perception of the role of the IT leader from cost managers and maintainers to strategic enablers of growth and profitability ...