Skip to main content

The Overconfidence Effect in Cloud Cost Management is Real (and Expensive)

Kash Shaikh
Virtana

"If I had asked people what they wanted, they would have said faster horses." - Henry Ford

In other words, do not try to make horses go incrementally faster, create the automobile (like the Ford Model T). Digital transformation, including its underpinning cloud infrastructure, is meant to deliver innovations that leapfrog the status quo.

The 21st century version uses digital technology and automation to overcome human limitations. Example: Performing calculations in the blink of an eye that would take a person lifetimes to complete. We are often blind to the ways our human limitations can creep into various systems and processes — and this overconfidence in the status quo can be limiting.

According to the results of a recent survey of 350 IT leaders at global organizations, we found evidence of the overconfidence effect in cloud migration.

The overconfidence effect is a well-established cognitive bias where a person's subjective confidence in their judgment is greater than the objective accuracy of those judgments. In short, it is when we systematically overestimate our knowledge and ability to predict on a massive scale.

Here is what we found: 85% of cloud decision makers surveyed feel confident managing their cloud bills, but 82% of respondents have incurred unnecessary cloud costs. Clearly, there is no guarantee that confidence in your cloud cost management capabilities, even among tech-savvy leaders, means you can avoid needless spending.

So, what does this mean for companies in the race to digital transformation?

Obviously, the pandemic accelerated organizations' journey to the cloud to enable agile, on-demand, flexible access to resources, helping them align with a digital business's dynamic needs. We heard from many of our customers at the start of lockdown last year, saying they had to shift to a remote work environment, seemingly overnight, and this effort was heavily cloud-reliant. However, blindly forging ahead can backfire.

This latest survey reveals that in addition to the overconfidence effect, enterprises are facing additional challenges across the hybrid cloud thanks to disjointed point tools, silos, lack of visibility, unexpected costs, lack of programmatic optimization, and the role risk plays in cloud cost management.

Cloud Waste is a Massive Problem

One of the biggest challenges is how to manage workloads operating in the cloud without incurring unexpected and unnecessary costs, which can eat into budgets needed for other areas of transformation. Over the many years we have been working with customers to optimize their cloud infrastructures, we have found that up to a third of their cloud spend is waste.

Clearly this is a rampant problem that continues to plague enterprises. It is important to note that the 82% from the survey represents unnecessary costs that respondents are aware of. It is highly likely that companies are spending far more than they need to without even knowing it.

For example, 56% of respondents lack programmatic cloud cost management capabilities. This can mean either that teams are spending too much time managing cloud costs or that the cloud waste is allowed to fester. Either way, time and money are being spent unwisely. Not only that, it is likely that there is waste that is not being uncovered. And the more the overconfidence effect is at play within an organization, the less likely it is that this waste will be identified.

The lack of visibility across hybrid and multi-cloud environments also blurs the issue. 86% of respondents said they cannot get a global view of cloud costs within minutes, creating delays and potentially reducing agility. 71% of respondents agreed that limited visibility across the hybrid cloud environment hinders their ability to maximize value, creates inefficiencies, and wastes time.

Disjointed tools also present a challenge. 72% of respondents said they are fed up with piecing together disparate management tools to monitor and manage everything from infrastructure performance to migration readiness to cloud cost, and 62% report that they have to cobble together multiple tools, systems, and custom scripts to get a global view of cloud costs. Again, these are the very same respondents who said they are confident in their ability to manage cloud costs.

What Leads to Waste also Impedes Transformation

Lack of programmatic management, limited visibility, and disparate tools affect more than just cloud costs. IT leaders are also grappling with issues that could hamper a successful digital transformation. 68% of survey respondents stated that their teams operate in silos, and 70% said that limited collaboration hinders their ability to adapt quickly and improve business outcomes.

Additionally, 66% of respondents stated that it is hard to understand if they are delivering the service levels the business needs, and 65% agreed that when there is an issue, they are hard-pressed to identify the business impact.

Finally, 77% cited increased performance issues as one of the reasons that pressure on cloud teams is on the rise.

The bottom line is that if you do not know what is going on across your entire infrastructure, you do not know if you are adequately serving the needs of the business and you cannot deliver performance levels that the business demands, which means that you have not met some of the foundational needs of digital transformation.

Clearing the way to cost-effective transformation

Wherever you have "seams" in your cloud monitoring and management — whether that is a result of multiple clouds, siloed teams, disparate tools, or time lags, for example — you have blind spots where unnecessary costs and other problems can hide. Combine that with the overconfidence effect and those risks only go up. A single modular platform can eliminate those seams and create appropriate levels of confidence rooted in data to de-risk cloud migrations; deliver deep precision observability into workloads before, during, and after a move; and optimize and manage efficiently once workloads are in the cloud.

This is underscored by Archana Vankatarman, Associate Research Director of Cloud Data Management at IDC Europe who said, "The duct-taped point tools and silos can make cloud cost management complex. The belief that they are wasting at least 15% of their public cloud spending will drive enterprises to actively invest in cloud cost management to halve cloud waste."

The bottom line is that cloud management is always changing. If you feel confident about your cloud costs based on your gut, think again. Have the right tools to know before you go forward with any initiative. In the end, you will not be upset that you took the time to get it right — and save money.

Methodology: Arlington Research, commissioned by Virtana, surveyed 350 cloud decision makers in April 2021 at US- and UK-based organizations with 250+ employees to better understand multi-cloud deployment experiences.

Kash Shaikh is CEO and President of Virtana

Hot Topics

The Latest

From growing reliance on FinOps teams to the increasing attention on artificial intelligence (AI), and software licensing, the Flexera 2025 State of the Cloud Report digs into how organizations are improving cloud spend efficiency, while tackling the complexities of emerging technologies ...

Today, organizations are generating and processing more data than ever before. From training AI models to running complex analytics, massive datasets have become the backbone of innovation. However, as businesses embrace the cloud for its scalability and flexibility, a new challenge arises: managing the soaring costs of storing and processing this data ...

Despite the frustrations, every engineer we spoke with ultimately affirmed the value and power of OpenTelemetry. The "sucks" moments are often the flip side of its greatest strengths ... Part 2 of this blog covers the powerful advantages and breakthroughs — the "OTel Rocks" moments ...

OpenTelemetry (OTel) arrived with a grand promise: a unified, vendor-neutral standard for observability data (traces, metrics, logs) that would free engineers from vendor lock-in and provide deeper insights into complex systems ... No powerful technology comes without its challenges, and OpenTelemetry is no exception. The engineers we spoke with were frank about the friction points they've encountered ...

Enterprises are turning to AI-powered software platforms to make IT management more intelligent and ensure their systems and technology meet business needs for efficiency, lowers costs and innovation, according to new research from Information Services Group ...

The power of Kubernetes lies in its ability to orchestrate containerized applications with unparalleled efficiency. Yet, this power comes at a cost: the dynamic, distributed, and ephemeral nature of its architecture creates a monitoring challenge akin to tracking a constantly shifting, interconnected network of fleeting entities ... Due to the dynamic and complex nature of Kubernetes, monitoring poses a substantial challenge for DevOps and platform engineers. Here are the primary obstacles ...

The perception of IT has undergone a remarkable transformation in recent years. What was once viewed primarily as a cost center has transformed into a pivotal force driving business innovation and market leadership ... As someone who has witnessed and helped drive this evolution, it's become clear to me that the most successful organizations share a common thread: they've mastered the art of leveraging IT advancements to achieve measurable business outcomes ...

More than half (51%) of companies are already leveraging AI agents, according to the PagerDuty Agentic AI Survey. Agentic AI adoption is poised to accelerate faster than generative AI (GenAI) while reshaping automation and decision-making across industries ...

Image
Pagerduty

 

Real privacy protection thanks to technology and processes is often portrayed as too hard and too costly to implement. So the most common strategy is to do as little as possible just to conform to formal requirements of current and incoming regulations. This is a missed opportunity ...

The expanding use of AI is driving enterprise interest in data operations (DataOps) to orchestrate data integration and processing and improve data quality and validity, according to a new report from Information Services Group (ISG) ...

The Overconfidence Effect in Cloud Cost Management is Real (and Expensive)

Kash Shaikh
Virtana

"If I had asked people what they wanted, they would have said faster horses." - Henry Ford

In other words, do not try to make horses go incrementally faster, create the automobile (like the Ford Model T). Digital transformation, including its underpinning cloud infrastructure, is meant to deliver innovations that leapfrog the status quo.

The 21st century version uses digital technology and automation to overcome human limitations. Example: Performing calculations in the blink of an eye that would take a person lifetimes to complete. We are often blind to the ways our human limitations can creep into various systems and processes — and this overconfidence in the status quo can be limiting.

According to the results of a recent survey of 350 IT leaders at global organizations, we found evidence of the overconfidence effect in cloud migration.

The overconfidence effect is a well-established cognitive bias where a person's subjective confidence in their judgment is greater than the objective accuracy of those judgments. In short, it is when we systematically overestimate our knowledge and ability to predict on a massive scale.

Here is what we found: 85% of cloud decision makers surveyed feel confident managing their cloud bills, but 82% of respondents have incurred unnecessary cloud costs. Clearly, there is no guarantee that confidence in your cloud cost management capabilities, even among tech-savvy leaders, means you can avoid needless spending.

So, what does this mean for companies in the race to digital transformation?

Obviously, the pandemic accelerated organizations' journey to the cloud to enable agile, on-demand, flexible access to resources, helping them align with a digital business's dynamic needs. We heard from many of our customers at the start of lockdown last year, saying they had to shift to a remote work environment, seemingly overnight, and this effort was heavily cloud-reliant. However, blindly forging ahead can backfire.

This latest survey reveals that in addition to the overconfidence effect, enterprises are facing additional challenges across the hybrid cloud thanks to disjointed point tools, silos, lack of visibility, unexpected costs, lack of programmatic optimization, and the role risk plays in cloud cost management.

Cloud Waste is a Massive Problem

One of the biggest challenges is how to manage workloads operating in the cloud without incurring unexpected and unnecessary costs, which can eat into budgets needed for other areas of transformation. Over the many years we have been working with customers to optimize their cloud infrastructures, we have found that up to a third of their cloud spend is waste.

Clearly this is a rampant problem that continues to plague enterprises. It is important to note that the 82% from the survey represents unnecessary costs that respondents are aware of. It is highly likely that companies are spending far more than they need to without even knowing it.

For example, 56% of respondents lack programmatic cloud cost management capabilities. This can mean either that teams are spending too much time managing cloud costs or that the cloud waste is allowed to fester. Either way, time and money are being spent unwisely. Not only that, it is likely that there is waste that is not being uncovered. And the more the overconfidence effect is at play within an organization, the less likely it is that this waste will be identified.

The lack of visibility across hybrid and multi-cloud environments also blurs the issue. 86% of respondents said they cannot get a global view of cloud costs within minutes, creating delays and potentially reducing agility. 71% of respondents agreed that limited visibility across the hybrid cloud environment hinders their ability to maximize value, creates inefficiencies, and wastes time.

Disjointed tools also present a challenge. 72% of respondents said they are fed up with piecing together disparate management tools to monitor and manage everything from infrastructure performance to migration readiness to cloud cost, and 62% report that they have to cobble together multiple tools, systems, and custom scripts to get a global view of cloud costs. Again, these are the very same respondents who said they are confident in their ability to manage cloud costs.

What Leads to Waste also Impedes Transformation

Lack of programmatic management, limited visibility, and disparate tools affect more than just cloud costs. IT leaders are also grappling with issues that could hamper a successful digital transformation. 68% of survey respondents stated that their teams operate in silos, and 70% said that limited collaboration hinders their ability to adapt quickly and improve business outcomes.

Additionally, 66% of respondents stated that it is hard to understand if they are delivering the service levels the business needs, and 65% agreed that when there is an issue, they are hard-pressed to identify the business impact.

Finally, 77% cited increased performance issues as one of the reasons that pressure on cloud teams is on the rise.

The bottom line is that if you do not know what is going on across your entire infrastructure, you do not know if you are adequately serving the needs of the business and you cannot deliver performance levels that the business demands, which means that you have not met some of the foundational needs of digital transformation.

Clearing the way to cost-effective transformation

Wherever you have "seams" in your cloud monitoring and management — whether that is a result of multiple clouds, siloed teams, disparate tools, or time lags, for example — you have blind spots where unnecessary costs and other problems can hide. Combine that with the overconfidence effect and those risks only go up. A single modular platform can eliminate those seams and create appropriate levels of confidence rooted in data to de-risk cloud migrations; deliver deep precision observability into workloads before, during, and after a move; and optimize and manage efficiently once workloads are in the cloud.

This is underscored by Archana Vankatarman, Associate Research Director of Cloud Data Management at IDC Europe who said, "The duct-taped point tools and silos can make cloud cost management complex. The belief that they are wasting at least 15% of their public cloud spending will drive enterprises to actively invest in cloud cost management to halve cloud waste."

The bottom line is that cloud management is always changing. If you feel confident about your cloud costs based on your gut, think again. Have the right tools to know before you go forward with any initiative. In the end, you will not be upset that you took the time to get it right — and save money.

Methodology: Arlington Research, commissioned by Virtana, surveyed 350 cloud decision makers in April 2021 at US- and UK-based organizations with 250+ employees to better understand multi-cloud deployment experiences.

Kash Shaikh is CEO and President of Virtana

Hot Topics

The Latest

From growing reliance on FinOps teams to the increasing attention on artificial intelligence (AI), and software licensing, the Flexera 2025 State of the Cloud Report digs into how organizations are improving cloud spend efficiency, while tackling the complexities of emerging technologies ...

Today, organizations are generating and processing more data than ever before. From training AI models to running complex analytics, massive datasets have become the backbone of innovation. However, as businesses embrace the cloud for its scalability and flexibility, a new challenge arises: managing the soaring costs of storing and processing this data ...

Despite the frustrations, every engineer we spoke with ultimately affirmed the value and power of OpenTelemetry. The "sucks" moments are often the flip side of its greatest strengths ... Part 2 of this blog covers the powerful advantages and breakthroughs — the "OTel Rocks" moments ...

OpenTelemetry (OTel) arrived with a grand promise: a unified, vendor-neutral standard for observability data (traces, metrics, logs) that would free engineers from vendor lock-in and provide deeper insights into complex systems ... No powerful technology comes without its challenges, and OpenTelemetry is no exception. The engineers we spoke with were frank about the friction points they've encountered ...

Enterprises are turning to AI-powered software platforms to make IT management more intelligent and ensure their systems and technology meet business needs for efficiency, lowers costs and innovation, according to new research from Information Services Group ...

The power of Kubernetes lies in its ability to orchestrate containerized applications with unparalleled efficiency. Yet, this power comes at a cost: the dynamic, distributed, and ephemeral nature of its architecture creates a monitoring challenge akin to tracking a constantly shifting, interconnected network of fleeting entities ... Due to the dynamic and complex nature of Kubernetes, monitoring poses a substantial challenge for DevOps and platform engineers. Here are the primary obstacles ...

The perception of IT has undergone a remarkable transformation in recent years. What was once viewed primarily as a cost center has transformed into a pivotal force driving business innovation and market leadership ... As someone who has witnessed and helped drive this evolution, it's become clear to me that the most successful organizations share a common thread: they've mastered the art of leveraging IT advancements to achieve measurable business outcomes ...

More than half (51%) of companies are already leveraging AI agents, according to the PagerDuty Agentic AI Survey. Agentic AI adoption is poised to accelerate faster than generative AI (GenAI) while reshaping automation and decision-making across industries ...

Image
Pagerduty

 

Real privacy protection thanks to technology and processes is often portrayed as too hard and too costly to implement. So the most common strategy is to do as little as possible just to conform to formal requirements of current and incoming regulations. This is a missed opportunity ...

The expanding use of AI is driving enterprise interest in data operations (DataOps) to orchestrate data integration and processing and improve data quality and validity, according to a new report from Information Services Group (ISG) ...