Data Issues Take 2 Days to Identify and Fix
July 11, 2023
Share this

Companies experience a median of five to ten data incidents over a period of three months, according to the 2023 State of Data Quality report from Bigeye.

Respondents reported at least two "severe" data incidents in the last six months, which created damage to the business/bottom line and were visible at the C-level. And 70% reported at least two data incidents that diminished the productivity of their teams.


Source: Bigeye

They also said incidents take an average of 48 hours to troubleshoot. And even though data issues most commonly take approximately 1-2 days to identify and fix, the issues can cause problems that last as long as weeks or even months.

Organizations with more than five data incidents per month are going from incident to incident, with little ability to trust data or invest in larger data infrastructure projects. They are performing reactive over proactive data quality work.

The report also found that data engineers are the first line of defense in managing data issues, followed closely by software engineers. The role of data engineer has moved on par with software engineering. Like software engineers, data engineers are in charge of a product — the data product — that increasingly demands software-like levels of process, maintenance, and code review.

Survey respondents say it takes an estimated 37,500 man hours to build in-house data quality monitoring, equating to about one year of work for 20 engineers

Those who used third-party data monitoring solutions found about a 2x to 3x ROI over in-house solutions. They also noted that at full utilization, third-party data monitoring solved for two issues: fractured infrastructure and anomalous data. They further reported that third-party data monitoring solutions had better test libraries, and a broader perspective on data problems.

"Data quality issues are the biggest blockers preventing data teams from being successful," said Kyle Kirwan, Bigeye CEO and co-founder. "We've heard that around 250-500 hours are lost every quarter, just dealing with data pipeline issues."

Methodology: The report consisted of answers from 100 survey respondents. At least 63 came from mid-to-large cloud data warehouse customers (with a spend of more than $500k per annum) who have some form of data monitoring in place, whether third-party or built in-house.

Share this

The Latest

December 05, 2023

Industry experts offer thoughtful, insightful, and often controversial predictions on how APM, AIOps, Observability, OpenTelemetry and related technologies will evolve and impact business in 2024. Part 2 covers more on Observability ...

December 04, 2023

The Holiday Season means it is time for APMdigest's annual list of Application Performance Management (APM) predictions, covering IT performance topics. Industry experts — from analysts and consultants to the top vendors — offer thoughtful, insightful, and often controversial predictions on how APM, observability, AIOps and related technologies will evolve and impact business in 2024. Part 1 covers APM and Observability ...

November 30, 2023

To help you stay on top of the ever-evolving tech scene, Automox IT experts shake the proverbial magic eight ball and share their predictions about tech trends in the coming year. From M&A frenzies to sustainable tech and automation, these forecasts paint an exciting picture of the future ...

November 29, 2023
The past few years have presented numerous challenges for businesses: a pandemic, rising interest rates, supply chain disruptions, and geopolitical conflict that sent shockwaves across the global economy. But change may finally be on the horizon. According to a recent report by Endava ... a majority of executives confirmed they are feeling optimistic about the current business climate, and as a result, are forecasting larger IT budgets, increased technology funding and rollout, and prioritized innovation in the coming year ...
November 28, 2023

Incident management processes are not keeping pace with the demands of modern operations teams, failing to meet the needs of SREs as well as platform and ops teams. Results from the State of DevOps Automation and AI Survey, commissioned by Transposit, point to an incident management paradox. Despite nearly 60% of ITOps and DevOps professionals reporting they have a defined incident management process that's fully documented in one place and over 70% saying they have a level of automation that meets their needs, teams are unable to quickly resolve incidents ...

November 27, 2023

Today, in the world of enterprise technology, the challenges posed by legacy Virtual Desktop Infrastructure (VDI) systems have long been a source of concern for IT departments. In many instances, this promising solution has become an organizational burden, hindering progress, depleting resources, and taking a psychological and operational toll on employees ...

November 22, 2023

Within retail organizations across the world, IT teams will be bracing themselves for a hectic holiday season ... While this is an exciting opportunity for retailers to boost sales, it also intensifies severe risk. Any application performance slipup will cause consumers to turn their back on brands, possibly forever. Online shoppers will be completely unforgiving to any retailer who doesn't deliver a seamless digital experience ...

November 21, 2023

Black Friday is a time when consumers can cash in on some of the biggest deals retailers offer all year long ... Nearly two-thirds of consumers utilize a retailer's web and mobile app for holiday shopping, raising the stakes for competitors to provide the best online experience to retain customer loyalty. Perforce's 2023 Black Friday survey sheds light on consumers' expectations this time of year and how developers can properly prepare their applications for increased online traffic ...

November 20, 2023

This holiday shopping season, the stakes for online retailers couldn't be higher ... Even an hour or two of downtime for a digital storefront during this critical period can cost millions in lost revenue and has the potential to damage brand credibility. Savvy retailers are increasingly investing in observability to help ensure a seamless, omnichannel customer experience. Just ahead of the holiday season, New Relic released its State of Observability for Retail report, which offers insight and analysis on the adoption and business value of observability for the global retail/consumer industry ...

November 16, 2023

As organizations struggle to find and retain the talent they need to manage complex cloud implementations, many are leaning toward hybrid cloud as a solution ... While it's true that using the cloud is not a "one size fits all" proposition, it is clear that both large and small companies prefer a hybrid cloud model ...