2024 DataOps Predictions - Part 1
January 24, 2024
Share this

Industry experts offer predictions on how DataOps will evolve and impact IT and business in 2024.

Data observability becomes mandatory

Data observability will become mandatory as organizations seek to drive smarter automation and faster decision-making in 2024. As the volume of data has continued to double every two years, organizations are urgently seeking to ingest and analyze it faster and at a greater scale. However, the cost and risk of poor-quality data is greater than ever. In a recent survey, 57% of DevOps practitioners said the absence of data observability makes it difficult to drive automation in a compliant way. As a result, there will be an increased demand for solutions that provide data observability to enable organizations to rapidly and securely ingest high-quality and reliable data that is ready for analytics on demand. Increased data observability will enable users to understand not only the availability of data, but also the structure, distribution, relationships, and lineage of that data across all sources. This is essential to generating insights that users can trust, by ensuring its freshness, identifying anomalies, and eliminating duplicates that could lead to errors.
Bernd Greifeneder
CTO and Founder, Dynatrace

AI DRIVES Data Observability

In 2024 large enterprises will rapidly increase investment in AI and LLM technologies. This will create a greater need for data observability to validate that data feeding AI initiatives is accurate and complete. As a result, data observability vendors will be expected to expand support from predominantly cloud-native data environments to larger, more traditional enterprise data stacks and provide native solutions for LLM data pipeline monitoring and validation.
Kyle Kirwan
Co-Founder and CEO, Bigeye

AI DRIVES DATAOPS

2024 will see the impact of rapid AI/ML advances on DataOps — empowering developers and DevOps teams by making data analytics that much more accessible, accurate, and definitive. AI/ML and new data science techniques will enable new operating models, data analytics tools, and technologies that enable data-driven decisions. Specific benefits will manifest in everything from customer recommendation engines to insight-optimized business operations, including streamlined development practices and DevOps processes. In this way, DataOps will provide enterprises with decisive advantages over competitors who lack that data command and clarity.
Anil Inamdar
VP & Head of Data, Instaclustr, part of Spot by NetApp

AI DRIVES DATA INFRASTRUCTURE MODERNIZATION

The continuous and rapid adoption of AI will force organizations to modernize their data infrastructure in 2024. Enterprises are examining their data and it's pushing them to have a better handle on it so technologies like AI can be properly used. Organizations will double down on data management and data integrity to ensure third-party applications are seamlessly integrated. Data practitioners will look for solutions that continuously keep data clean to quickly act on workflows. Better data means better-trained models on less data, as well as a better ability to leverage that data in AI applications that incorporate retrieval.
Matt Wallace
Technical Advisor, Faction

AI WILL NOT REPLACE DATA ENGINEERS

Data engineering will evolve — and be highly valued — in an AI world. There's been a lot of chatter that the AI revolution will replace the role of data engineers. That's not the case, and in fact their data expertise will be more critical than ever — just in new and different ways. To keep up with the evolving landscape, data engineers will need to understand how generative AI adds value. The data pipelines built and managed by data engineers will be perhaps the first place to connect with large language models for organizations to unlock value. Data engineers will be the ones who understand how to consume a model and plug it into a data pipeline to automate the extraction of value. They will also be expected to oversee and understand the AI work.
Jeff Hollan
Director of Product Management, Snowflake

BREAKING DOWN DATA SILOS

The traditional silos between IT, legal, and business departments will crumble in 2024, giving way to a more collaborative approach. This cross-functional synergy will ensure that data, including unstructured data, is not just collected but strategically utilized to drive business value.
Rohit Choudhary
CEO, Acceldata

Data as Business Asset

In many senses, data is the new oil. It's a finite resource that needs to be mined and managed strategically, and its value is highly dependent on your ability to refine and manipulate it for specific applications. For this reason, we see 2024 as being a critical year in the transition of data from being 1s and 0s on a screen to an actual asset to be managed, tracked, and optimized within an enterprise.
Jackie McGuire
Senior Security Strategist, Cribl

Data management will evolve beyond mere data storage in 2024. Organizations will recognize the strategic value of weaving data, including unstructured data, into their business strategies. This shift will unlock a wealth of insights and redefine business decision-making.
Rohit Choudhary
CEO, Acceldata

In 2024, the technology landscape will witness a transformative shift as data evolves from being a valuable asset to the lifeblood of thriving enterprises. Organizations that overlook data quality, integrity, and lineage will be challenged to not only make informed decisions but also realize the full potential of generative AI, LLM and ML applications and use cases. As the year unfolds, I predict that organizations neglecting to craft robust data foundations and strategies will find it increasingly challenging to stay afloat in the swiftly evolving tech industry. Those who fail to adapt and prioritize data fundamentals will struggle to outpace their competitors and may even risk survival in this highly competitive environment.
Armon Petrossian
CEO and Co-Founder, Coalesce

Data as innovation asset

The rapid expansion of data will continue to be a dominant trend in 2024. However, the ability to efficiently gather, process, and utilize this data will become the critical factor limiting or accelerating innovation within organizations. The challenge will lie in developing methods to quickly and securely assimilate this growing data influx, converting it into actionable insights. Companies that can effectively manage this data deluge, turning it into a strategic asset for innovation, will gain a competitive edge in the increasingly data-driven business landscape.
David Boskovic
Founder and CEO, Flatfile

Data as a product

Until recently, only large companies had the expertise and resources needed to create reusable data assets that can be easily repurposed across different teams and applications. Thanks to advancements in the governance products required to build these assets, in 2024 more companies will be able to create reusable data products, greatly accelerating efficiency and data innovation. Multiple teams can benefit from having access to the same data to build a service or application. However, this data must be presented in a way that is secure, well-contextualized, and understandable for users who weren't involved in its production. As data moves farther away from its initial source, you have to do more checks, which becomes increasingly expensive. Starting the data governance process at the source is not only less expensive but is a better way to understand the data's source and how it's schematized. New data governance capabilities, pre-built into products such as cloud data warehouses, databases, and other data infrastructure services, can meet these needs. Developers no longer need to manually build the infrastructure to create and share reusable data products. As a result, reusable data products will no longer be restricted to companies with large, sophisticated data engineering teams. With more companies building reusable data products, in 2024, developers will increase the value of their data and spend more time building innovative data applications and services.
Andrew Sellers
Head of Technology Strategy, Confluent

Go to: 2024 DataOps Predictions - Part 2

Share this

The Latest

February 21, 2024

Generative AI will usher in advantages within various industries. However, the technology is still nascent, and according to the recent Dynatrace survey there are many challenges and risks that organizations need to overcome to use this technology effectively ...

February 20, 2024

In today's digital era, monitoring and observability are indispensable in software and application development. Their efficacy lies in empowering developers to swiftly identify and address issues, enhance performance, and deliver flawless user experiences. Achieving these objectives requires meticulous planning, strategic implementation, and consistent ongoing maintenance. In this blog, we're sharing our five best practices to fortify your approach to application performance monitoring (APM) and observability ...

February 16, 2024

In MEAN TIME TO INSIGHT Episode 3, Shamus McGillicuddy, VP of Research, Network Infrastructure and Operations, at Enterprise Management Associates (EMA) discusses network security with Chris Steffen, VP of Research Covering Information Security, Risk, and Compliance Management at EMA ...

February 15, 2024

In a time where we're constantly bombarded with new buzzwords and technological advancements, it can be challenging for businesses to determine what is real, what is useful, and what they truly need. Over the years, we've witnessed the rise and fall of various tech trends, such as the promises (and fears) of AI becoming sentient and replacing humans to the declaration that data is the new oil. At the end of the day, one fundamental question remains: How can companies navigate through the tech buzz and make informed decisions for their future? ...

February 14, 2024

We increasingly see companies using their observability data to support security use cases. It's not entirely surprising given the challenges that organizations have with legacy SIEMs. We wanted to dig into this evolving intersection of security and observability, so we surveyed 500 security professionals — 40% of whom were either CISOs or CSOs — for our inaugural State of Security Observability report ...

February 13, 2024

Cloud computing continues to soar, with little signs of slowing down ... But, as with any new program, companies are seeing substantial benefits in the cloud but are also navigating budgetary challenges. With an estimated 94% of companies using cloud services today, priorities for IT teams have shifted from purely adoption-based to deploying new strategies. As they explore new territories, it can be a struggle to exploit the full value of their spend and the cloud's transformative capabilities ...

February 12, 2024

What will the enterprise of the future look like? If we asked this question three years ago, I doubt most of us would have pictured today as we know it: a future where generative AI has become deeply integrated into business and even our daily lives ...

February 09, 2024

With a focus on GenAI, industry experts offer predictions on how AI will evolve and impact IT and business in 2024. Part 5, the final installment in this series, covers the advantages AI will deliver: Generative AI will become increasingly important for resolving complicated data integration challenges, essentially providing a natural-language intermediary between data endpoints ...

February 08, 2024

With a focus on GenAI, industry experts offer predictions on how AI will evolve and impact IT and business in 2024. Part 4 covers the challenges of AI: In the short term, the rapid development and adoption of AI tools and products leveraging AI services will lead to an increase in biased outputs ...

February 07, 2024

With a focus on GenAI, industry experts offer predictions on how AI will evolve and impact IT and business in 2024. Part 3 covers the technologies that will drive AI: The question on every leader's mind in 2023 was - how soon will I see the return on my AI investment? The answer may lie in quantum computing ...