Dataflow Complexity is the New Normal
April 07, 2017

Pat Patterson
StreamSets

Share this

The process of wrangling big data is fraught with pitfalls for enterprises. Data-driven enterprises are buckling under the burden of gathering, analyzing and making actionable an incredible and growing amount of data flowing in from a variety of sources. It's not just the amount of big data that is confounding data-driven companies: The speed at which data must be collected and analyzed, and the variety of data types (think: IoT sensors, log files, web clickstreams) are overwhelming enterprise data architectures, which are increasingly defined by a complex tangle of big data sources and processing systems. Topping all this is the problem of data drift, the unexpected changes that consistently plague big data sources and result in corrupt and unusable data.

In short, the complexity of data in motion is growing and risks undermining the success of the modern data-driven enterprise. A recent survey of data engineers and architects, conducted by StreamSets, sought to bring some perspective to the new reality in the enterprise, leading to some interesting insights about the enterprise data landscape.

As we expected, use of streaming data has become quite common, with a high number of respondents — 72 percent — collecting this data for a variety of uses. Of these, two-thirds (48 percent) collect a combination of batch and streaming data, since real-time data requires context to provide intelligence. In contrast, 28 percent move batch data only, and 24 percent ingest streaming data only.

Survey results also showed that enterprises are gathering data from a range of sources: 61 percent collect from transactional databases, 53 percent from log files, 42 percent from analytics databases, 27 percent from clickstream data and 18 percent from IoT devices.

Moving on from their use of streaming data, the survey reveals that enterprises are also experiencing a sense of data urgency — that is, expeditious analysis of their incoming data sets. In fact, according to the survey, 56 percent of respondents say they require data analysis within minutes of receiving the data, and 16 percent require analysis within seconds. The world has certainly evolved from the daily or weekly business intelligence report to a live dashboard, or even analysis that drives automated actions like website personalization that can have a direct impact on a business' effectiveness in engaging with its customers. These requirements put extreme pressure on enterprise data architectures not necessarily designed to deliver consumption-ready data with this type of speed.

Our survey responses indicate that enterprises funnel their data into a range of destinations, making them much more complicated and expensive to manage than ever before. In addition, respondents keep some of their data on premises (58 percent), some in private clouds (48 percent) and some in public clouds (27 percent). The combination of diverse data stores and multiple deployment models is a new phenomenon we call data sprawl, and it is a key driver of dataflow complexity.

The challenges of increased dataflow complexity are here and now and, given the unprecedented growth of data each day, must be considered the new normal. With this information as a bird's-eye view of the state of data in motion, savvy enterprises will adopt technologies and solutions that will help them evolve with the big data landscape.

Pat Patterson is Community Champion at StreamSets.

Share this

The Latest

June 27, 2017

In today's everchanging IT industry, network engineers face a slew of challenges when it comes to network management. As networks continue to grow and become more complex, many IT professionals struggle to get a grasp on key workflows in which network engineers still rely on manual processes, including network documentation, troubleshooting, change management and cybersecurity ...

June 26, 2017

Many organizations are struggling to resolve customer-impacting incidents quickly enough to preserve brand loyalty and revenue, according to PagerDuty's recent State of Digital Operations Report ...

June 23, 2017

"Become the Automator, Not the Automated." While it's a simple enough phrase, it speaks directly to how today's organizations and IT teams must innovate to remain competitive. A critical aspect of innovation is acknowledging the digital transformation of businesses. The move to digitalization enables organizations to more effectively unlock the power of information technology (IT) to fuel and accelerate business innovation. It is a competitive weapon and a survival imperative ...

June 22, 2017

Executives in the US and Europe now place broad trust in Artificial Intelligence (AI) and machine learning systems, designed to protect organizations from more dynamic pernicious cyber threats, according to Radware's 2017 Executive Application & Network Security Survey ....

June 21, 2017

While IT service management (ITSM) has too often been viewed by the industry as an area of reactive management with fading process efficiencies and legacy concerns, a new study by Enterprise Management Associates (EMA) reveals that, in many organizations, ITSM is becoming a hub of innovation ...

June 20, 2017

Cloud is quickly becoming the new normal. The challenge for organizations is that increased cloud usage means increased complexity, often leading to a kind of infrastructure "blind spot." So how do companies break the blind spot and get back on track? ...

June 19, 2017

Hybrid IT is becoming a standard enterprise model, but there’s no single playbook to get there, according to a new report by Dimension Data entitled The Success Factors for Managing Hybrid IT ...

June 16, 2017

Any mobile app developer will tell you that one of the greatest challenges in monetizing their apps through video ads isn't finding the right demand or knowing when to run the videos; it's figuring out how to present video ads without slowing down their apps ...

June 15, 2017

40 percent of UK retail websites experience downtime during seasonal peaks, according to a recent study by Cogeco Peer 1 ...

June 14, 2017

Predictive analytics is a popular ITOA technology that you can leverage to improve your business by leaps and bounds. Predictive analytics analyzes relationships among various data points to predict behavioral trends, growth opportunities and risks, which can add critical value to your business. Here are a few questions to help you decide if predictive analytics is right for your business ...