8 Big Data Pain Points and How to Address Them - Part 1
August 02, 2018

Kamesh Pemmaraju

Share this

The word "Big" in Big Data doesn't even come close to capturing what is happening today in our industry and what is yet to come. The volume, velocity, and variety of data that is being generated has overwhelmed the capabilities of infrastructure and analytics we have today.

We are now experiencing Moore's law for data growth: data is doubling every 18 months. No wonder IDC forecasts that the global datasphere will grow to 163 zettabytes (a trillion gigabytes) by 2025. That's ten times the data generated in 2016.

Data scientists typically may have to simultaneously combine data from various sources with different volume, variety, and velocity needs to gain useful insights, but that in turn puts different demands on processing power, storage and network performance, latencies etc. Here's a quick look at the different types of Big Data sources:

Unstructured data: The type of data generated by sources such as social media, log files, and sensor data is not very structured and hence is generally not amenable to traditional database analysis methods. A large variety of Big Data tools, techniques, and approaches have emerged in the last few years to ingest, analyze, and extract customer sentiment from social media data. Newer approaches include Natural Language Processing, News Analytics, unstructured text analysis, etc.

Semi-structured data: Some unstructured data may in fact have some structure to them. Examples include email, call center logs, and IoT data. Some in the industry have coined a new term, "Semi-structured data," to describe these data sources. These may require a combination of traditional databases and newer Big Data tools to extract useful insights from these types of data.

Streaming data brings in the dimension of higher velocity and real-time processing constraints. The velocity of data varies widely depending on the type of application: IoT data tends to be small packets of data regularly streamed at low velocity, while 4K video streams stretch the velocity to the highest end of the spectrum.

The alluring promise of these new use cases – and associated emerging technologies and tools – is that they can generate useful insights faster so that companies can take actions to achieve better business outcomes, improve customer experience, and gain significant competitive advantage.

No wonder Big Data projects have been on the CIO top ten initiatives for the past decade – almost 70 percent of Fortune 1000 firms rate big data as important to their businesses; over 60 percent already have at least one big data project in place.

While data scientists are dealing with the complexity of how to derive value from diverse data sources, IT practitioners need to figure out the most efficient way to deal with the infrastructure requirements of Big Data projects. Traditional bare-metal infrastructure, with its siloed management of servers, storage, and networks, is not flexible enough to tackle the dynamic nature of the new Big Data workloads. This is where cloud-based systems shine. However, many challenges remain to be addressed in the areas of workload scaling, performance and latency, data migration, bandwidth limitations, and application architectures.

There are many pain points that companies experience when they try to deploy and run Big Data applications in their complex environments or use public or private cloud platforms, and there are also some best practices companies can use to address those pain points.


As data amounts grow from terabyte to petabyte and beyond, the time it takes to transport this data closer to compute resources and perform data processing and analytics takes longer and longer, impeding the agility of the organization. Public cloud vendors like AWS, who are all about centralized data centers, want to get your data into their cloud and go to extreme lengths (see AWS snowmobile) to get it. Furthermore, data transfer fees are mostly unidirectional, i.e., only data that is going out of an AWS service is subject to data transfer fees. Not only is this a classic lock-in scenario, but it also goes against other key emerging trends:

Edge Computing and Artificial Intelligence, especially for use cases such as IoT, 5G, image/speech recognition, Blockchain, and others, where there is a need to place processing and data closer to each other and/or closer to where the user or device is. Edge computing delivers faster data analytics results with the data being closer to processing while simultaneously reducing the cost of transporting data to the cloud.

Artificial Intelligence systems are more effective the more data they are given. For example, in deep learning, the more cases (data) you give to the system, the more it learns and the more accurate its results become. This is a case where you need massive parallel processing (e.g., using GPUs) of large data sets. Big Data analytics and AI can complement each other to improve speed of processing and produce more useful and relevant results.

To address the need to get data to where the compute resources are, IT leaders should look for hyper-converged, scale-out solutions that bring together compute, storage, and networking, thus reducing data I/O latency and improving data processing and analytics times. For even better performance, they should look for solutions that can bring the computing units (VMs or containers) as close to the physical storage as possible, without losing the manageability of the storage solution and while maintaining multi-tenancy across the cluster. For example, a Hadoop Data Node VM running on the same physical host and accessing local SSDs will experience the highest performance and faster results overall without impacting other workloads running within other tenants.

IT leaders can take advantage of many emerging memory technologies such as persistent memory (a new memory technology between DRAM and flash that will be non-volatile, with low latency and higher capacity than DRAMs), NVMe, and faster flash drives. With prices falling rapidly, there seems little need for spinning disks for primary storage.

IT administrators should implement a central way to manage all the edge computing sites, with the ability to deploy and manage multiple data processing clusters within those sites. Access rights to each of these environments should be managed through strict BU-level and Project-level RBAC and security controls.


For data science development and testing use cases, companies do not build a single huge data processing cluster in a centralized data center for all of their big data teams spread around the world. Building such a cluster in one location has DR implications, not to mention latency and country-specific data regulation challenges. Typically, companies want to build out separate local/edge clusters based on location, type of application, data locality requirements, and the need for separate development, test, and production environments.

Having a central pane of glass for management becomes crucial in this situation for operational efficiency, simplifying deployment, and upgrading these clusters. Having strict isolation and role-based access control (RBAC) is often a security requirement.

IT administrators should implement a central way to manage diverse infrastructures in multiple sites, with the ability to deploy and manage multiple data processing clusters within those sites. Access rights to each of these environments should be managed through strict BU-level and Project-level RBAC and security controls.


Companies still run the majority of their Big Data workloads, particularly Hadoop-based workloads, on bare metal. This is obviously not as scalable, elastic, or flexible as a virtual or cloud platform. Traditional bare metal environments are famous for creating silos where various specialist teams (storage, networking, security) form fiefdoms around their respective functional areas. Silos impede velocity because they lead to complexity of operations, lack of consistency in the environment, and lack of automation. Automating across silos turns into an exercise of custom scripts and lot of "glue and duct tape," which makes maintenance and change management complex, slow, and error-prone.

A virtualized environment for Big Data allows data scientists to create their own Hadoop, Spark or Cassandra clusters and to evaluate their algorithms. These clusters need to be self-service, elastic and high performing. IT should be able to control the resource allocation to data scientists and teams using quotas and role-based access control.

Better yet, IT managers should look for an orchestration platform that can deal with both bare metal and virtual environments, so IT can place workloads in the best target environment based on performance and latency requirements.

Read 8 Big Data Pain Points and How to Address Them - Part 2, to learn about 5 more big data pain points.

Kamesh Pemmaraju is VP of Product at ZeroStack
Share this

The Latest

August 17, 2018

As a Network Operations professional, you know how hard it is to ensure optimal network performance when you’re unsure of how end-user devices, application code, and infrastructure affect performance. Identifying your important applications and prioritizing their performance is more difficult than ever, especially when much of an organization’s web-based traffic appears the same to the network. You need insight to maximize performance — not inefficient troubleshooting, longer time to resolution, and an overall lack of application intelligence. But you can stay ahead. Follow these 10 steps to maximize the performance of your applications and underlying network infrastructure ...

August 16, 2018

IT organizations are constantly trying to optimize operations and troubleshooting activities and for good reason. Let's look at one example for the medical industry. Networked applications, such as electronic medical records (EMR), are vital for hospitals to provide outstanding service to their patients and physicians. However, a networking team can often not be aware of slow response times on the remotely hosted EMR application until a physician or someone else calls in to complain ...

August 15, 2018

In 2014, AWS Lambda introduced serverless architecture. Since then, many other cloud providers have developed serverless options. What’s behind this rapid growth? ...

August 14, 2018

This question is really two questions. The first would be: What's really going on in terms of a confusion of terms? — as we wrestle with AIOps, IT Operational Analytics, big data, AI bots, machine learning, and more generically stated "AI platforms" (… and the list is far from complete). The second might be phrased as: What's really going on in terms of real-world advanced IT analytics deployments — where are they succeeding, and where are they not? This blog will look at both questions as a way of introducing EMA's newest research with data ...

August 13, 2018

Consumers will now trade app convenience for security, according to a study commissioned by F5 Networks, The Curve of Convenience – The Trade-Off between Security and Convenience ...

August 10, 2018

Gartner unveiled the CX Pyramid, a new methodology to test organizations’ customer journeys and forge more powerful experiences that deliver greater customer loyalty and brand advocacy ...

August 09, 2018

Nearly half (48 percent) of consumers report that they currently use, or have used in the past, services of organizations that were involved in a publicly disclosed data breach and, of those, 48 percent have stopped using the services of an organization because of a breach, according to Global State of Digital Trust Survey and Index 2018, a new report from CA Technologies ...

August 08, 2018

Here's the problem: IT teams are in the dark. The only information they have available to them is based on what users decide to tell them about through calls to the help desk ...

August 07, 2018

Over the past year, the enterprise network grew significantly more complicated, creating new challenges for network professionals, according to IDG’s 8th annual State of the Network study. Internet of Things (IoT) projects, the demands of an increasingly mobile workforce, and an explosion of apps prompted network professionals to enhance their network infrastructure and the skillsets needed to support it. Network professionals are now being asked to help shape IT strategy ...

August 06, 2018

Retailers are already busy prepping to avoid an Amazon Prime type meltdown during the holiday shopping season. However, rather than focusing efforts on coping with surges in traffic to your website, you also need to be thinking about the ongoing speed of your site ...