As data insights continue to be the key factor in driving business innovation and growth, organizations are constantly refining their data strategies, exploring frameworks like data mesh to give more users self-service access in producing and consuming data. Recent data shows that nearly half of data leaders identified data mesh as a primary area of investment for 2023.
Data mesh has been revolutionary for many data-driven organizations. Before data mesh, teams had to coordinate data access and use through centralized IT bottlenecks, leading to long waiting times to access data and ultimately resulting in frustration for both data producers and consumers. With data mesh, teams can independently develop and manage their own data products through the decentralized data ownership and the enablement of domain experts. Not only does this allow organizations to boost their data-driven initiatives, but it also helps them enhance everything from data democratization to alignment between business operations and data resources, and sustain growth at scale.
Whilst the distribution and delegation of responsibilities promoted through data mesh has many benefits, achieving an implementation that fulfills data security standards is not without its challenges. Many organizations run into issues around data access, governance, and privacy. Let's explore these issues further and some steps organizations can take to help overcome them.
Data Mesh Security Challenges
The need to secure data mesh is only becoming more urgent as regulations increase and the US federal government continues to roll out more data privacy actions. What makes this a complex process is the fledgling nature of the data mesh combined with its distributed composition. Data security must be applied in a way that simultaneously protects individual domains and the entire ecosystem, without hindering data accessibility and innovation.
As data mesh implementations become more regular, there are three main challenges I see organizations experiencing when it comes to implementing and securing the architecture:
■ Decentralized ownership and access control. While the decentralized data ownership that comes with data mesh offers a range of benefits, it can also be difficult to keep track of who owns what when it comes to data collection, processing, sharing and use. Systems that are built to enable cross-domain data discovery, access and sharing can help address this challenge, but can also lead to a larger attack surface for bad actors. On top of this, it becomes more challenging for rules to stay consistent across data products, risking the security of the data.
■ Data governance. Along the same lines, it's no surprise that centralized ecosystems are easier to protect when it comes to data governance and compliance with regulations as opposed to decentralized environments. With distributed domains and data locations, security requirements become much more complex, requiring additional governance policies for each domain, and a way to oversee the security and compliance of the entire domain-based framework. Access and governance requirements are also federated, making it more difficult to consistently and effectively protect data.
■ Privacy in a self-service environment. Less oversight into data access and outdated/inadequate controls can also increase the likelihood of data misuse across domains. This is a growing concern as today's organizations collect increasing volumes of sensitive, personal data in order to provide consumers with more personalized products, experiences and services. Because this personal data — if exposed or accessed by an unauthorized party — can easily harm the data subject, modern data rules and regulations are requiring stricter privacy protections be enforced on data ecosystems.
Three Steps That Can Help
There are a few best practices organizations can follow to help achieve an efficient, secure, and distributed data ecosystem.
1. Maintaining consistent metadata. The first step in securing any data mesh architecture should be creating and applying a consistent metadata identification and tagging system. Why? Because users and administrators must have a thorough understanding of the resources at their disposal in order to protect and secure any data ecosystem.
Metadata allows them to identify and understand these ecosystem parts, from data sets to data users, by providing critical contextual information about resources or users that is vital to the system's operation. This helps with efficient access management, analytics, monitoring and compliance — all critical elements of data mesh. However, metadata is ineffective unless it can be consistently attributed and understood across domains.
To maintain consistent metadata, organizations can leverage tools that offer sensitive data discovery (SDD) capabilities, enabling teams to assess their data and ensure that it is tagged and classified appropriately. This helps data teams gain a better, holistic view of the resources across their distributed data mesh for enhanced data security.
2. Employing global & local policy management. Establishing a balance of both horizontal (global) and vertical (local) policies is crucial to data mesh security and governance. In the distributed domains of data mesh architectures, policies can be applied locally within specific domains. But these rely solely on domain-based policies, limiting consistency across the data ecosystem and requiring great manual effort and time to maintain. Applying policies globally across domains is not a perfect solution either — it improves consistency, but overlooks the unique requirements of each domain's purpose, users and specific data resources.
That's why finding the right balance of horizontal and vertical policy management, and maintaining them as scale, is key. A federated governance framework can help create, apply, and maintain policies at both the global and local level. Within this framework, domain-level policy management is delegated to the teams that own the data, and the responsibility of global policies remains with security and governance teams.
All teams must maintain rigorous activity monitoring across domains so that they can have complete oversight of global policy application and local policy enforcement within specific domains. This helps them respond to and manage security incidents as quickly as possible, and effectively manage both global and local data security.
3. Foster organizational alignment. Adopting a data mesh framework is an organizational change. In order to achieve true success, data teams, engineers and leaders must also be aligned and behind the initiative as well.
A large part of this involves organizations identifying internal data mesh champions to lead the charge and help teams adopt a data mesh mindset. Leaders must also learn to effectively collaborate and communicate with one another. Once technical, security, business and compliance stakeholders are aligned, virtually any organization can establish an effective and secure data mesh framework.
At the end of the day, every organization will have a different approach to data mesh depending on their industry, business needs, and data demands. But security remains a critical component across all data mesh strategies, at any stage. By following these steps, businesses can effectively jumpstart their data mesh strategy.