I work full-time for the University of Michigan, helping leaders to leverage data for decision making, but on top of that, I also write books about different data-centric technologies. One of my favorite parts about the latter is the large variety of topics I encounter on a day-to-day basis. My latest title for O'Reilly, The Rise of Logical Data Management, was an eye-opener for me. I'd never heard of "logical data management," even though it's been around for several years, but it makes some extraordinary promises, like the ability to manage data without having to first move it into a consolidated repository, which changes everything. Now, with the demands of AI and other modern use cases, logical data management is on the rise, so it's "new" to many. Here, I'd like to introduce you to it and explain how it works.

The Traditional Approach: What's Missing?

Normally, data needs to be in one physical place before it can be queried, reported on, visualized, or leveraged in any meaningful way for decision-making or analytics. Years ago, the role of this one physical place was performed by the on-premises data warehouse, which has recently been upstaged by cloud data warehouses, data lakes, and data lakehouses. Each of these architectures have their own strengths and weaknesses, but they all rely on data being replicated from multiple source systems into one core repository, before it can be leveraged for analysis.

Mostly, this replication is accomplished through batch-oriented extract, transform, and load (ETL) processes and other, faster methods. The problem is, one doesn't always want to replicate data, even if it can be done in real time. I'll give you a few examples: Some privacy regulations may limit the number of times personal information can be copied, or they may restrict personal information from being copied across borders. During merger and acquisition (M&A) activity, one department may wish to combine certain data from both companies but it might not be able to happen if the acquiring company hasn't yet determined the structure of the newly combined company. I could go on and on.

Because some data will always be siloed to an extent or distributed, organizations are realizing that the traditional data management approach, though increasingly powerful, are struggling to support AI and other demanding use cases such as true self-service access to data for business users. This is because such use-cases require not only live data, but also live data that is trusted, semantically standardized, and well-governed. What is missing is a way to provide that, while also not requiring replication, which always has a cost in terms of disk space, security, and back-up provisions. This is exactly what logical data management provides.

How Logical Data Management Works

Logical data management leverages data virtualization to create a virtual model of all applicable data sources, made available to an organization as an enterprise-wide logical layer. To query the data in any of the underlying data sources, which could include not only traditional on-premises databases but also cloud systems, software-as-a-service (SaaS) applications, data warehouses and data lakehouses, a user would not even need to know where the data is actually stored or how to access it; a user would only need to query the logical layer, which would get the necessary data for the user during the moment of the query. In this sense, the logical data layer enabled by logical data management abstracts users from the complexities of accessing the individual data sources.

One of the most profound benefits of this architecture is that by establishing an enterprise-wide logical layer above an organization's disparate data sources, it also enables organizations to establish a powerful semantic layer within that logical layer, to standardize semantics across the entire organization in an immediate, highly effective manner rather than at each of the different data sources individually.

In the same way, logical data management also enables organizations to implement data governance and security controls across the logical layer, to effectively control all of the underlying data sources from a single interface.

A Flexible Solution

Unlike data lakehouses, cloud data warehouses, and other powerful data platforms, logical data management is not a monolithic platform that needs to replace existing infrastructure. Instead, it is relatively "light" in that it can be implemented above any existing data estate to add tremendous value in terms of support for AI and other modern use cases. It's like a layer of intelligence that can make the best use of existing systems to enable flexible new capabilities.