Kevin Kline, Database Technology Evangelist at SolarWinds, explains how data integration is evolving in the age of generative AI.
If you’ve ever had to hot-foot it from the suburbs of a sprawling metropolis to a city centre for work or to go shopping, you’ll know it’s not always easy. A centralised city layout may look good on paper, but if it’s swamped by people and traffic, it can quickly grind to a halt.
The same is true of traditional data management systems. Built around a centralised hub, it can provide fast and easy access for all of your needs, but it too can become gridlocked under immense load.
In much the same way that town planners have sought to decentralise cities and towns, so too have those responsible for the free flow of data. And to address the problems associated with a traditional centralised data management system, they’re turning to a data mesh architecture.
Just as a city has different areas – residential, commercial, and industrial, all interconnected by roads – a data mesh has different domains usually aligned with specific departments within an enterprise. These domain teams manage their own analytics data, linked by sharing standards, governance, and technology.
Decentralisation for domain-oriented data ownership
For businesses or large organisations, it’s an approach that is gaining real traction. By adopting this architecture, each domain within a business – such as marketing, sales, or customer service – manages its own analytics data while receiving support from one or more data platform teams.
The advantages of this are clear. Each domain has a vested interest in ensuring the accuracy and quality of its own data. However, while it may seem that the data mesh approach to data management breeds more data silos within an enterprise, in fact, the company ensures that its data is managed by those who benefit from it the most.
By giving each domain in the enterprise direct ownership of its data and analytics – combined with sharing that data across other domains – silos are broken down and business agility is greatly accelerated due to the self-service nature of data mesh.
As part of the data mesh architecture, knowledge workers have the freedom of self-service access to their data. They’re also able to use enterprise data from whenever they need it without having to employ sneaky shadow IT tactics or go through territorial DBAs, IT specialists, or data subject-matter experts.
Navigating data governance in a decentralised world
This may ring alarm bells among those concerned that this could lead to a free-for-all that quickly descends into chaos. This is why implementing a data mesh goes hand-in-hand with sufficient checks and balances, along with well-planned governance processes and workflows.
From the outset, it’s important to set guidelines for classifying, handling, using, and protecting data across all departments. This ensures that guardrails are in place so everyone knows what they can and cannot do. These guardrails also ensure that data is visible to only those workers who have proper permissions to use it.
Enterprises are well aware of the importance of maintaining and monitoring their IT infrastructure. As part of the data mesh strategy, it’s important to assign responsibility for the governance and workflow management needed to ensure data is accurate, secure and used appropriately, according to the company’s rules. In effect, just as enterprises need IT security – usually under the purview of a CISO – so too do enterprises need similar staff in this role to act as an ‘enforcer’ of data security, keeping data secure while ensuring that the right people can easily access the data they need to do their jobs.
They also need to monitor the management of their information systems to see that data rules are followed. Access to data is governed by a variety of legal jurisdictions, laws, and regulations. That means data access and utilisation must also be audited regularly and, if needs be, action needs to be taken against those who flout the rules.
But while rules, governance, and compliance are important, so too is communication. If a data mesh is to work well, then it’s essential to encourage and facilitate communication between different departments to ensure everyone is aligned and making decisions from consistent and universally reliable information.
Generative AI in data management
Increasingly, that can be assisted by introducing AI-powered observability and service management solutions. Until now, many have incorporated AI models, typically using machine learning (ML), that are purpose-built by IT operation teams. But thanks to the large language models (LLMs), the use of generative AI in IT operations is helping to eradicate tedious, repetitive tasks such as organising and cleaning data.
When planning your best path forward for your organisation’s data strategy, a data mesh architecture is the best of all worlds. The data is owned and curated by the people best placed to carry out that task, while easy-to-use tools eliminate bottlenecks, allowing decision-makers to do make choices with trustworthy data. As a result, the future of data management will be more integrated, secure, and user-friendly, enabling businesses to leverage data more effectively for their growth and innovation.
