Simon Spring, account director EMEA at WhereScape, explores how we can embrace complexity through the power of data warehouse automation.
As organisations across the economy broaden their investment in data warehouse technologies, most are aiming to build curated repositories that provide users with access to the information they require in a format that meets their needs.
For instance, businesses the world over routinely require data-led insight for both reports and forecasts, and as a result, data warehouses have become invaluable in this and many other use cases.
While they offer a proven approach to the high-level challenges of storing, managing and analysing large and expanding datasets, the complexities associated with manual data warehouse management and administration mean that automation technologies are becoming a minimum requirement for organisations focused on time to value.
Ideally, as data enters the data warehouse environment, it needs to be cleansed, transformed, categorised and tagged so that it can be more effectively managed and monitored.
In this context, automation can help organisations focus on time to value by accelerating both data ingestion and processing.
But that’s just the start and in a holistic approach, Data Warehouse Automation (DWA) tools should be used to optimise the entire data warehousing process, as opposed to a strategy which relies on bringing together a collection of tools to solve niche problems in the data warehousing lifecycle.
End-to-end automation means organisations then don’t need teams of specialists at each stage of the process with manual handoffs between them, and where any miscommunication can result in delays or errors.
Instead, by providing automated templates, users can add their own data sources and model the data to suit its needs, repetitive tasks are eliminated while leaving IT teams fully in control of the process.
The impact of this approach can be extremely significant. As Gartner explained in its report ‘Assessing the Capabilities of Data Warehouse Automation (DWA)’, the template-driven approach reduces operational and compliance risks and, “The template-driven approach for data warehouse development reduces operational and compliance risks and is a disciplined process for delivering quality data warehouses incorporating all the best practices.”
Data warehouse automation is also helping organisations address the burgeoning levels of complexity that, if left unresolved, can impede their ability to get the most out of their data.
Think of it this way – as teams look to innovate to broaden their use of data and provide access to the right data at the right time, complexity is a likely by-product.
But for those with the job of designing and running the data ecosystem, the continuing, widespread use of legacy ETL tools and hand-coding exacerbates the impact of increasing complexity.
As Gartner explains, “Automating these elements’ design plays a critical and essential role in data warehouse modernisation and agile data warehousing.”
And looking at the role data warehouses play in wider efforts to innovate and become more agile, they have become increasingly important to teams focused on DevOps, DataOps and other Agile methodologies.
But, with DWA tools addressing time-consuming manual tasks and the impact of complexity, data teams can more effectively prioritise the delivery of strategic infrastructure projects and/or projects that rely on agile timeframes.
In turn, this can put them in a much stronger position to transform the way data is available to and used by the wider organisation.
DWA can also have a positive impact on the collaboration that should exist between IT and the business, so that critical processes, such as prototyping, can be implemented more rapidly and effectively.
For instance, using data-driven design so developers can create prototypes with real company data, can help illustrate how processes will work in the production version of the data warehouse.
The point is, business and IT teams can look at the data together, using inputs and feedback before creating the model.
With the benefit of an iterative approach, data warehouse developers can then accelerate the development of multiple prototypes before implementing the solution – knowing in advance that they will be able to meet the business requirements. The method provides flexibility for deployment as well as management of changes to the data with flexible updates.
These are crucial considerations. In the rush to become more data driven, automation has become a proven route to efficiency, impact and vitally, time to value.
Those organisations who successfully embrace the complexities of today’s digital ecosystems will quickly become masters of their data, while others remain limited by a long-term reliance on outdated manual processes.
