Dan Sommer, senior director & global market intelligence lead at Qlik, highlights the importance of an agile, automated and agnostic data pipeline.

In recent years, we’ve seen significant growth and investment in augmented analytics and machine learning as organisations seek to the get the most value from their data.

In fact, since Gartner first included Augmented Analytics in its Hype Cycle in 2017, we’ve seen rapid adoption which led it to predict that in 2020, it will be a “dominant driver of new purchases of analytics and business intelligence, as well as data science and machine learning platforms”.

However, while many organisations are investing heavily in improving their abilities to analyse and derive actionable insights from their data, most companies are typically using just a small percentage of it. 

Shockingly, IDC previously reported that as much as 90% of unstructured data is never analysed.

Why is this? If we were to compare data to water, we might think of the new analytics solutions as the faucet that gives us the output we want – a glass of water. However, what we can’t see is the path that the water takes to get there. We don’t know if the pipes have any leaks – and how much water was lost from the source – or how clean the water is, which ultimately impacts what we consume.

Similarly, the true business value of data can only be achieved when it is underpinned by an agile, automated and agnostic data pipeline that closes the gaps – and stops the leaks – by working across any cloud, system and data source in real-time. Without it, much of the data available to an enterprise remains unused or underutilised – and its potential is lost.

Identifying the gaps in your data pipeline

A new global survey conducted by IDC of 1,200 C-level executives and senior business decision makers, commissioned by Qlik, has revealed that ‘leaking’ data pipelines are a pervasive issue affecting every industry.

The majority of organisations struggle to create and find valuable data that could support better insights. Over 60% of business leaders reported experiencing significant challenges in investing in technology to create data, assess the value of data and identify valuable data sources.

Once identified, organizations face further challenges in processing or transforming the data into an analytics-ready form. Nearly half (42%) struggle to assess data correctness, while 37%  struggle to integrate disparate data sets into standard formats, and the same number come up against missing or incomplete data sets.

What fixing the pipeline is worth to your business

Those companies that invest in improving their data pipeline significantly increase their ability to use data to make decisions: analysis by IDC identified that most organisations with a strong data pipeline (87%) received the highest decision-making scores.

A clear association was then identified between these companies that were more capable of turning data into insights for decision-making and better business outcomes. In fact, the business leaders surveyed pointed to a significant impact on the bottom line from investing in a strong data pipeline: three quarters of organisations reported that operational efficiency and revenue improved by an average of 17%, while a similar number of companies indicated that profit had increased by an average of 17%.

This reaffirms the findings of the Data Literacy Index, conducted by the Wharton School on behalf of Qlik, which found that companies with employees who were empowered to make data-driven decisions – supported by greater access to data and the skills to confidently read, question and analyse it – were associated with a 3-5% increase in their enterprise value. For the global organisations that participated in the study, this represented an additional US$320-$534 million.

How to plug the gaps in your organisation

“The magnitude of the impact of different challenges throughout the data pipeline will vary from organisation to organisation – but there is no question that it will significantly reduce all firms’ ability to use their insights for better decision-making,” commented Dan Vesset, group vice president, Analytics and Information Management at IDC.  

“To achieve greater enterprise intelligence and, in turn, improved business outcomes, the whole pipeline needs to be robust. It doesn’t matter whether the faults occur at the well, in the pump, at the filtration facility, in the pipe, or at the faucet. The result is that the final data consumer doesn’t get what they expect or need.”

To help close the gaps in organisations’ data pipelines, there are five recommendations that business leaders should consider:

Invest in technology solutions that will improve the data flow at every stage of the pipeline

Companies that focus solely on the analytics tools, and not how they are getting clean data to them, will get just a fraction of the insights and, in turn, business outcomes from their data.

Hire a team of data, analytics and business subject-matter experts

Nearly half of organisations (44%) stated that finding talent was one of their greatest challenges for executing data analytics. While data expertise is undoubtedly a hot market, you must invest in growing your data team, both through external hires and by using data literacy education to upskill existing employees.

Establish a culture of collaboration in your data team

Make sure that your entire team, from the data architects and data engineers, to the designers and developers, are collaborating on creating robust and agile data pipelines that will underpin the new generation of enterprise intelligence.

Design a pipeline that embraces change

The rapid growth in augmented analytics over just the past few years will only be mirrored by new machine learning, cognitive and artificial intelligence capabilities. Ensure your data architectures and cloud technologies provide the necessary agility to take advantage of new innovations for competitive differentiation.

Make your data drive action and outcomes

To take advantage of the augmented analytics and machine learning capabilities coming to the market, businesses need an agile, automated and agnostic data pipeline that closes the gaps by working across any cloud, system and data source in real-time.

Only by plugging these gaps can organisations transform their data pipelines from passive, inform-only processes to active intelligence that drives action and outcomes to benefit the entire business.