In collaboration with MIT Tech Review, Databricks has released its new report, Building a high-performance data and AI organisation, diving into how data management strategies and technologies are evolving and the challenges organisations face when it comes to managing their data.
The global study surveyed 351 chief data officers, chief analytics officers, chief information officers, and other senior technology executives, as well as in-depth interviews with data and analytics leaders at organisations including Total, The Estée Lauder Companies, McDonald’s, L’Oréal, CVS Health and Northwestern Mutual.
According to MIT Technology Review Insights, high-performance data management is critical for delivering business results with cloud data and AI platforms.
So, where are businesses falling flat?
Scaling ML is a challenge for many organisations
Machine learning’s business impact is limited by difficulties managing its end-to-end lifecycle.Scaling machine learning use cases is exceedingly complex for many organisations. The most significant challenge, according to 55% of respondents, is the lack of a central place to store and discover machine learning models.
Enterprises seek cloud-native platforms
Organisations’ top data priorities over the next two years fall into three areas, all supported by wider adoption of cloud platforms: improving data management, enhancing data analytics and machine learning, and expanding the use of all types of enterprise data, including streaming and unstructured data.
43% of respondents said increasing the adoption of cloud platforms that support data management, analytics, and machine learning is one of the most important enterprise wide-data strategy initiatives over the next two years.
Lack of ROI on analytics or BI initiatives
Only 12% of respondents said they’ve achieved optimal price/performance for their analytics workloads.
Open-source and open-data formats are the missed opportunity
Open standards are the top requirements of future data architecture strategies.If respondents could build a new data architecture for their business, the most critical advantage over the existing architecture would be a greater embrace of open-source standards and open data formats.
50% of respondents cited leveraging open-source standards and open-data formats if they could do it all again and build a new data architecture for their business.
The secrets to success?
Attention to detail
Just 13% of organisations excel at delivering on their data strategy. This select group of “high-achievers” deliver measurable business results across the enterprise. They are succeeding thanks to their attention to the foundations of sound data management and architecture, which enable them to “democratise” data and derive value from machine learning.
Technology-enabled collaboration is creating a working data culture
The chief data officers interviewed for the study ascribe great importance to democratising analytics and machine learning capabilities. Pushing these to the edge with advanced data technologies will help end-users to make more informed business decisions – the hallmarks of a strong data culture.
“Managing data is highly complex and can be a real challenge for organizations. But creating the right architecture is the first step in a huge business transformation,” comments Francesca Fanshawe, editor of the report.
“There are many models an enterprise can adopt, but ultimately the aim should be to create a data architecture that’s simple, flexible, and well-governed.”
“The past year has been an accelerant of change as data-driven organisations look to adapt, innovate, and future proof their technology and architecture investments,” adds Chris D’Agostino, global principal technologist at Databricks.
“Now more than ever, enterprises need a modern data analytics strategy that is open, flexible, and empowers everyone across the organisation to make faster, more informed decisions with a unified view of all their data – whether that’s using machine learning and AI algorithms or straightforward SQL and BI reporting”
