With the digital transformation of almost everything, we are experiencing a so-called “data explosion.” The amount of unstructured data being generated is growing at a rate of up to 60% per year and is expected to make up 80 to 90% of all data by 2025.
According to a recent Forbes survey, an overwhelming majority (95%) of businesses expressed the necessity to manage their unstructured data, with over 40% of businesses saying they must do so on a regular basis.
To put this into perspective, here are some examples of the type and volume of data being collected and processed:
- Every year, a semiconductor manufacturer produces over a billion image scans documenting the manufacturing process of 4,000 wafers every week, which mean results in petabytes of data that must be stored for multiple years.
- An autonomous car produces up to 2 TBs of data every hour of use that must be maintained for several years for potential safety analysis or remodeling.
- Every human genome sequenced is unique and consumes 100 GBs of storage. Each genome holds immense value to both current and future medical research – the value in this data for future medical research is the main reason behind storing this data for decades to come.
As data growth increases, companies’ storage budgets and data storage capacities are struggling to keep up. The challenge of storing unstructured and inactive data cannot be ignored. This is where cold storage comes in.
So, what is the deal about cold storage?
Just like a story, data also has a beginning, a middle and an end. Depending on where the data is in its life cycle and its value to a current project, it can be classified as ‘hot’, ‘warm’ or ‘cold data’. Data that is ‘on demand’ and actively being used almost daily for a project, is called ‘hot’ data. Then there is ‘warm’ data, which needs to be accessed regularly but not daily. However, ‘cold’ data or inactive data is data that is no longer or very rarely needed. This data makes up over 60% of all stored data.
Typically, cold data was preserved by companies to comply with regulations or internal policies. However, this is not always the case anymore. Increasingly, companies are choosing to preserve cold data not for data compliance reasons but because they realise the long-term strategic value it holds, the future potential of the data to be re-used for projects, as well as the chance to enrich the data over time to augments its value.
Traditionally, cold data has been written to tape media, moved offline, and transferred to a storage facility for use in the future. Cold data storage is a highly durable, secure, and cheaper way to preserve data in the long-term. The emergence of new data analysis methods, increase in computing power, and the rapid acceleration of digital transformation, means there is a greater demand for data to be stored online and made easily accessible. However, concerns around long-term durability, data security, and storage costs still need to be addressed.
Is a cold storage strategy essential?
It is predicted that data created during the next three years will amount to more than all of the data created during the past 30 years. So, as organisations are waking up to the value of cold data, along with the rapid surge in data generation, a cold storage strategy is vital if companies want to stay ahead of the data curve. Companies need to prioritise architecting meaningful and scalable solutions to harvest data-driven innovations and opportunities to limit the common issues around massive data accumulation and continued data growth.
So, what are the advantages of deploying cold storage?
Recent innovations in cold data storage have enhanced performance, data durability and storage efficiency, raising the bar on performance and affordability expectations. Storing vast amounts of data can be extremely expensive for organisations. However, cold data storage is considerably cheaper than the NVMe and solid-state disk technologies used for high performance access to hot data. One of the main reasons for this is because cold data can be stored on lower performing and cheaper storage infrastructures, either in-house or in the cloud. This means organisations can cost-effectively store more of their growing data sets.
Modern cold storage archives have been created by some of the world’s largest cloud solution providers, but with emerging architectures and services, cold storage solutions are now deployable within an organisation’s data centre, colocation facility or hosted IT environment that meets data sovereignty and data residency requirements. This means that the data is more easily accessible with no access fees and retrieval times are minutes not days. Additionally, new erasure coding algorithms are now optimised specifically for cold storage which helps to reduce the storage overhead compared to the outdated practice of storing multiple copies.
Looking to the future of data and its value
Cold storage is not only a way of storing growing data sets, but it also helps organisations to securely maintain in-house control of these invaluable assets and easily retrieve them for future projects. In this new data era, it is becoming increasingly evident that data can no longer be disregarded if it doesn’t serve any immediate purpose. It needs to be preserved both for its historic relevance and value in the future. Thanks to new advances in cold storage, companies don’t have to sacrifice inactive data for active data. Whether an organisation is looking to reduce costs or increase their data’s value, cold storage should not be disregarded.
