Data – much like drinking water – is a critical resource. Since developers want to use data to build innovative applications and products, they must use as much as possible to deliver their end goals. But having too much data flowing around a modern distributed enterprise environment is a risky business.
It’s a conundrum often described in terms of the ‘tragedy of the commons’, a situation where an individual is incentivised to consume as much of a limited, shared resource as possible.
It’s a term often used with regard to the overuse of scarce natural resources. In the natural world, for example, this overconsumption might be water. Or it could be the depletion of marine life by overfishing.
In the modern IT environment, data is a different tragedy of the commons.
The data developers often rely on includes customers’ personal data like names and birth dates. The risks should this information being damaged – or stolen – can be catastrophic.
This is where database administrators (DBAs) – who are tasked with managing and securing this data – come in. But there’s an inherent tension between developers (who want as much access as possible) and DBAs, who want to limit this access.
Knowing how data is used within an enterprise is critical because it must be used to create new solutions and be kept secure. On the one hand, an overly open data policy could lead to breaches and leaks. On the other, an unduly restrictive policy limits innovation. Thankfully, there’s a solution to help ease this tension.
Observability can automatically analyse massive amounts of information across an entire IT environment. It can pinpoint the causes of outages or performance issues and provide actionable insights so teams can quickly resolve them. Additionally, observability solutions use artificial intelligence (AI) to predict and proactively prevent problems before they occur.
DBAs and developers can utilise observability solutions to balance boosting innovation and securing data. Observability can solve the tragedy of the data commons and ease the feud between DBAs and developers in three critical ways.
Ensuring data is secure
One of the biggest concerns DBAs have regarding data access is security. Sensitive production data is no longer stored exclusively on a company’s on-premises servers, where DBAs have complete visibility into how it’s used and who has access.
Instead, it’s now stored across various private and public clouds and on-premises environments, which has reduced visibility for DBAs.
This sensitive data is the same information developers often need to build and launch applications in a continuous delivery model. Developers build software in short cycles and can release it at any time.
This involves working across different clouds – including AWS and Azure – while leveraging complicated cloud-enabled technologies such as microservices and Docker containers. When data is deployed across multiple clouds, the attack surface increases and visibility decreases, raising new security concerns.
Observability gives DBAs a better understanding of how developer teams use data so they can help make sure it remains secure and protected.
Moreover, many modern IT environments deploy threat intelligence solutions to observe actions, detect threat signals, help predict security incidents, and respond before they happen.
Some threat intelligence applications are powered by AI, which can automatically gather and organise complex log data from the entire network. This data is then compared with an out-of-the-box threat database, helping to detect suspicious activity early.
By working in tandem, observability and threat intelligence provide a comprehensive overview to help relieve some security concerns for DBAs. This also creates a more secure environment for data to be shared between teams.
Remaining data compliant
Ensuring data is compliant is pivotal for organisations. Compliance regulations such as GDPR, CCPA, and HIPAA carry severe penalties in the event of data leaks, which means companies must maintain the integrity and availability of sensitive and regulated data.
Ensuring compliance while managing requests for information from developers working across geographies and clouds can be a daunting task for DBAs. They must simultaneously remain compliant and demonstrate continuous compliance – and this usually requires complex, dynamic, and automated systems.
Observability allows teams to clearly see and follow the complex flow of data as it moves between DBAs, developers, and other teams. Additionally, teams can deploy observability to help ensure sensitive data is anonymised before it’s shared.
DBAs can also use observability to track the lineage of data as it moves between teams to ensure it isn’t changed or transformed to help avoid the risk of a compliance violation.
Making compliance a regular part of the IT environment shouldn’t be a burden. In fact, it provides many benefits – such as improving data quality – which can be used to innovate further and build better products.
Making teams more productive
Observability has shown it can create harmony between DBAs and developers by helping mitigate security and compliance concerns. Observability helps address these issues and can make each team more productive. For DBAs, this means gaining additional insights to ensure applications and services are running correctly.
After all, one of the traits of observability is its ability to collect information from across hybrid and multi-cloud environments and present it in a single view to allow DBAs to respond quickly and resolve issues.
When DBAs gain the insights observability provides, they can better manage critical data and anticipate future problems before they arise.
Observability is essential to helping developers build applications as productively as possible. The ongoing ‘shift left’ has necessitated observability to gain single-pane-of-glass visibility into the build process.
Additionally, observability gives developers insight into the performance of web applications and helps them move quickly to address problems.
Finally, observability improves the speed at which teams can innovate, providing live code profiling they can use to predict and prevent future problems.
There’s little doubt observability helps ensure data doesn’t fall victim to the tragedy of the commons, where it might harm an entire organisation. Instead, it empowers teams to use data securely to work more productively and build innovative applications.
