Skip to main content

CTRL+ALT+DELETE: 5 Tips for Avoiding Data Disasters

Yaniv Yehuda

It's every system administrator's worse nightmare. An attempt to restore a database results in empty files, and there is no way to get the data back, ever.

Despite the fear and panic created by data loss, more often than not it's due to simple things that are under our control and can be prevented. Studies have shown that the single largest cause for data outages is human error. Regardless of how much you try, there are still going to be mistakes and you have to account for them in the way database changes are managed.

Here are five simple tips for keeping things running smoothly and minimizing risk.

1. Define roles and responsibilities

Safeguards need to be put in place to ensure that only authorized people have access to the production database.

The level of access shouldn't be determined only by an employee's position but also by the level of seniority. A famous story made the rounds last year when a developer shared that while following instructions in a new employee manual, he accidentally deleted the production database. To make things worse, the backup was 6 hours old and took all too long to locate. You might be shaking your head in disapproval right now over how the company could have been so irresponsible to let this happen, but it turns out … it's really not uncommon (check out the comments on this tweet).

To prevent unauthorized changes in the database that can result in utter disaster, it is essential to define, assign, and enforce distinct roles for all employees. If you need to, set roles and permissions per project to avoid any accidental spillover.

2. Confirm back up procedures

You need a well-planned backup strategy to protect databases against data loss caused by different types of hardware, software, and human errors.

You'd be surprised by how often backups simply aren't happening. In one case a sys admin complained that bringing hard drives home with backed up data was inconvenient, so the company invested in an expensive remote system; the same sys admin never got around to creating the new procedure, so the latest version of the backed-up data was 3 months old.

Another employee discovered at his new job there hadn't been a single back-up for the past three years.

Knowing the back-ups are happening isn't enough. You also need to also check to make sure they are usable and include all the data that's needed. It's worth restoring and then checking that the restored database is an exact match to the production data. A check such as "Is the most recent backup size within x bytes of the previous one" is a simple solution to make sure the restored database matches the production database.

3. Adopt version control best practices

Version control practices have long since been adopted in other code development environments, ensuring the integrity of code as only one person can work on a segment at any given time.

Version control provides the ability to identify which changes have been made, when, and by whom. It protects the integrity of the database by labeling each piece of code, so a history of changes can be kept and developers can revert to a previous version.

Bringing these practices into the database is crucial for data loss prevention, especially in today's high-paced environment with increasingly shorter product release cycles. By tracking database changes across all development groups you are facilitating seamless collaboration, while enabling DevOps teams to build and ship better products faster.

4. Implement change policies

Databases are code repositories, so they need the same safeguards when changes are made. It's crucial to have clear policies on which changes are allowed and how they are administered and tracked.

Is the action of dropping an index in a database allowed? How about a table? Do you prohibit production database deployments during daytime hours? All of these policies should not only be practiced by participating teams, but enforced on the database level, too. Keep track of all the changes and attempted changes made. A detailed audit can help detect problems and potential security issues.

5. Automate releases

By taking advantage of comprehensive automated tools, DBAs and developers can move versions effortlessly from one environment to the next. Database development solutions allow DBAs to implement consistent, repeatable processes while becoming more agile to keep pace with fast changing business environments.

Automation also enables DBAs to focus instead on the broader activities that require human input and can deliver value to the business, such as database design, capacity planning, performance monitoring and problem resolution.

Databases often hold the backbone of an organization, a priceless container for the transactions, customers, employee info and financial data of both the company and its customers. All this information needs to be protected by following clear procedures for managing database changes. Reducing the likelihood of data loss due to human error can help everyone sleep better at night.

The Latest

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 5 covers the infrastructure and hardware supporting AI ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 4 covers advancements in AI technology ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 3 covers AI's impact on employees and their roles ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 2 covers the challenges presented by AI, as well as solutions to those problems ...

In the final part of APMdigest's 2025 Predictions Series, industry experts offer predictions on how AI will evolve and impact technology and business in 2025 ...

E-commerce is set to skyrocket with a 9% rise over the next few years ... To thrive in this competitive environment, retailers must identify digital resilience as their top priority. In a world where savvy shoppers expect 24/7 access to online deals and experiences, any unexpected downtime to digital services can lead to significant financial losses, damage to brand reputation, abandoned carts with designer shoes, and additional issues ...

Efficiency is a highly-desirable objective in business ... We're seeing this scenario play out in enterprises around the world as they continue to struggle with infrastructures and remote work models with an eye toward operational efficiencies. In contrast to that goal, a recent Broadcom survey of global IT and network professionals found widespread adoption of these strategies is making the network more complex and hampering observability, leading to uptime, performance and security issues. Let's look more closely at these challenges ...

Image
Broadcom

The 2025 Catchpoint SRE Report dives into the forces transforming the SRE landscape, exploring both the challenges and opportunities ahead. Let's break down the key findings and what they mean for SRE professionals and the businesses relying on them ...

Image
Catchpoint

The pressure on IT teams has never been greater. As data environments grow increasingly complex, resource shortages are emerging as a major obstacle for IT leaders striving to meet the demands of modern infrastructure management ... According to DataStrike's newly released 2025 Data Infrastructure Survey Report, more than half (54%) of IT leaders cite resource limitations as a top challenge, highlighting a growing trend toward outsourcing as a solution ...

Image
Datastrike

Gartner revealed its top strategic predictions for 2025 and beyond. Gartner's top predictions explore how generative AI (GenAI) is affecting areas where most would assume only humans can have lasting impact ...

CTRL+ALT+DELETE: 5 Tips for Avoiding Data Disasters

Yaniv Yehuda

It's every system administrator's worse nightmare. An attempt to restore a database results in empty files, and there is no way to get the data back, ever.

Despite the fear and panic created by data loss, more often than not it's due to simple things that are under our control and can be prevented. Studies have shown that the single largest cause for data outages is human error. Regardless of how much you try, there are still going to be mistakes and you have to account for them in the way database changes are managed.

Here are five simple tips for keeping things running smoothly and minimizing risk.

1. Define roles and responsibilities

Safeguards need to be put in place to ensure that only authorized people have access to the production database.

The level of access shouldn't be determined only by an employee's position but also by the level of seniority. A famous story made the rounds last year when a developer shared that while following instructions in a new employee manual, he accidentally deleted the production database. To make things worse, the backup was 6 hours old and took all too long to locate. You might be shaking your head in disapproval right now over how the company could have been so irresponsible to let this happen, but it turns out … it's really not uncommon (check out the comments on this tweet).

To prevent unauthorized changes in the database that can result in utter disaster, it is essential to define, assign, and enforce distinct roles for all employees. If you need to, set roles and permissions per project to avoid any accidental spillover.

2. Confirm back up procedures

You need a well-planned backup strategy to protect databases against data loss caused by different types of hardware, software, and human errors.

You'd be surprised by how often backups simply aren't happening. In one case a sys admin complained that bringing hard drives home with backed up data was inconvenient, so the company invested in an expensive remote system; the same sys admin never got around to creating the new procedure, so the latest version of the backed-up data was 3 months old.

Another employee discovered at his new job there hadn't been a single back-up for the past three years.

Knowing the back-ups are happening isn't enough. You also need to also check to make sure they are usable and include all the data that's needed. It's worth restoring and then checking that the restored database is an exact match to the production data. A check such as "Is the most recent backup size within x bytes of the previous one" is a simple solution to make sure the restored database matches the production database.

3. Adopt version control best practices

Version control practices have long since been adopted in other code development environments, ensuring the integrity of code as only one person can work on a segment at any given time.

Version control provides the ability to identify which changes have been made, when, and by whom. It protects the integrity of the database by labeling each piece of code, so a history of changes can be kept and developers can revert to a previous version.

Bringing these practices into the database is crucial for data loss prevention, especially in today's high-paced environment with increasingly shorter product release cycles. By tracking database changes across all development groups you are facilitating seamless collaboration, while enabling DevOps teams to build and ship better products faster.

4. Implement change policies

Databases are code repositories, so they need the same safeguards when changes are made. It's crucial to have clear policies on which changes are allowed and how they are administered and tracked.

Is the action of dropping an index in a database allowed? How about a table? Do you prohibit production database deployments during daytime hours? All of these policies should not only be practiced by participating teams, but enforced on the database level, too. Keep track of all the changes and attempted changes made. A detailed audit can help detect problems and potential security issues.

5. Automate releases

By taking advantage of comprehensive automated tools, DBAs and developers can move versions effortlessly from one environment to the next. Database development solutions allow DBAs to implement consistent, repeatable processes while becoming more agile to keep pace with fast changing business environments.

Automation also enables DBAs to focus instead on the broader activities that require human input and can deliver value to the business, such as database design, capacity planning, performance monitoring and problem resolution.

Databases often hold the backbone of an organization, a priceless container for the transactions, customers, employee info and financial data of both the company and its customers. All this information needs to be protected by following clear procedures for managing database changes. Reducing the likelihood of data loss due to human error can help everyone sleep better at night.

The Latest

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 5 covers the infrastructure and hardware supporting AI ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 4 covers advancements in AI technology ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 3 covers AI's impact on employees and their roles ...

Industry experts offer predictions on how AI will evolve and impact technology and business in 2025. Part 2 covers the challenges presented by AI, as well as solutions to those problems ...

In the final part of APMdigest's 2025 Predictions Series, industry experts offer predictions on how AI will evolve and impact technology and business in 2025 ...

E-commerce is set to skyrocket with a 9% rise over the next few years ... To thrive in this competitive environment, retailers must identify digital resilience as their top priority. In a world where savvy shoppers expect 24/7 access to online deals and experiences, any unexpected downtime to digital services can lead to significant financial losses, damage to brand reputation, abandoned carts with designer shoes, and additional issues ...

Efficiency is a highly-desirable objective in business ... We're seeing this scenario play out in enterprises around the world as they continue to struggle with infrastructures and remote work models with an eye toward operational efficiencies. In contrast to that goal, a recent Broadcom survey of global IT and network professionals found widespread adoption of these strategies is making the network more complex and hampering observability, leading to uptime, performance and security issues. Let's look more closely at these challenges ...

Image
Broadcom

The 2025 Catchpoint SRE Report dives into the forces transforming the SRE landscape, exploring both the challenges and opportunities ahead. Let's break down the key findings and what they mean for SRE professionals and the businesses relying on them ...

Image
Catchpoint

The pressure on IT teams has never been greater. As data environments grow increasingly complex, resource shortages are emerging as a major obstacle for IT leaders striving to meet the demands of modern infrastructure management ... According to DataStrike's newly released 2025 Data Infrastructure Survey Report, more than half (54%) of IT leaders cite resource limitations as a top challenge, highlighting a growing trend toward outsourcing as a solution ...

Image
Datastrike

Gartner revealed its top strategic predictions for 2025 and beyond. Gartner's top predictions explore how generative AI (GenAI) is affecting areas where most would assume only humans can have lasting impact ...