Backups you have never restored are not backups

Every infrastructure review we run includes the same question, and the range of answers tells us more about an organisation than any diagram: when did you last restore from backup, and how long did it take?

The most common answer is that the job runs nightly and reports success. The second most common is a specific date, several years ago, usually in a lab.

Neither is a recovery capability. One is a monitoring signal, the other is an anecdote.

Why backup success and restore success are different properties

A backup job validates that data left the source. It says nothing about whether that data can be reconfigured into a running system, which involves a chain of assumptions the job never tests:

• That the backup is complete. Incremental chains with a missing or corrupted segment are a classic — the job succeeds because it wrote what it had.
• That the restore tooling still works against a backup of this age. Restore formats change, and a backup from two years ago may require a tool version you no longer run.
• That the target environment matches closely enough. Different kernel, different storage layout, different permissions — each is a place where a restore fails, and each is invisible until it is attempted.
• That the credentials needed to decrypt or fetch the backup still exist and are still valid. This one has ended more recovery attempts than any technical fault.

None of these show up in the job output. All of them show up during an actual recovery, which is precisely the moment you cannot afford to discover them.

A method for turning this into a number

The exercise is short and unpleasant and produces the only number that matters.

Pick a backup at random from at least a month ago. Not yesterday’s, not the one you know is good. Old backups are where the assumptions fail.

Restore it to a clean environment — not the original host, and not one that already has the software installed and configured.

Bring the service up against the restored data, and make it serve a real request or run a real query.

Time the whole thing, from “we decided to restore” to “the service is serving”. Include finding the credentials, finding the runbook, and finding the person who knows how the restore works.

Have someone other than the backup author perform it. If only one person can, that is the finding.

That measured duration is your real recovery time objective. Almost every organisation that runs this exercise discovers the number is larger than the one in their documentation — often by a factor that would change a business decision.

The failure modes worth designing against

The backup target is reachable with production credentials. This is how modern ransomware incidents become unrecoverable. The attacker does not need to break encryption; they need to delete the copies, and the same identity that can read production can usually reach the backup storage.

The countermeasure is a copy that cannot be deleted by that identity — object locking, append-only storage, or a genuinely offline copy. The test is not “does it exist” but “what credential would be required to delete it, and where does that credential live”. If the answer is the production identity, you do not have the copy you think you have.

Restoring is slow because it was never optimised. Restores are usually the least-exercised path in the storage system, and storage is optimised for steady-state reads. A full restore can be an order of magnitude slower than the capacity planning assumed, because nobody ever planned capacity for it.

The recovery procedure is undocumented or wrong. A runbook that assumes a tool version you no longer run, or a directory layout that has changed, converts a quick recovery into an improvised one. Runbooks decay, and the only maintenance that detects decay is using them.

Nobody has authority to declare the drill. Recovering into a clean environment requires someone to decide it is worth the effort in a week that is already busy. If that decision has no owner, the drill does not happen, and the capability quietly expires.

What good actually looks like

An organisation with a real recovery capability usually shows these properties:

• A measured restore time, performed within the last quarter, by someone other than the person who wrote the backup job.
• At least one copy that production credentials cannot delete.
• A runbook that has been used recently enough that its decay was detected and corrected.
• A documented RPO, and a restore that was validated against it rather than assumed to meet it.
• A drill scheduled like any other operational commitment, because it is one.

None of this is expensive. The work is mostly attention, and the reason it does not happen is that a green job is a satisfying thing to look at. The job is not lying to you — it is answering a narrower question than the one you are asking it.