A 1 TB copy and a restore that took several days

The client arrived with a copy that formally existed. Size: 1 TB. The job ran, retention was set, the panel showed success. On the day of the outage the only restore path was the whole disk. The few files without which the application does not start sat in the same set as media, logs and archive.

What the client brought

The production service had one backup policy: a copy of the machine or the volume, in full. On data loss or host damage the procedure sounded the same. Start a restore of the full image. Wait until 1 TB has passed over the network and through verification. Then start the system and check whether the application comes up.

What the service actually needed

The first request did not need the whole terabyte. Config, a database dump or the engine data directory, keys and the application directory without which the process does not start were enough. The rest could come back later: history, media, logs, archives.

The team knew this from practice. It did not have a list, a runbook or a separate job. Single-file restore was not in the procedure. An exercise, if it existed at all, meant restoring everything on another host.

How long it took

Transfer, chunk verification and bringing the machine up took several days. RTO in the contract counted time until the system is up. In practice the clock ran from the start of a full restore. The service stayed down because the procedure could not pull a few megabytes out of a terabyte.

Why a full disk blocks RTO

A 1 TB copy makes sense. It is history, a full rebuild, material for audit. It is not a start path. Network, I/O queue and verification scale with volume. Restore time follows the terabyte, even when megabytes are enough for traffic.

One copy, one path

When there is one job and one path, every outage looks the same. Restore everything, then check the service. A logical database dump, if it exists at all, sits in the same image and waits with the rest of the set.

Without a start set nobody is entitled to say which files are a condition of traffic. Then the only safe decision is: restore the whole thing. That is correct for completeness. It is expensive for RTO.

How we arranged it

We started with a list, not with new storage. Together with the client we wrote down a start set: the files and datasets without which the service does not take traffic. A short list, with an owner and a place in the copy. The full 1 TB image stays. It stops being the only path.

Two jobs, two times

A logical database dump and a copy of the start set went on a separate job, with a tighter RPO. Single-file restore is in the procedure. We mount the copy or pull a file from the backup server, for example Proxmox Backup Server, without restoring the whole machine.

The full disk has its own runbook. It is for rebuild and for audit, not for the first start. Once the service takes traffic, the rest of the set can return in the background.

An exercise that measures the right number

On the calendar we exercise the start set, not 1 TB. We measure time until the service takes traffic. Separately we record time until the whole set is in place. Those are two numbers. Mixing them in one RTO was the source of several days of waiting.

Result

The service comes up from a few files. The full 1 TB copy returns afterwards, once start is done. Start RTO is counted in hours, not in days waiting for the whole disk. The same layout, file restore and full-machine restore on the calendar, is described in the data security service.

Summary

A large copy is not a mistake. The mistake is a single restore path when the service only needs a short list of files to run. Writing down the start set, a separate job and an exercise of that path change RTO without throwing away a terabyte of history.