Every host says they take backups. Very few say when they last restored one. The gap between those two claims is where data loss lives, because backup pipelines fail silently in ways that only a restore reveals: truncated dumps that exit zero, uploads directories excluded by a stale ignore rule, databases backed up mid-transaction.
The hourly drill
Every hour we pick a random account, restore its most recent backup into an isolated environment, boot it, request its homepage, and diff the file tree and row counts against production. It costs us a modest amount of compute and it is the single highest-value thing we do.
$ srvd backup drill --randomselected acct_31882 · snapshot 2026-04-27T04:00Zrestore 14.2 GB in 3m 41sboot ✓ HTTP 200 in 1.9sfiles ✓ 41,208 / 41,208 matchdatabase ✓ 26 tables · 812,441 rows matchdrill PASS · 4m 06s totalThe nine failures it caught
- Three cases of a mysqldump exceeding a container memory limit and producing a truncated file with exit code 0.
- Two cases of an .rsync-filter rule left behind by an engineer excluding wp-content/uploads for a subset of accounts.
- Two cases of a database backed up without --single-transaction, producing a restore with referentially inconsistent rows.
- One case of a snapshot succeeding but landing in the wrong region bucket, so it existed but was not where the restore path looked.
- One case of an encryption key rotation that made snapshots older than 40 days unreadable. That one was found nine hours after the rotation.
“Every one of those nine would have been discovered by a customer, on the worst day of their year, if we had not been drilling.”
Your own restores are self-service and take about the same time as ours: pick a snapshot, restore to a staging URL first if you want to check it, then promote. We keep 30 days on shared plans and 90 on VPS and above, replicated to a second region in a different jurisdiction.