A tested system and a proven system are different things, and most companies say covered when they mean tested.
Built without a budget
Building a disaster recovery site was one of the main reasons I was hired, and it became my first major responsibility. There was no budget for it.
I had money for machines. I had nothing for people, and I could not hire anyone to do the work.
So I sent the people I already had out to the site periodically to build a piece of it. A trip here, a trip there, across years.
There was never a project. Never a headcount request, never an approval for the thing as a whole, never a line item anyone could point at and cut.
It existed because somebody kept sending people out there.
That is how resilience usually gets built
The vendor version is a program with a start date, a budget, and a completion milestone. The version that happens is a person who decides it matters and keeps chipping at it.
No single trip needs approval, so no single trip gets refused.
That approach depends entirely on one person continuing to care. It also survives budget cycles, because there is nothing on the budget to cut.
Forty miles was the limit, not a preference
The site sat about forty miles from the main computer room. That distance was as far as we could go and still keep the systems copying properly.
Connectivity started as a few circuits, became a faster one, and eventually became something much faster. Bandwidth was the binding constraint on how good the copy could be.
Today distance is nearly free. A company can put a copy of its data anywhere, and because the tradeoff disappeared, almost nobody reasons about it. When a constraint is forced, people think about it. When it vanishes, they stop.
Twenty desks
Ours had a duplicate machine running, receiving data, building logs. That part gets all the attention and it is the part everyone plans.
It also had laptops, computers, and a full twenty-desk setup, ready to go.
A disaster that takes out a computer room usually takes out the building the computer room is in, and the people who run the business need somewhere to be. A recovery plan that restores the systems and leaves forty people with nowhere to sit has solved half the problem.
We tested it. We never failed over live.
The testing was real. Systems were exercised, the copy was verified, the equipment was checked. That work was not theater and it caught things.
What never happened was the business running on it. No day where the main site was gone and the company operated from forty miles away.
My word for that is fortunately, and I mean it.
Testing establishes that the parts work under known conditions. A live cutover establishes that the whole thing carries real load, real users, and real decisions made under pressure by people who have never done it before. Only the second is proof, and almost nobody has it.
The version that works for a smaller company
A genuine failover test means taking the business down deliberately, accepting that something might not come back, and having somebody authorize a self-inflicted outage on a normal Tuesday.
The narrower version: fail over one system rather than everything, during a quiet period, and have the people who would run the recovery do it rather than the person who designed it.
That last detail matters most. A recovery executed by its designer proves the design. A recovery executed by whoever is on duty proves the documentation, and the documentation is what you will have.
Telling a board the truth about readiness
We have a recovery site and we test it regularly is accurate and honest. We are covered implies something stronger.
Saying it plainly costs nothing and changes the conversation. A board told that the parts are tested and the whole has never been exercised will usually ask what a full test would take, which is exactly the question you want asked.
The full article is on my site: https://thewritingking.com/tested-is-not-proven/. It runs longer, carries the diagrams, and answers the questions readers ask most.
These are AI-made summaries of longer articles on my site, written with AI assistance from my own interviews and my own career. Nothing goes out that I have not read and approved.