When a vendor has been working on your problem for a week without a theory, that is your signal to bring in somebody else.
The layer nobody watches
Monitoring shows software, load, and data volume, so that is where everyone looks first. Those three are visible and everyone knows how to talk about them.
The floor underneath is invisible by comparison. Cables, ports, controllers, links, and the physical arrangement of equipment do not appear on any dashboard, and they can change the behavior of everything above them.
A team can spend a very long time examining the visible layer while the answer sits in the layer nobody is instrumented to see.
The mirror and the forty miles
Our production system copied its disks continuously to an identical system at a disaster recovery site about forty miles away.
The copying ran asynchronously, which means production wrote its data, carried on working, and the copy caught up shortly afterward. Production never waited.
The alternative is synchronous, where every write waits for confirmation that the remote site has it too. Synchronous gives you a perfect copy at every instant. It also means every single write pays the cost of the distance.
Over forty miles, that cost is significant. Asynchronous was the correct design and it worked for years.
A thousand percent slower
Everything went slow at once. Operations that took minutes were taking hours. The business could not function at that speed, and nothing about the software had changed.
The cause was a bad cable.
That cable had pushed the mirror from asynchronous to synchronous. Every write to production was now waiting for a confirmation from a site forty miles away, and the system was doing exactly what its configuration told it to do.
The software was fine. One failed piece of copper at the bottom of the stack had changed how the entire business behaved.
A week with no theory
I called in a consulting firm first. They looked at it, and looked at it, and could not work it out. A week went by with no answer and no hypothesis.
I want to be fair to them. The cause was unusual and it lived in a layer most people do not examine. They were not lazy and I have no complaint about their effort.
But a week of effort with no theory is information. A team that knows a system produces a hypothesis early, tests it, and either narrows or discards it. A team without one produces activity, and activity looks like progress from the outside.
Ask what the current theory is. Ask what has been ruled out. If neither question has an answer after several days, the issue is not effort.
Three hours, no invoice
Somebody recommended a second firm, US Technical Services, so I called them. I told them plainly that I had a disaster, that I was stuck, and that the firm I had already hired was not helping. No positioning, no defending the earlier decision.
They came in with their whole team, about five people at the time. They worked for a few hours. Then they pointed at a cable and said that is the problem, switch it over.
I asked what it would cost. He said nothing.
I asked what he meant, because the answer made no sense to me. He said that when your neighbor’s house is on fire, you come and put the fire out. You do not charge them for it. You just put out the fire.
He became my consultant of choice permanently. That firm went on to run our private cloud, operate the computer room, take the overnight calls, and eventually rebuild the entire room to triple redundancy without taking anything down.
Years of work came out of a decision not to send an invoice for three hours.
Procurement will not find these
Every company has a list of approved suppliers selected on price, references, and a scoring matrix. That process produces adequate vendors, because it is built to avoid bad outcomes rather than to find exceptional ones.
Ask your network who they call when something has gone badly wrong. That question gets a different answer than asking who they use.
The full article is on my site: https://thewritingking.com/one-bad-cable/. It runs longer, carries the diagrams, and answers the questions readers ask most.
These are AI-made summaries of longer articles on my site, written with AI assistance from my own interviews and my own career. Nothing goes out that I have not read and approved.