8 min read ·
The slowness was load-bearing
Make a system faster and it can break. Fast was never the problem. The slowness was doing a job that nobody wrote down.
I call it load-bearing latency. Latency is the delay between asking for something and getting it. Load-bearing is the wall you can't knock down without the ceiling coming with it. Put the two together and you get a delay that nobody designed, nobody measured, and everybody depended on.
It's a folded napkin under a wobbly table. It isn't on the blueprint. Then someone tidies up, throws the napkin away, and the coffee ends up on the floor.
My favourite example is older than a lot of the people who will read this.
Programs built with Borland's Turbo Pascal 7 often used a library called CRT for the screen and keyboard. When such a program started, CRT timed how many trips a tiny empty loop could make in 55 milliseconds, so that its Delay procedure would know how long a millisecond was on your machine. It divided that count by 55 and kept the answer in a 16-bit variable. CelesTrak's write-up of the bug points out the cruel part: the calibration ran in every program that used CRT, whether or not it ever called Delay.
A 16-bit number tops out at 65,535. On a fast enough processor (the write-up's example is a 233 MHz Pentium II) the answer blew past it, and the program died before it drew a single character, with "Runtime Error 200", a division by zero.
Nobody changed a line of those programs. They were correct on the machines they were written for. The machines got faster, and it turned out the slowness had been part of the spec all along. Just not the written part.
You don't need a 1990s compiler to hit this. Here is a version you could hit this week.
Say you have a browser test that opens a dropdown menu and clicks "Export". The menu slides open over a few hundred milliseconds. Your test tool clicks the moment it's told to; it doesn't wait for the page to settle. It also sits far away from the browser it drives, so every command pays a slow round trip, and by the time the click arrives the menu has long stopped moving. The test passes for a year.
Then someone moves the tool next to the browser to make test runs faster. Now the click lands while the menu is still sliding. It hits the spot where "Export" was a frame ago, which now holds a different item or nothing at all, and the test starts failing one run in five.
Read the test line by line and it's correct: open the menu, click the item. The bug lives in the gap between those two lines, and for a year the network filled that gap for free. Nothing in the code mentions distance.
This is what makes load-bearing latency so nasty. The change that exposes it is usually an improvement. A faster CPU, a closer server, a new cache, a lighter page. Nobody wants to revert an improvement, so the person who shipped it gets a bug report about code they never touched.
I build browser agents, which means I spend a lot of my life watching software click on things that are still moving. I've stopped believing in "it passed yesterday" as evidence of anything.
It cuts the other way too. Slow down one piece of a system and you can expose the same kind of race, from the opposite side.
The biggest public example I know of happened two weeks ago. Just before midnight Pacific time on October 19, 2025, DynamoDB, one of Amazon's managed databases, started failing in AWS's us-east-1 region. Plenty of other AWS services depend on DynamoDB, and the trouble spread through them for hours. AWS's post-event summary names the mechanism, which postmortems rarely do.
DynamoDB keeps its DNS records current with two pieces of automation. DNS is the phone book of the internet: it turns a name like dynamodb.us-east-1.amazonaws.com into the addresses of real servers. A Planner watches the load balancers and keeps producing new plans for what that phone book entry should say. Three Enactors, one in each of three Availability Zones (separate groups of data centres), take the latest plan and apply it.
Then one Enactor got slow. It hit unusually long delays while retrying its updates. Meanwhile the Planner kept writing newer plans, and a second, faster Enactor picked up a much newer one and applied it everywhere.
The slow Enactor finally wrote its plan, a much older one, over the new one on the region's main DynamoDB name.
It had checked first. Before it started, it confirmed that its plan was newer than the one in place, and when it ran, that check was true. By the time it wrote, the check was stale. It had asked "am I up to date?" once, then waited, then acted on the answer as if no time had passed.
Then the fast Enactor's cleanup ran. Its job was to delete plans that were many generations old, and the old plan was now the live one. Deleting it removed every address behind that name. The name resolved to nothing.
The summary also says the system was left in a state that blocked further automatic updates, so engineers had to repair it by hand. AWS calls it a "latent race condition", and latent is the word to hold on to. The gap between checking and writing was always there. As long as the Enactors ran at roughly the usual speed, the gap was too short for anything to fit inside it. The delay didn't create the bug. It opened the door the bug had been standing behind. AWS turned that automation off worldwide and says it will fix the race and add protections before turning it back on.
The Turbo Pascal bug and the DynamoDB bug have the same shape. A system depended on how fast one part ran compared with another. Change either side, faster or slower, and the hidden contract breaks.
Hyrum Wright put the general rule better than anyone. Hyrum's Law: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody."
Timing is an observable behaviour. You never promised that a request takes 200 milliseconds, or that one job always finishes before another starts. Somebody depends on it anyway. Often that somebody is you, a year ago.
So how do you find the napkins before someone throws them away?
Change the speed on purpose, in both directions. Chrome DevTools lets you build custom network profiles with your own latency, and it can throttle the CPU. Toxiproxy, from Shopify, sits between your service and something it depends on and adds latency that varies from one call to the next. Run your tests slow. Then run them fast. A suite that only passes at one speed is telling you about a contract you never wrote.
Wait for conditions, not durations. Playwright, Microsoft's browser-testing tool, gets this right. Before it clicks, it checks that the element is visible, enabled, receiving events and stable, and "stable" means it "has maintained the same bounding box for at least two consecutive animation frames". That is the sliding menu, handled. Its own docs on a fixed wait are blunt: "Never wait for timeout in production. Tests that wait for time are inherently flaky."
Every sleep() is a guess about someone else's speed. Replace the guess with the thing you were actually waiting for.
Make the check and the act one step. If you check that something is true and then act on it, anything can happen in between, and on a bad day a lot will. Ask the database to do both at once: write this only if what's there is still the version I saw. Databases call this a conditional write. A write like that fails loudly instead of winning quietly. The ordering problem underneath is old, and there's no clean "now" to lean on once more than one machine is involved.
When you stretch a wait, never repeat the action. When something is slow, the tempting fix is to try it again. On a button that submits a form, a second click is a second submit. Give the page longer. Don't fire twice. A retry is a write, and it should be designed like one.
None of this is exotic. It's writing down the timing you were already relying on, in code, where the next person can see it. People will call the failures flaky. I call them the system telling you what it depended on, at exactly the moment it stopped getting it.
When it breaks, the error will point at a line of code, because that's all an error can point at. The cause will be a distance, or a clock, or a machine that got faster.
A stack trace has no field for a place or an hour.