Flaky is a lazy word

"Flaky" sounds like a property of the test, as if some code were born nervous. Mostly it means the test read something nobody wrote down: the hour, the machine it ran on, the test that ran before it. And the run that failed kept no record of any of it.

Google defines a flaky result as a test that shows both a passing and a failing result with the same code. In that 2016 post, John Micco wrote that about 1.5% of test runs at Google reported a flaky result, almost 16% of tests had some flakiness, and about 84% of the pass-to-fail transitions in their CI, the system that runs the tests on every change, involved a flaky test.

Most times a test went red, somebody had to stop and work out whether the red was real.

The same post has the most honest sentence I've read about CI: "It is human nature to ignore alarms when there is a history of false signals coming from a system."

That's what the word buys you. Permission to ignore the alarm.

Same code, different inputs

The trouble with "same code" is that a test reads far more than its code. It reads the clock and the machine's time zone. It reads where that machine sits, how busy it is, what the previous test left in the database, and what the random number generator felt like that morning.

None of that is in the change under review, and most of it never reaches the CI log. The log tells you which commit ran. It rarely tells you when, where, or after what.

The research agrees. A 2014 study went through 201 commits that appeared to fix flaky tests in 51 open-source projects and sorted them by cause. In Lamyaa Eloussi's summary of the paper she co-wrote, the top three were async wait at 45%, concurrency at 20% and test order dependency at 12%.

Async wait is a test that starts something in the background and doesn't wait for it properly, sometimes with a hopeful sleep. Concurrency is two parts of the program racing each other. Order dependency is a test that only passes if some other test ran before it, or didn't.

None of those is bad luck. Each one is code reacting to an input you didn't control: how long something took, which side of the race won, what ran first. That's a function with a parameter you forgot to write down, and you can find a forgotten parameter.

The clock

At 4:00 PM Pacific time on February 28, 2012, a bug went off inside Windows Azure. Microsoft's write-up says a component called the Guest Agent set certificates to expire a year out by "simply taking the current date and adding one to its year." In UTC, the one clock servers agree on, it was already midnight on February 29. One year later was February 29, 2013, a date that doesn't exist, and certificate creation failed.

Look at the hour. In Seattle it was late afternoon on the 28th. The calendar had flipped somewhere else.

Say your test checks that an order placed today shows "Arrives tomorrow". The server works out "today" in UTC; the test works it out in local time. In New York in winter, the UTC date changes at 7 pm. So the test passes all day and fails for whoever runs it after dinner. They hit re-run the next morning, it goes green, and someone files it under flaky.

Nothing flaky about it. The test checks what time it is in London.

Place works the same way. Move a test runner closer to what it drives and a timing assumption can break overnight; The slowness was load-bearing covers the mechanism. Order too: a test that passes because the test before it happened to create a user is passing on a gift.

The re-run button

Every CI system I've used has a re-run button. It's the most dangerous button in the building, because it's the cheapest one.

Re-run is a fine way to ask a question: does this fail again? It's a terrible way to answer one. When the retry goes green, the conversation ends, and whatever made the first run red goes with it.

Google built the institutional version and wrote down the side effects. Micco describes marking a test as flaky so that it reports a failure only after three in a row, which "encourages developers to ignore flakiness in their own tests." He describes automatic quarantine, which stops a flaky test from blocking anyone's work and "could easily mask a real race condition or some other bug in the code being tested."

Modern tools add a quieter problem. Playwright, a popular browser testing framework, files a test that "failed on the first run, but passed when retried" under a status it calls flaky. The config file it generates for a new project records a trace on the first retry, "but not on the first run." A trace is the full recording of a run: every click, every request, every screen.

So when the retry passes, the one run you have a full recording of is the run that worked.

That's a security camera that switches on after the burglary and films an empty room.

The fix is one setting. Playwright's retain-on-failure mode records every run and keeps the ones that fail; the docs say "a failed run's trace is kept even when a later retry passes." Turn it on. Disk is cheaper than your afternoon.

Flake insurance

Self-healing test tools take the re-run button and put it inside the test.

Healenium is an open-source one for Selenium, the old browser-automation workhorse. Its docs describe the loop: when the test can't find a button, Healenium catches the error, compares the page with the last version where the button was found, and carries on with the closest match. To its credit, it writes a report afterwards, with a screenshot. The run is still green.

I call this flake insurance. You pay a premium on every step so that nobody has to read a trace.

mabl's docs say it won't attempt its advanced auto-heal until a test has passed in a plan at least five times, and when the best match is low confidence, the step fails instead of healing. That's a healing vendor telling you that healing without a history is a guess.

Here's what the guess costs. Imagine a suite that tests an online shop's checkout. On Monday the run times out at "Apply coupon". On Tuesday it's "Save address". On Wednesday, "Place order". Three runs, three step names, and it looks like three flaky buttons.

Put the three red runs side by side and a shape appears. Every click goes through one shared helper. The helper waits a fixed two seconds for a button to appear, and on a busy CI machine some button, any button, takes longer to draw. Whichever one is slow that day is the step that fails. One bug, in one file.

Now give the same suite a healer. It can't find the button, goes looking for the closest match, and by the time it has scored the page, the button is there. It clicks and gets through. You get three green runs, three entries in a report nobody opens, and the same bug. And you never learn what the failures were telling you: on a loaded machine, your app can take more than two seconds to put a button on the screen.

A red step leaves a step name and a screenshot. A healed step leaves a green tick.

It feels like diligence. It's laziness, billed per step.

A healer has a place: a page nobody has mapped yet, or a demo. Even there, keep it off by default and make every heal loud. A heal is a failure with a good excuse, and it belongs in front of a person, not in a report.

Write the inputs down

The cure for "flaky" is boring. Treat the hidden inputs as inputs.

Log them with every run: the time and the machine's time zone, the region and host, the random seed (the number a shuffle starts from), the order the tests ran in, and how many attempts it took. pytest-randomly, a plugin for Python's test runner, shuffles your tests and prints the seed so you can replay the order that failed. Copy that pattern: whatever adds randomness hands you the number that replays it.

Pin what you can. Fake the clock in any test that cares about dates. Set the time zone on purpose; Playwright takes a timezoneId. Then run the suite once in a zone that isn't yours and watch what falls out.

Quarantine a test only with an owner and a date. A quarantined test with no owner is a deleted test that still burns CPU.

Put your timeouts in one place. When every wait carries its own magic number, the slowest machine in the fleet decides which test is flaky today.

And stop saying the word in standups. "Flaky" ends the conversation. "Fails after 7 pm in New York" starts one.

"Flaky" is the word for a question nobody asked.