Flaky Tests: What They Are, Why They Happen, How to Fix

A test goes red in CI. You rerun the job without changing a line, and it goes green. Most teams shrug and move on, but that shrug is how flaky tests take over a test suite.

A flaky test is a test that passes and fails on the same code, with nothing changed. Its result depends on something outside the code under test, such as timing, test order or the environment. Because a flaky failure can’t be trusted, flaky tests teach teams to ignore red builds.

What is a flaky test?

The working definition comes from Google: a flaky result is one where a test “exhibits both a passing and a failing result with the same code.” That is the whole meaning of the term. The test isn’t wrong every time, and it isn’t right every time. It is unreliable.

That definition matters because it rules out two things people often lump together with flakiness:

A test that always fails. That is a broken test or a real bug. It is easy to find.
A test that fails after a code change. That is the test doing its job.

A flaky test sits between the two. It fails sometimes, on code that hasn’t changed, and passes when you try again.

Flaky test vs. a real failure

The hard part is that a single red result looks the same either way. You only learn which kind it was by running the test again, or by looking at its history. If the same test, on the same commit, has both passed and failed, you are looking at flakiness. If it fails every time since a specific commit, you are looking at a regression.

Flaky test vs. real failure: random passes and fails on one commit, versus failing every time after one commit — A flaky test changes result with no code change. A real regression starts failing at one commit and keeps failing.

Why flaky tests are worse than they look

One flaky test is an annoyance. Hundreds of them change how a team works.

Google’s numbers show the scale. In a 2016 post on its Testing Blog, Google reported that about 1.5% of all its test runs returned a flaky result, and that almost 16% of its tests had some level of flakiness. Worse, about 84% of the pass-to-fail transitions its CI observed involved a flaky test. Most “new” failures were noise.

GitHub saw the same pattern in its main application. In 2020 it reported that 1 in 11 commits, about 9%, had at least one red build caused by a flaky test.

The real cost is trust. When most failures turn out to be noise, people start dismissing red builds by reflex, including the ones that point to real bugs. Google called this out directly: developers sometimes dismiss a failure as flaky, only to find out later it was a legitimate bug.

What causes flaky tests

The most cited study on the subject, Luo et al. (FSE 2014), examined 201 commits that fixed flaky tests across 51 open-source projects. Of the 161 commits they sorted by root cause, three causes stood out:

Async wait: 74 commits (45%). The test starts something asynchronous and checks the result before it is ready.
Concurrency: 32 commits (20%). Threads or processes interact in an order the test didn’t expect.
Test order dependency: 19 commits (12%). The test passes or fails depending on which tests ran before it.

Together, those three made up 77% of the commits they sorted.

Bar chart of what causes flaky tests: async wait 45%, concurrency 20%, test order dependency 12%, other causes 22% — Root causes of 161 flaky-test fixes in Luo et al. (FSE 2014).

Later work, including a 2022 survey of 76 papers on flaky tests by Parry et al., describes the same broad picture, with environment and infrastructure problems added on top.

For browser tests written with Playwright or Cypress, these causes usually show up in a handful of recognizable ways:

Symptom	Likely cause	Typical fix
Fails only in CI, passes locally	Slower CI machines expose a timing gap	Wait for a condition, not for a fixed time
Fails when run with the full suite, passes alone	Test order dependency or shared state	Give each test its own data and a fresh session
Fails more often with more workers	Concurrency, tests competing for the same account or record	Isolate test data per worker
Fails around midnight, month end or in other time zones	Time dependency	Freeze or control the clock in the test
Fails with timeouts on network calls	A real third-party service or slow backend	Mock or stub the external call
Fails after an element “moves” or re-renders	The test grabs an element before the page settles	Use locators and assertions that retry

Flaky test detection: how to find them

You can’t fix what you can’t see. Flaky test detection comes down to running tests more than once and keeping a record of what happened.

Rerun failures and label passes on retry

Most modern test runners can retry a failed test automatically. Playwright sorts every result into three groups: passed (passed on the first run), flaky (failed first, then passed on a retry) and failed (failed every attempt). By default, failing tests are not retried, so you have to turn this on:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    // Keep a trace of the retry so the flaky failure leaves evidence.
    trace: 'on-first-retry',
  },
});

Here is what that looks like in practice. We ran a small checkout suite with one test that waits a fixed second for a response that takes between 0.4 and 1.6 seconds. Over several runs, the test sometimes failed on its first attempt and passed on the retry, and Playwright’s HTML report labelled it flaky:

Playwright HTML report with one flaky test, saves a card, next to three passed tests — The Playwright HTML report counts the test as flaky, not passed or failed.

Opening the test shows both attempts. The first run failed because the status text was still empty after the fixed wait. The retry passed:

Playwright report for a flaky test: the first run failed on a fixed waitForTimeout, retry number 1 passed — Run 1 failed, Retry #1 passed. The error points straight at the fixed wait.

Cypress has the same idea with its retries setting, which takes separate values for cypress run and cypress open:

// cypress.config.js
const { defineConfig } = require('cypress');

module.exports = defineConfig({
  retries: { runMode: 2, openMode: 0 },
});

Retries are a detection tool, not a fix. A test that passes on retry is still flaky. The retry just gives you the label.

Repeat a suspect test on purpose

When you suspect a test, run it many times in a row. A test that passes once can still fail on the seventh run:

npx playwright test checkout.spec.ts --repeat-each=20 --workers=4

Running with several workers matters. Some flakiness only appears when tests run in parallel and compete for the same data.

Track a flake rate over time

A single run tells you little. The useful number is a test’s flake rate: how often it produced a flaky result out of all its runs over a period, for example the last 30 days. Tracking it per test lets you:

rank tests by how much noise they cause,
notice when a stable test suddenly becomes flaky, and
check that a fix actually worked.

GitHub’s experience shows why this pays off. Its earlier detection approaches caught only 25% of flaky failures. A retry-based system that recorded results identified about 90% of them, and once flaky tests were tracked and assigned, commits with flaky builds dropped from about 1 in 11 to less than half a percent, or 1 in 200 commits.

How to fix flaky tests

Fixes depend on the cause, which is why the table above is worth keeping close. A few changes solve most browser-test flakiness.

Replace fixed waits with conditions

A fixed wait is a guess about how long something takes. On a slow CI machine, the guess is wrong.

// Flaky: hopes two seconds is enough.
await page.getByRole('button', { name: 'Save' }).click();
await page.waitForTimeout(2000);
expect(await page.getByText('Saved').isVisible()).toBe(true);

// Stable: waits until the text appears, up to the assertion timeout.
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByText('Saved')).toBeVisible();

In Cypress, the same idea means waiting on the request instead of a number of milliseconds:

cy.intercept('POST', '/api/settings').as('save');
cy.contains('button', 'Save').click();
cy.wait('@save');
cy.contains('Saved').should('be.visible');

Isolate state between tests

Each test should create the data it needs and not rely on what an earlier test left behind. Playwright already gives every test a fresh browser context, so cookies and local storage don’t leak between tests. Shared accounts, database rows and files are still yours to isolate. A common pattern is to create a unique user or record per test, or per worker.

Control time and the network

If a test depends on today’s date, freeze the clock. Playwright’s clock API lets you set a fixed time before the page loads. If a test calls a real third-party service, mock it, and keep a smaller set of contract tests that check the real service separately.

A flaky testing policy that holds up

Fixing tests one at a time is not enough if new flaky tests keep arriving. Google noted that its rate of new flaky tests roughly matched its fix rate. A team needs a few simple rules.

Quarantine, but with a limit. Martin Fowler recommends moving non-deterministic tests out of the main suite so they stop blocking everyone, then fixing them quickly. He suggests a cap, such as no more than 8 tests in quarantine, or a time limit, such as a week.
Make flakiness visible in CI. Since version 1.52, Playwright can exit with an error when any test is marked flaky, using failOnFlakyTests in the config or --fail-on-flaky-tests on the command line. Turning it on for the main branch stops flaky passes from going unnoticed.
Give every flaky test an owner. GitHub’s system scored each flaky test by impact and automatically opened an issue for the people who most recently changed the test or the related code.

Flaky test automation: what to automate

Most of the work above can run without anyone thinking about it. Flaky test automation usually means four things:

Retries in CI that label pass-on-retry results as flaky instead of hiding them.
A stored history of every run, so you can calculate flake rates per test, per branch and per environment.
Alerts or issues when a test crosses a flake-rate threshold.
Context on each result, such as browser, worker, operating system and test data, so you can see what the flaky runs have in common.

Built-in reports from Playwright and Cypress cover the first point well. The other three need results from many runs kept in one place, which is what test analytics tools are for.

FAQ

What does flaky test mean?

A flaky test is one that passes and fails on the same code without any change. The result depends on something outside the code, such as timing or test order.

Are flaky tests a real problem or just noise?

They are a real problem. At Google, about 84% of observed pass-to-fail transitions involved a flaky test, which made real failures easy to miss.

Should I just add retries to fix flaky tests?

No. Retries help you detect flaky tests and keep pipelines moving, but they don’t remove the cause. Treat a pass on retry as a signal to fix the test.

What is the most common cause of flaky tests?

Waiting incorrectly for asynchronous work. In the Luo et al. study, async wait issues made up 45% of the flaky-test fixes they sorted by cause.

How do I detect flaky tests in Playwright?

Turn on retries in playwright.config.ts. Playwright then reports tests that fail first and pass on a retry as flaky. Use --repeat-each to stress a suspect test.

Conclusion

Flaky tests are tests you can’t trust, and an untrusted test is worse than no test because it trains people to ignore failures. Most flakiness comes from a small set of causes: async waits, concurrency and test order. Detect it with retries and a flake rate per test, fix it at the cause, and keep a short quarantine with an owner for every test in it. The goal isn’t zero flaky tests forever. It is noticing them fast and fixing them before the team stops looking.