Flaky Tests

The 30-second version

Here is a flaky test you can run yourself. The test clicks Save, waits, then checks that a "Saved" message appeared. The save is like a real network call: usually quick, occasionally slow. Run it on your laptop, then on a busy CICI (continuous integration)A server that builds your code and runs its tests automatically on every push (GitHub Actions, GitLab CI, …), so mistakes are caught minutes after they're made. A flaky test there blocks everyone's merges. machine (the shared server that runs your tests on every push), and then change how the test waits:

Three things to notice:

1. On the laptop, sleep(100 ms) looks fine. The test passes, you commit it, and nobody suspects anything.
2. On the busy machine, the same test goes red on the runs where the save happened to be slow. Nothing in the code changed.
3. A longer sleep isn't a fix. It makes every run slower and still loses on the slowest saves. Waiting for the "Saved" message itself is always green, and faster than any sleep long enough to be green: each run waits exactly as long as that save took, no more.

Why one flaky test is a big deal

A build is green only if every test passes. 500 tests that each flake once in 1,000 runs make about 39% of builds red for no reason. Why & What to Test §11 has a calculator for that. The worse cost is what people learn: red usually means nothing. They click "re-run" without reading. One day a real bug fails, gets re-run until it passes by luck, and ships.

What's in this topic

• §02 — the hidden inputs a test reads without you noticing.
• §03–§07 — the causes one by one, each with real failing output and the fix: sleeps, shared state and test order, clocks and randomness, concurrency, ports and leftover work.
• §08 — how to make a flaky test fail on purpose, so you can debug it.
• §09–§11 — retries, quarantine and measuring flake rate: what they cost and how to run them without lying to yourself.

Where the randomness comes from

A good test is DeterministicDeterministicGives the same result every time you run it with the same input. A deterministic test never passes on Monday and fails on Tuesday with no code change.: same code in, same result out (Why & What to Test §04). To break that, a test has to read something that isn't the code. Here are the usual suspects:

Every flaky test you will meet reads at least one of these. When you see a flake, the first question is always the same: which hidden input did this test read?

A worked example

Take a test that checks an order total. It's 12 lines, and it looks like it only reads the code. But:

• it calls time.Now() to stamp the order → reads the clock;
• it uses a package-level tax rate that another test also changes → reads test order;
• it starts a goroutine to email the receipt and sleeps 50 ms before checking → reads machine speed.

Three hidden inputs, any of which can turn it red on a bad day. None of them is visible in the assertion.

Sometimes the "flaky test" is a real bug

Don't assume the test is the problem. If the code really has a race, so that two requests at once sometimes lose an update, then the test that fails 1 time in 50 is doing its job. It's showing you an Intermittent bugIntermittent bugA real bug in the product that only shows up on some runs, for example a race that loses an update now and then. It looks exactly like a flaky test, which is why “just retry it” is dangerous. that users will hit too. That's why "just add a retry" is dangerous (§09). Before you fix a flaky test, decide which one it is:

Sleeps: betting on the clock

The test that always passes on your machine

SendAsync sends an email in the background and returns at once. The test can't see when the background work finishes, so it sleeps a little and hopes:

package outbox

import (
    "slices"
    "sync"
)

// Outbox sends emails in the background.
type Outbox struct {
    mu   sync.Mutex // guards sent
    sent []string   // addresses we've sent to
}

func (o *Outbox) SendAsync(to string) {
    go func() {          // runs in the background
        render(to)       // builds the email: ~1.5 ms of work
        o.mu.Lock()
        o.sent = append(o.sent, to)
        o.mu.Unlock()
    }()
}

// Sent returns a copy of the addresses sent so far.
func (o *Outbox) Sent() []string {
    o.mu.Lock()
    defer o.mu.Unlock()
    return slices.Clone(o.sent)
}

package outbox

import (
    "testing"
    "time"
)

func TestSendAsync(t *testing.T) {
    var o Outbox
    o.SendAsync("ana@example.com")
    time.Sleep(5 * time.Millisecond) // ← "surely it's done by now"
    if got := len(o.Sent()); got != 1 {
        t.Fatalf("sent %d emails, want 1", got)
    }
}

We ran both for real, measured on this machine (4 CPU cores):

The Go failure, when it comes:

--- FAIL: TestSendAsync (0.01s)
    outbox_test.go:13: sent 0 emails, want 1

Nothing changed between the green runs and the red ones except how busy the machine was. Even the failure rate under load isn't one number: we measured 30% one evening and 6–9% another, and our reviewer got 5%. That spread is the point. On a quiet machine the email takes about 1.5 ms, well inside the 5 ms sleep. When 8 copies fight over 4 cores, the background goroutine sometimes waits its turn for longer than 5 ms.

How long work takes isn't one number. It's a spread: usually quick, with a Long tailLong tail (of timings)The few runs that take far longer than usual. Most saves take 40 ms, but 1 in 100 takes 300 ms. Averages hide the tail; flaky tests live in it. of slow runs. A sleep picks a cut-off inside that spread, so some runs always land on the wrong side of it. That's exactly what the histogram in §01 showed.

Fix 1: wait for the event itself

The real fix is to make "the work is done" something the test can wait for. In Go, the outbox counts its running sends with a sync.WaitGroup and offers a Wait method. In JS, sendAsync returns its promise. Tests await it. Callers who don't need to wait should still attach .catch(logError): a promise that fails with nobody listening is an Unhandled rejectionUnhandled rejectionA promise failed and no code ever caught the error (no await in try/catch, no .catch). Node ≥ 15 crashes the process when this happens., which crashes Node and fails Vitest runs (§07).

type Outbox struct {
    mu   sync.Mutex
    sent []string
    wg   sync.WaitGroup // counts sends still running
}

func (o *Outbox) SendAsync(to string) {
    o.wg.Add(1)            // one more send in flight
    go func() {
        defer o.wg.Done()  // …and one fewer when this returns
        render(to)
        o.mu.Lock()
        o.sent = append(o.sent, to)
        o.mu.Unlock()
    }()
}

// Wait blocks until every SendAsync so far has finished.
func (o *Outbox) Wait() { o.wg.Wait() }

// in the test:
o.SendAsync("ana@example.com")
o.Wait() // ← wait for the event, not for the clock

Result, measured on this machine: the Go version ran 4,961 times with 8 copies at once and 0 failures; with the Race detectorRace detectorGo's built-in checker (go test -race). It watches every memory access while the program runs and reports two goroutines touching the same memory with nothing ordering them. on (-race), 100 runs, 0 failures. The JS version passed 1,000 of 1,000 in 0.6 s in total.

Fix 2: when you can't get an event, poll with a deadline

Sometimes the code under test really gives you nothing to wait on: a third-party library, a file another process writes. Then check repeatedly, with a deadline. That's PollingPollingChecking a condition again and again, with a short pause in between, until it becomes true or a deadline passes. Slower than being told when something happens, but works when nothing tells you.: the test passes as soon as the condition is true, and fails with a clear message only if it's still false after a generous limit.

// waitFor checks cond every millisecond until it's true,
// or fails the test after 2 s.
func waitFor(t *testing.T, cond func() bool) {
    t.Helper()
    deadline := time.Now().Add(2 * time.Second)
    for !cond() {
        if time.Now().After(deadline) {
            t.Fatal("condition not met within 2s")
        }
        time.Sleep(time.Millisecond)
    }
}

func TestSendAsyncPolling(t *testing.T) {
    var o Outbox
    o.SendAsync("ana@example.com")
    waitFor(t, func() bool { return len(o.Sent()) == 1 })
}

Polling is the fallback, not the first choice. Notice the slight sleep inside the loop: the deadline is generous (2 s) because it's only reached on failure, and the check interval is small because it's paid on every run. Get the interval wrong and polling gets slow: our 1,000 vi.waitFor runs took 51 s, against 0.6 s for awaiting the promise. The first check almost always came too early, so nearly every run paid one full 50 ms interval. Pass { interval: 5 } if you poll something that's usually quick.

The same bug in the browser

End-to-end tests are where sleeps breed. Here's a page whose "Saved" message appears 50–400 ms after a click, tested two ways with PlaywrightPlaywrightA library that drives real browsers (Chromium, Firefox, WebKit) from code, used for end-to-end tests and automation.:

test("shows Saved (sleep)", async ({ page }) => {
  await page.getByRole("button", { name: "Save" }).click();
  await page.waitForTimeout(200); // ← a guess
  expect(await page.getByRole("status").textContent()).toBe("Saved");
});

test("shows Saved (web-first assertion)", async ({ page }) => {
  await page.getByRole("button", { name: "Save" }).click();
  await expect(page.getByRole("status")).toHaveText("Saved"); // ← re-checks until true, up to 5 s
});

With --repeat-each=20 (run each test 20 times): the sleep version failed 9 of 20, with Expected: "Saved" and Received: "". The second version passed all 20. await expect(locator).toHaveText(…) is a Web-first assertionWeb-first assertionAn assertion like await expect(locator).toHaveText('Saved') that keeps re-checking the live page until it passes or a timeout (5 s by default) runs out. The opposite of reading a value once and checking it., so the polling loop above is built in. Reading a value once with textContent() and then asserting on it throws that away. End-to-End Testing covers locators and auto-waiting in depth.

Shared state and test order

A polluter in three tests

taxRate is a package-level variable: one copy, shared by every test in the package. One test changes it and never changes it back:

var taxRate = 0.10 // package-level: shared by every test

func SetTaxRate(r float64) { taxRate = r }

// Total adds tax to a price in cents.
func Total(cents int) int { return cents + int(float64(cents)*taxRate) }

func TestTotal(t *testing.T) {
    if got := Total(1000); got != 1100 {
        t.Errorf("Total(1000) = %d, want 1100", got)
    }
}

func TestTotalGermany(t *testing.T) {
    SetTaxRate(0.19) // ← changes shared state, never puts it back
    if got := Total(1000); got != 1190 {
        t.Errorf("Total(1000) = %d, want 1190", got)
    }
}

func TestTotalZero(t *testing.T) { /* Total(0) == 0 */ }

Run normally, all three pass, every time. Go and Vitest run tests in the order they appear in the file, and TestTotal happens to come before the polluter. The bug is real but hidden. It only shows up when someone adds a test below TestTotalGermany that expects 10%, or when tests run in another order.

Make the order random on purpose

Both tools can shuffleShuffled test orderRunning tests in a random order each time (go test -shuffle=on, vitest --sequence.shuffle) to expose order-dependent tests. The tool prints the seed it used, so you can replay exactly that order. the order. That's the single best way to flush out Order-dependent testOrder-dependent testA test whose result depends on which tests ran before it, because they share some state (a global variable, a database row, a file, an environment variable). tests. Two flags appear in the commands below: -count=1 turns off Go's test result cacheTest cache (Go)go test remembers results of packages whose code, tests and inputs haven't changed, and prints (cached) instead of running them again. -count=1 turns the cache off for that run. so the tests really run (Unit Testing §10 explains the cache), and -v (verbose) prints every test's name as it runs, so you can see the order. We ran the Go version six times with -shuffle=on. Four runs failed. Here's one:

$ go test -count=1 -shuffle=on -v ./pricing
-test.shuffle 1793396763043929742
=== RUN   TestTotalZero
--- PASS: TestTotalZero (0.00s)
=== RUN   TestTotalGermany
--- PASS: TestTotalGermany (0.00s)
=== RUN   TestTotal
    pricing_test.go:7: Total(1000) = 1190, want 1100
--- FAIL: TestTotal (0.00s)
FAIL

The first line is the seedSeedThe starting number for a pseudo-random generator. The same seed always produces the same “random” sequence, so printing the seed lets you replay a random run exactly.: the number the shuffle started from. Pass it back and you get exactly the same order, so the failure becomes repeatable:

$ go test -count=1 -shuffle=1793396763043929742 ./pricing
-test.shuffle 1793396763043929742
--- FAIL: TestTotal (0.00s)
    pricing_test.go:7: Total(1000) = 1190, want 1100
FAIL

Vitest works the same way (version 5.0.5 here). One of four shuffled runs failed:

$ npx vitest run --sequence.shuffle
      Running tests with seed "1793396793271"
   × adds 10% tax 5ms
AssertionError: expected 1190 to be 1100 // Object.is equality

$ npx vitest run --sequence.shuffle --sequence.seed=1793396793271   # same order again

Now try it yourself. These four tests really run in your browser, against one shared module:

// pricing.js — module-level state, shared by every test in the file
let taxRate = 0.1;
const cart = [];
export const setTaxRate = (r) => { taxRate = r; };
export const total = (cents) => cents + Math.trunc(cents * taxRate);
export const addItem = (item) => cart.push(item);
export const cartSize = () => cart.length;
export const reset = () => { taxRate = 0.1; cart.length = 0; };

// pricing.test.js
it("A: adds 10% tax", () => expect(total(1000)).toBe(1100));
it("B: adds 19% tax in Germany", () => {
  setTaxRate(0.19);                  // never put back
  expect(total(1000)).toBe(1190);
});
it("C: a new cart is empty", () => expect(cartSize()).toBe(0));
it("D: adds a book", () => {
  addItem("book");                   // never removed
  expect(cartSize()).toBe(1);
});

The fix: every test cleans up after itself

The quick fix is to restore what you change. Go's t.Cleanup registers a function that runs when the test ends, pass or fail. Vitest's afterEach (or beforeEach) runs around every test in the file:

// useTaxRate sets the rate for one test and restores it afterwards.
func useTaxRate(t *testing.T, r float64) {
    t.Helper()
    old := taxRate
    SetTaxRate(r)
    t.Cleanup(func() { SetTaxRate(old) }) // ← runs when the test ends
}

func TestTotalGermany(t *testing.T) {
    useTaxRate(t, 0.19)
    if got := Total(1000); got != 1190 {
        t.Errorf("Total(1000) = %d, want 1190", got)
    }
}

With the Go helper in place, 30 shuffled runs in a row passed. t.Cleanup is better than a line at the end of the test because it runs even when the test fails halfway through.

The better fix is to have nothing shared to clean up. If Total took the rate as a parameter, or lived on a Pricer struct that each test creates, the tests couldn't interfere at all. That's Dependency injectionDependency injection (DI)Giving a piece of code the things it needs (a database, a clock, a logger) from the outside — usually as constructor or function arguments — instead of letting it create or look them up itself.; Dependency Injection shows how to get from globals to injected values step by step.

It's not just variables

Anything that outlives a single test is shared state. The usual ones, with the tool that resets each:

Databases are the biggest source in practice. Integration Testing covers per-test transactions and schemas. For mocks that leak, see Mocks, Stubs & Fakes §07.

Clocks, time zones and random data

Time-based flakes have their own signature: the test fails at certain times. At night, at the end of the month, at weekends. Or only on the laptop of the one teammate in another country. Each of the four tests below reads the real clock. Pick when they run and on which machine:

// written on a Tuesday
expect(shipsToday(new Date())).toBe(true);

const d = new Date();
d.setMonth(d.getMonth() + 1); // same day, next month
expect(d.getMonth()).toBe(nextMonthNumber);

const now = new Date();
const key = now.toISOString().slice(0, 10); // the UTC date
expect(key).toBe(localDate(now));           // the local date

const tomorrow = new Date(Date.now() + 24 * 3600 * 1000);
expect(tomorrow.getDate()).toBe(tomorrowsDate);

Run the sweep for each machine. The weekend test fails 29% of the time everywhere. The month-end test fails for 7 days a year. The UTC-date test never fails on the UTC server, but fails 29% of the time in Hanoi, every day between midnight and 07:00. West of UTC it fails in the evening instead: about 18% of the time in New York, from 19:00 or 20:00 until midnight. The +24 h test only fails in New York, on the two nights a year when Daylight saving timeDaylight saving time (DST)Moving clocks forward one hour in spring and back in autumn, as many countries do (but not Vietnam). On those days a calendar day is 23 or 25 hours long, so “+24 hours” isn't always “tomorrow”. changes the clocks.

Real failure 1: "passes on CI, fails on my laptop"

toISOString() always gives the date in UTCUTCCoordinated Universal Time: the world's reference time, with no time zones and no daylight saving. Servers and databases should store and log times in UTC and only convert to local time for display., not the local date. CI servers usually run in UTC, so the bug is invisible there:

const todayKey = (now = new Date()) => now.toISOString().slice(0, 10);

afterEach(() => vi.useRealTimers()); // fake timers are shared state too (§04)

it("todayKey is the date on the calendar", () => {
  vi.useFakeTimers();
  vi.setSystemTime(new Date(2026, 9, 30, 6, 30)); // 30 Oct, 06:30 local time (months count from 0)
  expect(todayKey()).toBe("2026-10-30");
});

$ TZ=UTC npx vitest run                 # the CI server
      Tests  1 passed (1)
$ TZ=Asia/Ho_Chi_Minh npx vitest run    # a laptop in Hanoi
AssertionError: expected '2026-10-29' to be '2026-10-30' // Object.is equality
$ TZ=America/New_York npx vitest run
      Tests  1 passed (1)

At 06:30 in Hanoi it's still 23:30 the previous day in UTC. The TZ environment variableEnvironment variableA named setting the operating system hands a program when it starts, like DATABASE_URL=postgres://… Read in Go with os.Getenv and in Node with process.env. sets the time zone for the process. Running a suite with different TZ values is the quickest way to find these bugs.

Real failure 2: the end-of-month bomb

func TestRenewalIsNextMonth(t *testing.T) {
    now := time.Now() // ← the hidden input
    got := now.AddDate(0, 1, 0) // same day next month
    want := now.Month()%12 + 1
    if got.Month() != want {
        t.Errorf("renewal %s is in %s, want %s", got.Format("2006-01-02"), got.Month(), want)
    }
}

We ran it on 30 October 2026: it passed. On 31 October it fails, because there is no 31 November, so Go rolls over to the 1st of December:

--- FAIL: TestRenewalIsNextMonth (0.00s)
    dates_test.go:13: renewal 2026-12-01 is in December, want November

(To get that output today, we replaced time.Now() with time.Date(2026, time.October, 31, …), which is the fix below anyway.) JavaScript's setMonth rolls over the same way. Notice that this test found a real product question: what should renewal be for someone who signed up on 31 January? The flake was hiding it.

The fix: time is a parameter

Code that needs "now" should receive it, and code that needs a calendar date should be told which time zone it's in. Then tests pass fixed moments, including the nasty ones (31 January, 06:30 in Hanoi, the night the clocks change):

// TodayKey is the date of now in the shop's time zone, e.g. "2026-10-30".
func TodayKey(now time.Time, loc *time.Location) string {
    return now.In(loc).Format("2006-01-02")
}

// test: a fixed moment and an explicit zone, so the result never changes
func TestTodayKeyEarlyMorningInHanoi(t *testing.T) {
    hanoi, err := time.LoadLocation("Asia/Ho_Chi_Minh")
    if err != nil {
        t.Fatal(err)
    }
    now := time.Date(2026, time.October, 29, 23, 30, 0, 0, time.UTC) // 06:30 on the 30th in Hanoi
    if got := TodayKey(now, hanoi); got != "2026-10-30" {
        t.Errorf("TodayKey = %q, want %q", got, "2026-10-30")
    }
}

Both pass with TZ set to UTC, Asia/Ho_Chi_Minh and America/New_York. The machine's zone no longer matters. In production, the caller passes time.Now() / new Date(). How to thread a clock through a bigger program is covered in Dependency Injection §07. For code that waits (timeouts, retries), use a Fake clockFake clockA test double for time: code asks it for “now” instead of the real clock, and the test moves it forward (or backward) by hand. Makes timeouts and expiry testable instantly and deterministically., see Mocks, Stubs & Fakes §09.

Random data: fine, if you can replay it

Random test data is useful: it tries inputs you wouldn't think of. But a random failure you can't reproduce is a flaky test. The rule: choose a seedSeedThe starting number for a pseudo-random generator. The same seed always produces the same “random” sequence, so printing the seed lets you replay a random run exactly., print it, and let the test accept it back.

var seed = flag.Uint64("seed", 0, "random seed (0 = pick one)")

func TestAveragePrice(t *testing.T) {
    s := *seed
    if s == 0 {
        s = uint64(time.Now().UnixNano())
    }
    t.Logf("seed=%d (replay with -seed=%d)", s, s) // ← shown when the test fails
    r := rand.New(rand.NewPCG(s, 0))               // math/rand/v2

    qty := r.IntN(10) // 0..9 — sometimes 0!
    got := AveragePrice(qty*250, qty)
    if got != 250 {
        t.Errorf("AveragePrice(%d, %d) = %d, want 250", qty*250, qty, got)
    }
}

On our machine the Go test failed about 1 run in 10 (when qty came out 0), and the log said exactly how to replay it:

--- FAIL: TestAveragePrice (0.00s)
    cart_test.go:17: seed=1793397134037197166 (replay with -seed=1793397134037197166)
panic: runtime error: integer divide by zero [recovered]

$ go test ./cart -seed=1793397134037197166   # fails the same way, every time

The JS version, replayed: SEED=248324260 npx vitest run → AssertionError: expected NaN to be 250, every time. Here, again, the random test found a real bug: nothing stops a quantity of 0. Keep the random test. Add a fixed test for qty = 0 as a Regression testRegression testA test added after a bug is fixed, so the same bug can't quietly come back., and fix the code. If you want this kind of exploration done properly, that's what FuzzingFuzzingFeeding a function huge numbers of random or mutated inputs to find crashes and edge cases. Go has it built in (go test -fuzz). and property-based testsProperty-based testA test that states a rule that must always hold (e.g. sorting twice equals sorting once) and lets the tool generate hundreds of random inputs to try to break it. are for; see Unit Testing §12.

Concurrency: an order nobody promised

Go randomizes map order on purpose

Ranging over a Go map visits the keys in an order that changes from run to run. That's deliberate: the Go team made it random so nobody could come to depend on it. This test depends on it anyway:

// Names returns the tag names of a post.
func Names(counts map[string]int) []string {
    var out []string
    for name := range counts { // ← map order: random on purpose
        out = append(out, name)
    }
    return out
}

func TestNames(t *testing.T) {
    got := Names(map[string]int{"go": 3, "js": 5, "sql": 1})
    want := []string{"go", "js", "sql"}
    if !slices.Equal(got, want) {
        t.Errorf("Names() = %v, want %v", got, want)
    }
}

$ go test -count=100 -run 'TestNames$' ./tags
    tags_test.go:12: Names() = [js sql go], want [go js sql]
...

30 of 100 runs failed on this machine. Not 5 of 6, as you might expect from 6 possible orders: small maps aren't shuffled evenly, so the "right" order still comes up most of the time. That's exactly what makes this flake sneaky. It passes often enough to get merged.

Ask what Names promised. If nobody needs the names in order, the test is wrong: compare without order. If callers do need them sorted, the code is wrong: sort inside Names and keep the strict test.

func TestNames(t *testing.T) {
    got := Names(map[string]int{"go": 3, "js": 5, "sql": 1})
    slices.Sort(got) // ← the test only cares WHICH names, not their order
    if want := []string{"go", "js", "sql"}; !slices.Equal(got, want) {
        t.Errorf("Names() = %v, want %v", got, want)
    }
}

(JS objects keep their keys in insertion order, so the JS names wouldn't flake on this. JS has its own version of the problem, next.)

JavaScript: finish order isn't start order

Start three file reads at once, and they can finish in any order. This loader pushes each price when its read finishes:

export async function loadPrices(files) {
  const prices = [];
  await Promise.all(
    files.map(async (f) => {
      prices.push(Number(await readFile(f, "utf8"))); // ← pushed in FINISH order
    }),
  );
  return prices;
}

We ran expect(await loadPrices(files)).toEqual([100, 250, 399]) 500 times, three separate times: 54, 116 and 138 runs failed.

AssertionError: expected [ 250, 100, 399 ] to deeply equal [ 100, 250, 399 ]

This time the bug is in the code: whoever calls loadPrices([a, b, c]) surely expects the prices in that order. Promise.all already returns its results in input order, however they finish, so the fix is to use its return value:

export function loadPrices(files) {
  return Promise.all(files.map(async (f) => Number(await readFile(f, "utf8"))));
}

Same 500 × 3 runs: 0 failures.

Real races: the test is right, the code is wrong

The worst concurrency flakes are data racesData raceTwo threads or goroutines access the same memory, at least one of them writes, and nothing orders the two accesses (no lock, channel or atomic between them). The result is unpredictable.: two goroutines changing the same memory with nothing making them take turns, so an update is sometimes lost. The test that fails 1 time in 50 is reporting a real bug. Don't touch the test; fix the code.

Go has a tool that turns "sometimes" into "reliably": the Race detectorRace detectorGo's built-in checker (go test -race). It watches every memory access while the program runs and reports two goroutines touching the same memory with nothing ordering them., go test -race. It watches memory accesses while the tests run and reports two goroutines touching the same memory without a lock, even on runs where the result came out right. Unit Testing §11 shows it catching a racy cache, and Race Conditions explains how it works.

JavaScript runs your code on one thread, so it can't have Go-style data races. It can still have logical races: two awaits interleaving so that one overwrites the other's result. That's the loadPrices bug above in another form. Race Conditions has more.

Ports, leftovers and the network

Two tests, one port

go test ./... runs several packages at the same time, each in its own process. Vitest does the same with test files, using separate workers. Two packages that each start a server on port 8089:

func TestServer(t *testing.T) {
    srv := &http.Server{Addr: "127.0.0.1:8089"} // ← same port in every package
    errc := make(chan error, 1)
    go func() { errc <- srv.ListenAndServe() }()
    select {
    case err := <-errc: // ListenAndServe returned at once: it couldn't start
        t.Fatal(err)
    case <-time.After(300 * time.Millisecond): // pretend to run some requests
    }
    srv.Close()
}

$ go test -count=1 ./ports/...
--- FAIL: TestServer (0.00s)
    server_test.go:15: listen tcp 127.0.0.1:8089: bind: address already in use
FAIL	example.com/shop/ports/orders	0.013s
ok  	example.com/shop/ports/users	0.305s

Whichever package grabbed the port first wins. Run them one at a time (go test ./ports/users) and both pass, which is why this flake "can't be reproduced" locally. Node fails the same way, with listen EADDRINUSE: address already in use 127.0.0.1:8089. (chan, <- and select are Go's channels; see Channels §02. Here they mean "whichever happens first: the server fails to start, or 300 ms pass".)

The fix: listen on Port 0Port 0Asking the operating system to listen on port 0 means “give me any free port”. The server then reads back which port it got. Tests that do this never collide with each other., which asks the operating system for any free port, then read back the port you got:

func TestHealth(t *testing.T) {
    srv := httptest.NewServer(handler) // ← listens on 127.0.0.1:<a free port>
    defer srv.Close()

    resp, err := http.Get(srv.URL + "/health") // srv.URL = "http://127.0.0.1:41873"
    if err != nil {
        t.Fatal(err)
    }
    defer resp.Body.Close()
    // … check resp.StatusCode …
}

The same goes for anything with a fixed name: temp files (t.TempDir(), mkdtemp), database names, Docker container names, queue names. Give every test its own.

Work that outlives its test

A test that starts background work and returns without waiting for it leaves that work running during the next test. When it eventually fails, it fails over there.

func TestStartsWorker(t *testing.T) {
    go func() {
        time.Sleep(10 * time.Millisecond) // background work…
        t.Error("worker failed")          // …that outlives the test
    }()
} // ← returns at once; nobody waits for the goroutine

func TestSomethingSlow(t *testing.T) {
    time.Sleep(50 * time.Millisecond)
}

$ go test -count=1 -v ./leak
=== RUN   TestStartsWorker
--- PASS: TestStartsWorker (0.00s)
=== RUN   TestSomethingSlow
panic: Fail in goroutine after TestStartsWorker has completed

The whole test binary crashes in the middle of TestSomethingSlow. And it's flaky: if TestStartsWorker is the last test, the program exits before the goroutine wakes up, and everything passes. Run alone, it passes.

JavaScript has the same problem with a promise nobody awaits. Vitest catches it, and is honest that it can't tell who's to blame:

⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯
Error: database is closed
 ❯ saveInBackground leakjs/a.test.js:5:9
This error originated in "leakjs/a.test.js" test file. It doesn't mean the error
was thrown inside the file itself, but while it was running.
The last test to run before this error was "something slower". This means either:
- the error was thrown while Vitest was running this test, or
- the error was thrown after the test completed, and this was the most recent test at that point.

"Something slower" never touched the database. The culprit is the earlier test that called saveInBackground() without await.

The fix: a test waits for everything it starts. In Go, use a sync.WaitGroup or a done channel (§03), and wait in the test or in t.Cleanup. In JS, await every promise, and make sure afterEach stops timers, closes servers and database connections. Two guard rails help:

• Vitest already fails the run on an unhandled rejection (that's the Errors 1 error line). Don't turn that off.
• In Go, the third-party package go.uber.org/goleak can check at the end of each test (or package) that no unexpected goroutines are still running. It turns a goroutine that silently outlives its test into a clear failure in the test that started it.

The real network

A test that calls a real API (a payment provider, a weather service, your staging server) reads that service's mood: slow responses, rate limits, outages, changed data. You can't fix that from the test. Unit and integration tests should talk to a fake: httptest.Server in Go, MSW in JS, both shown in Mocks, Stubs & Fakes §08. Check that the fake still matches the real thing with a few separate contract testsContract testAn automated test that checks a service still answers the way its callers expect (same fields, same meanings), so a provider can't break a consumer without CI noticing. (Mocks, Stubs & Fakes §11). Run those on a schedule rather than on every push, so the provider's bad day doesn't block your merges.

Make it fail on purpose

Step 0: read what you already have

Before running anything, look at the failures you've already got. CI history usually answers half the question:

• When did it start failing? Find the first red run, then look at what was merged just before it.
• Where does it fail? Only on CI, only on one runner type, only in the nightly build?
• When in the day does it fail? Clustered at night or at month end points straight at §05.
• With what? Does it fail alone, or only in the full run? Was a seed printed?
• What's the error? A timeout, a wrong value, "address already in use", a panic in a different test? Each points at a different section of this topic.

Step 1: repeat it

Repetition turns a 1% flake into a guaranteed failure: 1,000 runs at 1% fail about 10 times. Run only the suspect test so each run is quick:

# run one test 1,000 times; stop at the first failure
go test -run '^TestSendAsync$' -count=1000 -failfast ./outbox

# the same, with the race detector watching
go test -run '^TestSendAsync$' -count=1000 -failfast -race ./outbox

-count=N runs each test N times (and, as a side effect, skips the result cache). -failfast stops at the first failure, so you get the error at once instead of after 1,000 runs. With -race on, our §03 test failed on the very first run: the race detector slows everything down, and slowness is exactly the push a timing flake needs. Vitest reports repeats like this; the [1/7] at the end means 7 of the 201 runs failed:

 ❯ rep/outbox.test.js (1 test | 1 failed) 1476ms
   × sends the email 1474ms (repeat x200)
 …
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/7]⎯

Step 2: add load

Plain repetition on a quiet laptop might not be enough: §03's Go test passed 1,000 of 1,000 there. CI machines are busy, so make yours busy too. The Go team's stress tool runs a Test binaryTest binarygo test compiles a package's tests into a normal program and runs it. go test -c saves that program to a file, so you can run it many times directly (./pkg.test -test.run=TestX). over and over, several copies at once, and prints a running tally:

$ go install golang.org/x/tools/cmd/stress@latest
$ go test -c -o outbox.test ./outbox          # compile the tests into a program
$ stress -p 8 ./outbox.test -test.run=TestSendAsync
5s: 2890 runs so far, 674 failures (23.32%), 8 active
10s: 6058 runs so far, 1417 failures (23.39%), 8 active

-p 8 means 8 copies at once. On 4 cores, that took the §03 test from 0% to 23% failures. Each failure's full output goes to a temp file, so you get the logs too. The flags after the binary name are the test's own flags, spelled -test.run instead of -run because you're running the compiled binary directly. (Recent versions of the tool need a recent Go to install: ours pulled in Go 1.26 automatically.)

Other ways to change the machine's speed: -cpu=1,2,4 runs each test three times, with GOMAXPROCSGOMAXPROCSHow many OS threads may run Go code at the same instant. Defaults to the number of CPU cores. set to 1, 2 and 4 (how many threads may run Go code at once), which changes how goroutines get interleaved; docker run --cpus=1 limits a container to one core; and running your other test suites at the same time is a cheap stand-in for a crowded CI runner.

Step 3: change the order

If the test passes alone and fails in the full run, it's order-dependent (§04). Shuffle until it fails, then replay the seed so it fails every time:

go test -shuffle=on ./pricing                  # prints: -test.shuffle 1793396763043929742
go test -shuffle=1793396763043929742 ./pricing # the same order again

npx vitest run --sequence.shuffle              # prints: Running tests with seed "…"
npx vitest run --sequence.shuffle --sequence.seed=1793396793271

Now you know that some earlier test pollutes the victim, but not which one. With 8 tests before it, you could try them one by one. BisectingBisectingFinding a culprit by halving the list of suspects on each try: run half; if the problem shows up, it's in that half, otherwise in the other. 100 suspects take about 7 tries. is faster: run half of them, then half of the guilty half, and so on.

1# 1. the victim alone
2go test -run '^TestTotal$' ./pricing
3# 2. first half of the suspects + the victim
4go test -run '^(TestLogin|TestSignup|TestCart|TestCheckout|TestTotal)$' ./pricing
5# 3. second half + the victim
6go test -run '^(TestCoupons|TestGermanTax|TestRefund|TestInvoice|TestTotal)$' ./pricing
7# 4. halve again
8go test -run '^(TestCoupons|TestGermanTax|TestTotal)$' ./pricing
9# 5. and again
10go test -run '^(TestCoupons|TestTotal)$' ./pricing
11# 6. the last suspect
12go test -run '^(TestGermanTax|TestTotal)$' ./pricing

Two cautions. First, -run keeps the file order, so this works when the failing order is the file order. With a shuffled order, replay the seed and read the order from the -v output, then bisect in that order. Second, the polluter may be in a different package or file that shares a database. Then the suspects are "everything that ran on the same database before it".

Step 4: change the time and the zone

If the history says "fails at night" or "fails on one laptop", don't wait for midnight:

TZ=Asia/Ho_Chi_Minh go test ./...        # Go reads TZ at start-up (§05)
TZ=Pacific/Kiritimati npx vitest run     # UTC+14: far from most CI servers

To pretend it's a different date, the code has to take its time from somewhere you control: a now parameter, or vi.setSystemTime(…) in Vitest. If neither is possible, that's the bug to fix first.

Step 5: record more for next time

Some flakes refuse to reproduce locally. Then make the next CI failure tell you more:

• Log the hidden inputs: the seed, the time and time zone, the order (-v), the port, how long the slow step took. A failure message like sent 0 emails after 5 ms (render took 7.2 ms) diagnoses itself.
• Playwright traces: set trace: "on-first-retry" in playwright.config.js. When a test fails and is retried, Playwright records a TraceTrace (Playwright)A zip file recorded during a test run with every action, a DOM snapshot before and after each one, the network requests, console messages and timings. You open it in the Trace Viewer to replay a failure step by step.: every action, a snapshot of the page before and after it, network requests and console logs. CI keeps the zip; npx playwright show-trace trace.zip opens it on your machine, so you can step through the failing run as if you'd been watching. End-to-End Testing shows the viewer.
• Keep the artifacts: screenshots, server logs and database dumps from the failing job.

Retries: a painkiller, not a cure

The arithmetic

With one retry, a test only turns the build red if it fails twice in a row. For a test that flakes 1% of the time, that's 1% × 1% = 0.01%: the flake practically disappears. Now take an Intermittent bugIntermittent bugA real bug in the product that only shows up on some runs, for example a race that loses an update now and then. It looks exactly like a flaky test, which is why “just retry it” is dangerous., a real race in your code that makes a test fail 30% of the time. It needs to fail twice in a row too: 30% × 30% = 9%. With two retries it's 30% × 30% × 30% = 2.7%. The bug gets through 97 times out of 100.

That's the trade. Retries buy a green build by lowering the sensitivity of the whole suite. Any failure that's ever intermittent, flake or bug, gets filtered out the same way.

How each tool retries, and what it tells you

Vitest retries with retry (in the config, per test, or --retry=2 on the command line). Here is our §03 sleep test, 300 runs with --retry=2. The normal summary says:

      Tests  3 failed | 297 passed (300)

That looks like 297 clean passes. Only the verbose reporter (--reporter=verbose) shows that 27 of them needed a second try:

 ✓ sleep/outbox.test.js > sends the email #34 22ms (retry x1)
 ✓ sleep/outbox.test.js > sends the email #61 16ms (retry x1)
 …

Vitest 5 can also limit retries to errors you've decided are outside your control. In a test's options or in the config, retry: { count: 2, condition: /ECONNRESET/ } matches the regular expression against the error message. We checked: a test that threw read ECONNRESET once passed on its retry, and a test with a wrong value failed at once, without retrying.

Playwright retries with retries: 2 in the config or --retries=2. It is more honest: a test that failed and then passed is reported in its own flaky group. Our §03 sleep test, 10 repeats with 2 retries:

  3 failed
    e2e/save.spec.js:4:1 › shows Saved (sleep) ──────────
    …
  4 flaky
    e2e/save.spec.js:4:1 › shows Saved (sleep) ──────────
    …
  3 passed (21.0s)

Flaky tests don't fail the run by default. --fail-on-flaky-tests (or failOnFlakyTests: true) makes them fail it, which is useful on main once you've cleaned up.

Go has no built-in retry, on purpose. The popular wrapper gotestsum adds one: gotestsum --format testname --rerun-fails=2 --packages ./... -- -count=1 (--format testname prints one line per test; the default format only shows a line per package). When the re-run passes, the exit code is 0 (green), but the output still lists the first failure:

FAIL tags.TestNames (0.00s)
PASS tags.TestNames (re-run 1) (0.00s)

=== Failed
=== FAIL: tags TestNames (0.00s)
    tags_test.go:12: Names() = [sql go js], want [go js sql]

DONE 2 runs, 2 tests, 1 failure in 0.300s

A retry policy that doesn't lie

Retries aren't evil. Browsers, containers and real networks do hiccup, and blocking a merge on a dropped connection helps nobody. The rules that keep retries honest:

1. No retries for unit tests. They have no network and no browser, so a flaky unit test is always a bug in the test or the code. Fix it.
2. Retries where infrastructure is involved (E2E, some integration tests). Keep the count low (1–2), and where the tool allows, limit them to infrastructure errors.
3. Every retried pass is recorded. A "flaky" result goes into the same dashboard as failures (§11) and counts against that test. A test that needed a retry passed, but it isn't healthy.
4. Retries come with an owner. A test that keeps needing retries goes to quarantine (§10) with someone assigned to fix it, rather than being retried forever.

Quarantine without forgetting

Why not just skip it?

Everyone has seen a t.Skip("flaky") or it.skip from three years ago. That's not quarantine; that's deletion with extra steps. The feature it covered has been untested ever since, and nobody remembers why. Real quarantineQuarantine (tests)Moving a known-flaky test out of the blocking build so it can't turn builds red, while still running and tracking it until someone fixes or deletes it. keeps three things a bare skip throws away:

1. It still runs, in a separate job that can't block anyone, so you collect data on how flaky it is and notice when a fix works.
2. It's tracked: an issue, an owner and a date, written right next to the test.
3. It ends: fixed and back in the blocking build, or consciously deleted.

Marking a test as quarantined

// quarantine skips a known-flaky test unless RUN_QUARANTINED=1.
func quarantine(t *testing.T, issue string) {
    t.Helper()
    if os.Getenv("RUN_QUARANTINED") != "1" {
        t.Skipf("quarantined: %s", issue)
    }
}

func TestUploadPreview(t *testing.T) {
    quarantine(t, "#412, owner: ana, until 2026-11-15")
    // … the flaky test, unchanged …
}

test("upload shows a preview (#412, owner: ana)", { tag: "@quarantine" }, async ({ page }) => {
  // …
});
// blocking job:   npx playwright test --grep-invert @quarantine
// quarantine job: npx playwright test --grep @quarantine

All three checked for real. Go prints --- SKIP: TestUploadPreview with the reason, and runs it normally with RUN_QUARANTINED=1. Vitest 5's tags report 1 passed | 1 skipped in the blocking run. Vitest checks tag names against the list in the config, so a typo like quarantien is an error, not a silently unskipped test.

Two CI jobs

Here it is in GitHub Actions, where CI is configured by a YAML file in the repository and each job runs on its own machine. The blocking job runs everything else. The quarantine job runs only quarantined tests, repeated to gather evidence, and is allowed to fail:

jobs:
  test:                       # blocks merging
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-go@v7
        with: { go-version: "1.26" }
      - run: go test -race -shuffle=on ./...

  quarantine:                 # informs, never blocks
    runs-on: ubuntu-latest
    continue-on-error: true   # ← a red result here doesn't fail the workflow
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-go@v7
        with: { go-version: "1.26" }
      - run: RUN_QUARANTINED=1 go test -race -count=20 -run 'TestUploadPreview' ./...

(-count=20 runs the quarantined test 20 times per build. That's where "is it fixed?" gets its answer.)

Rules that stop quarantine becoming a graveyard

• Quarantine needs an issue, an owner and a date. Put them in the test name or the skip message, so they show up in every log.
• Cap the ward. For example, no more than 1% of tests or 20 tests, whichever is smaller. When it's full, fixing comes before new features for that team.
• Coming back needs proof. For example, 100 clean runs in a row under load (-count=100, --repeat-each=100) before the quarantine marker comes off.
• Past the deadline, decide. Fix it, rewrite it at a lower level (often an E2E test that should have been a unit test), or delete it and write down what is now untested. Deleting on purpose is fine. Forgetting isn't.
• Remember what's uncovered. While a test is quarantined, the behaviour it checked is not protected. If it's checkout or login, that's a reason to fix it this week, not this quarter.

Measuring flake rate

You can't manage flakiness by feel. "The build is flaky lately" starts arguments. "TestUploadPreview failed and then passed on 31 of 400 commits this month" starts a fix. That needs a definition and some data.

What counts as a flake

The useful definition: a test flaked on a commit if it both passed and failed on that commit. Same code, two answers. The second answer can come from a retry (§09), from someone pressing "re-run", or from a nightly job that runs everything again.

Per test, the Flake rateFlake rateHow often a test (or a build) fails with no relevant code change. Usually measured as: runs that failed and then passed on the same commit, divided by all runs. is then:

commits where the test both passed and failed ÷ commits where the test ran

Two things this definition gets right. A test that always fails on a commit is just failing; that's a real break, not a flake. And a test that failed once and was never re-run is unknown. That's why recording retried passes and running the nightly job below matters: without a second run, you can't tell.

Getting the data out

Every test runner can write machine-readable results. Pick one format, upload it from every CI job, and keep it:

Here's what go test -json looks like, and a quick tally with jq (a command-line tool for filtering JSON). The package holds §06's broken TestNames and its sorted fix, saved side by side as TestNamesSorted:

$ go test -json -count=50 ./tags | head -2
{"Time":"2026-10-30T21:54:30.892528032Z","Action":"start","Package":"example.com/shop/tags"}
{"Time":"2026-10-30T21:54:30.894119115Z","Action":"run","Package":"example.com/shop/tags","Test":"TestNames"}

$ go test -json -count=50 ./tags \
    | jq -r 'select(.Test and (.Action=="pass" or .Action=="fail")) | "\(.Test) \(.Action)"' \
    | sort | uniq -c
      7 TestNames fail
     43 TestNames pass
     50 TestNamesSorted pass

Computing flake rate per test

Say CI saves one line per test attempt: { "commit": "a1f3", "test": "adds 19% tax", "status": "fail" }. Test names often contain spaces (Vitest and Playwright names are sentences), so the code must never build a key by gluing name and commit together with a space and splitting it later. Group by test and commit, and look for commits with both results:

type key struct{ test, commit string }
type stat struct{ commits, flaky, broken int }

// seen[(test, commit)] = the statuses seen for that test on that commit
seen := map[key]map[string]bool{}
// for each JSON line r read from the file:
k := key{r.Test, r.Commit}
if seen[k] == nil {
    seen[k] = map[string]bool{} // ← the inner map must be made first
}
seen[k][r.Status] = true

stats := map[string]*stat{}
for k, st := range seen {
    s := stats[k.test]
    if s == nil {
        s = &stat{}
        stats[k.test] = s
    }
    s.commits++
    switch {
    case st["fail"] && st["pass"]: // ← same code, both results
        s.flaky++
    case st["fail"]: // failed every time: a real failure
        s.broken++
    }
}

On a 10-line sample file with two Vitest-style names, the JS version prints this (the full Go program is about 60 lines and prints the same two lines, most flaky first):

adds 10% tax   flaky on 0/4 commits (0%), broken on 1
adds 19% tax   flaky on 2/4 commits (50%), broken on 0

Look at adds 10% tax: it failed once on commit c2d4 and was never re-run, so it shows as "broken". It may really be a flake. Without a second run, the data can't tell, which is the argument for the next idea.

A nightly deflake run

Once a night, run the whole suite several times against the same, unchanged main commit, with shuffling and the race detector on. Nothing changed between those runs, so by definition every failure is a flake. The nightly run finds them before they annoy anyone, and it settles every "unknown" from the day:

go test -race -shuffle=on -count=5 ./... -json > nightly.jsonl
npx vitest run --sequence.shuffle --repeats=4 --reporter=junit --outputFile=nightly.xml
npx playwright test --repeat-each=5 --reporter=json > nightly.json

What to do with the numbers

• Rank, don't average. Sort tests by flaky commits. In most suites a few tests cause most of the flakes. Fix the top three and the build visibly calms down.
• Set a budget for the whole build. For example, "fewer than 2% of builds on main fail and then pass on re-run". It works like an SLOSLO (service level objective)A reliability target you measure, like “99.9% of requests succeed in under 300 ms over 30 days”. The measured number (here, the share of fast successful requests) is the SLI; the 0.1% you're allowed to miss is the error budget. (a reliability target a team commits to, used for production services) for your test suite: inside the budget, carry on; over it, fixing flakes comes before features. Why & What to Test §11 shows how fast per-test flake rates add up to a red build.
• Act on a threshold. For example, a test that flakes on 3 commits in 7 days goes to quarantine (§10) with an issue. That rule can be a script reading the same data.
• Count the retries. If the dashboard only counts red builds, retries (§09) make the problem disappear from it. Count retried passes as flakes.

Check yourself

Answer before you click. Q1 is covered in §03, Q2 in §04 and §08, Q3 in §06, Q4 in §09 and Q5 in §07.

func TestKeys(t *testing.T) {
    m := map[string]int{"a": 1, "b": 2, "c": 3}
    var got []string
    for k := range m {
        got = append(got, k)
    }
    if !slices.Equal(got, []string{"a", "b", "c"}) {
        t.Fail()
    }
}

Cheat sheet

Diagnosis commands

Symptom → cause