Flaky Playwright suite: hard sleeps, racing navigation and ambiguous selectors

Our storefront team runs about 210 Playwright tests against a preview environment on every pull request. Over March, the suite failed on roughly 12 percent of runs, and a rerun almost always passed. Engineers had stopped reading the failures; "just rerun it" was the standard comment on pull requests, and twice a real regression was merged because its failure looked like the usual noise.

We exported a month of CI results and grouped the failures. Three messages accounted for 81 percent of them:

Error: expect(received).toBe(expected) // Object.is equality

Expected: true
Received: false

  at tests/checkout/coupon.spec.ts:41:52

Error: expect(received).toContain(expected) // indexOf

Expected substring: "/orders/"
Received string:    "https://preview.example.com/checkout/payment"

Error: locator.click: Error: strict mode violation: getByText('Save') resolved to 2 elements:
    1) <button type="submit" class="btn btn-primary">Save</button> aka getByRole('button', { name: 'Save', exact: true })
    2) <button type="button" class="btn btn-link">Save as draft</button> aka getByRole('button', { name: 'Save as draft' })

Reproducing it

Flaky tests hide on a fast laptop. We made them fail on demand by repeating them under CI-like conditions: one worker per CPU we gave the container, the preview environment instead of a local server, and many repetitions:

npx playwright test tests/checkout --repeat-each=30 --workers=4 --retries=0 --trace=retain-on-failure

Out of 30 repetitions of coupon.spec.ts, 5 failed. The trace of one failure, opened with npx playwright show-trace, showed the problem clearly: the test asserted on the toast at 2,000 ms, and the network panel showed the POST /api/cart/coupon response arriving at 2,340 ms.

What we ruled out

• An unstable preview environment. Error rates and latency on the preview API were flat during failing runs; failures did not cluster in time.
• Browser-specific bugs. The failures were spread across Chromium and WebKit projects in proportion to how many tests each ran.
• Test order dependencies. Running with --shard and reversed file order did not change the failure rate, though we did find one shared-account issue later (below).

Root cause

The suite was written with habits from an older Selenium codebase, and each one is a race:

1. Fixed sleeps followed by instant checks. await page.waitForTimeout(2000) and then expect(await locator.isVisible()).toBe(true). If the app takes 2,001 ms, the test fails; if it takes 200 ms, we waste 1.8 seconds per step.
2. Reading page.url() right after a click that triggers navigation. The click resolves when the click happens, not when the next page has loaded.
3. Text selectors that match more than one element. getByText('Save') matches substrings, so when a "Save as draft" button was added in February, Playwright's strict mode refused to guess. That one was not flaky so much as a failure that only appeared on pages where both buttons rendered.

What is happening under the hood

Playwright has two kinds of checks, and they behave very differently:

Actions such as click() and fill() already wait for the element to be attached, visible, stable, enabled and receiving events. So a test written with locators and web-first assertions waits exactly as long as needed and no longer. A fixed sleep can only be too short (flaky) or too long (slow), and on shared CI runners it is usually both on different days. Playwright's own documentation discourages waitForTimeout in tests for this reason.

Locators are strict: an action on a locator that matches several elements throws instead of picking the first one. That is a feature. The fix is a locator that describes what the user sees, such as a role and accessible name, not a looser one.

Environment

Minimal reproduction

import { test, expect } from '@playwright/test';

test('applies a coupon', async ({ page }) => {
  await page.goto('/cart');
  await page.getByLabel('Coupon code').fill('SPRING10');
  await page.getByText('Apply').click();

  await page.waitForTimeout(2000);                       // too short on a slow run
  expect(await page.locator('.toast').isVisible()).toBe(true); // checked exactly once

  await page.getByText('Checkout').click();
  expect(page.url()).toContain('/checkout');             // races the navigation
});

Add a 2.5 second delay to the coupon endpoint (a page.route handler that waits before route.continue() works) and it fails every time.

Who is affected

Any end-to-end suite, in any framework, that mixes fixed waits with one-shot checks. Playwright makes the correct style easy, but it still runs the incorrect style without complaint. Suites ported from Selenium or Puppeteer, and tests generated by recording and then edited by hand, are the most common sources.

The fix

1. Replace sleeps and snapshots with web-first assertions.

-  await page.getByText('Apply').click();
-  await page.waitForTimeout(2000);
-  expect(await page.locator('.toast').isVisible()).toBe(true);
+  await page.getByRole('button', { name: 'Apply coupon' }).click();
+  await expect(page.getByRole('status')).toHaveText('Coupon SPRING10 applied');
+  await expect(page.getByTestId('cart-total')).toHaveText('$90.00');

Asserting on the resulting total rather than only on the toast also proves the coupon took effect.

2. Assert navigation with the page, not with a URL read.

-  await page.getByText('Pay now').click();
-  expect(page.url()).toContain('/orders/');
+  await page.getByRole('button', { name: 'Pay now' }).click();
+  await expect(page).toHaveURL(/\/orders\/\d+$/);
+  await expect(page.getByRole('heading', { name: 'Thank you for your order' })).toBeVisible();

When a test needs to wait for a specific API response, we wait for that response, started before the click so it cannot be missed:

const saved = page.waitForResponse(
  (r) => r.url().endsWith('/api/addresses') && r.request().method() === 'POST',
);
await page.getByRole('button', { name: 'Save', exact: true }).click();
expect((await saved).status()).toBe(201);

3. Use user-facing locators. We replaced getByText on interactive elements with getByRole(..., { name }), used exact: true where names share a prefix, and scoped locators to a region when a page repeats the same control:

const shipping = page.getByRole('region', { name: 'Shipping address' });
await shipping.getByRole('button', { name: 'Save', exact: true }).click();

4. Isolate data. While repeating tests we found two specs logging in as the same seeded user and editing the same cart in parallel. Each worker now creates its own user through the API in a worker-scoped fixture.

5. Make lint enforce it. eslint-plugin-playwright flags most of these patterns. We enabled no-wait-for-timeout, prefer-web-first-assertions and no-useless-await as errors.

Alternatives and trade-offs

1. Raise timeouts or sleeps. Lowers the failure rate, raises the suite duration, and the next slow day brings the failures back.
2. More retries. Retries are a measurement tool, not a fix. With retries: 3, a test that fails half the time still passes 94 percent of runs and is reported as "flaky" in a column nobody reads. We moved to retries: 2 in CI only, and we track flaky tests explicitly (below).
3. test.slow() or per-assertion timeouts for genuinely slow flows, such as PDF generation. Legitimate, but only once the wait is a web-first assertion.

How to test the fix

The same repetition command that exposed the flakiness now proves its absence:

npx playwright test tests/checkout --repeat-each=50 --workers=4 --retries=0

All 50 repetitions of every checkout spec passed. We also ran the suite with the API artificially slowed by 1.5 seconds per request through a page.route fixture; the tests got slower but did not fail, which is exactly what web-first assertions should do.

Confirming in production

Over the four weeks after the change, across 1,140 CI runs:

• Suite failure rate on unchanged code fell from about 12 percent to 0.7 percent.
• Tests reported as flaky (passed on retry) fell from 9 to 10 per day to 1 or 2 per week, each one now triaged with its trace.
• Median suite duration dropped from 14 minutes to 9, because 380 seconds of fixed sleeps were removed.

Regression guard

Our configuration now records evidence on retries and keeps local runs honest:

// playwright.config.ts (excerpt)
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: { trace: 'on-first-retry' },
  reporter: [['html'], ['json', { outputFile: 'results.json' }]],
});

A nightly job runs the full suite with --fail-on-flaky-tests, so a test that only passes on retry turns the nightly build red and creates a ticket for the owning team. The ESLint rules above run on every pull request, and a weekly job runs the slowest 20 specs with --repeat-each=20.

Security implications

None beyond the bug itself. Two hygiene points came up while fixing it: traces and videos capture everything on screen and in network requests, so test accounts must never use real customer data, and CI artifacts containing traces should have short retention. Per-worker test users are created with random passwords and deleted after the run.

Prevention checklist

• Never use waitForTimeout in tests; wait for a state the user can see.
• Use await expect(...) web-first assertions instead of reading values and comparing.
• Prefer getByRole with an accessible name; add exact: true when names overlap.
• Give each worker its own data so parallel tests cannot interfere.
• Keep retries in CI only, record traces on retry, and fail a scheduled run on flaky tests.

References

• Playwright documentation, Best practices — https://playwright.dev/docs/best-practices
• Playwright documentation, Assertions (auto-retrying) — https://playwright.dev/docs/test-assertions
• Playwright documentation, Locators and strictness — https://playwright.dev/docs/locators
• Playwright documentation, Trace viewer — https://playwright.dev/docs/trace-viewer
• Playwright documentation, Retries — https://playwright.dev/docs/test-retries
• Playwright documentation, Command line (--repeat-each, --fail-on-flaky-tests) — https://playwright.dev/docs/test-cli