Playwright with produces false positives with toMatchAriaSnapshot on GitHub

1 week ago 6
ARTICLE AD BOX

Playwright is producing false positives on GitHub.

I assume that this is because:

it is a clean server; and (therefore)

it is creating a snapshot and falsely not using the string literal defined in the test expectation:

await expect(this.page.getByRole('navigation', { name: 'Menu' })).toMatchAriaSnapshot(`Expected ARIA String Literal`)

I can get the tests to fail locally.

baseURL=http://localhost:8080 CI=true npx playwright test --retries=0 --update-snapshots --grep "My Test 001" --max-failures=1 --reporter=line

and I can get same to pass (false positive):

baseURL=http://localhost:8080 CI=true npx playwright test --retries=0 --update-snapshots --grep "My Test 001" --max-failures=1 --reporter=line --update-snapshots

The key difference between the pass and fail above is the use of the --update-snapshots flag.

How can I get the tests to fail on the server, as expected? I have tried --update-snapshots=none; this does not work.

Playwright version: Version 1.58.2 Node Version: v22.13.1 Environments: GitHub Runner, Workstation OS: Linux Ubuntu
Read Entire Article