ARTICLE AD BOX
Playwright is producing false positives on GitHub.
I assume that this is because:
it is a clean server; and (therefore)
it is creating a snapshot and falsely not using the string literal defined in the test expectation:
await expect(this.page.getByRole('navigation', { name: 'Menu' })).toMatchAriaSnapshot(`Expected ARIA String Literal`)I can get the tests to fail locally.
baseURL=http://localhost:8080 CI=true npx playwright test --retries=0 --update-snapshots --grep "My Test 001" --max-failures=1 --reporter=lineand I can get same to pass (false positive):
baseURL=http://localhost:8080 CI=true npx playwright test --retries=0 --update-snapshots --grep "My Test 001" --max-failures=1 --reporter=line --update-snapshotsThe key difference between the pass and fail above is the use of the --update-snapshots flag.
How can I get the tests to fail on the server, as expected? I have tried --update-snapshots=none; this does not work.
Playwright version: Version 1.58.2 Node Version: v22.13.1 Environments: GitHub Runner, Workstation OS: Linux Ubuntu