fix(test): read the test.skip and test.fail condition by truthiness
Inside a test body, test.skip(condition, reason) and test.fail(condition, reason) treated every condition that was not a boolean as true. So test.skip(process.env.CI, 'only on CI') skipped on a machine where CI is unset, and test.fail(undefined, 'x') marked a passing test, which then failed with UNEXPECTED_PASS. Playwright reads the condition by truthiness. The maintainer asked for this fix in its own PR in #1887 ("Your separate PR for it is welcome").
parseModifierCall(packages/e2e/src/internal/skip.ts) now gets the call's own arguments (skip(...args)), so it can tellskip(undefined, reason)fromskip(reason?). With a condition, it reads the first argument by truthiness and takes the reason from the second argument only, as Playwright does.- The one-argument forms keep e2e's
skip(reason?: string)overload:test.skip(),test.skip('reason'), andtest.skip(undefined)still always skip. - A Promise condition (a missing
await) or a function condition used to skip or mark the test every time. Inside a body both now fail the test withINVALID_ARGUMENT. Playwright also throws on a function there. It goes through thereadConditionthattest.skipIfandtest.runIfalready used, moved tointernal/skip.tswith the error it throws as a parameter, soskipIfandrunIfkeepCOLLECTION_ERROR. Outside a body,test.skip(promise, reason)keeps main's error: the call belongs in a test body.isPromiseLikemoved tointernal/promise-like.tswith its own unit test, now counts a function with a callablethenas a thenable, and the MCP schema check uses it instead of its own copy. - Docs:
reference/test.mdxand theTestAPIJSDoc say "truthy", name the one-argument difference from Playwright, and list the Promise case underINVALID_ARGUMENT. The ja and zh Types blocks carry the same JSDoc. The Playwright migration guide names the one-argument difference: writetest.skip(!!process.env.CI).
test('sync', async () => {
test.skip(process.env.CI, 'flaky on CI'); // CI unset: runs (main: skipped)
test.fail(flags.beta, 'beta gap'); // flag missing: runs as normal (main: UNEXPECTED_PASS)
test.skip(await isSingleTenant(), 'x'); // still skips when true
test.skip(isSingleTenant(), 'x'); // missing await: INVALID_ARGUMENT (main: always skipped)
test.skip(() => isBeta, 'x'); // a function: INVALID_ARGUMENT (main: always skipped)
});
The condition stays typed boolean, as in Playwright, so TypeScript users hit the falsy cases only through any or a cast. JavaScript suites hit them directly.
Issue
Fixes #1966
Overlap with open PRs
git merge-tree of this head against each open PR that touches the same files:
| PR | Conflicting files |
|---|---|
#1887 test.fixme (mine) | internal/skip.ts, collect/registry.ts, reference/test.mdx, migrate/playwright.mdx, runner-lifecycle.test.ts, sdk-types.ts |
#1914 test.skip() from teardown | internal/skip.ts, reference/test.mdx |
#1925 test.describe.fail | collect/registry.ts, sdk-types.ts |
#1858 test.todo | collect/registry.ts, sdk-types.ts, reference/test.mdx (en, ja, zh) |
- #1887 is the one with a semantic dependency. It adds a third caller with the old
parseModifierCall(first, second)signature, which no longer compiles once this lands. Whichever lands second movestest.fixmetofixme(...args)withparseModifierCall('test.fixme()', args)and addstest.fixme(undefined, 'x')and Promise cases. Passing[first, second]would compile but break a baretest.fixme(). I'll do that on #1887 if this one lands first. - The other three are text conflicts only: none calls
parseModifierCallorreadConditionin a new place.
For the maintainer
- A lone
test.skip(undefined)ortest.skip('')still skips, because it matches e2e'sskip(reason?: string)overload. Playwright does not skip on either. Following Playwright here means narrowing the overload toskip(): void; skip(reason: string): void, which breaks callers that passstring | undefined. I kept today's behavior and documented the difference. Say if you want parity instead. - Widening the condition type to
SynchronousCondition<Condition>, asskipIfhas, would lettest.skip(process.env.CI, 'x')type-check. That is a public API change, so it is not in this PR.
Verified
Ran it locally: yes
- testbed, built CLI (
node node_modules/e2e/dist/cli/bin.js run tests/zz-falsy-repro.e2e.ts --reporter list), one scratch test file (not committed),origin/mainbuild against this branch:
| Call in the test body | origin/main (bc2193ac) | this branch |
|---|---|---|
test.skip(process.env.UNSET, 'flaky on CI') | skipped [flaky on CI] | passed |
test.skip(flags.beta, 'beta only'), flag missing | skipped [beta only] | passed |
test.skip(0, 'no items') | skipped [no items] | passed |
test.fail(process.env.UNSET, 'known CI gap'), passing body | failed, UNEXPECTED_PASS | passed |
test.skip('yes', 'the real reason') | skipped [yes] | skipped [the real reason] |
test.skip() | skipped | skipped |
test.skip('waiting on the API') | skipped [waiting on the API] | skipped [waiting on the API] |
test.skip(undefined) | skipped | skipped |
test.skip(Promise.resolve(false), 'async check') | skipped [async check] | failed, test.skip() got a Promise as its condition; await it before the call |
test.skip(() => false, 'fn condition') | skipped (not run on main here; main's parser counts any non-boolean as true) | failed, test.skip() got a function as its condition; call it and pass the result |
- Outside a body,
test.skip(Promise.resolve(false), 'async check')at module level stops the run withCOLLECTION_ERROR: test.skip(condition, reason) skips the running test and must be called inside a test body, the same error as on main. - With
FALSY_REPRO_CI=1set, the same file still skips the CI-only test and marks the CI-onlytest.fail(its passing body then fails withUNEXPECTED_PASS), so a truthy condition still applies.
🤖 Generated with Claude Code