fix(test): read the test.skip and test.fail condition by truthiness

Draft
SA
2h ago
+291-59

Inside a test body, test.skip(condition, reason) and test.fail(condition, reason) treated every condition that was not a boolean as true. So test.skip(process.env.CI, 'only on CI') skipped on a machine where CI is unset, and test.fail(undefined, 'x') marked a passing test, which then failed with UNEXPECTED_PASS. Playwright reads the condition by truthiness. The maintainer asked for this fix in its own PR in #1887 ("Your separate PR for it is welcome").

  • parseModifierCall (packages/e2e/src/internal/skip.ts) now gets the call's own arguments (skip(...args)), so it can tell skip(undefined, reason) from skip(reason?). With a condition, it reads the first argument by truthiness and takes the reason from the second argument only, as Playwright does.
  • The one-argument forms keep e2e's skip(reason?: string) overload: test.skip(), test.skip('reason'), and test.skip(undefined) still always skip.
  • A Promise condition (a missing await) or a function condition used to skip or mark the test every time. Inside a body both now fail the test with INVALID_ARGUMENT. Playwright also throws on a function there. It goes through the readCondition that test.skipIf and test.runIf already used, moved to internal/skip.ts with the error it throws as a parameter, so skipIf and runIf keep COLLECTION_ERROR. Outside a body, test.skip(promise, reason) keeps main's error: the call belongs in a test body. isPromiseLike moved to internal/promise-like.ts with its own unit test, now counts a function with a callable then as a thenable, and the MCP schema check uses it instead of its own copy.
  • Docs: reference/test.mdx and the TestAPI JSDoc say "truthy", name the one-argument difference from Playwright, and list the Promise case under INVALID_ARGUMENT. The ja and zh Types blocks carry the same JSDoc. The Playwright migration guide names the one-argument difference: write test.skip(!!process.env.CI).
test('sync', async () => {
  test.skip(process.env.CI, 'flaky on CI'); // CI unset: runs (main: skipped)
  test.fail(flags.beta, 'beta gap');        // flag missing: runs as normal (main: UNEXPECTED_PASS)
  test.skip(await isSingleTenant(), 'x');   // still skips when true
  test.skip(isSingleTenant(), 'x');         // missing await: INVALID_ARGUMENT (main: always skipped)
  test.skip(() => isBeta, 'x');             // a function: INVALID_ARGUMENT (main: always skipped)
});

The condition stays typed boolean, as in Playwright, so TypeScript users hit the falsy cases only through any or a cast. JavaScript suites hit them directly.

Issue

Fixes #1966

Overlap with open PRs

git merge-tree of this head against each open PR that touches the same files:

PRConflicting files
#1887 test.fixme (mine)internal/skip.ts, collect/registry.ts, reference/test.mdx, migrate/playwright.mdx, runner-lifecycle.test.ts, sdk-types.ts
#1914 test.skip() from teardowninternal/skip.ts, reference/test.mdx
#1925 test.describe.failcollect/registry.ts, sdk-types.ts
#1858 test.todocollect/registry.ts, sdk-types.ts, reference/test.mdx (en, ja, zh)
  • #1887 is the one with a semantic dependency. It adds a third caller with the old parseModifierCall(first, second) signature, which no longer compiles once this lands. Whichever lands second moves test.fixme to fixme(...args) with parseModifierCall('test.fixme()', args) and adds test.fixme(undefined, 'x') and Promise cases. Passing [first, second] would compile but break a bare test.fixme(). I'll do that on #1887 if this one lands first.
  • The other three are text conflicts only: none calls parseModifierCall or readCondition in a new place.

For the maintainer

  • A lone test.skip(undefined) or test.skip('') still skips, because it matches e2e's skip(reason?: string) overload. Playwright does not skip on either. Following Playwright here means narrowing the overload to skip(): void; skip(reason: string): void, which breaks callers that pass string | undefined. I kept today's behavior and documented the difference. Say if you want parity instead.
  • Widening the condition type to SynchronousCondition<Condition>, as skipIf has, would let test.skip(process.env.CI, 'x') type-check. That is a public API change, so it is not in this PR.

Verified

Ran it locally: yes

  • testbed, built CLI (node node_modules/e2e/dist/cli/bin.js run tests/zz-falsy-repro.e2e.ts --reporter list), one scratch test file (not committed), origin/main build against this branch:
Call in the test bodyorigin/main (bc2193ac)this branch
test.skip(process.env.UNSET, 'flaky on CI')skipped [flaky on CI]passed
test.skip(flags.beta, 'beta only'), flag missingskipped [beta only]passed
test.skip(0, 'no items')skipped [no items]passed
test.fail(process.env.UNSET, 'known CI gap'), passing bodyfailed, UNEXPECTED_PASSpassed
test.skip('yes', 'the real reason')skipped [yes]skipped [the real reason]
test.skip()skippedskipped
test.skip('waiting on the API')skipped [waiting on the API]skipped [waiting on the API]
test.skip(undefined)skippedskipped
test.skip(Promise.resolve(false), 'async check')skipped [async check]failed, test.skip() got a Promise as its condition; await it before the call
test.skip(() => false, 'fn condition')skipped (not run on main here; main's parser counts any non-boolean as true)failed, test.skip() got a function as its condition; call it and pass the result
  • Outside a body, test.skip(Promise.resolve(false), 'async check') at module level stops the run with COLLECTION_ERROR: test.skip(condition, reason) skips the running test and must be called inside a test body, the same error as on main.
  • With FALSY_REPRO_CI=1 set, the same file still skips the CI-only test and marks the CI-only test.fail (its passing body then fails with UNEXPECTED_PASS), so a truthy condition still applies.

🤖 Generated with Claude Code