feat(evals): add runner tapes for recording and replaying experiment runner output

CO
2d ago
+2616-50

PR Checklist

  • The commit message follows our guidelines

Bugs / Features

  • Related issue(s) linked (no existing issue)
  • Tests for the changes have been added
  • Docs have been added / updated
  • Changesets have been added

What is the current behavior?

Changing scorers in runExperiment means re-running the target runner. onItem fires after scoring, so if scoring or the process is interrupted, outputs from targets that already finished are lost. Separately, a runner or scorer that throws a falsy value (undefined / null / 0 / false) is counted as a pass, and a per-score status: "error" without an error field is not detected.

What is the new behavior?

  • createRecordingRunner / createReplayRunner: plug into the existing runExperiment and record native ExperimentRunner output before scoring. Replay feeds recorded output through new scorers without calling the target again. Slots are keyed by item id + original index and bound to a caller-declared manifest (runner/dataset revisions); identity mismatches are rejected.
  • createMemoryTapeStore / createDirectoryTapeStore: the directory store fsyncs a claim (file + parent dir) before invoking the target, publishes terminal results via exclusive temp file + no-clobber link, and fails closed on corrupt or partial files. No automatic retry; an unresolved slot is reported explicitly and needs a new tape.
  • inspectTape: counts and indices only, no content.
  • Original error messages/stacks are not persisted (a capture failure keeps the live error as cause in memory only).
  • Fix: falsy throws are treated as errors via explicit runnerFailed / scoringFailed flags; per-score error status is matched on status.
  • All serialization goes through safeStringify; its fallback string is rejected rather than persisted.

Notes for reviewers

  • A mid-flight abort waits for the target to actually settle (so the outcome can be recorded); this differs from native cooperative cancellation and is documented.
  • The directory store refuses Windows filesystems where directory fsync fails (DURABILITY_UNSUPPORTED); the memory store works everywhere.
  • No examples/ package was added because it would require a lockfile change; the docs page has a runnable snippet. Happy to add one if you prefer.
  • Verified locally: @voltagent/evals vitest 68/68, tsc --noEmit, build, and biome check on changed files. No model calls.

I used AI tools to assist; I reviewed and can explain every line.

🤖 Generated with Claude Code

Summary by cubic

Adds runner tapes to @voltagent/evals that record native ExperimentRunner output before scoring and replay it through new scorers without calling the target again. Also fixes runExperiment counting falsy thrown values (undefined, null, 0, false) from a runner or scorer as success.

  • createRecordingRunner and createReplayRunner plug into the existing runExperiment; slots are keyed by item ID and original index and bound to a caller-declared manifest, with mismatches rejected.
  • Evidence is stored in createMemoryTapeStore (process-local) or createDirectoryTapeStore, which claims each slot with fsync before the target runs and fails closed on corrupt or partial files.
  • inspectTape reports outcome counts and indices only, never content; error messages and stacks are not persisted.
  • Unresolved slots are reported explicitly and require a new tape; there is no automatic retry.

Summary by CodeRabbit

  • New Features
    • Added runner tapes to record experiment outputs and replay them with different scorers, without rerunning the target.
    • Added in-memory and directory-based tape storage, plus inspection summaries of outcomes and indices without exposing recorded content.
    • Added documentation for recording, replaying, storing, and inspecting tapes.
  • Bug Fixes
    • Experiment runs now correctly report errors when runners or scorers throw falsy values.