fix(evals): propagate experiment callback failures

EL
1d ago
+300-4

PR Checklist

  • The commit message follows the contribution guidelines.

Bugs / Features

  • Related issue linked: not applicable; this is a source-discovered bug with the reproduction below. No matching issue/PR was found.
  • Tests for the changes have been added.
  • Docs added/updated: not applicable; the existing callback error contract and public API are preserved.
  • A patch changeset for @voltagent/evals has been added.

What is the current behavior?

runExperiment registers task cleanup with void task.finally(...). If onItem or onProgress throws or rejects, that ignored derived promise rejects as well. A caller can catch the runExperiment error and still have Node terminate due to an unhandled rejection.

There is a second consequence of the same cleanup: a rejected task is deleted from the active set while the loop is awaiting a slow asynchronous dataset. If the dataset finishes before reaching the concurrency limit, runExperiment no longer awaits the rejected task and can report the run as successful.

Public-API reproduction (Node 24.13.0, built workspace on 72a46c76b4507e1f98b41d5050a92bd30587eb70):

import { setImmediate } from "node:timers/promises";
import { createExperiment, runExperiment } from "@voltagent/evals";

const expected = new Error("onItem failed");
const experiment = createExperiment({
  id: "callback-error-reproduction",
  dataset: { items: [{ id: "item-1", input: "hello" }] },
  runner: async () => ({ output: "world" }),
});
try {
  await runExperiment(experiment, {
    onItem: async () => { throw expected; },
  });
} catch (error) {
  if (error !== expected) throw error;
  console.log("caller caught the original callback error");
}
await setImmediate();
console.log("caller recovered without an unhandled rejection");

Run with node --unhandled-rejections=strict: before the fix, it prints the first message and exits 1 with the already-caught callback error. After the fix, it prints both messages and exits 0. No process-level rejection listeners are installed.

What is the new behavior?

Use a two-handler then for cleanup, so the cleanup chain resolves on either outcome. Fulfilled tasks are removed as before; failed tasks remain observable and signal their original rejection to the scheduling loop. The existing catch path records a failed VoltOps run and rethrows the original callback error.

The review follow-up also handles a dataset whose next item never arrives. Each dataset read races against the first task failure, with its abort listener removed afterward. A callback failure now rejects the experiment without waiting for that read; Error, undefined, and null rejection reasons are preserved. No further dataset item is processed after failure.

When a dataset read itself throws or rejects, preserve that error and skip iterator.return(), matching for await...of. A failing or stalled cleanup must not replace or delay that read error.

On other early exits, iterator cleanup is requested. After a task failure it is observed without awaiting it: an async generator's return() can itself wait for the pending next(). A cleanup rejection cannot replace the callback error or become an unhandled rejection. The underlying source may retain one pending read until its own I/O settles; this does not cancel that I/O or drain other already-running items.

Notes for reviewers

Validation in the actual frozen-lockfile pnpm 8.10.5 workspace:

  • Added ten regression cases: synchronous/asynchronous onItem and onProgress failures at concurrency 1 and 3, plus callback failures while waiting for the next asynchronous dataset item. The unchanged implementation produces two failed assertions and ten unhandled rejection errors; the other eleven tests in the file pass.
  • Full @voltagent/evals suite: 32 tests passed in 3 files, no unhandled rejection errors. The tests also verify failed VoltOps reporting and that successful completion is not reported after callback failure.
  • Independent built public-API runs: both callbacks at concurrency 1 and 3 exit 0 under Node strict unhandled-rejection handling.
  • Independently reproduced the review finding on the initial PR head 8a241ab: both callback failures remain pending while a dataset read stalls (2 assertions failed; the other 13 tests in that intermediate file passed). Added six stalled-read cases covering both callbacks and Error/undefined/null, plus a successful bounded-concurrency/ordered-results regression.
  • Twelve additional independent processes exercise the built public API under node --unhandled-rejections=strict: both callbacks, all three rejection reasons, and successful/throwing source cleanup. Every case rejects before the stalled read is released, preserves the reason, processes only the first item, and exits 0.
  • Four further regressions independently reproduce the latest review finding on f46cb84: synchronous/asynchronous next() failure with rejecting/stalled return(); all four fail before the fix. Another twelve independent strict Node processes verify those read-failure combinations with Error/undefined/null, preserving the original reason without invoking return(). The twelve callback/stalled-source process cases above were also rerun on the final build.
  • Recursive build of @voltagent/evals and its workspace dependencies passed; the changed evals package was rebuilt afterward. Evals tsc --noEmit and a separate typecheck of the changed test file passed. The evals package was rebuilt for the review follow-up.
  • Repository biome ci . passed for the 854 files in this sparse checkout, with 5 existing complexity warnings outside the changed files. Changed-file Biome and git diff --check passed.
  • The full monorepo test suite and external-service/Docker tests were not run.

AI assistance: OpenAI Codex helped discover, reproduce, implement, and validate this change. The stalled-dataset review finding was independently reproduced before preparing the follow-up fix.

Summary by CodeRabbit

  • Bug Fixes
    • Experiment runs now surface failures from synchronous and asynchronous callbacks, including during concurrent runs.
    • When an experiment task fails while an asynchronous dataset is waiting for its next item, the run now reports the original failure without waiting indefinitely for dataset cleanup. Failed work is reported as failed rather than being marked complete.