fix(scheduling): settle expandable-node request correlation on non-dispatch paths

Merged
BN
2mo ago
+735-44

Problem

MC-3: submit mints a full request-correlation record set for every non-caller job, including needs-deferred dynamic-matrix nodes. Those nodes route to expansion and never dispatch to a runner, so no completion, result patch, or disconnect ever settles their records. On every path where such a node concludes without dispatching, the record leaked in inflight_requests and session_active_requests with result: None — staying renewable and resolvable to a job that expansion had already deleted from the run.

Fix

Settle the record (result recorded, out of inflight + every session) exactly as the completion path would, on all non-dispatch conclusions:

  • run cancel (cancel_run_inner): use a run-wide expandable_job_ids set instead of the narrow caller_plans-only collection.
  • job cancel (cancel_job_inner): settle via is_expandable_node.
  • submit-time workflow-concurrency overflow cancel (submit_run_inner).
  • dependency skip / if: eval error (promote_ready_jobs Skip/Error arm).

Reusable callers mint no records at submit, so retiring them is a no-op; Settle is idempotent, so overlapping paths cannot double-settle harmfully. id_token_grants / oidc_job_contexts are deliberately kept (mirrors completed/cancelled real jobs).

Tests

Adds coverage for all four non-dispatch conclusions:

  • cancelled_deferred_matrix_node_settles_submit_requests
  • cancelled_deferred_matrix_node_job_cancel_settles_requests
  • overflowed_run_settles_deferred_matrix_node_requests
  • dependency_skipped_deferred_matrix_node_settles_requests
  • dependency_error_deferred_matrix_node_settles_requests

Verification

Rebased onto current main. cargo build, all 5 MC-3 tests, cargo fmt --check, and cargo clippy clean on preloop-runner-server.

Summary by cubic

Fixes MC-3: request-correlation records for needs-deferred dynamic-matrix nodes kept leaking in inflight_requests and session_active_requests when the node concluded without ever dispatching. These submit-time records are now settled exactly like a completed job — result recorded, removed from inflight and every session — on all non-dispatch conclusion paths.

Paths settled

  • Run cancel gathers a run-wide expandable_job_ids set from all queues, replacing the narrow caller_plans-only collection.
  • Job cancel settles expandable nodes via is_expandable_node.
  • Workflow-concurrency overflow cancels at submit time now settle expandable nodes.
  • The promote_ready_jobs Skip/Error arm settles expandable nodes with Skipped or Failure.

Settlement uses each node's own concluded status from the run, so an already-terminal expandable node (like a reusable caller that finished Success while the run stayed active) keeps its verdict instead of being overwritten to Cancelled. Settle is idempotent, and reusable callers mint no records at submit, so overlapping paths and caller retirement are no-ops; id_token_grants and oidc_job_contexts are deliberately kept to mirror completed jobs. Adds six tests covering each non-dispatch conclusion plus completed-caller preservation.

[!NOTE]

Settle expandable-node request correlations on run/job cancel and dependency skip paths

  • Expandable nodes (deferred-matrix placeholders and reusable callers) had their submit-time request records left inflight when a run or job was cancelled or when a dependency decision skipped or errored the node. This PR settles those records with the node's actual concluded status and clears inflight/session claims.
  • cancel_run_inner and cancel_job_inner now use expandable_job_ids / is_expandable_node to detect all expandable nodes and call retire_node_requests with node_settle_status, preserving terminal verdicts like Success for already-completed reusable callers instead of forcing Cancelled.
  • promote_ready_jobs settles request records for expandable nodes on dependency Skip/Error outcomes; submit_run_inner settles expandable nodes when a run is immediately cancelled on acceptance.
  • RequestRetirement::Settle retains id_token_grants and oidc_job_contexts for settled nodes while clearing inflight and session-active claims.
  • Behavioral Change: cancellation and dependency-skip paths now settle expandable-node request records with their actual job status (via node_settle_status in runtime_scheduling.rs) rather than leaving them inflight or hardcoding Cancelled; verify consumers of plan_requests, agent_job_requests, and timeline_requests still resolve to settled placeholders.

Macroscope summarized 2842849.

Summary by CodeRabbit

  • Bug Fixes

    • Improved cancellation handling for deferred matrix and reusable workflow jobs, ensuring requests are settled cleanly.
    • Preserved successful results from completed nested workflow calls when remaining work is cancelled.
    • Fixed workflow concurrency overflow cleanup so deferred jobs are marked cancelled promptly.
    • Prevented cancelled operations from overwriting completed successful outcomes.

  • Tests

    • Added regression coverage for nested workflow cancellation and successful result preservation.