feat(export): build manifests from Dynamo request traces

CR
9d ago
+1547-55

Adds a second input to batchbench-export-plans: --dynamo-traces <dir|file|glob> reads the request_end rows a Dynamo frontend writes with request tracing enabled (dynamo.request.trace.v1, gzipped JSONL) and writes the same schema v2 manifest as the prompt-chain source. With this, replays of Dynamo-served models don't need a separate gateway capture.

How it works

  • Window. Trace files are named by upload time and can hold rows received much earlier, so every file is read and rows are windowed by request_received_ms (--start/--end, plus the link window before --start for link targets).
  • Retries. Rows that share an x_request_id are attempts of one request. The exporter keeps the last attempt that finished and counts the others.
  • Sessions. It uses the same strict-prefix rule as prompt chains, applied to input_sequence_hashes. Rows link only within one model, block size, principal and agent session id. Rows with different trace_block_size are never linked or merged.
  • Hashes. Dynamo hashes every full block, then the partial last block when tokens remain, so there are ceil(input_length / trace_block_size) hashes. The exporter splits off the partial hash, so it is never used for linking or cutting; it only seeds the tail block. Rows that hash only full blocks also load. A row with more hashes than that counts as malformed_hashes.
  • Blocks. Each prompt is cut wherever a chain of the same model and block size ends or diverges, using full blocks only. Each segment becomes a user block with seed = the segment's last hash and tokens = trace_block_size × its hash count, so shared prefixes replay with identical bytes.
  • Tail. The partial tail past the last full block becomes a final user block, not overhead. That keeps the replayed prompt at the recorded length, which matters when block sizes run to thousands of tokens. So overhead_tokens is 0.
  • Extensions. A request that extends the previous request in its trajectory repeats that request's emitted blocks, then a live assistant block of the previous output_tokens, then the new content. Only the newest reply is live. Earlier replies repeat as plain assistant blocks under their original seeds, and the agent replays an already-sent seed with the same bytes, real reply included. A reply that would not fit in the extension is generated (split where segments end) rather than replayed live, because a live block always replays the whole real reply. live_clamped counts these; it is typical for reasoning models, whose output_tokens include reasoning that clients don't send back.
  • Branches. Any other request in a trajectory starts with the blocks already sent for the longest prefix it shares with an earlier request. reset_before clears the agent's cache, so generated blocks come out the same from their seeds, but copies of earlier replies are generated rather than real.
  • Optional join. --analytics-table (needs --clickhouse-url) or --analytics-rows-jsonl joins on x_request_id (--analytics-id-column, default x_request_id). The analytics side is bounded by time, and the id is compared as a string. A match supplies the principal (user_id, else api_key_id) and the alias (in trajectory metadata). Without a join, the exporter runs on the traces alone, and --principal-id is rejected.
  • Stats. The summary line adds trace counters: files, unreadable files, skipped rows by reason, duplicate attempts, join matches, live and clamped blocks, and rows per block size.

docs/export-plans.md documents the source, the join, and the fidelity lost compared with prompt chains: no roles, no tools array, template overhead merged and double counted, block granularity, large blocks hiding divergence and splitting sessions, approximate reply length, no stream/max_tokens, and timing that starts at Dynamo.

link_parents now groups by a link_key property (the principal, for chain records). It also finds candidate parents with a set intersection, because trace chains run to thousands of hashes. The behaviour is unchanged for chain records.

Testing

  • python -m unittest discover -s tests: 37 tests pass, including the existing ones. The new tests generate cumulative hashes in the test, in the production ceil shape. They cover shared prefixes, divergent branches, partial tails (ceil, floor and exact shapes, plus a turn that fills the previous partial block), three-turn sessions, branch reuse, a 1056-token block size, two block sizes with colliding hashes, a row received before its file's upload time, a truncated file, retried attempts, and rows with and without x_request_id, joined and unjoined.
  • The exported manifest dry-runs in batchbench-agent with both live blocks replayed.
  • A synthetic scale check (19.5k rows, 12M hashes, 98 files) ran in about 3 s with about 200 MB RSS.