feat(trace): capture Codex skills and spec materials

Merged
MI
10h ago
+1975-23

Codex sessions now capture used skill definitions and supporting files. They store each distinct pasted image once and replace supported spec copies with material pointers.

Pointers replace copies in viewer messages and retained native events. Images remain inline if decoding or storage fails.

Skills and spec pointers remain Partial in the compatibility table. Skill capture covers filesystem skills in the root session. Transformed output, mixed or computed patches, and unsupported formats remain inline. Child sessions remain outside scope.

Code-mode patch replacement accepts one awaited tools.apply_patch call with a literal argument or an immediately preceding const literal. The call can use the text wrapper. Other scripts remain inline because their patch arguments can change before the write.

Failure checks inspect JSON text, nested command results, promise results, and tool error flags. Failed writes retain their original transcript content. Valid source documents that start with Error: still support pointers.

The author requested an update to the current milestone 2 spec. R-001 through R-006 remain unchanged. R-007 through R-010 and D-004 define milestone 2 and supersede the original exclusion. The spec update follows the eng-spec workflow.

Refs #3657. This PR delivers milestone 2. The parent issue remains open until both milestones pass their required checks.

tested manually too

image image

Checks:

  • make -C app/cli test passes, including the new regression cases for computed patches and failed writes.
  • Codex package lint and lint for the new changes pass. Full CLI lint reports existing violations outside the changed files.
  • A local test with an actual pasted screenshot stored one 345 KB PNG and replaced its inline copies with material pointers.
  • The existing live milestone 2 results cover skills in 3/3 sessions, spec pointers in 3/3 sessions, and pasted images in 1/1 session.
  • The existing full Codex suite passed all 25 tested feature checks with CLI 0.163.0 and openrouter/openai/gpt-6-luna.

Assisted-by: Codex

Provider checklist

PR #3684 delivered the core provider and CLI wiring. This PR adds the three optional extensions and connects them to session evidence. Codex has no dedicated skill tool. The existing lifecycle hooks scan the rollout for skill loads.

Provider package (app/cli/internal/trace/<agent>/)

  • Define Name and check var _ trace.Provider = (*Provider)(nil) at compile time.
  • Keep InstallHooks idempotent and preserve user hooks.
  • Omit the session-end hook from InstallHooksForTraceRun.
  • Return the installed settings path from SettingsFile.
  • Remove only Chainloop hooks in UninstallHooks.
  • Return true from HooksInstalled only for Chainloop hooks.
  • Support session-start, user-prompt, and session-end events where the agent provides them.
  • Support hooks before and after file and shell tools. Use existing lifecycle hooks for Codex skill tracking.
  • Read hook input into a typed struct with a size limit.
  • Require absolute FilePath, Cwd, and TranscriptPath values.
  • Read ToolUseID, AgentID, and Command. Read AgentVersion and Model when the transcript omits them.
  • Use one Go slice for each tool list in IsFileWritingTool and IsCommandTool, including any plugin list.
  • Select snapshots through CaptureFileSnapshot, ResolveBeforeContent, and CleanupAfterEdit, or reverse edits through HookInput.Edits.
  • Use the same ToolUseID for shell pre and post hooks.
  • Keep CopySessionData safe to repeat. Check payload paths. Preserve hook execution when the agent or transcript is missing.
  • Parse only the private copy in opts.SessionDir.
  • Keep CopySessionData and ParseSession independent of the control plane and network.
  • Access Trace state only through *state.Store so trace run --local works outside Git repositories.
  • Record the agent name and version, session ID and times, primary model, used models, and model provider.
  • Record input, output, total, cache-read, and cache-write tokens.
  • Use agent costs or an embedded price table. Use zero with a warning when neither source supplies a cost.
  • Record tools, conversation counts, and subagents where the agent supports them.
  • Keep RawSession["main"] non-nil. Each line must contain a JSON object.
  • Warn for each unreadable data item.
  • Return one JSON document for each nonempty session-start, prompt-submit, or post-tool response. Emit nothing for an empty message.
  • Report supported channels accurately through the Supports* methods.
  • Return trace.ErrAnnounceUnsupported from AnnounceToUser when no channel exists.
  • Implement SkillTracker, SpecCopyReplacer, and PastedImageFinder, with a compile-time check for each interface.
  • Test hooks, parsing with real testdata/, announcement responses, and copying.

CLI wiring

  • Register the provider in providers/registry.go.
  • Add the field and provider name to traceProviderFlags, any(), and names().
  • Add --<agent> to trace init and trace run, with help text.
  • Add chainloop trace hook <agent> … commands to cmd/trace_hook.go with skipActionOptsInit.
  • Emit the pre-tool response first if the agent requires it before tool execution.
  • Update providers/capabilities_test.go.
  • Update cmd/trace_init_providers_test.go and cmd/trace_init_test.go.
  • Check whether the CLI reference requires make generate. This PR leaves the commands and wiring from #3684 unchanged.

End-to-end tests and documentation

  • Implement e2e.Harness in e2e/<agent>.go.
  • Register the harness in e2e/e2e_test.go.
  • Record the version and models in e2e/config.yaml.
  • Run the suite for the provider column and check the results.
  • Add the provider column to each feature table in README.md, using suite results.
  • Explain each non-Yes cell under "Why not Yes" in README.md.
  • Update the provider list and the "How each agent is hooked in" table in README.md.
  • Add the provider spec in docs/specs/ and cite its requirement IDs in the PR.
  • Update the license header year in each changed file.
  • Pass full CLI lint and tests. Tests pass. Full lint reports existing violations elsewhere. Codex package lint and lint for the new changes pass.