Harbor Benchmark Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Harbor Benchmark Integration serves as the bridge between the plugsuits Code Editing Agent (CEA) and the terminal-bench@2.0 evaluation framework. Its primary purpose is to adapt the agent's internal event-driven execution into the Agent Trajectory Interchange Format (ATIF-v1.6) required for standardized benchmarking.
This integration allows for rigorous testing of the agent's ability to handle complex coding tasks, such as modernizing scientific stacks or performing compaction-heavy long-context operations. It relies on a specialized headless runner that emits structured JSONL events, which are then consumed and mapped to trajectories by a Python-based adapter.
Sources: packages/cea/benchmark/AGENTS.md:1-5, README.md:54-58, AGENTS.md:68-71
Architecture and Data Flow
The integration operates through a multi-stage pipeline where the agent runs within a Docker container, produces structured logs, and is subsequently validated by the Harbor framework.
Event Transformation Pipeline
The data flow follows a specific sequence from the internal runtime to the final validated trajectory:
- Execution: The
headless.tsrunner executes the task and emits JSONL events tooutput.jsonl. - Mapping: The
harbor_agent.pyadapter reads these events. - Conversion:
StepEventobjects are mapped directly to ATIF trajectory steps. - Validation: The resulting
trajectory.jsonis verified using Harbor's validation utilities.
The following diagram illustrates the flow of events from the headless runner to the final ATIF trajectory:
Brief explanation: This diagram shows the sequential transformation of internal JSONL events into the standardized ATIF-v1.6 trajectory format used by Harbor.
Sources: packages/cea/benchmark/AGENTS.md:32-46, packages/cea/benchmark/AGENTS.md:65-70
Event System
The integration relies on a structured JSONL event format. Every event is a single line of JSON with a mandatory type field.
Event Type Definitions
| Type | Description | Key Fields |
|---|---|---|
metadata | Emitted once at session start to identify the agent and session. | session_id, agent, timestamp |
step | Represents a sequential action (user or agent) in the ATIF trajectory. | step_id, source, message, tool_calls, metrics |
compaction | Logs history compaction events for debugging/analysis. | event, tokensBefore, tokensAfter, durationMs |
error | Signals a fatal termination of the agent. | error, timestamp |
Sources: packages/cea/benchmark/AGENTS.md:48-55, packages/cea/src/AGENTS.md:21-25
Trajectory Conversion Logic
The conversion from output.jsonl to trajectory.json is handled by harbor_agent.py. Key implementation constraints include:
- Sequential ID Management:
step_idis generated by the headless runner, not the adapter, ensuring strict incrementing sequences. - Observation Bundling: Tool results are not separate steps; they are captured and included in the
observationfield of the step immediately following the tool call. - Metric Accuracy: Token counts and execution metrics are pulled directly from the AI SDK
metricsfield rather than being estimated.
Sources: packages/cea/benchmark/AGENTS.md:72-83, AGENTS.md:120-123
Configuration and Environment
The behavior of the agent during benchmarking is controlled via specific environment variables passed to the Harbor runner.
| Variable | Effect |
|---|---|
AGENT_ENABLE_THINKING | Enables the --think flag, allowing the agent to capture and export reasoning_content in the trajectory. |
AGENT_ENABLE_TOOL_FALLBACK | Enables the --tool-fallback flag, which uses XML-based tool calling for models without native tool support. |
The integration also supports a CONTEXT_LIMIT_OVERRIDE when COMPACTION_DEBUG=1 is set, which is crucial for testing the agent's performance in "compaction stress" tasks where the context window is intentionally constrained (e.g., to 6,000 or 32,000 tokens).
Sources: packages/cea/benchmark/AGENTS.md:23-30, README.md:143-154, packages/cea/benchmark/tasks/compaction-stress-search/instruction.md:1-5
Verification and Validation
Post-execution, the integration provides specific commands to verify the integrity of the benchmark run:
- Event Distribution:
jq -r '.type' | sort | uniq -con the output file should show exactly one metadata event and zero error lines for a successful run. - Sequence Check: Step IDs must be strictly increasing.
- Schema Validation: The
harbor.utils.trajectory_validatortool is used to ensure the final JSON file adheres to ATIF-v1.6 specifications.
Sources: packages/cea/benchmark/AGENTS.md:57-63, packages/cea/benchmark/AGENTS.md:85-87
Summary
The Harbor Benchmark Integration is a critical component for the plugsuits project, providing a standardized way to measure agent performance. By decoupling the agent's internal logic from the benchmark's reporting requirements via a structured JSONL event stream, it ensures that metrics like token usage, reasoning depth, and tool accuracy are captured with high fidelity. This architecture allows the Code Editing Agent to be evaluated across diverse models and providers within the Harbor ecosystem.
Sources: packages/cea/benchmark/AGENTS.md:1-5, README.md:24-34