Benchmarking Agents

Relevant source files

The following files were used as context for generating this wiki page:

• loom-py/src/loom/benchmark/README.md
• loom-py/src/loom/cli/benchmark.py
• loom-py/src/loom/benchmark/metrics.py
• ROADMAP.md
• loom-py/tests/unit/test_benchmark.py
• loom-py/src/loom/benchmark/init.py

The benchmark module provides the infrastructure for quantitative evaluation of Loom agents against industry-standard benchmarks. Its primary purpose is to measure agent performance, calculate the impact of context engineering on token usage and cost, and verify the correctness of agent-generated solutions. This system is crucial for analyzing cost/quality trade-offs and ensuring that optimizations like context reduction and compaction do not negatively affect task completion rates.

The strategic focus for benchmarking is on general AI assistant tasks (GAIA) and multi-agent tool coordination (TAU-bench), as these align better with Loom's core runtime features than one-shot coding tasks. While an MVP for SWE-bench exists, it is considered a lower priority.

Sources: loom-py/src/loom/benchmark/README.md:5-13, ROADMAP.md:100-110

Architecture and Data Flow

The benchmark system is composed of several key components that work together to execute tasks and report results. The BenchmarkRunner orchestrates the process, using a dataset-specific Adapter to load and prepare tasks. The agent performs the task, and its performance is captured in TaskMetrics, which are then aggregated into BenchmarkResults.

Sources: loom-py/src/loom/benchmark/README.md:175-184

This diagram illustrates the data flow through the benchmark system.

Components

• BenchmarkRunner (runner.py): The main orchestrator that executes benchmark tasks, manages agent lifecycles, and collects metrics.
• Metrics (metrics.py): Contains data classes (TaskMetrics, BenchmarkResults, ComparisonReport) for capturing, aggregating, and serializing performance data.
• Adapters (adapters/): Dataset-specific modules responsible for loading tasks, preparing the execution environment, and evaluating results. An adapter for swe_bench is provided as an example.
• CLI (cli/benchmark.py): Provides command-line access to run benchmarks, compare results, and generate reports.

Sources: loom-py/src/loom/benchmark/README.md:175-181, loom-py/src/loom/cli/benchmark.py

Command-Line Interface

The loom benchmark command provides a suite of tools for running evaluations from the terminal.

Sources: loom-py/src/loom/cli/benchmark.py:18-24

loom benchmark run

Executes benchmark tasks for a given agent and dataset, collecting metrics.

Usage:

loom benchmark run swe-bench \
    --agent-path ./apps/chat-assistant \
    --dataset ./datasets/swe-bench-lite \
    --max-tasks 3 \
    --output results/baseline.json \
    --verbose

Sources: loom-py/src/loom/benchmark/README.md:16-22

Arguments:

Argument	Description	Required	Default
benchmark	The name of the benchmark to run (e.g., 'gaia', 'tau-bench').	Yes	-
--agent-path	Path to the agent project directory containing loom.toml.	Yes	-
--dataset	Path to the benchmark dataset.	Yes	-
--max-tasks	Maximum number of tasks to run.	No	10
--no-context-engineering	Disables context engineering to run a baseline.	No	false
--output	Path to save the output results JSON file.	No	-
--workspace	Workspace path for task execution.	No	-
-v, --verbose	Print detailed progress during the run.	No	false

Sources: loom-py/src/loom/cli/benchmark.py:32-80

loom benchmark compare

Runs a benchmark twice—once with context engineering and once without—to generate a comparison report.

Usage:

loom benchmark compare swe-bench \
    --agent-path ./apps/chat-assistant \
    --dataset ./datasets/swe-bench-lite \
    --max-tasks 5 \
    --output results/comparison.json

Sources: loom-py/src/loom/benchmark/README.md:28-34

This sequence diagram shows the flow of the compare command.

loom benchmark report

Generates a human-readable report (console or markdown) from a saved results file.

Usage:

loom benchmark report results/comparison.json --format markdown

Sources: loom-py/src/loom/benchmark/README.md:37-42

Arguments:

Argument	Description	Required	Default
input	Path to the benchmark results JSON file.	Yes	-
--format	The output format for the report.	No	console
--output	Path to save the output markdown file.	No	-

Sources: loom-py/src/loom/cli/benchmark.py:122-139

Metrics Collection

The framework collects a detailed set of metrics for each task, which are then aggregated to provide a summary of the entire benchmark run.

Sources: loom-py/src/loom/benchmark/metrics.py:7-10

Metrics Data Model

The core data structures for metrics are TaskMetrics, BenchmarkResults, and ComparisonReport.

This class diagram shows the relationships between the metrics classes.

Per-Task Metrics (TaskMetrics)

The TaskMetrics dataclass captures performance data for a single task run.

Metric	Description
task_id	Unique identifier for the task.
success	Whether the agent completed the task without errors.
iterations	Number of cognitive loop iterations.
total_tokens	Total tokens used (prompt + completion).
prompt_tokens	Total input tokens sent to the LLM.
completion_tokens	Total output tokens received from the LLM.
total_cost_usd	Estimated cost based on LLM pricing (e.g., DeepSeek at $0.14/$0.28 per 1M tokens).
execution_time_sec	Wall-clock execution time.
offloaded_files	Number of files created for data offloading.
compacted_steps	Number of thought/action steps compressed in the history.
token_savings_pct	Estimated token reduction percentage due to context engineering.
correct_answer	Whether the agent's solution passed automated tests or other verification.

Sources: loom-py/src/loom/benchmark/README.md:61-75, loom-py/src/loom/benchmark/metrics.py:13-46, ROADMAP.md:95-96

Programmatic Usage

In addition to the CLI, the benchmark runner can be used programmatically for more complex evaluation scenarios or integration into a CI/CD pipeline.

Basic Benchmark Run

from loom.benchmark import BenchmarkRunner
from loom.cognitive import CognitiveAgent, CognitiveConfig
from loom.llm import LLMProvider
from pathlib import Path

# Initialize LLM
llm = LLMProvider.from_name("deepseek")

# Create agent factory
def agent_factory(ctx, llm_provider):
    return CognitiveAgent(
        ctx=ctx,
        llm=llm_provider,
        config=CognitiveConfig(
            system_prompt="You are a software engineering assistant.",
            max_iterations=20,
        ),
        workspace_path=Path("./workspace"),
    )

# Create runner
runner = BenchmarkRunner(
    agent_factory=agent_factory,
    llm=llm,
    benchmark="swe-bench",
    dataset_path="./datasets/swe-bench-lite",
)

# Run benchmark
results = await runner.run(max_tasks=50, verbose=True)

# Save results
results.save(Path("results/run.json"))

Sources: loom-py/src/loom/benchmark/README.md:97-133

Supported Benchmarks

The framework is designed to be extensible with new benchmarks via custom adapters. The current priorities are aligned with evaluating Loom's core strengths in general assistance and multi-agent collaboration.

Benchmark	Status	Description
GAIA	📋 Planned (P0)	General AI assistant tasks to evaluate tool coordination.
TAU-bench	📋 Planned (P0)	Multi-agent scenarios to test EventBus and collaboration.
SWE-bench	✅ Implemented (MVP)	Software engineering tasks from GitHub issues. (Lower priority).
WebArena	📋 Planned	Real-world web interaction tasks.

Sources: loom-py/src/loom/benchmark/README.md:151-173, ROADMAP.md:100-110

Conclusion

The benchmarking system is a core component for validating the development of Loom agents. It provides robust, quantitative feedback on performance, cost, and quality. By enabling direct comparisons, it proves the value of advanced features like context engineering, ensuring that token and cost reductions are achieved without sacrificing the agent's ability to correctly complete its tasks. The framework's command-line and programmatic interfaces make it a versatile tool for developers and for automated quality assurance.