Testing Strategies

Relevant source files

The following files were used as context for generating this wiki page:

• loom-py/tests/README.md
• core/tests/README.md
• loom-py/src/loom/benchmark/README.md
• core/src/dashboard/README.md
• core/tests/dashboard_unit_test.rs
• loom-py/tests/unit/test_cli_ui.py
• core/tests/filesystem_tool_test.rs
• core/tests/mcp_test.rs
• loom-py/tests/unit/test_compactor.py
• loom-py/tests/unit/test_tool_descriptor.py

The Loom project employs a multi-layered testing strategy to ensure the reliability, performance, and correctness of its event-driven, multi-language architecture. This strategy encompasses unit, integration, end-to-end, pressure, and benchmark tests, covering both the Rust Core and the Python SDK. The primary goal is to validate the "Brain/Hand" separation, ensuring that the stable Rust infrastructure and the fast-iterating Python cognitive components work together seamlessly.

Testing is integral to the development lifecycle, with tests running automatically on every commit and pull request via GitHub Actions. This comprehensive approach validates everything from individual functions to complex multi-agent collaboration scenarios, context engineering pipelines, and user-facing CLI outputs.

Sources: ROADMAP.md, loom-py/tests/README.md:3-5, core/tests/README.md:3-4

Testing Layers

Loom's testing is structured into several distinct layers, each with a specific focus. This layered approach ensures that issues are caught at the most appropriate level, from isolated component logic to system-wide performance.

This diagram shows the hierarchy of testing layers within the Loom project.

• Unit Tests: Validate the smallest parts of the application in isolation, such as individual functions or classes.
• Integration Tests: Verify the interaction between different components, like the ToolOrchestrator and ActionBroker.
• End-to-End (E2E) Tests: Test the entire application flow, from an agent receiving an event to executing a tool and producing a result.
• Pressure Tests: Stress-test components like the EventBus to measure throughput, latency, and behavior under heavy load.
• Benchmark Tests: Quantitatively evaluate agent performance against industry-standard benchmarks like GAIA and TAU-bench.

Sources: core/tests/README.md:6-20, loom-py/tests/README.md:12-40, loom-py/src/loom/benchmark/README.md:6-16

Rust Core Testing

The Rust Core ("Hands") is tested extensively to guarantee the stability of the underlying infrastructure. Tests are organized to mirror the source code structure, with each module in core/src/ having a corresponding <module>_test.rs file.

Sources: core/tests/README.md:3-5

Unit & Integration Test Structure

Unit tests focus on specific modules, while integration tests validate the collaboration between them. The tests are organized into submodules for clarity, especially for complex areas like pressure and end-to-end testing.

The following table summarizes the key test files and their coverage:

Test File	Source Module	Coverage
event_test.rs	src/event.rs	EventBus pub/sub, QoS, backpressure
action_broker_test.rs	src/action_broker.rs	Capability registration, invocation, timeout
agent_runtime_test.rs	src/agent/runtime.rs	Agent lifecycle, mailbox distribution
tool_orchestrator_test.rs	src/llm/tool_orchestrator.rs	Tool call parsing, ActionBroker integration
filesystem_tool_test.rs	tools/native/	ReadFileTool, WriteFileTool, ListDirTool, DeleteFileTool
mcp_test.rs	tools/mcp/	MCP server configuration, manager, and JSON-RPC structures
dashboard_unit_test.rs	src/dashboard/	EventBroadcaster, FlowTracker, TopologyBuilder
integration_test.rs	Core Pipeline	End-to-end event → agent → action → result flow

Sources: core/tests/README.md:8-20, core/tests/filesystem_tool_test.rs, core/tests/mcp_test.rs, core/tests/dashboard_unit_test.rs

A typical Rust unit test uses the #[tokio::test] attribute for async support and often involves creating temporary resources to ensure isolation.

// Source: core/tests/filesystem_tool_test.rs:17-34
#[tokio::test]
async fn test_write_file_creates_file() {
    let workspace = create_temp_workspace();
    let tool = WriteFileTool::new(workspace.path().to_path_buf());

    let result = tool
        .call(json!({
            "path": "test.txt",
            "content": "Hello, World!"
        }))
        .await;

    assert!(result.is_ok());
    let value = result.unwrap();
    assert_eq!(value["success"], true);
    assert_eq!(value["bytes_written"], 13);

    // Verify file exists
    let content = fs::read_to_string(workspace.path().join("test.txt"))
        .await
        .unwrap();
    assert_eq!(content, "Hello, World!");
}

Pressure Testing

Pressure tests are a critical part of ensuring the EventBus can handle high-load scenarios. These tests are run serially to avoid resource conflicts and measure key performance indicators.

Performance Targets:

Metric	Target	Status
Throughput	10k events/sec	✅ ~175k events/sec
P50 Latency	<100ms	✅ <1ms
P99 Latency	<500ms	✅ ~1ms

These tests are modularized under core/tests/pressure/ and cover throughput, QoS behavior, backpressure, and latency.

Sources: core/tests/README.md:100-120

Dashboard Testing

The real-time dashboard has its own suite of unit and integration tests to ensure observability components are reliable.

This diagram shows the components of the dashboard that are unit tested.

• EventBroadcaster: Tests SSE event broadcasting to subscribers.
• FlowTracker: Validates the tracking of communication flows between agents and the cleanup of old data.
• TopologyBuilder: Ensures the correct generation of agent topology snapshots from the AgentDirectory.
• DashboardConfig: Verifies environment variable parsing and default values.

Sources: core/src/dashboard/README.md:104-118, core/tests/dashboard_unit_test.rs

Python SDK Testing

The Python SDK (loom-py), which forms the "Brain", is tested using pytest. The tests are organized into unit, integration, and e2e directories, focusing on context engineering, the cognitive loop, and the CLI.

Sources: loom-py/tests/README.md:3-5, loom-py/pyproject.toml:60-75

Test Organization and Coverage

The Python test suite is comprehensive, with a clear separation between unit and integration tests.

Module	Unit Tests	Integration Tests	Total
context/step.py	10	0	10
context/reducer.py	22	2	24
context/compactor.py	17	2	19
context/offloader.py	26	2	28
cognitive/agent.py	13	2	15
cli/chat.py & cli/ui.py	35+	3	38+

Note: Test counts are derived from loom-py/tests/README.md, test_compactor.py, and test_cli_ui.py and may vary as the codebase evolves.

Sources: loom-py/tests/README.md:43-53, loom-py/tests/unit/test_compactor.py, loom-py/tests/unit/test_cli_ui.py

Test Design Principles

The Python tests adhere to a set of design principles to ensure they are effective and maintainable:

1. Test Public Interface: Focus on observable behavior rather than internal implementation details.
2. Test Integration Points: Validate the data flow and contracts between different components.
3. Test Edge Cases: Cover optional features and backward compatibility.
4. Test User-Facing Behavior: Verify what the user sees, such as CLI output.

Sources: loom-py/tests/README.md:101-128

A typical Python test uses pytest fixtures to create reusable test data and capsys to capture and assert CLI output.

# Source: loom-py/tests/README.md:123-128
# ✅ GOOD: Test what users see
def test_display_offloaded_step(capsys):
    print_stream_step_complete(step)
    captured = capsys.readouterr()
    assert "Data offloaded" in captured.out  # User sees this

Benchmarking Strategy

Loom includes a dedicated benchmark module to quantitatively evaluate agent performance and the impact of context engineering. This allows for rigorous analysis of cost, quality, and token savings.

Sources: loom-py/src/loom/benchmark/README.md:6-12

Benchmark Flow

The benchmark system follows a clear data flow from dataset to final report.

This diagram illustrates the data flow within the benchmark module.

Metrics Collected

The benchmark runner collects a wide range of metrics to provide a holistic view of agent performance.

Metric	Description
total_tokens	Total tokens used (prompt + completion)
total_cost_usd	Estimated cost based on actual pricing
execution_time_sec	Wall-clock execution time
offloaded_files	Files created for data offloading
compacted_steps	Steps compressed in history
token_savings_pct	Estimated token reduction percentage
correct_answer	Whether the solution passed automated tests

Sources: loom-py/src/loom/benchmark/README.md:58-74

Benchmarks can be executed via the loom CLI, allowing for comparisons between different agent configurations, such as with and without context engineering enabled.

# Run a benchmark
loom benchmark run swe-bench --max-tasks 3

# Compare baseline vs. optimized
loom benchmark compare swe-bench --max-tasks 5

Sources: loom-py/src/loom/benchmark/README.md:19-31

Running Tests

Tests can be executed using standard toolchains for Rust and Python.

Rust

# Run all core unit tests
cargo test --lib --tests

# Run a specific test file
cargo test --test event_pressure_test

# Run pressure tests serially to avoid resource conflicts
cargo test --test event_pressure_test -- --test-threads=1 --nocapture

Sources: core/tests/README.md:86-98

Python

# Run all Python tests
cd loom-py
pytest

# Run specific test groups with verbose output
pytest tests/unit/test_step.py::TestStepAttributes -v

# Run tests with coverage report
pytest tests/ --cov=loom.context --cov-report=html

Sources: loom-py/tests/README.md:58-81

Conclusion

Loom's testing strategy is a cornerstone of its development process, providing a robust framework for ensuring quality and performance. By combining unit, integration, E2E, pressure, and benchmark testing across both its Rust core and Python SDK, the project maintains a high degree of confidence in its complex, asynchronous, and multi-agent architecture. This comprehensive approach enables rapid iteration while safeguarding the stability and correctness of the entire system.