Observability and Monitoring

Relevant source files

The following files were used as context for generating this wiki page:

• observability/README.md
• observability/docker-compose.observability.yaml
• observability/otel-collector-config.yaml
• observability/grafana/dashboards/loom-overview.json
• core/examples/mcp_integration.rs
• core/src/dashboard/README.md
• loom-py/src/loom/cli/README.md

Loom provides a comprehensive, OpenTelemetry-based observability stack for monitoring the health and performance of the runtime and the agents it orchestrates. This stack offers distributed tracing, metrics collection, and visualization through industry-standard tools. Alongside this, Loom includes a built-in, real-time dashboard for immediate, terminal-adjacent insights into event flows and agent activity.

This document outlines the architecture of both the full observability stack and the built-in dashboard, details the available metrics, and provides guidance on configuration and usage.

Sources: observability/README.md, core/src/dashboard/README.md

Observability Stack Architecture

The primary observability solution is built around a collection of standard, interoperable tools that receive telemetry data from Loom Core and Python agents via the OpenTelemetry Protocol (OTLP).

The diagram below illustrates the data flow from a Loom application to the various components of the observability stack.

Components

The stack is defined in docker-compose.observability.yaml and consists of four main services:

Service	Port(s)	Image	Description
OTel Collector	4317 (gRPC), 4318 (HTTP), 8889 (Prometheus Metrics)	otel/opentelelemetry-collector	Central hub that receives, processes, and exports telemetry data.
Jaeger	16686 (UI)	jaegertracing/all-in-one	Provides a UI for visualizing end-to-end distributed traces.
Prometheus	9090 (UI)	prom/prometheus	A time-series database (TSDB) for storing and querying metrics using PromQL.
Grafana	3000 (UI)	grafana/grafana	A visualization platform for creating dashboards from Prometheus metrics and Jaeger traces.

Sources: observability/docker-compose.observability.yaml:7-41, observability/README.md:48-101

OpenTelemetry Collector Configuration

The OTel Collector is the pipeline's entry point. Its configuration in otel-collector-config.yaml defines how data is received, processed, and exported.

This diagram shows the OTel Collector's internal pipelines for traces and metrics.

• Receivers: It listens for OTLP data over gRPC (:4317) and HTTP (:4318).
• Processors: It batches data before export for efficiency.
• Exporters: It sends traces to Jaeger and exposes metrics for Prometheus to scrape.
• Service Pipelines: Separate pipelines are defined for traces and metrics, routing data from receivers to the appropriate exporters.

Sources: observability/otel-collector-config.yaml:1-41

Built-in Loom Dashboard

For more immediate, real-time feedback without a full Docker stack, Loom Core provides a built-in web dashboard. It focuses on live event streams and agent topology.

Sources: core/src/dashboard/README.md:3-5

Dashboard Architecture

The dashboard runs within the Loom Core process and serves a pre-built React frontend.

• EventBroadcaster: Publishes a stream of events from the EventBus over Server-Sent Events (SSE).
• FlowTracker: Records and aggregates communication flows between components, pruning edges after 60 seconds of inactivity.
• TopologyBuilder: Reads the AgentDirectory to provide a snapshot of registered agents and their subscriptions.
• React Frontend: A Vite-built single-page application that consumes the API endpoints to render the live view.

Sources: core/src/dashboard/README.md:30-39

Dashboard Configuration

The built-in dashboard is disabled by default and can be enabled and configured via environment variables.

Environment Variable	Default	Description
LOOM_DASHBOARD	false	Set to true to enable the dashboard.
LOOM_DASHBOARD_HOST	127.0.0.1	Bind address for the HTTP server.
LOOM_DASHBOARD_PORT	3030	Listening port for the dashboard UI.

Sources: core/src/dashboard/README.md:55-62

Telemetry Implementation

Telemetry can be enabled in both Rust and Python applications by setting a few environment variables.

Enabling Telemetry

To enable telemetry, you must set the OTEL_EXPORTER_OTLP_ENDPOINT variable. The init_telemetry() function in Rust or the equivalent setup in Python will then establish a connection to the OTel Collector.

Rust Example (mcp_integration.rs):

// core/examples/mcp_integration.rs:18-27
// Set OpenTelemetry environment variables (optional - comment out to disable)
std::env::set_var("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317");
std::env::set_var("OTEL_SERVICE_NAME", "loom-mcp-example");

// Initialize telemetry BEFORE creating Loom
// This will set up both logging and OpenTelemetry tracing
if std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").is_ok() {
    loom_core::telemetry::init_telemetry()
        .map_err(|e| format!("Failed to initialize telemetry: {}", e))?;
}

Configuration Variables

Variable	Default	Description
OTEL_EXPORTER_OTLP_ENDPOINT	http://localhost:4317	The gRPC endpoint of the OTel Collector.
OTEL_SERVICE_NAME	loom-core	The service name that appears in traces and metrics.
OTEL_TRACE_SAMPLER	always_on	The trace sampling strategy (e.g., always_on, always_off, traceidratio=0.1).
DEPLOYMENT_ENV	development	An environment label added to telemetry data.

Sources: observability/README.md:144-152, loom-py/src/loom/cli/README.md:162-165

Metrics and Dashboards

Loom Core is instrumented to export a wide range of metrics, which are visualized in a pre-configured Grafana dashboard.

Key Metrics

The following metrics provide insight into the core components of the Loom runtime.

Component	Metric Name	Description
Event Bus	loom_event_bus_published_total	Total events published.
	loom_event_bus_delivered_total	Total events successfully delivered.
	loom_event_bus_dropped_total	Total events dropped.
	loom_event_bus_publish_latency	Histogram of event publishing latency.
Action Broker	loom_action_broker_invocations_total	Total tool/capability invocations.
	loom_action_broker_invoke_latency	Histogram of invocation latency.
	loom_action_broker_errors_total	Total errors during invocation.
Agent Runtime	loom_agent_runtime_agents_active	Current number of active agents.
MCP Manager	loom_mcp_manager_servers_active	Current number of active MCP server connections.
Router	loom_router_decisions_total	Total routing decisions made, labeled by route.

Note: Metrics may appear with a loom_loom_ prefix due to namespace configuration in the OTel collector.

Sources: observability/README.md:103-141

Grafana "Loom Overview" Dashboard

A pre-built dashboard is provisioned in Grafana to provide an at-a-glance overview of the system's health.

The dashboard layout includes the following panels:

This dashboard visualizes key metrics such as:

• Active Agents: loom_agent_runtime_agents_active
• Event Bus Throughput: rate(loom_event_bus_published_total[5m])
• Action Broker Latency: histogram_quantile(0.99, rate(loom_action_broker_invoke_latency_bucket[5m]))
• Routing Decisions: sum by (route) (loom_router_decisions_total)
• Tool Invocation Rate: rate(loom_action_broker_invocations_total[5m])
• Error Rate: sum(rate(loom_action_broker_errors_total[5m])) / sum(rate(loom_action_broker_invocations_total[5m]))

Sources: observability/grafana/dashboards/loom-overview.json:5-150

Quick Start

Follow these steps to launch the observability stack and view telemetry from a Loom application.

1. Start the Observability Stack:
   From the observability directory, run the Docker Compose file.

   cd observability
   docker compose -f docker-compose.observability.yaml up -d
   

2. Run Loom with Telemetry:
   Navigate to the core directory, set the required environment variables, and run an example.

   cd ../core
   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
   export OTEL_SERVICE_NAME=loom-example
   cargo run --example mcp_integration
   

3. View Your Data:

   • Jaeger (Traces): http://localhost:16686
   • Prometheus (Metrics): http://localhost:9090
   • Grafana (Dashboards): http://localhost:3000 (login: admin/admin)

Sources: observability/README.md:14-25