Monitoring & Logging

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• ethos-server/src/main.rs
• ethos-server/src/http.rs
• ethos-cli/src/main.rs
• ethos-core/src/config.rs
• ethos-server/src/subsystems/consolidate.rs
• ethos-server/src/subsystems/decay.rs

Ethos implements a multi-layered monitoring and logging strategy designed for production-grade reliability. This system encompasses structured tracing, real-time health checks, and detailed reporting from background subsystems such as the consolidation and decay engines. The primary goal is to provide visibility into the "neuromorphic" processes—remembering, consolidating, and forgetting—ensuring that the memory service remains healthy and performant.

Monitoring is facilitated through a dedicated HTTP REST API and a CLI utility, both of which allow operators to query the system state. Structured logs are emitted using the tracing ecosystem, providing granular insights into internal operations, while background tasks generate detailed reports on memory lifecycle events.

Sources: README.md:52, ethos-server/src/http.rs:13-22

System Health Monitoring

Ethos provides comprehensive health monitoring through the ethos-server binary and the HTTP API. This includes verifying connectivity to the PostgreSQL database and the availability of the pgvector extension, which is critical for semantic search.

Health Check Mechanisms

The system supports two primary ways to perform a health check:

1. CLI Check: Running ethos-server --health performs an immediate validation of the database connection and extension versions before exiting.
2. API Check: A GET request to /health returns a JSON payload containing the current status of the server, database version, and socket path.

Sources: ethos-server/src/main.rs:43-67, ethos-server/src/http.rs:96-126

Health Status Schema

The health check endpoint provides a detailed view of the system's infrastructure:

Field	Type	Description
status	String	Global health status (e.g., "healthy", "unhealthy")
version	String	Current Ethos server version
postgresql	String	PostgreSQL version string
pgvector	String	Version of the pgvector extension
socket	String	Path to the Unix domain socket for IPC

Sources: ethos-server/src/http.rs:114-124, ethos-cli/src/main.rs:179-183

Structured Logging & Tracing

The Ethos server utilizes the tracing and tracing-subscriber crates to provide structured, levels-based logging. The logging level is configurable via environment variables (e.g., RUST_LOG) or the ethos.toml configuration file.

Subsystem Reporting

Background subsystems, specifically those managing the memory lifecycle, generate Report structures that summarize the work performed during a cycle. These reports are emitted to the logging system to provide visibility into background data processing.

Consolidation Reporting

The consolidation engine (DMN) processes raw episodic traces into semantic facts. Every cycle produces a ConsolidationReport:

• Episodes Scanned: Count of candidates evaluated for promotion.
• Episodes Promoted: Count of episodes successfully turned into facts.
• Fact Statistics: Breakdown of facts created, updated, superseded, or flagged for review.

Sources: ethos-server/src/subsystems/consolidate.rs:33-41, ethos-server/src/subsystems/consolidate.rs:104-108

Decay Sweep Reporting

The decay engine, which manages Ebbinghaus decay and Long-Term Potentiation (LTP), tracks the pruning of memories. The DecaySweepReport includes:

• Updated Count: Records whose salience was recalculated.
• Pruned Count: Records that fell below the prune_threshold and were soft-deleted.
• Elapsed Time: Total time in milliseconds the sweep took to complete.

Sources: ethos-server/src/subsystems/decay.rs:36-44, ethos-server/src/subsystems/decay.rs:64-73

External Monitoring Tools

Ethos CLI Status

The ethos-cli utility acts as a monitoring client, allowing developers to check the status of a running server from the terminal.

// ethos-cli/src/main.rs:168-175
fn do_status(server: &str) -> anyhow::Result<()> {
    let client = reqwest::blocking::Client::builder()
        .timeout(std::time::Duration::from_secs(10))
        .build()?;

    let url = format!("{}/health", server);
    let resp = client.get(&url).send();
    // ... processes and prints status fields
}

Sources: ethos-cli/src/main.rs:168-197

Performance Tracking

The HTTP API search endpoint tracks request latency using the took_ms field. This provides real-time performance monitoring for semantic retrieval operations.

Configuration for Monitoring

Monitoring behavior is influenced by several parameters in the EthosConfig structure, particularly thresholds for background tasks and service connection paths.

Parameter	Type	Context	Description
log_level	String	Service	Global level for tracing output (INFO, DEBUG, etc.)
socket_path	String	Service	The IPC path monitored for connections
idle_threshold_seconds	u64	Consolidation	Minimum inactivity before monitoring triggers DMN
cpu_threshold_percent	u8	Consolidation	CPU ceiling; tasks won't run if system is busy

Sources: ethos-core/src/config.rs:19-55, ethos-server/src/subsystems/consolidate.rs:125-156

Monitoring and logging in Ethos are designed to be "fire-and-forget" for the ingest pipeline while providing high-fidelity diagnostic data for the server's internal memory management processes. By combining structured logs, health endpoints, and detailed lifecycle reports, operators can ensure the neuromorphic engine operates correctly within an AI agent's infrastructure.