Consolidation Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Consolidation Engine is a background subsystem in Ethos responsible for the "episodic to semantic promotion" process. Inspired by the human brain's Default Mode Network (DMN), it periodically distills raw conversation turns—stored as episodic traces—into durable, structured facts called semantic facts. This process ensures that important decisions, user preferences, and frequently accessed information are preserved as long-term knowledge while reducing noise in the memory system.
The engine typically runs every 15 minutes when the system is idle. It evaluates episodic traces based on importance scores, retrieval frequency, and specific keyword patterns (e.g., "we decided to", "I prefer"). Once a fact is extracted, the engine performs conflict resolution to determine if the new information should update, supersede, or be flagged against existing knowledge.
Sources: README.md:16-20, ethos-server/src/subsystems/consolidate.rs:1-15
Architecture and Data Flow
The Consolidation Engine operates as a background loop within the ethos-server. It monitors the state of the database and system load to ensure consolidation occurs during idle periods to minimize impact on real-time conversation ingestion.
The diagram above illustrates the high-level flow of data from raw episodic traces to structured semantic facts.
Sources: README.md:92-105, ethos-server/src/subsystems/consolidate.rs:80-120
System Idle Detection
To prevent performance degradation during active use, the engine checks for system idleness before starting a cycle. A system is considered idle if:
- No new
session_eventshave been recorded within a configured threshold (default 60 seconds). - The CPU load (via
/proc/loadavg) is below a specified percentage (default 80%).
Sources: ethos-server/src/subsystems/consolidate.rs:134-168, ethos-core/src/config.rs:46-56
Fact Extraction Logic
Fact extraction uses rule-based pattern matching (without requiring an LLM in the primary loop) to identify significant information within episodic traces.
Promotion Criteria
Episodes are scanned for promotion if they meet any of the following conditions:
- Importance:
importance >= 0.8. - Repetition: High retrieval count (
retrieval_count >= 5). - Keywords: Presence of decision keywords ("decided", "plan is"), preference keywords ("prefers", "hates", "favorite"), or explicit markers ("remember this", "important:").
Sources: ethos-server/src/subsystems/consolidate.rs:213-248, ethos-core/src/config.rs:58-67
Pattern Categories
| Category | Example Patterns | Resulting Predicate |
|---|---|---|
| Decision | "we decided to use [object]", "let's go with [object]" | uses, plan |
| Preference | "[subject] prefers [object]", "[subject] hates [object]" | prefers, loves, hates |
| Explicit | "remember this: [statement]", "important: [statement]" | is |
Sources: ethos-server/src/subsystems/consolidate.rs:260-320
Conflict Resolution
When a new fact is extracted, the engine compares it against existing semantic_facts sharing the same subject and predicate.
The conflict resolution flow ensures data integrity when new information contradicts old knowledge.
Sources: ethos-server/src/subsystems/consolidate.rs:372-422, ethos-core/src/config.rs:88-91
Resolution Types
- Refinement: If objects are compatible (e.g., "Rust" and "Rust language"), the existing fact is updated with merged content and a confidence boost.
- Supersession: If a new explicit decision is made or if the new fact has significantly higher confidence (delta > 0.15), the old fact is marked as
superseded_bythe new ID. - Flagging: If a contradiction is ambiguous, both facts are marked
flagged_for_review, and an entry is written to a Markdown review inbox.
Sources: ethos-server/src/subsystems/consolidate.rs:425-460, migrations/001_initial_schema.sql:107-111
Configuration Parameters
Consolidation behavior is governed by the ConsolidationConfig and ConflictResolutionConfig structures.
| Option | Type | Default | Description |
|---|---|---|---|
interval_minutes | u64 | 15 | Frequency of the background consolidation loop. |
idle_threshold_seconds | u64 | 60 | Required silence in session events to trigger. |
cpu_threshold_percent | u8 | 80 | Max CPU load allowed for consolidation to start. |
importance_threshold | f32 | 0.8 | Minimum importance score for automatic promotion. |
retrieval_threshold | u32 | 5 | Minimum retrievals for automatic promotion. |
auto_supersede_confidence_delta | f64 | 0.15 | Required confidence gap to automatically replace a fact. |
review_inbox | String | - | Path to the Markdown file for flagged conflicts. |
Sources: ethos-core/src/config.rs:46-70, ethos-core/src/config.rs:88-91
Integration with Decay
Immediately following a successful consolidation cycle, the engine triggers a Decay Sweep. This process applies Ebbinghaus forgetting curves to all memory types, including the newly created semantic facts. This ensures that while facts are "durable," they still require periodic reinforcement (retrieval) to maintain high salience and avoid being pruned.
Sources: ethos-server/src/subsystems/consolidate.rs:104-107, ethos-server/src/subsystems/decay.rs:37-50
API Endpoints
While typically automated, consolidation can be manually triggered via the HTTP API.
POST /consolidate
Triggers an immediate consolidation cycle, bypassing the idle check.
- Payload:
ConsolidateRequest { session: Option<String>, reason: Option<String> } - Response:
ConsolidationReportcontaining counts of episodes scanned, promoted, and facts created/updated.
Sources: ethos-server/src/http.rs:191-213, ethos-server/src/subsystems/consolidate.rs:33-46
Summary
The Consolidation Engine serves as the long-term memory architect of Ethos. By transforming transient episodic traces into structured semantic facts, it enables AI agents to retain critical knowledge across sessions. Its robust conflict resolution and integration with the Ebbinghaus decay system create a dynamic memory environment where important information is promoted and refined, while contradictions are safely managed or surfaced for human review.