Decay & Pruning
Relevant source files
The following files were used as context for generating this wiki page:
The Decay & Pruning system in Ethos is a neuromorphic memory management subsystem inspired by the human hippocampus and the Ebbinghaus forgetting curve. Its primary purpose is to ensure that the memory engine remains performant and relevant by naturally fading unimportant information over time while reinforcing frequently accessed data through Long-Term Potentiation (LTP).
Memories in Ethos are not just stored; they exist in a state of constant flux. Raw conversation turns, episodic traces, and semantic facts all possess salience scores that diminish over time unless they are reinforced by retrieval. When a memory's salience or confidence falls below a specific threshold, it is marked as "pruned"—a soft-delete state that removes it from active context while preserving the underlying data.
Sources: README.md:16-24, ethos-server/src/subsystems/decay.rs:1-15
Architecture and Logic
The system operates through two primary mechanisms: Continuous Reinforcement (LTP) and Scheduled Decay Sweeps.
The Decay Formula
The salience of a memory at time $t$ is calculated using a modified Ebbinghaus formula:
$S(t) = S_0 \times e^{-t/\tau_{eff}} \times (1 + \alpha \times f) \times (1 + \beta \times E)$
| Variable | Description | Source |
|---|---|---|
| $S_0$ | Initial or current salience/importance. | decay.rs:3 |
| $t$ | Time since last access (measured in days). | decay.rs:4 |
| $\tau_{eff}$ | Effective decay time constant, boosted by retrieval count ($base_tau \times ltp_mult^{count}$). | decay.rs:5-6 |
| $\alpha$ | Frequency weight (Default: 0.3). | decay.rs:7 |
| $f$ | Normalized access frequency (retrieval count / days since creation). | decay.rs:8 |
| $\beta$ | Emotional weight (Default: 0.2). | decay.rs:9 |
| $E$ | Emotional tone (scale of 0.0 to 1.0). | decay.rs:10 |
Sources: ethos-server/src/subsystems/decay.rs:1-15, README.md:38-40
Spreading Activation and LTP
When a memory is retrieved, the record_retrieval function is called. This implementation of Long-Term Potentiation (LTP) increases the retrieval_count, updates the last_retrieved_at timestamp, and applies a direct salience boost (typically 5-10%).
The retrieval process directly counters the decay effect, extending the memory's half-life significantly with each access.
Sources: ethos-server/src/subsystems/decay.rs:60-112, README.md:42-47
Decay Sweeps
Decay sweeps are periodic background tasks that iterate through the memory tables to update salience scores and prune stale data.
Sweep Execution Flow
A sweep is typically triggered by the run_consolidation_loop every 15 minutes when the system is idle. It processes three primary tables in order: memory_vectors, episodic_traces, and semantic_facts.
Pruning Criteria
A memory item is marked as pruned (soft-deleted) if it meets any of the following conditions:
- Salience Threshold: The calculated salience (or confidence for facts) falls below the
prune_threshold(default 0.05). - Explicit Expiration: For
memory_vectors, if theexpires_attimestamp is in the past. - Supersession: For
semantic_facts, if a newer fact has superseded the old one during consolidation.
Sources: ethos-server/src/subsystems/decay.rs:141-155, migrations/001_initial_schema.sql:98-100, migrations/003_story_010_pruned_flag.sql:7-10
Data Model Extensions
The decay system relies on specific fields in the database schema to track the "biological" age and strength of memories.
| Table | Column | Type | Description |
|---|---|---|---|
episodic_traces | salience | FLOAT | Current decay-adjusted strength. |
episodic_traces | pruned | BOOLEAN | Flag for hidden/soft-deleted state. |
semantic_facts | confidence | FLOAT | Decays alongside salience for facts. |
memory_vectors | importance | FLOAT | Mapping of salience for vector embeddings. |
memory_vectors | pruned | BOOLEAN | Added in Story 010 for decay support. |
Sources: migrations/001_initial_schema.sql:59-60, migrations/003_story_010_pruned_flag.sql:7-10, ethos-server/src/subsystems/decay.rs:197-205
Configuration Parameters
The decay behavior is governed by the DecayConfig struct, which defines the "physics" of forgetting in the engine.
| Parameter | Default/Type | Description |
|---|---|---|
base_tau_days | 7.0 (f64) | The base time constant (half-life factor). |
ltp_multiplier | 1.5 (f64) | How much each retrieval extends the time constant. |
frequency_weight | 0.3 (f64) | Impact of access frequency on salience. |
emotional_weight | 0.2 (f64) | Impact of emotional tone on retention. |
prune_threshold | 0.05 (f64) | The salience floor before pruning occurs. |
Sources: ethos-server/src/subsystems/decay.rs:242-250, README.md:42-47
Implementation Notes
The implementation uses batch processing (limit 500) for decay sweeps to prevent long-held database locks. The calculate_salience function is implemented as a pure function to allow for extensive unit testing of the decay curves without requiring database connectivity.
// ethos-server/src/subsystems/decay.rs:116-118
pub fn calculate_salience(
current_salience: f64,
retrieval_count: i32,
created_at: DateTime<Utc>,
last_accessed: Option<DateTime<Utc>>,
emotional_tone: f64,
config: &DecayConfig,
) -> f64 { ... }
Sources: ethos-server/src/subsystems/decay.rs:116-130, ethos-server/src/subsystems/decay.rs:141-144
The Decay & Pruning system ensures Ethos scales indefinitely by focusing computational and context resources on the most relevant "living" memories while allowing noise and outdated information to fade into the background.