Event Data Lifecycle & State Memory

Relevant source files

The following files were used as context for generating this wiki page:

• logic_graph.md
• infrastructure_planning/CODEBASE_GUIDE.md
• plans/reasoning_engine_analysis.md
• README.md
• infrastructure_planning/milestone_50_tasks.md

NeuralDBG utilizes a specialized lifecycle for managing training data, transforming raw tensor metrics into high-level Semantic Events. This system is designed for memory efficiency, avoiding the storage of full tensors and instead focusing on the state-based tracking of gradient health and activation regimes. The lifecycle transitions from real-time monitoring through PyTorch hooks to post-mortem causal analysis and hypothesis generation.

The core philosophy of this system is Causal Compression, which reduces thousands of individual training observations into a compact set of root causes and propagation patterns. This allows researchers to understand the "why" of a training failure without the overhead of traditional metric tracking or large-scale data dumps.

Sources: README.md, infrastructure_planning/CODEBASE_GUIDE.md

1. Data Transformation Flow

The event data lifecycle begins during the training loop. Raw statistics are captured at the module boundaries of a neural network and evaluated against predefined thresholds. If a significant shift is detected, a persistent event is created and stored in the engine's memory.

The diagram above illustrates the transformation of ephemeral tensor data into persistent semantic structures for reasoning.
Sources: logic_graph.md

1.1 Extraction and Detection

Events are extracted using non-invasive PyTorch hooks. The system monitors two primary data streams:

• Forward Pass: Observes activation statistics to identify regime shifts (e.g., neurons becoming "dead" or "saturated").
• Backward Pass: Observes gradient norms to identify health transitions (e.g., stable to vanishing or exploding).

Event Source	Data Captured	Transition Types
Forward Hook	Activation Mean/Var/Sparsity	NORMAL -> SATURATED, DEAD, ANOMALOUS
Backward Hook	Gradient Norms	HEALTHY -> VANISHING, EXPLODING, SATURATED

Sources: logic_graph.md, infrastructure_planning/CODEBASE_GUIDE.md, plans/reasoning_engine_analysis.md:15-25

2. State Memory Architecture

NeuralDBG maintains a internal state memory through the NeuralDbg class. This memory is designed to be compiler-aware and non-invasive, operating safely within torch.compile boundaries.

2.1 The SemanticEvent Schema

Every event stored in memory follows a structured schema that includes temporal and structural metadata. This metadata is essential for the later stages of the lifecycle, specifically for coupling detection and causal chain tracing.

Field	Type	Description
event_type	Enum	The category of event (Gradient Health, Activation Shift).
layer_name	String	The identifier for the specific model layer.
step	Integer	The training step number when the event occurred.
from_state	Stats	The metric state prior to the transition.
to_state	Stats	The metric state after the transition.
confidence	Float	The certainty score of the detection.

Sources: logic_graph.md, infrastructure_planning/CODEBASE_GUIDE.md, README.md

2.2 Memory Management and Pruning

Currently, events are stored in an unbounded list within the NeuralDbg.events attribute. However, architectural plans identify this as a potential bottleneck. Proposed improvements include implementing circular buffers and event pruning based on confidence scores to prevent memory leaks during long training sessions.

Proposed Memory Efficiency Measures:

• Circular Buffer: Using collections.deque(maxlen=10000) to cap memory usage.
• Event Pruning: Automatically discarding low-confidence events when a threshold is reached.
• Weak References: Using weakref for model tracking to avoid preventing garbage collection of the target network.

Sources: plans/reasoning_engine_analysis.md:150-185

3. Post-Mortem Lifecycle Stages

Once training finishes or fails, the data lifecycle moves from "Collection" to "Reasoning." This stage involves compressing the stored events into a coherent narrative.

This sequence shows the progression of data from raw event storage to finalized user explanations.
Sources: logic_graph.md, README.md, plans/reasoning_engine_analysis.md:15-25

3.1 Causal Reasoning Components

The "State Memory" is utilized by the reasoning engine to perform three key functions:

1. Temporal Pattern Recognition: Identifying if failures in Layer A preceded failures in Layer B (Causal Cascades).
2. Coupling Detection: Finding structural or temporal links between disparate events within a short step window (e.g., 5 steps).
3. Hypothesis Ranking: Assigning confidence scores to various explanations based on the evidence chain found in the event logs.

Sources: logic_graph.md, plans/reasoning_engine_analysis.md:200-220, infrastructure_planning/CODEBASE_GUIDE.md

4. Lifecycle Integration and Automation

As the project progresses toward Milestone 50, the lifecycle is being expanded to include external persistence and resource tracking.

• MLflow Integration: Automating the export of SemanticEvent objects and causal hypotheses to a persistent tracking server.
• Resource Profiling: Adding lightweight GPU and memory utilization metrics to the SemanticEvent metadata to identify hardware-related bottlenecks.
• DVC Tracking: Ensuring that the data used to generate these events (synthetic datasets) is versioned alongside the training runs for full reproducibility.

Sources: infrastructure_planning/milestone_50_tasks.md

Summary

The Event Data Lifecycle in NeuralDBG is a transition from ephemeral tensor monitoring to persistent, high-level reasoning. By utilizing a Hub & Spokes architecture, the core reasoning engine remains isolated from the noise of raw data, relying instead on a structured state memory of SemanticEvent objects. This approach ensures that training failures are not just tracked as metrics, but explained as a sequence of causal transitions.