Introduction & Quick Start
Relevant source files
The following files were used as context for generating this wiki page:
NeuralDBG is a causal inference engine specifically designed for deep learning training dynamics. Unlike traditional monitoring tools that focus on raw metric tracking, NeuralDBG extracts semantic events and builds causal hypotheses to explain why a model failed during training. It treats training as a semantic trace of learning dynamics, providing structured explanations for phenomena like vanishing gradients or dead neurons without requiring raw tensor inspection.
Sources: README.md:3-12, infrastructure_planning/CODEBASE_GUIDE.md:8-12
The system operates as a non-invasive wrapper for PyTorch training loops. It utilizes module-boundary monitoring to ensure compatibility with torch.compile and maintains efficiency by extracting high-level transitions rather than storing full tensors. This approach allows researchers and engineers to identify the root causes of training instabilities through automated post-mortem reasoning.
Sources: README.md:15-22, plans/reasoning_engine_analysis.md:14-22
Core Architecture: Hub & Spokes
NeuralDBG follows a "Hub & Spokes" architectural design to maintain a stable core while allowing for modular extensions. The "Hub" contains the pure business logic and foundational data structures of the causal inference engine, while "Spokes" act as adapters for external dependencies like PyTorch, UI components, and CI/CD infrastructure.
Sources: copilot-instructions.md:28-34, infrastructure_planning/CODEBASE_GUIDE.md:73-86
The following diagram illustrates the relationship between the core engine and its surrounding infrastructure:
The Hub & Spokes model ensures that the core logic (reasoning, event classification) remains isolated from framework-specific noise.
Sources: infrastructure_planning/CODEBASE_GUIDE.md:73-86
Key Engine Components
The core engine in neuraldbg.py is structured into several classes that manage the lifecycle of a training analysis:
| Class | Description |
|---|---|
NeuralDbg | The main engine class; a context manager that wraps PyTorch models with hooks. |
EventType | Enum defining semantic events (gradient transitions, activation shifts). |
ActivationHealth | Tracks layer states such as NORMAL, SATURATED, DEAD, or ANOMALOUS. |
GradientHealth | Categorizes gradient norms as HEALTHY, VANISHING, EXPLODING, or SATURATED. |
SemanticEvent | Representation of a detected failure event including layer ID and step range. |
CausalHypothesis | The result of the reasoning engine, linking evidence into a causal chain. |
Sources: infrastructure_planning/CODEBASE_GUIDE.md:31-40
The Reasoning Workflow
NeuralDBG follows an Event-Driven Abductive Reasoning Model. Instead of analyzing raw data directly, it processes metrics through a multi-stage pipeline to generate explanations.
The reasoning flow converts raw tensor statistics into high-level semantic events before performing pattern detection.
Sources: plans/reasoning_engine_analysis.md:8-12
Detection Mechanisms
The engine employs specific thresholds to classify the health of the network during training:
- Vanishing Gradients: Detected when gradient norm drops below
1e-6. - Exploding Gradients: Detected when gradient norm exceeds
1e3. - Coupled Failures: Identified based on temporal proximity, typically within a window of 5 steps.
Sources: plans/reasoning_engine_analysis.md:14-22, plans/reasoning_engine_analysis.md:267-272
Quick Start Guide
Installation
NeuralDBG requires Python 3.8+ and PyTorch.
pip install neuraldbg
Sources: README.md:30-34
Basic Implementation
To use NeuralDBG, wrap your standard PyTorch training loop with the NeuralDbg context manager. This automatically attaches forward and backward hooks to the model layers.
import torch
import torch.nn as nn
from neuraldbg import NeuralDbg
model = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 1))
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
# Wrap training loop
with NeuralDbg(model) as dbg:
for inputs, targets in dataloader:
optimizer.zero_grad()
outputs = model(inputs)
loss = nn.MSELoss()(outputs, targets)
loss.backward()
optimizer.step()
# Query for ranked causal explanations
explanations = dbg.explain_failure()
Sources: README.md:38-58, infrastructure_planning/CODEBASE_GUIDE.md:43-45
Analysis API
Once training is complete or a failure has been detected, the following methods are used to extract insights:
dbg.explain_failure(): The main entry point for causal reasoning, returning ranked explanations.dbg.get_root_causes(): Identifies the specific layer and step where the failure first originated.dbg.trace_causal_chain(event_type): Queries the specific chain of events leading to a failure.dbg.export_mermaid_causal_graph(): Generates a Mermaid-compatible string to visualize the failure trace.
Sources: README.md:62-72, infrastructure_planning/CODEBASE_GUIDE.md:49-54
Project Roadmap & Constraints
NeuralDBG development is governed by strict milestones and guardrails to ensure depth of implementation over breadth.
| Milestone | Target Progress | Focus Area |
|---|---|---|
| Phase 1 | 25% | Core validation and 60% test coverage. |
| Phase 2 | 35% | torch.compile hardening and module boundary validation. |
| Phase 3 | 45% | Enhanced demo scenarios and causal compression documentation. |
| Phase 4 | 50% | Security hardening (Bandit/Safety) and CI/CD validation. |
Sources: ROADMAP.md:10-53
Operational Guardrails
- No UI: The engine is strictly CLI/Inference API based for the MVP.
- Framework Limit: Currently supports PyTorch only.
- No Full Tensor Storage: To prevent memory bloat, the engine avoids storing raw tensors.
- Pessimistic Progress: Progress reporting must be realistic; if a feature is 50% done, it is scored as 30% in tracking.
Sources: ROADMAP.md:62-71, copilot-instructions.md:143-150
Conclusion
NeuralDBG provides a structured approach to debugging deep learning models by shifting focus from raw metrics to causal reasoning. By integrating semantic event extraction with automated pattern detection, it allows developers to move beyond identifying that a model failed to understanding why it failed. The "Hub & Spokes" architecture ensures that this diagnostic capability remains framework-agnostic at its core, while the non-invasive hook system makes it easy to integrate into existing PyTorch workflows.