Testing Strategy & Validation Gates
Relevant source files
The following files were used as context for generating this wiki page:
The NeuralDBG project employs a multi-layered testing strategy and strict validation gates to ensure the reliability of its causal inference engine. This strategy integrates automated technical checks, security hardening, and human-centric validation (The Mom Test) into a unified delivery pipeline. The system is designed around "Hard Milestone Locks," where development must stop at specific progress intervals (25%, 50%, 75%, 90%, 95%) until all validation criteria are met.
Sources: copilot-instructions.md:9, GAD.md:R20, README.md
Testing Hierarchy and Minimum Coverage
The technical testing architecture follows a traditional Testing Pyramid with specific mandatory coverage requirements. Every code addition must maintain a minimum of 60% test coverage.
The Testing Pyramid
NeuralDBG allocates testing effort according to the following distribution:
- Unit Tests (70%): Testing individual components, classes, and logic in isolation.
- Integration Tests (20%): Verifying that different modules (e.g., semantic event extraction and causal compression) work together.
- E2E Tests (10%): Validating the entire training-to-explanation flow using end-to-end scenarios.
Sources: copilot-instructions.md:Technical Preferences, copilot-instructions.md:Advanced Testing & Analysis, GAD.md: Advanced Testing & Analysis
Specialized Testing Methods
Beyond standard tests, the project mandates advanced analysis techniques:
- Fuzz Testing: Using tools like AFL (American Fuzzy Lop) on critical parser or data-handling paths.
- Load Testing: Using Locust.io to verify performance under stress.
- Mutation Testing: Using Stryker (or language equivalents) to verify the efficacy of the test suite by injecting intentional faults.
- Continuous Analysis: Integration of CodeQL, SonarQube, and Codacy into the CI/CD pipeline for deep static analysis.
Sources: copilot-instructions.md:Advanced Testing & Analysis, GAD.md: Advanced Testing & Analysis
CI/CD Validation Gates
The CI/CD pipeline acts as an "unbreakable digital bouncer," preventing the merger of code that fails to meet quality or security standards.
The flow above illustrates how automated gates block code based on coverage, security, and style requirements.
Sources: infrastructure_planning/milestone_0_tasks.md:The CI/CD "Gate" Concept Explained
Automated Check Requirements
| Check Type | Tooling | Requirement |
|---|---|---|
| Unit/Integration | pytest | Minimum 60% total coverage |
| Security (Static) | bandit | Zero high-severity issues |
| Dependency Audit | safety | Zero known vulnerabilities |
| Static Analysis | CodeQL, Codacy | Mandatory for CI/CD pipeline |
| Linting | black, isort, flake8 | Strict adherence to style rules |
Sources: copilot-instructions.md:Advanced Testing & Analysis, infrastructure_planning/milestone_0_tasks.md:1. Cross-Platform CI/CD, README.md
Milestone Validation Gates
NeuralDBG enforces "Hard Milestone Locks." Development is strictly limited to one feature per cycle, and progress cannot continue until specific qualitative and quantitative gates are cleared.
Qualitative Validation (The Mom Test)
Before production code is written (during the 0-10% research phase), and at every subsequent milestone, user research must validate the problem-solution fit.
| Milestone | Required Validation |
|---|---|
| 25% | Mom Test follow-up (3+ users), Marketing Test (landing page views) |
| 50% | Mom Test validation (5+ new users), Marketing Test (conversion metrics) |
| 75% | Mom Test expansion (different segments), Marketing Test (pricing) |
| 90% | Final Mom Test, Marketing Test (launch readiness) |
| 95% | Pre-launch validation (all criteria met) |
Sources: copilot-instructions.md:Periodic Validation, GAD.md:Periodic Validation
Technical Milestone Requirements
At each validation gate (25, 50, 75, 90, 95%), the project must provide:
- 2 Working Demos: Runnable demonstrations showing different aspects of the product.
- Security Scan Proof: Explicit proof (e.g., Bandit output) that security rules are followed.
- Rule 33 Verification: Confirmation that all AI rule files (AGENTS.md, GAD.md, etc.) are synchronized across branches.
Sources: copilot-instructions.md:Working Demos, GAD.md:Working Demos, infrastructure_planning/RULE_SYNC_PROCEDURE.md:Layer 2
Rule 39: CI/CD Debugging First
A critical operational gate is Rule 39, which mandates that any failing GitHub Actions check takes absolute priority over feature work.
This sequence ensures that the "Hub" (core repository) remains stable by fixing the pipeline before introducing new logic.
Sources: copilot-instructions.md:RULE 39, GAD.md:RULE 39
Deep Understanding Gate
Before transitioning to a new development phase, a "Deep Understanding" gate must be cleared. The developer must demonstrate knowledge of:
- Mechanism: How the feature works under the hood.
- 2nd Order Consequences: Effects in production and potential edge cases.
- 3rd Order Consequences: Long-term effects and future dependencies.
- Critical Thinking: Answering probing questions such as "What would happen if 10x more users used this?" or "What assumptions are we making that might be wrong?"
Sources: copilot-instructions.md:Deep Understanding Before Phase Transition, GAD.md:Deep Understanding Before Phase Transition
Causal Reasoning Validation
Specific to the NeuralDBG engine, testing includes validating the accuracy of causal hypotheses generated by the engine. This involves testing against known failure patterns:
- Vanishing Gradients: Validated using deep Tanh networks.
- Exploding Gradients: Validated using high learning rates or poor initialization.
- Dead Neurons: Validated using ReLU activations with high bias.
Sources: plans/reasoning_engine_analysis.md:3. Activation Functions Deep Dive, plans/reasoning_engine_analysis.md:5. Deep Model Testing Gaps
The Testing Strategy and Validation Gates within NeuralDBG ensure that every feature is not only technically sound and secure but also provides genuine value to users. By combining high-coverage automated testing with strict milestone-based research gates and mandatory CI/CD debugging protocols, the project maintains a stable and high-quality core.