Product Validation & User Research
Relevant source files
The following files were used as context for generating this wiki page:
- mom_test_template.md
- interview_collection_guide.md
- AGENTS.md (referenced as Global AI Rules/AGENTS.md in context)
- GAD.md
- ROADMAP.md
- plans/reasoning_engine_analysis.md
Product Validation and User Research within NeuralDBG are governed by the Prime Directive: Value over Code. This principle dictates that no production code should be written before a feature has been validated by addressing a real user problem, primarily through the "Mom Test" methodology. The goal is to ensure that the development of the causal inference engine remains grounded in actual machine learning research pain points rather than technical speculation.
The research phase is strictly enforced during the initial 0-10% of project progress. During this period, the focus is exclusively on user interviews and market intelligence gathering. Development is halted at specific milestones (25%, 50%, 75%, 90%, and 95%) to perform follow-up validation, ensuring the product maintains alignment with user needs as it evolves from a concept to a Minimum Viable Product (MVP).
Sources: GAD.md:14-16, copilot-instructions.md:144-155, ROADMAP.md:7-11
The Mom Test Framework
The project utilizes the "Mom Test" as its primary validation gate. This framework focuses on discovering the "truth" of user problems by avoiding leading questions about the product idea and instead focusing on the user's past behaviors and existing workarounds.
Mandatory Interview Scripting
Interviews must follow a specific structure to ensure unbiased data collection. The core questions are designed to extract concrete evidence of pain points:
- Contextual Inquiry: "Tell me about the last time [problem] happened to you."
- Effort Quantification: "How much time did you spend solving it?"
- Current Workarounds: "What did you concretely do to solve it?"
- Market Search: "Have you already searched for or built a solution?"
- Frequency: "How frequently does this problem occur?"
Sources: mom_test_template.md:95-105, interview_collection_guide.md:46-52
Evaluation Signals
The research process categorizes user feedback into three signal types to determine project viability:
| Signal Type | Indicators | Action |
|---|---|---|
| Positive | Spontaneous mention of the problem; custom scripts built; abandoned projects due to the issue. | Continue/Deepen |
| Neutral | Acknowledges the problem but it is not painful enough to warrant a new tool. | Pivot Research |
| Negative | Problem does not exist or existing solutions (e.g., MLFlow, TensorBoard) are satisfactory. | Stop/Major Pivot |
Sources: mom_test_template.md:111-120, interview_collection_guide.md:11-15
Validation Milestones & Gates
Development is governed by a "Hard Milestone Lock." The system requires a stop at specific completion percentages to perform mandatory validation tasks before proceeding.
Periodic Validation Requirements
The diagram above illustrates the mandatory stop-and-validate gates required by the project directives.
Sources: copilot-instructions.md:144-155, GAD.md:139-148
Validation Criteria for MVP
For the project to move past the 50% progress mark, specific criteria must be checked:
- Minimum of 5 complete interviews.
- At least 3 respondents must mention the problem spontaneously.
- At least 2 respondents must have already attempted to build or search for a solution.
Sources: mom_test_template.md:162-167, ROADMAP.md:74-82
Market Intelligence & Outreach
At each milestone, the project must research at least three external sources to maintain "Market Intelligence." This involves monitoring active communities where deep learning practitioners discuss training instabilities.
Target Research Channels
The following platforms are designated for sourcing interview candidates and observing "natural" problem discussions:
- Reddit: r/MachineLearning, r/pytorch, and r/deeplearning (using keywords like "vanishing gradients debug" or "NaN loss").
- Discord: PyTorch, FastAI, and Hugging Face servers, specifically within #debugging and #training-issues channels.
Sources: interview_collection_guide.md:21-44, copilot-instructions.md:15-16
Causal Inference Gap Analysis
Technical research identifies a gap in existing tools that user research aims to exploit. While current users rely on metric tracking (TensorBoard/W&B), NeuralDBG's research focuses on the "pain" of interpreting these metrics into causal reasons.
| Competitor Type | User Activity | Pain Point (Validation Focus) |
|---|---|---|
| Metric Trackers | Monitoring loss curves | "Why" is the loss spiking? |
| Manual Hooks | Printing tensor norms | Cognitive load of manual inspection |
| Simple Heuristics | Rule-based alerts | High false positives; no causal chain |
Sources: plans/reasoning_engine_analysis.md:15-35, interview_collection_guide.md:76-85
Decision Protocol
Following the collection of interview data, the lead orchestrator must make a formal decision documented in the mom_test_template.md.
This sequence shows the interaction between the research findings and the AI's enforcement of the project gates.
Sources: GAD.md:149-152, mom_test_template.md:169-175
Conclusion
Product Validation in NeuralDBG is not a one-time event but a continuous cycle of user feedback and technical verification. By strictly adhering to the Mom Test and mandatory milestone locks, the project minimizes technical debt and maximizes the relevance of its causal reasoning engine for the machine learning community.
Sources: ROADMAP.md:84-88, GAD.md:12-14