Dogfood Protocol
Relevant source files
The following files were used as context for generating this wiki page:
The Noesis Dogfood Protocol is a four-week empirical testing program designed to validate the project's central product thesis: that earlier work performed by the system should demonstrably improve later related work. It serves as a source of directional evidence from a primary user rather than a broad validation study, ensuring that product evolution is steered by real-world friction and advantages rather than hypothetical personas or invented fixtures.
The protocol is sequenced to begin after the execution of the High-Leverage Correction Plan, specifically once the "causal spine" of the system—the path allowing a correction in one session to change a later session—is operational. It gates critical evidence-dependent choices, such as posture heuristics and learning thresholds, without blocking underlying correctness or structural architecture work.
Sources: plans/noesis-dogfood-protocol.md:3-5, plans/noesis-high-leverage-correction-plan.html:phase-1, plans/compounding-partnership-product-loop.html:handoff
Core Methodology and Timeline
The protocol follows a strict four-week schedule. The primary participant utilizes Noesis with the Pi runtime as their primary interface for genuine builder, philosopher, learner, and researcher tasks.
Protocol Lifecycle
The lifecycle ensures consistent data collection and iterative reporting before final conclusions are drawn.
Sources: plans/noesis-dogfood-protocol.md:7-13, plans/noesis-dogfood-protocol.md:39-42
Data Collection: The Friction Log
A critical requirement of the protocol is that the friction log must be maintained outside the Noesis workspace and repository. This ensures that Noesis cannot edit, rank, or suppress entries that document its own failures or irritations.
Friction Log Schema
| Field | Description |
|---|---|
| Timestamp | The exact time of the event. |
| Work Scope | The specific project or activity being performed. |
| Intended Posture | Whether the user intended "Work with me" or "Do for me". |
| What happened | Observable behavior, recorded without interpretation. |
| Friction/Advantage | Description of what became harder or easier. |
| Prior Work | The specific earlier session, correction, or artifact expected to help. |
| Did it help? | Binary/Mixed result: Yes, No, Mixed, or N/A. |
| Signature | A stable short label for recurring corrections (e.g., "Voice preservation"). |
Sources: plans/noesis-dogfood-protocol.md:15-28, plans/noesis-high-leverage-correction-plan.html:measure
Performance Metrics (The Compounding Claim)
The protocol utilizes a compounding-metrics read model to quantify improvement. These metrics distinguish genuine advantage from mere "accumulation" (growth in context size without quality gain).
The diagram shows how settled turns are re-evaluated to compute improvement over the baseline.
Sources: plans/noesis-high-leverage-correction-plan.html:measure, plans/noesis-autonomous-compounding-implementation-plan.html:lifecycle
Primary Metrics
- Served-revision win rate: Blind-judged paired wins of the active revision over the pinned baseline.
- Scope leakage rate: The share of turns where an adaptation was served to an unrelated scope.
- Context tax: The cost in tokens/bytes for injected context and prompt layers.
- Correction recurrence: The rate at which specific friction signatures recur after an adaptation is activated.
Sources: plans/noesis-high-leverage-correction-plan.html:measure
Readout and Decision Gating
The protocol mandates weekly readouts followed by a comprehensive final report. These readouts serve as the empirical basis for adjusting system heuristics.
Weekly Reporting Requirements
- Count of real work sessions and eligible replay turns.
- Replay coverage and specific exclusion reasons (e.g., secret data).
- Recurring friction signatures and exposure counts.
- Directional examples of successful transfers vs. leakage into unrelated work.
- Current metrics from the compounding read model.
Sources: plans/noesis-dogfood-protocol.md:30-37
Decision Gates
The final four-week readout is the authoritative gate for:
- Posture Heuristics: Non-obvious rules for detecting if a user needs collaboration or execution.
- Scope/Threshold Choices: Granularity of learning and thresholds for automatic activation.
- Metric Interpretation: Final calibration of what constitutes "success" in the context of the thesis.
- Product Validation Claims: The formal statement on whether the compounding thesis holds.
Sources: plans/noesis-dogfood-protocol.md:39-49, plans/compounding-partnership-product-loop.html:handoff
Integration with Architecture
The protocol interacts directly with the WorkspaceStore and EvaluationLab. While the log is external, the system's internal telemetry provides the data for the "Shadow Replay" mechanism.
| Component | Role in Protocol |
|---|---|
| WorkspaceStore | Authoritative source for turn IDs and session history for replays. |
| EvaluationLab | Conducts effect-free paired shadow replays using recorded tool results. |
| AgentHarness | Used with credential-free controlled providers to verify protocol assertions in CI. |
Sources: plans/noesis-high-leverage-correction-plan.html:HL-10, AGENTS.md:architecture, plans/noesis-autonomous-compounding-implementation-plan.html:evaluation
Conclusion
The Dogfood Protocol ensures that Noesis remains a research-led project where "rigidity must earn its keep." By enforcing a strict period of real-world usage and external logging, the project avoids building complex machinery that does not serve a proven need. A null or negative result from the protocol is considered a valid finding, leading to revision of the product doctrine rather than the optimization of a false metric.
Sources: plans/noesis-dogfood-protocol.md:51, plans/noesis-high-leverage-correction-plan.html:done