Evaluations & Continuous Feedback
Relevant source files
The following files were used as context for generating this wiki page:
Evaluations and Continuous Feedback constitute the core evidence-driven loop of Noesis, ensuring that the system's autonomous compounding is both measurable and safe. The system moves from ordinary work to evidence signals, which then drive LLM-based reflection and the authorship of new CapabilityRevisions. These revisions undergo a rigorous preflight evaluation before activation and are continuously monitored against a set of four compounding metrics to verify that earlier work demonstrably makes later work better.
Sources: plans/noesis-autonomous-compounding-implementation-plan.html:150-165, plans/noesis-high-leverage-correction-plan.html:105-115
Evaluation Architecture
The evaluation system is structured as an "LLM-native laboratory" designed to test candidate revisions against baselines using both historical and generated data. The architecture enforces a strict separation of roles—such as the blind judge, trial agent, and case generator—to prevent "gaming" the evaluation process.
The Evaluation Lifecycle
The lifecycle follows a structured path from signal detection to either the retention, revision, or reversion of a capability.
The diagram shows the transition from a detected signal to the final determination of a capability's success in real-world use.
Sources: plans/noesis-autonomous-compounding-implementation-plan.html:435-445, plans/noesis-first-complete-iteration.html:700-740
Key Components of the Evaluation Lab
| Component | Responsibility |
|---|---|
| Case Generator | Uses behavioral objectives and evidence to create source variations and negative cases. |
| Trial Agent | Executes a single case (arm) and records output, artifacts, and traces. |
| Blind Judge/Critic | Compares randomized A/B outputs against a rubric to determine a winner with reasoning. |
| Protected Rails | Hard gates that check for authority violations, resource failures, or regressions. |
Sources: plans/noesis-autonomous-compounding-implementation-plan.html:504-525, plans/noesis-first-complete-iteration.html:742-760
Continuous Feedback & Compounding Metrics
To ensure the system is genuinely improving, Noesis employs "Effect-free paired shadow replay." This mechanism materializes contemporaneous served and baseline arms from an immutable turn plan, replaying them with recorded tool results to calculate specific metrics.
The Four Compounding Metrics
| Metric | Definition | Success Indicator |
|---|---|---|
| Win Rate | Blind-judged wins of served revision vs. pinned baseline on replays. | Durably above chance. |
| Scope Leakage | Share of turns where an adaptation was served to an unrelated scope. | Trends toward zero. |
| Context Tax | Token/byte cost of injected layers and marginal served input. | Flat or falling. |
| Recurrence | Rate at which the same user correction signature recurs. | Falls after activation. |
Sources: plans/noesis-high-leverage-correction-plan.html:128-158, plans/noesis-dogfood-protocol.md:50-70
Feedback Data Flow
The system attributes real-world feedback (corrections, retries, aborts) back to the specific experiment and serving revision.
The sequence illustrates how user interactions are translated into durable metrics via paired replay and persistence in SQLite.
Sources: plans/compounding-partnership-product-loop.html:840-880, plans/noesis-high-leverage-correction-plan.html:118-125
Preflight and Activation
Before a CapabilityRevision is activated, it must pass a preflight evaluation. This report is byte-stable and pins the exact evidence used for the decision.
Preflight Logic
- Case Composition: Includes historical source cases, LLM-generated variations, and user corrections.
- Paired Execution: Both the baseline and candidate are given the same evidence and budget.
- Rail Enforcement: A high quality judgment cannot override a protected rail (e.g., unauthorized permission requests).
Sources: plans/noesis-autonomous-compounding-implementation-plan.html:530-560, plans/compounding-partnership-product-loop.html:1320-1340
Activation Transaction
Activation is an atomic SQLite transaction that:
- Materializes the staged revision bytes.
- Updates the activation pointer to the new complete revision set.
- Records the activity in the
activity_log.
Sources: plans/noesis-autonomous-compounding-implementation-plan.html:420-430, plans/compounding-partnership-product-loop.html:610-630
Dogfooding & Friction Logging
A specific four-week dogfooding protocol is used to provide directional evidence for the product thesis. Users are required to maintain an external friction log that Noesis cannot edit or suppress.
Friction Log Structure
Users record entries for every interaction where they feel friction or observe an advantage, noting:
- Intended Posture: "Work with me" or "Do for me".
- Correction Signature: A stable label for recurring issues.
- Prior Work Expected to Help: The specific adaptation or memory that failed or succeeded.
Sources: plans/noesis-dogfood-protocol.md:20-45, plans/noesis-high-leverage-correction-plan.html:265-275
Security & Isolation
Evaluation is performed with strict boundary controls to prevent generated code from modifying protected components.
- Evaluator Isolation: Judges receive randomized arms and escaped artifacts; candidate authors cannot see judge decisions or protected cases.
- Out-of-Process Execution: Generated tools run in a sanitized, temporary child process backend with bounded time and output.
- Receipt Verification: Every protected mutation (activation, revert) requires an
AuthorityReceiptverified at the deep protected adapter.
Sources: plans/noesis-autonomous-compounding-implementation-plan.html:610-635, plans/noesis-high-leverage-correction-plan.html:360-385, packages/domain/test/architecture.test.ts:95-130
The Evaluations & Continuous Feedback system ensures that every Noesis adaptation is grounded in empirical evidence, remains reversible through stable revision snapshots, and provably benefits the user's workflow over time.