Metrics & Decision Audit
Relevant source files
The following files were used as context for generating this wiki page:
Metrics and Decision Audit is a core observability framework within M31A designed to track session performance, tool usage, LLM costs, and architectural decisions. It provides a structured mechanism for logging decision receipts—complete with rationale and alternatives—while collecting high-fidelity metrics across the seven-phase workflow. This system ensures transparency in autonomous operations and enables cross-session learning via the learning ledger.
Sources: README.md:144-150, internal/tools/search/metrics.go:17-21
Architectural Overview
The system operates through an in-memory decision logger and a dedicated metrics collector. The decision logger captures "receipts" that include the rationale for specific actions, cost tracking, and automatic secret redaction. These records are managed via a non-blocking ring-buffer to ensure that logging operations do not interfere with the primary execution flow of the agent.
The diagram shows the flow of observability data from the Workflow Engine to persistence layers and the subsequent feedback loop into the engine.
Sources: README.md:145-155, internal/engine/workflow/ship.go:120-130
Decision Logging
Decisions are captured as structured receipts throughout the lifecycle of a session. These receipts are categorized to distinguish between planning, execution, and shipping actions. During the Plan phase, the engine logs decisions regarding task generation and versioning, including the cost in terms of attempts. During the Ship phase, these decisions are summarized and appended to the session summary.
Decision Categories
| Category | Description | Context of Use |
|---|---|---|
CategoryPlan | Decisions related to implementation strategy and task lists. | Generated during Plan and Plan Checker iterations. |
CategoryExecute | Decisions made during tool usage and task implementation. | Captured during autonomous execution cycles. |
CategoryShip | Decisions involving final commits and ledger updates. | Logged during session finalization. |
Sources: internal/engine/workflow/plan.go:193-200, internal/engine/workflow/ship.go:420-435, README.md:150-155
Decision Receipt Structure
// Example logic for logging a plan decision
e.LogDecision(decision.DecisionReceipt{
Decision: fmt.Sprintf("plan generated: %d tasks (v%d)", len(tasks), e.state.planVersion),
Rationale: "plan phase complete",
Category: decision.CategoryPlan,
Cost: decision.Cost{
Attempts: e.state.planVersion,
},
})
Sources: internal/engine/workflow/plan.go:193-200
Metrics Collection
The MetricsTool provides an interface for querying session observability data. It interfaces with a Collector to retrieve snapshots of session performance, tool efficiency, and LLM expenditures.
Query Modes
The metrics system supports four distinct query modes through the MetricsTool:
- Summary: Provides an overall session overview including tool success rates and total LLM costs.
- Tool-Stats: Details per-tool call counts, success/failure ratios, and average durations.
- Phase-Stats: Tracks durations for each phase, transition counts, and triggers for "self-heal" or "bisect" operations.
- Cost-Report: A granular breakdown of token usage (prompt vs. completion) and costs per phase.
Sources: internal/tools/search/metrics.go:17-64
Data Models
| Field | Type | Description |
|---|---|---|
CallCount | int64 | Total number of times a tool was invoked. |
SuccessCount | int64 | Number of tool invocations that returned without error. |
DurationMs | int64 | Time taken for a specific phase or tool execution. |
TotalTokens | int64 | Cumulative token usage for LLM interactions. |
HealTrigger | int | Number of times a failure triggered a self-healing attempt. |
Sources: internal/tools/search/metrics.go:85-150, internal/engine/workflow/ship.go:190-200
Session Finalization and Audit
During the Ship phase, the engine performs a final audit of the session's metrics and decisions. This includes generating a ShipSummary and updating the MEMORY.md file for cross-session recall.
The sequence diagram illustrates the final audit process where execution data is persisted into git history, the learning ledger, and long-term memory.
Sources: internal/engine/workflow/ship.go:210-250, internal/engine/workflow/ship.go:290-310
Git Diff Auditing
The audit system uses git diff --numstat to accurately track changes made by the agent. It classifies files as Added, Deleted, or Modified by comparing the current HEAD against the sessionStartHash.
Sources: internal/engine/workflow/ship.go:265-300
Learning Ledger & Memory
The system records specific failure modes and successful patterns to improve future performance.
- Failed Tasks: Records Task IDs, status, and the number of heal attempts.
- File Patterns: Tracks file extensions touched to build context for future relevance scoring.
- Efficiency: Logs the round-trip duration of the session.
Sources: internal/engine/workflow/ship.go:170-200, internal/engine/workflow/ship.go:215-230
Conclusion
The Metrics & Decision Audit system is fundamental to M31A's autonomy, providing the necessary telemetry to validate agent actions and optimize costs. By combining real-time metric collection with structured decision logging, the project maintains a transparent record of AI reasoning that serves both as a debugging aid for developers and a knowledge base for future sessions.