Quality Gates & Verification
Relevant source files
The following files were used as context for generating this wiki page:
Quality Gates and Verification in M31A represent a multi-layered defensive architecture designed to ensure code correctness, architectural integrity, and security throughout the autonomous development lifecycle. This system operates across three primary stages: pre-execution planning checks, per-task execution gates, and final post-implementation verification.
The purpose of this system is to prevent common AI failure modes such as "code-as-text" hallucinations, broken imports, and security anti-patterns. It integrates automated shell-based testing, LLM-based plan checking, and specialized heuristics to validate both the intent (Plan phase) and the output (Execute/Verify phases) of the agent's work.
Core Architecture and Workflow
The verification logic is embedded directly into the Engine and spans several workflow phases. It utilizes a combination of static analysis, runtime checks, and self-healing loops to maintain high standards.
This diagram shows the end-to-end flow of quality checks from plan inception to final reporting.
Sources: internal/engine/workflow/execute.go:132-150, internal/engine/workflow/verify.go:21-60, internal/engine/workflow/prompts/plan-check.md:1-10
Implementation Gates
1. Plan Quality Checker
Before execution begins, the system reviews implementation plans against specific dimensions to ensure they are actionable and verifiable.
- Task Granularity: Ensures tasks are under 200 lines of code and avoid "unbounded" definitions (0 files and 0 criteria).
- Acceptance Criteria: Validates that criteria are concrete (e.g., "npm test exits 0") rather than subjective ("looks correct").
- Dependency Correctness: Checks for missing orderings when tasks modify the same files.
- File Coverage: Identifies gaps between proposed changes and actual task assignments.
Sources: internal/engine/workflow/prompts/plan-check.md:10-50, internal/engine/workflow/coverage_gates_test.go:10-45
2. Execution-Time Quality Gates
During the Execute phase, M31A enforces strict "Read-Before-Write" rules and per-task quality gates.
| Gate Type | Description | File References |
|---|---|---|
| Code-as-Text Detection | Hard enforcement against models outputting code in chat instead of using FileWrite. Triggers a forced retry. | execute.go:193-220 |
| Loop Detection | Tracks repetitive tool calls to prevent infinite loops during task implementation. | execute.go:136-146, execute.go:303-315 |
| Acceptance Re-check | After a task is "healed", the engine re-runs quality gates before allowing a commit. | execute.go:151-179 |
Sources: internal/engine/workflow/execute.go:136-146, internal/engine/workflow/execute.go:193-220, internal/engine/workflow/execute.go:303-315
Verification and Self-Healing
The Verify phase serves as the final gate. If a task fails verification, M31A initiates a self-healing process.
Automated Self-Healing Logic
If runVerify detects syntax errors or test failures, it passes the error context back to the LLM. If standard healing fails, the engine attempts tryBisectHeal. This uses git bisect to identify the specific commit that introduced the regression, providing the agent with the exact diff context needed to fix the issue.
Sequence illustrating the interaction between the verification engine, Git bisect, and the LLM for recovery.
Sources: internal/engine/workflow/verify.go:42-80, internal/engine/workflow/verify.go:153-210
Security Scanning & Reporting
The system generates a structured VerifyReport that includes automated security scanning.
Security Anti-Pattern Detection
The scanSecurityFindings function parses task-modified files for common vulnerabilities using pattern matching:
- High Severity: Hardcoded passwords, secret keys, API key prefixes (e.g.,
sk-), and insecure TLS configurations. - Medium Severity: Usage of
eval(),exec(), or "TODO: security" markers. - Low Severity: Binding to
0.0.0.0or potential XSS sinks likeinnerHTML.
Sources: internal/engine/workflow/verify_report.go:81-115, internal/engine/workflow/verify_report_test.go:42-75
Acceptance Suite (scripts/verify_v1.sh)
The project includes a shell-based verification harness that sets the baseline for quality:
check "Binary builds cleanly" bash -c 'CGO_ENABLED=0 go build -o /tmp/m31a_verify ./cmd/m31a'
check "go vet passes" go vet ./...
check "All tests pass" go test -race ./...
Sources: scripts/verify_v1.sh:30-33
Summary
Quality Gates and Verification in M31A integrate architectural checks with autonomous recovery. By combining static plan analysis, execution monitoring (loop and code-as-text detection), and post-commit Git-based bisection, the system ensures that the agent's output is not only functional but also adheres to the project's security and structural standards. Every successful workflow concludes with a VerifyReport summarizing pass rates, errors, and security findings.
Sources: internal/engine/workflow/verify_report.go:121-140, internal/engine/workflow/verify.go:211-230