Automated PR Reviews & Comment Modes
Relevant source files
The following files were used as context for generating this wiki page:
The Automated PR Review system is a core component of the AI Reviewer application, responsible for analyzing pull request diffs and generating structured feedback. It utilizes Large Language Models (LLMs) enriched with codebase context to identify bugs, security vulnerabilities, and quality issues. The system operates primarily in a non-blocking COMMENT mode, ensuring that AI feedback acts as an advisory tool rather than a hard gate for merging.
The system encompasses two main phases: the initial automated review triggered by PR events and the subsequent interactive processing of developer feedback. This lifecycle ensures that the AI not only provides initial findings but also learns from developer interactions through a dedicated "Review Memory" layer.
Review Orchestration Flow
The review process is managed by an orchestrator that coordinates between the GitHub API, the LLM provider, and the internal database. When a pull request event is received, the system filters files, retrieves relevant RAG (Retrieval-Augmented Generation) context, and invokes the LLM for analysis.
Core Review Logic
The orchestrator performs several critical steps:
- Configuration Check: Validates that reviews are enabled for the repository and checks for ignored paths or authors.
- Context Retrieval: Queries the database for similar code patterns (Layer 1), past review decisions (Layer 2), and repo-specific knowledge.
- LLM Invocation: Generates comments for each file, ensuring that suggestions are pinned to valid, commentable lines in the diff.
- Persistence & Posting: Saves review findings to the database and submits a bundled review to GitHub.
Sources: packages/review/src/orchestrator.ts:77-240, README.md:160-193
The following sequence diagram illustrates the lifecycle of a PR review from webhook intake to posting on GitHub:
Comment Modes and Intent Classification
The system supports different interaction modes and uses AI to classify the intent of developer replies to AI-generated comments.
Review Modes
The project defines specific modes of operation via the ReviewMode enum:
| Mode | Description |
|---|---|
COMMENT | Posts inline comments only. Advisory only; never blocks merges. |
REQUEST_CHANGES | Planned feature to block merges if CRITICAL issues are identified. |
Sources: packages/db/prisma/schema.prisma:450-456, packages/github/src/review-client.ts:12-16
Intent Classification
When a developer replies to an AI comment, the comment-processor uses the LLM to classify the intent into one of five categories:
- QUESTION: Developer seeks clarification. The system generates and posts an automated explanation.
- ACCEPTED: Developer agrees with the finding. This updates the "Review Memory" to increase confidence in similar future flags.
- DISMISSED: Developer disagrees. Updates memory to avoid flagging similar patterns unless they are high severity.
- CODEBASE_KNOWLEDGE: Developer provides factual info about the architecture. This is extracted and stored for future RAG context.
- UNRELATED: No action taken.
Sources: packages/review/src/prompts.ts:133-144, packages/review/src/comment-processor.ts:70-112
Data Models and Schemas
The system relies on structured data to maintain consistency between the LLM output, the database, and the GitHub API.
Review Structures
The following Zod schemas define the contract for LLM outputs:
export const FileReviewCommentSchema = z.object({
line: z.number().int().positive(),
side: z.enum(['LEFT', 'RIGHT']).default('RIGHT'),
severity: z.enum(['INFO', 'WARNING', 'ERROR', 'CRITICAL']),
body: z.string().min(1).max(5000),
suggestion: z.string().max(2000).optional(),
});
Sources: packages/review/src/schemas.ts:14-23
Database Entity Relationships
The relationship between reviews, comments, and the learning memory is captured in the database schema:
Severity and Feedback Loop
The system uses a severity guide to prioritize findings and manage developer experience.
| Severity | Description | Action |
|---|---|---|
| INFO | Suggestions for improvement or minor style nits. | Informational only. |
| WARNING | Potential bugs or code smells. | Standard review comment. |
| ERROR | Definite bugs or security issues. | High-priority comment. |
| CRITICAL | Data loss or security breach risks. | Highest priority; may trigger REQUEST_CHANGES in future modes. |
Sources: packages/review/src/prompts.ts:14-17, packages/review/src/orchestrator.ts:245-252
The Learning Moat
A key differentiator of this system is its ability to learn per-repository. When a developer marks a comment as ACCEPTED or DISMISSED, this outcome is stored in the ReviewMemory table. During subsequent reviews, the orchestrator retrieves these past decisions and provides them to the LLM as "Past review decisions on similar patterns," creating a compounding quality improvement for the specific codebase.
Sources: CLAUDE.md:54-65, packages/review/src/prompts.ts:86-101, packages/review/src/memory-search.ts:28-50
Conclusion
The Automated PR Review system integrates deep codebase context with LLM analysis to provide actionable feedback. By supporting interactive comment processing and a learning memory layer, the system moves beyond generic linting to provide repo-specific architectural and logic-aware reviews.