Code Chunking & Embeddings
Relevant source files
The following files were used as context for generating this wiki page:
The Code Chunking & Embeddings system is a core component of the project's RAG (Retrieval-Augmented Generation) pipeline. Its primary purpose is to transform raw source code into manageable, semantically meaningful units (chunks) and represent them as high-dimensional vectors (embeddings). This process enables the AI reviewer to retrieve relevant codebase context, past review decisions, and repository-specific knowledge during the Pull Request review process.
By utilizing Abstract Syntax Tree (AST) parsing instead of simple character-based splitting, the system ensures that code context remains intact by chunking at function, class, and method boundaries. These chunks are then stored in a PostgreSQL database with pgvector support, allowing for efficient similarity searches tailored to specific repositories.
Sources: CLAUDE.md:46-51, packages/chunking/src/ast-chunker.ts:101-110
AST-Based Chunking Logic
The project employs tree-sitter for AST parsing to identify logical boundaries within source code. This approach prevents splitting a single function or class across multiple chunks, which would degrade the LLM's understanding of the code.
Supported Languages and Node Types
The chunker specifically targets structural elements based on the programming language of the file.
| Language | Target AST Node Types |
|---|---|
| TypeScript / TSX / JavaScript | function_declaration, method_definition, class_declaration |
| Python | function_definition, class_definition |
Sources: packages/chunking/src/ast-chunker.ts:15-32
Chunking Workflow
When a file is processed, the system walks the AST to find matched nodes. For each matched node, it extracts the content and includes up to 3 lines of preceding context to capture decorators, comments, or assignments.
The diagram shows the logical flow of the AST chunker from language detection to the final extraction of code units.
Sources: packages/chunking/src/ast-chunker.ts:110-142
Ingestion and Embedding Pipeline
The ingestion system handles the lifecycle of repository indexing, moving through states such as DISCOVERING, CHUNKING, and ACTIVE.
Repository Indexing Process
- Discovery: The system fetches the recursive file tree from GitHub and filters for supported code files under 150KB.
- Batch Processing: Files are processed in batches (default size 15) to manage rate limits.
- Chunking & Embedding: Each file is chunked via the AST chunker, and the resulting content is sent to an embedding provider (e.g., OpenAI
text-embedding-3-smallorvoyage-code-2). - Storage: Chunks are saved in the
Chunktable, and their vectors are inserted intoChunkEmbeddingusing raw SQL to handle thevectortype.
Sources: packages/ingestion/src/index-repository.ts:40-100, CLAUDE.md:120-123
Data Structure for Retrieval
The system stores specific metadata for each chunk to facilitate filtered retrieval:
repo_id: Ensures context never leaks across repositories.file_path: Tracks the source of the code.function_name: Captured from the ASTnameoridentifierfield.start_line/end_line: Precise location in the source file.
Sources: CLAUDE.md:60-70, packages/chunking/src/ast-chunker.ts:79-84
Retrieval Layers
The system utilizes embeddings across four distinct "Knowledge Layers" to provide comprehensive context during a review.
The diagram illustrates the parallel retrieval of context from different embedding-backed layers during a review.
Sources: packages/review/src/orchestrator.ts:145-175, packages/review/src/memory-search.ts:35-50
Retrieval Components summary
| Layer | Source Data | Retrieval Strategy |
|---|---|---|
| Codebase Context | Indexed files (Chunk table) | Cosine similarity on code embeddings. |
| Review Memory | Past AI comments + dev outcomes | Similarity search, weighted to surface ACCEPTED outcomes. |
| Repo Knowledge | Extracted dev comments | Search for architectural rules or conventions. |
Sources: CLAUDE.md:75-105, packages/review/src/memory-search.ts:55-70
Database Schema (Vector Entities)
The PostgreSQL schema uses pgvector to store embeddings separately from the main chunk data to optimize scan performance.
The ER diagram shows the relationship between code artifacts and their corresponding embedding vectors in the database.
Sources: packages/db/prisma/schema.prisma:295-345, packages/db/prisma/schema.prisma:485-515
Summary
The Code Chunking & Embedding system provides the technical foundation for the project's "compounding moat"—a review system that learns per-repo. By combining AST-aware chunking with multi-layer vector retrieval, the system moves beyond generic AI reviews to provide context-aware suggestions that respect existing patterns and past team decisions.
Sources: CLAUDE.md:160-165, packages/review/src/orchestrator.ts:255-265