Legal RAG & Corpus Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Legal RAG (Retrieval-Augmented Generation) & Corpus Engine provides the infrastructure for analyzing Telegram messages against the Criminal Code of the Russian Federation (УК РФ). It processes raw legal texts into a searchable vector database and implements a multi-stage pipeline to identify potential legal violations in chat history.
The system combines deterministic text processing, vector similarity search via Cloudflare Vectorize, and LLM-based verification to ensure high-accuracy legal references.
Core Architecture
The engine operates through three main phases: ingestion of legal documents, semantic retrieval of relevant articles, and a tiered classification process.
The diagram shows the data flow from legal text ingestion to real-time message analysis.
Corpus Preparation
The Corpus Processor transforms raw legal documents into structured, searchable units. It normalizes text and removes obsolete legal fragments to maintain an accurate corpus.
Document Processing Logic
- Normalization: The engine normalizes whitespace and required fields (lawCode, title, versionDate) to ensure idempotency.
Sources: src/features/legal-rag/corpus.ts:40-45, tests/legal-rag-corpus.test.ts:41-55 - Fragment Stripping: The system removes obsolete fragments, such as articles or paragraphs marked "Утратил силу" (Repealed), using regular expressions.
Sources: src/features/legal-rag/corpus.ts:107-124 - Chunking: Articles are split into chunks with a default maximum length of 1800 characters to optimize vector embedding performance.
Sources: src/features/legal-rag/corpus.ts:38, src/features/legal-rag/corpus.ts:88-105 - Deterministic Identification: Each chunk receives a
vectorIdbased on the law code, article number, index, and a SHA-256 hash of its content.
Sources: src/features/legal-rag/corpus.ts:63-70
Obsolete Fragment Removal
| Pattern Type | Description |
|---|---|
| Repealed Articles | Entire articles explicitly marked as having lost force. |
| Numbered Clauses | Specific numbered items within articles with effective dates. |
| Notes | Parenthetical or footer notes regarding legal updates. |
Sources: src/features/legal-rag/corpus.ts:107-130, tests/legal-rag-corpus.test.ts:79-99
Ingestion Pipeline
The ingestion process populates the vector index and relational metadata. The pipeline supports batch processing and version-scoped cleanup.
Data Ingestion Steps
- Batch Ingest: The endpoint
handleLegalRagIngestBatchreceives structured documents and chunks.
Sources: tests/features/legal-rag-ingest.test.ts:162-175 - Embeddings Generation: The engine uses the
@cf/baai/bge-m3model via Cloudflare Workers AI to generate embeddings for each text chunk.
Sources: src/core/providers/legal-rag-provider.ts:30, tests/features/legal-rag-ingest-e2e.test.ts:168 - Vector Persistence: Embeddings are stored in the
LEGAL_RAG_INDEXVectorize index with metadata includinglaw_codeanddocument_checksum.
Sources: tests/features/legal-rag-ingest.test.ts:187-195 - Metadata Storage: relational data (article titles, full text, source URLs) is stored in the
legal_chunkstable in D1.
Sources: tests/features/legal-rag-ingest-e2e.test.ts:110-120 - Scoped Cleanup: If
replaceExistingis true, the system deletes vector IDs and D1 records for previous versions of the specified law code after the new version is successfully persisted.
Sources: tests/features/legal-rag-ingest.test.ts:210-230
Retrieval Engine
The LegalRagProvider handles semantic lookups during message analysis. It uses multi-variant queries to increase recall.
Sequence diagram showing the retrieval of legal references based on message context.
Key Provider Functions
- buildRetrievalTexts: Constructs up to 3 query variants using the target text, the semantic prefilter search query, and a compact context window.
Sources: src/core/providers/legal-rag-provider.ts:91-103 - retrieveMatches: Queries the vector index for each variant and deduplicates results, keeping the highest score for each hit.
Sources: src/core/providers/legal-rag-provider.ts:124-143 - loadLegalReferences: Joins the resulting vector IDs with relational data from D1 to provide human-readable article information.
Sources: src/core/providers/legal-rag-provider.ts:175-207
Analysis Pipeline
The CriminalCodeAnalyzerDO orchestrates a tiered analysis to balance cost and accuracy.
Analysis Tiers
- Local Prefilter: A regex-based check to identify obvious threats or commands that should be ignored.
Sources: src/durable-objects/criminal-code-analyzer-do.ts:255-260 - Semantic Prefilter (LLM): A "nano" model (e.g., GPT-4o-mini) evaluates if the message contains potential criminal signs and generates a
searchQuery.
Sources: src/durable-objects/criminal-code-analyzer-do.ts:553-580, src/durable-objects/criminal-code-analyzer-do.ts:655-675 - Legal RAG Retrieval: Fetches relevant articles from the vector database.
Sources: src/core/providers/legal-rag-provider.ts:71-85 - Final Judge (LLM): If RAG returns hits, a high-reasoning model (e.g., GPT-5-nano) performs the final classification, checking if the message context fits the specific disposition of the retrieved legal articles.
Sources: src/durable-objects/criminal-code-analyzer-do.ts:984-1000, src/durable-objects/criminal-code-analyzer-do.ts:1037-1065
Configuration Options
| Variable | Type | Default | Description |
|---|---|---|---|
LEGAL_RAG_TOP_K | Number | 5 | Number of legal chunks to retrieve. |
LEGAL_RAG_MIN_SCORE | Number | 0.55 | Minimum similarity score for RAG hits. |
CRIMINAL_FINAL_JUDGE_ENABLED | Boolean | true | Enable LLM verification of RAG results. |
CRIMINAL_PREFILTER_MIN_CONFIDENCE | Number | 0.55 | Confidence threshold to trigger RAG. |
Sources: src/core/providers/legal-rag-provider.ts:31-33, src/durable-objects/criminal-code-analyzer-do.ts:705-710, src/durable-objects/criminal-code-analyzer-do.ts:1005-1010
Summary
The Legal RAG & Corpus Engine enables the bot to move beyond simple keyword matching. By using vector embeddings and a multi-stage LLM pipeline, it identifies complex legal violations while maintaining high performance through aggressive pre-filtering and deterministic corpus processing. The engine's reliance on versioned checksums ensures that updates to the Russian Criminal Code can be ingested without corrupting existing historical data.