Document Processing Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Document Processing Pipeline is a core module within the SecondLayer project designed to transform raw legal files into structured, machine-readable data. It handles the ingestion, parsing, semantic structuring, and analysis of various document formats, including PDF, DOCX, and HTML. The pipeline is utilized across multiple services such as Legal Research, Due Diligence, and the Vault to facilitate AI-powered legal document analysis and risk detection.
The system employs a multi-stage approach: initially converting binary files into text via specialized parsers (including OCR where necessary), then logically segmenting the text into semantic sections, and finally performing high-level analysis such as clause extraction, summarization, and version comparison.
Sources: lexconfig/services_tooling_map.txt:31-41, mcp_backend/src/api/document-analysis-tools.ts:63-75
Pipeline Architecture and Flow
The pipeline follows a structured sequence of operations to ensure data integrity and analytical depth. It begins with intent classification and ends with a validated response supported by citations.
This diagram represents the "Canonical Pipeline (v1)" used for processing legal questions and documents.
Sources: lexconfig/pipeline_contracts.txt:13-21, mcp_backend/src/api/document-analysis-tools.ts:257-285
Document Ingestion and Parsing
The parsing stage is responsible for extracting raw text and metadata from various file formats. It uses a tiered strategy to ensure maximum accuracy, particularly for complex PDF and HTML sources.
Parsing Strategies
| Format | Primary Method | Fallback/Advanced Method |
|---|---|---|
| Native Text Extraction | OCR via Playwright + Google Vision API | |
| DOCX | Mammoth (native) | OCR |
| HTML | Screenshot | OCR |
Sources: mcp_backend/src/api/document-analysis-tools.ts:66-70, mcp_backend/src/api/tests/document-analysis-e2e.test.ts:114-162
Key Components
- DocumentParser: The primary service that coordinates file conversion. It supports Ukrainian, Russian, and English languages.
- parse_document Tool: An MCP tool that accepts a Base64-encoded file and returns a
ParsedDocumentobject containing the extracted text, page count, and source metadata (e.g., 'ocr' or 'native').
Sources: mcp_backend/src/api/document-analysis-tools.ts:98-124, mcp_backend/src/services/document-parser.ts:1-20
Semantic Sectioning
Once raw text is extracted, the extract_document_sections tool (implemented via the SemanticSectionizer) transforms it into logical sections. This is critical for legal documents where meaning is derived from specific parts such as "Facts," "Reasoning," and "Decision."
The sectionizer identifies boundaries in court cases and legislation to enable granular retrieval.
Sources: lexconfig/pipeline_contracts.txt:140-165, lexconfig/spec.txt:28-30
Analytical Tools
After the document is structured, the pipeline provides tools for deep legal analysis. These tools often leverage LLMs (Large Language Models) like GPT-4o, guided by ModelSelector for budget-aware execution.
Key Analysis Tools
- extract_key_clauses: Identifies specific legal clauses (e.g., payments, liability, termination) and assigns a risk level (low/medium/high).
- summarize_document: Generates both an Executive Summary for leadership and a Detailed Summary segmented by sections.
- compare_documents: Performs a semantic comparison between two versions of a document, classifying changes as critical (amounts/dates), significant (new clauses), or minor (formatting).
Sources: mcp_backend/src/api/document-analysis-tools.ts:130-186, mcp_backend/src/api/document-analysis-tools.ts:237-255
Change Importance Classification
The pipeline classifies the importance of document changes based on specific patterns:
private classifyChangeImportance(text: string): 'critical' | 'significant' | 'minor' {
const lowerText = text.toLowerCase();
// Critical: money, dates, numbers, legal terms
if (/\d+[\s]*грн|uah|usd|eur|\$|€/i.test(text) ||
/термін|строк|penalty|штраф|відповідальність/i.test(lowerText)) {
return 'critical';
}
// Significant: new clauses, rights, obligations
if (/право|обов'язок|obligation|зобов'язання/i.test(lowerText) && text.length > 50) {
return 'significant';
}
return 'minor';
}
Sources: mcp_backend/src/api/document-analysis-tools.ts:316-335
Quality Control and Validation
To prevent hallucinations and ensure technical accuracy, the pipeline includes a final validation step.
- validate_response: This MCP tool checks the synthesized answer against the original document sections. Every key claim MUST have a corresponding citation or anchor within the source document.
- Citation Requirements: Citations must include
source_id,quote, andurlto be considered valid. If a claim is unsupported, the validator returnsissuesthat flag the response for human review.
Sources: lexconfig/pipeline_contracts.txt:216-248, lexconfig/services_tooling_map.txt:100-103
Summary
The Document Processing Pipeline provides a robust framework for handling legal data, progressing from raw binary ingestion to semantic understanding and risk analysis. By combining native extraction with OCR and LLM-powered validation, it ensures that legal professionals receive structured, cited, and accurate information from their documents.
Sources: mcp_backend/src/api/tests/document-analysis-e2e.test.ts:246-285, lexconfig/spec.txt:72-78