AI Document Extraction
Relevant source files
The following files were used as context for generating this wiki page:
AI Document Extraction automates the population of record fields by analyzing uploaded documents such as PDFs and images. The system processes files through a multi-stage pipeline that transitions from local text extraction to regex heuristics and, finally, to Large Language Models (LLMs) when necessary.
The primary goal of this feature is to reduce data-entry friction for users while maintaining human-in-the-loop control. Extracted data is presented as suggestions that you must review and confirm before the system commits them to the database.
Sources: PRODUCT.md:25-29, PRODUCT.md:88-91
Extraction Pipeline Architecture
The extraction process follows a three-stage escalating pipeline. The system attempts each stage in order and stops once it finds a sufficient number of fields.
- Text Extraction: The system reads the raw text layer from PDFs or performs OCR on scans and photos.
- Heuristic Extraction: Built-in regex rules match specific patterns in the text to identify fields like dates, costs, and providers.
- Cloud/Local AI: If heuristics fall short, the system sends the document bytes to a configured AI provider (e.g., Anthropic, OpenAI) or a local Ollama instance.
Sources: CLAUDE.md:144-156, src/lib/documents/homeItemFieldExtraction.ts:63-95
Pipeline Flow Diagram
The following diagram illustrates the escalation logic used when a document is processed:
The pipeline ensures privacy by attempting local extraction before escalating to cloud-based AI providers.
Sources: CLAUDE.md:144-156, src/lib/documents/homeItemFieldExtraction.ts:88-93
Stage 1: Text Extraction
The textExtraction.ts module handles the initial processing of various file formats. It uses system-level binaries to extract content.
- PDFs: Uses
pdftotextfor documents with an embedded text layer. - Scans/Images: Uses
pdftoppmto rasterize PDF pages andtesseractfor Optical Character Recognition (OCR). - Word Documents:
.docand.docxfiles skip extraction and are attached as-is.
To prevent resource exhaustion, the system limits concurrent extractions to two at a time.
Sources: src/lib/documents/textExtraction.ts:16-20, src/lib/documents/textExtraction.ts:39-65, src/lib/documents/textExtraction.ts:85-87
Stage 2: Heuristic Extraction
Heuristic extraction uses domain-specific logic to parse extracted text. For example, homeItemFieldExtraction.ts searches for keywords like "repair" or "improvement" to classify a document type and uses regex to find "invoice date" or numerical cost values.
The system uses a LOW_CONFIDENCE_THRESHOLD (typically 2 fields). If heuristics find fewer fields than this threshold, the pipeline prepares to escalate to Stage 3.
Sources: src/lib/documents/homeItemFieldExtraction.ts:24-41, src/lib/documents/homeItemFieldExtraction.ts:84-86
Stage 3: AI Provider Integration
The system supports multiple "Bring Your Own Key" (BYOK) providers. Configuration is household-wide and stored encrypted at rest.
Supported Providers
| Provider | Method |
|---|---|
| Anthropic | Cloud-based LLM |
| OpenAI | Cloud-based LLM |
| Gemini | Cloud-based LLM |
| OpenRouter | Cloud-based aggregator |
| Ollama | Local self-hosted LLM |
Sources: src/lib/ai/extract.ts:21-27, src/lib/appSettings.ts, PRIVACY.md:27-31
Security and Prompt Injection
Every extraction request includes an UNTRUSTED_DOCUMENT_NOTICE. This instruction directs the model to treat document content strictly as data and to ignore any embedded instructions that might attempt prompt injection.
Sources: src/lib/ai/extract.ts:51-56
User Interface and Review
Hearth distinguishes between data sources using visual highlights. This allows you to identify which fields were found via patterns versus those inferred by an LLM.
- Heuristic Highlights: Use the "accent" color (purple/violet).
- AI Highlights: Use a neutral grey.
A "Needs Review" state is applied to records filled automatically without a person present. The highlights and "Auto" badges clear automatically the moment you edit a field.
Sources: src/lib/autoFillHighlight.ts:5-13, src/lib/autoFillHighlight.ts:24-30, GLOSSARY.md:95-97
UI Component Data Flow
The client-side applyIfEmpty function ensures that existing manual entries are not overwritten by the extraction results.
Sources: src/lib/autoFillHighlight.ts:63-68, src/components/ContractForm.tsx:125-144
Summary
AI Document Extraction in Hearth provides a privacy-first automation layer. By combining local OCR, heuristic regex matching, and optional cloud AI, the system balances convenience with data sovereignty. All extracted data requires explicit user confirmation, ensuring that the household's "system of record" remains accurate and verified.
Sources: PRODUCT.md:16-19, PRODUCT.md:88-91