Email Ingestion via IMAP
Relevant source files
The following files were used as context for generating this wiki page:
Email Ingestion via IMAP is a core feature of the KnowledgeVault (ragappv3) system designed to automate the collection of technical documents. It allows the system to poll specified IMAP mailboxes, extract attachments, and route them to appropriate vaults for indexing and RAG-based retrieval.
The system utilizes an asynchronous polling mechanism to check for new emails at configurable intervals. This ensures that documentation sent via email is automatically incorporated into the knowledge base without manual user intervention in the web UI.
Sources: README.md:16, README.md:330, backend/app/config.py:355-364
System Architecture
The Email Ingestion system is integrated into the backend as a dedicated service. It interacts with the global configuration for connection parameters and utilizes the standard document processing pipeline once an attachment is retrieved.
Data Flow Diagram
The following diagram illustrates how an email moves from the mail server to the KnowledgeVault vector store.
The flow starts with the EmailService polling the external IMAP server. If attachments are found and validated, they are passed into the standard document processing and vectorization pipeline.
Sources: README.md:47-60, backend/app/config.py:365-385
Configuration Options
The IMAP ingestion service is highly configurable through environment variables, allowing administrators to define polling intervals, security settings, and allowed file types.
| Option | Type | Default | Description |
|---|---|---|---|
IMAP_ENABLED | boolean | false | Master switch to enable email ingestion. |
IMAP_HOST | string | "" | Hostname of the IMAP server. |
IMAP_PORT | integer | 993 | Server port (993 for SSL). |
IMAP_USERNAME | string | "" | Username for the email account. |
IMAP_PASSWORD | SecretStr | "" | Password for the email account. |
IMAP_USE_SSL | boolean | true | Whether to use SSL/TLS for the connection. |
IMAP_POLL_INTERVAL | integer | 60 | Polling frequency in seconds. |
IMAP_MAX_ATTACHMENT_SIZE | integer | 10485760 | Max size per attachment (default 10MB). |
Sources: backend/app/config.py:355-364, README.md:330-340
Document Validation and Security
To maintain system integrity and prevent malicious file ingestion, the Email Ingestion service enforces strict validation rules on all incoming attachments.
Allowed MIME Types
The system only processes specific file formats relevant to technical documentation. These include:
- Documents: PDF, TXT, Markdown, DOCX.
- Data/Code: CSV, JSON, SQL, Python, JavaScript.
- Spreadsheets/Slides: XLS, XLSX, PPTX.
- Web/Config: HTML, CSS, XML, YAML, Log files.
Security Hardening
Following the project's security hardening specifications, uploaded documents (including those from email) undergo "Magic Byte" validation to prevent extension spoofing. For example, a file ending in .pdf must contain the %PDF header bytes to be accepted.
Sources: backend/app/config.py:365-385, specs/security-ux-hardening/SPEC.md:52-57, frontend/src/lib/api.ts:255-275
Implementation Details
The backend logic for this feature is centered around the EmailService and the email.py API routes.
Polling Logic
The polling interval is governed by imap_poll_interval. At every interval, the system:
- Connects to the server defined in
imap_host. - Authenticates using
imap_usernameand the encryptedimap_password. - Selects the mailbox defined in
imap_mailbox(defaults to "INBOX"). - Iterates through unread messages to find attachments matching
imap_allowed_mime_types.
Code Snippet: Configuration Model
# backend/app/config.py:355-364
imap_enabled: bool = False
imap_host: str = ""
imap_port: int = 993
imap_username: str = ""
imap_password: SecretStr = SecretStr("")
imap_use_ssl: bool = True
imap_mailbox: str = "INBOX"
imap_poll_interval: int = 60 # seconds
Sources: backend/app/config.py:355-364, README.md:95-105
Integration with Vaults
Documents ingested via email are routed to specific vaults. While the system supports multi-tenancy, email routing ensures that documents are placed in the correct vault_uploads_dir, which is a canonical per-vault storage directory keyed by an integer ID. This maintains the logical separation of data between different organizations or projects within the same KnowledgeVault instance.
Sources: backend/app/config.py:476-485, README.md:360-370
Conclusion
Email Ingestion via IMAP provides a seamless, automated bridge between standard communication channels and the KnowledgeVault RAG pipeline. By combining configurable polling with rigorous security checks like MIME type validation and magic byte verification, the system ensures that the knowledge base remains current and secure with minimal administrative overhead.