Email Flow (Ingestion to Display)

Relevant source files

The following files were used as context for generating this wiki page:

• services/api/src/smtp.ts
• services/api/src/queue/emailQueue.ts
• services/api/src/worker.ts
• services/api/src/routes/messages.ts
• blueprint.md
• services/api/test/smtp.e2e.test.ts
• services/api/src/routes/filters.ts
• README.md

The email processing pipeline is a core component of the platform, designed for robust and scalable handling of inbound emails. It follows an asynchronous, queue-based architecture to decouple the initial email reception from the more intensive tasks of parsing, validation, storage, and filtering. This ensures that the SMTP server remains highly responsive while background workers handle the processing load.

The flow begins when the self-hosted SMTP server receives an email. Instead of processing it synchronously, the server quickly writes the raw email content to a temporary file and dispatches a job to a Redis-backed queue (BullMQ). A dedicated worker process picks up this job, parses the email, enforces security and usage policies (rate limits, quotas, spam/virus checks), persists the message and its attachments to the PostgreSQL database, and applies any user-defined filters. Once stored, the email data is made available to the frontend application via a set of RESTful API endpoints.

Sources: blueprint.md, services/api/src/smtp.ts, services/api/src/worker.ts

High-Level Inbound Flow

The following diagram illustrates the complete journey of an email from reception to being viewable by the user.

1. SMTP Ingestion

The entry point for all inbound email is a custom SMTP server built using the smtp-server library. Its primary responsibility is to accept the email connection, receive the email data, and quickly hand it off for asynchronous processing.

Sources: services/api/src/smtp.ts, README.md

Key Components

• smtp-server: A Node.js library used to create the SMTP server. It listens on the configured port (e.g., 2525 for local dev) for incoming email.
• onData handler: This is the crucial event handler that fires when the server has received the complete email content.
• Temporary File Storage: To avoid holding large emails in memory, the raw email stream is piped directly to a temporary file on disk.
• emailQueue: A BullMQ queue instance that manages jobs for the email worker.

Ingestion Sequence Diagram

This diagram shows the sequence of events during the initial ingestion phase.

2. Asynchronous Worker Processing

Once a job is placed on the emailQueue, a background worker process, defined in worker.ts, picks it up. This worker performs all the heavy lifting, ensuring the main API remains unblocked.

Sources: services/api/src/worker.ts

Worker Logic Flow

The worker executes a series of steps to process, validate, and store the email.

Key Processing Steps

1. Parsing: The raw email content is parsed using mailparser's simpleParser to extract headers, body (text and HTML), and attachments.
   Sources: services/api/src/worker.ts:157
2. Validation & Security:
   • Blocked Senders: The sender's domain is checked against a configurable blocklist (blockedSenderDomains).
   • Abuse Rules: Custom rules for sender/recipient domains and IPs are evaluated.
   • Rate Limits: Inbound rates are checked per IP, per domain, and per inbox.
   • Quotas: The total number of messages per inbox and per domain is checked against configured limits.
   • Spam/Virus: The email is scanned for spam using Rspamd and for viruses using ClamAV (if configured).
     Sources: services/api/src/worker.ts:176-224
3. Persistence:
   • A new Message record is created in the PostgreSQL database via Prisma.
   • Attachments are saved using the storageService (to disk or S3) and Attachment records are created. Attachments are also checked against size and MIME type allowlists.
     Sources: services/api/src/worker.ts:80-117, services/api/src/worker.ts:226-245
4. Post-processing:
   • Filters: User-defined filters (e.g., move to folder, add label, delete) are applied to the newly created message.
   • Cleanup: The temporary raw email file is deleted.
     Sources: services/api/src/worker.ts:260-276

3. Data Model and Storage

The core data is stored in a PostgreSQL database, managed by Prisma ORM. The schema is designed to link domains, inboxes, messages, and attachments.

Sources: blueprint.md

Entity Relationship Diagram

This diagram shows the relationships between the main database tables involved in the email flow.

4. API Endpoints for Retrieval

After an email is successfully processed and stored, it can be retrieved through the API. The messageRoutes define the endpoints for accessing message and attachment data.

Sources: services/api/src/routes/messages.ts

Key Retrieval Endpoints

Method	Endpoint	Description
GET	/inboxes/:id/messages	Retrieves a paginated list of messages for a specific inbox. Supports filtering by query, date, read status, and attachments.
GET	/messages/:id	Fetches a single message by its ID, including its attachments and inbox details.
DELETE	/messages/:id	Soft-deletes a message and its associated attachments.
PATCH	/messages/:id/read	Marks a message as read or unread.
GET	/attachments/:id/download	Downloads the content of a specific attachment.
GET	/messages/:id/export	Exports the full message, including headers and attachments, as a standard .eml file.

Sources: services/api/src/routes/messages.ts

Client Data Fetching Sequence

This diagram illustrates how a web client fetches and displays a list of messages for an inbox.

5. Frontend Display

The frontend application, built with React, consumes the API endpoints to render the email data. When a user selects a message, the application fetches the full message details, including the HTML body.

For security, the HTML content of the email (htmlBody) is typically rendered within a sandboxed <iframe> to prevent malicious scripts from executing in the context of the main application. Attachments are displayed as a list of links that point to the /attachments/:id/download endpoint.

Sources: services/api/src/routes/messages.ts:211-225, README.md