SMTP Server (Ingestion)

Relevant source files

The following files were used as context for generating this wiki page:

• services/api/src/smtp.ts
• services/api/src/worker.ts
• blueprint.md
• README.md
• services/api/test/smtp.e2e.test.ts
• services/api/src/services/spamFilter.ts
• deployment_guide.md
• services/api/prisma/migrations/0002_abuse_controls/migration.sql

The SMTP Ingestion system is the primary entry point for all inbound emails into the platform. Its core purpose is to receive emails for user-configured domains, process them asynchronously, and store them for display in the web application. The architecture is designed for an "inbound-only" model, focusing on receiving and parsing emails rather than sending them, which minimizes deliverability complexities.

The system is built around a lightweight Node.js SMTP server that hands off processing to a robust background worker via a message queue. This decoupled design ensures that the SMTP server can accept emails quickly without being blocked by potentially long-running tasks like spam scanning, attachment processing, or database writes.

Sources: blueprint.md:73-77, README.md:21-22

Architecture and Data Flow

The inbound email pipeline consists of several key components that work together to receive, process, and store emails. The process begins when an external mail server sends an email to a domain managed by the platform.

Sources: blueprint.md:79-88

Inbound Pipeline Overview

The following diagram illustrates the high-level flow of an email from the internet to the application's database.

Core Components

Component	Technology	Purpose	Source File
MTA (Receiver)	Postfix (recommended)	Receives email from the internet via port 25 and forwards it to the internal SMTP service.	blueprint.md:84
SMTP Ingestion	smtp-server (Node.js)	A lightweight server that listens for incoming mail, saves the raw email to disk, and queues a job for processing.	services/api/src/smtp.ts:19-64
Message Queue	BullMQ (Redis)	Decouples the ingestion server from the processing logic, providing persistence and retry capabilities for email jobs.	services/api/src/smtp.ts:39-48
Processing Worker	BullMQ Worker	Pulls jobs from the queue, parses the raw email, performs validation, runs spam checks, and persists the data.	services/api/src/worker.ts
Spam Filtering	Rspamd	An optional but recommended service that provides spam scoring for each message.	services/api/src/services/spamFilter.ts, services/api/src/worker.ts:200-209
Database	PostgreSQL	Stores email metadata, inbox information, domains, and user data.	README.md:20
Storage	Disk / S3-compatible	Stores raw .eml files temporarily and email attachments permanently.	services/api/src/smtp.ts:24-27, services/api/src/worker.ts:107-109

SMTP Ingestion Service

The ingestion service is implemented in services/api/src/smtp.ts using the smtp-server library. It is intentionally minimal to ensure high availability and fast response times.

Functionality

1. Listens for Connections: The server listens on a configured port (default 2525) for SMTP connections. Authentication (AUTH) is explicitly disabled as it's an inbound-only receiver.
2. Receives Email Data: The onData event handler streams the raw email content directly to a temporary .eml file in the configured storage directory (storage/raw).
3. Queues Processing Job: Once the stream ends, it adds a job to the emailQueue (BullMQ). The job payload contains the path to the raw .eml file and envelope data like the recipient address and sender's remote IP.
4. Responds to Client: It immediately responds to the sending MTA, confirming receipt, without waiting for the email to be fully processed.

Sources: services/api/src/smtp.ts:19-51

// services/api/src/smtp.ts

export const startSmtpServer = (logger: Logger, port = appConfig.smtpPort) => {
  const server = new SMTPServer({
    disabledCommands: ["AUTH"],
    logger: false,
    async onData(stream, session, callback) {
      try {
        const rawDir = await ensureRawStorageDir();
        const filename = `${Date.now()}-${generateToken()}.eml`;
        const filePath = path.join(rawDir, filename);

        const writeStream = (await import("fs")).createWriteStream(filePath);
        stream.pipe(writeStream);

        stream.on("end", async () => {
          // ...
            await emailQueue.add("inbound", {
              rawPath: filePath,
              envelope: {
                rcptTo: session.envelope.rcptTo,
                remoteAddress: session.remoteAddress,
              },
            });
            logger.info({ filePath }, "queued outbound email");
            callback();
          // ...
        });
      } catch (err) {
        // ...
      }
    },
  });

  server.listen(port, () => {
    logger.info(`SMTP ingest listening on :${port}`);
  });

  return server;
};

This code snippet shows the core logic of the SMTP server, which saves the incoming email stream to a file and queues it for background processing.

Sources: services/api/src/smtp.ts:19-64

Email Processing Worker

The worker, defined in services/api/src/worker.ts, is responsible for all the heavy lifting. It picks up jobs from the BullMQ queue and executes a series of steps to validate, enrich, and store the email.

Processing Sequence

The following diagram details the sequence of events within the worker for a single email job.

Key Worker Logic

• Parsing: Uses mailparser.simpleParser to convert the raw .eml content into a structured ParsedMail object.
• Recipient Handling: Identifies the recipient and uses ensureDomainAndInbox to find the corresponding domain and inbox in the database. If ALLOW_AUTO_DOMAIN_CREATION is true, it will create missing domains and inboxes on the fly.
• Abuse Control:
  • Sender Blocklist: Rejects emails from domains listed in BLOCKED_SENDER_DOMAINS.
  • Dynamic Rules: Evaluates rules from the Rule table, which can block emails based on sender domain, email, recipient, or source IP.
  • Rate Limiting: Enforces limits on the number of emails received per IP, per domain, and per inbox within a configured time window.
  • Quotas: Checks if the inbox or domain has exceeded its total message quota.
• Spam & Virus Scanning:
  • Integrates with Rspamd via checkSpam to get a spam score. If the score exceeds a threshold or the action is reject, the job fails.
  • Scans attachments for viruses using scanBuffer. If a virus is detected, the job fails.
• Persistence:
  • Saves the parsed email metadata (subject, from, to, body, headers) to the Message table in PostgreSQL.
  • Saves attachments to the configured storage (disk or S3) and records their metadata in the Attachment table.
• Cleanup: Deletes the temporary raw .eml file after successful processing.

Sources: services/api/src/worker.ts, services/api/src/services/spamFilter.ts, services/api/prisma/migrations/0002_abuse_controls/migration.sql

Configuration

The behavior of the SMTP ingestion system is controlled by several environment variables.

Variable	Description	Default	Source
SMTP_PORT	The port for the SMTP ingestion server to listen on.	2525	services/api/src/smtp.ts:20
STORAGE_DIR	The base directory for storing temporary raw emails and attachments.	./storage	services/api/src/smtp.ts:13
ALLOW_AUTO_DOMAIN_CREATION	If true, domains and inboxes are created automatically upon receiving an email. Set to false in production.	true (in compose)	README.md:104, services/api/src/worker.ts:40
RSPAMD_URL	URL of the Rspamd service for spam checking.	http://rspamd:11333	services/api/src/services/spamFilter.ts:11
SMTP_RATE_LIMIT_PER_IP	Max emails from a single IP in the time window.	300	services/api/src/worker.ts:145
SMTP_RATE_LIMIT_PER_DOMAIN	Max emails for a single domain in the time window.	500	services/api/src/worker.ts:148
SMTP_RATE_LIMIT_PER_INBOX	Max emails for a single inbox in the time window.	200	services/api/src/worker.ts:151
QUOTA_MESSAGES_PER_INBOX	Maximum number of messages an inbox can hold.	(not specified)	services/api/src/worker.ts:191
QUOTA_MESSAGES_PER_DOMAIN	Maximum number of messages a domain can hold.	(not specified)	services/api/src/worker.ts:195

Testing

End-to-end tests for the SMTP ingestion flow are located in services/api/test/smtp.e2e.test.ts. These tests use nodemailer to act as an SMTP client and send a real email to the running SMTP server. After sending, the test uses the API to verify that the message was correctly processed and stored in the database.

// services/api/test/smtp.e2e.test.ts

it("accepts inbound SMTP and surfaces message via API", async () => {
    const transport = nodemailer.createTransport({
      host: SMTP_HOST,
      port: smtpPort,
      secure: false,
      tls: { rejectUnauthorized: false },
    });

    await transport.sendMail({
      from: "sender@test.local",
      to: `hello@${TEST_DOMAIN}`,
      subject: "Hello inbound",
      text: "This is a test email.",
    });

    // allow ingest to persist
    await new Promise((resolve) => setTimeout(resolve, 300));

    const inboxes = await request(app.server)
      .get(`/inboxes?domain=${TEST_DOMAIN}`)
      .set("Authorization", `Bearer ${token}`);
    // ... assertions
});

This test case demonstrates how to send an email to the local SMTP server and then query the API to confirm its arrival.

Sources: services/api/test/smtp.e2e.test.ts:68-96

Summary

The SMTP Ingestion system is a critical, well-architected component of the email platform. By using a decoupled, queue-based architecture, it achieves high throughput and resilience. The processing worker enforces a multi-layered defense against abuse through rate limiting, quotas, and spam filtering, ensuring the platform remains stable and secure. The entire flow is designed to be asynchronous, from the moment an email is received until it is fully processed and stored.