Alerts & SMTP Notifications

Relevant source files

The following files were used as context for generating this wiki page:

• internal/alert/service.go
• internal/alert/smtp.go
• internal/alert/settings.go
• internal/api/alerts.go
• web/src/pages/alerts-page.tsx

The Alerts system implements a state-driven monitoring engine that detects failures in critical operations such as subscription refreshes, proxy group availability, and data plane health. It uses a state machine to track firing and resolved alerts, persisting this state in SQLite to prevent notification storms across service restarts.

SMTP Notifications serve as the primary external communication channel for these alerts. The system sends an email when an alert first fires and a recovery notice once the condition clears. Users can manage these notifications via a dedicated UI in the web dashboard, which facilitates SMTP configuration and manual alert acknowledgment.

System Architecture

The alerting system consists of four primary components: Detectors, the Alert Service, the Repository, and Notification Channels. Detectors report the current state of the system, while the Alert Service reconciles these findings with the stored state.

The Alert Service evaluates findings at regular intervals. It uses a repeat interval (defaulting to 6 hours) to re-notify for ongoing critical issues unless they are acknowledged.
Sources: internal/alert/service.go:1-60, internal/alert/service.go:94-110

Alert Lifecycle and State Machine

Alerts transition between firing and resolved statuses. When a Finding is reported by a detector that does not match an existing record, the system creates a new alert and sends a notification. If a previously firing alert is no longer reported in subsequent evaluations, it is marked as resolved and a recovery email is dispatched.

SMTP Configuration & Security

The SMTP channel supports three security modes: plain (none), STARTTLS, and implicit TLS (SMTPS). Settings are stored using AEAD encryption to ensure that sensitive credentials, such as SMTP passwords, are never exposed in logs or plain database text.

Configuration Fields

Field	Type	Description
Host	String	SMTP server hostname or IP address.
Port	Integer	Server port (e.g., 587 for STARTTLS, 465 for TLS).
Security	String	Encryption method: none, starttls, or tls.
Username	String	SMTP authentication username.
Password	String	SMTP authentication password (write-only).
From	String	The sender email address.
To	[]String	List of recipient email addresses.

Sources: internal/alert/settings.go:12-21, internal/alert/smtp.go:21-50

Security Implementation

The system uses an internal Cipher interface to Seal and Open SMTP settings using the alert:smtp-settings additional authenticated data (AAD) tag. When users update settings, leaving the password field empty in the UI preserves the previously stored secret, preventing accidental credential loss.
Sources: internal/alert/settings.go:3-10, internal/alert/settings.go:68-75

API Endpoints

The Alert API provides endpoints for listing alerts, acknowledging active issues, and managing SMTP settings.

Endpoint	Method	Action
/api/v1/alerts	GET	Lists alert records (filtered by status and limit).
/api/v1/alerts/:id/ack	POST	Acknowledges a firing alert to silence repeat notifications.
/api/v1/alerts/settings	GET	Retrieves redacted SMTP configuration.
/api/v1/alerts/settings	PUT	Updates SMTP configuration.
/api/v1/alerts/settings/test	POST	Sends a test email to verify the SMTP configuration.

Sources: internal/api/alerts.go:42-132

SMTP Delivery Logic

The SMTPChannel performs a specific sequence of operations to deliver notifications:

1. Dial: Establishes a TCP connection with a 10-second timeout.
2. Security: Upgrades to TLS if the mode is tls.
3. Handshake: Initializes the SMTP client and checks for STARTTLS or AUTH extensions.
4. Authentication: Performs PlainAuth if a username is provided.
5. Data Transfer: Formats a MIME message with standard headers (From, To, Subject, Content-Type).
6. Cleanup: Closes the writer and sends the QUIT command.

UI Integration

The web interface provides a real-time view of alerts and a configuration panel for SMTP.

• Firing Alerts: Displays the rule name, target, severity, message, and trigger time.
• History: Shows the last 30 resolved alerts with their recovery timestamps.
• Acknowledgment: Users can click "Confirm" to mark an alert as acknowledged, which updates the acknowledged boolean in the backend and stops repeat notifications.
• Auto-refresh: The dashboard polls the alert status every 30 seconds.

Sources: web/src/pages/alerts-page.tsx:28-60, web/src/pages/alerts-page.tsx:84-120

Conclusion

The Alerts and SMTP Notifications module provides a robust mechanism for monitoring system health and ensuring operators are informed of critical failures. By combining persistent state management with encrypted configuration and flexible SMTP support, the system balances reliability with security. Operators should regularly check the "Alerts" dashboard to acknowledge known issues and verify SMTP connectivity using the integrated test tool.