RSS Feed Automation

Introduction

The RSS Feed Automation system is a core feature of the newsletter program, designed to automatically monitor specified RSS feeds, detect new content, and generate and distribute newsletters based on that content. This automates the process of content curation and delivery, ensuring that subscribers receive timely updates from various sources.

The system is built around three main components: an RSSMonitor for polling feeds, a ContentProcessor for transforming RSS items into email-friendly HTML, and a NewsletterSender for distributing the generated content to all active subscribers. The entire process is configurable and designed to be robust, with error handling and retry logic.

System Architecture

The automation process follows a linear flow, starting from polling RSS feeds and ending with sending emails to subscribers. Each step is handled by a dedicated component.

This diagram shows the high-level interaction between the components of the RSS automation system.

Core Components

Sources: newsletter-program/src/tests/integration/rss-monitoring-automation.test.js:133-135, 303-304, 355-356, newsletter-program/src/core/content-processor.js:4-9

RSS Feed Polling

The RSSMonitor component is responsible for polling the configured RSS feeds. It fetches the XML content, parses it to extract items, and identifies new entries that have not been processed before.

Polling and Detection Flow

The following diagram illustrates the logic for checking a feed for new items.

Key Operations

• Fetching: Uses fetch to retrieve the XML content from the feed URL. It includes error handling for network issues.
• Parsing: Extracts <item> elements from the RSS XML. Each item typically contains a title, description, link, guid, and pubDate.
• New Item Detection: Compares the guid of fetched items against a stored cache of previously seen items for that feed. Any item not in the cache is considered new.
• Caching: After a check, the RSSMonitor updates a cache file with the latest items to prevent them from being processed again. The cache stores lastChecked and lastItemDate timestamps, along with a list of item guids.

Sources: newsletter-program/src/tests/integration/rss-monitoring-automation.test.js:133-157, 180-226, 247-270

Feed Configuration

RSS feeds are configured in the theme's configuration files. Each feed requires a URL and can have its own template and sending frequency.

newsletter:
  rss_feeds:
    - url: https://your-blog.com/feed.xml
      template: newsletter
      frequency: weekly

Sources: themes/matrix-flow/README-newsletter.md:111-115

Content Processing and Email Generation

Once new RSS items are detected, the ContentProcessor takes over to convert them into an HTML email.

ContentProcessor Class

This class is responsible for sanitizing content and rendering templates.

Email Generation Sequence

The process involves loading a template, sanitizing the RSS content, and injecting the data into the template placeholders.

Content Sanitization

To prevent XSS attacks and ensure email client compatibility, all content from the RSS feed is sanitized. The sanitizeContent method uses the sanitize-html library to strip dangerous tags and attributes.

• Allowed Tags: p, br, strong, em, ul, ol, li, a, img, and heading tags (h1-h6).
• Allowed Attributes: href, src, alt, title, etc. Dangerous attributes like onerror are removed.

Sources: newsletter-program/src/core/content-processor.js:47-64, newsletter-program/src/tests/integration/rss-monitoring-automation.test.js:401-410

Email Template

The system uses an HTML template to structure the final email. The template contains placeholders like {{title}} and {{description}} that are replaced with data from the RSS item. It also supports looping through multiple items using {{#each items}}.

An example from the rss-newsletter.html template:

<div class="rss-item">
    <h2><a href="{{link}}" target="_blank">{{title}}</a></h2>
    
    <div class="rss-meta">
        <span class="date">{{formatted_date}}</span>
        {{#author}}<span class="author">{{author}}</span>{{/author}}
    </div>
    
    {{#description}}
    <div class_name="rss-description">
        {{{description}}}
    </div>
    {{/description}}
    
    <a href="{{link}}" class="read-more" target="_blank">Read full article →</a>
</div>

Sources: newsletter-program/src/templates/rss-newsletter.html:171-190, newsletter-program/src/core/content-processor.js:18-26

Automatic Email Sending

After the email is composed, the NewsletterSender component sends it to all active subscribers.

Sending Process

1. Get Subscribers: Retrieves a list of all active subscribers.
2. Iterate and Send: Loops through each subscriber and sends the composed email.
3. Retry Logic: If an email fails to send (e.g., due to a rate limit error), the system will automatically retry the send operation a configured number of times before marking it as a failure.
4. Record Analytics: The results of the send operation (number of sent and failed emails) are logged for analytics.

Sources: newsletter-program/src/tests/integration/rss-monitoring-automation.test.js:325-353, 355-375

This sequence diagram illustrates the sending and retry logic.