RSS Data Caching

Relevant source files

The following files were used as context for generating this wiki page:

• newsletter-program/src/tests/integration/rss-monitoring-automation.test.js
• newsletter-program/src/core/content-processor.js
• newsletters/README.md
• newsletter-program/src/templates/rss-newsletter.html
• themes/matrix-flow/README-newsletter.md

Introduction

The RSS Data Caching system is a core component of the automated newsletter generation feature. Its primary purpose is to monitor specified RSS feeds, detect new content, and cache previously seen items to prevent duplicate processing. This system polls RSS feeds, compares the fetched items against a local cache, and identifies new entries to be processed for inclusion in newsletters.

The caching mechanism is file-based, relying on JSON files for both feed configuration and item caching. This design avoids external database dependencies and simplifies the overall architecture. The system is designed to be robust, handling fetch errors gracefully and ensuring that only new content triggers the newsletter composition workflow.

System Architecture

The RSS caching and processing flow involves several key components that work together to fetch, parse, cache, and transform RSS data into email content. The RSSMonitor class is central to this process, orchestrating the polling and caching logic.

The following diagram illustrates the high-level data flow from polling the RSS feed to the final email generation.

This diagram shows the process starting with the RSSMonitor, which fetches and parses the feed, compares items against the cache, updates the cache with new items, and then triggers the ContentProcessor to generate and send a newsletter.

Sources: newsletter-program/src/tests/integration/rss-monitoring-automation.test.js:150-170, themes/matrix-flow/README-newsletter.md

Data Structures

The system uses two primary JSON files for configuration and caching.

Feed Configuration (rss-feeds.json)

This file (or equivalent in-memory configuration) defines the list of RSS feeds to be monitored. Each feed object contains metadata and settings for polling and processing.

Field	Type	Description
id	String	A unique identifier for the feed.
url	String	The URL of the RSS feed.
title	String	The title of the feed.
isActive	Boolean	If true, the feed is actively monitored.
checkInterval	Number	The interval in minutes for polling the feed.
settings	Object	Contains settings for automatic email generation.
settings.autoSend	Boolean	If true, automatically sends a newsletter for new items.
settings.subjectPrefix	String	A prefix to add to the email subject line.

Sources: newsletter-program/src/tests/integration/rss-monitoring-automation.test.js:153-164

Cache Data (rss-cache.json)

The cache file stores the state of each monitored feed, including the last check time and a list of items that have already been seen. This prevents reprocessing of old content.

Field	Type	Description
lastChecked	String	ISO 8601 timestamp of the last time the feed was checked.
lastItemDate	String	ISO 8601 timestamp of the most recent item found.
items	Array	An array of objects representing cached items.
items.guid	String	The unique identifier for the RSS item.
items.title	String	The title of the RSS item.
items.pubDate	String	The publication date of the item.

Sources: newsletter-program/src/tests/integration/rss-monitoring-automation.test.js:237-248

Data Relationship

The diagram below shows the relationship between a feed configuration and its corresponding cache entry.

Caching and New Item Detection

The core of the system is its ability to detect new items in an RSS feed by comparing them against the stored cache.

Process Flow

The RSSMonitor class manages this process. The detection logic is simulated in the integration tests.

1. Fetch Feed: The monitor fetches the raw XML content from the feed's URL.
2. Parse Items: It parses the XML to extract a list of items, each with a title, link, guid, and pubDate.
3. Get Cache: The monitor retrieves the existing cache for the given feed ID.
4. Compare: It compares the guid of each fetched item against the list of guids in the cache.
5. Identify New Items: Items whose guids are not found in the cache are considered new.
6. Update Cache: The cache is updated with the new items, and the lastChecked and lastItemDate fields are refreshed.

Sources: newsletter-program/src/tests/integration/rss-monitoring-automation.test.js:173-200, 232-252

Sequence Diagram

This sequence diagram illustrates the interaction between the RSSMonitor and the cache to detect new items.

Key Functions

• updateCache(feedId, cacheData): Writes the updated cache data for a specific feed to the filesystem.
• getCache(feedId): Reads and returns the cache data for a specific feed from the filesystem.

These functions are demonstrated in the integration test, showing how cache data is stored and retrieved.

Sources: newsletter-program/src/tests/integration/rss-monitoring-automation.test.js:232-252

Content Processing and Templating

Once new RSS items are identified, they are passed to the ContentProcessor to be transformed into HTML email content.

ContentProcessor Class

This class is responsible for sanitizing content and rendering it into an HTML template.

Key methods include:

• rssToEmail(rssItem, template, options): Populates an email template with data from an RSS item.
• sanitizeContent(content): Cleans the HTML content from the RSS description to prevent XSS attacks. It allows a specific set of tags and attributes.
• renderTemplate(template, data): A simple template engine that replaces {{variable}} placeholders with data.
• renderMultipleItems(template, data): Handles templates with {{#each items}} loops for digest newsletters.

Sources: newsletter-program/src/core/content-processor.js:1-150

Content Processing Flow

The following flowchart shows how an RSS item is processed into a final HTML email.

HTML Sanitization

Content from RSS feeds is considered untrusted. The sanitizeContent method uses the sanitize-html library to ensure safety.

• Allowed Tags: p, br, strong, em, u, h1-h6, ul, ol, li, a, img.
• Allowed Attributes: href, name, target for <a> tags; src, alt, title for <img> tags.
• Disallowed: Potentially dangerous tags like <script> and attributes like onerror are removed.

Sources: newsletter-program/src/core/content-processor.js:46-64

Email Template

The rss-newsletter.html file serves as the template for emails generated from RSS items. It uses {{variable}} placeholders that are replaced by the ContentProcessor. It also supports basic looping for digest emails containing multiple items.

<!-- snippet from rss-newsletter.html -->
{{#items}}
<div class="rss-item">
    <h2><a href="{{link}}" target="_blank">{{title}}</a></h2>
    
    <div class="rss-meta">
        <span class="date">{{formatted_date}}</span>
        {{#author}}<span class="author">{{author}}</span>{{/author}}
    </div>
    
    {{#description}}
    <div class_name="rss-description">
        {{{description}}}
    </div>
    {{/description}}
    
    <a href="{{link}}" class="read-more" target="_blank">Read full article →</a>
</div>
{{/items}}

Sources: newsletter-program/src/templates/rss-newsletter.html:198-216, newsletter-program/src/core/content-processor.js:84-111.

Summary

The RSS Data Caching system is an integral part of the portfolio's newsletter automation. By maintaining a file-based cache of processed RSS items, it efficiently detects new content without relying on external databases. The system is designed for automation, where new articles from monitored feeds can be automatically sanitized, formatted into a branded HTML email using a template, and sent to subscribers. This process is validated through comprehensive integration tests that mock the entire flow, from fetching feeds to sending emails.

Sources: themes/matrix-flow/README-newsletter.md, newsletter-program/src/tests/integration/rss-monitoring-automation.test.js