Building the RSS Feed
Relevant source files
The following files were used as context for generating this wiki page:
The RSS feed is a core feature of this project, designed to aggregate news, releases, data breaches, and other relevant content into a single, consumable XML feed. The process is managed by a series of Node.js scripts orchestrated via npm commands. It involves collecting data from multiple sources, composing them into RSS items, inserting them into the main feed.xml file, and then finalizing, formatting, and validating the feed.
The overall architecture is based on a GitHub Action that can be triggered manually or by a cron job. This action executes a series of scripts to collect, process, format, and validate the new information, ultimately generating a pull request for review and merging.
Sources: README.md:16-18, README.md:41-47, package.json:6-12
Architecture and Process Flow
The RSS feed generation is a multi-stage process executed by different scripts. The main stages are Data Collection, Feed Building, and Feed Formatting. These stages are typically run in sequence to produce the final feed.xml.
The following diagram illustrates the high-level process flow.
This diagram shows the sequence of npm scripts used to build the RSS feed.
Sources: package.json:6-12, README.md:41-47
NPM Scripts
The process is managed through a set of scripts defined in package.json.
| Script | Command | Description |
|---|---|---|
rss:build | node scripts/build-rss.js | Updates the <lastBuildDate> in feed.xml and the lastCheckTimestamp in config.json. |
rss:format | node scripts/format.js>log.txt | Parses, filters, sorts, and formats the items in feed.xml. |
rss:validate | node scripts/validate.js | Validates the feed.xml against the W3C Feed Validation Service. |
collect:* | node scripts/collect-*.js | A series of scripts to collect data from various sources. |
Sources: package.json:6-12
Data Collection
The first step in the process is to collect new content from various sources. Each source is handled by a dedicated script in the scripts/ directory. These scripts fetch new data, format it into an RSS <item> structure, and inject it into the feed.xml file.
The insertion point for new content is marked by a delimiter </image> in the feed.xml file. All collection scripts read the feed, split it by this delimiter, and insert the new items.
Sources: scripts/collect-discussions.js:38-40, config.json:10
The diagram below shows the different data sources that contribute to the feed.
This diagram illustrates how various collection scripts gather data from different sources and populate feed.xml.
Sources: scripts/collect-releases.js, scripts/collect-issues.js, scripts/collect-discussions.js, scripts/collect-retrospective.js, scripts/collect-manual-posts.js
Collection Script Details
collect-releases.js: Fetches new repository releases from thethomas-iniguez-visioliuser account on GitHub. It filters releases newer thanlastCheckTimestamp.
Sources: scripts/collect-releases.js:6-12, scripts/collect-releases.js:24-25collect-issues.js: Collects new comments from specific GitHub issues defined inconfig.json. It usesgh-gotto paginate through comments.
Sources: scripts/collect-issues.js:6-15collect-discussions.js: Collects new comments from specific GitHub discussions using the GitHub GraphQL API.
Sources: scripts/collect-discussions.js:6-29collect-retrospective.js: A complex script that fetches data from multiple external RSS feeds (likehaveibeenpwned.com) and also scrapes the websitebonjourlafuite.eu.orgfor data breach information.
Sources: scripts/collect-retrospective.js:19-40, scripts/collect-retrospective.js:145-148collect-manual-posts.js: Reads structured post data from a localmanual-posts.jsonfile, allowing for the manual addition of feed items.
Sources: scripts/collect-manual-posts.js:5-18
All these scripts use the composeFeedItem utility to generate the XML for each new item.
Sources: utils/index.js:156-175
Feed Finalization (rss:build)
After data collection, the scripts/build-rss.js script is run. Its main purpose is to update the feed's metadata.
The script performs the following steps:
- Reads the current content of
feed.xml. - Reads the
config.jsonfile. - Checks if the feed has changed by comparing the length of the current XML content with the
lastFeedContentstored in the configuration. - If there are changes, it updates the
<lastBuildDate>tag infeed.xmlto the current time. - It then updates
config.jsonwith the new timestamp (lastCheckTimestamp) and the new feed content (lastFeedContent).
This logic ensures that the lastBuildDate is only updated when new content has actually been added to the feed.
Sources: scripts/build-rss.js:3-25
The following flowchart details the logic within build-rss.js.
This diagram shows the conditional logic for updating the feed's build date.
Sources: scripts/build-rss.js:11-25
Feed Formatting and Sorting (rss:format)
The scripts/format.js script is responsible for cleaning, sorting, and formatting the entire feed.xml file. This ensures the feed is well-structured, chronologically sorted, and free of duplicate entries.
The process is as follows:
- The script reads the raw
feed.xml. - It uses the
rss-parserlibrary to parse the XML string into a JavaScript object. - The parsed items are passed to the
filterFeedItemsutility, which removes items that are incomplete or are duplicates based on theirguid,link, ortitle. - The filtered items are then sorted in descending chronological order based on their
isoDate. - Each sorted item is passed back to
composeFeedItemto be re-rendered as a clean XML string. - The newly generated list of items is injected back into the XML template.
- Finally, the
xml-formatterlibrary is used to pretty-print the entire XML content before overwritingfeed.xml.
Sources: scripts/format.js:1-41
Sequence of Formatting
This diagram shows the interactions between format.js and other modules to process the feed.
Sources: scripts/format.js:28-41, utils/index.js:231-243
Core Utilities and Configuration
The entire build process relies on a set of shared utilities and a central configuration file.
Utilities (utils/index.js)
The utils/index.js file provides helper functions used across all scripts.
| Function | Description |
|---|---|
composeFeedItem | Creates an XML <item> block from a given object containing title, description, etc. |
buildRFC822Date | Formats a date string into an RFC 822 compliant date for the <pubDate> tag. |
getFeedContent | Reads the content of feed.xml. |
overwriteFeedContent | Writes content to feed.xml. |
getConfig | Reads and parses config.json. |
overwriteConfig | Writes a configuration object to config.json. |
filterFeedItems | Filters an array of feed items to remove duplicates (by guid, link, or title) and incomplete entries. |
cleancontent | Cleans a string by removing blacklisted keywords. |
md2html | Converts a Markdown string to HTML using remark. |
Sources: utils/index.js
Configuration (config.json)
The config.json file stores state and configuration parameters for the build process.
| Key | Description |
|---|---|
lastCheckTimestamp | A Unix timestamp (in milliseconds) of the last successful feed build. Used by collection scripts to fetch only new content. |
breakDelimiter | The string (</image>) used as a marker to inject new items into feed.xml. |
lastFeedContent | A string containing the full content of feed.xml from the last build. Used to detect changes. |
discussionsInScope | An array of objects defining which GitHub discussions to monitor. |
issuesInScope | An array of objects defining which GitHub issues to monitor. |
retrospective | An object containing state for the retrospective collection script. |
Sources: config.json
Summary
The RSS feed building process is a robust, automated system that aggregates content from diverse sources. It relies on a series of specialized scripts for collecting, composing, finalizing, and formatting feed items. Configuration is managed centrally in config.json, which also maintains the state between runs, ensuring that only new content is added. The final output is a well-formed, sorted, and validated feed.xml file, ready for consumption by news readers and other services.
Sources: README.md:16-27, package.json:6-12