Collecting Retrospectives
Relevant source files
The following files were used as context for generating this wiki page:
The "Collecting Retrospectives" feature is a core content aggregation module within the Node.js News Feeder project. Its primary purpose is to automatically fetch news items, specifically data breach notifications and other security-related posts, from multiple external sources. It achieves this by parsing RSS feeds and scraping HTML content from predefined websites.
This script, collect-retrospective.js, is responsible for transforming the collected external data into a standardized RSS item format and injecting it into the main feed.xml file. This process enriches the news feeder with timely and relevant security news from sources beyond the immediate Node.js ecosystem, such as data breach announcements. The collected items are later sorted and sanitized by the formatting script.
Configuration
The behavior of the collection script is partially guided by config.json, although many key parameters are hardcoded directly within the script itself.
Sources: config.json, scripts/collect-retrospective.js
| Key | Type | Description | Source |
|---|---|---|---|
retrospective | Object | Contains lastDay and nextDay properties. This seems to be a remnant of a previous implementation and is no longer actively used by the main logic in collect-retrospective.js. | config.json:6-9 |
breakDelimiter | String | An XML tag (</image>) used as a marker to determine where new feed items should be injected into the feed.xml file. | config.json:10 |
Data Collection Process
The script employs two distinct methods for collecting data: scraping a specific HTML website and parsing multiple external RSS feeds.
Sources: scripts/collect-retrospective.js
Scraping bonjourlafuite.eu.org
The script directly scrapes the website bonjourlafuite.eu.org to extract data breach information. This process involves a direct HTTP GET request, parsing the returned HTML, and extracting structured data from the page's DOM.
The data flow for this scraping process is as follows:
This diagram illustrates the scraping and parsing workflow for bonjourlafuite.eu.org.
Sources: scripts/collect-retrospective.js:56-118
A notable implementation detail is the use of a staticDnsAgent to resolve the hostname, which suggests a workaround for potential DNS resolution issues.
Sources: scripts/collect-retrospective.js:6-16, scripts/collect-retrospective.js:57-59
Fetching from Multiple RSS Feeds
The script also fetches and processes items from a hardcoded list of external RSS feeds. This is handled by the fetchAllFeeds and updateFeedWithAllItems functions.
The hardcoded feed URLs are:
https://haveibeenpwned.com/feed/breaches/https://thomas-iniguez-visioli.github.io/retro-weekly/feed.xml
Sources: scripts/collect-retrospective.js:183-186
The following sequence diagram shows the interaction for fetching these feeds:
This diagram outlines how the script concurrently fetches, parses, and processes multiple external RSS feeds.
Sources: scripts/collect-rospective.js:188-223
Data Processing and Integration
Once data is collected from all sources, it is processed and integrated into the main feed.xml.
Feed Item Composition
The composeFeedItem utility function is used to construct a standard XML <item> block for the RSS feed. It takes a structured object and returns an XML string.
Sources: utils/index.js:133-154
The structure of the object passed to composeFeedItem is:
| Parameter | Description |
|---|---|
title | The title of the news item. |
description | The content of the news item, wrapped in <![CDATA[...]]>. |
pubDate | The publication date, formatted to RFC822 standard. |
link | The URL to the original source article. |
guid | A unique identifier for the item, usually the same as the link. |
The function then generates an XML snippet like this:
<item>
<title>${cleanTitle}</title>
<description>${cleanDescription}</description>
<pubDate>${pubDate}</pubDate>
<link>${cleancontent(link,{})}</link>
<guid>${guid}</guid>
</item>
Sources: utils/index.js:147-153
Feed Content Update
New items are not simply appended to the end of the file. Instead, the script reads the existing feed.xml, splits its content using the breakDelimiter (</image>) from config.json, inserts the new items, and then writes the content back.
This process is visualized below:
This diagram shows the logic for injecting new items into the existing RSS feed file.
Sources: scripts/collect-retrospective.js:42-45, utils/index.js:160-162
Post-Collection Processing
After the collect-retrospective.js script runs, the npm run rss:format command is intended to be executed. This command runs scripts/format.js, which reads the entire feed.xml, sorts all items by date, filters for duplicates, and reformats the XML for consistency.
Key steps in scripts/format.js:
- Parse XML: Reads
feed.xmland parses it usingrss-parser. - Filter Items: Uses the
filterFeedItemsutility to remove duplicates and incomplete entries. - Sort Items: Sorts all items in descending order based on their
isoDate. - Recompose Feed: Rebuilds the XML content with the sorted items.
- Format XML: Uses
xml-formatterto prettify the final XML output. - Overwrite File: Saves the formatted, sorted, and cleaned content back to
feed.xml.
Sources: scripts/format.js, package.json:11
Execution
The script is executed via an npm script defined in package.json.
"scripts": {
"collect:retrospective": "node scripts/collect-retrospective.js",
...
}
Sources: package.json:8
Running npm run collect:retrospective will trigger the entire collection process.
Summary
The "Collecting Retrospectives" module is a powerful, multi-source data aggregator. It combines web scraping of bonjourlafuite.eu.org and parsing of RSS feeds like haveibeenpwned.com to gather security-related news. The collected data is standardized into RSS items and injected into the project's main feed.xml. This automated collection, followed by a separate formatting and sorting step, ensures the news feeder remains populated with fresh and relevant content from diverse external sources.