Data Flow

Relevant source files

The following files were used as context for generating this wiki page:

• scripts/collect-releases.js
• scripts/build-rss.js
• scripts/build-website.js
• utils/index.js
• config.json
• scripts/collect-issues.js
• scripts/collect-discussions.js
• scripts/collect-retrospective.js
• scripts/collect-manual-posts.js
• server.js

The nodejs-news-feeder project is designed to aggregate news and updates from various sources into a unified RSS feed (feed.xml) and a static HTML website (index.html). The data flow is orchestrated through a series of Node.js scripts that collect, process, and format information. This process is typically automated via GitHub Actions, which run the scripts on a schedule or manual trigger.

The core architecture involves multiple data collection scripts, each targeting a specific source like GitHub releases, issues, discussions, or external RSS feeds. These scripts read configuration from config.json, fetch new data since the last run, format it into a standard RSS <item> structure, and inject it into the main feed.xml file. Finally, other scripts build the final RSS feed and the static website from this aggregated data.

Overall Data Flow

The entire process can be visualized as a pipeline that starts from diverse data sources and ends with the generated feed and website.

This diagram shows the high-level data flow, from data collection to final output generation.

Configuration

The data flow is heavily controlled by the config.json file. This file acts as the central point for configuring data sources, pagination limits, and tracking the state of the collection process.

Key	Description	Example Value
lastCheckTimestamp	A Unix timestamp indicating the last time the feed was successfully built. Collection scripts use this to fetch only new content.	1763032644743
reposPaginationLimit	The maximum number of repositories to fetch when collecting releases.	250
releasePaginationLimit	The maximum number of releases to fetch per repository.	10
commentsPaginationLimit	The maximum number of comments to fetch per issue.	100
breakDelimiter	An XML tag string (</image>) used as a marker to inject new <item> blocks into feed.xml.	"</image>"
discussionsInScope	An array of objects specifying which GitHub discussions to monitor for new comments.	[]
issuesInScope	An array of objects specifying which GitHub issues to monitor for new comments.	[]
lastFeedContent	A cached version of the feed.xml content from the last build, used to detect changes.	"<?xml ..."

Sources: config.json, scripts/build-rss.js:15-16, utils/index.js:134-138

Data Collection (Ingestion)

Data is collected from multiple sources by dedicated scripts. Each script filters content based on the lastCheckTimestamp from config.json to avoid duplicates.

1. GitHub Releases

The scripts/collect-releases.js script fetches new releases from repositories belonging to a specific GitHub user.

1. It reads the lastCheckTimestamp and pagination limits from config.json.
2. It fetches a list of repositories for the user thomas-iniguez-visioli.
3. For each repository, it fetches the latest releases.
4. It filters these releases, keeping only those with a published_at date greater than lastCheckTimestamp.
5. Each new release is formatted into an RSS <item> using composeFeedItem.
6. The generated items are injected into feed.xml.

Sources: scripts/collect-releases.js:3-33

2. GitHub Issues & Discussions

New content is also sourced from comments on specific GitHub issues and discussions, which are defined in config.json.

• Issues: scripts/collect-issues.js uses the gh-got library to fetch comments from repositories listed in issuesInScope. It filters comments where updated_at is newer than lastCheckTimestamp.
• Discussions: scripts/collect-discussions.js uses the @octokit/graphql library to execute a GraphQL query. It fetches the last 100 comments for each discussion in discussionsInScope and filters them by publishedAt.

In both cases, the new comments are formatted and appended to feed.xml.

Sources: scripts/collect-issues.js:6-25, scripts/collect-discussions.js:6-34

3. External Feeds and Web Scraping

The scripts/collect-retrospective.js script is responsible for fetching content from external sources.

1. External RSS Feeds: It fetches and parses multiple RSS feeds, such as haveibeenpwned.com/feed/breaches/. It uses rss-parser to process the feeds and filterFeedItems to remove duplicates.
2. Web Scraping: It scrapes the website bonjourlafuite.eu.org to extract data breach information. It makes an HTTP request, parses the resulting HTML with node-html-parser, and extracts data from specific DOM elements (div.timeline-entry).

The collected items from both sources are then formatted and added to feed.xml, with a check to prevent adding items with a duplicate GUID.

This diagram illustrates the data collection flow within collect-retrospective.js.

4. Manual Posts

The system includes a simple Express.js server to allow for manual entry of posts.

1. The server, defined in server.js, serves add-post.html.
2. This page contains a form that submits new post data (title, description, link) to the /api/posts endpoint.
3. The POST /api/posts handler reads the existing posts from manual-posts.json, appends the new post with a timestamp, and writes the file back.
4. The scripts/collect-manual-posts.js script reads manual-posts.json, formats each entry into an RSS item, and injects it into feed.xml.

This flow provides a way to add content that doesn't originate from an automated source.

Data Processing and Formatting

The utils/index.js file contains helper functions that are critical for processing and formatting data before it's added to the feed.

Function	Description
composeFeedItem	Constructs an XML <item> block from a JavaScript object containing title, description, pubDate, link, and guid.
md2html	Converts Markdown text to HTML using remark and remark-html.
buildRFC822Date	Formats a date string into the RFC 822 format required for RSS feeds (e.g., Mon, 29 Nov 2021 01:00:00 BST).
cleancontent	Sanitizes text by removing blacklisted keywords found in blacklist.txt.
escapeXmlCharacters	Escapes special XML characters like &, <, > to their corresponding entities.
filterFeedItems	Filters an array of feed items to remove entries with missing fields or duplicates based on GUID, link, or title.

Sources: utils/index.js

The composeFeedItem function is central to this stage:

// Source: utils/index.js:140-161
export function composeFeedItem({ title, description, pubDate, link, guid,source }) {
  // ... cleaning logic ...
  const cleanTitle = cleancontent(title, { /* ... */ });
  const cleanDescription = cleancontent(description, { /* ... */ });

  return `
    <item>
      <title>${cleanTitle}</title>
      <description>${cleanDescription}</description>
      <pubDate>${pubDate}</pubDate>
      <link>${cleancontent(link,{})}</link>
      <guid>${guid}</guid>
    </item>
  `
}

Feed and Website Generation

After all collection scripts have run, the feed.xml file contains a mix of old and new items. The final build steps ensure the feed is correctly formatted and the website is updated.

RSS Feed Build

The scripts/build-rss.js script finalizes the RSS feed.

1. It reads the current feed.xml and the lastFeedContent from config.json.
2. If the content has changed, it updates the <lastBuildDate> tag in feed.xml to the current time.
3. It then updates config.json with the new lastCheckTimestamp and the new lastFeedContent. This ensures that the next run will only fetch content created after this point.

Sources: scripts/build-rss.js:3-23

Website Build

The scripts/build-website.js script generates the static index.html page.

1. It reads the final feed.xml content.
2. It parses the XML using @ulisesgascon/rss-feed-parser to get a structured JSON object of the feed metadata and items.
3. It reads the EJS template from templates/index.html.ejs.
4. It renders the template with the parsed feed data, creating the final index.html file.

Sources: scripts/build-website.js:1-12

Summary

The data flow of nodejs-news-feeder is a multi-stage pipeline that automates the aggregation of information. It begins with configurable data collection from diverse sources, including APIs, external feeds, and manual inputs. The collected data is then sanitized, formatted, and incrementally injected into a central feed.xml file. Finally, build scripts finalize this RSS feed and generate a user-facing static website, with a configuration file tracking the state to ensure data is processed efficiently on subsequent runs.