Home
Relevant source files
The following files were used as context for generating this wiki page:The Node.js News Feeder is an automated system designed to aggregate news, releases, discussions, and other relevant content from the Node.js ecosystem. Its primary goal is to provide a centralized and timely source of information for news sites, blogs, and the community. The project generates a primary RSS feed (feed.xml) and a corresponding static website (index.html) for easy consumption.
The workflow is driven by a series of scripts, orchestrated by GitHub Actions. These scripts collect data from various sources, including GitHub APIs and external RSS feeds. The collected data is processed, formatted, and compiled into a single feed. A key feature of the workflow is a manual curation step, where all new content is submitted as a pull request, allowing for review and approval before being published.
Sources: README.md:5-12, README.md:28-32
Architecture Overview
The system operates in a four-stage process, starting from content creation by the community and ending with consumption by end-users. A GitHub Action, triggered by a cron job or manually, automates the collection and processing steps. The final content is reviewed by curators before being merged and published.
Sources: README.md:26-32
The following diagram illustrates the high-level architecture of the news feeder system.
Sources: README.md:26-32, package.json:6-11
Core Components
The project is composed of several key parts: data collection scripts, a local web server for manual entry, utility functions for data processing, and build scripts for generating the final artifacts.
Data Collection Scripts
Data is aggregated from multiple sources using a collection of Node.js scripts. These scripts are defined in package.json and are the primary mechanism for fetching new content. The lastCheckTimestamp in config.json is used by most scripts to ensure only new items are fetched.
Sources: package.json:6-11, config.json:2, scripts/collect-issues.js:15
| Script Command | File | Description |
|---|---|---|
npm run collect:releases | scripts/collect-releases.js | Collects new releases from configured GitHub repositories. |
npm run collect:issues | scripts/collect-issues.js | Collects new comments from specific GitHub issues defined in config.json. |
npm run collect:discussions | scripts/collect-discussions.js | Collects new comments from specific GitHub discussions using the GraphQL API. |
npm run collect:retrospective | scripts/collect-retrospective.js | A complex script that fetches items from multiple external RSS feeds (e.g., haveibeenpwned.com) and scrapes content from bonjourlafuite.eu.org. |
npm run collect:manual | scripts/collect-manual-posts.js | Reads manually created posts from manual-posts.json and adds them to the feed. |
Sources: package.json:6-11, scripts/collect-releases.js, scripts/collect-issues.js, scripts/collect-discussions.js, scripts/collect-retrospective.js, scripts/collect-manual-posts.js
Data Processing and Formatting
Once data is collected, it is processed and formatted into a standardized structure before being added to the RSS feed.
Utility Functions
The utils/index.js file provides helper functions for cleaning, formatting, and manipulating content.
| Function | Description |
|---|---|
md2html(md) | Converts a Markdown string to HTML. |
escapeXmlCharacters(text) | Escapes special XML characters (&, <, >, ", '). |
stripHtmlTags(text) | Removes all HTML tags from a string. |
cleancontent(text, options) | A master function that cleans content, including removing blacklisted keywords from blacklist.txt. |
buildRFC822Date(dateString) | Formats a date string into the RFC 822 format required for RSS feeds. |
composeFeedItem({...}) | Constructs an XML <item> block from title, description, pubDate, link, and guid. |
filterFeedItems(items, opts) | Filters an array of feed items to remove duplicates (by guid, link, or title) and incomplete entries. |
Sources: utils/index.js
Feed Generation
The collected items are injected into the feed.xml file. The scripts use a delimiter, </image>, defined in config.json to find the correct insertion point.
// Example from scripts/collect-discussions.js
const feedContent = getFeedContent()
const [before, after] = feedContent.split(breakDelimiter)
const updatedFeedContent = `${before}${breakDelimiter}${relevantComments}${after}`
overwriteFeedContent(updatedFeedContent)
Sources: scripts/collect-discussions.js:33-36, config.json:10
After all collection scripts have run, npm run rss:format is executed. This script reads the entire feed.xml, sorts all items by date, filters out duplicates, and formats the XML for readability.
Sources: scripts/format.js, package.json:14
Manual Post Entry
For news that isn't captured automatically, a local Express server allows for manual post creation.
- Server:
server.jsruns a simple Express server on port 3000. - Frontend:
add-post.htmlprovides a form to submit a new post's title, description, link, and publication date. - Storage: Submitted posts are sent via a
POSTrequest to/api/postsand stored inmanual-posts.json.
Sources: server.js, add-post.html
The following diagram shows the sequence for adding a manual post.
Sources: server.js:12-45, add-post.html:31-64
Website and RSS Build
The final artifacts (index.html and feed.xml) are generated by dedicated build scripts.
scripts/build-rss.js: This script updates the<lastBuildDate>infeed.xmlto the current time and updates thelastCheckTimestampinconfig.jsonto prevent reprocessing of the same items.scripts/build-website.js: This script reads the finalfeed.xml, parses it using@ulisesgascon/rss-feed-parser, and renders theindex.htmlpage using an EJS template (templates/index.html.ejs).
Sources: scripts/build-rss.js, scripts/build-website.js
This diagram shows how the website is built from the RSS feed.
Sources: scripts/build-website.js:5-10
Configuration
The system's behavior is controlled by config.json. This file stores timestamps, API limits, and the scope of content to be collected.
| Key | Type | Description |
|---|---|---|
lastCheckTimestamp | Number | A Unix timestamp indicating the last time the feed was successfully updated. Used to fetch only new content. |
reposPaginationLimit | Number | The maximum number of repositories to paginate through when fetching releases. |
releasePaginationLimit | Number | The maximum number of releases to fetch per repository. |
commentsPaginationLimit | Number | The maximum number of comments to fetch per issue/discussion. |
breakDelimiter | String | The XML tag (</image>) used as a marker for injecting new <item> blocks into feed.xml. |
discussionsInScope | Array | An array of objects specifying which GitHub discussions to monitor. |
issuesInScope | Array | An array of objects specifying which GitHub issues to monitor. |
lastFeedContent | String | A copy of the last generated feed.xml content, used to detect if changes have occurred. |
Sources: config.json
Consuming the News
The generated news feed is available through several channels:
- RSS Feed: The primary output is an RSS 2.0 feed available at
https://nodejs.github.io/nodejs-news-feeder/feed.xml. - Web Browser: A user-friendly HTML version can be viewed at
https://nodejs.github.io/nodejs-news-feeder/. - Terminal: Users can read the latest news directly in their terminal with a
curlcommand. - Slack: A Slack channel (
#nodejs-news-feeder) is available for subscribing to updates.
Sources: README.md:15-25