Project Goals and Vision
Relevant source files
The following files were used as context for generating this wiki page: - [README.md](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/README.md) - [package.json](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/package.json) - [scripts/collect-retrospective.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/scripts/collect-retrospective.js) - [scripts/collect-releases.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/scripts/collect-releases.js) - [scripts/collect-issues.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/scripts/collect-issues.js) - [scripts/collect-discussions.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/scripts/collect-discussions.js) - [scripts/collect-manual-posts.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/scripts/collect-manual-posts.js) - [server.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/server.js) - [add-post.html](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/add-post.html) - [config.json](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/config.json) - [index.html](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/index.html) - [scripts/build-website.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/scripts/build-website.js) - [scripts/build-rss.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/scripts/build-rss.js) - [scripts/format.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/HEAD/scripts/format.js)The nodejs-news-feeder project is an automated system designed to aggregate information from various sources, process it, and publish it as a unified RSS feed and a static website. While the README.md states the original intention was to track news within the Node.js organization, the current implementation is focused on aggregating cybersecurity news, specifically data breaches. The system is built to be extensible, combining automated data collection from APIs and external feeds with a manual entry and curation process.
The primary goal is to provide a centralized, timely, and accessible source of information. It solves the problem of monitoring multiple disparate sources by collecting content, creating a curated feed, and distributing it through various channels like RSS, a web page, and potentially Slack.
Sources: README.md:5-10, scripts/collect-retrospective.js:125-129, index.html:6-15
High-Level Architecture
The project operates on a multi-stage pipeline, primarily orchestrated through Node.js scripts and GitHub Actions. The process begins with data collection from various sources, moves to a processing and curation stage, and ends with the publication of the updated content.
The following diagram illustrates the overall workflow from content creation to consumption.
This flow ensures that all new content is collected automatically, but still passes through a manual review step (via Pull Request) before being published, allowing for quality control and content curation.
Sources: README.md:25-34, package.json:7-13
Data Aggregation
The system is designed to pull data from multiple distinct sources. Each source type is handled by a dedicated script.
Data Sources
The project collects information from automated API calls, external RSS feeds, and manual user input. The current implementation is heavily focused on cybersecurity news.
| Data Source | Collection Script | Description |
|---|---|---|
| GitHub Releases | scripts/collect-releases.js | Fetches new releases from configured GitHub repositories. |
| GitHub Issues | scripts/collect-issues.js | Fetches new comments from specific, configured GitHub issues. |
| GitHub Discussions | scripts/collect-discussions.js | Fetches new comments from specific, configured GitHub discussions. |
| External Feeds | scripts/collect-retrospective.js | Fetches and parses items from external RSS/JSON feeds, such as haveibeenpwned.com and bonjourlafuite.eu.org. |
| Manual Posts | scripts/collect-manual-posts.js | Reads manually created posts from the local manual-posts.json file. |
Sources: scripts/collect-releases.js, scripts/collect-issues.js, scripts/collect-discussions.js, scripts/collect-retrospective.js, scripts/collect-manual-posts.js
Collection Process
The following diagram shows how the various collection scripts contribute to the final feed.
Each collection script fetches data, formats it into a standard item using composeFeedItem, and injects it into the feed.xml file. Subsequent scripts then format, sort, and build the final outputs.
Sources: package.json:7-17, utils/index.js:115-135, scripts/format.js
Manual Content Curation
To supplement automated collection, the project includes a local web server for manually adding news items. This is useful for including sources that do not have an API or RSS feed.
- Backend: An Express server defined in
server.jsruns onlocalhost:3000. - Frontend: A simple form in
add-post.htmlallows users to submit a title, description, link, and publication date. - Storage: Submitted posts are appended to a
manual-posts.jsonfile. - Integration: The
npm run collect:manualscript readsmanual-posts.jsonand adds its contents to the mainfeed.xml.
Sources: server.js, add-post.html, scripts/collect-manual-posts.js
Manual Post Workflow
The sequence diagram below details the process of adding a new post manually.
Sources: server.js:28-48, add-post.html:43-70
Consumption Channels
The aggregated news is made available to end-users through several channels, ensuring wide accessibility.
- RSS Feed: The primary output is an RSS 2.0 feed available at
feed.xml. This is the canonical source for all other outputs. - Web Page: A static HTML page is generated from the RSS feed, providing a user-friendly, readable view of the latest news in a browser. This is available at
index.html. - Terminal: Users can consume the feed directly in their terminal using tools like
curlandgrep. - Slack: The
README.mdmentions a Slack channel#nodejs-news-feederfor receiving updates, implying a separate integration pushes new items from the RSS feed to Slack.
Sources: README.md:13-23, scripts/build-website.js
Core Scripts and Commands
The project's functionality is exposed through a series of npm scripts defined in package.json.
| Script | Description |
|---|---|
collect:* | A set of scripts (collect:releases, collect:issues, etc.) that run the corresponding data collection logic. |
start | Starts the local Express server (server.js) for adding manual posts. |
rss:validate | Validates the feed.xml against the W3C Feed Validation Service. |
rss:build | Updates the <lastBuildDate> timestamp in feed.xml and saves the current feed state to config.json. |
rss:format | Sorts all items in feed.xml by date, filters duplicates, and applies consistent XML formatting. |
website:build | Generates the index.html static site from the content of feed.xml using an EJS template. |
test | Runs the project's test suite using Node.js's native test runner. |
lint | Lints the codebase for style consistency using standard. |
Sources: package.json:6-22, scripts/validate.js, scripts/build-rss.js, scripts/format.js, scripts/build-website.js
Configuration
Key operational parameters are managed in config.json. This centralized configuration allows for easy modification of the aggregator's behavior without changing the code.
| Parameter | Description |
|---|---|
lastCheckTimestamp | A Unix timestamp indicating the last time the feed was successfully updated. Scripts use this to fetch only new content. |
breakDelimiter | An XML tag (</image>) used as a stable marker to inject new items into feed.xml. |
*PaginationLimit | Limits the number of items fetched from APIs (e.g., reposPaginationLimit, releasePaginationLimit). |
discussionsInScope | An array of objects defining which GitHub Discussions to monitor. |
issuesInScope | An array of objects defining which GitHub Issues to monitor. |
lastFeedContent | A complete copy of the feed.xml from the last successful build, used to detect if changes have occurred. |
Sources: config.json:2-10, scripts/build-rss.js:16-21