GitHub Actions Workflow
Relevant source files
The following files were used as context for generating this wiki page:This project utilizes a series of scripts, orchestrated by GitHub Actions, to automatically aggregate news and updates from various sources into a unified RSS feed and a static HTML website. The workflow is designed to be triggered either on a schedule (via a cron job) or manually, providing flexibility for content curators.
The core process involves collecting data from GitHub (releases, issue comments, discussion comments), external RSS feeds, and scraped web pages. This data is then processed, formatted, and inserted into a master feed.xml file. Finally, this XML file is used to build a user-facing website. The entire process culminates in a Pull Request, allowing for human review and curation before the new content is published.
Sources: README.md:25-35
Architecture Overview
The workflow follows a multi-stage pipeline that begins with data collection and ends with the publication of new content.
This diagram illustrates the high-level data flow from content creation to consumption.
Sources: README.md:25-35
Workflow Components & Scripts
The automation is powered by a collection of Node.js scripts defined in package.json. These scripts handle distinct stages of the workflow, from data collection to final deployment.
Sources: package.json:7-20
Key Scripts
This table provides a summary of the main scripts involved in the workflow.
| Command | Script Path | Description |
|---|---|---|
npm run collect:releases | scripts/collect-releases.js | Collects new repository releases from GitHub. |
npm run collect:issues | scripts/collect-issues.js | Collects new comments from specific GitHub issues. |
npm run collect:discussions | scripts/collect-discussions.js | Collects new comments from specific GitHub discussions. |
npm run collect:retrospective | scripts/collect-retrospective.js | Collects items from external RSS feeds and scrapes web pages. |
npm run collect:manual | scripts/collect-manual-posts.js | Collects posts from the local manual-posts.json file. |
npm run rss:build | scripts/build-rss.js | Updates the <lastBuildDate> in feed.xml and the lastCheckTimestamp in config.json. |
npm run rss:format | scripts/format.js | Sorts and formats the feed.xml file. |
npm run rss:validate | scripts/validate.js | Validates the feed.xml against the W3C feed validator. |
npm run website:build | scripts/build-website.js | Generates the index.html website from the feed.xml content. |
Sources: package.json:7-20
Data Collection
Data collection is the first automated step, executed by several specialized scripts. Each script targets a different data source and filters for new content using the lastCheckTimestamp value stored in config.json.
The following diagram shows the general process for how each collection script works.
GitHub Data Sources
- Releases: The
collect-releases.jsscript usesgh-gotto paginate through repositories and their releases, filtering for any published afterlastCheckTimestamp. - Issue Comments: The
collect-issues.jsscript fetches comments from a predefined list of issues inconfig.json(issuesInScope) and filters for new comments. - Discussion Comments: The
collect-discussions.jsscript uses@octokit/graphqlto query for new comments in specific discussions defined inconfig.json(discussionsInScope).
Sources: scripts/collect-releases.js:19-22, scripts/collect-issues.js:14-17, scripts/collect-discussions.js:10-28, config.json
External Data Sources
The collect-retrospective.js script is responsible for aggregating content from multiple external sources:
- RSS Feeds: It fetches and parses a list of RSS feeds, including
haveibeenpwned.com/feed/breaches/. - Web Scraping: It performs an HTTP GET request to
bonjourlafuite.eu.organd usesnode-html-parserto scrape timeline entries from the HTML, converting them into feed items.
This script uses a custom HTTPS agent to handle DNS resolution for the scraped site. It checks for duplicates by GUID before adding new items to the feed.
Sources: scripts/collect-retrospective.js:10-33, 67-101, 137-140
Manual Posts
The collect-manual-posts.js script provides a way to inject content directly into the feed by reading from a manual-posts.json file. This is useful for adding one-off announcements or articles that don't originate from an automated source. An associated add-post.html and server.js provide a simple web interface to add entries to this JSON file.
Sources: scripts/collect-manual-posts.js, server.js, add-post.html
Feed Generation and Processing
Once data is collected, it must be processed and built into the final artifacts (feed.xml and index.html).
Item Composition
All collection scripts use the composeFeedItem utility function to structure the data into a standard XML <item> block. This function also uses a cleancontent helper to sanitize the title, description, and link.
Sources: utils/index.js:140-153, utils/index.js:104-123
Feed Building and Formatting
rss:build: Thebuild-rss.jsscript updates the<lastBuildDate>infeed.xmlto the current time. It also updates thelastCheckTimestampinconfig.jsonto ensure subsequent runs only fetch newer content. This step only runs if the feed content has changed.rss:format: Theformat.jsscript reads the entirefeed.xml, parses it, sorts all items by publication date in descending order, and then overwrites the file with a cleanly formatted and sorted version. This ensures the feed is always chronologically ordered.
Sources: scripts/build-rss.js, scripts/format.js
Website Generation
The build-website.js script is the final step in the build process. It parses the generated feed.xml and uses an EJS template (templates/index.html.ejs) to render the final index.html page, displaying the news items for browser consumption.
Sources: scripts/build-website.js
Configuration
The behavior of the workflow is controlled by config.json. This file stores state and configuration parameters.
| Key | Type | Description |
|---|---|---|
lastCheckTimestamp | number | A Unix timestamp of the last successful run. Used to filter for new content. |
breakDelimiter | string | An XML tag (</image>) used as a placeholder to inject new items into feed.xml. |
discussionsInScope | Array<Object> | A list of GitHub discussions to monitor for new comments. |
issuesInScope | Array<Object> | A list of GitHub issues to monitor for new comments. |
lastFeedContent | string | A copy of the feed.xml content from the last build, used to detect changes. |
Sources: config.json:2-12, scripts/build-rss.js:15
Summary
The GitHub Actions workflow for nodejs-news-feeder is a robust, multi-source aggregation system. It automates the collection of news from GitHub, external feeds, and scraped websites. Through a series of modular scripts, it processes, formats, and validates this content, ultimately generating an RSS feed and a static website. The inclusion of a manual PR review step ensures content quality before publication, blending automation with human oversight.