Running the Scripts

Relevant source files

The following files were used as context for generating this wiki page:

• package.json
• README.md
• scripts/collect-releases.js
• scripts/collect-retrospective.js
• scripts/build-rss.js
• scripts/format.js
• scripts/build-website.js
• scripts/validate.js
• utils/index.js
• server.js
• config.json

This document provides a technical overview of the scripts used in the nodejs-news-feeder project. These scripts automate the process of collecting news from various sources, generating an RSS feed, and building a static website to display the news. The entire workflow is designed to be executed via NPM commands, which are defined in the package.json file.

The core of the system is a set of data collection scripts that fetch information from GitHub (releases, issue comments, discussion comments), external RSS feeds, and manually added posts. This data is then processed, formatted, and used to build the final feed.xml and index.html files. The process is typically triggered by a GitHub Action, either on a schedule or manually.

Sources: README.md, package.json

Available Scripts

The project contains several scripts for development, data collection, and building the feed/website. They are all run via npm run <script_name>.

Script Name	Command	Description
collect:releases	node scripts/collect-releases.js	Collects new repository releases from GitHub.
collect:issues	node scripts/collect-issues.js	Collects new comments from specific GitHub issues.
collect:discussions	node scripts/collect-discussions.js	Collects new comments from specific GitHub discussions.
collect:retrospective	node scripts/collect-retrospective.js	Collects items from external RSS feeds and scrapes websites.
collect:manual	node scripts/collect-manual-posts.js	Collects posts from the local manual-posts.json file.
start	node server.js	Starts a local Express server for adding manual posts.
rss:validate	node scripts/validate.js	Validates the feed.xml file against the W3C Feed Validation Service.
rss:build	node scripts/build-rss.js	Updates the lastBuildDate in feed.xml and the lastCheckTimestamp in config.json.
rss:format	node scripts/format.js>log.txt	Sorts and formats the items in feed.xml.
rss:format-check	node scripts/format-check.js	Checks if rss:format would produce any changes.
website:build	node scripts/build-website.js	Generates the index.html file from the RSS feed content.
lint / lint:fix	standard / standard --fix	Runs the linter to check/fix code style.
test / test:coverage / test:watch	node --test	Runs the test suite using Node.js's built-in test runner.

Sources: package.json:5-22

Core Workflow

The overall process involves collecting data from multiple sources, aggregating it into the feed.xml file, and then building the static website from that feed.

This diagram illustrates the high-level data flow from collection to final output.

Data Collection

Data is collected from several sources using a series of collect:* scripts. Each script appends new items to the feed.xml file, inserting them after the </image> delimiter.

Sources: config.json:10, scripts/collect-releases.js:37-39

GitHub Content (Releases, Issues, Discussions)

These scripts fetch updates from the nodejs organization on GitHub. They use the lastCheckTimestamp from config.json to only retrieve content created since the last run.

• collect:releases: Uses gh-got to paginate through repositories for a specified user (thomas-iniguez-visioli) and fetches their releases. It filters for releases published after lastCheckTimestamp.
• collect:issues: Uses gh-got to fetch comments from a predefined list of issues in config.json under issuesInScope.
• collect:discussions: Uses @octokit/graphql to fetch comments from discussions specified in config.json under discussionsInScope.

Sources: scripts/collect-releases.js:6-28, scripts/collect-issues.js:4-17, scripts/collect-discussions.js:4-29, config.json:2,11-12

The following sequence diagram shows the process for collecting releases.

External Feeds and Web Scraping

The collect:retrospective script is responsible for fetching content from external sources. It demonstrates two methods:

1. RSS Parsing: It fetches and parses multiple RSS feeds, such as haveibeenpwned.com/feed/breaches/. It uses the rss-parser library.
2. Web Scraping: It scrapes the website bonjourlafuite.eu.org by making an HTTP GET request and then parsing the HTML response with node-html-parser to extract timeline entries.

For each found item, it checks if the GUID already exists in feed.xml to avoid duplicates before adding it.

Sources: scripts/collect-retrospective.js:10-33, 50-93, 131-155

Manual Posts

The system allows for adding posts manually via a simple web interface.

1. Run npm start to launch a local Express server defined in server.js.
2. Navigate to http://localhost:3000 which serves add-post.html.
3. The form on this page submits a new post to the POST /api/posts endpoint.
4. The server receives the post, adds a timestamp and ID, and appends it to manual-posts.json.
5. Running npm run collect:manual reads manual-posts.json and adds its contents to the main feed.xml.

This flow is depicted below:

Feed and Website Generation

After all data is collected into feed.xml, other scripts are used to finalize and publish the content.

Building and Formatting the RSS Feed

• rss:format: This is a crucial step. The script reads the entire feed.xml, parses it, and then performs two key actions:
  1. Filtering: It uses the filterFeedItems utility function to remove duplicate items based on their GUID, link, or title.
  2. Sorting: It sorts all items in descending order by their publication date (isoDate).
     The script then reconstructs the XML content and overwrites feed.xml with a clean, sorted, and formatted version using xml-formatter.
• rss:build: This script updates the <lastBuildDate> tag in feed.xml to the current time. It also updates the lastCheckTimestamp in config.json, which is essential for ensuring the collection scripts only fetch new content on the next run.

Sources: scripts/format.js, scripts/build-rss.js, utils/index.js:203-215

Validating the RSS Feed

The rss:validate script sends the content of feed.xml to the https://validator.w3.org/feed/check.cgi service. It then parses the HTML response to check if the feed is valid and logs the result. This acts as a quality gate to ensure the generated feed is compliant.

Sources: scripts/validate.js

Building the Website

The website:build script generates the static index.html page.

1. It reads the final feed.xml.
2. It parses the feed using @ulisesgascon/rss-feed-parser.
3. It reads the EJS template from templates/index.html.ejs.
4. It renders the template, passing the parsed feed's metadata and items to generate the final HTML.
5. The generated HTML is written to index.html in the root directory.

Sources: scripts/build-website.js

Configuration

Script behavior is controlled by config.json.

Key	Type	Description
lastCheckTimestamp	Number	A Unix timestamp (milliseconds) of the last successful run. Used to fetch only new items.
reposPaginationLimit	Number	The maximum number of repositories to fetch when collecting releases.
releasePaginationLimit	Number	The maximum number of releases to fetch per repository.
commentsPaginationLimit	Number	The maximum number of comments to fetch per issue/discussion.
breakDelimiter	String	The XML tag (</image>) used as a marker to inject new feed items.
discussionsInScope	Array	An array of objects specifying which GitHub discussions to monitor.
issuesInScope	Array	An array of objects specifying which GitHub issues to monitor.

Sources: config.json:2-12

Conclusion

The scripting system for nodejs-news-feeder provides a robust and extensible framework for news aggregation. By defining clear data sources in config.json and using a series of distinct npm scripts for collection, formatting, and building, the project achieves a high degree of automation. The workflow is idempotent, preventing duplicate content, and includes validation steps to ensure the quality of the final RSS feed and website.