Collecting GitHub Discussions

Relevant source files

The following files were used as context for generating this wiki page:

• scripts/collect-discussions.js
• config.json
• utils/index.js
• package.json
• README.md

The collect-discussions.js script is a core component of the Node.js News Feeder's content aggregation system. Its primary purpose is to automatically fetch recent comments from specified GitHub Discussions, transform them into a standardized format, and inject them as new items into the project's RSS feed (feed.xml). This process ensures that updates posted in designated discussion threads are captured and distributed as news.

The entire operation is driven by a GitHub Action that can be triggered manually or by a CRON job. This script, along with others like collect-issues.js and collect-releases.js, populates the feed with content from various sources, which is then curated before being published.

Sources: README.md, scripts/collect-discussions.js, package.json:4,8

Architecture and Data Flow

The script follows a linear, multi-step process to collect and integrate discussion comments. It begins by reading its configuration, queries the GitHub API for new data, processes the results, formats them into XML, and finally updates the main feed file.

This diagram illustrates the high-level data flow for the collect-discussions.js script.

Configuration

The script's behavior is entirely controlled by values in config.json. The getConfig utility function is used to load these settings at runtime.

Sources: scripts/collect-discussions.js:4, utils/index.js:120-122

Key	Type	Description
discussionsInScope	Array<Object>	An array of objects, each specifying a GitHub Discussion to monitor. Each object contains a discussionId (the discussion number) and a team name for attribution.
lastCheckTimestamp	Number	A Unix timestamp (in milliseconds) representing the last time the script was run. Comments published after this timestamp are considered new.
breakDelimiter	String	A string (</image>) used as a marker in feed.xml to determine where new <item> elements should be injected.

Sources: config.json:2,10,11, scripts/collect-discussions.js:6

An example of the discussionsInScope configuration. Note that in the provided config.json file, this array is empty, meaning the script will not fetch any discussions until it is populated.

{
  "discussionsInScope": [],
}

Sources: config.json:10

Data Fetching

To retrieve comments, the script uses the @octokit/graphql library to send a query to the GitHub GraphQL API. It authenticates using a GITHUB_TOKEN from the environment variables.

Sources: scripts/collect-discussions.js:1,26-28, package.json:37

GraphQL Query

For each discussion ID defined in discussionsInScope, the script executes a GraphQL query to fetch the last 100 comments. The query retrieves the comment's body, publication and update timestamps, and its unique database ID.

{
  repository(name: "node", owner: "nodejs") {
    discussion(number: ${discussionId}) {
      comments(last: 100) {
        edges {
          node {
            body
            publishedAt
            updatedAt
            databaseId
          }
        }
      }
    }
  }
}

Sources: scripts/collect-discussions.js:9-24

The following sequence diagram shows the interaction between the script and the GitHub API.

Data Processing and Formatting

Once the comments are fetched, they undergo several processing steps before being added to the feed.

Filtering and Augmentation

1. Filtering: The script iterates through the returned comments and keeps only those whose publishedAt date is more recent than the lastCheckTimestamp from the configuration. This prevents duplicate entries in the feed.
2. Augmentation: For each relevant comment, the script attaches the team and discussionId from the corresponding discussionsInScope object. This contextual information is used to build the feed item's title and link.

Sources: scripts/collect-discussions.js:31-33

Feed Item Generation

Each filtered comment is transformed into an RSS <item> string using the composeFeedItem utility function.

The following diagram shows how a comment object is mapped to a feed item.

The key transformations are:

• Title: Created by combining the team name with a formatted date from buildTitleDate(comment.publishedAt).
• Description: The Markdown body of the comment is converted to HTML using md2html and wrapped in a CDATA block to prevent XML parsing issues.
• Publication Date: The publishedAt timestamp is converted to the RFC 822 format required by RSS feeds using buildRFC822Date.
• Link & GUID: A permanent link to the specific comment is constructed using the discussionId and the comment's databaseId. This URL is used for both the <link> and <guid> tags to ensure uniqueness.

Sources: scripts/collect-discussions.js:36-40, utils/index.js:15, 116, 172

Updating the RSS Feed

The final step involves writing the newly generated feed items to the feed.xml file.

1. The script reads the entire content of feed.xml into memory using getFeedContent().
2. It splits the content at the breakDelimiter (</image>), creating before and after sections.
3. The newly generated relevantComments string (containing all new <item> blocks) is concatenated between the before and after sections.
4. The complete, updated feed content is written back to feed.xml using overwriteFeedContent().

This method ensures that new items are always added to the same place in the document, preserving the overall structure of the RSS feed.

Sources: scripts/collect-discussions.js:43-46, utils/index.js:150, 158, config.json:11

Summary

The "Collecting GitHub Discussions" feature is an automated pipeline for sourcing news content. It leverages a simple yet effective configuration in config.json to target specific discussions. By using the GitHub GraphQL API, it efficiently fetches recent comments, filters them to avoid duplicates, and formats them into valid RSS items. The final step of injecting these items into feed.xml ensures that team updates are seamlessly integrated into the main news feed, contributing to the project's goal of providing timely and relevant Node.js news.

Sources: README.md, scripts/collect-discussions.js