Collecting GitHub Issues

Relevant source files

The following files were used as context for generating this wiki page:

• scripts/collect-issues.js
• config.json
• utils/index.js
• package.json
• README.md

The "Collecting GitHub Issues" feature is a core component of the Node.js News Feeder automation. Its primary purpose is to fetch new comments from specific, predefined GitHub issues and transform them into structured items for the project's RSS feed. This process is handled by the scripts/collect-issues.js script, which is executed via the npm run collect:issues command.

This automated collection is part of a larger workflow that aggregates content from various sources, including GitHub releases and discussions, to provide timely updates for the Node.js community. The collected comments are treated as news updates from different teams or working groups.

Sources: README.md, scripts/collect-issues.js, package.json:8

Architecture and Data Flow

The script operates by reading a configuration file, making requests to the GitHub API, processing the retrieved data, and then injecting the formatted content into the main feed.xml file.

High-Level Process Flow

The following diagram illustrates the end-to-end process of the collect-issues.js script.

Component Interaction

This sequence diagram shows how collect-issues.js interacts with other parts of the system, such as the configuration file, the GitHub API, and the local feed.xml file.

Configuration

The script's behavior is controlled by values stored in config.json. These are retrieved at runtime using the getConfig() utility function.

Sources: scripts/collect-issues.js:4, utils/index.js:125-127

Key	Type	Description
lastCheckTimestamp	Number	A Unix timestamp (in milliseconds) indicating the last time the script was run. Only comments updated after this time are processed.
issuesInScope	Array	An array of objects, where each object specifies a GitHub issue to monitor. Each object contains an issue (e.g., owner/repo#number) and a team name for context.
commentsPaginationLimit	Number	The maximum number of comments to fetch per issue. This limits the scope of the API request to avoid excessive data retrieval.
breakDelimiter	String	A string (e.g., </image>) used as a marker in feed.xml to determine where new <item> elements should be injected.

Sources: scripts/collect-issues.js:6, config.json:2-10

An example of the issuesInScope configuration:

{
  "issuesInScope": [
    {
      "issue": "nodejs/some-repo/issues/123",
      "team": "Working Group A"
    }
  ]
}

(Note: The actual issuesInScope array in config.json is empty, but its structure is implied by its usage in collect-issues.js).
Sources: config.json:10, scripts/collect-issues.js:9, 21

Data Fetching

The script iterates through each entry in the issuesInScope array and fetches its comments from the GitHub API.

API Request

• Library: gh-got is used to make authenticated requests to the GitHub API.
• Endpoint: The script targets the repos/{issue}/comments endpoint. The {issue} part is dynamically replaced with the repository and issue number from the configuration.
• Authentication: The request is authenticated using a GITHUB_TOKEN from the environment variables.
• Pagination: The pagination.countLimit option is set to the commentsPaginationLimit from the configuration to control how many comments are fetched.

Sources: scripts/collect-issues.js:10-16, package.json:33

// scripts/collect-issues.js:10-16
const issueComments = await ghGot(
            `repos/${issue}/comments`,
            {
              token: process.env.GITHUB_TOKEN,
              pagination: { countLimit: commentsPaginationLimit }
            }
).json()

Data Processing and Transformation

Once the comments are fetched, they are filtered and transformed into a format suitable for the RSS feed.

Filtering

Comments are filtered to include only those that have been updated since the last run of the script. This is achieved by comparing the comment.updated_at timestamp with the lastCheckTimestamp from the configuration.

Sources: scripts/collect-issues.js:20-21

// scripts/collect-issues.js:20-21
.filter(comment => new Date(comment.updated_at).getTime() > lastCheckTimestamp)
.map(comment => ({ ...comment, issue, team }))

Formatting

Each relevant comment is formatted into an RSS <item> XML block using the composeFeedItem utility function. The fields are constructed as follows:

Field	Value Construction	Utility Functions Used
title	A string combining the team name and the creation date of the comment.	buildTitleDate()
description	The Markdown body of the comment, converted to HTML and wrapped in <![CDATA[...]]>.	md2html()
pubDate	The creation date of the comment, formatted to the RFC 822 standard.	buildRFC822Date()
link	The direct URL to the comment on GitHub (comment.html_url).	cleancontent()
guid	The same URL as the link, used as a unique identifier for the feed item.	None

Sources: scripts/collect-issues.js:24-30, utils/index.js:133-156, 119-122, 180-195, 14-16

Feed Generation

The final step involves updating the feed.xml file with the newly generated items.

1. The existing content of feed.xml is read into memory using getFeedContent().
2. The content is split using the breakDelimiter defined in config.json. This delimiter acts as an injection point.
3. The newly generated RSS items are concatenated between the two parts of the original feed.
4. The complete, updated XML content is written back to feed.xml using overwriteFeedContent().

Sources: scripts/collect-issues.js:32-35, utils/index.js:158-160, 166-168, config.json:9

Summary

The "Collecting GitHub Issues" script is a vital automation tool that polls designated GitHub issues for new comments. It leverages a configuration file to define its scope, uses the GitHub API to fetch data, and employs a set of utility functions to transform comments into valid RSS feed items. This process ensures that updates posted as comments in specific tracking issues are automatically syndicated to the project's central news feed, improving communication and visibility.