Configuring `config.json`
Relevant source files
The following files were used as context for generating this wiki page:The config.json file is the central configuration hub for the nodejs-news-feeder application. It defines the parameters and state necessary for the various data collection scripts to run, fetch new content, and update the RSS feed. This file controls everything from which GitHub repositories and discussions to monitor to the stateful tracking of when the last successful data fetch occurred.
Properly configuring this file is essential for the correct operation of the news feeder. The utility functions getConfig() and overwriteConfig() are used throughout the scripts to read from and write to this file, ensuring a consistent state across different processes.
Sources: utils/index.js:124-130, config.json
Configuration Parameters
The config.json file contains several key-value pairs that control the behavior of the data collection and feed generation scripts.
Configuration Schema
The following table provides an overview of the main configuration keys, their purpose, and the scripts that utilize them.
| Key | Type | Description | Used By |
|---|---|---|---|
lastCheckTimestamp | Number | A Unix timestamp (in milliseconds) of the last time the feed was successfully updated. This is used to fetch only new items. | build-rss.js, collect-releases.js, collect-issues.js, collect-discussions.js |
reposPaginationLimit | Number | The maximum number of repositories to fetch from the organization. | collect-releases.js |
releasePaginationLimit | Number | The maximum number of releases to fetch per repository. | collect-releases.js |
commentsPaginationLimit | Number | The maximum number of comments to fetch from a GitHub issue. | collect-issues.js |
retrospective | Object | An object containing lastDay and nextDay strings to manage fetching of retrospective content. | collect-retrospective.js |
breakDelimiter | String | The string (</image>) used as a marker in feed.xml to inject new <item> elements. | All collect-*.js scripts |
discussionsInScope | Array | An array of objects specifying which GitHub Discussions to scrape for comments. | collect-discussions.js |
issuesInScope | Array | An array of objects specifying which GitHub Issues to scrape for comments. | collect-issues.js |
lastFeedContent | String | A stringified copy of the entire feed.xml from the previous run. Used to detect if any changes have occurred. | build-rss.js |
Sources: config.json:2-11, scripts/collect-releases.js:4, scripts/collect-issues.js:4, scripts/collect-discussions.js:4, scripts/build-rss.js:15
Configuration Management Flow
The scripts interact with config.json in a clear read-modify-write cycle, primarily managed by utility functions. This diagram illustrates the flow of configuration data.
This diagram shows that various scripts read the configuration via getConfig(). After processing, the build-rss.js script is responsible for updating the state (e.g., lastCheckTimestamp) by calling overwriteConfig().
Sources: utils/index.js:124-130, scripts/build-rss.js:20-24, scripts/collect-issues.js:4
State Management
config.json is not just for static configuration; it also maintains the state between script executions. This is crucial for ensuring that only new content is added to the feed.
lastCheckTimestamp
This is the most critical state-keeping parameter. It is a Unix timestamp in milliseconds that marks the last time the feed was successfully built.
- Usage: Each data collection script (
collect-releases.js,collect-issues.js,collect-discussions.js) uses this timestamp to filter results from the GitHub API, requesting only items that have been published or updated since the last check.
// Example from scripts/collect-issues.js
const { lastCheckTimestamp, breakDelimiter, issuesInScope, commentsPaginationLimit } = getConfig()
// ...
// Select only comments that are newer than the last check and add issue context
return issueComments
.filter(comment => new Date(comment.updated_at).getTime() > lastCheckTimestamp)
.map(comment => ({ ...comment, issue, team }))
Sources: scripts/collect-issues.js:4,15-16, config.json:2
- Update Mechanism: The
lastCheckTimestampis updated to the current time by thescripts/build-rss.jsscript after all collection scripts have run and the feed has been modified.
// From scripts/build-rss.js
const config = getConfig()
const now = new Date()
// ...
if (hasChanged) {
// ...
overwriteConfig({
...config,
lastCheckTimestamp: now.getTime(),
lastFeedContent: updatedXml // sauvegarde le contenu mis à jour
})
}
Sources: scripts/build-rss.js:4,18-24
lastFeedContent
This parameter stores a complete string of the feed.xml content from the previous run. Its primary purpose is to determine if any changes were made during the current run. The scripts/build-rss.js script compares the length of the new feed content with lastFeedContent. If they differ, it indicates new items were added, and the <lastBuildDate> in the RSS feed should be updated.
Sources: config.json:11, scripts/build-rss.js:15
The following sequence diagram illustrates the state update process managed by build-rss.js.
This diagram shows how build-rss.js reads both the feed and the config, checks for changes, and updates both files if new content has been added.
Sources: scripts/build-rss.js
Content Source Configuration
These parameters define where the collector scripts should look for new content.
issuesInScope and discussionsInScope
These arrays contain objects that point the collector scripts to specific GitHub issues or discussions. The scripts iterate over these arrays and use the GitHub API to fetch comments.
issuesInScope: Each object contains anissue(e.g.,"nodejs/node/issues/12345") and ateamname for context in the feed title.discussionsInScope: Each object contains adiscussionId(number) and ateamname.
An example of the structure within config.json:
{
"discussionsInScope": [],
"issuesInScope": []
}
Sources: config.json:10-11, scripts/collect-issues.js:5, scripts/collect-discussions.js:4
retrospective
This object is used by scripts/collect-retrospective.js to fetch content from external RSS feeds and other sources like bonjourlafuite.eu.org. The lastDay and nextDay properties appear to be related to a specific "retro-weekly" source, but the script also fetches from a hardcoded list of other feed URLs.
Sources: config.json:7-10, scripts/collect-retrospective.js:11, 155-158
Feed Generation Parameters
These settings control how the feed is constructed and paginated.
breakDelimiter
This string, </image>, is a simple yet effective marker within the feed.xml file. All data collection scripts read the feed content, split it using this delimiter, and inject the new <item> XML blocks they have generated. This ensures new content is consistently added to the same place in the feed structure before it is later sorted by date.
// Example from scripts/collect-releases.js
const feedContent = getFeedContent()
const [before, after] = feedContent.split(breakDelimiter)
const updatedFeedContent = `${before}${breakDelimiter}${relevantReleases}${after}`
overwriteFeedContent(updatedFeedContent)
Sources: config.json:10, utils/index.js:142-144, scripts/collect-releases.js:34-36
Pagination Limits
The reposPaginationLimit, releasePaginationLimit, and commentsPaginationLimit keys are used to limit the amount of data fetched from the GitHub API in a single run. This helps prevent rate-limiting issues and controls resource usage. For example, collect-releases.js uses reposPaginationLimit when fetching the list of all repositories and releasePaginationLimit for fetching releases within each repository.
Sources: config.json:3-6, scripts/collect-releases.js:7,15, scripts/collect-issues.js:8