Internal Data Schema

Relevant source files

The following files were used as context for generating this wiki page:

• utils/index.js
• scripts/build-rss.js
• config.json
• server.js
• scripts/collect-manual-posts.js
• scripts/collect-discussions.js
• scripts/collect-issues.js
• add-post.html
• feed.xml

The nodejs-news-feeder project aggregates information from multiple sources into a standardized RSS feed. The internal data schema defines the structure of configuration files, the format of data collected from various sources, and the final unified structure used to generate the feed.xml output. This ensures consistency and simplifies the process of adding new data sources.

The core of the schema is a unified "Feed Item" object, which acts as a common intermediate representation for all news entries, regardless of their origin. Configuration is managed through config.json, and manual entries are handled via a simple Express server and a corresponding JSON file, manual-posts.json.

Data Flow Overview

The system collects data from GitHub (issues, discussions) and a local file for manual posts. This data is then transformed into a common Feed Item structure, which is used to build the final feed.xml file.

This diagram illustrates the high-level data flow from sources to the final RSS feed.

Configuration Schema (config.json)

The config.json file is the central point for configuring the data collection scripts. It defines which repositories, issues, and discussions to monitor, and it maintains state between runs.

Sources: config.json, utils/index.js:132-134

Key	Type	Description
lastCheckTimestamp	Number	A Unix timestamp indicating the last time the feed was populated. Used to fetch only new content.
reposPaginationLimit	Number	The maximum number of repositories to paginate through.
releasePaginationLimit	Number	The maximum number of releases to paginate through.
commentsPaginationLimit	Number	The maximum number of comments to paginate through for issues and discussions.
breakDelimiter	String	A string (</image>) used as a marker in feed.xml to inject new items.
discussionsInScope	Array	An array of objects, each specifying a GitHub discussion (discussionId) and team name to collect updates from.
issuesInScope	Array	An array of objects, each specifying a GitHub issue (issue as owner/repo/issues/number) and team name to collect updates from.
lastFeedContent	String	A cached copy of the feed.xml content.

Core Data Schema: Feed Item

All data collected from various sources is normalized into a standard "Feed Item" object before being converted to XML. This is handled by the composeFeedItem function.

Sources: utils/index.js:139-159

The following diagram shows the structure of the Feed Item object passed to composeFeedItem.

composeFeedItem Function

This utility function takes the structured object and generates an XML <item> block for the RSS feed.

Parameter	Type	Description
title	String	The title of the feed item. It is cleaned to strip HTML.
description	String	The main content of the feed item. It is also cleaned.
pubDate	String	The publication date, formatted as an RFC 822 string.
link	String	The URL pointing to the original content.
guid	String	A unique identifier for the item, usually the same as the link.
source	String	The origin of the data (not directly used in the output XML but passed to the function).

Sources: utils/index.js:139-159

Final XML Item Schema

The composeFeedItem function produces an XML snippet with the following structure, which is then injected into feed.xml.

<item>
  <title>Cleaned Title</title>
  <description>Cleaned Description</description>
  <pubDate>RFC 822 Formatted Date</pubDate>
  <link>https://example.com/source</link>
  <guid>https://example.com/source</guid>
</item>

Sources: utils/index.js:150-158, feed.xml

Manual Post Schema

The system allows for adding posts manually via a simple web interface and a backing Express server. These posts are stored in manual-posts.json.

Sources: server.js, add-post.html, scripts/collect-manual-posts.js

Manual Post API

An Express server provides an API to add and retrieve manual posts.

Endpoint	Method	Description
/api/posts	GET	Retrieves all posts from manual-posts.json.
/api/posts	POST	Adds a new post to manual-posts.json.

Sources: server.js:15-25, server.js:27-48

Manual Post Submission Flow

The following sequence diagram shows how a user adds a new post through the web interface.

manual-posts.json Schema

The manual-posts.json file contains an array of post objects. When a new post is submitted, the server adds an id and a standardized published_at timestamp.

The structure of a post object in the JSON file is as follows:

[
  {
    "title": "Example Manual Post",
    "description": "This is the content of the post.",
    "link": "https://example.com/manual-post",
    "published_at": "2023-10-27T10:00:00.000Z",
    "id": 1698397200000
  }
]

Sources: server.js:39-43, scripts/collect-manual-posts.js:5-15