Data Curation Process

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• scripts/collect-issues.js
• scripts/collect-discussions.js
• scripts/collect-releases.js
• scripts/collect-retrospective.js
• scripts/collect-manual-posts.js
• utils/index.js
• config.json
• scripts/build-rss.js
• scripts/build-website.js
• server.js
• add-post.html
• package.json

The Data Curation Process is a semi-automated system designed to aggregate news and updates from various sources into a centralized RSS feed and a static website. It aims to improve communication and awareness by collecting information from GitHub repositories, external RSS feeds, and manual submissions, processing it, and presenting it for human review before publication.

The core of the process is a set of data collection scripts orchestrated by a GitHub Action. This action, triggered on a schedule or manually, gathers new content, formats it into a standard feed item structure, and updates the main feed.xml file. The changes are then submitted as a pull request, allowing curators to review and approve the content, ensuring its quality and relevance before it is published.

Sources: README.md:9-15, README.md:31-37

Architecture Overview

The process can be broken down into four main stages: Data Collection, Data Processing, Curation, and Publication. A GitHub Action serves as the entry point, executing a series of scripts to perform these stages.

The following diagram illustrates the high-level workflow of the data curation process.

Sources: README.md:31-37, package.json:7-13

Data Collection

Data is collected from multiple sources using a series of Node.js scripts. Each script is responsible for fetching data from a specific source type, filtering for new content, and preparing it for processing. The lastCheckTimestamp from config.json is a critical parameter used by most scripts to ensure only new content is fetched.

Sources: config.json:2, scripts/collect-issues.js:15, scripts/collect-releases.js:21

Data Sources

The system aggregates data from the following primary sources:

Source Type	Collection Script	Description
GitHub Releases	scripts/collect-releases.js	Fetches new releases from repositories defined in config.json.
GitHub Issues	scripts/collect-issues.js	Fetches new comments from specific issues defined in config.json.
GitHub Discussions	scripts/collect-discussions.js	Fetches new comments from specific discussions defined in config.json.
External Feeds	scripts/collect-retrospective.js	Fetches items from external RSS/JSON feeds and scrapes HTML from websites.
Manual Posts	scripts/collect-manual-posts.js	Reads manually created posts from manual-posts.json.

Sources: package.json:7-11, config.json

GitHub Data Fetching

Scripts for collecting data from GitHub use the @octokit/graphql and gh-got libraries to interact with the GitHub API.

• Releases (collect-releases.js): This script paginates through repositories belonging to the thomas-iniguez-visioli user, fetches releases for each, and filters them based on the lastCheckTimestamp.
• Issues (collect-issues.js): This script iterates through issuesInScope from config.json and fetches comments newer than lastCheckTimestamp.
• Discussions (collect-discussions.js): This script uses a GraphQL query to fetch the last 100 comments from discussions specified in discussionsInScope and filters them by publishedAt date against the lastCheckTimestamp.

Sources: scripts/collect-releases.js:7-22, scripts/collect-issues.js:6-16, scripts/collect-discussions.js:6-28

External & Scraped Data Fetching

The collect-retrospective.js script is responsible for fetching data from a variety of external sources.

• RSS/JSON Feeds: It fetches and parses items from a predefined list of feed URLs, such as haveibeenpwned.com/feed/breaches/. It uses the rss-parser and got libraries for this.
• HTML Scraping: The script also scrapes content directly from https://bonjourlafuite.eu.org/. It fetches the HTML, uses node-html-parser to query for div.timeline-entry elements, and extracts the title, content, and source link for each data breach entry.

The following diagram shows the flow for fetching external data.

Sources: scripts/collect-retrospective.js:19-35, scripts/collect-retrospective.js:40-88, scripts/collect-retrospective.js:176-203

Manual Post Submission

A simple Express server provides a web interface for manually adding posts.

1. A user navigates to add-post.html.
2. The user fills out a form with a title, description, link, and publication date.
3. On submission, a POST request is sent to the /api/posts endpoint handled by server.js.
4. The server reads the existing manual-posts.json, appends the new post with a generated timestamp, and writes the file back to disk.
5. The collect-manual-posts.js script later reads this JSON file to include these posts in the main feed.

Sources: server.js, add-post.html, scripts/collect-manual-posts.js

Data Processing and Formatting

Once data is collected, it is processed by utility functions in utils/index.js before being inserted into the feed.xml.

Feed Item Composition

The composeFeedItem function is the central component for creating a standardized RSS <item> block. It takes an object with properties like title, description, pubDate, link, and guid and formats them into an XML string.

// file: utils/index.js:145-159
export function composeFeedItem({ title, description, pubDate, link, guid,source }) {
  // ...
  const cleanTitle = cleancontent(title, { /* ... */ });
  const cleanDescription = cleancontent(description, { /* ... */ });

  return `
    <item>
      <title>${cleanTitle}</title>
      <description>${cleanDescription}</description>
      <pubDate>${pubDate}</pubDate>
      <link>${cleancontent(link,{})}</link>
      <guid>${guid}</guid>
    </item>
  `
}

Sources: utils/index.js:145-159

Content Sanitization and Formatting

Several utility functions are used to clean and format the content to be XML-safe and correctly presented.

Function	Description	File
cleancontent()	The main cleaning function that removes blacklisted keywords from text. It reads keywords from blacklist.txt.	utils/index.js:98-116
md2html()	Converts Markdown text to HTML using remark and remark-html.	utils/index.js:14-16
escapeXmlCharacters()	Escapes special XML characters like &, <, >, ", '.	utils/index.js:27-33
stripHtmlTags()	Removes all HTML tags from a string.	utils/index.js:40-52
buildRFC822Date()	Formats a date string into the RFC 822 format required for RSS feeds (e.g., Mon, 29 Nov 2021 01:00:00 BST).	utils/index.js:189-203
filterFeedItems()	Filters an array of feed items to remove duplicates based on guid, link, or title, and excludes entries with missing required fields.	utils/index.js:218-232

The sequence diagram below shows how a collection script uses these utilities to process a piece of content.

Sources: utils/index.js, scripts/collect-discussions.js:32-38

Configuration

The entire process is driven by config.json. This file contains crucial parameters that control the behavior of the collection scripts.

Parameter	Type	Description
lastCheckTimestamp	Number	A Unix timestamp indicating the last time the feed was successfully updated. Used to fetch only new content.
reposPaginationLimit	Number	The maximum number of repositories to fetch when collecting releases.
releasePaginationLimit	Number	The maximum number of releases to fetch per repository.
commentsPaginationLimit	Number	The maximum number of comments to fetch per issue.
breakDelimiter	String	An XML tag (</image>) used as a marker to inject new items into feed.xml.
discussionsInScope	Array	An array of objects specifying which GitHub discussions to monitor.
issuesInScope	Array	An array of objects specifying which GitHub issues to monitor.
lastFeedContent	String	A stored copy of the last generated feed.xml content, used to detect changes.

Sources: config.json:2-10, config.json:11

Feed Generation and Publication

After all collection scripts have run, the new items are aggregated and injected into the main feed.xml file.

1. Injection: Each collection script reads the existing feed.xml, splits it at the breakDelimiter (</image>), and inserts the newly generated <item> blocks in between.
2. Date Update: The scripts/build-rss.js script runs last. It checks if the content of feed.xml has changed by comparing its length to the lastFeedContent stored in config.json. If it has changed, it updates the <lastBuildDate> tag to the current time.
3. Timestamp and Cache Update: If changes were made, build-rss.js also updates the lastCheckTimestamp in config.json to the current time and saves the new XML content to lastFeedContent for the next run.
4. Website Build: The scripts/build-website.js script parses the final feed.xml using @ulisesgascon/rss-feed-parser, renders the items into an HTML page using an EJS template, and overwrites index.html.

This automated process ensures that the RSS feed, website, and configuration state are all kept in sync upon successful data collection.

Sources: scripts/collect-issues.js:24-27, scripts/build-rss.js:14-25, scripts/build-website.js:5-11