Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:- README.md
- package.json
- config.json
- utils/index.js
- server.js
- add-post.html
- scripts/collect-releases.js
- scripts/collect-issues.js
- scripts/collect-discussions.js
- scripts/collect-retrospective.js
- scripts/collect-manual-posts.js
- scripts/build-rss.js
- scripts/build-website.js
- scripts/format.js
- scripts/validate.js
This document provides a technical overview of the nodejs-news-feeder project architecture. The system is designed to aggregate news and updates from various sources, format them into a standardized RSS feed, and generate a static HTML website for browser consumption. The primary goal is to provide a centralized and consumable feed of content relevant to the Node.js community.
The architecture is based on a series of Node.js scripts that collect, process, and build the final artifacts (feed.xml and index.html). The process is typically triggered by a GitHub Action, which can be run on a schedule (CRON) or manually. This automated workflow ensures that the feed remains up-to-date with minimal human intervention, while a pull request review process allows for content curation before publication.
Sources: README.md:10-14, README.md:27-33
High-Level Data Flow
The system operates in a four-stage process, starting from content generation and ending with consumption by end-users. A GitHub Action orchestrates the collection, processing, and formatting of data, which is then reviewed and merged to become publicly available.
Sources: README.md:27-33
This diagram illustrates the end-to-end flow of information through the system.
Sources: README.md:27-33, package.json:8-12
Core Components
The system is composed of several key components: data collection scripts, a build process for the RSS feed and website, a configuration file, and a set of utility functions.
Configuration (config.json)
The config.json file is the central point of configuration for the data collection and build scripts. It defines which sources to poll, pagination limits, and stateful information like the timestamp of the last check.
| Key | Description | Example Value |
|---|---|---|
lastCheckTimestamp | A Unix timestamp (in milliseconds) indicating the last time the feed was successfully updated. Used to fetch only new content. | 1763032644743 |
reposPaginationLimit | The maximum number of repositories to fetch when collecting releases. | 250 |
releasePaginationLimit | The maximum number of releases to fetch per repository. | 10 |
commentsPaginationLimit | The maximum number of comments to fetch from issues/discussions. | 100 |
breakDelimiter | An XML tag string (</image>) used as a placeholder to inject new feed items into feed.xml. | "</image>" |
discussionsInScope | An array of objects specifying GitHub discussions to monitor for new comments. | [] |
issuesInScope | An array of objects specifying GitHub issues to monitor for new comments. | [] |
lastFeedContent | A cached string of the entire feed.xml content from the last build, used to detect if any changes occurred. | "<?xml ..." |
Sources: config.json, scripts/build-rss.js:14-19, utils/index.js:108
Utility Functions (utils/index.js)
A collection of helper functions in utils/index.js provides common functionalities used across all scripts, such as data transformation, file I/O, and content cleaning.
| Function | Description |
|---|---|
getConfig() | Reads and parses config.json. |
overwriteConfig(config) | Stringifies and writes the updated configuration back to config.json. |
getFeedContent() | Reads the current feed.xml file. |
overwriteFeedContent(content) | Writes new content to feed.xml. |
composeFeedItem({...}) | Creates a single <item> XML block from a title, description, date, and link. |
md2html(md) | Converts a Markdown string to HTML using remark. |
buildRFC822Date(dateString) | Formats a date string into the RFC 822 format required for RSS feeds. |
filterFeedItems(items, opts) | Filters an array of feed items to remove duplicates and entries that don't match keywords or categories. |
cleancontent(text, opts) | Cleans text by removing blacklisted keywords and optionally stripping HTML. |
Sources: utils/index.js
Data Collection
Data is collected from multiple sources using a set of scripts defined in the scripts/ directory. Each script is responsible for a specific data source and appends new content to the feed.xml file.
The general workflow for a collection script is as follows:
- Read configuration from
config.json, especiallylastCheckTimestamp. - Fetch data from the source (e.g., GitHub API, external RSS feed).
- Filter the fetched data to include only items newer than
lastCheckTimestamp. - For each new item, format it into an RSS
<item>structure usingcomposeFeedItem. - Read the existing
feed.xml, find thebreakDelimiter, and inject the new items. - Write the updated content back to
feed.xml.
Sources: scripts/collect-releases.js, scripts/collect-issues.js, scripts/collect-discussions.js
Collection Scripts
The following scripts are used to gather content:
| Script | npm run command | Source | Description |
|---|---|---|---|
collect-releases.js | collect:releases | GitHub Repositories | Fetches new releases from repositories owned by a specific user. |
collect-issues.js | collect:issues | GitHub Issues | Fetches new comments from a predefined list of issues. |
collect-discussions.js | collect:discussions | GitHub Discussions | Fetches new comments from a predefined list of discussions using the GraphQL API. |
collect-retrospective.js | collect:retrospective | External Feeds & Websites | Scrapes content from bonjourlafuite.eu.org and parses external RSS feeds like haveibeenpwned.com. |
collect-manual-posts.js | collect:manual | manual-posts.json | Adds posts that were manually created and stored in a local JSON file. |
Sources: package.json:8-12, scripts/collect-releases.js, scripts/collect-issues.js, scripts/collect-discussions.js, scripts/collect-retrospective.js, scripts/collect-manual-posts.js
Manual Post Submission
For content that cannot be collected automatically, the project includes a simple Express.js server to add posts manually. Running npm start launches this server.
The server provides a web interface and a simple API:
- UI: Serves
add-post.html, which contains a form to submit a new post. - API Endpoints:
GET /api/posts: Returns all posts frommanual-posts.json.POST /api/posts: Receives new post data from the form, adds a timestamp and ID, and appends it tomanual-posts.json.
Sources: server.js, add-post.html
This sequence diagram shows the flow for adding a manual post.
Sources: server.js:28-46, add-post.html:49-67
Build and Deployment Process
After all collection scripts have run, a series of build, format, and validation scripts are executed to finalize the feed.xml and generate the index.html website.
RSS Feed Generation
The RSS feed is built and maintained through two main scripts:
-
build-rss.js: This script is responsible for updating the<lastBuildDate>infeed.xmlto the current time. It also saves the new content of the feed intoconfig.jsonunderlastFeedContentto track changes for future builds. This prevents unnecessary updates if no new items were added.
Sources: scripts/build-rss.js -
format.js: This script cleans up the feed. It parses the entirefeed.xml, sorts all items by publication date in descending order, and filters out duplicates. It then reconstructs the XML file and formats it for readability usingxml-formatter.
Sources: scripts/format.js
This diagram shows the feed formatting and build process.
Sources: scripts/format.js, scripts/build-rss.js
Website Generation
The static website is generated by scripts/build-website.js. This script:
- Reads the final
feed.xml. - Parses the RSS content into metadata and an array of items using
@ulisesgascon/rss-feed-parser. - Reads the EJS template from
templates/index.html.ejs. - Renders the template with the parsed feed data.
- Writes the final output to
index.html.
Sources: scripts/build-website.js
Feed Validation
To ensure the generated feed.xml is compliant with RSS standards, the scripts/validate.js script sends the XML content to the W3C Feed Validation Service (https://validator.w3.org/feed/check.cgi) and reports whether the feed is valid. This serves as a crucial quality check in the CI/CD pipeline.
Sources: scripts/validate.js
Summary
The nodejs-news-feeder architecture is a script-driven, automated system for content aggregation and distribution. It leverages Node.js for fetching data from diverse sources, including REST and GraphQL APIs, external RSS feeds, and scraped web pages. The system is configured via a central JSON file and uses a set of utility functions for common tasks. The final outputs are a standardized feed.xml and a static index.html, with a manual curation step via pull requests to ensure content quality. This modular and automated approach allows for efficient and reliable delivery of news to the Node.js community.
Sources: README.md, package.json## FULL FILE CONTEXT SUMMARY
Query: You are an expert code analyst examining the github repository: https://github.com/thomas-iniguez-visioli/nodejs-news-feeder (nodejs-news-feeder).
You provide direct, concise, and accurate information about code repositories.
You NEVER start responses with markdown headers or code fences.
IMPORTANT:You MUST respond in English language.
Total Files Included: 20
Total Files Skipped: 0
Total Context Tokens: 97,450 / 800,000 (12.2%)
Remaining Token Budget: 702,550
Included Files:
- CONTRIBUTING.md (473 tokens)
- README.md (869 tokens)
- CODE_OF_CONDUCT.md (43 tokens)
- package.json (660 tokens)
- scripts/collect-discussions.js (393 tokens)
- utils/tests/utils.test.js (2,530 tokens)
- utils/index.js (1,799 tokens)
- scripts/collect-issues.js (317 tokens)
- scripts/build-website.js (135 tokens)
- add-post.html (652 tokens)
- scripts/collect-manual-posts.js (258 tokens)
- server.js (353 tokens)
- scripts/collect-releases.js (389 tokens)
- index.html (23,787 tokens)
- scripts/build-rss.js (326 tokens)
- scripts/collect-retrospective.js (1,865 tokens)
- tmp.html (10,356 tokens)
- config.json (51,614 tokens)
- scripts/validate.js (214 tokens)
- scripts/format.js (417 tokens)
Code of Conduct
The project has a
Code of Conduct
to which all contributors must adhere.
See details on our policy on Code of Conduct.
Issues
You can use the issues to:
- Asking for General Help
- Discussing non-technical topics
- Submitting a Bug Report
Pull Requests
Feel free to submit a pull request (PR) if you see something that can be improved. We will provide you support if you are new to open source and/or contributing to open source projects.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
================================================================================
# Official Node.js News Feeder
Usage
As the Node.js organization continues to grow and expand, it becomes increasingly difficult to keep track of all the ongoing projects, teams, and working groups. We've recognized this challenge and have been actively seeking solutions to improve communication and awareness within our community.
This feed was established with the intention of offering support for news sites, blogs, and other creators that aim to deliver timely updates on Node.js news, releases, and other content relevant to the Node.js community.
Consume the feed
The RSS feed is available at https://nodejs.github.io/nodejs-news-feeder/feed.xml, optionally you can subscribe to the Slack channel #nodejs-news-feeder to receive the latest news.
Use your terminal to read the latest news
You can also read the latest news directly from your terminal using the following command:
curl -s https://nodejs.github.io/nodejs-news-feeder/feed.xml | grep -E '(<title>|<link>|<pubDate>)' | sed 's/<[^>]*>//g' | sed 's/^[ \t]*//;s/[ \t]*$//'
Read the news on your browser
You can also read the latest news on your browser using the following link: https://nodejs.github.io/nodejs-news-feeder/
In order to update the RSS feed, you need to trigger the Github Action Populate Feed manually or wait for the CRON job.
This process will generate a PR with the latest news, so we can change the content and decide when to merge it.
How it works
- The community creates releases and also generates content by replying to specific issues and discussions.
- A GitHub Action, triggered by a cron job or manual input, collects issues/releases/discussions, processes, formats, validates, and stores all the new information as feed items.
- The changes are then presented in a pull request, which is thoroughly reviewed by curators who ensure that the content meet our expectations, before is publicly available and promoted.
- Once the changes are approved and included in the main branch, the updated content can be accessed by readers via a feed or a specific channel on Slack.
Development
git clone https://github.com/nodejs/nodejs-news-feeder
cd nodejs-news-feeder
nvm use
npm install
Scripts
Code linter
npm run lint
npm run lint:fix
Testing
npm run test
npm run test:watch
npm run test:coverage
RSS formatter
Update the feed.xml file format
npm run rss:format-check
npm run rss:format
RSS Build
Update the feed.xml file with the latest news
npm run rss:build
RSS Validate
Check the current feed.xml against the https://validator.w3.org/feed/check.cgi
npm run rss:validate
License
MIT License
================================================================================
The Node.js Code of Conduct applies to this repo.
The Node.js Code of Conduct document can be found at
https://github.com/nodejs/admin/blob/main/CODE_OF_CONDUCT.md
================================================================================
{
"name": "nodejs-news-feeder",
"version": "1.0.0",
"description": "",
"private": true,
"type": "module",
"scripts": {
"collect:releases": "node scripts/collect-releases.js",
"collect:issues": "node scripts/collect-issues.js",
"collect:discussions": "node scripts/collect-discussions.js",
"collect:retrospective": "node scripts/collect-retrospective.js",
"collect:manual": "node scripts/collect-manual-posts.js",
"start": "node server.js",
"rss:validate": "node scripts/validate.js",
"rss:build": "node scripts/build-rss.js",
"rss:format": "node scripts/format.js>log.txt",
"rss:format-check": "node scripts/format-check.js",
"website:build": "node scripts/build-website.js",
"lint": "standard",
"lint:fix": "standard --fix",
"test": "env TZ='Europe/Amsterdam' node --test",
"test:coverage": "env TZ='Europe/Amsterdam' node --test --test-reporter=lcov --test-reporter-destination=lcov.info",
"test:watch": "env TZ='Europe/Amsterdam' node --test --watch"
},
"repository": {
"type": "git",
"url": "git+https://github.com/nodejs/nodejs-news-feeder.git%22
},
"author": "",
"license": "MIT",
"engines": {
"node": ">=22.0.0",
"npm": ">=9.6.4"
},
"bugs": {
"url": "https://github.com/nodejs/nodejs-news-feeder/issues%22
},
"homepage": "https://github.com/nodejs/nodejs-news-feeder#readme",
"devDependencies": {
"standard": "17.1.2"
},
"dependencies": {
"@octokit/graphql": "8.2.2",
"@ulisesgascon/rss-feed-parser": "^1.0.1",
"cacheable-request": "^13.0.12",
"cheerio": "^1.1.2",
"ejs": "3.1.10",
"express": "^5.1.0",
"extend": "^3.0.2",
"gh-got": "10.0.0",
"got": "13.0.0",
"jsdom": "22.1.0",
"luxon": "^2.3.2",
"node-html-parser": "^6.1.13",
"remark": "14.0.3",
"remark-html": "15.0.2",
"rss-parser": "^3.13.0",
"unified": "^11.0.5",
"xml-formatter": "3.6.7"
}
}
================================================================================
import { graphql } from '@octokit/graphql'
import { getConfig, composeFeedItem, buildTitleDate, md2html, buildRFC822Date, getFeedContent, overwriteFeedContent } from '../utils/index.js'
const { discussionsInScope, breakDelimiter, lastCheckTimestamp } = getConfig()
const comments = await Promise.all(discussionsInScope.map(async ({ discussionId, team }) => {
const { repository } = await graphql(
{ repository(name: "node", owner: "nodejs") { discussion(number: ${discussionId}) { comments(last: 100) { edges { node { body publishedAt updatedAt databaseId } } } } } } ,
{
headers: {
authorization: token ${process.env.GITHUB_TOKEN}
}
}
)
return repository.discussion.comments.edges
.filter(comment => new Date(comment.node.publishedAt).getTime() > lastCheckTimestamp)
.map(comment => ({ ...comment.node, team, discussionId }))
}))
const relevantComments = comments.flat().map(comment => composeFeedItem({
title: ${comment.team} update on ${buildTitleDate(comment.publishedAt)},
description: <![CDATA[${md2html(comment.body)}]]>,
pubDate: buildRFC822Date(comment.publishedAt),
link: https://github.com/orgs/nodejs/discussions/%24%7Bcomment.discussionId%7D#discussioncomment-${comment.databaseId},
guid: https://github.com/orgs/nodejs/discussions/%24%7Bcomment.discussionId%7D#discussioncomment-${comment.databaseId}
})).join('')
const feedContent = getFeedContent()
const [before, after] = feedContent.split(breakDelimiter)
const updatedFeedContent = ${before}${breakDelimiter}${relevantComments}${after}
overwriteFeedContent(updatedFeedContent)
================================================================================
import { describe, it } from 'node:test'
import { strictEqual } from 'node:assert/strict'
import {
buildRFC822Date,
escapeXmlCharacters,
stripHtmlTags,
escapeHtmlTags,
normalizeLineEndings,
normalizeWhitespace,
removeControlCharacters
} from '../index.js'
describe('Utils', () => {
describe('buildRFC822Date', () => {
it('should return a date in RFC822 format', () => {
strictEqual(buildRFC822Date('2021-11-29T00:00:00.000Z'), 'Mon, 29 Nov 2021 01:00:00 BST')
strictEqual(buildRFC822Date('2021-09-08T00:00:00.000+01:00'), 'Wed, 08 Sep 2021 01:00:00 BST')
})
})
describe('escapeXmlCharacters', () => {
it('should escape XML special characters', () => {
strictEqual(escapeXmlCharacters('Hello & World'), 'Hello & World')
strictEqual(escapeXmlCharacters('Price < 100'), 'Price < 100')
strictEqual(escapeXmlCharacters('Value > 50'), 'Value > 50')
strictEqual(escapeXmlCharacters('Say "Hello"'), 'Say "Hello"')
strictEqual(escapeXmlCharacters("It's working"), 'It's working')
})
it('should escape multiple XML characters in one string', () => {
strictEqual(escapeXmlCharacters('A & B < C > "D" & \'E\''), 'A & B < C > "D" & 'E'')
strictEqual(escapeXmlCharacters('<tag attr="value">Content & more</tag>'), '<tag attr="value">Content & more</tag>')
})
it('should handle empty and edge case strings', () => {
strictEqual(escapeXmlCharacters(''), '')
strictEqual(escapeXmlCharacters('No special chars'), 'No special chars')
strictEqual(escapeXmlCharacters('&&&'), '&&&')
})
it('should handle non-string inputs', () => {
strictEqual(escapeXmlCharacters(null), null)
strictEqual(escapeXmlCharacters(undefined), undefined)
strictEqual(escapeXmlCharacters(123), 123)
})
})
describe('stripHtmlTags', () => {
it('should remove simple HTML tags', () => {
strictEqual(stripHtmlTags('
Hello World
'), 'Hello World')strictEqual(stripHtmlTags('Content'), 'Content')
strictEqual(stripHtmlTags('Text'), 'Text')
})
it('should remove self-closing tags', () => {
strictEqual(stripHtmlTags('Line 1<br/>Line 2'), 'Line 1Line 2')
strictEqual(stripHtmlTags('Image: <img src="test.jpg"/>'), 'Image: ')
strictEqual(stripHtmlTags('Break<hr>here'), 'Breakhere')
})
it('should remove tags with attributes', () => {
strictEqual(stripHtmlTags('<a href="http://example.com">Link</a>'), 'Link')
strictEqual(stripHtmlTags('<div class="container" id="main">Content</div>'), 'Content')
strictEqual(stripHtmlTags('<img src="image.jpg" alt="description" />'), '')
})
it('should remove nested tags', () => {
strictEqual(stripHtmlTags('<div><p><strong>Bold</strong> text</p></div>'), 'Bold text')
strictEqual(stripHtmlTags('<ul><li>Item 1</li><li>Item 2</li></ul>'), 'Item 1Item 2')
})
it('should handle malformed HTML', () => {
strictEqual(stripHtmlTags('<p>Unclosed tag'), 'Unclosed tag')
strictEqual(stripHtmlTags('Text with < and > symbols'), 'Text with symbols')
strictEqual(stripHtmlTags('<>Empty tag</>'), 'Empty tag')
})
it('should handle edge cases', () => {
strictEqual(stripHtmlTags(''), '')
strictEqual(stripHtmlTags('No HTML here'), 'No HTML here')
strictEqual(stripHtmlTags('<><><>'), '')
})
it('should handle non-string inputs', () => {
strictEqual(stripHtmlTags(null), null)
strictEqual(stripHtmlTags(undefined), undefined)
strictEqual(stripHtmlTags(123), 123)
})
})
describe('escapeHtmlTags', () => {
it('should escape HTML tag brackets', () => {
strictEqual(escapeHtmlTags('
Hello World
'), '<p>Hello World</p>')strictEqual(escapeHtmlTags('Content'), '<div>Content</div>')
strictEqual(escapeHtmlTags('
'), '<br/>')
})
it('should escape tags with attributes', () => {
strictEqual(escapeHtmlTags('<a href="link">Text</a>'), '<a href="link">Text</a>')
strictEqual(escapeHtmlTags('<img src="image.jpg" alt="desc" />'), '<img src="image.jpg" alt="desc" />')
})
it('should escape nested tags', () => {
strictEqual(escapeHtmlTags('<div><p>Content</p></div>'), '<div><p>Content</p></div>')
})
it('should handle standalone brackets', () => {
strictEqual(escapeHtmlTags('Price < 100 > 50'), 'Price < 100 > 50')
strictEqual(escapeHtmlTags('Math: 5 < 10 and 15 > 10'), 'Math: 5 < 10 and 15 > 10')
})
it('should handle edge cases', () => {
strictEqual(escapeHtmlTags(''), '')
strictEqual(escapeHtmlTags('No brackets here'), 'No brackets here')
strictEqual(escapeHtmlTags('<>'), '<>')
})
it('should handle non-string inputs', () => {
strictEqual(escapeHtmlTags(null), null)
strictEqual(escapeHtmlTags(undefined), undefined)
strictEqual(escapeHtmlTags(123), 123)
})
})
describe('normalizeLineEndings', () => {
it('should convert Windows line endings to Unix', () => {
strictEqual(normalizeLineEndings('Line 1\r\nLine 2'), 'Line 1\nLine 2')
strictEqual(normalizeLineEndings('First\r\nSecond\r\nThird'), 'First\nSecond\nThird')
})
it('should handle mixed line endings', () => {
strictEqual(normalizeLineEndings('Line 1\r\nLine 2\nLine 3\r\nLine 4'), 'Line 1\nLine 2\nLine 3\nLine 4')
})
it('should preserve Unix line endings', () => {
strictEqual(normalizeLineEndings('Line 1\nLine 2\nLine 3'), 'Line 1\nLine 2\nLine 3')
})
it('should handle multiple consecutive line endings', () => {
strictEqual(normalizeLineEndings('Line 1\r\n\r\nLine 2'), 'Line 1\n\nLine 2')
strictEqual(normalizeLineEndings('\r\n\r\n\r\n'), '\n\n\n')
})
it('should handle edge cases', () => {
strictEqual(normalizeLineEndings(''), '')
strictEqual(normalizeLineEndings('No line endings'), 'No line endings')
strictEqual(normalizeLineEndings('\r\n'), '\n')
})
it('should handle non-string inputs', () => {
strictEqual(normalizeLineEndings(null), null)
strictEqual(normalizeLineEndings(undefined), undefined)
strictEqual(normalizeLineEndings(123), 123)
})
})
describe('normalizeWhitespace', () => {
it('should collapse multiple spaces to single space', () => {
strictEqual(normalizeWhitespace('Hello World'), 'Hello World')
strictEqual(normalizeWhitespace('Multiple spaces here'), 'Multiple spaces here')
})
it('should collapse mixed whitespace characters', () => {
strictEqual(normalizeWhitespace('Tab\t\tand spaces'), 'Tab and spaces')
strictEqual(normalizeWhitespace('Line\n\nbreaks and spaces'), 'Line breaks and spaces')
})
it('should handle leading and trailing whitespace', () => {
strictEqual(normalizeWhitespace(' Leading spaces'), ' Leading spaces')
strictEqual(normalizeWhitespace('Trailing spaces '), 'Trailing spaces ')
strictEqual(normalizeWhitespace(' Both sides '), ' Both sides ')
})
it('should handle various whitespace combinations', () => {
strictEqual(normalizeWhitespace('Word1\t\n \r\nWord2'), 'Word1 Word2')
strictEqual(normalizeWhitespace('A\t\t\tB\n\n\nC D'), 'A B C D')
})
it('should handle edge cases', () => {
strictEqual(normalizeWhitespace(''), '')
strictEqual(normalizeWhitespace('NoWhitespace'), 'NoWhitespace')
strictEqual(normalizeWhitespace(' '), ' ')
strictEqual(normalizeWhitespace('\t\n\r'), ' ')
})
it('should handle non-string inputs', () => {
strictEqual(normalizeWhitespace(null), null)
strictEqual(normalizeWhitespace(undefined), undefined)
strictEqual(normalizeWhitespace(123), 123)
})
})
describe('removeControlCharacters', () => {
it('should remove basic control characters', () => {
strictEqual(removeControlCharacters('Hello\x00World'), 'HelloWorld')
strictEqual(removeControlCharacters('Text\x01\x02\x03'), 'Text')
strictEqual(removeControlCharacters('\x1FBefore\x7FAfter\x9F'), 'BeforeAfter')
})
it('should preserve allowed control characters', () => {
strictEqual(removeControlCharacters('Line1\tTabbed\nLine2\rReturn'), 'Line1\tTabbed\nLine2\rReturn')
strictEqual(removeControlCharacters('Keep\x09tab\x0Anewline\x0Dreturn'), 'Keep\ttab\nnewline\rreturn')
})
it('should handle mixed control and regular characters', () => {
strictEqual(removeControlCharacters('Normal\x00text\x01with\x02controls\x03'), 'Normaltextwithcontrols')
strictEqual(removeControlCharacters('A\x00B\tC\nD\x1FE'), 'AB\tC\nDE')
})
it('should handle extended control characters', () => {
strictEqual(removeControlCharacters('Text\x80\x81\x9FEnd'), 'TextEnd')
strictEqual(removeControlCharacters('\x7F\x80\x90\x9F'), '')
})
it('should handle edge cases', () => {
strictEqual(removeControlCharacters(''), '')
strictEqual(removeControlCharacters('Normal text'), 'Normal text')
strictEqual(removeControlCharacters('\x00\x01\x02'), '')
})
it('should handle non-string inputs', () => {
strictEqual(removeControlCharacters(null), null)
strictEqual(removeControlCharacters(undefined), undefined)
strictEqual(removeControlCharacters(123), 123)
})
})
})
================================================================================
import { readFileSync, writeFileSync, existsSync} from 'fs'
import { join } from 'path'
import { createHash } from 'crypto'
import * as remark from 'remark'
import remarkHtml from 'remark-html'
import { DateTime } from 'luxon'
const dateRegex = /(\d*-\d*-\d*)/gm
const xmlFile = join(process.cwd(), 'feed.xml')
const configFile = join(process.cwd(), 'config.json')
const websiteFile = join(process.cwd(), 'index.html')
const websiteTemplate = join(process.cwd(), 'templates', 'index.html.ejs')
import got from 'got'
export function md2html(md) {
return remark.remark().use(remarkHtml).processSync(md).toString()
}
// XML character escape mapping
const xmlEscapeMap = {
'&': '&',
'<': '<',
'>': '>',
'"': '"',
"'": ''',
"< ":"","<=":""
}
/**
- Escapes XML special characters in text
- @param {string} text - The text to escape
- @returns {string} - Text with XML characters escaped
*/
export function escapeXmlCharacters(text) {
if (typeof text !== 'string') {
return text
}
return text.replace(/[&<>"']/g, (match) => xmlEscapeMap[match])
}
/**
- Strips HTML tags from text completely
- @param {string} text - The text to process
- @returns {string} - Text with HTML tags removed
*/
export function stripHtmlTags(text) {
if (typeof text !== 'string') {
return text
}
// Remove HTML tags (both opening and closing) using repeated replacement to avoid incomplete multi-character sanitization
let previous;
do {
previous = text;
text = text.replace(/<[^>]*>/g, '');
} while (text !== previous);
return text;
}
/**
- Escapes HTML tags as text (converts < and > to entities)
- @param {string} text - The text to process
- @returns {string} - Text with HTML tags escaped as text
*/
export function escapeHtmlTags(text) {
if (typeof text !== 'string') {
return text
}
// Escape HTML tags by converting < and > to entities
return text.replace(/</g, '<').replace(/>/g, '>')
}
const getkeys = () => {
if(existsSync(join(process.cwd(), "./blacklist.txt"))){
return readFileSync(join(process.cwd(), "./blacklist.txt"), "utf-8").split("\n").map((key) => key.trim().replace(/\r/g, ""))
}else{
writeFileSync(join(process.cwd(), "./blacklist.txt"), "")
return []
}
}
/**
-
Main cleancontent function that orchestrates all cleaning operations
-
@param {any} text - The content to clean (will be converted to string)
-
@param {object} options - Configuration options
-
@returns {string} - Clean XML-safe string
*/
export function cleancontent(text, options = {}) {
let validtext=text
const keys=getkeys()
//console.log(keys)
for(const key of keys){while (validtext.includes(key)){
//console.log(validtext.includes(key))
validtext=validtext.replace(key,"")
}
}
return validtext
// Merge and validate configuration options
}
export function buildTitleDate(timestamp) {
const [date, time] = new Date(timestamp).toISOString().split('T')
// Format: YYYY-MM-DD HH:MM:SS
return ${date} ${time.slice(0, 8)}
}
export function getConfig() {
return JSON.parse(readFileSync(configFile, 'utf8'))
}
export function overwriteConfig(config) {
writeFileSync(configFile, JSON.stringify(config, null, 2))
}
export function composeFeedItem({ title, description, pubDate, link, guid,source }) {
// console.log(pubDate)
//console.log(source)
// Clean title and description using cleancontent function
const cleanTitle = cleancontent(title, {
stripHtml: true,
preserveLineBreaks: false,
normalizeWhitespace: true
})
const cleanDescription = cleancontent(description, {
stripHtml: true,
preserveLineBreaks: false,
normalizeWhitespace: true
})
return <item> <title>${cleanTitle}</title> <description>${cleanDescription}</description> <pubDate>${pubDate}</pubDate> <link>${cleancontent(link,{})}</link> <guid>${guid}</guid> </item>
}
export function getFeedContent() {
return readFileSync(xmlFile, 'utf8')
}
export function getWebsiteTemplate() {
return readFileSync(websiteTemplate, 'utf8')
}
export function overwriteFeedContent(content) {
writeFileSync(xmlFile, content)
}
export function overwriteWebsiteContent(content) {
writeFileSync(websiteFile, content)
}
export function getFeedHash() {
const xml = getFeedContent()
return createHash('sha256').update(xml).digest('hex')
}
// @see: https://whitep4nth3r.com/blog/how-to-format-dates-for-rss-feeds-rfc-822/
export function addLeadingZero(num) {
num = num.toString()
while (num.length < 2) num = '0' + num
return num
}
export function buildRFC822Date(dateString) {
const dayStrings = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']
const monthStrings = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
const timeStamp = Date.parse(dateString)
const date = new Date(timeStamp)
const day = dayStrings[date.getDay()]
const dayNumber = addLeadingZero(date.getDate())
const month = monthStrings[date.getMonth()]
const year = date.getFullYear()
const time = ${addLeadingZero(date.getHours())}:${addLeadingZero(date.getMinutes())}:00
const timezone = date.getTimezoneOffset() === 0 ? 'GMT' : 'BST'
//console.log(${day}, ${dayNumber} ${month} ${year} ${time} ${timezone})
// Wed, 02 Oct 2002 13:00:00 GMT
return ${day}, ${dayNumber} ${month} ${year} ${time} ${timezone}
}
export async function generateRetroRequestUrl() {
//console.log(JSON.parse(await got.get("https://thomas-iniguez-visioli.github.io/retro-weekly/feed.json").text()))
return JSON.parse(await got.get("https://thomas-iniguez-visioli.github.io/retro-weekly/feed.json").text()).items
}
export function generateRetroUIUrl(dateString) {
return https://thomas-iniguez-visioli.github.io/retro-weekly/${dateString}/
}
export function parseRetrospectiveContent(data) {
const [rawTitle, , description] = data.split('\n')
const title = rawTitle.replace('# ', '').replaceAll('`', '').trim()
const dates = title.split(dateRegex)
//console.log(dates)
return { title, description, lastDay: dates[1], nextDay: dates[3] }
}
// Filtrage centralisé des items RSS
export function filterFeedItems(items, { keywords = [], categories = [] } = {}) {
const seen = new Set();
return items.filter(item => {
// Exclure les entrées incomplètes
if (!item.title || !item.link || !item.pubDate) return false;
// Filtrage par mot-clé
if (keywords.length && !keywords.some(k => item.title.includes(k) || (item.description && item.description.includes(k)))) return false;
// Filtrage par catégorie
if (categories.length && (!item.categories || !item.categories.some(c => categories.includes(c)))) return false;
// Doublons par guid ou lien ou titre
const key = (item.guid || item.link || item.title).trim();
if (seen.has(key)) return false;
seen.add(key);
return true;
});
}
================================================================================
import ghGot from 'gh-got'
import { md2html, buildTitleDate, buildRFC822Date, composeFeedItem, getFeedContent, overwriteFeedContent, getConfig } from '../utils/index.js'
const { lastCheckTimestamp, breakDelimiter, issuesInScope, commentsPaginationLimit } = getConfig()
// Collect all the comments for the issues in scope
const comments = await Promise.all(issuesInScope.map(async ({ issue, team }) => {
const issueComments = await ghGot(
repos/${issue}/comments,
{
token: process.env.GITHUB_TOKEN,
pagination: { countLimit: commentsPaginationLimit }
}
).json()
// Select only comments that are newer than the last check and add issue context
return issueComments
.filter(comment => new Date(comment.updated_at).getTime() > lastCheckTimestamp)
.map(comment => ({ ...comment, issue, team }))
}))
const relevantComments = comments.flat().map(comment => composeFeedItem({
title: ${comment.team} update on ${buildTitleDate(comment.created_at)},
description: <![CDATA[${md2html(comment.body)}]]>,
pubDate: buildRFC822Date(comment.created_at),
link: comment.html_url,
guid: comment.html_url
})).join('')
const feedContent = getFeedContent()
const [before, after] = feedContent.split(breakDelimiter)
const updatedFeedContent = ${before}${breakDelimiter}${relevantComments}${after}
overwriteFeedContent(updatedFeedContent)
================================================================================
import { getFeedContent, overwriteWebsiteContent, getWebsiteTemplate } from '../utils/index.js'
import ejs from 'ejs'
import { rssParse } from '@ulisesgascon/rss-feed-parser'
const template = getWebsiteTemplate()
const feed = getFeedContent()
try{
const { metadata, items } = rssParse(feed)
console .log(items)
const html = ejs.render(template, { metadata, items:items.map((e)=>{
e.link.replace('https://thomas-iniguez-visioli.github.io/nodejs-news-feeder/','')
return e
}) })
overwriteWebsiteContent(html)
}catch(err){console.log(err)}
================================================================================
Add Leak Post
Title Description Link Publication Date Add Post<h2>Current Manual Posts</h2>
<div id="posts-list"></div>
<script>
const form = document.getElementById('add-post-form');
const postsList = document.getElementById('posts-list');
async function getPosts() {
const response = await fetch('/api/posts');
const posts = await response.json();
postsList.innerHTML = '';
posts.forEach(post => {
const postElement = document.createElement('div');
postElement.innerHTML = `
<h3>${post.title}</h3>
<p>${post.description}</p>
<a href="${post.link}">${post.link}</a>
<p><small>Published on: ${new Date(post.published_at).toLocaleString()}</small></p>
<hr>
`;
postsList.appendChild(postElement);
});
}
form.addEventListener('submit', async (event) => {
event.preventDefault();
const formData = new FormData(form);
const data = Object.fromEntries(formData.entries());
const response = await fetch('/api/posts', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify(data)
});
if (response.ok) {
form.reset();
getPosts();
} else {
alert('Failed to add post');
}
});
getPosts();
</script>
================================================================================
import { md2html, buildRFC822Date, composeFeedItem, getFeedContent, overwriteFeedContent, getConfig } from './utils/index.js';
import fs from 'fs/promises';
import path from 'path';
const { breakDelimiter } = getConfig();
const postsFilePath = path.join(process.cwd(), 'manual-posts.json');
async function getManualPosts() {
try {
const postsData = await fs.readFile(postsFilePath, 'utf-8');
return JSON.parse(postsData);
} catch (error) {
if (error.code === 'ENOENT') {
return [];
}
throw error;
}
}
const manualPosts = await getManualPosts();
const relevantPosts = manualPosts.map(post => composeFeedItem({
title: post.title,
description: <![CDATA[${md2html(post.description)}]]>,
pubDate: buildRFC822Date(post.published_at),
link: post.link,
guid: post.link
})).join('');
if (relevantPosts) {
const feedContent = getFeedContent();
const [before, after] = feedContent.split(breakDelimiter);
const updatedFeedContent = ${before}${breakDelimiter}${relevantPosts}${after};
overwriteFeedContent(updatedFeedContent);
}
================================================================================
import express from 'express';
import fs from 'fs/promises';
import path from 'path';
const app = express();
const port = 3000;
const postsFilePath = path.join(process.cwd(), 'manual-posts.json');
app.use(express.json());
app.use(express.static(process.cwd()));
app.get('/', (req, res) => {
res.sendFile(path.join(process.cwd(), 'add-post.html'));
});
app.get('/api/posts', async (req, res) => {
try {
const posts = await fs.readFile(postsFilePath, 'utf-8');
res.json(JSON.parse(posts));
} catch (error) {
if (error.code === 'ENOENT') {
res.json([]);
} else {
res.status(500).send('Error reading posts');
}
}
});
app.post('/api/posts', async (req, res) => {
try {
let posts = [];
try {
const existingPosts = await fs.readFile(postsFilePath, 'utf-8');
posts = JSON.parse(existingPosts);
} catch (error) {
if (error.code !== 'ENOENT') throw error;
}
const newPost = {
...req.body,
id: Date.now(),
published_at: req.body.published_at ? new Date(req.body.published_at).toISOString() : new Date().toISOString()
};
posts.push(newPost);
await fs.writeFile(postsFilePath, JSON.stringify(posts, null, 2));
res.status(201).json(newPost);
} catch (error) {
res.status(500).send('Error saving post');
}
});
app.listen(port, () => {
console.log(Server running at http://localhost:${port});
});
================================================================================
================================================================================
FILE: