Relevant source files

The following files were used as context for generating this wiki page: - [README.md](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/README.md) - [package.json](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/package.json) - [config.json](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/config.json) - [utils/index.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/utils/index.js) - [server.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/server.js) - [add-post.html](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/add-post.html) - [scripts/collect-releases.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/scripts/collect-releases.js) - [scripts/collect-issues.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/scripts/collect-issues.js) - [scripts/collect-discussions.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/scripts/collect-discussions.js) - [scripts/collect-retrospective.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/scripts/collect-retrospective.js) - [scripts/collect-manual-posts.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/scripts/collect-manual-posts.js) - [scripts/build-rss.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/scripts/build-rss.js) - [scripts/build-website.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/scripts/build-website.js) - [scripts/format.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/scripts/format.js) - [scripts/validate.js](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/blob/main/scripts/validate.js)

# Architecture Overview This document provides a technical overview of the `nodejs-news-feeder` project architecture. The system is designed to aggregate news and updates from various sources, format them into a standardized RSS feed, and generate a static HTML website for browser consumption. The primary goal is to provide a centralized and consumable feed of content relevant to the Node.js community. The architecture is based on a series of Node.js scripts that collect, process, and build the final artifacts (`feed.xml` and `index.html`). The process is typically triggered by a GitHub Action, which can be run on a schedule (CRON) or manually. This automated workflow ensures that the feed remains up-to-date with minimal human intervention, while a pull request review process allows for content curation before publication. Sources: [README.md:10-14](), [README.md:27-33]() ## High-Level Data Flow The system operates in a four-stage process, starting from content generation and ending with consumption by end-users. A GitHub Action orchestrates the collection, processing, and formatting of data, which is then reviewed and merged to become publicly available. Sources: [README.md:27-33]() This diagram illustrates the end-to-end flow of information through the system. ```mermaid graph TD subgraph "1. Content Generation" A[GitHub Releases] B[GitHub Issues/Discussions] C[External RSS Feeds] D[Manual Posts] end subgraph "2. Data Aggregation (GitHub Action)" E[Collection Scripts] --> F{Process & Format} F --> G[Validate Content] G --> H[Store as Feed Items] end subgraph "3. Curation & Publication" I(Create Pull Request) --> J{Review & Curate} J --> K[Merge to main branch] end subgraph "4. Consumption" L[RSS Feed (feed.xml)] M[Website (index.html)] N[Slack Channel] end A --> E B --> E C --> E D --> E H --> I K --> L K --> M L --> N ``` Sources: [README.md:27-33](), [package.json:8-12]() ## Core Components The system is composed of several key components: data collection scripts, a build process for the RSS feed and website, a configuration file, and a set of utility functions. ### Configuration (`config.json`) The `config.json` file is the central point of configuration for the data collection and build scripts. It defines which sources to poll, pagination limits, and stateful information like the timestamp of the last check. | Key | Description | Example Value | | --- | --- | --- | | `lastCheckTimestamp` | A Unix timestamp (in milliseconds) indicating the last time the feed was successfully updated. Used to fetch only new content. | `1763032644743` | | `reposPaginationLimit` | The maximum number of repositories to fetch when collecting releases. | `250` | | `releasePaginationLimit` | The maximum number of releases to fetch per repository. | `10` | | `commentsPaginationLimit`| The maximum number of comments to fetch from issues/discussions. | `100` | | `breakDelimiter` | An XML tag string (``) used as a placeholder to inject new feed items into `feed.xml`. | `""` | | `discussionsInScope` | An array of objects specifying GitHub discussions to monitor for new comments. | `[]` | | `issuesInScope` | An array of objects specifying GitHub issues to monitor for new comments. | `[]` | | `lastFeedContent` | A cached string of the entire `feed.xml` content from the last build, used to detect if any changes occurred. | `"` XML block from a title, description, date, and link. | | `md2html(md)` | Converts a Markdown string to HTML using `remark`. | | `buildRFC822Date(dateString)` | Formats a date string into the RFC 822 format required for RSS feeds. | | `filterFeedItems(items, opts)` | Filters an array of feed items to remove duplicates and entries that don't match keywords or categories. | | `cleancontent(text, opts)` | Cleans text by removing blacklisted keywords and optionally stripping HTML. | Sources: [utils/index.js]() ## Data Collection Data is collected from multiple sources using a set of scripts defined in the `scripts/` directory. Each script is responsible for a specific data source and appends new content to the `feed.xml` file. The general workflow for a collection script is as follows: 1. Read configuration from `config.json`, especially `lastCheckTimestamp`. 2. Fetch data from the source (e.g., GitHub API, external RSS feed). 3. Filter the fetched data to include only items newer than `lastCheckTimestamp`. 4. For each new item, format it into an RSS `` structure using `composeFeedItem`. 5. Read the existing `feed.xml`, find the `breakDelimiter`, and inject the new items. 6. Write the updated content back to `feed.xml`. Sources: [scripts/collect-releases.js](), [scripts/collect-issues.js](), [scripts/collect-discussions.js]() ### Collection Scripts The following scripts are used to gather content: | Script | `npm run` command | Source | Description | | --- | --- | --- | --- | | `collect-releases.js` | `collect:releases` | GitHub Repositories | Fetches new releases from repositories owned by a specific user. | | `collect-issues.js` | `collect:issues` | GitHub Issues | Fetches new comments from a predefined list of issues. | | `collect-discussions.js`| `collect:discussions` | GitHub Discussions | Fetches new comments from a predefined list of discussions using the GraphQL API. | | `collect-retrospective.js`| `collect:retrospective` | External Feeds & Websites | Scrapes content from `bonjourlafuite.eu.org` and parses external RSS feeds like `haveibeenpwned.com`. | | `collect-manual-posts.js`| `collect:manual` | `manual-posts.json` | Adds posts that were manually created and stored in a local JSON file. | Sources: [package.json:8-12](), [scripts/collect-releases.js](), [scripts/collect-issues.js](), [scripts/collect-discussions.js](), [scripts/collect-retrospective.js](), [scripts/collect-manual-posts.js]() ### Manual Post Submission For content that cannot be collected automatically, the project includes a simple Express.js server to add posts manually. Running `npm start` launches this server. The server provides a web interface and a simple API: - **UI**: Serves `add-post.html`, which contains a form to submit a new post. - **API Endpoints**: - `GET /api/posts`: Returns all posts from `manual-posts.json`. - `POST /api/posts`: Receives new post data from the form, adds a timestamp and ID, and appends it to `manual-posts.json`. Sources: [server.js](), [add-post.html]() This sequence diagram shows the flow for adding a manual post. ```mermaid sequenceDiagram participant User as "User" participant Browser as "Browser (add-post.html)" participant Server as "Express Server (server.js)" participant JSONFile as "manual-posts.json" User->>Browser: Fills and submits form Browser->>Server: POST /api/posts (JSON payload) Server->>JSONFile: Read existing posts JSONFile-->>Server: Returns post array Server->>Server: Add new post to array Server->>JSONFile: Write updated post array Server-->>Browser: 201 Created Browser->>User: Resets form, updates list ``` Sources: [server.js:28-46](), [add-post.html:49-67]() ## Build and Deployment Process After all collection scripts have run, a series of build, format, and validation scripts are executed to finalize the `feed.xml` and generate the `index.html` website. ### RSS Feed Generation The RSS feed is built and maintained through two main scripts: 1. **`build-rss.js`**: This script is responsible for updating the `` in `feed.xml` to the current time. It also saves the new content of the feed into `config.json` under `lastFeedContent` to track changes for future builds. This prevents unnecessary updates if no new items were added. Sources: [scripts/build-rss.js]() 2. **`format.js`**: This script cleans up the feed. It parses the entire `feed.xml`, sorts all items by publication date in descending order, and filters out duplicates. It then reconstructs the XML file and formats it for readability using `xml-formatter`. Sources: [scripts/format.js]() This diagram shows the feed formatting and build process. ```mermaid flowchart TD A[Start `npm run rss:format`] --> B{Read feed.xml}; B --> C[Parse XML to JSON items]; C --> D[Filter for duplicates]; D --> E[Sort items by date]; E --> F[Re-compose XML items]; F --> G[Inject into XML template]; G --> H[Format with xml-formatter]; H --> I{Overwrite feed.xml}; I --> J[Start `npm run rss:build`]; J --> K{Read feed.xml}; K --> L{Check for changes vs. config}; L -- Changes found --> M[Update lastBuildDate]; M --> N{Overwrite feed.xml}; N --> O[Update lastCheckTimestamp in config.json]; L -- No changes --> P[End]; ``` Sources: [scripts/format.js](), [scripts/build-rss.js]() ### Website Generation The static website is generated by `scripts/build-website.js`. This script: 1. Reads the final `feed.xml`. 2. Parses the RSS content into metadata and an array of items using `@ulisesgascon/rss-feed-parser`. 3. Reads the EJS template from `templates/index.html.ejs`. 4. Renders the template with the parsed feed data. 5. Writes the final output to `index.html`. Sources: [scripts/build-website.js]() ### Feed Validation To ensure the generated `feed.xml` is compliant with RSS standards, the `scripts/validate.js` script sends the XML content to the W3C Feed Validation Service (`https://validator.w3.org/feed/check.cgi`) and reports whether the feed is valid. This serves as a crucial quality check in the CI/CD pipeline. Sources: [scripts/validate.js]() ## Summary The `nodejs-news-feeder` architecture is a script-driven, automated system for content aggregation and distribution. It leverages Node.js for fetching data from diverse sources, including REST and GraphQL APIs, external RSS feeds, and scraped web pages. The system is configured via a central JSON file and uses a set of utility functions for common tasks. The final outputs are a standardized `feed.xml` and a static `index.html`, with a manual curation step via pull requests to ensure content quality. This modular and automated approach allows for efficient and reliable delivery of news to the Node.js community. Sources: [README.md](), [package.json]()## FULL FILE CONTEXT SUMMARY Query: You are an expert code analyst examining the github repository: https://github.com/thomas-iniguez-visioli/nodejs-news-feeder (nodejs-news-feeder). You provide direct, concise, and accurate information about code repositories. You NEVER start responses with markdown headers or code fences. IMPORTANT:You MUST respond in English language. Total Files Included: 20 Total Files Skipped: 0 Total Context Tokens: 97,450 / 800,000 (12.2%) Remaining Token Budget: 702,550 ### Included Files: 1. CONTRIBUTING.md (473 tokens) 2. README.md (869 tokens) 3. CODE_OF_CONDUCT.md (43 tokens) 4. package.json (660 tokens) 5. scripts/collect-discussions.js (393 tokens) 6. utils/__tests__/utils.test.js (2,530 tokens) 7. utils/index.js (1,799 tokens) 8. scripts/collect-issues.js (317 tokens) 9. scripts/build-website.js (135 tokens) 10. add-post.html (652 tokens) 11. scripts/collect-manual-posts.js (258 tokens) 12. server.js (353 tokens) 13. scripts/collect-releases.js (389 tokens) 14. index.html (23,787 tokens) 15. scripts/build-rss.js (326 tokens) 16. scripts/collect-retrospective.js (1,865 tokens) 17. tmp.html (10,356 tokens) 18. config.json (51,614 tokens) 19. scripts/validate.js (214 tokens) 20. scripts/format.js (417 tokens) ================================================================================ FILE: CONTRIBUTING.md Relevance Score: 0.990 | Chunks Retrieved: 1 | Tokens: 473 ================================================================================ # Contributing * [Code of Conduct](#code-of-conduct) * [Issues](#issues) * [Pull Requests](#pull-requests) * [Developer's Certificate of Origin 1.1](#developers-certificate-of-origin) ## [Code of Conduct](./doc/contributing/code-of-conduct.md) The project has a [Code of Conduct](CODE_OF_CONDUCT.md) to which all contributors must adhere. See [details on our policy on Code of Conduct](CODE_OF_CONDUCT.md). ## [Issues](./doc/contributing/issues.md) You can use the issues to: - Asking for General Help - Discussing non-technical topics - Submitting a Bug Report ## [Pull Requests](./doc/contributing/pull-requests.md) Feel free to submit a pull request (PR) if you see something that can be improved. We will provide you support if you are new to open source and/or contributing to open source projects. ## Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

 (a) The contribution was created in whole or in part by me and I
     have the right to submit it under the open source license
     indicated in the file; or

 (b) The contribution is based upon previous work that, to the best
     of my knowledge, is covered under an appropriate open source
     license and I have the right under that license to submit that
     work with modifications, whether created in whole or in part
     by me, under the same open source license (unless I am
     permitted to submit under a different license), as indicated
     in the file; or

 (c) The contribution was provided directly to me by some other
     person who certified (a), (b) or (c) and I have not modified
     it.

 (d) I understand and agree that this project and the contribution
     are public and that a record of the contribution (including all
     personal information I submit with it, including my sign-off) is
     maintained indefinitely and may be redistributed consistent with
     this project or the open source license(s) involved.

================================================================================ ================================================================================ FILE: README.md Relevance Score: 0.980 | Chunks Retrieved: 1 | Tokens: 869 ================================================================================ [![Dependabot Updates](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/actions/workflows/dependabot/dependabot-updates/badge.svg)](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/actions/workflows/dependabot/dependabot-updates)[![Analyze with Semgrep and Fix with Pixee](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/actions/workflows/t.yml/badge.svg)](https://github.com/thomas-iniguez-visioli/nodejs-news-feeder/actions/workflows/t.yml)# Official Node.js News Feeder ### Usage As the Node.js organization continues to grow and expand, it becomes increasingly difficult to keep track of all the ongoing projects, teams, and working groups. We've recognized this challenge and have been actively seeking solutions to improve communication and awareness within our community. This feed was established with the intention of offering support for news sites, blogs, and other creators that aim to deliver timely updates on Node.js news, releases, and other content relevant to the Node.js community. #### Consume the feed The RSS feed is available at https://nodejs.github.io/nodejs-news-feeder/feed.xml, optionally you can subscribe to the Slack channel `#nodejs-news-feeder` to receive the latest news. #### Use your terminal to read the latest news You can also read the latest news directly from your terminal using the following command: ```bash curl -s https://nodejs.github.io/nodejs-news-feeder/feed.xml | grep -E '(|<link>|<pubDate>)' | sed 's/<[^>]*>//g' | sed 's/^[ \t]*//;s/[ \t]*$//' ``` #### Read the news on your browser You can also read the latest news on your browser using the following link: [https://nodejs.github.io/nodejs-news-feeder/](https://nodejs.github.io/nodejs-news-feeder/) In order to update the RSS feed, you need to trigger the Github Action `Populate Feed` [manually](https://github.com/nodejs/nodejs-news-feeder/actions/workflows/populate_feed.yml) or wait for the CRON job. This process will generate a PR with the latest news, so we can change the content and decide when to merge it. ### How it works ![architecture overview](https://blog.ulisesgascon.com/_next/image?url=%2Fimg%2Farch-nodejs-news-feeder.png&w=1920&q=75) 1. The community creates releases and also generates content by replying to specific issues and discussions. 2. A GitHub Action, triggered by a cron job or manual input, collects issues/releases/discussions, processes, formats, validates, and stores all the new information as feed items. 3. The changes are then presented in a pull request, which is thoroughly reviewed by curators who ensure that the content meet our expectations, before is publicly available and promoted. 4. Once the changes are approved and included in the main branch, the updated content can be accessed by readers via a feed or a specific channel on Slack. [More details](https://blog.ulisesgascon.com/nodejs-news-feeder) ### Development ```bash git clone https://github.com/nodejs/nodejs-news-feeder cd nodejs-news-feeder nvm use npm install ``` ### Scripts #### Code linter ```bash npm run lint npm run lint:fix ``` #### Testing ```bash npm run test npm run test:watch npm run test:coverage ``` #### RSS formatter Update the `feed.xml` file format ```bash npm run rss:format-check npm run rss:format ``` #### RSS Build Update the `feed.xml` file with the latest news ```bash npm run rss:build ``` #### RSS Validate Check the current `feed.xml` against the https://validator.w3.org/feed/check.cgi ```bash npm run rss:validate ``` ### License MIT License ================================================================================ ================================================================================ FILE: CODE_OF_CONDUCT.md Relevance Score: 0.970 | Chunks Retrieved: 1 | Tokens: 43 ================================================================================ # Code of Conduct The Node.js Code of Conduct applies to this repo. The Node.js Code of Conduct document can be found at https://github.com/nodejs/admin/blob/main/CODE_OF_CONDUCT.md ================================================================================ ================================================================================ FILE: package.json Relevance Score: 0.960 | Chunks Retrieved: 1 | Tokens: 660 ================================================================================ { "name": "nodejs-news-feeder", "version": "1.0.0", "description": "", "private": true, "type": "module", "scripts": { "collect:releases": "node scripts/collect-releases.js", "collect:issues": "node scripts/collect-issues.js", "collect:discussions": "node scripts/collect-discussions.js", "collect:retrospective": "node scripts/collect-retrospective.js", "collect:manual": "node scripts/collect-manual-posts.js", "start": "node server.js", "rss:validate": "node scripts/validate.js", "rss:build": "node scripts/build-rss.js", "rss:format": "node scripts/format.js>log.txt", "rss:format-check": "node scripts/format-check.js", "website:build": "node scripts/build-website.js", "lint": "standard", "lint:fix": "standard --fix", "test": "env TZ='Europe/Amsterdam' node --test", "test:coverage": "env TZ='Europe/Amsterdam' node --test --test-reporter=lcov --test-reporter-destination=lcov.info", "test:watch": "env TZ='Europe/Amsterdam' node --test --watch" }, "repository": { "type": "git", "url": "git+https://github.com/nodejs/nodejs-news-feeder.git%22 }, "author": "", "license": "MIT", "engines": { "node": ">=22.0.0", "npm": ">=9.6.4" }, "bugs": { "url": "https://github.com/nodejs/nodejs-news-feeder/issues%22 }, "homepage": "https://github.com/nodejs/nodejs-news-feeder#readme", "devDependencies": { "standard": "17.1.2" }, "dependencies": { "@octokit/graphql": "8.2.2", "@ulisesgascon/rss-feed-parser": "^1.0.1", "cacheable-request": "^13.0.12", "cheerio": "^1.1.2", "ejs": "3.1.10", "express": "^5.1.0", "extend": "^3.0.2", "gh-got": "10.0.0", "got": "13.0.0", "jsdom": "22.1.0", "luxon": "^2.3.2", "node-html-parser": "^6.1.13", "remark": "14.0.3", "remark-html": "15.0.2", "rss-parser": "^3.13.0", "unified": "^11.0.5", "xml-formatter": "3.6.7" } } ================================================================================ ================================================================================ FILE: scripts/collect-discussions.js Relevance Score: 0.950 | Chunks Retrieved: 1 | Tokens: 393 ================================================================================ import { graphql } from '@octokit/graphql' import { getConfig, composeFeedItem, buildTitleDate, md2html, buildRFC822Date, getFeedContent, overwriteFeedContent } from '../utils/index.js' const { discussionsInScope, breakDelimiter, lastCheckTimestamp } = getConfig() const comments = await Promise.all(discussionsInScope.map(async ({ discussionId, team }) => { const { repository } = await graphql( ` { repository(name: "node", owner: "nodejs") { discussion(number: ${discussionId}) { comments(last: 100) { edges { node { body publishedAt updatedAt databaseId } } } } } } `, { headers: { authorization: `token ${process.env.GITHUB_TOKEN}` } } ) return repository.discussion.comments.edges .filter(comment => new Date(comment.node.publishedAt).getTime() > lastCheckTimestamp) .map(comment => ({ ...comment.node, team, discussionId })) })) const relevantComments = comments.flat().map(comment => composeFeedItem({ title: `${comment.team} update on ${buildTitleDate(comment.publishedAt)}`, description: `<![CDATA[${md2html(comment.body)}]]>`, pubDate: buildRFC822Date(comment.publishedAt), link: `https://github.com/orgs/nodejs/discussions/%24%7Bcomment.discussionId%7D#discussioncomment-${comment.databaseId}`, guid: `https://github.com/orgs/nodejs/discussions/%24%7Bcomment.discussionId%7D#discussioncomment-${comment.databaseId}` })).join('') const feedContent = getFeedContent() const [before, after] = feedContent.split(breakDelimiter) const updatedFeedContent = `${before}${breakDelimiter}${relevantComments}${after}` overwriteFeedContent(updatedFeedContent) ================================================================================ ================================================================================ FILE: utils/__tests__/utils.test.js Relevance Score: 0.940 | Chunks Retrieved: 2 | Tokens: 2,530 ================================================================================ import { describe, it } from 'node:test' import { strictEqual } from 'node:assert/strict' import { buildRFC822Date, escapeXmlCharacters, stripHtmlTags, escapeHtmlTags, normalizeLineEndings, normalizeWhitespace, removeControlCharacters } from '../index.js' describe('Utils', () => { describe('buildRFC822Date', () => { it('should return a date in RFC822 format', () => { strictEqual(buildRFC822Date('2021-11-29T00:00:00.000Z'), 'Mon, 29 Nov 2021 01:00:00 BST') strictEqual(buildRFC822Date('2021-09-08T00:00:00.000+01:00'), 'Wed, 08 Sep 2021 01:00:00 BST') }) }) describe('escapeXmlCharacters', () => { it('should escape XML special characters', () => { strictEqual(escapeXmlCharacters('Hello & World'), 'Hello & World') strictEqual(escapeXmlCharacters('Price < 100'), 'Price < 100') strictEqual(escapeXmlCharacters('Value > 50'), 'Value > 50') strictEqual(escapeXmlCharacters('Say "Hello"'), 'Say "Hello"') strictEqual(escapeXmlCharacters("It's working"), 'It's working') }) it('should escape multiple XML characters in one string', () => { strictEqual(escapeXmlCharacters('A & B < C > "D" & \'E\''), 'A & B < C > "D" & 'E'') strictEqual(escapeXmlCharacters('<tag attr="value">Content & more</tag>'), '<tag attr="value">Content & more</tag>') }) it('should handle empty and edge case strings', () => { strictEqual(escapeXmlCharacters(''), '') strictEqual(escapeXmlCharacters('No special chars'), 'No special chars') strictEqual(escapeXmlCharacters('&&&'), '&&&') }) it('should handle non-string inputs', () => { strictEqual(escapeXmlCharacters(null), null) strictEqual(escapeXmlCharacters(undefined), undefined) strictEqual(escapeXmlCharacters(123), 123) }) }) describe('stripHtmlTags', () => { it('should remove simple HTML tags', () => { strictEqual(stripHtmlTags('<p>Hello World</p>'), 'Hello World') strictEqual(stripHtmlTags('<div>Content</div>'), 'Content') strictEqual(stripHtmlTags('<span>Text</span>'), 'Text') }) it('should remove self-closing tags', () => { strictEqual(stripHtmlTags('Line 1<br/>Line 2'), 'Line 1Line 2') strictEqual(stripHtmlTags('Image: <img src="test.jpg"/>'), 'Image: ') strictEqual(stripHtmlTags('Break<hr>here'), 'Breakhere') }) it('should remove tags with attributes', () => { strictEqual(stripHtmlTags('<a href="http://example.com">Link</a>'), 'Link') strictEqual(stripHtmlTags('<div class="container" id="main">Content</div>'), 'Content') strictEqual(stripHtmlTags('<img src="image.jpg" alt="description" />'), '') }) it('should remove nested tags', () => { strictEqual(stripHtmlTags('<div><p><strong>Bold</strong> text</p></div>'), 'Bold text') strictEqual(stripHtmlTags('<ul><li>Item 1</li><li>Item 2</li></ul>'), 'Item 1Item 2') }) it('should handle malformed HTML', () => { strictEqual(stripHtmlTags('<p>Unclosed tag'), 'Unclosed tag') strictEqual(stripHtmlTags('Text with < and > symbols'), 'Text with symbols') strictEqual(stripHtmlTags('<>Empty tag</>'), 'Empty tag') }) it('should handle edge cases', () => { strictEqual(stripHtmlTags(''), '') strictEqual(stripHtmlTags('No HTML here'), 'No HTML here') strictEqual(stripHtmlTags('<><><>'), '') }) it('should handle non-string inputs', () => { strictEqual(stripHtmlTags(null), null) strictEqual(stripHtmlTags(undefined), undefined) strictEqual(stripHtmlTags(123), 123) }) }) describe('escapeHtmlTags', () => { it('should escape HTML tag brackets', () => { strictEqual(escapeHtmlTags('<p>Hello World</p>'), '<p>Hello World</p>') strictEqual(escapeHtmlTags('<div>Content</div>'), '<div>Content</div>') strictEqual(escapeHtmlTags('<br/>'), '<br/>') }) it('should escape tags with attributes', () => { strictEqual(escapeHtmlTags('<a href="link">Text</a>'), '<a href="link">Text</a>') strictEqual(escapeHtmlTags('<img src="image.jpg" alt="desc" />'), '<img src="image.jpg" alt="desc" />') }) it('should escape nested tags', () => { strictEqual(escapeHtmlTags('<div><p>Content</p></div>'), '<div><p>Content</p></div>') }) it('should handle standalone brackets', () => { strictEqual(escapeHtmlTags('Price < 100 > 50'), 'Price < 100 > 50') strictEqual(escapeHtmlTags('Math: 5 < 10 and 15 > 10'), 'Math: 5 < 10 and 15 > 10') }) it('should handle edge cases', () => { strictEqual(escapeHtmlTags(''), '') strictEqual(escapeHtmlTags('No brackets here'), 'No brackets here') strictEqual(escapeHtmlTags('<>'), '<>') }) it('should handle non-string inputs', () => { strictEqual(escapeHtmlTags(null), null) strictEqual(escapeHtmlTags(undefined), undefined) strictEqual(escapeHtmlTags(123), 123) }) }) describe('normalizeLineEndings', () => { it('should convert Windows line endings to Unix', () => { strictEqual(normalizeLineEndings('Line 1\r\nLine 2'), 'Line 1\nLine 2') strictEqual(normalizeLineEndings('First\r\nSecond\r\nThird'), 'First\nSecond\nThird') }) it('should handle mixed line endings', () => { strictEqual(normalizeLineEndings('Line 1\r\nLine 2\nLine 3\r\nLine 4'), 'Line 1\nLine 2\nLine 3\nLine 4') }) it('should preserve Unix line endings', () => { strictEqual(normalizeLineEndings('Line 1\nLine 2\nLine 3'), 'Line 1\nLine 2\nLine 3') }) it('should handle multiple consecutive line endings', () => { strictEqual(normalizeLineEndings('Line 1\r\n\r\nLine 2'), 'Line 1\n\nLine 2') strictEqual(normalizeLineEndings('\r\n\r\n\r\n'), '\n\n\n') }) it('should handle edge cases', () => { strictEqual(normalizeLineEndings(''), '') strictEqual(normalizeLineEndings('No line endings'), 'No line endings') strictEqual(normalizeLineEndings('\r\n'), '\n') }) it('should handle non-string inputs', () => { strictEqual(normalizeLineEndings(null), null) strictEqual(normalizeLineEndings(undefined), undefined) strictEqual(normalizeLineEndings(123), 123) }) }) describe('normalizeWhitespace', () => { it('should collapse multiple spaces to single space', () => { strictEqual(normalizeWhitespace('Hello World'), 'Hello World') strictEqual(normalizeWhitespace('Multiple spaces here'), 'Multiple spaces here') }) it('should collapse mixed whitespace characters', () => { strictEqual(normalizeWhitespace('Tab\t\tand spaces'), 'Tab and spaces') strictEqual(normalizeWhitespace('Line\n\nbreaks and spaces'), 'Line breaks and spaces') }) it('should handle leading and trailing whitespace', () => { strictEqual(normalizeWhitespace(' Leading spaces'), ' Leading spaces') strictEqual(normalizeWhitespace('Trailing spaces '), 'Trailing spaces ') strictEqual(normalizeWhitespace(' Both sides '), ' Both sides ') }) it('should handle various whitespace combinations', () => { strictEqual(normalizeWhitespace('Word1\t\n \r\nWord2'), 'Word1 Word2') strictEqual(normalizeWhitespace('A\t\t\tB\n\n\nC D'), 'A B C D') }) it('should handle edge cases', () => { strictEqual(normalizeWhitespace(''), '') strictEqual(normalizeWhitespace('NoWhitespace'), 'NoWhitespace') strictEqual(normalizeWhitespace(' '), ' ') strictEqual(normalizeWhitespace('\t\n\r'), ' ') }) it('should handle non-string inputs', () => { strictEqual(normalizeWhitespace(null), null) strictEqual(normalizeWhitespace(undefined), undefined) strictEqual(normalizeWhitespace(123), 123) }) }) describe('removeControlCharacters', () => { it('should remove basic control characters', () => { strictEqual(removeControlCharacters('Hello\x00World'), 'HelloWorld') strictEqual(removeControlCharacters('Text\x01\x02\x03'), 'Text') strictEqual(removeControlCharacters('\x1FBefore\x7FAfter\x9F'), 'BeforeAfter') }) it('should preserve allowed control characters', () => { strictEqual(removeControlCharacters('Line1\tTabbed\nLine2\rReturn'), 'Line1\tTabbed\nLine2\rReturn') strictEqual(removeControlCharacters('Keep\x09tab\x0Anewline\x0Dreturn'), 'Keep\ttab\nnewline\rreturn') }) it('should handle mixed control and regular characters', () => { strictEqual(removeControlCharacters('Normal\x00text\x01with\x02controls\x03'), 'Normaltextwithcontrols') strictEqual(removeControlCharacters('A\x00B\tC\nD\x1FE'), 'AB\tC\nDE') }) it('should handle extended control characters', () => { strictEqual(removeControlCharacters('Text\x80\x81\x9FEnd'), 'TextEnd') strictEqual(removeControlCharacters('\x7F\x80\x90\x9F'), '') }) it('should handle edge cases', () => { strictEqual(removeControlCharacters(''), '') strictEqual(removeControlCharacters('Normal text'), 'Normal text') strictEqual(removeControlCharacters('\x00\x01\x02'), '') }) it('should handle non-string inputs', () => { strictEqual(removeControlCharacters(null), null) strictEqual(removeControlCharacters(undefined), undefined) strictEqual(removeControlCharacters(123), 123) }) }) }) ================================================================================ ================================================================================ FILE: utils/index.js Relevance Score: 0.930 | Chunks Retrieved: 2 | Tokens: 1,799 ================================================================================ import { readFileSync, writeFileSync, existsSync} from 'fs' import { join } from 'path' import { createHash } from 'crypto' import * as remark from 'remark' import remarkHtml from 'remark-html' import { DateTime } from 'luxon' const dateRegex = /(\d*-\d*-\d*)/gm const xmlFile = join(process.cwd(), 'feed.xml') const configFile = join(process.cwd(), 'config.json') const websiteFile = join(process.cwd(), 'index.html') const websiteTemplate = join(process.cwd(), 'templates', 'index.html.ejs') import got from 'got' export function md2html(md) { return remark.remark().use(remarkHtml).processSync(md).toString() } // XML character escape mapping const xmlEscapeMap = { '&': '&', '<': '<', '>': '>', '"': '"', "'": ''', "< ":"","<=":"" } /** * Escapes XML special characters in text * @param {string} text - The text to escape * @returns {string} - Text with XML characters escaped */ export function escapeXmlCharacters(text) { if (typeof text !== 'string') { return text } return text.replace(/[&<>"']/g, (match) => xmlEscapeMap[match]) } /** * Strips HTML tags from text completely * @param {string} text - The text to process * @returns {string} - Text with HTML tags removed */ export function stripHtmlTags(text) { if (typeof text !== 'string') { return text } // Remove HTML tags (both opening and closing) using repeated replacement to avoid incomplete multi-character sanitization let previous; do { previous = text; text = text.replace(/<[^>]*>/g, ''); } while (text !== previous); return text; } /** * Escapes HTML tags as text (converts < and > to entities) * @param {string} text - The text to process * @returns {string} - Text with HTML tags escaped as text */ export function escapeHtmlTags(text) { if (typeof text !== 'string') { return text } // Escape HTML tags by converting < and > to entities return text.replace(/</g, '<').replace(/>/g, '>') } const getkeys = () => { if(existsSync(join(process.cwd(), "./blacklist.txt"))){ return readFileSync(join(process.cwd(), "./blacklist.txt"), "utf-8").split("\n").map((key) => key.trim().replace(/\r/g, "")) }else{ writeFileSync(join(process.cwd(), "./blacklist.txt"), "") return [] } } /** * Main cleancontent function that orchestrates all cleaning operations * @param {any} text - The content to clean (will be converted to string) * @param {object} options - Configuration options * @returns {string} - Clean XML-safe string */ export function cleancontent(text, options = {}) { let validtext=text const keys=getkeys() //console.log(keys) for(const key of keys){ while (validtext.includes(key)){ //console.log(validtext.includes(key)) validtext=validtext.replace(key,"") } } return validtext // Merge and validate configuration options } export function buildTitleDate(timestamp) { const [date, time] = new Date(timestamp).toISOString().split('T') // Format: YYYY-MM-DD HH:MM:SS return `${date} ${time.slice(0, 8)}` } export function getConfig() { return JSON.parse(readFileSync(configFile, 'utf8')) } export function overwriteConfig(config) { writeFileSync(configFile, JSON.stringify(config, null, 2)) } export function composeFeedItem({ title, description, pubDate, link, guid,source }) { // console.log(pubDate) //console.log(source) // Clean title and description using cleancontent function const cleanTitle = cleancontent(title, { stripHtml: true, preserveLineBreaks: false, normalizeWhitespace: true }) const cleanDescription = cleancontent(description, { stripHtml: true, preserveLineBreaks: false, normalizeWhitespace: true }) return ` <item> <title>${cleanTitle} ${cleanDescription} ${pubDate} ${cleancontent(link,{})} ${guid} ` } export function getFeedContent() { return readFileSync(xmlFile, 'utf8') } export function getWebsiteTemplate() { return readFileSync(websiteTemplate, 'utf8') } export function overwriteFeedContent(content) { writeFileSync(xmlFile, content) } export function overwriteWebsiteContent(content) { writeFileSync(websiteFile, content) } export function getFeedHash() { const xml = getFeedContent() return createHash('sha256').update(xml).digest('hex') } // @see: https://whitep4nth3r.com/blog/how-to-format-dates-for-rss-feeds-rfc-822/ export function addLeadingZero(num) { num = num.toString() while (num.length < 2) num = '0' + num return num } export function buildRFC822Date(dateString) { const dayStrings = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat'] const monthStrings = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'] const timeStamp = Date.parse(dateString) const date = new Date(timeStamp) const day = dayStrings[date.getDay()] const dayNumber = addLeadingZero(date.getDate()) const month = monthStrings[date.getMonth()] const year = date.getFullYear() const time = `${addLeadingZero(date.getHours())}:${addLeadingZero(date.getMinutes())}:00` const timezone = date.getTimezoneOffset() === 0 ? 'GMT' : 'BST' //console.log(`${day}, ${dayNumber} ${month} ${year} ${time} ${timezone}`) // Wed, 02 Oct 2002 13:00:00 GMT return `${day}, ${dayNumber} ${month} ${year} ${time} ${timezone}` } export async function generateRetroRequestUrl() { //console.log(JSON.parse(await got.get("https://thomas-iniguez-visioli.github.io/retro-weekly/feed.json").text())) return JSON.parse(await got.get("https://thomas-iniguez-visioli.github.io/retro-weekly/feed.json").text()).items } export function generateRetroUIUrl(dateString) { return `https://thomas-iniguez-visioli.github.io/retro-weekly/${dateString}/` } export function parseRetrospectiveContent(data) { const [rawTitle, , description] = data.split('\n') const title = rawTitle.replace('# ', '').replaceAll('`', '').trim() const dates = title.split(dateRegex) //console.log(dates) return { title, description, lastDay: dates[1], nextDay: dates[3] } } // Filtrage centralisé des items RSS export function filterFeedItems(items, { keywords = [], categories = [] } = {}) { const seen = new Set(); return items.filter(item => { // Exclure les entrées incomplètes if (!item.title || !item.link || !item.pubDate) return false; // Filtrage par mot-clé if (keywords.length && !keywords.some(k => item.title.includes(k) || (item.description && item.description.includes(k)))) return false; // Filtrage par catégorie if (categories.length && (!item.categories || !item.categories.some(c => categories.includes(c)))) return false; // Doublons par guid ou lien ou titre const key = (item.guid || item.link || item.title).trim(); if (seen.has(key)) return false; seen.add(key); return true; }); } ================================================================================ ================================================================================ FILE: scripts/collect-issues.js Relevance Score: 0.920 | Chunks Retrieved: 1 | Tokens: 317 ================================================================================ import ghGot from 'gh-got' import { md2html, buildTitleDate, buildRFC822Date, composeFeedItem, getFeedContent, overwriteFeedContent, getConfig } from '../utils/index.js' const { lastCheckTimestamp, breakDelimiter, issuesInScope, commentsPaginationLimit } = getConfig() // Collect all the comments for the issues in scope const comments = await Promise.all(issuesInScope.map(async ({ issue, team }) => { const issueComments = await ghGot( `repos/${issue}/comments`, { token: process.env.GITHUB_TOKEN, pagination: { countLimit: commentsPaginationLimit } } ).json() // Select only comments that are newer than the last check and add issue context return issueComments .filter(comment => new Date(comment.updated_at).getTime() > lastCheckTimestamp) .map(comment => ({ ...comment, issue, team })) })) const relevantComments = comments.flat().map(comment => composeFeedItem({ title: `${comment.team} update on ${buildTitleDate(comment.created_at)}`, description: ``, pubDate: buildRFC822Date(comment.created_at), link: comment.html_url, guid: comment.html_url })).join('') const feedContent = getFeedContent() const [before, after] = feedContent.split(breakDelimiter) const updatedFeedContent = `${before}${breakDelimiter}${relevantComments}${after}` overwriteFeedContent(updatedFeedContent) ================================================================================ ================================================================================ FILE: scripts/build-website.js Relevance Score: 0.910 | Chunks Retrieved: 1 | Tokens: 135 ================================================================================ import { getFeedContent, overwriteWebsiteContent, getWebsiteTemplate } from '../utils/index.js' import ejs from 'ejs' import { rssParse } from '@ulisesgascon/rss-feed-parser' const template = getWebsiteTemplate() const feed = getFeedContent() try{ const { metadata, items } = rssParse(feed) console .log(items) const html = ejs.render(template, { metadata, items:items.map((e)=>{ e.link.replace('https://thomas-iniguez-visioli.github.io/nodejs-news-feeder/','') return e }) }) overwriteWebsiteContent(html) }catch(err){console.log(err)} ================================================================================ ================================================================================ FILE: add-post.html Relevance Score: 0.900 | Chunks Retrieved: 1 | Tokens: 652 ================================================================================ Add Leak Post

Add Leak Post

Current Manual Posts