Relevant source files

The following files were used as context for generating this wiki page: - [index.js](https://github.com/alphaleadership/feed/blob/main/index.js) - [themes/freemind/_config.yml](https://github.com/alphaleadership/feed/blob/main/themes/freemind/_config.yml) - [package.json](https://github.com/alphaleadership/feed/blob/main/package.json) - [public/index.html](https://github.com/alphaleadership/feed/blob/main/public/index.html) - [scripts/import.js](https://github.com/alphaleadership/feed/blob/main/scripts/import.js) - [update_breach_info.js](https://github.com/alphaleadership/feed/blob/main/update_breach_info.js)

# RSS/Atom Feed Generation The project features a multi-stage system for aggregating data from external sources and publishing it as content on a Hexo-based static site. A key output of this process is a public-facing Atom feed (`atom.xml`), which syndicates the generated content. The core of this system is a Node.js script that consumes an external RSS feed, processes its items, enriches them with other data, and creates markdown files that serve as the source for the Hexo site's pages and its own feed. This system effectively acts as a content pipeline, transforming structured data from various feeds and APIs into a unified, aggregated blog and Atom feed. The process is automated via scripts defined in `package.json` and configured through YAML files. ## System Architecture The overall process involves fetching data, processing and creating content files, and finally, generating the static site and its associated Atom feed. The main workflow is orchestrated by the `index.js` script, which is executed as part of the `build` process. The following diagram illustrates the high-level data flow from external sources to the final generated `atom.xml` file. ```mermaid flowchart TD subgraph External_Sources A[External RSS Feed
nodejs-news-feeder/feed.xml] B[HaveIBeenPwned API] C[Bonjourlafuite.eu.org] end subgraph Processing_Scripts D[index.js] E[update_breach_info.js] end subgraph Local_Data_and_Content F[source/_data/breaches.json] G[source/_posts/*.md] end subgraph Hexo_Generation H[Hexo Build Process] I[hexo-generator-feed] end subgraph Public_Output J[Public Website] K[public/atom.xml] end A --> D B --> D C --> E --> F D --> F D --> G F --> G G --> H H -- uses --> I I --> K H --> J ``` *Sources: [index.js](), [update_breach_info.js](), [package.json:5](), [themes/freemind/_config.yml:6-15]()* ## Configuration The generation of the Atom feed is configured within the theme's configuration file. The `hexo-generator-feed` plugin reads these settings to create the final `atom.xml`. *Sources: [themes/freemind/_config.yml:6-15](), [public/index.html:15]()* ### Feed Settings The primary settings are located under the `feed` key in `themes/freemind/_config.yml`. | Parameter | Value | Description | | :--- | :--- | :--- | | `enable` | `true` | Enables the feed generation plugin. | | `type` | `atom` | Specifies the feed format. | | `path` | `atom.xml` | The output filename for the generated feed. | | `limit` | `0` | The maximum number of posts to include (0 means all). | | `order_by` | `-date` | Sorts posts by date in descending order. | | `icon` | `icon.png` | The icon for the feed. | | `autodiscovery` | `true` | Adds a `` tag in the HTML head for feed discovery. | *Sources: [themes/freemind/_config.yml:6-15]()* ### Example Configuration ```yaml # themes/freemind/_config.yml feed: enable: true type: atom path: atom.xml limit: 0 hub: content: content_limit: 140 content_limit_delim: ' ' order_by: -date icon: icon.png autodiscovery: true template: ``` *Sources: [themes/freemind/_config.yml:6-15]()* ## Core Processing: `index.js` This script is the heart of the content aggregation system. It fetches data, processes it, and generates the markdown files that Hexo uses to build the site. The following sequence diagram shows the interactions within `index.js`. ```mermaid sequenceDiagram participant Script as "index.js" participant HIBP_API as "HaveIBeenPwned API" participant RSS_Source as "External RSS Feed" participant Filesystem as "Local Filesystem" Script->>HIBP_API: GET /api/v3/breaches HIBP_API-->>Script: Breach Data Script->>Filesystem: Update source/_data/breaches.json Script->>RSS_Source: parseURL(rssUrl) RSS_Source-->>Script: Feed Items loop For each unique item Script->>Script: Group items by company & date Script->>Filesystem: Read existing .md files for context Script->>Script: Generate new post content Script->>Filesystem: Write/Update .md file in source/_posts/ Script->>Filesystem: Update _config.yml & build.yml end ``` *Sources: [index.js]()* ### 1. Data Ingestion The script begins by fetching data from two primary external sources: 1. **HaveIBeenPwned API**: It makes a GET request to `https://haveibeenpwned.com/api/v3/breaches` to get an updated list of data breaches. This data is used to enrich the local `source/_data/breaches.json` file. *Sources: [index.js:10-20]()* 2. **External RSS Feed**: It uses the `rss-parser` library to parse an XML feed from `https://thomas-iniguez-visioli.github.io/nodejs-news-feeder/feed.xml`. This is the main source of new content. *Sources: [index.js:8](), [index.js:70]()* ### 2. Data Aggregation and Deduplication After fetching, the script processes the feed items: - **Deduplication**: Items are filtered to ensure uniqueness based on their `guid`, `link`, or `title`. - **Grouping**: Unique items are grouped into a `postsMap` object. The key for this map is a combination of the company name (derived from the link) and the publication date (e.g., `company#YYYY-M-D`). This allows multiple news items about the same company on the same day to be grouped. - **Merging with Existing Content**: The script reads existing markdown files from the corresponding company directory (`source/_posts/{company}/`) to merge previously processed items with new ones, preventing data loss and creating a comprehensive list. *Sources: [index.js:73-77](), [index.js:81-122]()* ### 3. Content Generation For each group of items in `postsMap`, the script generates one or more markdown files. - **File Path**: A destination directory is determined by the `l()` function, which typically resolves to `source/_posts/{company_name}`. - **File Naming**: If there's only one item for a company on a given day, the file is named `{Dir}#{dateS}.md`. If there are multiple, they are indexed: `{Dir}#{dateS}#{idx}.md`. - **Front-matter**: Each markdown file is created with YAML front-matter containing essential metadata. ```yaml title: ${Dir} fuite du ${dateS} date: ${dateS} lien: "${item.guid}" categories: - ${item.category1} - ${Dir} ``` *Sources: [index.js:212-217]()* - **Content Body**: The body of the post includes the cleaned content from the RSS item (`contentSnippet`). It also appends a list of links to other posts related to the same company ("Autres fuites pour ce dossier"). *Sources: [index.js:189-205]()* ## Final Feed Publication Once the markdown files are generated in `source/_posts/`, the standard Hexo build process takes over. ### Hexo Plugin The `hexo-generator-feed` plugin, listed as a dependency in `package.json`, is responsible for generating the `atom.xml` file. It iterates through the site's posts (which were created by `index.js`) and builds the feed according to the settings in `themes/freemind/_config.yml`. | Dependency | Purpose | | :--- | :--- | | `rss-parser` | To parse the incoming external RSS feed. | | `hexo-generator-feed` | To generate the site's own `atom.xml` from Hexo posts. | *Sources: [package.json:25](), [package.json:32]()* ### HTML Autodiscovery To allow browsers and feed readers to automatically find the feed, a `` tag is inserted into the `` of the generated HTML pages. This is enabled by the `autodiscovery: true` setting. ```html ``` *Sources: [public/index.html:14-15](), [themes/freemind/_config.yml:14]()* ## Supporting Data Pipelines Besides the main `index.js` flow, other scripts contribute to the data that ultimately appears in the posts and the feed. ### Breach Data Import (`scripts/import.js`) This script is designed to import data from local markdown files located in `source/_posts/{directory}` into the central `source/_data/breaches.json` file. It reads the front-matter from `.md` files, merges it with a default schema, and appends it to the `breaches.json` array. This allows for manual or semi-automated additions to the breach database, which is then used by other parts of the site. *Sources: [scripts/import.js]()* ### Breach Data Enrichment (`update_breach_info.js`) This script enhances the `source/_data/breaches.json` file by scraping victim counts from `https://bonjourlafuite.eu.org/`. It normalizes breach names to find matches between the local JSON data and the scraped data, then updates the corresponding records with the victim count (`pwnCountFr`) and source URL. This enriched data can then be displayed on the generated pages. *Sources: [update_breach_info.js]()* ## Conclusion The RSS/Atom feed generation in this project is a sophisticated content aggregation pipeline. It leverages a custom Node.js script to consume and process an external RSS feed, enriches this data with information from other APIs and web scrapers, and programmatically generates markdown files. These files are then used by the Hexo static site generator and the `hexo-generator-feed` plugin to build the final website and its public-facing `atom.xml` feed, effectively syndicating the aggregated and processed information.```json {"thought": "I need to create a technical wiki page about \"RSS/Atom Feed Generation\" for the `alphaleadership/feed` repository.\n\nHere's my plan:\n1. **Start with the `

` block.** I'll list the provided files (`feed.xml`, `_config.yml`) and find at least three more relevant files.\n * `index.js`: This is the core script that parses an RSS feed, processes the items, and creates Hexo posts. It's highly relevant.\n * `themes/freemind/_config.yml`: This theme configuration file has settings for the RSS feed (`feed` and `rss` keys).\n * `public/index.html`: This shows how the feed is linked in the final HTML output (``).\n * `package.json`: Lists `rss-parser` and `hexo-generator-feed` as dependencies, which are crucial for this feature.\n * `update_breach_info.js`: While not directly about RSS generation, it fetches data that is then used in the posts, which are part of the feed. It shows the data pipeline.\n * `scripts/import.js`: This script imports markdown files and adds them to `breaches.json`, which is a data source for the site content that eventually goes into the feed.\n\n I will use `index.js`, `themes/freemind/_config.yml`, `public/index.html`, `package.json`, and `scripts/import.js`. This gives me 5 files. I will also reference the original `_config.yml` if it's different from the theme's one, but the theme's config is more specific here.\n\n2. **Write the Introduction.** I'll explain that the project uses a Node.js script (`index.js`) to consume an external RSS feed, process its items, and generate markdown files for a Hexo static site. The Hexo site then generates its own Atom feed from these posts.\n\n3. **Create Detailed Sections.**\n * **## System Overview:** I'll provide a high-level flowchart of the entire process: from fetching the external RSS, to creating markdown files, to Hexo generating the site and the final `atom.xml`.\n * **## Configuration:** I'll detail the configuration options found in `themes/freemind/_config.yml` related to the feed. I'll create a table for these settings.\n * **## Core Processing Logic (`index.js`):** This is the main part.\n * **### RSS Feed Consumption:** Explain how `rss-parser` is used to fetch `https://thomas-iniguez-visioli.github.io/nodejs-news-feeder/feed.xml`.\n * **### Data Aggregation and Grouping:** Describe how the script groups feed items by \"entreprise\" (company) and date. It also reads existing markdown files to merge old and new data.\n * **### Post Generation:** Detail the logic for creating `.md` files in the `source/_posts` directory. I'll explain the file naming convention (`#.md` or `##.md`).\n * **### Content and Front-matter:** Show the structure of the generated markdown files, including the YAML front-matter (`title`, `date`, `lien`, `categories`).\n * **### Linking Related Posts:** Explain the logic that generates \"Autres fuites pour ce dossier\" links.\n * **## Hexo Feed Generation:** Explain that the `hexo-generator-feed` plugin (listed in `package.json`) is responsible for creating the final `atom.xml` file based on the generated posts. I'll reference the configuration in `themes/freemind/_config.yml`.\n * **## HTML Integration:** Show how the generated feed is linked in the `` of the HTML pages, as seen in `public/index.html`.\n\n4. **Incorporate Mermaid Diagrams.**\n * A `flowchart TD` for the overall data flow.\n * A `sequenceDiagram` for the `index.js` script's interaction with the RSS source and the local filesystem.\n\n5. **Use Tables.**\n * A table for the feed configuration options from `themes/freemind/_config.yml`.\n * A table for the key dependencies from `package.json`.\n\n6. **Add Code Snippets (Optional but good).**\n * The `feed` configuration from `themes/freemind/_config.yml`.\n * The `parser.parseURL` call from `index.js`.\n * The front-matter structure of a generated post from `index.js`.\n * The `` tag from `public/index.html`.\n\n7. **Cite Sources.** I will meticulously add `Sources: [filename:line-start-end]()` after every piece of information, diagram, table, and code snippet. I will ensure I cite at least 5 different files.\n\n8. **Write a Conclusion.** I'll summarize the dual-feed nature of the system: consuming one RSS feed to programmatically generate content, which is then used to publish a new, aggregated Atom feed.\n\n9. **Review and Refine.** I'll check for accuracy, clarity, and adherence to all the instructions, especially the \"no preamble\" and exact `

` block format. The language must be English.\n\nLet's start building the content.\n\n**File Selection:**\n1. `index.js` - Core logic.\n2. `themes/freemind/_config.yml` - Feed generation configuration.\n3. `package.json` - Dependencies like `rss-parser` and `hexo-generator-feed`.\n4. `public/index.html` - Final HTML output showing the feed link.\n5. `scripts/import.js` - Shows another data pipeline that feeds into the content used by the RSS feed.\n6. `update_breach_info.js` - Shows data enrichment from external sources.\n\nThis is a good set of files. I'll use these.\n\n**Structure:**\n- `

` block\n- `# RSS/Atom Feed Generation`\n- `## Introduction`\n- `## System Architecture`\n - Mermaid Flowchart\n- `## Configuration`\n - Table of `themes/freemind/_config.yml` settings\n- `## Core Processing: `index.js`\n - Mermaid Sequence Diagram\n - `### 1. Data Ingestion` (RSS parsing, HaveIBeenPwned API)\n - `### 2. Data Aggregation and Deduplication`\n - `### 3. Content Generation` (Markdown file creation)\n - `### 4. Configuration Updates` (`_config.yml` and `build.yml`)\n- `## Final Feed Publication`\n - `### Hexo Plugin` (mentioning `hexo-generator-feed`)\n - `### HTML Autodiscovery` (linking in the HTML head)\n- `## Supporting Data Pipelines`\n - `### Breach Data Import (`scripts/import.js`)`\n - `### Breach Data Enrichment (`update_breach_info.js`)`\n- `## Conclusion`\n\nThis structure seems logical and covers all aspects based on the provided files. I will now write the content for each section, ensuring to add citations for every claim.","output":"

Relevant source files

\nThe following files were used as context for generating this wiki page:\n\n- [index.js](https://github.com/alphaleadership/feed/blob/main/index.js)\n- [themes/freemind/_config.yml](https://github.com/alphaleadership/feed/blob/main/themes/freemind/_config.yml)\n- [package.json](https://github.com/alphaleadership/feed/blob/main/package.json)\n- [public/index.html](https://github.com/alphaleadership/feed/blob/main/public/index.html)\n- [scripts/import.js](https://github.com/alphaleadership/feed/blob/main/scripts/import.js)\n- [update_breach_info.js](https://github.com/alphaleadership/feed/blob/main/update_breach_info.js)\n

\n\n# RSS/Atom Feed Generation\n\nThe project features a multi-stage system for aggregating data from external sources and publishing it as content on a Hexo-based static site. A key output of this process is a public-facing Atom feed (`atom.xml`), which syndicates the generated content. The core of this system is a Node.js script that consumes an external RSS feed, processes its items, enriches them with other data, and creates markdown files that serve as the source for the Hexo site's pages and its own feed.\n\nThis system effectively acts as a content pipeline, transforming structured data from various feeds and APIs into a unified, aggregated blog and Atom feed. The process is automated via scripts defined in `package.json` and configured through YAML files.\n\n## System Architecture\n\nThe overall process involves fetching data, processing and creating content files, and finally, generating the static site and its associated Atom feed. The main workflow is orchestrated by the `index.js` script, which is executed as part of the `build` process.\n\nThe following diagram illustrates the high-level data flow from external sources to the final generated `atom.xml` file.\n\n```mermaid\nflowchart TD\n subgraph External_Sources\n A[External RSS Feed
nodejs-news-feeder/feed.xml]\n B[HaveIBeenPwned API]\n C[Bonjourlafuite.eu.org]\n end\n\n subgraph Processing_Scripts\n D[index.js]\n E[update_breach_info.js]\n end\n\n subgraph Local_Data_and_Content\n F[source/_data/breaches.json]\n G[source/_posts/*.md]\n end\n\n subgraph Hexo_Generation\n H[Hexo Build Process]\n I[hexo-generator-feed]\n end\n\n subgraph Public_Output\n J[Public Website]\n K[public/atom.xml]\n end\n\n A --> D\n B --> D\n C --> E --> F\n D --> F\n D --> G\n F --> G\n G --> H\n H -- uses --> I\n I --> K\n H --> J\n```\n*Sources: [index.js](), [update_breach_info.js](), [package.json:5](), [themes/freemind/_config.yml:6-15]()*\n\n## Configuration\n\nThe generation of the Atom feed is configured within the theme's configuration file. The `hexo-generator-feed` plugin reads these settings to create the final `atom.xml`.\n\n*Sources: [themes/freemind/_config.yml:6-15](), [public/index.html:15]()*\n\n### Feed Settings\n\nThe primary settings are located under the `feed` key in `themes/freemind/_config.yml`.\n\n| Parameter | Value | Description |\n| :--- | :--- | :--- |\n| `enable` | `true` | Enables the feed generation plugin. |\n| `type` | `atom` | Specifies the feed format. |\n| `path` | `atom.xml` | The output filename for the generated feed. |\n| `limit` | `0` | The maximum number of posts to include (0 means all). |\n| `order_by` | `-date` | Sorts posts by date in descending order. |\n| `icon` | `icon.png` | The icon for the feed. |\n| `autodiscovery` | `true` | Adds a `` tag in the HTML head for feed discovery. |\n\n*Sources: [themes/freemind/_config.yml:6-15]()*\n\n### Example Configuration\n\n```yaml\n# themes/freemind/_config.yml\nfeed:\n enable: true\n type: atom\n path: atom.xml\n limit: 0\n hub:\n content:\n content_limit: 140\n content_limit_delim: ' '\n order_by: -date\n icon: icon.png\n autodiscovery: true\n template:\n```\n*Sources: [themes/freemind/_config.yml:6-15]()*\n\n## Core Processing: `index.js`\n\nThis script is the heart of the content aggregation system. It fetches data, processes it, and generates the markdown files that Hexo uses to build the site.\n\nThe following sequence diagram shows the interactions within `index.js`.\n\n```mermaid\nsequenceDiagram\n participant Script as \"index.js\"\n participant HIBP_API as \"HaveIBeenPwned API\"\n participant RSS_Source as \"External RSS Feed\"\n participant Filesystem as \"Local Filesystem\"\n\n Script->>HIBP_API: GET /api/v3/breaches\n HIBP_API-->>Script: Breach Data\n Script->>Filesystem: Update source/_data/breaches.json\n\n Script->>RSS_Source: parseURL(rssUrl)\n RSS_Source-->>Script: Feed Items\n\n loop For each unique item\n Script->>Script: Group items by company & date\n Script->>Filesystem: Read existing .md files for context\n Script->>Script: Generate new post content\n Script->>Filesystem: Write/Update .md file in source/_posts/\n Script->>Filesystem: Update _config.yml & build.yml\n end\n```\n*Sources: [index.js]()*\n\n### 1. Data Ingestion\n\nThe script begins by fetching data from two primary external sources:\n\n1. **HaveIBeenPwned API**: It makes a GET request to `https://haveibeenpwned.com/api/v3/breaches` to get an updated list of data breaches. This data is used to enrich the local `source/_data/breaches.json` file.\n *Sources: [index.js:10-20]()*\n2. **External RSS Feed**: It uses the `rss-parser` library to parse an XML feed from `https://thomas-iniguez-visioli.github.io/nodejs-news-feeder/feed.xml`. This is the main source of new content.\n *Sources: [index.js:8](), [index.js:70]()*\n\n### 2. Data Aggregation and Deduplication\n\nAfter fetching, the script processes the feed items:\n- **Deduplication**: Items are filtered to ensure uniqueness based on their `guid`, `link`, or `title`.\n- **Grouping**: Unique items are grouped into a `postsMap` object. The key for this map is a combination of the company name (derived from the link) and the publication date (e.g., `company#YYYY-M-D`). This allows multiple news items about the same company on the same day to be grouped.\n- **Merging with Existing Content**: The script reads existing markdown files from the corresponding company directory (`source/_posts/{company}/`) to merge previously processed items with new ones, preventing data loss and creating a comprehensive list.\n\n*Sources: [index.js:73-77](), [index.js:81-122]()*\n\n### 3. Content Generation\n\nFor each group of items in `postsMap`, the script generates one or more markdown files.\n\n- **File Path**: A destination directory is determined by the `l()` function, which typically resolves to `source/_posts/{company_name}`.\n- **File Naming**: If there's only one item for a company on a given day, the file is named `{Dir}#{dateS}.md`. If there are multiple, they are indexed: `{Dir}#{dateS}#{idx}.md`.\n- **Front-matter**: Each markdown file is created with YAML front-matter containing essential metadata.\n\n```yaml\ntitle: ${Dir} fuite du ${dateS}\ndate: ${dateS}\nlien: \"${item.guid}\"\ncategories:\n - ${item.category1}\n - ${Dir}\n```\n*Sources: [index.js:212-217]()*\n\n- **Content Body**: The body of the post includes the cleaned content from the RSS item (`contentSnippet`). It also appends a list of links to other posts related to the same company (\"Autres fuites pour ce dossier\").\n*Sources: [index.js:189-205]()*\n\n## Final Feed Publication\n\nOnce the markdown files are generated in `source/_posts/`, the standard Hexo build process takes over.\n\n### Hexo Plugin\n\nThe `hexo-generator-feed` plugin, listed as a dependency in `package.json`, is responsible for generating the `atom.xml` file. It iterates through the site's posts (which were created by `index.js`) and builds the feed according to the settings in `themes/freemind/_config.yml`.\n\n| Dependency | Purpose |\n| :--- | :--- |\n| `rss-parser` | To parse the incoming external RSS feed. |\n| `hexo-generator-feed` | To generate the site's own `atom.xml` from Hexo posts. |\n\n*Sources: [package.json:25](), [package.json:32]()*\n\n### HTML Autodiscovery\n\nTo allow browsers and feed readers to automatically find the feed, a `` tag is inserted into the `` of the generated HTML pages. This is enabled by the `autodiscovery: true` setting.\n\n```html\n\n\n```\n*Sources: [public/index.html:14-15](), [themes/freemind/_config.yml:14]()*\n\n## Supporting Data Pipelines\n\nBesides the main `index.js` flow, other scripts contribute to the data that ultimately appears in the posts and the feed.\n\n### Breach Data Import (`scripts/import.js`)\n\nThis script is designed to import data from local markdown files located in `source/_posts/{directory}` into the central `source/_data/breaches.json` file. It reads the front-matter from `.md` files, merges it with a default schema, and appends it to the `breaches.json` array. This allows for manual or semi-automated additions to the breach database, which is then used by other parts of the site.\n\n*Sources: [scripts/import.js]()*\n\n### Breach Data Enrichment (`update_breach_info.js`)\n\nThis script enhances the `source/_data/breaches.json` file by scraping victim counts from `https://bonjourlafuite.eu.org/`. It normalizes breach names to find matches between the local JSON data and the scraped data, then updates the corresponding records with the victim count (`pwnCountFr`) and source URL. This enriched data can then be displayed on the generated pages.\n\n*Sources: [update_breach_info.js]()*\n\n## Conclusion\n\nThe RSS/Atom feed generation in this project is a sophisticated content aggregation pipeline. It leverages a custom Node.js script to consume and process an external RSS feed, enriches this data with information from other APIs and web scrapers, and programmatically generates markdown files. These files are then used by the Hexo static site generator and the `hexo-generator-feed` plugin to build the final website and its public-facing `atom.xml` feed, effectively syndicating the aggregated and processed information."} ```