Utility Functions
Relevant source files
The following files were used as context for generating this wiki page:
The utils/index.js module provides a collection of shared, reusable helper functions that support various operations throughout the nodejs-news-feeder application. These utilities are primarily focused on data processing, content transformation, date formatting, and file system interactions, ensuring consistency and reducing code duplication across different scripts.
The scope of these functions ranges from low-level string sanitization to higher-level tasks like composing RSS feed items and managing configuration files. They form the core processing backbone for scripts that collect, format, and build the final news feed.
Core Concepts
The utility functions can be grouped into several logical categories based on their functionality.
String Manipulation and Sanitization
These functions are responsible for cleaning and formatting text content to ensure it is safe for XML and HTML rendering.
Sources: utils/index.js:19-95, utils/tests/utils.test.js:22-108
| Function | Description |
|---|---|
escapeXmlCharacters(text) | Escapes special XML characters (&, <, >, ", ') to their corresponding entities. |
stripHtmlTags(text) | Removes all HTML tags from a string, leaving only the inner text content. |
escapeHtmlTags(text) | Converts HTML tag brackets (<, >) into their entity equivalents (<, >). |
cleancontent(text, options) | A primary sanitization function that removes blacklisted keywords from a string. It reads a list of keys from blacklist.txt and removes all occurrences of them from the input text. |
The cleancontent function's behavior is illustrated by the following diagram:
This diagram shows the process of the cleancontent function, which reads a blacklist and iteratively removes forbidden keywords from the input text.
Sources: utils/index.js:68-95
Date and Time Formatting
These utilities handle the conversion of date strings into formats required by the RSS feed and item titles.
Sources: utils/index.js:97-103, 157-179, utils/tests/utils.test.js:12-20
| Function | Description |
|---|---|
buildTitleDate(timestamp) | Formats an ISO timestamp into a YYYY-MM-DD HH:MM:SS string, suitable for use in item titles. |
buildRFC822Date(dateString) | Converts a date string into the RFC 822 format (e.g., Mon, 29 Nov 2021 01:00:00 BST) required for the <pubDate> tag in RSS feeds. It uses a helper function addLeadingZero to pad day and time values. |
File and Configuration Handling
A set of simple functions for synchronous reading and writing to the project's main data and configuration files.
The diagram below shows the relationship between the file utility functions and the files they manage.
Content Transformation and Generation
These functions are central to the application's purpose of transforming collected data into feed items.
Sources: utils/index.js:15-17, 113-134
md2html(md)
This function converts a Markdown string into an HTML string. It uses the remark and remark-html libraries to perform the transformation. It is used by data collection scripts to process content from GitHub issues and discussions.
Sources: utils/index.js:15-17, scripts/collect-issues.js:25
composeFeedItem(itemObject)
This function constructs a single <item> block for the RSS feed from a given object. It orchestrates calls to cleancontent to sanitize the title and link before embedding them into the XML structure.
// Source: utils/index.js:113-134
export function composeFeedItem({ title, description, pubDate, link, guid,source }) {
// ...
const cleanTitle = cleancontent(title, {
stripHtml: true,
preserveLineBreaks: false,
normalizeWhitespace: true
})
const cleanDescription = cleancontent(description, {
stripHtml: true,
preserveLineBreaks: false,
normalizeWhitespace: true
})
return `
<item>
<title>${cleanTitle}</title>
<description>${cleanDescription}</description>
<pubDate>${pubDate}</pubDate>
<link>${cleancontent(link,{})}</link>
<guid>${guid}</guid>
</item>
`
}
The workflow for this function is as follows:
- Accepts an object with properties like
title,description,pubDate,link, andguid. - Cleans the
titleanddescriptionusingcleancontent. - Cleans the
link. - Constructs and returns an XML string for the
<item>.
Sources: utils/index.js:113-134, scripts/format.js:29-37
Feed Filtering
The filterFeedItems function is used to process an array of feed items, removing duplicates and filtering by keywords or categories.
The logic is as follows:
- It ensures essential fields (
title,link,pubDate) are present. - It filters items based on optional
keywordsandcategoriesarrays. - It removes duplicates by tracking a unique key for each item, which can be its
guid,link, ortitle. ASetis used for efficient lookup of seen items.
Sources: utils/index.js:200-215, scripts/format.js:20-25
Usage in Scripts
The utility functions are imported and used extensively by the scripts in the scripts/ directory to perform the main logic of the application.
The following sequence diagram illustrates how scripts/collect-issues.js uses the utility functions to fetch data from GitHub, process it, and update the local feed.xml file.
This diagram shows the interaction between a collection script and the utility module to read configuration, process fetched data, and write the results back to the file system.
Sources: scripts/collect-issues.js, utils/index.js
Conclusion
The utility functions in utils/index.js are fundamental to the nodejs-news-feeder project. They provide a robust and centralized toolkit for handling common data manipulation and file I/O tasks. By abstracting these operations, the utilities enable the data collection and feed generation scripts to focus on their primary logic, leading to a cleaner, more maintainable, and consistent codebase.