Using the Blacklist

The blacklist is a content filtering mechanism used within the nodejs-news-feeder project. Its primary purpose is to sanitize the text content of news items before they are published to the RSS feed (feed.xml) and the corresponding website (index.html). This is achieved by removing specific, predefined strings from the title, description, and link of each feed item.

The entire process is managed through a simple text file and a set of utility functions. The blacklist.txt file serves as the single source of truth for all strings that need to be removed. Utility functions read this file, process the content, and ensure that the final output is clean. This system is integral to the data collection scripts that generate the feed items.

Architecture and Data Flow

The blacklist functionality is centered around the cleancontent() function in utils/index.js. This function is called during the creation of each RSS feed item to ensure that unwanted strings are stripped out.

The general data flow is as follows:

1. A list of keywords to be removed is maintained in the blacklist.txt file.
2. The getkeys() utility function reads blacklist.txt and provides an array of these keywords.
3. When a data collection script (e.g., collect-retrospective.js) runs, it calls composeFeedItem() to create a new news entry.
4. composeFeedItem() in turn calls cleancontent() to sanitize the title, description, and link fields.
5. The cleancontent() function uses the keywords from getkeys() to remove all occurrences of those strings from the provided text.
6. The cleaned content is then used to build the final XML for the RSS feed.

This diagram illustrates the process of cleaning content for an RSS item.

Core Components

The blacklist system is composed of a configuration file and several key JavaScript functions.

The blacklist.txt File

This file is the data source for the blacklist. It is a simple text file where each line contains a string that should be removed from the feed content. The getkeys() function reads this file, trims whitespace from each line, and handles carriage returns (\r) to ensure cross-platform compatibility. If the file does not exist, it is created automatically.

Sources: blacklist.txt, utils/index.js:88-95

The getkeys() Function

Located in utils/index.js, this function is responsible for reading and parsing the blacklist.txt file.

• Path: utils/index.js
• Logic:
  1. Checks if blacklist.txt exists in the project's root directory.
  2. If it exists, it reads the file content.
  3. The content is split into an array of strings by newlines (\n).
  4. Each string (key) is trimmed of whitespace and has carriage return characters removed.
  5. If the file does not exist, it is created as an empty file, and an empty array is returned.

// utils/index.js:88-95
const getkeys = () => {
  if(existsSync(join(process.cwd(), "./blacklist.txt"))){
    return readFileSync(join(process.cwd(), "./blacklist.txt"), "utf-8").split("\n").map((key) => key.trim().replace(/\r/g, ""))
  }else{
    writeFileSync(join(process.cwd(), "./blacklist.txt"), "")
    return []
  }
}

Sources: utils/index.js:88-95

The cleancontent() Function

This is the main function that performs the content sanitization. It takes a text string as input and removes all occurrences of the blacklisted keywords.

• Path: utils/index.js
• Logic:
  1. It receives a text argument to be cleaned.
  2. It calls getkeys() to retrieve the list of blacklisted strings.
  3. It iterates through each key in the blacklist.
  4. For each key, it enters a while loop that continues as long as the key is found in the text.
  5. Inside the loop, it replaces the first occurrence of the key with an empty string. The loop ensures that all occurrences are removed, not just the first one.
  6. After checking all keys, it returns the modified validtext.

// utils/index.js:103-115
export function cleancontent(text, options = {}) {
  let validtext=text
  const keys=getkeys()
  //console.log(keys)
  for(const key of keys){
    
    while (validtext.includes(key)){
      //console.log(validtext.includes(key))  
      validtext=validtext.replace(key,"")
    }
  }

  return validtext
}

Sources: utils/index.js:103-115

Integration in composeFeedItem()

The cleancontent() function is applied within composeFeedItem(), which is responsible for building each <item> block for the feed.xml. It is specifically used to clean the title, description, and link of the item.

The following sequence diagram shows how a data collection script triggers the cleaning process.

Summary of Key Functions

This table summarizes the functions involved in the blacklist process.

Sources: utils/index.js:88-147

Conclusion

The blacklist feature provides a straightforward and maintainable way to filter unwanted content from the news feed. By centralizing the keywords in blacklist.txt, it allows for easy updates without changing any application code. The cleancontent function ensures that all specified strings are thoroughly removed from the key fields of each news item, maintaining the quality and relevance of the content presented in both the feed.xml and the index.html website.