RSS & Substack Collectors

Relevant source files

The following files were used as context for generating this wiki page:

• lib/collectors/rss.ts
• lib/collectors/substack.ts
• lib/communities/catalog.ts
• test/rss-community-bounds.test.ts
• test/rss-conditional-get.test.ts
• README.md

RSS and Substack Collectors ingest external content into the Vantage platform by parsing standard syndication formats. The RSS collector provides the core logic for fetching and normalizing XML-based feeds, while the Substack collector extends this functionality to handle publication-specific routing and platform metadata.

These collectors are part of the "free-lane" interface, enabling content acquisition from public feeds without requiring specialized scraping agents or browser-based extraction.

RSS Collector Architecture

The rssCollector implements the Collector interface and serves as the foundation for XML-based ingestion. It supports both RSS and Atom formats by mapping common XML tags to the internal NewDocument schema.

Core Logic and Execution Flow

The collector performs a series of validation and transformation steps:

1. URL Validation: Extracts and cleans the feedUrl from the source configuration.
2. Conditional GET: Includes If-None-Match (ETag) and If-Modified-Since headers if state exists from previous runs.
3. Response Bounding: Limits the total response size to prevent memory exhaustion, defaulting to 512 KB with a hard ceiling of 3 MB.
4. Parsing: Uses fast-xml-parser to transform the XML text into a JavaScript object.
5. Normalization: Iterates through the latest 50 entries and maps fields such as pubDate, guid, and content:encoded to standard document fields.

Sources: lib/collectors/rss.ts:51-68, test/rss-community-bounds.test.ts:15-18

XML to Document Mapping

The collector normalizes various XML structures into a consistent document format.

Vantage Field	XML Source Paths (Priority Order)
title	item.title, entry.title
urlCanonical	item.link, item.link.@_href, item.id, item.guid
contentMd	item.content:encoded, item.content, item.summary, item.description
postedAt	item.pubDate, item.published, item.updated, item.dc:date
authorRef	item.author, item.dc:creator, item.author.name

Sources: lib/collectors/rss.ts:80-108

Execution Flow Diagram

This diagram illustrates the process from configuration extraction to the final document normalization.

Substack Collector Logic

The substackCollector acts as a wrapper around the core RSS collector. It simplifies configuration by allowing users to provide a Substack subdomain or a direct publication URL, which it then converts into the standard Substack RSS feed path.

Substack Feed Resolution

If a publication or subdomain is provided in the configuration, the collector constructs the feed URL by appending /feed to the host. If the input does not contain a dot, it defaults to the substack.com domain.

Sources: lib/collectors/substack.ts:11-25

Platform-Specific Normalization

After the base RSS collector processes the feed, the Substack collector overrides specific fields:

• platform: Set to substack.
• contentHash: Re-calculated using the "substack" platform prefix.
• metadata.kind: Specifically marked as "post".

Sources: lib/collectors/substack.ts:38-51

Conditional GET and Efficiency

To minimize bandwidth and processing time, the collectors implement HTTP conditional requests.

The collector stores the etag and last-modified headers in the nextState object, which is passed back to the pipeline to be reused in the subsequent poll.

Sources: lib/collectors/rss.ts:63-78, test/rss-conditional-get.test.ts:12-28

Configuration Parameters

Collectors are configured via the SourceType and associated configuration objects, as defined in the community catalog.

Parameter	Type	Description	Default / Limit
feedUrl	string	The direct URL to the RSS/Atom/Substack feed.	Required
maxResponseBytes	number	Maximum size of the XML response to download.	512,000 (Max 3,000,000)
publication	string	(Substack only) Subdomain or full publication URL.	N/A
url	string	Alias for feedUrl.	N/A

Sources: lib/collectors/rss.ts:36-41, lib/collectors/substack.ts:14-20, lib/communities/catalog.ts:24-40

Implementation Details

Data Integrity and Security

• Hashing: Documents use a content hash generated from the platform, canonical URL, and markdown content to detect duplicates.
• URL Sanitization: Errors regarding feed URLs are sanitized to remove credentials (username/password) or sensitive query parameters before being thrown.
• Pagination: The current implementation processes the latest 50 entries per poll; historical imports require separate pagination logic.

Sources: lib/collectors/rss.ts:43-52, lib/collectors/rss.ts:109-112, test/rss-conditional-get.test.ts:35-45

Source Catalog Integration

RSS and Substack sources are integrated into the COMMUNITY_SOURCE_CATALOG as "free" lane sources. Examples include:

• RSS: DEV.to tags (AI, LLM), Lobsters, Hugging Face Blog.
• Substack: Latent Space, Ahead of AI, Import AI.

Sources: lib/communities/catalog.ts:24-45

Summary

The RSS and Substack collectors provide a robust, resource-efficient mechanism for ingesting public syndication feeds. By leveraging conditional GET and response bounding, the system ensures high-performance content acquisition while maintaining data normalization across diverse XML structures. The Substack collector further streamlines this process for newsletter-specific content through automatic feed resolution and platform tagging.