Web Scraping & Crawlers
Relevant source files
The following files were used as context for generating this wiki page:
The Web Scraping and Crawler systems in ZINC-Fusion-V15 serve as the primary ingestion engine for alternative data and news intelligence. The system is designed to monitor over 25 agricultural, trade, and policy sources, ranging from structured RSS feeds and APIs to unstructured web content requiring headless browser automation.
These crawlers gather high-fidelity data to support the project's "Big-11" specialist models, which generate signals for commodity procurement forecasting. By monitoring sources like Pro Farmer, the White House, and various agricultural agencies, the system ensures that the ML pipeline has access to real-time market-moving events and policy shifts.
Sources: scripts/ingest_news_sources.py:11-35, AGENTS.md:73-86
System Architecture
The scraping architecture is bifurcated into two main implementation strategies: high-frequency Python-based ingestion and deep-crawl Node.js/Puppeteer scripts.
Core Ingestion Pipeline (Python)
The primary production pipeline is a server-scheduled Python job (ingest_news_sources.py). It uses a multi-priority approach to fetch data based on the urgency of the source.
- P0 Critical: Every 2 hours (e.g., Farm Policy News, USDA Press).
- P1 High: Every 4 hours (e.g., Federal Reserve, White House).
- P2 Medium: Every 6 hours (e.g., MPOB Malaysia, CBOE Insights).
Sources: scripts/ingest_news_sources.py:15-30
Headless Browser Automation (Node.js)
For complex sources requiring authentication or dynamic content rendering (specifically Pro Farmer), the system utilizes Puppeteer. These scripts handle login sessions, cookie management, and deep pagination through sitemaps and topic archives.
Sources: scripts/_deprecated/scrape_profarmer_sitemap_comprehensive.js:18-50, scripts/_deprecated/scrape_full_metadata.js:20-42
Data Flow Diagram
The following diagram illustrates how raw web data is transformed into structured specialist signals.
This diagram represents the flow from various external endpoints through specialized scrapers into the alt database schema.
Sources: scripts/ingest_news_sources.py:145-175, frontend/src/inngest/whitehouse-press.ts:311-370
Implementation Details
Specialist Routing & Classification
A critical component of the scraping logic is the classification of articles into one of the 11 specialist "buckets." This is handled through rule-based keyword matching and source-specific mapping.
| Specialist Bucket | Signal Focus | Key Scraped Sources |
|---|---|---|
| crush | Soybean Fundamentals | FarmDoc Daily, USDA, AgWeb |
| china | Trade Flows | Reuters China, MOFCOM |
| tariff | Trade Policy | USTR, Federal Register |
| energy | Crude/Energy Complex | EIA Petroleum, EIA Today |
| trump_effect | Policy Volatility | Truth Social, Executive Orders |
| palm | Substitution Pressures | MPOB Malaysia, GAPKI Indonesia |
Sources: scripts/ingest_news_sources.py:53-140, AGENTS.md:73-86
Authentication and Session Management
For Pro Farmer scraping, the system implements a robust keyboard-driven login simulation to bypass standard bot detection.
// Example from scripts/_deprecated/scrape_profarmer_sitemap_comprehensive.js
await page.goto('https://www.profarmer.com/r/sign-in', { waitUntil: 'networkidle2' });
await page.evaluate(() => {
const email = document.querySelector('input[type="email"]');
if (email) email.focus();
});
await page.keyboard.type(user, { delay: 80 });
await page.keyboard.press('Tab');
await page.keyboard.type(pass, { delay: 80 });
await page.keyboard.press('Enter');
Sources: scripts/_deprecated/scrape_profarmer_sitemap_comprehensive.js:34-45
White House Ingestion System
The White House ingestion system is implemented as an Inngest serverless function (whitehouse-press.ts). It monitors over 20 targeted URLs categorized by document type.
Document Type Inference
The system automatically classifies documents based on URL patterns and content:
- executive_order: Presidential actions and EO paths.
- proclamation: Proclamation headers and link segments.
- fact_sheet: Press briefing fact sheets.
- nomination_appointment: Personnel announcements.
Sequence of White House Scraping
The Inngest sequence ensures that critical policy documents are categorized and persisted for specialist model consumption.
Sources: frontend/src/inngest/whitehouse-press.ts:251-370
Data Persistence & Deduplication
To prevent redundant processing, every scraped item generates a row_hash (SHA-256) based on its primary content (usually Title + Link or Title + Content).
Storage Schemas
The system routes data into the alt schema across four primary event tables:
alt.policy_news_event: Default for trade and general policy.alt.executive_actions_event: Presidential orders and memoranda.alt.econ_news_event: Macro and central bank news.alt.profarmer_news_event: Specialized agricultural intelligence.
Sources: scripts/ingest_news_sources.py:180-215, AGENTS.md:33-38
Metadata Extraction logic
Comprehensive metadata extraction targets the following structured data formats:
- JSON-LD: Extracting
datePublished,headline, andauthor. - OpenGraph: Fetching
og:titleandog:description. - Schema.org: Parsing
@graphobjects forNewsArticletypes. - Breadcrumbs: Identifying
article:sectionand category tags.
Sources: scripts/_deprecated/scrape_full_metadata.js:80-150, scripts/_deprecated/scrape_profarmer_sitemap_comprehensive.js:100-140
Summary
The Web Scraping and Crawler framework in ZINC-Fusion-V15 is a sophisticated multi-language system designed for maximum reliability and data depth. By combining Python's efficiency for RSS/API sources with Node.js/Puppeteer's ability to handle complex authenticated sessions, the system creates a comprehensive "knowledge layer" of alternative data. Every piece of ingested information is rigorously classified into specialist buckets, deduplicated via cryptographic hashes, and stored in a structured relational format to drive the downstream forecasting models.