Ingestion & Web Archival
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Ingestion and Web Archival represent the initial phases of the IDAHO-VAULT journalism operating system. The primary purpose of this system is to facilitate the rapid, structured intake of real-world information—ranging from legislative data to news articles—into a local-first environment for subsequent analysis and synthesis. By utilizing both automated scrapers and manual inputs, the ingestion layer ensures that external data is captured with high fidelity and linked to its original provenance.
The archival component focuses on preserving the integrity of these sources through web archival tools like the Wayback Machine. This ensures that the evidence supporting journalistic claims remains recoverable even if the original source URLs become dead or modified. This stage is metadata-driven and manifest-coordinated, ensuring that every document's transition from raw input to a structured vault record is traceable and human-verified.
Sources: IDAHO-VAULT — WORKFLOW.md:9-25, IDAHO-VAULT — END-TO-END JOURNALISM WORKFLOW (AI-ASSISTED, LOCAL-FIRST)v0.1.md:15-32
Ingestion Architecture & Logic
Data Intake Layers
The ingestion process is divided into logical stages that transform information from real-world input into structured vault units. The system operates on a single unit of work (one document, record, or note) at a time to ensure reliable transformation.
Stage 1: Capture
The goal of the Capture stage is to bring external information into the Vault. Inputs include legislative websites, PDFs (bills, fiscal notes), court records, RSS feeds, and press releases.
Key Characteristics:
- Location: Initial files are written to
/INBOX/. - Resulting Metadata:
status: raw. - Content: Raw content or a reference link, source, and minimal metadata.
Sources: IDAHO-VAULT — WORKFLOW.md:67-85, IDAHO-VAULT — END-TO-END JOURNALISM WORKFLOW (AI-ASSISTED, LOCAL-FIRST)v0.1.md:62-75
Stage 2: Structure
The Structure stage makes raw data usable by normalizing formatting and identifying key fields through automated agents.
Key Actions:
- Extract text from PDFs.
- Clean formatting.
- Identify key entities (e.g., bill numbers, sponsors, fiscal impact).
- Location: Files move to
/PROCESSED/. - Resulting Metadata:
status: structured.
Sources: IDAHO-VAULT — WORKFLOW.md:87-104, IDAHO-VAULT — END-TO-END JOURNALISM WORKFLOW (AI-ASSISTED, LOCAL-FIRST)v0.1.md:91-105
Ingest Methods & Automation
Automated Scrapers and Scripts
The vault employs several specialized Python scripts and GitHub Action workflows to maintain the ingestion pipeline.
| Script | Purpose | Trigger |
|---|---|---|
idaho_leg_scraper.py | Scrapes Idaho Legislature bill data | Daily 6 AM MT |
post_digest.py | Posts bill activity to GitHub Issues digest | Scraper workflow call |
wayback_audit.py | Audits URL preservation in Wayback Machine | Weekly Monday 8 AM UTC |
wayback-preserve.yml | Archives new URLs to Wayback Machine | Push to main branch |
Sources: VAULT-CONVENTIONS.from-pushable-2026-05-08.md:483-500, IDAHO-VAULT — END-TO-END JOURNALISM WORKFLOW (AI-ASSISTED, LOCAL-FIRST)v0.1.md:62-68
LSO Minidata Ingest
A specific intake contract exists for Idaho Legislature bill data provided via the Legislative Services Office (LSO). This data arrives as a daily email attachment (minidata.csv) and is treated as an authoritative session snapshot.
Ingest Logic:
- Encoding:
cp1252(required overutf-8). - Key Fields:
bill_id,title,status,vote_result. - Conflict Resolution: The ingest script updates
last_actionandstatusfields without destroying richer existing metadata like sponsors or committee history.
Sources: HANDOFF-LSO-MINIDATA-INGEST-2026-03-24.md:14-25, HANDOFF-LSO-MINIDATA-INGEST-2026-03-24.md:58-75
Web Archival & URL Preservation
Web archival is critical for maintaining "Truth Residency" within the vault. Automated workflows audit and archive URLs found in SOURCES, GOVERNMENTS, and TOPICS directories.
Coordination via Manifest
The manifest.json serves as the coordination layer for the ingestion workflow. Agents must read the manifest before acting and update it afterward to reflect the document's current processing state.
Manifest Roles:
- Tracks file identity and current status.
- Identifies the last agent to process the unit.
- Prevents duplicate processing or skipped stages.
Sources: IDAHO-VAULT — WORKFLOW.md:58-65, IDAHO-VAULT — END-TO-END JOURNALISM WORKFLOW (AI-ASSISTED, LOCAL-FIRST)v0.1.md:175-188
Metadata Standards for Ingested Records
Every record ingested into the vault must adhere to specific YAML frontmatter standards to ensure lifecycle traceability.
---
id: unique-id
type: document | bill | note
status: raw | structured | interpreted | verified
source: url or origin
created_at: timestamp
updated_at: timestamp
authority: "LOGAN"
---
Sources: IDAHO-VAULT — WORKFLOW.md:48-56, VAULT-METADATA-STANDARD.md:42-53
Summary
The Ingestion & Web Archival system within IDAHO-VAULT provides a structured gateway for real-world information. By transitioning data through standardized logical stages—Capture and Structure—and ensuring its permanence via Wayback Machine archival, the project maintains a traceable and verifiable record of truth. The reliance on a coordination manifest and strict metadata standards ensures that even as the project scales, the provenance and status of every journalistic unit remain clear.