Twitter Extraction & Archival
Relevant source files
The following files were used as context for generating this wiki page:
Twitter Extraction & Archival is a specialized module within the IDAHO-VAULT ecosystem designed to convert raw Twitter/X data exports into structured, linkable Markdown notes. This process ensures that social media contributions and historical context are preserved locally, adhering to the project's CORE MODEL of moving data from raw ingestion to structured insight.
The system utilizes a Python-based extraction pipeline that processes tweets.js files from a Twitter archive ZIP, applying sanitization, metadata extraction, and deduplication logic. The resulting files are integrated into the vault's tweets/ directory, organized by year, and enriched with YAML frontmatter to support the vault's knowledge graph.
Sources: twitter-extract.py:1-12, IDAHO-VAULT — END-TO-END JOURNALISM WORKFLOW (AI-ASSISTED, LOCAL-FIRST)v0.1.md:27-42
Extraction Pipeline Architecture
The extraction process follows a structured functional flow, beginning with targeted metadata extraction from a Twitter ZIP archive and ending with the creation of individual Markdown files.
Workflow Logic
The pipeline is designed to be resource-efficient by performing targeted extraction of metadata files (tweets.js and account.js) into a temporary directory before parsing. It employs a deduplication mechanism by scanning the existing OUTPUT_DIR for previously extracted tweet_id values to prevent redundant file creation.
The diagram shows the sequence of operations within the extraction script, emphasizing the deduplication and organizational logic.
Sources: twitter-extract.py:53-62, twitter-extract.py:68-78, twitter-extract.py:192-225
Key Functions and Components
| Component | Responsibility | Technical Implementation |
|---|---|---|
load_twitter_js | Parses .js archive files | Strips window.YTD prefix and uses json.loads |
get_existing_ids | Deduplication indexer | Scans .md files in output dir via regex |
sanitize_filename | Path safety | Removes URLs, mentions, and illegal NTFS/APFS chars |
build_tweet_markdown | Note generation | Assembles YAML frontmatter and Markdown body |
Sources: twitter-extract.py:30-37, twitter-extract.py:40-51, twitter-extract.py:90-108, twitter-extract.py:111-180
Data Structure and Sanitization
The extraction process transforms raw JSON objects into structured Markdown. This involves mapping Twitter entities to specific YAML fields and sanitizing text for both file system compatibility and vault readability.
YAML Frontmatter Schema
Each archived tweet includes standardized metadata used for vault linking and authority tracking.
---
date: YYYY-MM-DD
time: HH:MM:SS
tweet_id: "numeric_string"
type: retweet | reply | original
source: "App Name"
retweets: count
favorites: count
lang: en
authority: LOGAN
reply_to: "@username" (optional)
mentions: [list]
hashtags: [list]
related: [Twitter, lfinneytweets, hashtags...]
---
Sources: twitter-extract.py:138-164
Sanitization and Path Safety
The system adheres to the NETWEB Path Standard by cleaning filenames.
- Replacements: URLs are removed;
@mentionsare stripped from the start of replies to create cleaner slugs. - Illegal Characters: Characters like
< > : " / \ | ? *are replaced with spaces to ensure compatibility with Windows (NTFS) and macOS (APFS). - Truncation: Filenames are truncated to a maximum of 60 characters, ensuring the total path length stays within the 218-character repository limit.
Sources: twitter-extract.py:90-108, VAULT-CONVENTIONS.from-pushable-2026-05-08.md:467-486
Integration and Governance
Twitter data archival is governed by the project's constitutional principles regarding human-authored vs. machine-generated content and data privacy.
Constitutional Alignment
- Attribution: The Python extraction script is classified as "machine/procedural" product attributable to Claude, while the resulting Markdown files represent the "human product" (historical social record) attributable to Logan.
- Privacy: Archives must not contain raw personal data, secrets, or raw metadata from non-public services unless cleared for public-persistent storage.
- Directory Structure: Twitter notes are stored in
tweets/at the vault root, following the convention that root-heavy layouts are intentional and root-flat notes are a deliberate choice.
Sources: CONSTITUTION 1.md:15-19, PRIVACY.md:37-60, VAULT-CONVENTIONS.from-pushable-2026-05-08.md:65-75
Relationship Diagram
This diagram illustrates the relationship between individual tweet notes, their extracted entities, and the hierarchical storage structure within the vault.
Sources: twitter-extract.py:11-12, twitter-extract.py:228-233
Summary
Twitter Extraction & Archival provides a reliable bridge between transient social media activity and the permanent IDAHO-VAULT record. By automating the conversion of Twitter archives into deduplicated, sanitized Markdown notes, the system ensures that social context remains traceable, searchable, and compatible across different operating environments while maintaining the strict metadata standards required for journalistic integrity.