Legacy Data Extraction Scripts

Relevant source files

The following files were used as context for generating this wiki page:

• old/site-src/extract_pdf_text.py
• old/site-src/parse_csv_data.py
• old/site-src/parse_html_bookmarks.py
• old/site-src/parse_text_data.py
• PLAYLIST_EXTRACTION_GUIDE.md
• old/site-src/TODO.txt
• redesign/videos.js

Introduction

The Legacy Data Extraction Scripts are a collection of Python utilities and guides located in the old/site-src/ directory and project root. Their primary purpose was to perform one-time data ingestion from various unstructured and semi-structured sources like text files, PDFs, CSVs, HTML bookmarks, and YouTube. This process was crucial for bootstrapping the content for The Norm Macdonald Archive, populating it with jokes, video metadata, and transcripts.

As indicated by their location and the TODO.txt file, these scripts represent a completed, foundational task of content aggregation. The data they produced, particularly the consolidated_youtube_data.json, is now a static asset consumed by the frontend application. This document serves as an architectural reference for how the initial data was collected and processed.

Sources: old/site-src/TODO.txt, redesign/videos.js:65-83, PLAYLIST_EXTRACTION_GUIDE.md

System Overview

The extraction system is not a single, cohesive application but a set of independent scripts. Each script is designed to handle a specific file format. The general workflow involves running a script against a source file, which then prints a JSON-structured string to standard output. This output was likely redirected to files that were later consolidated into the final data assets used by the website.

The following diagram illustrates the high-level data flow from various sources to the final consolidated data file.

Script Summary

Script/Guide	Purpose	Input Format	Output Format
PLAYLIST_EXTRACTION_GUIDE.md	Extracts video metadata from a YouTube playlist.	YouTube Playlist	playlist_videos.json
parse_text_data.py	Parses structured text files to extract jokes.	.txt	JSON string (list of joke objects)
parse_csv_data.py	Converts CSV data into a JSON array.	.csv	JSON string (list of row objects)
parse_html_bookmarks.py	Parses browser bookmark files to extract links.	.html	JSON string (list of link objects)
extract_pdf_text.py	Extracts all raw text from a PDF document.	.pdf	Raw text string

Sources: PLAYLIST_EXTRACTION_GUIDE.md, old/site-src/parse_text_data.py, old/site-src/parse_csv_data.py, old/site-src/parse_html_bookmarks.py, old/site-src/extract_pdf_text.py

Detailed Processes

YouTube Playlist Extraction

This process is documented in PLAYLIST_EXTRACTION_GUIDE.md and was designed to overcome network restrictions in a development environment. It provides three methods to extract video metadata from a YouTube playlist.

The primary goal is to generate a playlist_videos.json file, which is then merged into the main consolidated_youtube_data.json by a merge_playlist_data.py script (not provided in the source files, but mentioned in the guide).

Sources: PLAYLIST_EXTRACTION_GUIDE.md:3-5, 80-86

The following flowchart shows the different paths a user can take to extract the playlist data.

Output JSON Structure

The browser console script generates a JSON array of objects. Most fields are placeholders, intended to be populated later or ignored.

Key	Type	Description	Example
Video url	String	The full URL to the YouTube video.	https://www.youtube.com/watch?v=VIDEO_ID
Title	String	The title of the video.	"Video Title Here"
Description	String	Placeholder text.	"Video url:"
Channel name	String	Placeholder text.	"N/A"
Duration	String	Placeholder text.	"N/A"
Views	String	Placeholder text.	"N/A"
Thumbnail url	String	Placeholder text.	"N/A"

Source: PLAYLIST_EXTRACTION_GUIDE.md:23-33, 114-125

Text File Parsing (parse_text_data.py)

This script is designed to parse .txt files that contain jokes formatted with specific metadata keys. It uses a simple delimiter (---) to separate individual joke entries.

Logic

1. The script reads the entire content of the input text file.
2. It splits the content into blocks using --- as the separator.
3. For each block, it uses regular expressions to find and extract Guest:, Episode:, and Timestamp:.
4. Any text not matching these patterns is treated as the joke content.
5. It constructs a list of dictionaries, with each dictionary representing a joke.
6. The final list is printed to stdout as a JSON string.

Source: old/site-src/parse_text_data.py:5-24

Output Data Structure

[
  {
    "joke": "The joke content itself...",
    "guest": "Guest Name",
    "episode": "Episode Title or Number",
    "timestamp": "00:00:00"
  }
]

Source: old/site-src/parse_text_data.py:20-23

CSV Data Parsing (parse_csv_data.py)

This is a generic utility to convert any CSV file with a header row into a JSON array of objects.

Logic

1. The script opens the specified CSV file.
2. It uses Python's csv.DictReader, which automatically uses the first row as keys for a dictionary for each subsequent row.
3. It iterates through the reader, creating a list of these dictionaries.
4. The final list is printed to stdout as a JSON string.

Source: old/site-src/parse_csv_data.py:5-10

HTML Bookmark Parsing (parse_html_bookmarks.py)

This script extracts links and their titles from a standard HTML bookmark export file.

Logic

1. It requires the BeautifulSoup4 library.
2. It reads the HTML file content.
3. It creates a BeautifulSoup object to parse the HTML.
4. It finds all anchor (<a>) tags within the document.
5. For each <a> tag, it extracts the href attribute (the URL) and the tag's text content (the title).
6. It compiles this data into a list of dictionaries and prints it to stdout as a JSON string.

Source: old/site-src/parse_html_bookmarks.py:6-15

PDF Text Extraction (extract_pdf_text.py)

This is the most basic of the parsers, designed for raw text extraction from PDF files.

Logic

1. It requires the PyPDF2 library.
2. It opens the PDF file in binary-read mode.
3. It creates a PdfReader object.
   4en. It iterates through each page of the PDF, extracts the text using .extract_text(), and appends it to a single string.
4. The final concatenated string of text from all pages is printed to stdout.

This script does not perform any structuring; it only dumps the raw text content.

Source: old/site-src/extract_pdf_text.py:5-13

Conclusion

The legacy data extraction scripts represent a pragmatic approach to content aggregation for a static-site project. They are a collection of specialized, single-purpose tools that transform data from various formats into a more usable, structured JSON format. While simple, they were effective in populating the initial dataset for the Norm Macdonald Archive, a task marked as "COMPLETED" in project documentation. The existence of these scripts provides valuable insight into the project's origins and the provenance of its core data.

Sources: old/site-src/TODO.txt:5