Data Flow and Architecture

Relevant source files

The following files were used as context for generating this wiki page:

• PLAYLIST_EXTRACTION_GUIDE.md
• extract_playlist.py
• merge_playlist_data.py
• redesign/script.js
• redesign/videos.js
• redesign/jokes.html
• redesign/videos.html

The Norm Macdonald Archive is a static website that relies on a pre-processing pipeline to gather and structure its content. The architecture separates data collection and processing from the frontend presentation layer. Data, primarily from YouTube and manually curated jokes, is extracted, consolidated into JSON files, and then consumed by the client-side JavaScript to dynamically render the content on static HTML pages.

This architecture ensures the main website is fast and simple to host, as it consists only of static assets (HTML, CSS, JS, JSON). All complex data processing is handled offline through a series of scripts, which generate the final data files used by the site.

1. Data Sources and Schemas

The application's content is driven by two primary JSON files: jokes-data.json and consolidated_youtube_data.json. These files serve as the database for the static site.

1.1. Joke Data

The jokes-data.json file contains an array of joke objects. This data is loaded by the "Jokes" page to display, filter, and search through Norm's jokes.

Sources: redesign/script.js:70-75, redesign/jokes.html

Schema: jokes-data.json

Field	Type	Description	Example
joke	String	The text of the joke. Can include newline characters for formatting.	"I think there's nothing cooler..."
guest	String	The guest present when the joke was told. Can be null.	"Adam Sandler"
episode	String	The episode identifier.	"S03E12"
time	String	The timestamp in the video where the joke occurs.	"1:02:34"
url	String	The YouTube video ID for the clip.	"p2Up2I388A0"

1.2. Video Data

The consolidated_youtube_data.json file contains an array of video metadata objects. It is the data source for the "Videos" page.

Sources: redesign/videos.js:51-64, redesign/videos.html

Schema: consolidated_youtube_data.json

Field	Type	Description	Example
Video url	String	The full URL of the YouTube video.	"https://www.youtube.com/watch?v=..."
Title	String	The title of the video.	"Norm Macdonald on Conan"
Description	String	The video's description.	"Video url:"
Channel name	String	The name of the YouTube channel.	"N/A"
Duration	String	The duration of the video.	"N/A"
Views	String	The view count.	"N/A"
Thumbnail url	String	The URL for the video's thumbnail image.	"N/A"

Note: Several fields in the video data schema are marked as "N/A", indicating they are part of the structure but not always populated by the extraction scripts.

Sources: PLAYLIST_EXTRACTION_GUIDE.md:30-36

2. Data Extraction and Consolidation Workflow

New video data is added to the archive through a multi-step, offline process. This process involves extracting video metadata from a YouTube playlist and merging it into the primary consolidated_youtube_data.json file.

Sources: PLAYLIST_EXTRACTION_GUIDE.md, extract_playlist.py, merge_playlist_data.py

The following diagram illustrates the overall workflow for updating video data.

2.1. Playlist Extraction

The first step is to extract video metadata from a given YouTube playlist. The PLAYLIST_EXTRACTION_GUIDE.md outlines three methods to accomplish this, all of which produce a playlist_videos.json file.

1. Browser Console Script: A JavaScript snippet is run directly in the browser console on a YouTube playlist page. It scrapes the DOM for video renderers (ytd-playlist-video-renderer) to extract titles and video IDs.
2. Python Script (extract_playlist.py): A more reliable method using the pytube library. The script iterates through a playlist URL, extracts video information, and saves it as playlist_videos.json.
3. Browser Tool (extract_playlist.html): A user-friendly HTML tool that guides the user through running a script and pasting the output to generate the JSON file.

Sources: PLAYLIST_EXTRACTION_GUIDE.md:10-53, extract_playlist.py

2.2. Data Merging

Once playlist_videos.json is created, the merge_playlist_data.py script is used to consolidate the new data.

The merging process is as follows:

1. Load the existing consolidated_youtube_data.json.
2. Create a set of existing video URLs for efficient duplicate checking.
3. Load the newly extracted playlist_videos.json.
4. Iterate through the new videos. If a video's URL is not in the set of existing URLs, it is added to the main data list.
5. Save the updated list back to consolidated_youtube_data.json.

Sources: merge_playlist_data.py:10-39

The following diagram shows the logic within merge_playlist_data.py.

3. Frontend Architecture

The frontend is a pure HTML, CSS, and JavaScript application with no frameworks. Each major content section (Jokes, Videos) has its own HTML page and a corresponding JavaScript file that fetches data and renders the UI.

Sources: redesign/README.md:30-34, redesign/script.js, redesign/videos.js

3.1. Client-Side Data Loading

On page load, the relevant JavaScript file initiates a fetch request to its corresponding JSON data file. Once the data is loaded, it's stored in a global state object, and the initial UI is rendered.

The following sequence diagram shows the data loading and rendering process for the "Jokes" page.

A similar flow occurs on the "Videos" page, but it uses videos.js to fetch consolidated_youtube_data.json.

Sources: redesign/videos.js:46-68, 281-286

3.2. State Management

Both redesign/script.js (for jokes) and redesign/videos.js (for videos) use a simple global state object to manage the application's state. This object holds the master list of items, the currently filtered list, pagination details, and user input like search terms.

Example state object from redesign/script.js:

const state = {
    jokes: [],
    filteredJokes: [],
    currentPage: 1,
    jokesPerPage: 10,
    searchTerm: '',
    selectedGuest: '',
    currentRandomJoke: null
};

Sources: redesign/script.js:5-13, redesign/videos.js:5-11

All user interactions, such as searching or filtering, call an applyFilters() function. This function updates the filteredJokes (or filteredVideos) array in the state object and then calls renderJokes() (or renderVideos()) to re-render the UI based on the new state. This creates a unidirectional data flow, where the UI is always a reflection of the current state.

Sources: redesign/script.js:93-106, redesign/videos.js:92-103