Extracting YouTube Playlist Data

Relevant source files

The following files were used as context for generating this wiki page:

• PLAYLIST_EXTRACTION_GUIDE.md
• extract_playlist.py
• extract_playlist.html
• merge_playlist_data.py
• videos.js
• videos.html

The YouTube Playlist Extraction system provides a set of tools and a defined workflow for fetching video metadata from a specific YouTube playlist and integrating it into the main data source for the Norm Macdonald Archive website. This process is essential for keeping the video archive up-to-date, particularly because the development environment has network restrictions that prevent direct API calls to YouTube.

The overall process involves three main stages: extracting basic video data into a playlist_videos.json file, merging this new data with the existing consolidated_youtube_data.json while handling duplicates, and finally, consuming the consolidated data on the website's video page.

Overall Data Flow

The following diagram illustrates the high-level workflow from data extraction to its final consumption by the website's front end.

This diagram shows the three phases: extracting data into playlist_videos.json using one of three methods, merging it into the main data file, and the website's JavaScript consuming the final result.
Sources: PLAYLIST_EXTRACTION_GUIDE.md, merge_playlist_data.py, videos.js

Extraction Methods

There are three distinct methods available to extract the initial playlist data. Each method produces the same output file, playlist_videos.json, but offers a different trade-off between ease of use and reliability.

Method	Description	Requirements	Key Benefit
Browser Console	A JavaScript snippet is pasted directly into the browser's developer console on the YouTube playlist page.	Web browser only	Easiest, no installation required.
Python Script	A Python script (extract_playlist.py) is run from the command line.	Python 3, pytube library	Most reliable and automated.
Browser Tool	A local HTML file (extract_playlist.html) provides a user-friendly interface to guide the process.	Web browser only	User-friendly with validation.
Sources: PLAYLIST_EXTRACTION_GUIDE.md:9-45

Method 1: Browser Console Script

This is the quickest method. The user navigates to the YouTube playlist, scrolls to load all videos, opens the developer console, and pastes in a provided JavaScript snippet. The script scrapes the DOM for video information and copies the resulting JSON to the clipboard.

The script iterates through all ytd-playlist-video-renderer elements, extracting the video title and URL. It constructs a JSON object for each video with a predefined structure.

This diagram shows the user interaction flow for the browser console method.

Method 2: Python Script (extract_playlist.py)

This is the most reliable method. It uses the pytube library to fetch playlist information directly from YouTube. The script handles fetching the playlist, iterating through each video, and saving the extracted data to playlist_videos.json.

This flowchart shows the logic of the extract_playlist.py script.

The script defaults to the playlist https://youtube.com/playlist?list=PLxTIDernmMHuIsNqSMOhKR1jlWimaLklk but can accept a different URL as a command-line argument.
Sources: extract_playlist.py, PLAYLIST_EXTRACTION_GUIDE.md:37-45

Method 3: Browser Tool (extract_playlist.html)

This method provides a user-friendly graphical interface via a local HTML file. The page contains instructions, the same JavaScript snippet from Method 1, and a text area to paste the JSON output. It includes client-side validation to ensure the pasted data is a valid JSON array of video objects before allowing the user to download the playlist_videos.json file.

The processJSON function in the HTML file is responsible for validation and file creation.

This sequence diagram illustrates the user flow for the browser tool.

Extracted Data Structure

All extraction methods produce a JSON file named playlist_videos.json containing an array of video objects. Each object has a minimal set of fields, as most detailed metadata (like duration, views, thumbnail) is not captured by the simple extraction scripts and is marked as "N/A".

The following table describes the fields in each video object.

Field	Type	Description	Example
Video url	string	The full watch URL for the YouTube video.	"https://www.youtube.com/watch?v=VIDEO_ID"
Title	string	The title of the video.	"Video Title Here"
Description	string	A placeholder description.	"Video url:"
Channel name	string	The name of the YouTube channel. Defaults to "N/A".	"N/A"
Duration	string	The duration of the video. Defaults to "N/A".	"N/A"
Views	string	The view count. Defaults to "N/A".	"N/A"
Thumbnail url	string	The URL for the video thumbnail. Defaults to "N/A".	"N/A"

Sources: extract_playlist.py:26-34, PLAYLIST_EXTRACTION_GUIDE.md:92-102

An example of the JSON structure:

[
  {
    "Video url": "https://www.youtube.com/watch?v=VIDEO_ID",
    "Title": "Video Title Here",
    "Description": "Video url:",
    "Channel name": "N/A",
    "Duration": "N/A",
    "Views": "N/A",
    "Thumbnail url": "N/A"
  }
]

Sources: PLAYLIST_EXTRACTION_GUIDE.md:92-102

Data Merging and Deduplication

After playlist_videos.json is created, the merge_playlist_data.py script is run to integrate the new videos into the main data file, consolidated_youtube_data.json.

The script's primary function is to prevent duplicate entries. It achieves this by:

1. Loading the existing videos from consolidated_youtube_data.json.
2. Creating a Python set of all existing video URLs for efficient lookup.
3. Iterating through the newly extracted videos from playlist_videos.json.
4. For each new video, it checks if its URL is already in the set of existing URLs.
5. Only videos with unique URLs are appended to the list of videos.
6. The combined list (old + new unique videos) is written back to consolidated_youtube_data.json, overwriting the old file.

This flowchart details the merging and deduplication logic.

Data Consumption

The final consolidated_youtube_data.json file is consumed by the front-end JavaScript of the website, specifically in videos.js, which powers the videos.html page.

The loadVideos function fetches this JSON file. It then processes and normalizes the data, mapping the raw JSON fields to a consistent internal state object. A key step in this process is the extractVideoId function, which parses the full Video url to get just the video ID, which is then used to construct thumbnail URLs and embed links. The processed data is then used to render the video cards on the page.

Sources: videos.js:46-73, videos.html

Conclusion

The YouTube playlist extraction workflow is a crucial semi-automated process for maintaining the content of the Norm Macdonald Archive. It provides multiple flexible methods for contributors to gather new video data locally, circumventing development environment limitations. The subsequent merge script ensures data integrity by preventing duplicates, and the final consolidated file serves as the single source of truth for the website's video archive section. This system allows the archive to grow systematically while maintaining a clean and robust dataset.
Sources: PLAYLIST_EXTRACTION_GUIDE.md, merge_playlist_data.py