Consolidating Playlist Data

Relevant source files

The following files were used as context for generating this wiki page:

• merge_playlist_data.py
• PLAYLIST_EXTRACTION_GUIDE.md
• extract_playlist.html
• videos.js
• old/site-src/TODO.txt
• README.md
• redesign/videos.js

The process of consolidating playlist data is a crucial workflow for updating the video content on The Norm Macdonald Archive. It involves extracting video metadata from a YouTube playlist and merging it into a central data file, consolidated_youtube_data.json. This master file serves as the single source of truth for the video archive displayed on the website.

This system ensures that new videos can be added to the archive without introducing duplicates. The workflow is designed to be semi-automated, with tools provided to extract data, followed by a script to handle the merging and deduplication logic. The final consolidated file is then consumed by the frontend to render the video gallery.

Sources: PLAYLIST_EXTRACTION_GUIDE.md, merge_playlist_data.py, videos.js, README.md:21-22

Overall Data Flow

The entire process, from data extraction to frontend rendering, follows a clear pipeline. A developer first extracts video data from a specific YouTube playlist using one of the provided tools. This generates a playlist_videos.json file. This new data is then merged into the existing consolidated_youtube_data.json using the merge_playlist_data.py script. Finally, the frontend application fetches the consolidated file to display the videos.

The following diagram illustrates the end-to-end data flow.

Playlist Data Extraction

Before the consolidation can occur, video data must be extracted from a YouTube playlist. The project provides several methods to accomplish this, catering to different user preferences and technical environments. The target playlist is https://youtube.com/playlist?list=PLxTIDernmMHuIsNqSMOhKR1jlWimaLklk.

Sources: PLAYLIST_EXTRACTION_GUIDE.md:5-7

Extraction Methods

Three primary methods are documented for generating the playlist_videos.json file.

Method	Description
Browser Console	The user pastes a JavaScript snippet into the browser console on the YouTube playlist page. This is the easiest method as it requires no installation.
Python Script	A pytube-based Python script (extract_playlist.py) is run locally. This is considered the most reliable method but may fail in restricted network environments.
Browser Tool	A user-friendly HTML tool (extract_playlist.html) guides the user through running the console script and downloading the resulting JSON file.

Sources: PLAYLIST_EXTRACTION_GUIDE.md:11-50, extract_playlist.html

Extracted Data Structure

All extraction methods produce a JSON file named playlist_videos.json containing an array of video objects. The structure is minimal, capturing the essential URL and title, with other fields acting as placeholders.

Key	Type	Description
Video url	String	The full URL to the YouTube video.
Title	String	The title of the video.
Description	String	Placeholder text, consistently "Video url:".
Channel name	String	Placeholder, "N/A".
Duration	String	Placeholder, "N/A".
Views	String	Placeholder, "N/A".
Thumbnail url	String	Placeholder, "N/A".

Sources: PLAYLIST_EXTRACTION_GUIDE.md:100-110, extract_playlist.html:105-116

The Merge Process (merge_playlist_data.py)

The core of the consolidation process is the merge_playlist_data.py script. Its purpose is to intelligently merge the newly extracted videos from playlist_videos.json into the master consolidated_youtube_data.json file, ensuring no duplicates are added.

Sources: merge_playlist_data.py

Merge Logic

The script follows a sequence of loading data, identifying new entries, and writing the merged result.

This flowchart shows the logic within the merge_videos function.

Deduplication

The key to preventing redundant data is the deduplication step. The script first loads all existing video URLs into a Python set for efficient lookup. It then iterates through the newly extracted videos and only appends those whose Video url is not already in the set.

# merge_playlist_data.py:41-51

# Extract existing video URLs for deduplication
existing_urls = {video.get('Video url') for video in existing_videos}
print(f"📋 Found {len(existing_urls)} unique URLs in existing data")

# Add only new videos
new_videos = []
duplicates = 0

for video in playlist_videos:
    video_url = video.get('Video url')
    if video_url and video_url not in existing_urls:
        new_videos.append(video)
        existing_urls.add(video_url)
    else:
        duplicates += 1

Sources: merge_playlist_data.py:41-51

Consumption by Frontend

The ultimate consumer of the consolidated_youtube_data.json file is the website's frontend, specifically the video archive page. The JavaScript file videos.js is responsible for fetching this data and rendering it.

Sources: videos.js, redesign/videos.js

Data Loading and Normalization

The loadVideos function in videos.js fetches the consolidated JSON file. Upon receiving the data, it normalizes each raw video object into a more structured format suitable for the application's state. This includes extracting the video ID from the URL.

This sequence diagram shows the frontend loading process.

The raw data from consolidated_youtube_data.json is mapped to a new object structure within the application's state.

State Key	Type	Description	Source Field
id	Number	The index of the video in the array.	(Generated)
url	String	The extracted YouTube video ID.	Video url
title	String	The video title.	Title
description	String	The video description.	Description
channel	String	The channel name.	Channel name
duration	String	The video duration.	Duration
views	String	The view count.	Views
thumbnail	String	The thumbnail URL.	Thumbnail url

Sources: videos.js:68-78, redesign/videos.js:68-78

Conclusion

The data consolidation pipeline is a fundamental part of the site's content management strategy. It provides a structured and reliable method for expanding the video archive. By separating the extraction and merging steps, and by implementing robust deduplication, the system ensures data integrity while remaining simple enough for developers to maintain. The final consolidated_youtube_data.json file acts as a stable "database" for the static site, enabling rich, data-driven features on the frontend.

Sources: PLAYLIST_EXTRACTION_GUIDE.md:60-64, old/site-src/TODO.txt:10-11