The SingleFile Integration

Relevant source files

The following files were used as context for generating this wiki page:

• packages/plugin/background/processor.ts
• packages/plugin/utils/singleFile.ts
• packages/server/src/api/pages.ts
• packages/plugin/lib/single-file-extension.js
• packages/plugin/lib/single-file-extension-background.js
• packages/plugin/lib/single-file-extension-editor.js
• packages/web/src/pages/(layout)/page.[slug].tsx

The SingleFile integration is the core mechanism by which the Web Archive browser plugin captures, processes, and uploads web pages for archiving. It leverages the powerful open-source SingleFile library to convert a live web page into a self-contained HTML file. This integration is managed primarily within the browser plugin's background and content scripts, which orchestrate the scraping process and subsequent upload to the Web Archive server.

The system is designed to be robust, managing the entire lifecycle of a page capture as a distinct task. This includes scraping the page content, handling potential failures, uploading the data, and persisting task status. The integration also extends to a client-side editor that allows for modification of the saved HTML content, providing features like highlighting and content removal.

Architecture Overview

The integration involves several key components across the browser plugin and the server. The browser plugin initiates the process, using SingleFile to scrape the page. The resulting HTML is then sent to the server for storage.

This diagram illustrates the high-level flow of the SingleFile integration.

Page Scraping and Task Management

The page scraping process is managed as a series of tasks within the plugin's background script. Each task represents a single page archival operation, from initiation to completion or failure.

Sources: packages/plugin/background/processor.ts

Task Lifecycle

When a user initiates a page save, a task is created and its state is tracked throughout the process. The task lifecycle is managed by the createAndRunTask function.

The state of a task is defined by the status property in the SeriableSingleFileTask interface.

Task Data Structure

Task information is stored locally using Browser.storage.local. The SeriableSingleFileTask interface defines the structure of a task object.

Property	Type	Description
uuid	string	Unique identifier for the task.
status	'init' | 'scraping' | 'uploading' | 'done' | 'failed'	The current status of the task.
progress	number	The progress of the task (currently not fully implemented, defaults to 0).
href	string	The URL of the page being archived.
tabId	number	The ID of the browser tab where the scraping occurs.
title	string	The title of the page.
pageDesc	string	The description of the page.
folderId	string	The ID of the folder to save the page into.
bindTags	string[]	An array of tags to associate with the page.
isShowcased	boolean	Whether the page should be added to the showcase.
startTimeStamp	number	Timestamp when the task was created.
endTimeStamp	number	Timestamp when the task finished (either done or failed).
errorMessage	string	An error message if the task failed.

Sources: packages/plugin/background/processor.ts:7-17

Scraping Process Flow

The createAndRunTask function in processor.ts orchestrates the entire flow.

This sequence diagram shows the interactions during the scraping and upload process.

The scrapePageData function is responsible for injecting and invoking the SingleFile library in the target tab.

// packages/plugin/background/processor.ts:66-72
async function scrapePageData(singleFileSetting: SingleFileSetting, tabId: number) {
  await Browser.scripting.executeScript({
    target: { tabId },
    files: ['/lib/single-file.js', '/lib/single-file-extension-core.js'],
  })
  const { content } = await sendMessage('scrape-page-data', singleFileSetting, `content-script@${tabId}`)
  return content
}

SingleFile Configuration

The scraping process is configured via the SingleFileSetting object. The project uses a default set of options to ensure pages are saved cleanly and efficiently.

Property	Default Value	Description
removeHiddenElements	true	Removes hidden elements like comments or modals.
removeUnusedStyles	true	Removes CSS rules that do not apply to any content.
removeUnusedFonts	true	Removes unused @font-face rules.
blockScripts	true	Prevents scripts from running in the saved page.
blockAudios	true	Prevents audio from being embedded.
blockVideos	true	Prevents video from being embedded.
compressHTML	true	Minifies the final HTML.
loadDeferredImages	true	Forces lazy-loaded images to load.
loadDeferredImagesMaxIdleTime	1500	Max time in ms to wait for lazy images.

Sources: packages/plugin/utils/singleFile.ts:22-59

Data Upload

After the page is successfully scraped into a single HTML string, the uploadPageData function in the background script sends it to the server.

Sources: packages/plugin/background/processor.ts:74-99, packages/server/src/api/pages.ts:12-92

API Endpoint: POST /api/pages/upload_new_page

This endpoint accepts a multipart/form-data request containing the page content and its associated metadata.

Field	Type	Required	Description
title	string	Yes	The title of the page.
pageUrl	string	Yes	The original URL of the page.
pageDesc	string	No	A description for the page.
folderId	number	Yes	The ID of the folder to save the page into.
pageFile	File	Yes	The Blob containing the HTML content of the page.
screenshot	File	No	A Blob of the page screenshot (image/webp).
bindTags	string	No	A JSON string array of tag names to associate.
isShowcased	string	No	'1' if the page should be showcased, '0' otherwise.

The server-side handler validates these fields, saves the pageFile and screenshot to an R2 bucket, and inserts a new record into the pages table in the D1 database.

Sources: packages/server/src/api/pages.ts:12-92, packages/plugin/background/processor.ts:74-99

Saved Page Editor

The project includes a sophisticated client-side editor for the saved HTML files, based on single-file-extension-editor.js. This editor is not part of the initial capture but is used when viewing and editing an already archived page from a .html file.

The editor provides functionalities such as:

• Adding, moving, and resizing notes.
• Highlighting text in various colors.
• Cutting/removing elements from the page.
• Formatting the page into a "reader mode" style.

When an HTML file saved by web-archive is opened, an embedded script initializes the editor. The init function within single-file-extension-editor.js is the entry point. It parses the document, deserializes shadow roots for custom elements (like notes), and attaches all necessary event listeners for editing features.

// packages/plugin/lib/single-file-extension-editor.js:1546-1619
async function init({ content, password, compressContent }, { filename, reset } = {}) {
    // ...
    // Detects if the page is a saved page
    if (detectSavedPage(contentDocument)) {
        // ...
        // Replaces the current document with the saved page's content
        document.replaceChild(contentDocument.documentElement, document.documentElement);
        // ...
        // Deserializes shadow roots for notes and other custom elements
        deserializeShadowRoots(document);
        // Attaches listeners to notes
        document.querySelectorAll(NOTE_TAGNAME).forEach(containerElement => attachNoteListeners(containerElement, true));
        // Inserts stylesheets for highlights
        insertHighlightStylesheet(document);
        // Initializes masks for editing UI
        maskPageElement = getMaskElement(PAGE_MASK_CLASS, PAGE_MASK_CONTAINER_CLASS);
        maskNoteElement = getMaskElement(NOTE_MASK_CLASS);
        // Attaches global event listeners for mouse, keyboard, and touch events
        document.documentElement.onmousedown = onMouseDown;
        // ...
    }
}

Sources: packages/plugin/lib/single-file-extension-editor.js:1546-1619

This powerful editing capability allows users to annotate and clean up their archived pages directly within the browser, with all changes being saved back into the HTML file itself.