Data Flow: Saving a Page

Relevant source files

The following files were used as context for generating this wiki page:

• packages/plugin/background/processor.ts
• packages/server/src/api/pages.ts
• packages/server/src/model/page.ts
• packages/plugin/popup/components/UploadPageForm.tsx
• packages/plugin/utils/singleFile.ts
• packages/web/src/pages/(layout)/page.[slug].tsx
• packages/plugin/popup/components/HistoryTaskList.tsx

The process of saving a web page is a core feature of the Web Archive application. It involves a browser extension that captures the page content, a background task processor that manages the capture and upload, and a server backend that stores the page data and metadata. This flow ensures that a complete, self-contained version of the web page is archived and can be viewed later through the web interface.

The entire process is asynchronous, managed by a task queue in the browser extension's background script. This allows users to continue browsing while pages are saved in the background. The status of these tasks can be monitored through the extension's popup.

High-Level Architecture

The page-saving mechanism is distributed across three main components: the browser plugin, the background script, and the server.

1. Browser Plugin (Popup UI): The user initiates the save process from the plugin's popup. The UI scrapes initial data, allows the user to add metadata, and triggers the background task.
2. Background Script: This script orchestrates the entire process. It uses the SingleFile library to scrape the full page content, manages a task queue for concurrent saves, and uploads the final data to the server.
3. Server (Cloudflare Worker): A Hono-based API running on Cloudflare Workers receives the uploaded data. It saves the page's HTML content and screenshot to an R2 bucket and writes the page's metadata to a D1 database.

This diagram illustrates the high-level interaction between these components.

Detailed Data Flow

1. Initiation and Data Collection (Plugin)

The process begins when the user opens the browser extension popup and navigates to the save page form.

1. Initial Data Scraping: The UploadPageForm component immediately calls scrapePageData to get basic information from the current tab. This includes the page title, description, and a screenshot. The screenshot is captured using takeScreenshot.
   Sources: packages/plugin/popup/components/UploadPageForm.tsx:70-85
2. User Input: The user selects a destination folder, adds tags, and can choose to feature the page in the "Showcase".
   Sources: packages/plugin/popup/components/UploadPageForm.tsx:96-125
3. Task Creation: Upon confirmation, the handleSavePage function is called. It sends an add-save-page-task message to the background script with all the collected data, including the page metadata and settings for the SingleFile library.
   Sources: packages/plugin/popup/components/UploadPageForm.tsx:135-154

2. Background Task Processing

The background script (processor.ts) is responsible for managing the entire archival process as an asynchronous task.

Task Definition and Lifecycle

A task is defined by the SeriableSingleFileTask interface. Each task has a status that progresses through a defined lifecycle.

Field	Type	Description
uuid	string	A unique identifier for the task.
status	'init' | 'scraping' | 'uploading' | 'done' | 'failed'	The current state of the task.
progress	number	The progress of the task (currently unused).
href	string	The URL of the page being saved.
tabId	number	The ID of the tab where the page is located.
title	string	The title of the page.
startTimeStamp	number	Timestamp when the task was created.
endTimeStamp	number	Timestamp when the task finished (successfully or not).
errorMessage	string	An error message if the task failed.

Sources: packages/plugin/background/processor.ts:5-18, packages/plugin/popup/components/HistoryTaskList.tsx

This flowchart shows the lifecycle of a save task.

Task Execution

The createAndRunTask function orchestrates the task execution:

1. Initialization: A new task is created with the status init and saved to local storage.
2. Scraping: The status is updated to scraping. The scrapePageData function injects and executes the SingleFile library (single-file.js) in the target tab to capture the full page HTML.
3. Uploading: The status changes to uploading. The uploadPageData function constructs a FormData object containing the page's HTML content, screenshot, and all metadata. This form data is then sent via a POST request to the /api/pages/upload_new_page endpoint on the server. A keep-alive mechanism is used to prevent the service worker from terminating during long uploads.
4. Completion: Upon a successful server response, the task status is set to done. If any step fails, the status is set to failed with an error message.

Sources: packages/plugin/background/processor.ts:125-174

3. Server-Side Storage

The server receives the FormData at the /api/pages/upload_new_page endpoint.

1. Validation: The endpoint first validates the incoming form data, ensuring required fields like title, pageUrl, pageFile, and folderId are present and correctly typed.
   Sources: packages/server/src/api/pages.ts:16-59
2. File Storage: The page's HTML (pageFile) and the screenshot are saved to the R2 bucket using the saveFileToBucket utility. This returns unique keys (contentUrl and screenshotId) for the stored objects.
   Sources: packages/server/src/api/pages.ts:80-83
3. Database Insertion: The insertPage function is called to write the page's metadata to the pages table in the D1 database. This includes the title, description, original URL, folder ID, and the R2 keys for the content and screenshot.
   Sources: packages/server/src/api/pages.ts:88-97, packages/server/src/model/page.ts:192-201
4. Tag Association: If any tags were provided, updateBindPageByTagName is called to associate the newly created page ID with the specified tags in the tags table.
   Sources: packages/server/src/api/pages.ts:98-99

Database Schema for pages

The pages table stores all metadata related to an archived page.

4. Viewing the Saved Page

Once a page is saved, it can be viewed through the web application.

1. Fetching Page Content: When a user navigates to /page/:slug, the ArchivePage component fetches the page's HTML content from the /api/pages/content?pageId=<slug> endpoint.
   Sources: packages/web/src/pages/(layout)/page.[slug].tsx:18-28
2. Server-Side Content Delivery: The /api/pages/content endpoint retrieves the page's metadata from the database using getPageById. It uses the contentUrl from the metadata to fetch the corresponding HTML file from the R2 bucket.
   Sources: packages/server/src/api/pages.ts:468-486
3. Rendering: The fetched HTML content is set as the src of an <iframe> in the IframePageContent component, rendering the archived page in a sandboxed environment.
   Sources: packages/web/src/pages/(layout)/page.[slug].tsx:122-123

Summary

The page saving data flow is a well-orchestrated process that leverages a browser extension for user interaction and page capture, a background task manager for robust, asynchronous processing, and a Cloudflare-based server for scalable storage and retrieval. The use of the SingleFile library ensures high-fidelity page captures, while the task queue in the background script provides a seamless user experience. The final archived page is stored securely in R2 and its metadata is managed in a D1 database, making it readily available for viewing through the web interface.