Archiving a Web Page

Relevant source files

The following files were used as context for generating this wiki page:

• packages/plugin/popup/components/UploadPageForm.tsx
• packages/plugin/background/processor.ts
• packages/server/src/api/pages.ts
• packages/web/src/pages/(layout)/page.[slug].tsx
• packages/plugin/lib/single-file-extension-background.js
• README.md

The "Web Archive" project allows users to save a complete, self-contained version of a webpage for offline viewing and sharing. This process is initiated from a browser plugin, which scrapes the active page's content, and sends it to a server backend for persistent storage. The server, built on Cloudflare Workers, stores the page's HTML content in an R2 storage bucket and its metadata in a D1 database.

The core of the archiving functionality relies on the SingleFile library to capture the webpage as a single HTML file. The browser plugin orchestrates this capture, collects additional metadata from the user, and manages the upload process as a background task. The server-side component is a Hono application responsible for receiving the uploaded data, validating it, and storing it across Cloudflare's infrastructure.

Architecture Overview

The archiving process spans three main components of the system: the Browser Plugin, the Server, and the Web Client. The data flows from the user's browser to the Cloudflare-based backend for storage, and is later retrieved by the web client for display.

End-to-End Archiving Flow

The following sequence diagram illustrates the complete process, from the user initiating the save action in the browser plugin to the page being stored on the server.

This diagram shows the interaction between the plugin's UI, background script, content script, and the server API, including interactions with Cloudflare R2 and D1.
Sources: packages/plugin/popup/components/UploadPageForm.tsx:100-119, packages/plugin/background/processor.ts:119-178, packages/server/src/api/pages.ts:10-101

Browser Plugin Components

The browser plugin is responsible for capturing the page and managing the upload process.

Upload Form

When a user decides to save a page, the plugin displays a form to collect metadata. This form is built using React.

Sources: packages/plugin/popup/components/UploadPageForm.tsx

Form Data

The UploadPageForm component manages the state for the page to be uploaded.

Field	Type	Description	Source File
title	string	The title of the page, scraped from the current tab.	packages/plugin/popup/components/UploadPageForm.tsx:49
pageDesc	string	The page description, scraped from the current tab.	packages/plugin/popup/components/UploadPageForm.tsx:50
href	string	The original URL of the page.	packages/plugin/popup/components/UploadPageForm.tsx:51
folderId	string	The ID of the folder to save the page into.	packages/plugin/popup/components/UploadPageForm.tsx:52
screenshot	string	A base64-encoded screenshot of the visible tab area.	packages/plugin/popup/components/UploadPageForm.tsx:53
bindTags	string[]	An array of tags to associate with the page.	packages/plugin/popup/components/UploadPageForm.tsx:54
isShowcased	boolean	A flag to indicate if the page should be featured on the showcase.	packages/plugin/popup/components/UploadPageForm.tsx:55

Upon submission, the handleSavePage function is triggered. It performs validation to ensure a folder is selected and then sends a message to the background script to start the archiving task.

// packages/plugin/popup/components/UploadPageForm.tsx:100-119
async function handleSavePage() {
  // todo await folderlist to check folder extists?
  if (isNil(uploadPageData.folderId)) {
    toast.error(t('folder-required'))
    return
  }
  const tab = await getCurrentTab()
  if (isNil(tab.id)) {
    toast.error(t('get-current-tab-info-failed'))
    return
  }
  await sendMessage('add-save-page-task', {
    tabId: tab.id,
    singleFileSetting: getSingleFileSetting(),
    pageForm: {
      // ... uploadPageData
    },
  })
  toast.success(t('add-save-page-task-success'))
  setActivePage('home')
}

Background Task Processing

The background script (processor.ts) is responsible for handling the entire archiving process asynchronously after the user initiates it. It creates and manages tasks, ensuring that even if the popup is closed, the archiving completes.

Sources: packages/plugin/background/processor.ts

Task State Management

A task is represented by the SeriableSingleFileTask interface and can be in one of several states. The progress of all tasks is stored in Browser.storage.local.

This flowchart shows the state transitions of an archiving task from initialization to completion or failure.
Sources: packages/plugin/background/processor.ts:160-178

Task Properties

Property	Type	Description
uuid	string	Unique identifier for the task.
status	string	Current status: init, scraping, uploading, done, failed.
progress	number	The progress of the task (currently unused, defaults to 0).
href	string	The URL of the page being archived.
tabId	number	The ID of the browser tab.
title	string	The title of the page.
startTimeStamp	number	Timestamp when the task was created.
endTimeStamp	number	Timestamp when the task finished.
errorMessage	string	Error message if the task failed.

Sources: packages/plugin/background/processor.ts:6-18

The createAndRunTask function orchestrates the two main steps:

1. scrapePageData: Injects and executes single-file.js and single-file-extension-core.js in the target tab to get the page's HTML content.
2. uploadPageData: Constructs a FormData object and sends it to the server's /api/pages/upload_new_page endpoint.

Sources: packages/plugin/background/processor.ts:119-178, packages/plugin/lib/single-file-extension-background.js

Server-Side Processing

The server is a Hono application running on Cloudflare Workers. It exposes an API endpoint to handle the page uploads.

Sources: packages/server/src/api/pages.ts

Upload Endpoint: /api/pages/upload_new_page

This endpoint is responsible for receiving the archived page and its metadata, and storing them.

Request Validation

The endpoint uses Hono's validator to ensure the incoming FormData contains all the required fields with the correct types.

Field	Type	Validation
title	string	Required.
pageDesc	string	Optional.
pageUrl	string	Required.
pageFile	File	Required. The HTML content of the page.
folderId	number	Required, must be a numeric string.
screenshot	File	Optional. A screenshot of the page.
bindTags	string[]	Optional, must be a JSON string array.
isShowcased	boolean	Optional, must be 0 or 1.

Sources: packages/server/src/api/pages.ts:11-55

Storage Logic

Upon successful validation, the endpoint performs the following actions:

1. Saves the page's HTML file to the Cloudflare R2 bucket using saveFileToBucket.
2. Saves the screenshot, if provided, to the R2 bucket.
3. Inserts the page's metadata (title, description, URL, R2 object keys, folder ID) into the Pages table in the D1 database via insertPage.
4. Associates the new page with the provided tags in the database using updateBindPageByTagName.

// packages/server/src/api/pages.ts:80-95
const [contentUrl, screenshotId] = await Promise.all([
  saveFileToBucket(c.env.BUCKET, pageFile),
  saveFileToBucket(c.env.BUCKET, screenshot),
])

if (isNil(contentUrl)) {
  return c.json({ status: 'error', message: 'Failed to upload file' })
}
const insertId = await insertPage(c.env.DB, {
  title,
  pageDesc,
  pageUrl,
  contentUrl,
  folderId,
  screenshotId,
  isShowcased,
})

Sources: packages/server/src/api/pages.ts:80-101

Viewing the Archived Page

Once a page is archived, it can be viewed through the web application.

Sources: packages/web/src/pages/(layout)/page.[slug].tsx

The ArchivePage component fetches the page's metadata and its HTML content separately.

1. It calls getPageDetail to retrieve metadata like title and folder ID from the server.
2. It fetches the raw HTML content from /api/pages/content?pageId=<slug>.
3. The HTML content is converted into a blob URL using the useObjectURL hook.
4. The content is then displayed in one of two modes:
   • Iframe Mode: Renders the original, self-contained HTML file inside an <iframe>.
   • Readability Mode: Processes the HTML with Mozilla's Readability library to provide a simplified, reader-friendly view.

// packages/web/src/pages/(layout)/page.[slug].tsx:17-27
async function getPageContent(pageId: string | undefined) {
  if (!pageId)
    return ''
  const url = `/api/pages/content?pageId=${pageId}`
  const res = await fetch(url, {
    method: 'GET',
    headers: {
      'Content-Type': 'text/html',
      'Authorization': `Bearer ${localStorage.getItem('token')}`,
    },
  })
  return await res.text()
}

This function demonstrates how the web client retrieves the stored HTML content for a given page ID.
Sources: packages/web/src/pages/(layout)/page.[slug].tsx:17-27