System Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
- README.md
- package.json
- packages/plugin/manifest.json
- packages/plugin/lib/single-file-extension-background.js
- packages/plugin/popup/components/HistoryTaskList.tsx
- packages/server/package.json
- packages/web/package.json
- packages/web/src/pages/(layout)/page.[slug].tsx
- packages/web/src/components/readability-page-content.tsx
- packages/web/src/pages/showcase/page.[slug].tsx
The Web Archive project is a self-hosted service for archiving, viewing, and sharing web pages. The system is built on a modern web stack, leveraging Cloudflare for server-side operations and React for the front-end client. It is architected as a monorepo containing three primary, decoupled components: a browser plugin for capturing web pages, a server for storage and retrieval, and a web client for displaying the archived content.
This document provides a high-level overview of the system's architecture, its core components, and the interactions between them. The project is designed to be a complete, self-contained solution for personal web archiving.
Sources: README.md, package.json:12-14
Core Components
The system is a monorepo managed by pnpm workspaces and is composed of several distinct packages. The main application logic is split into three parts:
- Browser Plugin (
@web-archive/plugin): A browser extension responsible for capturing web pages as single HTML files and uploading them to the server. - Server (
@web-archive/server): A Cloudflare Worker-based backend that handles API requests, stores page metadata in a D1 database, and saves page content to an R2 storage bucket. - Web Client (
@web-archive/web): A React-based single-page application for querying, managing, and viewing the archived web pages. - Shared (
@web-archive/shared): A shared library of UI components and utilities used by both the plugin and the web client.
This component-based architecture is defined in the root package.json file.
Sources: README.md:11-15, package.json:12-14
Monorepo Package Structure
The project uses a pnpm workspace to manage its multiple packages. This structure promotes code sharing and simplifies dependency management across the different parts of the application.
This diagram illustrates the high-level dependencies between the packages within the monorepo and their interaction with external Cloudflare services.
Sources: package.json:12-14, README.md:16-17
End-to-End Data Flow
The following diagram illustrates the complete lifecycle of archiving and viewing a page.
This sequence shows the two main user flows: archiving a new page via the plugin and viewing a previously archived page via the web client.
Sources: README.md:11-15, packages/web/src/pages/(layout)/page.[slug].tsx:18-27, packages/plugin/lib/single-file-extension-background.js
Browser Plugin (@web-archive/plugin)
The browser plugin is the entry point for capturing web content. It is built as a standard WebExtension (Manifest V3) and relies heavily on the SingleFile library to process and bundle a webpage into a single, self-contained HTML file.
Sources: packages/plugin/manifest.json, packages/plugin/lib/single-file-extension-background.js
Plugin Components
The plugin's structure is defined in its manifest file and includes several key scripts:
| Component | Path/File | Description |
|---|---|---|
| Manifest | packages/plugin/manifest.json | Defines the extension's properties, permissions, and scripts. |
| Background Script | background/background.js | The service worker that manages state, handles events, and orchestrates the page saving process. |
| Popup UI | popup/index.html | The main user interface, allowing users to initiate saves and view task history. |
| Content Scripts | lib/single-file-*.js | A suite of scripts from the SingleFile library injected into web pages to capture their content. |
| Permissions | activeTab, storage, tabs, scripting | Permissions required for accessing tab content, storing settings, and executing scripts. |
Sources: packages/plugin/manifest.json:9-12, 16-56, 57-62
Page Saving Process
The page saving process is managed by the background script. It maintains a queue of tasks and processes them. The popup UI can query this task list to display the status of ongoing and completed saves.
This flow shows how the popup UI communicates with the background script to get real-time updates on page-saving tasks.
Sources: packages/plugin/popup/components/HistoryTaskList.tsx:13-21, packages/plugin/lib/single-file-extension-background.js
Server (@web-archive/server)
The server is a lightweight API built with Hono and deployed as a Cloudflare Worker. Its primary role is to act as a bridge between the plugin/web client and the Cloudflare storage services (D1 and R2).
Sources: README.md:16-17, packages/server/package.json
Technologies
| Technology | Usage |
|---|---|
| Hono | A small, simple, and ultrafast web framework for the edge. Used for routing and handling API requests. |
| Cloudflare Workers | Provides the serverless execution environment. |
| Vite | Used as the build tool for the server package, with @hono/vite-cloudflare-pages. |
| Wrangler | The command-line tool for developing and deploying Cloudflare Workers. |
Sources: packages/server/package.json:11-17
API Endpoints
The server exposes several API endpoints for the web client to interact with. These are inferred from the client-side fetch requests.
| Endpoint | Method | Description |
|---|---|---|
/api/pages/content | GET | Fetches the raw HTML content of a specific archived page by its ID. |
/api/showcase/content | GET | Fetches the content for a page in the public showcase. |
/api/pages | DELETE | Deletes a specific page. The client shows a confirmation before calling this. |
Sources: packages/web/src/pages/(layout)/page.[slug].tsx:18-27, 81-84, packages/web/src/pages/showcase/page.[slug].tsx:14-23
Web Client (@web-archive/web)
The web client is a React-based Single Page Application (SPA) that provides the user interface for browsing, searching, and viewing archived pages.
Sources: packages/web/package.json, README.md:14-15
Page Rendering Modes
A key feature of the web client is its ability to display archived content in two different modes, which the user can toggle between.
-
Iframe Mode (Default): The original, full HTML content is loaded into an
<iframe>. This preserves the original layout and styling. To achieve this, the HTML content is fetched from the API, converted into a Blob, and then loaded into the iframe via a temporary object URL.Sources: packages/web/src/pages/(layout)/page.[slug].tsx:67-69, 102-103
-
Readability Mode: For a cleaner, article-focused view, the client uses the
@mozilla/readabilitylibrary. This library parses the raw HTML and extracts the main article content, stripping away ads, navigation, and other non-essential elements. The extracted content is then rendered directly into the DOM.Sources: packages/web/src/components/readability-page-content.tsx:8-10, 19, packages/web/package.json:8
The flow for displaying a page is as follows:
This diagram shows the client-side logic for rendering a page in either Iframe or Readability mode.
Sources: packages/web/src/pages/(layout)/page.[slug].tsx:67-74, 91, 101-103, packages/web/src/components/readability-page-content.tsx:8-10
Showcase Feature
The application includes a "Showcase" section, which appears to be a public-facing gallery of selected archived pages. It has its own API endpoints and navigation logic, including "next" and "previous" functionality.