Project Goals and Vision
Relevant source files
The following files were used as context for generating this wiki page:- README.md
- package.json
- packages/plugin/manifest.json
- packages/plugin/background/processor.ts
- packages/server/src/api/pages.ts
- packages/web/src/pages/(layout)/page.[slug].tsx
- packages/web/src/assets/locales/en/translation.json
- packages/plugin/assets/locales/en/translation.json
- packages/web/package.json
- packages/web/src/pages/showcase/page.[slug].tsx
The Web Archive project is a self-hostable service for archiving, managing, and sharing web pages. It is built upon a modern, decoupled architecture leveraging Cloudflare's infrastructure, including Workers, D1 database, and R2 storage. The project's primary goal is to provide a robust and personal web archiving solution that captures web content accurately as a single HTML file and offers powerful features for organization and retrieval.
The system is composed of three main parts: a browser plugin for capturing pages, a server for storage and API management, and a web client for querying and viewing the archived content. This modular design allows for flexible deployment and use. Key features include folder and tag-based classification, full-text search, a "showcase" mode for curated content, and an AI-powered tagging system.
Sources: README.md:9-14, package.json:5
Core Architecture
The project is structured as a monorepo with three distinct packages: plugin, server, and web. This architecture separates the concerns of capturing, storing, and presenting the archived web pages.
Sources: package.json:11-13
This diagram illustrates the high-level interaction between the browser plugin, the Cloudflare-based server, and the web client.
Sources: README.md:9-14
Web Archiving and Processing
The core of the archiving functionality lies within the browser plugin. It is responsible for capturing the current state of a webpage and uploading it to the server.
Page Saving Process
The plugin, described as "SingleFile with categories and exhibition pages," uses a sophisticated process to scrape the page content and its resources into a single HTML file. This process is initiated by the user through the plugin's UI.
The following sequence diagram shows the flow from initiating a save task to the final upload.
This flow is managed by the createAndRunTask function, which orchestrates scraping and uploading, while updating the task status along the way.
Sources: packages/plugin/background/processor.ts:107-142, packages/plugin/assets/locales/en/translation.json:20
Page Processing Options
The plugin provides several options to control how a page is saved, allowing users to optimize the final archived file. These options are based on the underlying SingleFile engine.
| Option | Description |
|---|---|
compress-HTML | Compresses the HTML content to reduce file size. |
block-audios | Prevents audio elements from being included in the archive. |
block-videos | Prevents video elements from being included in the archive. |
block-scripts | Prevents scripts from being included in the archive. |
remove-hidden-elements | Removes elements that are not visible on the page. |
remove-unused-styles | Strips out CSS rules that are not used. |
load-deferred-images | Attempts to load images that are loaded lazily. |
Sources: packages/plugin/assets/locales/en/translation.json:11-19
Server-Side Storage and Management
The server, built with Hono, runs on a Cloudflare Worker. It exposes a RESTful API to handle page uploads, queries, and management tasks. It uses Cloudflare D1 for metadata and R2 for file storage.
Sources: README.md:13-14, packages/server/src/api/pages.ts
Page Upload Endpoint
The primary endpoint for adding new archives is /api/pages/upload_new_page. It accepts multipart/form-data containing the page details and the HTML file itself.
This diagram shows the process of validating input, storing the page and screenshot in R2, and then recording the metadata and tag associations in the D1 database.
Sources: packages/server/src/api/pages.ts:20-101
API Endpoints
The server provides a comprehensive API for managing archived pages.
| Method | Endpoint | Description |
|---|---|---|
POST | /api/pages/upload_new_page | Uploads a new page archive, including HTML file, screenshot, and metadata. |
POST | /api/pages/query | Queries for pages based on folder, tag, or keyword. Supports pagination. |
GET | /api/pages/detail | Retrieves the details of a single page by its ID. |
GET | /api/pages/content | Retrieves the raw HTML content of an archived page. |
DELETE | /api/pages/delete_page | Marks a page as deleted. |
PUT | /api/pages/update_page | Updates a page's metadata, such as title, folder, and tags. |
POST | /api/pages/restore_page | Restores a previously deleted page. |
DELETE | /api/pages/clear_deleted | Permanently deletes all pages marked as deleted and their associated files from R2. |
GET | /api/pages/screenshot | Retrieves the screenshot image for a page. |
PUT | /api/pages/update_showcase | Adds or removes a page from the public showcase. |
Sources: packages/server/src/api/pages.ts
Web Client and Content Presentation
The web client is a React-based single-page application that allows users to browse, search, and view their archives. It interacts with the server's API to fetch data and render content.
Sources: package.json:7, packages/web/index.html:5
Page Viewing
When a user navigates to an archived page, the client fetches the page's HTML content from the /api/pages/content endpoint and displays it. The client offers two viewing modes:
- Iframe Mode: Renders the original, interactive HTML file within an
<iframe>. This preserves the original layout and scripts. - Read Mode: Uses the
@mozilla/readabilitylibrary to extract the core article content and presents it in a clean, readable format, stripping away ads, navigation, and other clutter.
Sources: packages/web/src/pages/(layout)/page.[slug].tsx:10-11, 75-78, packages/web/package.json:9
The user can toggle between these modes and also download the raw HTML file.
This diagram shows the logic for fetching and displaying an archived page in the web client.
Sources: packages/web/src/pages/(layout)/page.[slug].tsx:28-60
Showcase Feature
The project includes a "Showcase" feature, which appears to be a public-facing gallery of selected archived pages. Pages can be marked as "showcased" and are then viewable via a separate route (/showcase/page/:slug). This view includes navigation to the next showcase page and a link back to the main showcase folder.
Sources: packages/web/src/pages/showcase/page.[slug].tsx, packages/web/src/assets/locales/en/translation.json:51, packages/server/src/api/pages.ts:468-486
Other Key Features
Based on translation files and API definitions, the project aims to provide a rich user experience with the following features:
| Feature | Description | Source |
|---|---|---|
| Folder Classification | Organize archived pages into folders. | packages/web/src/assets/locales/en/translation.json:44 |
| Tagging | Assign multiple tags to pages for flexible organization. | packages/web/src/assets/locales/en/translation.json:47 |
| AI Tag Generation | Automatically generate tags for a page based on its title and description. | README.md:19, packages/web/src/assets/locales/en/translation.json:38 |
| Trash/Recycle Bin | Deleted pages are moved to a trash area and can be restored or permanently cleared. | packages/web/src/assets/locales/en/translation.json:67, packages/server/src/api/pages.ts:383-438 |
| Search | Search for pages within a specific folder by keyword. | packages/web/src/assets/locales/en/translation.json:55 |
| Mobile Adaptation | The service is designed to be responsive and usable on mobile devices. | README.md:18 |
Conclusion
The vision for Web Archive is to be a comprehensive, self-hosted, and modern solution for personal web archiving. By combining the power of the SingleFile engine for high-fidelity page capture with the scalability and ease of deployment of the Cloudflare ecosystem, the project provides a full-featured platform. It goes beyond simple saving by offering robust organizational tools like folders, tags, AI-driven classification, and curated showcases, making it a powerful tool for knowledge management and digital preservation.
Sources: README.md:9-14, package.json:5