Monorepo Structure

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• package.json
• packages/plugin/lib/single-file-extension-background.js
• packages/plugin/lib/single-file-extension-editor.js
• packages/plugin/manifest.firefox.json
• packages/plugin/manifest.json
• packages/plugin/package.json
• packages/plugin/popup/components/HistoryTaskList.tsx
• packages/plugin/popup/components/SavedPageList.tsx
• packages/server/package.json
• packages/server/src/api/pages.ts
• packages/server/src/api/tags.ts
• packages/shared/components/sheet.tsx
• packages/web/package.json
• packages/web/src/components/page-card.tsx
• packages/web/src/components/powerd-by.tsx
• packages/web/src/components/readability-page-content.tsx
• packages/web/src/pages/(layout)/page.[slug].tsx
• packages/web/src/pages/showcase/page.[slug].tsx

# Monorepo Structure

The web-archive project is a self-hosted web archiving service built on Cloudflare technologies. It is organized as a monorepo managed by pnpm, which allows for a modular architecture with shared code and streamlined development workflows. The project consists of three main parts: a browser plugin for capturing web pages, a server backend running on Cloudflare Workers, and a web client for viewing and managing the archives.

This structure facilitates code sharing between the different parts of the application, particularly through the shared package, and simplifies the build and deployment process for the entire system.

Sources: README.md:9-14, package.json:23-25

Packages Overview

The monorepo is configured via the root package.json to include all subdirectories within the packages/* directory as part of the pnpm workspace. This setup enables cross-package dependencies and centralized script execution.

Sources: package.json:23-25

Package Dependencies

The following diagram illustrates the dependencies between the different packages within the monorepo.

This diagram shows that the plugin, web, and server packages all depend on the shared package for common utilities and components.

Sources: packages/web/package.json:12, packages/server/package.json:16, packages/plugin/package.json:20

Package Descriptions

Package	Name	Description
plugin	plugin	A browser extension for Chrome and Firefox that captures web pages as single HTML files and uploads them to the server. It leverages the single-file library for page processing.
server	@web-archive/server	A Hono-based backend deployed on Cloudflare Pages. It provides APIs for uploading pages, managing folders and tags, and serving archived content.
web	@web-archive/web	A React-based web client for browsing, searching, and viewing archived pages. It includes features like a "Readability" mode and a showcase for public pages.
shared	@web-archive/shared	A package containing shared code, such as React components (Sheet, Card, etc.) and utility functions, used by other packages.

Sources: README.md:11-14, packages/plugin/package.json:2, packages/server/package.json:2, packages/web/package.json:2, packages/shared/components/sheet.tsx

Core Packages

plugin Package

The plugin package is a browser extension responsible for capturing web pages. It is built using Vite and the @crxjs/vite-plugin.

Sources: packages/plugin/package.json:15-17

Key Functionality

• Page Saving: Uses the single-file library to process and save the current web page as a single HTML file. This is handled by background scripts like single-file-extension-background.js.
• User Interface: Provides a popup UI (popup/index.html) with components like SavedPageList and HistoryTaskList to show previously archived pages for the current URL and the status of ongoing save tasks.
• Cross-Browser Support: Includes separate manifests for Chrome (manifest.json) and Firefox (manifest.firefox.json) to handle differences between the platforms, such as the background script definition (service_worker vs. scripts).
• Content Scripts: Injects scripts into web pages to handle page processing and communication with the background script.

Sources: packages/plugin/manifest.json:44-47, packages/plugin/manifest.firefox.json:47-50, packages/plugin/popup/components/SavedPageList.tsx, packages/plugin/popup/components/HistoryTaskList.tsx, packages/plugin/lib/single-file-extension-background.js

Page Saving Flow

The following diagram shows the sequence of events when a user saves a page using the browser plugin.

server Package

The server package is a backend service built with Hono and designed to run on Cloudflare. It handles all data persistence and business logic.

Sources: packages/server/package.json:12

API Endpoints

The server exposes several API endpoints for managing pages and tags. The primary endpoints for pages are defined in src/api/pages.ts.

Method	Endpoint	Description
POST	/upload_new_page	Uploads a new archived page, including the HTML file, screenshot, and metadata.
POST	/query	Queries for pages based on folder, tag, or keyword with pagination.
GET	/detail	Retrieves the details for a single page by its ID.
GET	/content	Retrieves the raw HTML content of a saved page for rendering.
DELETE	/delete_page	Marks a page as deleted.
PUT	/update_page	Updates a page's metadata, such as title, description, and associated tags.
POST	/restore_page	Restores a previously deleted page.
DELETE	/clear_deleted	Permanently deletes all pages that are marked as deleted from the database and R2 bucket.
GET	/screenshot	Retrieves the screenshot image for a given page.

Sources: packages/server/src/api/pages.ts

Other endpoints include tag management (/api/tags/*) for creating, updating, deleting, and generating tags using Cloudflare's AI.

Sources: packages/server/src/api/tags.ts

web Package

The web package is the user-facing frontend, built with React and Vite. It allows users to browse and view their archived content.

Sources: packages/web/package.json:3-4

Key Functionality

• Page Rendering: Displays archived pages. It offers two modes:
  1. Iframe Mode: Renders the saved HTML content directly within an <iframe> to preserve the original layout and styling.
  2. Readability Mode: Uses @mozilla/readability to extract the core article content from the HTML and presents it in a clean, readable format. This is implemented in the ReadabilityPageContent component.
• Routing: Uses @generouted/react-router for file-based routing. Dynamic routes like /page/[slug] and /showcase/page/[slug] handle the display of individual pages.
• UI Components: Displays pages in a grid using PageCard components, which show a title, description, tags, and a screenshot.
• Showcase: Provides a public-facing "showcase" section to display specially selected pages.

Sources: packages/web/src/pages/(layout)/page.[slug].tsx:9-10, 81-82, packages/web/src/components/readability-page-content.tsx:9-11, packages/web/package.json:6, packages/web/src/components/page-card.tsx

Page View Flow

This diagram shows the process of a user viewing an archived page.

Development and Build Scripts

The root package.json defines several pnpm scripts for managing the monorepo. These scripts use the --filter flag to target specific packages.

Script	Command	Description
dev:web	pnpm --filter=web dev	Starts the development server for the web package.
dev:server	pnpm --filter=server dev	Starts the development server for the server package.
dev:plugin	pnpm --filter=plugin dev	Starts the development build for the plugin package in watch mode.
build:service	pnpm --filter=server build && pnpm --filter=web build && ...	Builds the server and web packages for production.
build:plugin	pnpm --filter=plugin build	Builds the plugin package for Chrome.
build:plugin-firefox	pnpm --filter=plugin build:firefox	Builds the plugin package for Firefox.
lint	eslint .	Lints all code in the monorepo.

Sources: package.json:27-37

Conclusion

The web-archive project's monorepo structure, managed with pnpm workspaces, provides a clear and maintainable organization for its distinct but interconnected parts. The separation into plugin, server, web, and shared packages promotes modularity and code reuse. This architecture simplifies dependency management and provides a unified workflow for development, building, and linting across the entire application stack.