Feature: Web Clipper Extension
Relevant source files
The following files were used as context for generating this wiki page:
The Web Clipper is a browser extension designed to capture, highlight, and save content from the web directly into the Nucleus ecosystem. It supports clipping entire web pages, highlighting specific text passages, and parsing structured content from specific sites like Twitter. The extension is a key part of the "Capture" and "Clipping" feature sets, enabling users to gather information from various online sources with minimal friction.
The clipper's functionality includes web clipping, video clipping from platforms like YouTube, and social media clipping from sites such as Twitter and Reddit. It aims to provide a seamless experience for saving articles, persisting highlights on pages, and adding notes to captured content. This functionality is central to the knowledge management capabilities of the associated applications like Memotron.
Sources: client/landing/routes/compare/memotronWheel.ts:500-598
Architecture
The Web Clipper extension operates through a combination of background scripts, content scripts, and specialized parsers. These components work together to interact with web pages, process content, and communicate with the browser's extension framework.
The following diagram illustrates the high-level architecture of the Web Clipper extension.
Content Script (highlightV4.js):* Injected into web pages. It is responsible for direct DOM manipulation, such as creating and displaying text highlights, and for initiating the content parsing process.
- Parser Module (
parsers/): Contains the logic for extracting structured data from specific websites. A central dispatcher (parsers/index.ts) selects the appropriate parser based on the current page's URL. - Background Script: (Not present in the provided files, but its role is implied by the overall structure) Manages the extension's state, handles communication between components, and interacts with browser APIs for features like context menus.
- Configuration (
clipper.config.ts): Provides static configuration values used by various parts of the extension, such as highlight colors and CSS class names.
Sources: client/extensions/clipper/contentScripts/highlightV4.js, client/extensions/clipper/parsers/index.ts, client/extensions/clipper/clipper.config.ts
Configuration
The clipper's behavior is defined by a set of constants in clipper.config.ts. This file centralizes values for CSS classes, highlight colors, and other operational parameters.
| Constant | Description | Default Value |
|---|---|---|
CLIP_DELAY | Delay in milliseconds before performing a clip action. | 100 |
HIGHLIGHT_CLASS_NAME | The base CSS class name applied to all highlights. | memotron-highlight |
HIGHLIGHT_COLORS | An array of color objects available for highlighting. | [{ id: "yellow", hex: "#FFF8A1" }, ...] |
HIGHLIGHT_TAG_NAME | The HTML tag used to wrap highlighted content. | memotron-highlight |
HIGHLIGHTER_ID_PREFIX | A prefix for generating unique IDs for highlight elements. | memotron-highlighter- |
Sources: client/extensions/clipper/clipper.config.ts:1-21
Content Parsing
A core feature of the clipper is its ability to parse structured content from specific websites. This is managed by a modular parser system.
Parser Dispatching
The getParser function in parsers/index.ts acts as a dispatcher. It takes the window's location object and determines which, if any, specialized parser should be used for the current page. If no specific parser matches, it returns a defaultParser.
This flow shows how a parser is selected.
Case Study: Twitter Parser
The twitterParser.ts provides a concrete example of a site-specific parser. It is designed to extract data from individual tweet pages.
Key Functions
| Function | Description |
|---|---|
isTweetPage | Checks if the current URL corresponds to a single tweet page. |
getTweet | The main function that orchestrates the extraction of tweet data from the DOM. |
getTweetAuthor | Extracts the author's name, handle, and avatar URL. |
getTweetText | Retrieves the main text content of the tweet. |
getTweetMedia | Finds and extracts URLs for any images or videos in the tweet. |
getTweetTimestamp | Gets the tweet's creation date and time. |
Sources: client/extensions/clipper/parsers/twitterParser.ts
Data Extraction Flow
The getTweet function follows a specific sequence to gather all relevant information from a tweet's DOM structure.
This diagram shows the sequence of internal function calls within twitterParser to build a complete tweet object from the webpage's DOM.
Sources: client/extensions/clipper/parsers/twitterParser.ts:121-143
Text Highlighting
The content script highlightV4.js is responsible for all text highlighting functionality on the page. It uses the rangy library for robust cross-browser range and selection handling.
Core Highlighting Logic
The script allows for creating, removing, serializing, and deserializing highlights.
- Highlight Creation: When a user selects text, the
highlightfunction is called. It wraps the selected text range with a custom HTML element (<memotron-highlight>) and applies the appropriate color and class fromclipper.config.ts. - Serialization: The
serializefunction converts a highlight object into a string representation. This string contains the path to the highlighted text within the DOM (using an XPath-like format), start and end offsets, and the text content itself. This allows the highlight to be stored and recreated later. - Deserialization: The
deserializefunction takes a serialized string and attempts to recreate the highlight on the page by finding the original text nodes and re-applying the highlight wrapper.
The process of creating and serializing a highlight is shown below.
Competitor Feature Comparison
The highlighting feature is a key differentiator. The project tracks how its implementation compares to competitors like "Web Highlights". The goal is to support offline highlighting, persistence across page loads, and PDF annotation.
- Web Highlights: Noted for its specialization in highlighting any website or PDF, with offline support and persistence.
- Memotron (Goal): Aims for similar capabilities, including quick saving of webpages, persistent text highlights, and the ability to add notes.
Sources: client/cx/contemporaries/web-highlights.json:28-32, client/landing/routes/compare/memotronWheel.ts:510-530