Introduction to htmlparser2

Relevant source files

The following files were used as context for generating this wiki page:

• package.json
• README.md
• src/index.ts
• src/Parser.ts
• src/Tokenizer.ts
• src/Parser.spec.ts

htmlparser2 is a fast and forgiving HTML/XML parser for Node.js and the web. Its primary purpose is to provide a low-level callback interface for consuming documents with minimal memory allocations. It is optimized for performance, making it one of the fastest parsers available in the Node.js ecosystem, though it achieves this speed by taking certain shortcuts compared to strict spec compliance.

Sources: README.md:1-12, package.json:2-15

The library serves as a foundational component for a broader ecosystem of tools, including domhandler for creating DOM trees, domutils for DOM manipulation, and css-select for CSS selector queries. It supports standard HTML, XML, and specialized feed formats such as RSS, Atom, and RDF.

Sources: README.md:19-30, src/index.ts:79-88

Core Architecture and Data Flow

The architecture of htmlparser2 is centered around a two-stage processing pipeline consisting of a Tokenizer and a Parser. Data is pushed into the system in chunks, allowing for efficient streaming of large documents.

Parsing Pipeline

1. Input Handling: Data is received via the .write() method or through a stream interface (e.g., WritableStream).
2. Tokenization: The Tokenizer reads raw character data and identifies structural boundaries such as tag starts, attribute names, and entity sequences.
3. Callback Execution: As tokens are identified, the Tokenizer triggers callbacks on the Parser.
4. Logical Processing: The Parser maintains the state of the document (e.g., the element stack) and handles higher-level logic like implicit tag closing and namespace-aware casing adjustments.

Sources: src/Parser.ts:251-274, src/Tokenizer.ts:167-175, src/Parser.ts:639-653

The following diagram illustrates the flow of data from raw input to the final event handlers:

The diagram shows how the Parser acts as a coordinator between the raw Tokenizer states and the user-provided Handler callbacks.
Sources: src/Parser.ts:316-360, src/Tokenizer.ts:675-770

Key Components

The Parser Class

The Parser is the primary entry point for the library. It implements the Callbacks interface for the tokenizer and exposes the high-level Handler interface to the user.

Sources: src/Parser.ts:219-250

Component	Description
Handler	Interface for user-defined callbacks (e.g., onopentag, ontext).
ParserOptions	Configuration for XML mode, entity decoding, and casing.
Tokenizer	A state-machine based scanner that breaks input into tokens.
Element Stack	An internal array used to track open elements and handle implicit closures.

Sources: src/Parser.ts:153-217, src/Tokenizer.ts:123-157

Implicit Logic and HTML Constraints

In HTML mode, the parser handles several automatic behaviors defined by the HTML spec, such as:

• Void Elements: Tags like <img> or <br> that do not have closing tags.
• Open-Implies-Close: Certain tags (like <li> or <p>) will automatically close specific predecessors if they are still open on the stack.
• Foreign Context: Automatic switching into SVG or MathML contexts to handle case-sensitive tags and attributes correctly.

Sources: src/Parser.ts:18-87, src/Parser.ts:394-420

Configuration Options

Users can customize the behavior of the parser through the ParserOptions object.

Option	Type	Default	Description
xmlMode	boolean	false	Enables XML-specific behaviors like CDATA and self-closing tags.
decodeEntities	boolean	true	Whether to decode entities (e.g., & to &).
lowerCaseTags	boolean	!xmlMode	Converts all tag names to lowercase.
lowerCaseAttributeNames	boolean	!xmlMode	Converts all attribute names to lowercase.
recognizeSelfClosing	boolean	xmlMode	Allows tags to close via /> in HTML mode.

Sources: src/Parser.ts:160-201, README.md:120-130

Tokenizer State Machine

The Tokenizer uses a complex state machine to process characters. It supports a wide range of states to handle different parts of HTML/XML syntax.

This diagram depicts a simplified subset of the Tokenizer states involved in parsing a standard opening tag with attributes.
Sources: src/Tokenizer.ts:44-93, src/Tokenizer.ts:771-866

Integration and Utility Methods

The src/index.ts file provides several high-level wrapper functions to simplify common parsing tasks.

Document Parsing

The parseDocument function combines the Parser with a DomHandler to return a fully constructed DOM tree.

export function parseDocument(data: string, options?: Options): Document {
    const handler = new DomHandler(undefined, options);
    new Parser(handler, options).end(data);
    return handler.root;
}

Sources: src/index.ts:32-36

Feed Parsing

The parseFeed method is specifically designed for handling RSS, Atom, and RDF feeds, ensuring that xmlMode is appropriately enabled.

Sources: src/index.ts:79-88, src/FeedHandler.spec.ts:7-35

Summary

htmlparser2 is architected as a high-performance, streaming-first parser. By decoupling tokenization from logical stack management, it provides a flexible foundation for both simple event-driven parsing and complex DOM tree construction. Its ability to handle malformed HTML while maintaining speed makes it a standard choice for web scraping and document processing in the JavaScript ecosystem.

Sources: README.md:204-220, package.json:2-10