Stream Processing Patterns

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• src/WritableStream.ts
• src/WebWritableStream.ts
• src/Parser.ts
• src/Tokenizer.ts
• src/WritableStream.spec.ts
• src/WebWritableStream.spec.ts

Stream processing in htmlparser2 is designed to handle large HTML/XML documents by processing data in chunks rather than loading entire files into memory. This approach minimizes allocations and allows for high-performance parsing of live data feeds, such as RSS, Atom, and RDF. The library provides specialized wrappers for both Node.js Streams and the Web Streams API, ensuring compatibility across different runtimes.

The core of the streaming architecture involves a Parser that coordinates with a Tokenizer. The Parser manages stateful aspects like tag stacks and foreign contexts (SVG/MathML), while the Tokenizer handles the low-level character scanning. Input can be fed into this system through standard write and end calls, or via piped stream interfaces.

Sources: README.md:82-85, src/Parser.ts:250-256, package.json:4-13

Core Streaming Interfaces

The project provides three primary ways to interact with streaming data, depending on the environment and specific use case.

Node.js WritableStream

The WritableStream class extends Node.js's native Writable class. It uses a StringDecoder to handle potential fragmented multi-byte characters (like Unicode) that might be split across different data chunks.

Sources: src/WritableStream.ts:16-20, src/WritableStream.spec.ts:11-21

Web WritableStream

For modern environments (browsers, Deno, Node.js 18+), WebWritableStream implements the Web Streams API. This is particularly useful for piping fetch() response bodies directly into the parser.

Sources: src/WebWritableStream.ts:20-22, src/WebWritableStream.spec.ts:63-78

Manual Parser Injection

The Parser class itself exposes write(chunk: string) and end(chunk?: string) methods, allowing developers to manually push data into the state machine without a formal stream wrapper.

Sources: src/Parser.ts:708-735, README.md:46-70

Architecture and Data Flow

The data flow starts with an external source (like a file or network socket) and moves through decoders before reaching the tokenization logic.

Stream Control and State Management

Streaming requires the ability to handle backpressure or wait for external resources. The Parser and Tokenizer support pausing and resuming the flow.

Pausing and Resuming

When parser.pause() is called, the underlying tokenizer stops processing the buffer. Any subsequent calls to parser.write() will buffer the chunks internally. Calling parser.resume() flushes these buffered chunks into the tokenizer.

Buffer Handling

The Parser maintains an internal buffers array. As the Tokenizer consumes data, the Parser shifts these buffers to free memory. This is handled in the getSlice method, which identifies the correct buffer based on a bufferOffset.

Sources: src/Parser.ts:672-698

Component Summary

Component	Responsibility	Relevant Files
Parser	High-level state, tag stack management, foreign context detection.	src/Parser.ts
Tokenizer	Low-level state machine, character scanning, entity decoding.	src/Tokenizer.ts
WritableStream	Node.js stream compatibility using StringDecoder.	src/WritableStream.ts
WebWritableStream	Web Streams API compatibility using TextDecoder.	src/WebWritableStream.ts
Handler	Interface for receiving events like onopentag, ontext, etc.	src/Parser.ts

Sources: src/Parser.ts:164-245, src/Tokenizer.ts:145-163, src/WritableStream.ts:16-43

Technical Patterns in Tokenization

The Tokenizer is a state-driven scanner that operates on strings. It tracks its position via an index and uses a sectionStart to mark the beginning of the current token (e.g., a tag name or text node).

Fragmented Unicode Handling

A critical pattern in the streaming implementation is the handling of fragmented Unicode characters. In both WritableStream and WebWritableStream, decoders are configured to maintain state between chunks.

• Node.js: StringDecoder handles buffer fragments.
• Web: TextDecoder uses the { stream: true } option.

Sources: src/WritableStream.ts:33-35, src/WebWritableStream.ts:32-35, src/WebWritableStream.spec.ts:10-22

Entity Decoding in Streams

The Tokenizer utilizes an EntityDecoder from the entities package. When an ampersand (&) is encountered, the tokenizer enters the InEntity state. If the entity is fragmented across chunks, the EntityDecoder maintains its internal state until the entity is fully resolved or determined to be plain text.

Sources: src/Tokenizer.ts:585-612, src/Tokenizer.spec.ts:255-279

Conclusion

Stream processing in htmlparser2 is achieved through a combination of environment-specific writable stream wrappers and a robust internal buffering system. By separating character-level tokenization from high-level tree logic and providing built-in support for pausing/resuming, the library enables efficient processing of data-heavy applications like web crawlers and feed aggregators.

Sources: README.md:144-159, package.json:2-10