Node.js WritableStream

Relevant source files

The following files were used as context for generating this wiki page:

• src/WritableStream.ts
• src/WritableStream.spec.ts
• src/Parser.ts
• src/Tokenizer.ts
• src/WebWritableStream.ts
• package.json

The WritableStream class in htmlparser2 is a specialized implementation of the Node.js Writable stream interface. Its primary purpose is to adapt the core Parser logic into a standard streaming pipeline, allowing developers to pipe data directly from sources like fs.createReadStream into the parser. This implementation handles buffer decoding and state management to ensure that fragmented data—such as multi-byte Unicode characters split across chunks—is correctly processed.

This module acts as a wrapper around the Parser class, facilitating high-performance HTML/XML parsing within the Node.js ecosystem. While the Parser class itself provides a .write() method, the WritableStream ensures compatibility with the broader Node.js stream API, including events like finish and the .pipe() method.

Sources: src/WritableStream.ts:16-19, README.md:144-156

Architecture and Data Flow

The WritableStream architecture leverages a delegation pattern where the stream interface manages Node.js-specific overhead (decoding and buffering) while the internal Parser instance handles the actual tokenization and tree construction.

Internal Components

• Parser Instance: An internal Parser is initialized during construction with the provided handler callbacks and options.
• StringDecoder: A node:string_decoder is used to maintain state between chunks, specifically to handle fragmented UTF-8 sequences.
• _write implementation: Overrides the standard Writable._write method to pass decoded strings to the internal parser.

The diagram shows how raw input buffers are decoded into strings before being processed by the internal parser and tokenizer.

Sources: src/WritableStream.ts:17-43, src/Parser.ts:250-256

Implementation Details

The WritableStream extends the Node.js Writable base class. It is configured with decodeStrings: false to manually handle buffer decoding using the StringDecoder, which prevents corruption of multi-byte characters.

Key Methods

Method	Description
constructor(cbs, options)	Initializes the internal Parser and sets up the stream state.
_write(chunk, encoding, callback)	Processes incoming data. If the chunk is a Buffer, it uses _decoder.write() to transform it into a string.
_final(callback)	Called when the stream is ending. It flushes the remaining bytes from the decoder and calls parser.end().

export class WritableStream extends Writable {
    private readonly _parser: Parser;
    private readonly _decoder = new StringDecoder();

    constructor(cbs: Partial<Handler>, options?: ParserOptions) {
        super({ decodeStrings: false });
        this._parser = new Parser(cbs, options);
    }

    override _write(
        chunk: string | Buffer,
        encoding: string,
        callback: () => void,
    ): void {
        this._parser.write(
            isBuffer(chunk, encoding) ? this._decoder.write(chunk) : chunk,
        );
        callback();
    }
}

Sources: src/WritableStream.ts:17-38, src/WritableStream.spec.ts:10-21

Unicode Handling and Fragmentation

A critical feature of the WritableStream is its ability to handle fragmented Unicode characters. In streaming environments, a multi-byte character (like the Euro symbol €, 0xe2 0x82 0xac) might be split across two separate .write() calls. The StringDecoder ensures these fragments are buffered until a complete character is formed before passing it to the Parser.

The sequence diagram illustrates how fragmented bytes are held by the decoder until a valid character is available for the parser.

Sources: src/WritableStream.ts:33-36, src/WritableStream.spec.ts:10-21

Comparison with WebWritableStream

The project also provides a WebWritableStream which implements the Web Streams API. While the Node.js WritableStream uses node:stream and StringDecoder, the web version uses the TextDecoder and the native WritableStream interface available in modern runtimes (Node.js ≥18.0, Deno, and browsers).

Key Differences

Feature	Node.js WritableStream	WebWritableStream
Base Class	node:stream.Writable	Native WritableStream
Decoding	node:string_decoder	TextDecoder
Entry Point	htmlparser2/WritableStream	htmlparser2/WebWritableStream
Engine Support	Node.js (All)	Node.js ≥18, Browser, Deno

Sources: src/WebWritableStream.ts:25-50, package.json:35-42

Summary

The Node.js WritableStream provides a robust, stream-compliant entry point for htmlparser2. By wrapping the core Parser and utilizing the StringDecoder, it ensures that large HTML or XML documents can be processed efficiently and safely as they arrive over the network or from the file system, without losing data at chunk boundaries.

Sources: src/WritableStream.ts