Tokenization Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Tokenization Engine is the low-level core of htmlparser2, responsible for breaking down a stream of characters into discrete meaningful tokens such as tag names, attribute names, attribute values, text, comments, and declarations. It operates as a state machine that processes input incrementally, allowing it to handle data chunks as they arrive from a stream without needing the entire document in memory.
The engine is primarily implemented in the Tokenizer class, which communicates with a higher-level consumer (typically the Parser) via a set of predefined callbacks. It supports both standard HTML parsing and a strict xmlMode for processing XML, RSS, and Atom feeds.
Sources: src/Tokenizer.ts:211-236, README.md, src/Parser.ts:251-267
Architecture and State Machine
The Tokenizer is built around an extensive internal state machine. As characters are read from the buffer, the engine transitions between states to determine the context of the data.
Core States
The tokenizer uses a State enum to track its progress through the document. Key states include:
Text: The default state for character data.BeforeTagName: Triggered after encountering a<character.InTagName: Currently reading a tag name.BeforeAttributeName: Transitioned to after a tag name and whitespace.InAttributeValueDq/Sq/Nq: Reading attribute values inside double quotes, single quotes, or unquoted.InCommentLike: Handling comments and CDATA sections.
Sources: src/Tokenizer.ts:51-92
State Transition Flow
The following diagram illustrates the high-level transitions between primary states when parsing a standard tag.
This diagram shows the transition from raw text into tags, through attribute parsing, and back to text.
Sources: src/Tokenizer.ts:290-302, src/Tokenizer.ts:538-586, src/Tokenizer.ts:635-655
Core Components and Logic
Callbacks Interface
The Tokenizer does not return a tree; instead, it emits events through the Callbacks interface. The Parser implements these callbacks to build the higher-level representation.
| Callback | Purpose |
|---|---|
ontext | Emits a range of indices representing plain text. |
onopentagname | Emitted when a start tag name is identified. |
onattribname | Emitted when an attribute name is identified. |
onattribdata | Emitted for segments of attribute values. |
onclosetag | Emitted when a closing tag name is identified. |
oncomment | Emitted for comment content. |
Sources: src/Tokenizer.ts:121-139
Special Tag Handling
The engine contains specialized logic for "text-only" tags like <script>, <style>, <title>, and <textarea>. When these tags are opened, the engine enters a SpecialStartSequence state. It then looks for a specific end-sequence (e.g., </script) while treating everything in between as raw text or RCDATA (where entities are decoded but tags are not).
Sources: src/Tokenizer.ts:145-168, src/Tokenizer.ts:333-380
Entity Decoding
The Tokenizer integrates with the entities library to decode HTML and XML entities (e.g., &). It uses a baseState to remember where it was before it started decoding an entity, allowing it to return to either a Text state or an AttributeValue state once decoding is finished.
Sources: src/Tokenizer.ts:241-248, src/Tokenizer.ts:749-780
Data Flow: Streaming and Buffering
The engine is designed for incremental processing. The write(chunk) method appends new data and immediately triggers the parse() loop.
This sequence shows how data flows from an external source through the streaming interface into the core tokenization logic.
Sources: src/Tokenizer.ts:262-266, src/Parser.ts:912-924, src/WritableStream.ts:29-38
Configuration Options
The Tokenization Engine behavior is modified by the ParserOptions passed during construction.
| Option | Type | Default | Description |
|---|---|---|---|
xmlMode | boolean | false | If true, uses XML rules for entities, CDATA, and self-closing tags. |
decodeEntities | boolean | true | Whether to decode entities like >. |
recognizeSelfClosing | boolean | xmlMode | If true, tags ending in /> trigger a close event immediately. |
Sources: src/Tokenizer.ts:232-236, src/Parser.ts:193-241
Implementation Details: Index Management
The Tokenizer tracks positions using three main indices:
index: The current character being processed in the current buffer.sectionStart: The beginning of the current token segment.offset: The total number of characters processed before the current buffer.
When a callback is triggered, it provides the absolute indices (including the offset), which allows the Parser to calculate the exact position of elements in the document.
Sources: src/Tokenizer.ts:215-227, src/Parser.ts:285-294
Example: Handling Fragmented Data
If a tag name is split across two chunks (e.g., <di and v>), the Tokenizer maintains its InTagName state and sectionStart. When the second chunk arrives, it continues reading until the tag name ends, then calls onopentagname with the full range.
Sources: src/Tokenizer.ts:782-800, src/Parser.spec.ts:127-140
Conclusion
The Tokenization Engine provides a high-performance, low-level streaming interface for HTML and XML. By utilizing a robust state machine and index-based data slicing, it achieves fast parsing speeds while maintaining the ability to handle malformed markup and fragmented input streams. It serves as the foundation upon which the Parser builds higher-level features like DOM construction and feed parsing.