Shared Decoding Logic

Relevant source files

The following files were used as context for generating this wiki page:

• src/decode.ts
• src/index.ts
• scripts/trie/README.md
• scripts/write-decode-map.ts
• src/decode.spec.ts
• src/generated/decode-data-xml.ts

Shared Decoding Logic refers to the underlying architecture and algorithms used to transform HTML and XML character references back into their original characters. The system utilizes a highly optimized Radix Tree (trie) structure mapped to a Uint16Array to balance high-speed lookup with minimal memory consumption.

The logic supports various decoding modes, including strict semicolon-terminated entities for XML and legacy semicolon-optional entities for HTML. It provides both synchronous decoding for full strings and an EntityDecoder class for streaming data where entities might be split across multiple chunks.
Sources: src/decode.ts:1-74, scripts/trie/README.md:1-25

Decoding Architecture and Modes

The system distinguishes between XML and HTML entity levels. While XML follows a strict path for a small set of predefined entities, HTML utilizes a comprehensive trie-based lookup for thousands of named references.

Decoding Configuration

The behavior of the shared logic is governed by the DecodingMode and EntityLevel enums.

Option	Type	Description
EntityLevel.XML	Enum	Support only the five core XML entities (amp, lt, gt, quot, apos).
EntityLevel.HTML	Enum	Support full HTML entity set, including legacy variants.
DecodingMode.Strict	Enum	Requires all entities to be terminated with a semicolon (;).
DecodingMode.Legacy	Enum	Allows decoding of certain HTML entities without a semicolon.
DecodingMode.Attribute	Enum	Applies specific HTML spec rules for entities inside attribute values.
Sources: src/index.ts:10-64, src/decode.ts:76-88

Top-Level Flow

The main entry point decode determines the appropriate internal function based on the provided EntityLevel.

The logic differentiates between XML (always strict) and HTML (mode-dependent).
Sources: src/index.ts:66-83, src/decode.ts:707-797

Trie-Based Named Entity Lookup

Named entities are stored in a binary trie represented as a Uint16Array. This structure avoids the overhead of object allocation by using relative offsets and bit-packed headers.

Node Structure

Each node in the trie begins with a 16-bit header. The bits are allocated to provide information about value length, flags, and branch data.

• Bits 15-14: Value length (0 = none, 1 = inline, 2 = 1 word, 3 = 2 words).
• Bit 13: Dual-use flag. If a value is present, it indicates a semicolon-required (Strict) entity. If no value is present, it indicates a "Compact Run".
• Bits 12-7: Branch length or span.
• Bits 6-0: Jump table offset or the first character of a branch/run.
  Sources: scripts/trie/README.md:46-65

Compact Runs

To reduce pointer chasing in linear chains (e.g., &CounterClockwise...), the encoder collapses single-child paths into a "compact run". The header indicates the run length, and subsequent words store packed characters (two per 16-bit word).
Sources: scripts/trie/README.md:83-93, src/decode.ts:352-402

Nodes are processed iteratively by matching input characters against either branch tables or packed runs.
Sources: src/decode.ts:340-466

Numeric Entity Decoding

Numeric entities (e.g., { or {) are handled by the parseNumericEntity function. This logic supports both decimal and hexadecimal bases.

1. Prefix Detection: Identifies # followed by an optional x or X for hexadecimal.
2. Digit Accumulation: Parses digits until a non-digit character or semicolon is encountered.
3. Code Point Validation: Values exceeding 0x10FFFF are clamped to 0x110000 and eventually replaced by the replacement character U+FFFD.
4. Packing: The result is often packed into a 32-bit integer where the top 11 bits represent characters consumed and the bottom 21 bits represent the code point.
   Sources: src/decode.ts:588-640, src/decode.ts:252-286

Streaming Entity Decoder

The EntityDecoder class enables decoding of entities that are provided in chunks. It maintains an internal state machine to track progress across write calls.

Decoder State Machine

The decoder transitions through states as it consumes the entity body following the initial &.

State	Description
EntityStart	Checking for # (numeric) or alpha (named).
NumericStart	Checking for x (hex) or digits (decimal).
NumericDecimal	Consuming decimal digits.
NumericHex	Consuming hexadecimal digits.
NamedEntity	Traversing the trie for a matching name.
Sources: src/decode.ts:63-74, src/decode.ts:143-176

Sequence of Operation

The write method returns -1 if the current chunk does not contain enough data to complete or definitively reject an entity, allowing for buffered processing.
Sources: src/decode.ts:133-141, src/decode.spec.ts:178-220

Summary

The shared decoding logic provides a high-performance implementation for entity resolution. By utilizing a binary trie with optimizations like compact runs and inline values, the system minimizes memory usage. The dual-path approach ensures that simple XML entities are processed via a fast-path switch, while complex HTML entities benefit from efficient trie navigation, supporting both standard-compliant strict parsing and real-world legacy requirements.
Sources: scripts/trie/README.md:9-19, src/decode.ts:707-710