Entity Data Maps
Relevant source files
The following files were used as context for generating this wiki page:
Entity Data Maps represent the core knowledge base of the entities library, containing the definitions for XML and HTML character references. These maps are initially stored as JSON documents and then transformed into highly optimized, binary-packed Radix Trees (Tries) for high-performance encoding and decoding operations.
The library supports different "levels" of entity support, primarily XML and HTML. While XML defines a small set of five essential entities, HTML includes a vast superset of named character references, including legacy entities that do not strictly require a trailing semicolon.
Sources: src/index.ts:10-14, readme.md:43-52, scripts/trie/README.md:1-10
Data Source Hierarchy
The project categorizes entity data into three primary JSON maps located in the /maps directory. These serve as the ground truth for all generated binary structures.
| Map File | Purpose | Characteristics |
|---|---|---|
xml.json | Core XML entities | Contains 5 basic entities (amp, lt, gt, quot, apos). |
entities.json | Standard HTML5 entities | Comprehensive list of named character references defined by WHATWG. |
legacy.json | Semicolon-optional entities | Entities that can be decoded without a trailing semicolon for backward compatibility. |
Sources: maps/xml.json, maps/entities.json, maps/legacy.json, scripts/write-decode-map.ts:51-53
Trie Generation Architecture
To minimize memory consumption and maximize traversal speed, the raw JSON maps are processed through a build-time pipeline that converts them into Uint16Array structures.
The Transformation Pipeline
The process begins by building a logical trie where each node represents a character in an entity name. The generator identifies strict versus legacy entities: strict entities set a semiRequired flag, while legacy entities are represented as nodes where the semicolon is optional.
Sources: scripts/trie/trie.ts:16-52, scripts/write-decode-map.ts:220-234
Decode Map Structure
The decoding maps are optimized for a state-machine based traversal. Nodes are stored in a single Uint16Array, with each node starting with a 16-bit header word.
Node Header Bit Layout
The header uses bit-packing to store metadata efficiently:
- Bits 15-14: Value length (0: None, 1: Inlined, 2: 1-word, 3: 2-words).
- Bit 13: Dual-use flag (Semicolon-required if value exists; otherwise Compact Run flag).
- Bits 12-7: Branch length or span.
- Bits 6-0: Jump table offset or first character of a run.
Sources: scripts/trie/README.md:58-75
Compact Runs
To reduce pointer chasing, linear chains of at least three single-child nodes are collapsed into "Compact Runs." The characters are packed two per uint16 word following the header. This significantly reduces the size of the trie for long, non-branching entity names like ∳.
Sources: scripts/trie/README.md:95-104, src/decode.ts:251-285
Encode Map Optimization
While the decoding map is keyed by entity names (e.g., "amp"), the encoding map is keyed by the resulting characters or code points.
Encoding Data Flow
The encoding process uses a different optimization strategy to handle ASCII vs. Astral code points:
Fast-Path Table
For ASCII characters (0-127), the library uses a flat asciiEntities lookup table. This avoids the overhead of Map hashing for common characters like < or &. If an ASCII character is the start of a multi-code-point entity (e.g., < followed by \u20D2), the table points to a branch node to continue the search.
Sources: src/encode.ts:24-40, src/encode.ts:205-230
Dictionary Compression
For large tries (HTML), the library employs a dictionary-based compression during the serialization of the binary data.
- Dict1: The 61 most frequent values are assigned 1-character codes.
- Dict2: Remaining unique values are assigned 2-character codes.
- Delta Encoding: Values in the header are delta-encoded and run-length encoded (RLE) using a base-91 safe ASCII set to further reduce file size before Gzip.
Sources: scripts/write-decode-map.ts:28-48, scripts/write-decode-map.ts:114-145
Summary of Entity Levels
The following table summarizes how the data maps are applied based on the configured EntityLevel:
| Level | Decoder Logic | Encoder Logic |
|---|---|---|
| XML | Uses xmlDecodeTree; always strict (semicolon required). Includes fast-path for 5 core entities. | Escapes &, <, >, ", ' and non-ASCII characters. |
| HTML | Uses htmlDecodeTree; supports Legacy, Strict, and Attribute modes. | Extensive mapping using htmlTrie; handles multi-code-point entities. |
Sources: src/index.ts:74-118, src/decode.ts:503-575, src/encode.ts:138-155