Encoding API

Relevant source files

The following files were used as context for generating this wiki page:

• src/encode.ts
• src/encode.spec.ts
• src/escape.ts
• src/index.ts
• src/escape.spec.ts
• scripts/write-encode-map.ts

The Encoding API provides a set of highly optimized functions for converting raw strings into HTML or XML compatible formats by replacing special characters and non-ASCII characters with their corresponding entities or numeric references. The API supports various levels of encoding, from simple UTF-8 escaping to extensive HTML entity mapping. Sources: src/index.ts:1-12, readme.md

The system is designed for high performance, utilizing a combination of bitsets for fast ASCII checks, trie-based lookups for multi-code-point entities, and a hybrid scanning mechanism that switches between inline loops and regex jumps to skip "clean" spans of text efficiently. Sources: src/encode.ts:208-230, src/escape.ts:68-80

Core Encoding Functions

The API exposes several specialized functions depending on the target format and the desired character set.

Function	Description	Source
encodeHTML	Encodes all characters requiring HTML entities, including ASCII specials like #.	src/encode.ts:167-175
encodeNonAsciiHTML	Encodes non-ASCII characters and characters invalid in HTML, but keeps valid ASCII like #.	src/encode.ts:176-184
encodeXML	Encodes the five XML special characters and all non-ASCII characters.	src/escape.ts:60-109
escapeUTF8	Encodes only the five XML special characters (", &, ', <, >).	src/escape.ts:149-153
escapeAttribute	Encodes characters required for HTML attribute values (", &, \u00A0).	src/escape.ts:160-164
escapeText	Encodes characters required for HTML text nodes (&, <, >, \u00A0).	src/escape.ts:172-176

Encoding Strategy and Configuration

The top-level encode function acts as a wrapper that dispatches to specific implementations based on EncodingOptions. Sources: src/index.ts:108-142

Encoding Mode Logic

The diagram shows how the encode function selects the underlying implementation based on the provided mode and entity level. Sources: src/index.ts:119-141

Configuration Options

The EncodingOptions interface allows developers to customize the behavior:

• level: Specifies EntityLevel.XML (0) or EntityLevel.HTML (1). Sources: src/index.ts:9-13
• mode: An enum EncodingMode defining the escaping strategy (UTF8, ASCII, Extensive, Attribute, Text). Sources: src/index.ts:21-50

Internal Architecture

The Encoding Trie

For HTML encoding, the library uses a trie to handle both single and multi-code-point entities (e.g., < followed by \u20D2 becomes <⃒). Sources: src/encode.ts:9-16.

1. ASCII Fast Path: A flat asciiEntities lookup table (128 entries) stores pre-computed entities or branch nodes for characters 0–127. Sources: src/encode.ts:29-37.
2. Surrogate Pair Support: The trie handles astral characters by batch-inserting high-surrogate nodes that lead to maps of low-surrogate children. Sources: src/encode.ts:114-133.
3. Serialized Format: The trie is stored in a compact string format using delta-encoded gaps and recursive blocks {} for children. Sources: scripts/write-encode-map.ts:62-80

Hybrid Scan Mechanism

The encoding functions utilize a performance-oriented scanning algorithm to find characters that need escaping.

The scan uses an INLINE_SCAN_WINDOW of 16 characters. If no escapable character is found within this window, it switches to a native RegExp execution to skip larger clean spans of text. Sources: src/encode.ts:208-251, src/escape.ts:74-88

Implementation Details

Bitset Validation

The API uses bitsets for O(1) checks of ASCII characters. For example, the XML_BITSET_VALUE (0x50_00_00_c4) represents the five XML specials within the ASCII range 32-63. Sources: src/escape.ts:31-33. The HTML_BITSET uses four 32-bit words to cover the full 0-127 ASCII range. Sources: src/encode.ts:140-145

Surrogate Pair Handling

When a non-ASCII character is encountered, the encoder checks if it is the start of a surrogate pair using input.codePointAt(index). If a pair is detected, it is consumed as a single code point, and the index is advanced accordingly to skip the trailing surrogate. Sources: src/escape.ts:99-103, src/encode.ts:311-318

Code Example: Extensive Encoding

import { encode, EntityLevel, EncodingMode } from "entities";

// Extensive HTML encoding (encodes all non-ASCII and specials)
const result = encode("asdf & ÿ", { 
    level: EntityLevel.HTML, 
    mode: EncodingMode.Extensive 
});
// Result: "asdf & ÿ"

Sources: src/index.ts:108-142, src/encode.spec.ts:8-12

Summary

The Encoding API is a performance-critical component of the entities library. It leverages pre-compiled tries and a sophisticated scanning strategy to minimize overhead. By providing multiple entry points like escapeUTF8 for modern environments and encodeHTML for legacy compatibility, it ensures both speed and technical correctness across various web standards. Sources: src/index.ts, src/encode.ts, src/escape.ts