Code Point Handling

Relevant source files

The following files were used as context for generating this wiki page:

• src/decode-codepoint.ts
• src/decode.ts
• src/encode.ts
• src/escape.ts
• scripts/trie/README.md

Introduction

Code point handling within the entities library encompasses the logic for identifying, validating, and transforming Unicode code points during HTML and XML encoding and decoding processes. The system distinguishes between standard ASCII characters, astral plane characters (requiring surrogate pairs in UTF-16), and special character references that require remapping according to the HTML specification.

The primary goal of this module is to ensure that numeric and named character references are accurately converted to their corresponding Unicode code points while maintaining performance and security (e.g., by replacing invalid or out-of-range values with U+FFFD).

Code Point Validation and Replacement

The library implements specific logic to handle invalid Unicode ranges and C1 control characters (128–159). Following the HTML specification, certain code points are remapped to Windows-1252 equivalents or replaced with the Unicode replacement character (U+FFFD).

Validation Logic

The replaceCodePoint function serves as the central validator for all decoded numeric entities. It identifies three primary categories of "invalid" input that require substitution:

1. NUL Characters: Code point 0.
2. Surrogates: Values between 0xd800 and 0xdfff.
3. Out-of-Range: Values exceeding the maximum Unicode code point 0x10ffff.

Sources: src/decode-codepoint.ts:18-28

C1 Control Character Remapping

Code points in the range of 128 to 159 are remapped using a specific lookup table to their standard printable equivalents.

Original Code Point	Replacement (Decimal)	Description (Contextual)
128	8364	Euro Sign (€)
130	8218	Single Low-9 Quotation Mark (‚)
131	402	Latin Small Letter F with Hook (ƒ)
142	381	Latin Capital Letter Z with Caron (Ž)
159	376	Latin Capital Letter Y with Diaeresis (Ÿ)

Sources: src/decode-codepoint.ts:5-15, src/decode-codepoint.ts:30-32

Numeric Entity Parsing

Numeric entities (e.g., { or {) are parsed by extracting the numeric value from the input string and converting it into a 21-bit code point integer.

Data Flow for Numeric Decoding

The process involves detecting the base (decimal vs. hexadecimal), accumulating digits, and packing the result with metadata for the calling decoder.

The system packs the consumed length and the codePoint into a single integer to avoid object allocations. The code point is clamped to 0x110000 to prevent overflow in the 21-bit field before validation.
Sources: src/decode.ts:468-525

Multi-Code Point Entities

The library supports entities that decode into multiple code points. In the binary trie structure, these are represented by nodes with a valueLength of 3, indicating two uint16 words follow the header.

Decoding Multi-Byte Values

In the EntityDecoder class, the emitNamedEntityData method handles the emission of multiple code points by calling the emitCodePoint callback multiple times for the same consumed character count.
Sources: src/decode.ts:413-431

Encoding Multi-Byte Structures

During encoding, a specialized trie (represented by EncodeTrieNode) manages multi-character sequences. For example, the character < followed by U+20D2 is encoded as <⃒.
Sources: src/encode.ts:16-24, src/encode.ts:245-255

UTF-16 and Astral Plane Handling

Because JavaScript uses UTF-16, characters outside the Basic Multilingual Plane (BMP) are represented as surrogate pairs.

Surrogate Pair Extraction

The library provides internal helpers and uses native methods to correctly identify code points from surrogate pairs.

• String.prototype.codePointAt: Used during encoding to retrieve the full 21-bit code point from the current index, skipping the trailing surrogate if necessary.
• String.fromCodePoint: Used during decoding to convert the processed integer back into a JavaScript string.

Sources: src/encode.ts:269-272, src/decode.ts:546-550

Encoding Trie for Astral Characters

Astral entries are stored in the encoding trie as surrogate-pair nodes. Contiguous entries sharing the same high surrogate are grouped together to minimize trie depth and lookup overhead.
Sources: src/encode.ts:98-117

Implementation Summary

The following table summarizes the primary functions and variables involved in code point management:

Element	Location	Purpose
replaceCodePoint	src/decode-codepoint.ts	Validates and remaps numeric references.
parseNumericEntity	src/decode.ts	Extracts numeric values and packs them with consumption metadata.
asciiEntities	src/encode.ts	Fast lookup table for code points 0–127 to avoid Map overhead.
XML_BITSET_VALUE	src/escape.ts	Bitmask for XML special characters (", &, ', <, >).
valueLength	scripts/trie/README.md	Trie metadata determining if a node stores 1 or 2 code units.

Sources: src/decode-codepoint.ts, src/decode.ts:468, src/encode.ts:38, src/escape.ts:18, scripts/trie/README.md:57

Conclusion

Code point handling in this library is optimized for both spec-compliance and execution speed. By using binary tries for named entities and packed integers for numeric references, the system avoids costly allocations while strictly adhering to the HTML and XML standards for character substitution and surrogate pair management. Summary remapping of C1 control characters and clamping of overflow values provide a robust defense against malformed entity inputs.