The JSON Tokener (Parsing)

Relevant source files

The following files were used as context for generating this wiki page:

• json_tokener.c
• json_tokener.h
• json_object.h
• json_object.c
• json_util.c
• tests/test_parse.c

The JSON Tokener is the core component of the libfastjson library responsible for converting JSON-formatted strings into a C representation of JSON objects. It implements a reference-counting object model that allows developers to construct, manipulate, and parse JSON data efficiently. The tokener is designed to be fast and supports incremental parsing, allowing large or fragmented JSON streams to be processed in chunks.

The tokener operates as a state machine that consumes characters and transitions between various internal states (e.g., parsing numbers, strings, or nested objects). While it aims for high performance, it also provides a "strict" mode for more rigorous standards compliance.

Sources: README.html:11-15, json_tokener.c:161-165, json_tokener.h:118-124

Core Architecture and Data Structures

The tokener's state is encapsulated in the fjson_tokener structure. This structure maintains the current parsing position, the nesting depth, and a stack to handle nested objects and arrays.

The Tokener Structure

The fjson_tokener structure tracks the progress of a parsing operation.

Field	Type	Description
pb	struct printbuf*	Internal buffer for gathering token characters (e.g., string content).
max_depth	int	Maximum allowed nesting depth to prevent stack overflow.
depth	int	Current nesting level during parsing.
stack	struct fjson_tokener_srec*	Array of state records for each nesting level.
err	enum fjson_tokener_error	Current error state or fjson_tokener_success.
char_offset	int	Number of characters processed in the current input string.
flags	int	Configuration flags (e.g., FJSON_TOKENER_STRICT).

Sources: json_tokener.h:74-84, json_tokener.c:116-126

State Management

The parsing logic uses a state record (fjson_tokener_srec) for every level of nesting. This allows the tokener to "remember" where it was in a parent object or array when it descends into a child element.

Parsing Logic and Flow

The primary entry point for parsing is fjson_tokener_parse_ex(). This function iterates through the input string, using a large switch statement to handle transitions between fjson_tokener_state values.

The State Machine Flow

The following diagram illustrates the high-level transition flow when the tokener encounters different JSON elements.

Incremental Parsing

fjson_tokener_parse_ex supports incremental parsing. If a string is incomplete (e.g., a closing brace is missing), the function returns NULL and sets the error to fjson_tokener_continue. The caller can then provide more data and call the function again using the same fjson_tokener instance.

Sources: json_tokener.h:124-135, tests/test_parse.c:166-175

Key API Functions

The tokener provides several functions for lifecycle management and parsing.

Function	Description
fjson_tokener_new()	Allocates a new tokener with default depth (32).
fjson_tokener_reset()	Resets the tokener state, allowing reuse for a new JSON string.
fjson_tokener_parse()	High-level wrapper that parses a null-terminated string.
fjson_tokener_parse_ex()	The core parsing engine; accepts a pointer and length.
fjson_tokener_get_error()	Returns the error code from the last parsing operation.

Sources: json_tokener.h:92-116, json_tokener.c:100-114

Error Handling

When parsing fails, the tokener sets an error code of type enum fjson_tokener_error. Descriptions can be retrieved via fjson_tokener_error_desc().

Error Code	Meaning
fjson_tokener_success	Parsing finished successfully.
fjson_tokener_continue	More data is needed to complete the JSON object.
fjson_tokener_error_depth	The JSON structure exceeds the maximum allowed nesting depth.
fjson_tokener_error_parse_unexpected	Encountered a character that does not match JSON syntax.
fjson_tokener_error_parse_null	"null" was expected but not found.
fjson_tokener_error_parse_string	Invalid character sequence within a string.

Sources: json_tokener.h:20-36, json_tokener.c:48-64

Implementation Details

String and Unicode Handling

The tokener handles escape sequences (e.g., \n, \t, \") and Unicode escape sequences (\uXXXX). For Unicode, it supports decoding surrogate pairs into UTF-8. If an invalid surrogate pair is found, it appends the UTF-8 replacement character 0xEF 0xBF 0xBD.

Sources: json_tokener.c:85-88, json_tokener.c:441-480

Number Parsing

Number parsing is performed by gathering valid numeric characters (0-9, ., e, E, +, -) into a temporary buffer and then using fjson_parse_int64 or fjson_parse_double. The library includes workarounds for broken sscanf implementations in some environments to ensure numeric accuracy.

Sources: json_tokener.c:575-610, json_util.c:164-185

Utility Functions

JSON objects can also be parsed directly from file descriptors or files using utility wrappers that utilize the tokener internally.

Conclusion

The JSON Tokener in libfastjson provides a robust, state-based parsing mechanism capable of handling both simple and complex nested JSON structures. Through its support for incremental parsing and explicit state management, it offers the flexibility required for high-performance applications that process JSON data from various streaming or static sources.