Locale and Character Encoding Tests
Relevant source files
The following files were used as context for generating this wiki page:
Locale and Character Encoding Tests in libfastjson ensure that the library consistently parses and generates JSON data regardless of the system's regional settings or the complexity of Unicode input. These tests verify that floating-point numbers are handled correctly in locales where the decimal separator is not a period, and that UTF-8 sequences—including complex surrogate pairs—are accurately decoded and encoded.
The system relies on a combination of internal state management during parsing and a set of validation tests to maintain technical accuracy across different environments. Key areas of focus include LC_NUMERIC locale switching, Unicode escape sequence decoding (\uXXXX), and strict enforcement of character casing for JSON literals like true, false, and null.
Locale Management and Numeric Parsing
The library implements a locale-independent approach to parsing numeric values. Since the JSON standard requires a period (.) as a decimal separator, but many system locales use a comma (,), the library temporarily overrides the system's LC_NUMERIC setting during the parsing process.
Locale Switching Logic
When fjson_tokener_parse_ex is invoked, it captures the current locale and switches the LC_NUMERIC category to "C" to ensure consistent behavior of functions like sscanf and strtod.
The sequence ensures that internal calls to numeric conversion utilities do not produce unexpected results based on the host environment.
Sources: json_tokener.c:318-326, json_tokener.c:685-688
Numeric Consistency Features
| Feature | Implementation | File |
|---|---|---|
| Integer Range | Checks for ERANGE and handles INT64_MIN/MAX | json_util.c:165-225 |
| Double Precision | Uses %.17g for formatting to maintain precision | json_print.c:482 |
| Decimal Handling | Explicitly replaces , with . if localization leaks into output | json_print.c:489-495 |
Character Encoding and Unicode Handling
The library supports UTF-8 as its primary character encoding. The parsing logic specifically handles the transition from escaped Unicode sequences to raw UTF-8 bytes.
Unicode Decoding Process
The fjson_tokener handles \uXXXX sequences through the fjson_tokener_state_escape_unicode state. This state includes logic to decode 16-bit hex values and handle UTF-16 surrogate pairs.
The decoding logic handles the complexity of Unicode mapping, including error recovery by inserting the UTF-8 replacement character (0xEF 0xBF 0xBD) when invalid sequences or orphaned surrogates are encountered.
Sources: json_tokener.c:485-610, json_tokener.c:115
Binary and Null Character Support
The library supports strings containing embedded null characters (\0). Tests confirm that fjson_object_new_string_len accurately tracks the length of binary strings, and the tokener can re-parse these if they are represented as \u0000.
Sources: tests/test_null.c:14-25, tests/test_null.c:40-50
Character Case Sensitivity
In adherence to the JSON specification, particularly when FJSON_TOKENER_STRICT flags are enabled, libfastjson enforces case sensitivity for boolean and null literals.
Strict Mode Enforcement
Tests verify that title-cased or upper-cased literals (e.g., True, False, Null) are rejected when the strict flag is set.
/* tests/test_charcase.c:30-45 */
tok = fjson_tokener_new();
fjson_tokener_set_flags(tok, FJSON_TOKENER_STRICT);
new_obj = fjson_tokener_parse_ex(tok, "True", 4);
CHK(new_obj == NULL);
new_obj = fjson_tokener_parse_ex(tok, "False", 5);
CHK(new_obj == NULL);
Sources: tests/test_charcase.c:26-45
String Escaping and Output Validation
The library uses an optimized lookup table for string escaping to ensure that control characters and special symbols are safely encoded during serialization.
Escape Table Logic
The array char_needsEscape is used as a high-speed filter. Characters requiring special handling (like backslashes, quotes, and control characters) are identified through a single lookup.
| Character | Escaped Result | Requirement |
|---|---|---|
" | \" | Mandatory |
\ | \\ | Mandatory |
\b | \b | Control Char |
\n | \n | Control Char |
\r | \r | Control Char |
\t | \t | Control Char |
< 0x20 | \u00XX | Hex Escape |
Sources: json_print.c:245-285, json_print.c:303-317
Summary
The Locale and Character Encoding tests ensure that libfastjson remains a robust tool for cross-platform JSON processing. By strictly managing the LC_NUMERIC locale and providing a comprehensive state machine for Unicode decoding, the library guarantees that data integrity is maintained regardless of regional system configurations or complex international text input.