fix: treat function names and keywords as member names in dot notation per RFC 9535
Merged
TR
4d ago+175+175+012345678901234567890123456789-71-71-01234567890123456789
Why
The tokenizer treated length, count, match, search and value as function names whenever any non-identifier character followed them, and treated true, false and null as keywords everywhere. That meant valid RFC 9535 queries using those words as member names were rejected:
| Query | Before | After |
|---|---|---|
$.value[0], $.value[*], $.value[?@.id == 'x'] | rejected | accepted |
$.value.id, $..value[0], $[?@.value[0] == 1] | rejected | accepted |
$.length[0], $.count.x | rejected | accepted |
$.true, $.null.x | rejected | accepted |
$.value | accepted | accepted |
$.value only worked because the end-of-input branch skipped the function check. This was reported against openapi arazzo validate in speakeasy-api/openapi#301. Microsoft Graph and OData list responses put their arrays under value, so it hits a lot of real Arazzo workflows.
What changed
- A name directly after
.or..is always tokenized as a member name, which matches RFC 9535member-name-shorthand. - Otherwise a function name only becomes a
FUNCTIONtoken when(follows it directly, matchingfunction-expr = function-name "(" ...with no whitespace allowed. parseFunctionExprnow requires aFUNCTIONtoken instead of accepting any token followed by(.
Review notes
- The parser change is needed so whitespace between a function name and
(is still rejected (compliance suitewhitespace, functions, *cases). Before, the tokenizer caught that indirectly. - Side effect worth knowing: before this change, an unknown function such as
$[?foo(@.a) > 1]parsed silently aslength(), becausefunctionTypeLengthis the zero value of the enum. It's now a parse error, which is what RFC 9535 expects. - The four
Function call ...tokenizer tests used a method-call form (@.length()), which isn't RFC 9535 syntax and never got through the parser. I rewrote them as real function calls (length(),count(search(...))) so they still coverFUNCTIONtokenization.
Testing
go build ./...andgo test ./...(including the RFC 9535 compliance suite) pass locally.- Added parser round-trip cases, tokenizer cases and query evaluation cases for function names and keywords used as member names, plus the whitespace-before-
(rejection. - Against
speakeasy-api/openapiusing a localgo.work: the repro from #301 now validates, and theoverlay,arazzoandopenapipackage tests pass. One Arazzo stress test lists an expected$.length == 5error in its ignore list. It's still rejected, but now at the==(column 9) rather than the space (column 8), so that ignore string needs updating when openapi bumps to the release with this fix.