fix: treat function names and keywords as member names in dot notation per RFC 9535

Merged
TR
4d ago
+175-71

Why

The tokenizer treated length, count, match, search and value as function names whenever any non-identifier character followed them, and treated true, false and null as keywords everywhere. That meant valid RFC 9535 queries using those words as member names were rejected:

QueryBeforeAfter
$.value[0], $.value[*], $.value[?@.id == 'x']rejectedaccepted
$.value.id, $..value[0], $[?@.value[0] == 1]rejectedaccepted
$.length[0], $.count.xrejectedaccepted
$.true, $.null.xrejectedaccepted
$.valueacceptedaccepted

$.value only worked because the end-of-input branch skipped the function check. This was reported against openapi arazzo validate in speakeasy-api/openapi#301. Microsoft Graph and OData list responses put their arrays under value, so it hits a lot of real Arazzo workflows.

What changed

  • A name directly after . or .. is always tokenized as a member name, which matches RFC 9535 member-name-shorthand.
  • Otherwise a function name only becomes a FUNCTION token when ( follows it directly, matching function-expr = function-name "(" ... with no whitespace allowed.
  • parseFunctionExpr now requires a FUNCTION token instead of accepting any token followed by (.

Review notes

  • The parser change is needed so whitespace between a function name and ( is still rejected (compliance suite whitespace, functions, * cases). Before, the tokenizer caught that indirectly.
  • Side effect worth knowing: before this change, an unknown function such as $[?foo(@.a) > 1] parsed silently as length(), because functionTypeLength is the zero value of the enum. It's now a parse error, which is what RFC 9535 expects.
  • The four Function call ... tokenizer tests used a method-call form (@.length()), which isn't RFC 9535 syntax and never got through the parser. I rewrote them as real function calls (length(), count(search(...))) so they still cover FUNCTION tokenization.

Testing

  • go build ./... and go test ./... (including the RFC 9535 compliance suite) pass locally.
  • Added parser round-trip cases, tokenizer cases and query evaluation cases for function names and keywords used as member names, plus the whitespace-before-( rejection.
  • Against speakeasy-api/openapi using a local go.work: the repro from #301 now validates, and the overlay, arazzo and openapi package tests pass. One Arazzo stress test lists an expected $.length == 5 error in its ignore list. It's still rejected, but now at the == (column 9) rather than the space (column 8), so that ignore string needs updating when openapi bumps to the release with this fix.