Skip to content

feat(json): add json.null, json.decode_with_nulls and json.raw - #1088

Merged
wolfy-j merged 1 commit into
mainfrom
feat/json-null-raw
Nov 15, 2026
Merged

wolfy-j merged 1 commit into
mainfrom
feat/json-null-raw

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Defect

The json module cannot express two things JSON callers need:

  • A present key with a null value. json.decode('{"a":null}') returns {}, and Lua has no value that json.encode writes as null inside a table. A tool call or API payload with {"a": null} arrives as {}.
  • Object member order. json.encode writes table keys sorted, so a JSON document whose member order matters (for example a JSON Schema whose property order is meaningful to a model) cannot be embedded as written.

Cause

JSON null decodes to nil, and a nil table value is the absence of a key. Lua tables have no key order, and the encoder sorts keys on purpose so equal tables encode to identical bytes. Neither gap can be closed with plain tables.

The process boundary also drops any userdata: ExportPayload deep-copies function arguments, results and messages and replaces every userdata (except errors) with nil, and ToGoAny turns userdata into its tostring text. A sentinel added only to the json module would vanish at the first function call.

Fix

  • json.null: one immutable userdata shared by every Lua state (no metatable, no state, compared with ==). encode writes it as null at any depth.
  • json.decode_with_nulls(str): decodes like decode, with every JSON null decoded to json.null, so null members stay present and null array elements keep their positions; the result encodes back to the same JSON value. decode keeps its behavior: json.null is truthy, and turning every decoded null into it would change if x.field then checks in every existing caller.
  • json.raw(text): a userdata holding one valid JSON value; encode writes the text as is, member order included. The text is validated at construction (single value, unique member names, valid UTF-8, numbers in float64 range) and stored without insignificant whitespace, so top-level and nested output match. Invalid text returns nil, errors.INVALID.
  • The sentinel and raw value live in runtime/lua/engine/value (JSONNull, NewJSONRaw, JSONRawText, IsJSONValue) so the engine and the json module share one definition:
    • ExportPayload (both the copy and the immutable paths) passes them by reference; both are immutable.
    • ToGoAny converts json.null to nil (keeping the map key) and a raw value to the Go value of its JSON.
    • json.validate reads both as their JSON values, and validate/validate_string accept a raw value as the schema.
  • Type manifest and spec.md cover null, raw and decode_with_nulls.

Why raw passthrough rather than an ordered decode

An ordered decode needs a new container type (a table with key order, or a userdata with accessors), and every consumer of decoded values would have to handle it: indexing, pairs, the encoder, ToGoAny, payload export, validation. Order matters almost only when JSON is written back out, and the common case is embedding a document someone wrote (a tool or response schema) into a request. json.raw covers that with one opaque value, keeps decoded data as plain tables, and leaves the encoder's sorted-key output as it is. Code that needs to inspect the content decodes the text.

Why a shared userdata for null

go-lua has no light userdata. A single package-level *LUserData with no metatable is the closest equivalent: Lua code cannot change it (no setmetatable on userdata, no debug library in processes), it compares by identity in every state, and it costs nothing to pass between processes. A string marker (like sql.NULL's "SQL_NULL") would encode as a string through the existing userdata path.

Tests

  • runtime/lua/modules/json/null_raw_test.go: TestNullEncodesAsJSONNull, TestNullIsOneValueAcrossModuleBuilds, TestDecodeWithNullsKeepsNullMembersAndElements, TestDecodeWithNullsTopLevelNull, TestDecodeWithNullsRejectsInvalidInput, TestDecodeDropsNullMembers, TestValidateTreatsNullAsJSONNull, TestRawEncodesTextWithMemberOrder, TestRawStoresCompactText, TestRawRejectsInvalidJSON, TestValidateReadsRawAsItsJSONValue, TestValidateAcceptsRawSchema, TestNullAndRawTypes (type check).
  • runtime/lua/engine/value/json_test.go: TestToGoAnyJSONNull, TestToGoAnyJSONRaw, TestNewJSONRawCompactsAndValidates, TestIsJSONValue.
  • runtime/lua/engine/payload/json_values_test.go: TestExportPayloadKeepsJSONValues, TestProcessAndImmutabilizeKeepsJSONValues, TestLuaToGolangTranscodesJSONValues.

On main, 12 of the 13 json module tests fail (TestDecodeDropsNullMembers guards the unchanged decode). With only the new value definitions added, the engine tests fail where they should: export replaces both values with nil ({"list":[null,2]} instead of {"a":null,"list":[null,2],"schema":{"z":1,"a":2}}), and ToGoAny returns "userdata(value.jsonNull): 0x..." instead of nil.

Verification

  • go test -race passes on ./runtime/lua/modules/json/..., ./runtime/lua/engine/value/... and ./runtime/lua/engine/payload/....
  • go test -short passes on ./runtime/lua/... and ./boot/components/runtime/lua.
  • go vet and golangci-lint v2.15.0 (the make lint flags) report no issues on the touched packages.
  • The framework llm suite (2914 tests) passes against a build of this branch.

Note

Go data converted to Lua has no null members (a Go map's nil value becomes an absent key), so a value that passes through Go data, such as a msgpack transport, loses json.null. Lua-to-Lua transfer keeps it.

json.null is a shared immutable userdata that encodes as null and keeps a
key present with a null value. json.decode_with_nulls decodes JSON null to
json.null. json.raw wraps validated JSON text that encode writes as is,
member order included. Both values pass between Lua processes by reference,
convert to Go data as their JSON values, and validate as JSON.
@wolfy-j
wolfy-j requested a review from skhaz November 15, 2026 20:36
@wolfy-j
wolfy-j requested a review from skhaz November 15, 2026 20:37
@wolfy-j
wolfy-j merged commit 68867c8 into main Nov 15, 2026
39 of 40 checks passed
@wolfy-j
wolfy-j deleted the feat/json-null-raw branch November 15, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants