Repository navigation
feat(json): add json.null, json.decode_with_nulls and json.raw - #1088
Merged
Merged
Conversation
json.null is a shared immutable userdata that encodes as null and keeps a key present with a null value. json.decode_with_nulls decodes JSON null to json.null. json.raw wraps validated JSON text that encode writes as is, member order included. Both values pass between Lua processes by reference, convert to Go data as their JSON values, and validate as JSON.
skhaz
approved these changes
Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defect
The
jsonmodule cannot express two things JSON callers need:json.decode('{"a":null}')returns{}, and Lua has no value thatjson.encodewrites asnullinside a table. A tool call or API payload with{"a": null}arrives as{}.json.encodewrites table keys sorted, so a JSON document whose member order matters (for example a JSON Schema whose property order is meaningful to a model) cannot be embedded as written.Cause
JSON null decodes to
nil, and aniltable value is the absence of a key. Lua tables have no key order, and the encoder sorts keys on purpose so equal tables encode to identical bytes. Neither gap can be closed with plain tables.The process boundary also drops any userdata:
ExportPayloaddeep-copies function arguments, results and messages and replaces every userdata (except errors) withnil, andToGoAnyturns userdata into itstostringtext. A sentinel added only to the json module would vanish at the first function call.Fix
json.null: one immutable userdata shared by every Lua state (no metatable, no state, compared with==).encodewrites it asnullat any depth.json.decode_with_nulls(str): decodes likedecode, with every JSON null decoded tojson.null, so null members stay present and null array elements keep their positions; the result encodes back to the same JSON value.decodekeeps its behavior:json.nullis truthy, and turning every decoded null into it would changeif x.field thenchecks in every existing caller.json.raw(text): a userdata holding one valid JSON value;encodewrites the text as is, member order included. The text is validated at construction (single value, unique member names, valid UTF-8, numbers in float64 range) and stored without insignificant whitespace, so top-level and nested output match. Invalid text returnsnil, errors.INVALID.runtime/lua/engine/value(JSONNull,NewJSONRaw,JSONRawText,IsJSONValue) so the engine and the json module share one definition:ExportPayload(both the copy and the immutable paths) passes them by reference; both are immutable.ToGoAnyconvertsjson.nulltonil(keeping the map key) and a raw value to the Go value of its JSON.json.validatereads both as their JSON values, andvalidate/validate_stringaccept a raw value as the schema.spec.mdcovernull,rawanddecode_with_nulls.Why raw passthrough rather than an ordered decode
An ordered decode needs a new container type (a table with key order, or a userdata with accessors), and every consumer of decoded values would have to handle it: indexing,
pairs, the encoder,ToGoAny, payload export, validation. Order matters almost only when JSON is written back out, and the common case is embedding a document someone wrote (a tool or response schema) into a request.json.rawcovers that with one opaque value, keeps decoded data as plain tables, and leaves the encoder's sorted-key output as it is. Code that needs to inspect the content decodes the text.Why a shared userdata for null
go-lua has no light userdata. A single package-level
*LUserDatawith no metatable is the closest equivalent: Lua code cannot change it (nosetmetatableon userdata, nodebuglibrary in processes), it compares by identity in every state, and it costs nothing to pass between processes. A string marker (likesql.NULL's"SQL_NULL") would encode as a string through the existing userdata path.Tests
runtime/lua/modules/json/null_raw_test.go:TestNullEncodesAsJSONNull,TestNullIsOneValueAcrossModuleBuilds,TestDecodeWithNullsKeepsNullMembersAndElements,TestDecodeWithNullsTopLevelNull,TestDecodeWithNullsRejectsInvalidInput,TestDecodeDropsNullMembers,TestValidateTreatsNullAsJSONNull,TestRawEncodesTextWithMemberOrder,TestRawStoresCompactText,TestRawRejectsInvalidJSON,TestValidateReadsRawAsItsJSONValue,TestValidateAcceptsRawSchema,TestNullAndRawTypes(type check).runtime/lua/engine/value/json_test.go:TestToGoAnyJSONNull,TestToGoAnyJSONRaw,TestNewJSONRawCompactsAndValidates,TestIsJSONValue.runtime/lua/engine/payload/json_values_test.go:TestExportPayloadKeepsJSONValues,TestProcessAndImmutabilizeKeepsJSONValues,TestLuaToGolangTranscodesJSONValues.On main, 12 of the 13 json module tests fail (
TestDecodeDropsNullMembersguards the unchangeddecode). With only the newvaluedefinitions added, the engine tests fail where they should: export replaces both values withnil({"list":[null,2]}instead of{"a":null,"list":[null,2],"schema":{"z":1,"a":2}}), andToGoAnyreturns"userdata(value.jsonNull): 0x..."instead ofnil.Verification
go test -racepasses on./runtime/lua/modules/json/...,./runtime/lua/engine/value/...and./runtime/lua/engine/payload/....go test -shortpasses on./runtime/lua/...and./boot/components/runtime/lua.go vetand golangci-lint v2.15.0 (themake lintflags) report no issues on the touched packages.llmsuite (2914 tests) passes against a build of this branch.Note
Go data converted to Lua has no null members (a Go map's
nilvalue becomes an absent key), so a value that passes through Go data, such as a msgpack transport, losesjson.null. Lua-to-Lua transfer keeps it.