Skip to content

fix(utf8lib): port the Lua 5.4 utf8 library - #94

Merged
wolfy-j merged 2 commits into
mainfrom
fix/utf8lib-lua54
Nov 15, 2026
Merged

wolfy-j merged 2 commits into
mainfrom
fix/utf8lib-lua54

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Defect

The utf8 library departs from Lua 5.4 in several functions:

for p, c in utf8.codes("H\195\164") do print(p, c) end
-- error: invalid UTF-8 code; Lua 5.4: 1 72 / 2 228

utf8.codepoint("H\195\164llo", -1)   -- nothing, Lua 5.4: 111
utf8.codepoint("H\195\164llo", 1, -1) -- nothing, Lua 5.4: 72 228 108 108 111
utf8.len("abc", 5)                    -- 0, Lua 5.4: error, initial position out of bounds
utf8.offset("H\195\164llo", 1, 3)     -- 3, Lua 5.4: error, initial position is a continuation byte
utf8.offset("H\195\164llo", -1)       -- nil, Lua 5.4: 6
utf8.char(0xD800)                     -- "\239\191\189" (U+FFFD), Lua 5.4: "\237\160\128"
utf8.char(0x110000)                   -- error, Lua 5.4: "\244\144\128\128"
utf8.len("\237\160\128", 1, -1, true) -- nil 1, Lua 5.4: 1 (lax mode)

utf8.codes fails on any string that contains a multibyte character.

Cause

  • utf8CodesIter returns the 1-based position as the control value, but reads the incoming control value as a 0-based offset. Each step advances one byte and decodes a continuation byte.
  • utf8Codepoint neither resolves negative positions nor checks bounds, and it clamps j.
  • utf8Len and utf8Offset clamp positions instead of raising, and utf8Offset resolves negative positions with the wrong formula (len + i + 2).
  • The library is built on Go's unicode/utf8. EncodeRune replaces surrogates and values above U+10FFFF with U+FFFD, and the decoder has no lax mode.

Fix

utf8lib.go is a port of lutf8lib.c:

  • the utf8_decode decoder, strict (up to U+10FFFF, no surrogates) and lax (up to 0x7FFFFFFF);
  • luaO_utf8esc encoding for utf8.char (up to six bytes);
  • u_posrelat position handling;
  • the same argument errors (out of bounds, initial/final position out of bounds, position out of bounds, initial position is a continuation byte, value out of range, invalid UTF-8 code);
  • the optional lax argument of len, codepoint and codes;
  • the utf8.codes iterator from Lua 5.4.6. It skips the continuation bytes of the previous character, rejects a string that starts with a continuation byte, and rejects a stray continuation byte after a character.

charpattern is unchanged. The stdlib type signatures declare the optional lax parameter.

OpenUtf8 is exported and already in linit.go's luaLibs, so OpenLibs opens it. No embedder-facing change.

Tests

  • utf8lib_test.go: 39 result cases and 18 error cases for char, charpattern, len, codepoint, offset and codes, including negative positions, bounds, invalid, truncated, overlong and surrogate sequences, and lax mode. Expected values come from the Lua 5.4 reference interpreter; the two codes continuation-byte checks follow 5.4.6. 23 cases fail on main.
  • compiler/check/tests/regression/utf8_lax_param_test.go: the type checker accepts the lax argument. It fails on main with "too many arguments".

Verification

go vet ./..., golangci-lint v2.8.0 with the CI flags, go test ./... (full fixture corpus), the coverage collector run, and the race "other" shard (go test -race on all non-fixture tests and the non-regression fixture categories) pass. The regression-fixture race shards are left to CI.

The runtime picks this up through a go-lua dependency bump after merge.

utf8.codes advanced one byte per step, because the iterator treated its
1-based control value as a 0-based offset, so it raised "invalid UTF-8
code" on the first multibyte character. utf8.codepoint ignored negative
positions and silently clamped out-of-range ones, utf8.len and
utf8.offset clamped instead of raising, utf8.offset accepted a start on a
continuation byte, utf8.char encoded surrogates as U+FFFD and rejected
values above U+10FFFF, and the lax flag was not supported.

utf8lib.go now follows lutf8lib.c: the same decoder (strict and lax),
luaO_utf8esc encoding up to 0x7FFFFFFF, u_posrelat position handling, and
the same argument errors. The stdlib type signatures declare the optional
lax parameter.
@wolfy-j
wolfy-j requested a review from skhaz November 15, 2026 18:42
@wolfy-j
wolfy-j requested a review from skhaz November 15, 2026 18:48
@wolfy-j
wolfy-j merged commit f93f659 into main Nov 15, 2026
12 checks passed
@wolfy-j
wolfy-j deleted the fix/utf8lib-lua54 branch November 15, 2026 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants