Skip to content

fix(utf8): advance multibyte iterators and bound offset positions - #91

Merged
wolfy-j merged 1 commit into
mainfrom
fix/utf8-iteration-offset-boundaries
Nov 15, 2026
Merged

wolfy-j merged 1 commit into
mainfrom
fix/utf8-iteration-offset-boundaries

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

utf8.codes treated its 1-based control value as the next byte index without skipping continuation bytes. Iterating Aä€😀 returned the first multibyte codepoint, then raised invalid UTF-8 code on its continuation byte. Advance to the next leading byte while preserving the byte positions Lua returns.

The adjacent utf8.offset implementation indexed beyond the string for zero-offset queries at the end, including utf8.offset("a", 0, 2). It also used the wrong negative-index adjustment and the wrong default start for backwards traversal. Bounds now match Lua 5.4: invalid positions and continuation-byte starting positions raise Lua errors; backwards traversal starts at the end; offset zero accepts the end sentinel safely.

Regression scripts fail on the previous implementation, pass after the fix, and also pass against reference Lua 5.4. Coverage includes 1–4-byte characters, empty input, truncated UTF-8, the valid replacement character, backwards traversal, negative positions, the end sentinel and out-of-range positions. Focused UTF-8 tests pass with the race detector.

This is the dependency correction needed before exposing the UTF-8 library through runtime #1086. No interpreter refactor is included.

@wolfy-j
wolfy-j requested a review from skhaz November 15, 2026 18:19
@wolfy-j
wolfy-j merged commit 440a9bd into main Nov 15, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants