Skip to content

fix(vm,mathlib): return integers from #, select('#') and the math library - #97

Merged
wolfy-j merged 1 commit into
mainfrom
fix/integer-subtype-results
Nov 15, 2026
Merged

wolfy-j merged 1 commit into
mainfrom
fix/integer-subtype-results

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Defect

The length operator, select('#') and the math library return floats where Lua 5.4 returns integers, and math.random with a full-width range fails:

print(math.type(#{1, 2}))              -- float, Lua 5.4: integer
print(math.type(#"ab"))                -- float
print(math.type(select("#", 1)))       -- float
print(math.type(math.floor(3.7)))      -- float (the static type of math.floor is already integer)
print(math.type(math.abs(-3)))         -- float
print(math.type(math.max(1, 2)))       -- float
print(math.type(math.fmod(7, 3)))      -- float
print(math.type(math.modf(3.5)))       -- float
print(math.abs(math.mininteger))       -- 9.2233720368548e+18, Lua 5.4: math.mininteger
print(math.fmod(1, 0))                 -- nan, Lua 5.4: bad argument #2 (zero)
print(math.modf(math.huge))            -- inf  nan, Lua 5.4: inf  0.0
print(math.random(math.mininteger, math.maxinteger)) -- error "invalid argument to IntN"
print(math.random(1.5))                -- 1, Lua 5.4: number has no integer representation

Cause

  • OP_LEN stores string and table lengths with SetNumber, and baseSelect pushes the '#' count as LNumber.
  • mathAbs, mathFloor, mathCeil, mathFmod, mathModf, mathMax and mathMin read every argument with CheckNumber, which converts integers to float, and push LNumber.
  • mathRandom computes n-m+1 in int (overflows for wide ranges, and rand.IntN panics on the negative result), reads bounds with the truncating CheckInt, and pushes LNumber.

Fix

The library functions follow lmathlib.c:

  • floor/ceil return integer arguments unchanged and convert the rounded float to an integer when it fits (pushnumint); modf does the same for its integral part, and its fractional part is 0.0 when the argument is integral or infinite.
  • abs keeps integers (wrapping at math.mininteger), fmod on two integers uses truncating integer remainder with the "zero" argument error and the -1 special case.
  • max/min return the selected argument itself, compared with the VM's <, so the subtype of the winner is kept.
  • random(m, n)/random(n) read bounds with luaL_checkinteger semantics, draw uniformly over the full [m, n] range with unsigned arithmetic, and return an integer; random(0) returns a random integer; argument errors match Lua 5.4.
  • OP_LEN stores string and table lengths as integers and the __len result as returned. select('#') pushes an integer.

luaL_checkinteger already existed as formatInteger in stringlib_format.go; it moves to auxlib.go as checkInteger and both string.format and math.random use it.

Tests

integer_subtype_results_test.go checks the value and subtype of # (tables, strings, __len returning integer, float and non-number), select('#'), and every math function above for integer, float, mixed, numeric-string, out-of-range, negative-zero, infinity and math.mininteger arguments, the random bounds over repeated draws, and the argument errors of fmod, random, max and min. Expected values come from the Lua 5.4 reference interpreter. 37 of the 48 result cases, all 9 error cases and the bounds test fail on main.

Not covered here (tracked in the conformance issue): next/pairs array keys, the public CheckInt helpers, and exact mixed integer/float <, which max/min inherit once it lands. String library results are in #95.

Verification

go vet ./..., golangci-lint v2.8.0 with the CI flags, go test ./... (full fixture corpus) and the coverage collector run pass.

The runtime picks this up through a go-lua dependency bump after merge.

@wolfy-j
wolfy-j requested a review from skhaz November 15, 2026 20:23
@wolfy-j
wolfy-j merged commit 92ea89c into main Nov 15, 2026
12 checks passed
@wolfy-j
wolfy-j deleted the fix/integer-subtype-results branch November 15, 2026 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants