Skip to content

perf: take MOD and MULMOD remainder digits without a branch on ARM64 - #186

Merged
svlachakis merged 2 commits into
mainfrom
perf/arm64-mg-2by1-branchless
Nov 16, 2026
Merged

svlachakis merged 2 commits into
mainfrom
perf/arm64-mg-2by1-branchless

Conversation

@svlachakis

Copy link
Copy Markdown
Contributor

Changes

  • URem2By1: a remainder-only copy of UDivRem2By1 (Moller-Granlund 128-by-64 reciprocal digit). The first correction is a select (r1 = r1 > low ? r1 + d : r1) in the standard build, so the running remainder does not wait for a mispredicted branch. The zkEVM build keeps the branch (DivRemBranchlessCorrection => false), which costs fewer guest steps.
  • The 16 call sites that discard the quotient now call URem2By1: RemainderBy64Reciprocal (MOD by one limb, and the operand reduction of MULMOD by a 64-bit modulus), Remainder128By64Bits (the product reduction of MULMOD by a 64-bit modulus) and Remainder257By64Bits (ADDMOD by a 64-bit modulus). These paths run only without the x64 hardware divide.
  • UDivRem2By1 does not change. A select there (quotient live) made DIV 6-13% slower on Neoverse-N2, so the quotient paths keep the branch.
  • Reciprocal2By1 indexes the seed table with (byte)(d >> 55): for a normalised d this is the old (d >> 55) - 256, and the byte index needs no bounds check. A Debug.Assert keeps the normalisation precondition.
  • New ReciprocalDivisionBench (DIV, MOD and MULMOD by a 33- to 64-bit modulus, nine operand shapes) and ReciprocalDivisionTests.

Why the select: for random digits the first correction applies to 64-75% of digits (Python model of the digit, 20,000 digits per divisor class: 1e18, random 64-bit, 2- to 16-bit), so a branch on it is close to unpredictable.

Performance (arm64)

GitHub Actions ubuntu-24.04-arm, Neoverse-N2, 4 cores, .NET 10.0.15. Base (bench: add reciprocal division shapes) and head built in the same job and run on the same runner as A1 B1 A2 B2, BenchmarkDotNet --warmupCount 3 --iterationCount 10. 0 B allocated in every case. Mean ns, error in parentheses.

MULMOD by a 33- to 64-bit odd modulus (MultiplyModBy64; the shape names the x, y operands):

Shape A1 B1 Round 1 A2 B2 Round 2
WadScaled 32.65 (0.03) 30.05 (0.02) -8.0% 32.67 (0.06) 29.58 (0.01) -9.5%
WadWide 40.53 (0.04) 34.09 (0.01) -15.9% 40.62 (0.05) 34.08 (0.02) -16.1%
Small 43.86 (0.04) 34.89 (0.01) -20.5% 43.90 (0.02) 34.90 (0.01) -20.5%
Full256By64 42.92 (0.16) 35.25 (0.01) -17.9% 42.79 (0.06) 35.28 (0.01) -17.6%
ShortBy128 39.92 (0.02) 36.80 (0.01) -7.8% 39.88 (0.04) 36.83 (0.01) -7.6%
MulDivBy128 43.85 (0.03) 39.33 (0.01) -10.3% 43.96 (0.10) 39.33 (0.01) -10.5%
Full256By128 49.52 (0.04) 42.08 (0.02) -15.0% 49.37 (0.06) 42.08 (0.01) -14.8%
MulDivBy192 46.56 (0.09) 41.82 (0.02) -10.2% 46.70 (0.13) 41.90 (0.02) -10.3%
Full256By192 51.02 (0.03) 44.07 (0.01) -13.6% 51.14 (0.06) 44.04 (0.01) -13.9%

MOD:

Shape A1 B1 Round 1 A2 B2 Round 2
WadScaled 17.97 (0.01) 17.79 (0.00) -1.0% 17.88 (0.00) 17.79 (0.00) -0.5%
WadWide 23.24 (0.01) 23.19 (0.02) -0.2% 23.22 (0.01) 23.18 (0.01) -0.2%
Small 22.65 (0.01) 22.59 (0.00) -0.3% 22.66 (0.00) 22.59 (0.00) -0.3%
Full256By64 23.69 (0.01) 23.52 (0.00) -0.7% 23.71 (0.01) 23.52 (0.00) -0.8%
ShortBy128 19.85 (0.01) 19.80 (0.01) -0.3% 19.85 (0.01) 19.80 (0.01) -0.3%
MulDivBy128 24.67 (0.08) 24.34 (0.06) -1.3% 24.65 (0.04) 24.37 (0.01) -1.1%
Full256By128 28.98 (0.11) 28.89 (0.10) -0.3% 28.95 (0.17) 28.84 (0.09) -0.4%
MulDivBy192 25.20 (0.01) 25.19 (0.01) -0.0% 25.19 (0.01) 25.14 (0.01) -0.2%
Full256By192 33.59 (0.05) 32.82 (0.05) -2.3% 33.52 (0.05) 32.88 (0.06) -1.9%

DIV (code path not changed except the seed index):

Shape Round 1 Round 2
WadScaled -1.2% -1.2%
WadWide +0.9% +1.0%
Small +0.8% +0.7%
Full256By64 +0.6% +0.8%
ShortBy128, MulDivBy128, Full256By128, MulDivBy192, Full256By192 -0.3..+0.2% -0.3..+0.2%

A second run of the same jobs (https://github.com/NethermindEth/int256/actions/runs/43654848126) gives the same numbers within 0.5 points. The DIV-by-one-limb shapes are +0.6..+1.0% in both runs; their code is the same apart from the byte index, so I read this as a layout effect, but I cannot prove it.

Runs: https://github.com/NethermindEth/int256/actions/runs/43654808112 (arm64). In one of the four jobs of the second run the first B1 benchmark reported NA (harness); B2 of the same job and both rounds of the first run are complete.

x64: every changed call site is on the path without the hardware divide (X86Base.X64.IsSupported is false), so x64 runs the same code apart from the byte index. An x64 run (AMD EPYC, ubuntu-latest) of an earlier version of this change showed DIV, MOD and MULMOD flat (within +-0.7% in both rounds).

zkVM guests

One run per arm (the executors are deterministic), the 11 blocks of Nethermind.Stateless.ZiskGuest/inputs.json, Nethermind master 0d776d3b096 with the zkEVM build of Nethermind.Int256.dll from int256 base 115878b or this PR swapped into each guest before bflat links it. Every output is the same as the expected output in inputs.json on every arm (Airbender: make verify against the published digest; OpenVM: the public values). The guests take only the byte index (the select is off in the zkEVM build).

ZisK

ziskemu steps.

Block Base This PR Change Same output
25526356 86584277 86584193 -84 (-0.0001%) yes
25532382 92707071 92706971 -100 (-0.0001%) yes
25532442 103432700 103432606 -94 (-0.0001%) yes
25532454 110486684 110486634 -50 (-0.0000%) yes
25532553 97418720 97418620 -100 (-0.0001%) yes
25532642 101204539 101204451 -88 (-0.0001%) yes
11957686 252342479 252342479 +0 (+0.0000%) yes
11962410 148928577 148928577 +0 (+0.0000%) yes
11962415 180376686 180376686 +0 (+0.0000%) yes
11962427 152940271 152940271 +0 (+0.0000%) yes
11962485 157777805 157777805 +0 (+0.0000%) yes
Total 1484199809 1484199293 -516 (-0.0000%)

Total cost: 194152241037 -> 194152188267 (-0.0000%).

SP1

SP1 cycles (nethermindeth/sp1-runner:v6.9.0).

Block Base This PR Change Same output
25526356 153587685 153579275 -8410 (-0.0055%) yes
25532382 166822265 166816645 -5620 (-0.0034%) yes
25532442 257924332 257895756 -28576 (-0.0111%) yes
25532454 196004939 195996355 -8584 (-0.0044%) yes
25532553 419159331 419157723 -1608 (-0.0004%) yes
25532642 196036418 196025870 -10548 (-0.0054%) yes
11957686 387082900 387081736 -1164 (-0.0003%) yes
11962410 219680513 219677119 -3394 (-0.0015%) yes
11962415 262249034 262243940 -5094 (-0.0019%) yes
11962427 209154324 209146390 -7934 (-0.0038%) yes
11962485 218345880 218341582 -4298 (-0.0020%) yes
Total 2686047621 2685962391 -85230 (-0.0032%)

OpenVM

OpenVM instret (execute_metered_cost).

Block Base This PR Change Same output
25526356 130313114 130304598 -8516 (-0.0065%) yes
25532382 142728669 142722709 -5960 (-0.0042%) yes
25532442 193351773 193322769 -29004 (-0.0150%) yes
25532454 167532056 167522644 -9412 (-0.0056%) yes
25532553 150903519 150901721 -1798 (-0.0012%) yes
25532642 154124499 154113721 -10778 (-0.0070%) yes
11957686 318226296 318225050 -1246 (-0.0004%) yes
11962410 182725417 182721755 -3662 (-0.0020%) yes
11962415 232212327 232206913 -5414 (-0.0023%) yes
11962427 186222470 186214266 -8204 (-0.0044%) yes
11962485 194853165 194848609 -4556 (-0.0023%) yes
Total 2053193305 2053104755 -88550 (-0.0043%)

Total cost: 148330045774 -> 148328230499 (-0.0012%).

Airbender

Airbender cycles.

Block Base This PR Change Same output
25526356 394729754 394729670 -84 (-0.0000%) yes
25532382 463984492 463984392 -100 (-0.0000%) yes
25532442 468395126 468381510 -13616 (-0.0029%) yes
25532454 493876494 493876394 -100 (-0.0000%) yes
25532553 477627432 477627236 -196 (-0.0000%) yes
25532642 472664232 472664144 -88 (-0.0000%) yes
11957686 445037086 445037038 -48 (-0.0000%) yes
11962410 262084652 262084608 -44 (-0.0000%) yes
11962415 342891048 342890958 -90 (-0.0000%) yes
11962427 263715367 263715315 -52 (-0.0000%) yes
11962485 291926317 291926273 -44 (-0.0000%) yes
Total 4376932000 4376917538 -14462 (-0.0003%)

Runs: https://github.com/NethermindEth/nethermind/actions/runs/43655824720 (all four guests, this PR head before a comment and a test were added) and https://github.com/NethermindEth/nethermind/actions/runs/43652715912 (OpenVM).

Tests

  • ReciprocalDivisionTests: DIV, MOD, MULMOD and ADDMOD against BigInteger for divisors that cover all 256 seed-table entries (top nine bits of the normalised divisor) with low bits at both ends and random, ulong.MaxValue and 2^63, and 4,000 random one-limb divisors, with dividends next to a multiple of the divisor and next to the maximum quotient. A wrong seed index (d >> 56) fails it.
  • The full suite passes on arm64, arm64 with DOTNET_EnableHWIntrinsic=0, and the zkEVM build (602,346 / 602,346 / 602,320 tests).

The first Moller-Granlund correction of a 128-by-64 reciprocal digit applies to about two digits in three, so a branch on it mispredicts. Chains that keep only the remainder (MOD by one limb, MULMOD by a 64-bit modulus, the 257-bit remainder of ADDMOD) now use URem2By1, which selects the corrected remainder. The quotient paths keep UDivRem2By1 as it was: a select there made DIV slower. The zkEVM build keeps the branch, which costs fewer guest steps. The seed table index in Reciprocal2By1 is a byte, so the lookup needs no bounds check.

@LukaszRozmej LukaszRozmej left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. URem2By1 is UDivRem2By1 without the quotient, and the quotient adjustments never feed r1. For normalised d, (byte)(d >> 55) is equal to the old (d >> 55) - 256. ReciprocalDivisionTests checks every seed entry at every shift against BigInteger, and the arm64 runners (with and without HW intrinsics) run it.

Nit: before this change, a divisor that is not normalised threw IndexOutOfRange from the bounds check. Now Release builds silently read a wrong seed, because only Debug.Assert guards it. That can happen only if there is a bug in a caller, so this is not blocking.

🤖 AI agent (Claude Code / Opus 5.5) on behalf of @LukaszRozmej

@svlachakis

Copy link
Copy Markdown
Contributor Author

@LukaszRozmej Thanks. I kept it without a Release check. Every caller normalises the divisor before the lookup (shift by its leading zero count), the Debug.Assert guards that contract in Debug and in the test runs, and ReciprocalDivisionTests covers every seed entry at every shift. A Release check would add a compare and branch to each MOD and MULMOD remainder step, which is the path this PR makes branchless.

@LukaszRozmej LukaszRozmej left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 67f76443021444825df2cae1bc59d03d1a6ce50e.

Reviewed remainder-only corrections against the existing quotient/remainder algorithm, normalized seed indexing, std/zkVM selection, BigInteger vectors and benchmarks. No new correctness finding; reported ARM64/guest measurements were not rerun.

Scope (5 changed files)
  • src/Nethermind.Int256.Benchmark/ReciprocalDivisionBench.cs
  • src/Nethermind.Int256.Tests/ReciprocalDivisionTests.cs
  • src/Nethermind.Int256/UInt256.divmod.cs
  • src/Nethermind.Int256/UInt256.std.cs
  • src/Nethermind.Int256/UInt256.zkevm.cs

Findings

None.

Optimization

Optimization pass completed; no additional actionable opportunity identified.

Validation: Static review of the PR diff, relevant callers/contracts and existing threads. No project tests, native builds or benchmarks rerun.

🤖 AI agent (Codex / GPT-6) on behalf of @LukaszRozmej

@svlachakis
svlachakis merged commit 3400275 into main Nov 16, 2026
17 checks passed
@svlachakis
svlachakis deleted the perf/arm64-mg-2by1-branchless branch November 16, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants