Skip to content

feat(actor): preempt CPU-bound Lua actors with a tick budget - #893

Open
wolfy-j wants to merge 51 commits into
mainfrom
feature/lua-tick-preemption
Open

wolfy-j wants to merge 51 commits into
mainfrom
feature/lua-tick-preemption

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Reason for This PR

A Lua actor in a CPU-bound loop holds its scheduler worker until it blocks or finishes, starving every other actor on that worker; the only remedy today is cancelling it. This ports v2's tick-budget preemption to v1. Depends on wippyai/go-lua#60 (go.mod pins its commit; bump to the release tag once it is merged).

Description of Changes

  • API: process.Preemptible, StepPreempted + StepOutput.Preempt(), process.ErrStepLimitExceeded, process.StepAccounted, and api/runtime/lua execution options (ExecutionBudgets, EntryExecutionBudgets, ExecutionBudgetsFromOptions).
  • Options (v2 keys): tick_budget (0 = default 512, negative = unlimited) and max_steps (0 = unlimited) read from entry meta.options and overridable by process.with_options{...}. Unknown meta.options keys are rejected as in v2 (default_host is also accepted because v1's process functions read it); numeric parsing matches v2. Differences from v2: one tick budget covers a whole scheduler step across the actor's coroutines (v2 resets it per resume); max_steps keeps counting across upgrades so it caps an actor's lifetime (v2 starts a replacement fresh); entry options come from the entry being run, spawn options carry over upgrades.
  • Scheduler: enables preemption at admission and after upgrades; a preempted process stays on its worker when nothing else is ready, otherwise it is requeued behind ready work and a parked worker is woken. Upgrades start a new yield-completion epoch so completions of replaced code cannot reach the new code.
  • Lua process: module top-level code and imported libraries run as Lua frames of the main task (engine.DeferInitializer), so they are preemptible too; a failing module chunk or a missing method surfaces from Step (after any preemption of the initialization) rather than from Init (script parse errors still come from Init).
  • Limits of preemption: Lua inside Go callbacks (sort comparators, gsub functions) is not preemptible and does not consume max_steps; it stops on termination/cancellation.
  • Fixes in code this touches (several pre-existing on main): a pooled processor could be stepped half-initialized by a stale queue entry; Stop raced with admission and with cancel fields; failed upgrades closed a process twice; late async completions could wake a reused processor; pooled processors ignored cancellation and delayed Stop; pooled tasks, step outputs and completed initializers retained previous users' values; a declared import now wins over a library's _G export of the same name.

Review corrections

  • Shutdown's deadline now includes in-flight admission. Cooperative Init/OnStart calls are canceled on expiry; noncooperative callbacks cannot publish work after workers stop. A successful late OnStart receives exactly one completion/Close after returning; a rejected admission stays caller-owned.
  • Stop does not close a process concurrently with Init/OnStart. A native callback that ignores cancellation can outlive Stop; cleanup must wait for it to return. This does not make arbitrary in-process native code forcibly interruptible.
  • Retiring yield completions invokes the same EventDiscarder ownership hook as queue Close/Reset, leaving messages and queue generation intact.
  • Current-main integration uses the canonical execution-frame continuation and closes a failed replacement once, before releasing its frame-owned resources.
  • CPU-only preemption no longer creates external-yield bookkeeping.

Tests

  • Unit tests for budgets, options, max_steps, initializers, close ownership, stale completions, publication order, admission vs Stop, wakes.
  • Seeded chaos tests (system/scheduler/actor/chaos_test.go, runtime/lua/engine/chaos_test.go) checking exactly-once completion/close, no wrong-incarnation completions, max_steps, no stranded processes, counters and goroutines back to baseline; fuzz targets for option parsing.
  • Affected packages pass with -race; full app suite passes; lint clean on changed packages.

Integration with #887

Runtime #887 is now stacked on this PR and contains the verified handler-validation compatibility corrections. Companion go-lua #58 is stacked on #60. Existing PR branches now own the integration commits; redundant staging branches are removed.

Land this PR before #887. Combined integration was verified with race suites and full repository lint against the real dependency pin; updated #887 CI is rerunning. Nothing has been merged or released by this update.

A Lua actor in a long loop held its scheduler worker until it blocked or
finished, starving every other actor on that worker. Lua actors now run
each step under a VM tick budget (go-lua SetTickBudget) and are suspended
at a safepoint when it is used up.

- process.Preemptible (EnablePreemption) and StepPreempted. The actor
  scheduler enables preemption at admission and requeues a preempted
  process through the global FIFO behind other ready work, also after
  dispatching the same step's yields. Pool, Temporal and eval drivers do
  not enable it and are unchanged.
- The Lua process spends one budget per step across all its coroutines;
  tasks that did not finish run in the next step.
- Options match v2: tick_budget (0 = default 512, negative = unlimited)
  and max_steps (0 = unlimited, exceeding fails with
  process.ErrStepLimitExceeded), read from entry meta.options and
  overridden by process.with_options. Keys directly under meta are
  rejected; with_options validates values at the call site.
- go-lua bumped to the preemption commit (wippyai/go-lua#60).
@wolfy-j
wolfy-j requested a review from skhaz October 2, 2026 22:24
wolfy-j added 20 commits October 2, 2026 20:13
Upgrade replacements are admitted for preemption like any other process,
inherit the spawn lifecycle options of the actor, and continue its step
count so max_steps bounds the actor over its whole lifetime.

A preempted process goes straight back to its worker when nothing else is
ready and wakes another worker only when one is parked.
…he tick budget

Module chunks run inside the main task of the first step instead of Init, so
they are scheduled and preempted like any other Lua code while the exported
method cache keeps serving later executions. An exhausted budget ends the
step without walking the run queue, and the preempted coroutine queues behind
the waiting ones. A step rejected by max_steps releases the retention leases
of the messages it was delivered.
The test closed its process explicitly and again through the helper's
cleanup, putting one Process into the shared pool twice; a later test could
then receive a Process whose state another test released.
A receive that parks on the step's last tick now sees the messages held back
for it, so the process reports preempted instead of idle.
Library chunks no longer execute through PCall while the factory builds the
state. A binder registers them with DeferInitializer; the entry bootstrap runs
them as Lua frames of the main task in dependency order, so loops in library
code are preempted and cannot block the spawner or an upgrade. Chunk
environments resolve imports at lookup time. The bootstrap uses no guest-visible
global.
SyncExecute initializes the process's library chunks, in order and with the
same result handling, before it calls the main chunk.
A chunk environment decides which aliases already exist in the base when the
chunk first looks up a global, after its dependencies have run, so a library's
_G export under an alias name no longer overrides the import while helpers
installed later still resolve.
Each chunk environment seals its set of import aliases against the globals
as they are once every library has initialized, so a later sibling library's
_G export cannot override an earlier library's import regardless of when that
library first looked up a global.
…rnation

On timeout Stop read each processor's cancel field while Submit could still
be assigning it after publishing the processor, and while completion could be
clearing it for pooling. It now cancels through the processor's atomically
published signal reference, as Terminate does; the cancellation still wakes
parked processes through the wake registered at submit.
…ed admission

Completion owns closing a non-pooled process and the submitter owns a process
Submit rejects. The worker closed the process before completing it on every
failed upgrade, and Submit closed it on OnStart rejection while the caller
closed it again; a Lua process returns itself to its pool on each Close, so
the duplicate put could hand one process to two users.
Handlers received the pooled Processor as their receiver, and its CompleteYield
read the current queue generation. A completion that outlived its actor and
arrived after the slot was reused enqueued an event into, and woke, the new
actor. Handlers now receive the queue's generation-bound YieldCompleter, and
Processor no longer implements the receiver interface.
… state

The registry-held initializer list kept every completed Fn, Done and
OnInitialized callback, retaining chunk environments and their captures for the
life of the state. A completed entry is cleared and the callbacks are released
after they run; failed initializers stay registered for retry.
Pooled tasks truncated retBuf and resumeBuf without clearing, so the backing
arrays kept the previous execution's Lua values reachable.
Reset truncated the overflow slice without clearing it, so the pooled output
kept references to the previous step's commands.
Submit and CreateProcessor stored StateReady before setting the context,
cancel function, queue generation and completer, and the processor pool
built processors in the Ready state. A stale queue entry left from a pooled
processor's previous incarnation could win Ready->Running and step the new
process half-initialized, which the race detector reported as a write of the
context racing the worker's read. Processors are built and pooled as
Complete, and Ready is the last store before the processor is scheduled.
The test waited for sync.Pool to hand the finished processor to a new process,
which the race detector's randomized pool drops make nondeterministic. It now
asserts that yield handlers receive the incarnation's generation-bound
completer; the queue test covers that a stale completer is inert.
StepsUsed reported zero for actors without max_steps, so an upgrade into
a definition that sets a limit restarted the lifetime count.
wolfy-j added 24 commits October 3, 2026 10:33
Document the Set flags of ExecutionBudgets and drop a fairness comment
that contradicts guest preemption.
The registry helper wrapped a nil-safe lookup that attrs.Bag already
provides; api/runtime/lua no longer imports api/registry.
A present meta.options of another shape registered with default budgets,
and ExecutionBudgetsFromOptions dereferenced a nil interface.
Yield tags restart in the replacement, so a completion of the replaced code
could resume an unrelated task. EventQueue.RetireYieldCompletions makes
earlier completers inert and drops queued completions; messages and wakes
keep the queue generation.
…sition

A completion or message bound to an earlier incarnation could pass the
generation check, then wake or flag the incarnation that reused the slot,
costing it a spurious step. The check and the transition now run under one
per-processor lock that slot reuse also takes.
wakeAny notified the first worker whose token was free, which could be the
worker running the step that queued the work. A worker that then blocked in a
long step left the work stranded while another worker stayed parked. Workers
publish whether they are parked and only those are notified.
Submit published a processor to the routing maps before its admission ended,
so Stop could cancel and complete it while Submit still finished
initialization and published it Ready. Admissions are now registered under the
stopping check and Stop waits for them before it acts.
…n release

CreateProcessor did not register the context-cancel wake that Submit
registers, so Terminate or a cancelled context left an idle pooled
processor parked forever. ReleaseProcessor lowered the live count
without telling a draining Stop, so Stop waited out its whole deadline
after the last pooled processor was released.
Reject unknown meta.options fields on process entries (accepting the v2
process keys and default_host) and parse tick_budget and max_steps with
the v2 numeric carriers and ranges, so max_steps spans uint64. Add fuzz
targets for options parsing.
Integrate current main without restoring early processor publication or double-closing failed upgrades. Observe the shutdown deadline during Init/OnStart, cancel unfinished admissions, and pair successful late starts with completion only after the callback returns. Retire yield completion data through the canonical discard hook and allocate yield bookkeeping only for external yields. Add ownership, deadline, failed-frame, late-rejection and CPU-only allocation regressions.
@wolfy-j
wolfy-j requested a review from skhaz October 23, 2026 02:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants