Skip to content

feat(eval): supervised eval processes with ownership, limits and send capabilities - #895

Open
wolfy-j wants to merge 59 commits into
feature/lua-tick-preemptionfrom
feature/lua-eval-host
Open

wolfy-j wants to merge 59 commits into
feature/lua-tick-preemptionfrom
feature/lua-eval-host

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Reason for This PR

v1's eval_runner runs dynamic Lua synchronously inside the caller, with the caller's authority (bounded only by a step limit and cancellation). This ports v2's eval model: dynamic code (for example model-written code) runs as a supervised process admitted under an explicit policy, with execution limits, capability-restricted messaging and a lifetime bounded by its owner. eval_runner keeps its current surface and is deprecated. Stacked on #893; shares the upgrade continuation foundation with #909.

Description of Changes

  • Admission: process.Start.Admission starts a process from an in-memory factory. An admitted process starts from a clean frame: it inherits nothing from its caller except trace context (ctxapi.PropagatorPairs).
  • Ownership: process executions, and function calls when a process manager is present, have an ExecutionScope. Spawns with process.owned, and every spawn of an owned process, are owned and terminated when the owner completes, however it ends. Monitor-only evals are owned too; function calls and owned processes cannot spawn detached evals; detached evals own what they spawn and run under lua.eval.detached_lifetime (default 1h). These are stricter than v2. A child terminated because its owner ended completes with process.ErrOwnerEnded (explicit terminations and deadlines keep their errors); the owner's EXIT is not ordered after its children's.
  • Upgrades continue the execution: frame keys marked Execution (PID, spawn options, scope, ownership, child limit, send grants, tty port) are shared by every incarnation through ctxapi.ContinueFrameContext; hosts release the frame they created via ctxapi.ExecutionFrame.
  • Send capabilities: a restricted process may address its parent, processes it spawned, senders of messages it received, itself and (when its policy grants lookup) names it looked up; forged PIDs fail as not allowed. Unlike v2, PIDs passed as input are not granted (v1 PIDs are strings).
  • Eval host and eval module (v2 surface): compile/spawn/evict, program cache with pinning of running programs, policy hashing with typed bindings (plain data only; v2 also admits bytes and records), imports run as deferred initializers under the eval's budget, method overrides, v2 command IDs (plus lookup), tick_budget/max_steps/max_children (the child limit covers the whole spawned tree). memory_limit_bytes, heap_reserve_bytes and mailbox_capacity are accepted and not enforced.
  • Security: the eval policy denies every action it does not allow; process.upgrade and process.registry.lookup honour explicit denials (security.IsDenied); metrics writes are gated for evals.
  • Fixes found on the way: resource store cleanups raced with removal; stale leases of a pooled store reached its next user; Cloner values are copied across spawns; a failed owner start left owned children running; an owned spawn could adopt an existing named process.

Tests

  • Unit: api/context, api/process, api/security, system/process, system/scheduler/actor, runtime/lua/evalhost, runtime/lua/modules/eval, runtime/lua/modules/process, service/host.
  • Adversarial app tests in app.test.eval_module (sandbox surface, forged PIDs, registry lookup, child authority, descendant limits, detached lifetime with a spinning eval, upgrade denial, lifetime resistance) alongside spawn, program lifecycle, send capability, limit, admission and function-lifetime tests.
  • Leak checks: eval cycles across endings, upgrade chains and function calls release registrations, slots and frames.
  • Affected API, eval, host, process and scheduler packages pass with -race; the native application suite passes all 411 tests (environment-dependent suites retain their existing skips); full repository lint reports zero issues.
  • Review regressions cover clone-free continuation, owned replacements, restoring borrowed handles, concurrent same-key writes, 300-upgrade chains and failed Init cleanup. Value and ownership change in one atomic snapshot.

Known limits: memory is not bounded in v1 (a single eval can exhaust host memory), and Lua inside sort/gsub callbacks is bounded only by termination and the lifetime, not by max_steps.

Landing constraints from integration review

#887 and #893 pin divergent go-lua branches. Composing them with the current #893 dependency does not compile (ast.HasTypeSyntax is absent). Before landing this eval/preemption stack together with argument validation, it needs a go-lua revision containing both features and tests that validate the actual selected method after deferred bootstrap, before its body runs. Green individual-branch CI does not establish that combined compatibility.

The memory and mailbox settings above remain unenforced compatibility fields; this is not an OS-isolated or memory-safe sandbox for hostile code. Tick budgets also do not hard-preempt arbitrary Go-native builtins.

@wolfy-j
wolfy-j requested a review from skhaz October 3, 2026 13:41
@wolfy-j
wolfy-j force-pushed the feature/lua-eval-host branch from 450cab9 to 5bdd02b Compare October 3, 2026 14:05
@wolfy-j
wolfy-j force-pushed the feature/lua-eval-host branch 2 times, most recently from d0014b1 to 182e8d4 Compare October 3, 2026 18:00
wolfy-j added 23 commits October 3, 2026 17:57
process.Start gains Admission{Factory, Meta}: a process definition that is
not a registry entry, such as a compiled eval program. Only Go callers can
set it. The manager forwards an admission only to hosts that implement
process.AdmissionHost (local hosts; remote hosts cannot run an in-memory
factory), and the host builds the process from the admitted factory instead
of the registry. A named admission never takes the spawn-or-signal shortcut:
a taken name is an error carrying the existing PID.
A process whose frame carries ProcessSendGrants may only address PIDs the
runtime handed to it: itself, its parent (both seeded by whoever restricts
it), processes it spawned, senders of messages it received (msg:from()) and
registry lookup results. Addressing any other PID through process.send,
terminate, cancel, monitor or link fails with "not allowed to <op>".
Grants live in a non-inheritable frame key, so children start without them,
and checks happen in the sending process only: nothing changes on the wire.
Processes without grants behave exactly as before.
ChildSlots bounds how many child processes a process may have running at
once. ChildSlotResolver, a frame resolver, reserves a slot as a frame
attachment of each child process the limited process spawns: the slot is
released when the child's frame is released at exit and rolled back when the
spawn fails or targets a host that does not accept attachments. The limit
lives in a non-inheritable frame key.
api/host defines the eval host boundary with v2's names and shapes:
EvalPolicy, EvalCompileSpec, EvalSpawnSpec, EvalProgram and the eval
admission errors. evalhost.Admitter implements it:

- Compile validates source and policy with v2's limits and defaults,
  canonicalizes the policy and caches the compiled program keyed by source,
  policy and imported library sources (LRU, superseded on library change).
- Spawn starts the program as a real process through process.Start with an
  admission. The process frame carries its own actor and a deny-by-default
  scope built from the policy alone: process.send per send mode (cap mode
  defers to send grants seeded with the parent; explicit allows only
  send_targets) and the spawn family only with the spawn command, bounded by
  max_children child slots. tick_budget and max_steps become the process's
  execution budgets. Link modes follow v2 (linked by default, monitor_only,
  detached only when allowed).
- Modules are explicit; a module needs an allowed class and no process,
  storage or network class, except process, which is allowed by name.
  compile = "typed" type-checks strictly; jit and the upgrade command are
  rejected as unsupported. memory and mailbox limits are accepted and not
  enforced.

The compiler gains opt-in ExplicitModules, AllowModules and Strict; the eval
permission checks move to evalhost for both eval surfaces; process.pid()
grants the process's own PID; process.ErrInvalidHost makes a start on a
non-process host detectable, so eval can point at lua.eval.spawn_host.
The eval module exposes the eval host to Lua with v2's surface:
eval.compile(source, opts?) -> Program, eval.spawn(source_or_program,
opts?) -> pid, eval.evict(program), Program:spawn/evict and
tostring(Program). Options and their errors match v2 (closed option
records, nested policy, flat or nested limits, link modes, send modes).
Calls check eval.compile, eval.spawn, eval.module, eval.import and
eval.class.

Boot creates the admitter next to the existing eval host, registers the
child slot frame resolver and the eval module. New config keys:
lua.eval.spawn_host (process host for evals from callers that are not on a
process host, such as functions) and lua.eval.program_cache_size.

eval_runner is unchanged and marked deprecated in its description and
spec. App tests cover the new module end to end; the eval_runner suites
pass unchanged.
Store.Close walked the cleanup list outside its lock while a cleanup's
remover could relink the same nodes, a data race that could lose or corrupt
cleanups. Close now detaches every cleanup under the lock and runs them
afterwards; a remover called after Close finds its node detached.
Every execution has an ExecutionScope: a process for its lifetime, a
function call for one invocation. A spawn with process.owned (and every spawn
by an owned process) reserves a slot in the spawner's scope before the child
starts, as a frame attachment: it is rolled back when the child is not
admitted, rejected by hosts that cannot carry attachments, bound to the
child's PID once it starts and released when the child completes. When an
execution completes, its scope refuses new children and terminates the ones
still running, through process.Manager, for every process type.

Completion is signalled explicitly (process hosts via ctxapi.CompleteFrame,
the function executor when the call returns), not by frame reclamation,
which waits for frames forked from the execution. Child slots use the same
completion signal, so a child that leaves a grandchild running no longer
holds its parent's slot.
Send grants were not inherited, so a process spawned by a capability-
restricted process ran unrestricted under the inherited eval scope. Grants
are now inherited as an independent copy: a child can address what its
spawner could, and later acquisitions stay with the process that made them.
A restricted process may always address its parent.
- Evals are owned by the spawning execution unless detached (process.owned);
  a function call or an owned process cannot detach an eval, so a function
  never outlives its evals and an eval subtree stays contained.
- The eval scope no longer grants process.context or process.exec: named
  spawn-or-signal delivered messages past the send mode, and exec bypassed
  max_children.
- network selection requires network.select, as for process.with_options.
- Bindings are copied at admission as bounded plain-data trees (depth and
  table limits, shared tables and mixed tables rejected) on both the Lua and
  Go paths.
- Imports are compiled at admission and kept with the program: a spawn runs
  the library revision the program was admitted with.
- Typed compilation declares bindings and imports to the type checker.
- jit is not a compile mode.
- Spawning a program refreshes its cache recency.
A detached eval has no owner to end it, so it now runs under a finite
lifetime (lua.eval.detached_lifetime, default 1h): the Lua process reports
it through process.ExecutionTimeoutProvider and the scheduler cancels the
process at the deadline, waking it if it is blocked. Owned evals and all
other Lua processes report no lifetime and are unchanged.
A frame fork copies Cloner values, but PropagatedPairs, which spawns use to
carry inheritable values into a new process, passed the spawner's value
itself. A capability-restricted process and its children then shared one
send-grants set: acquisitions leaked between them and concurrent grants
raced on the map.
process.upgrade had no authorization, so eval code with the process module
could replace itself with any registry process and shed its admission
policy, budgets and ownership. process.upgrade is now refused when the
security scope explicitly denies process.upgrade (security.IsDenied), which
leaves processes under existing policies unchanged, and the eval policy
denies every action it does not allow instead of leaving it undefined.
An upgrade forked the new code's frame from the previous incarnation's frame
and made it the process context. Values set on the process frame without
inheritance (the execution scope, ownership, child limit, terminal port) were
lost to the upgraded code, completion ran on the incarnation frame so owned
children outlived the process, and the host's frame was never reclaimed.

Frame keys can now be marked Execution: ctxapi.ContinueFrameContext forks the
execution's frame and carries those values, which nothing else receives. The
scheduler keeps the execution's root context, runs each incarnation in a
continuation of it, releases the previous incarnation's frame on upgrade and
the last one after completion, and reports completion on the root context.
PID, spawn options, execution scope, the owned marker, child slots and the
tty port are execution values; the owned child's release attachment stays
with the frame that admitted it.
The admitter's binder ran each import chunk with PCall while creating the
process, before its budget, deadline or ownership applied, so an import that
never returned blocked eval.spawn. Imports are now deferred initializers that
run in the eval's first step, in admission order, preemptible and counted
against its tick budget and steps.
Admission accepted int8 and int16 bindings that payload conversion rejects at
spawn, hashed bindings as JSON so 1 and 1.0 shared a cached program although
math.type tells them apart, and copied a Go map or slice once per reference,
so a value repeated thousands of times multiplied copying and hashing under
the table limit. Bindings are now copied into the values a program receives
(int64, float64, strings, booleans, []any, map[string]any), a table reachable
twice is rejected as the Lua converter does, and the policy hash encodes
values with type tags; a spawn request's policy is normalized the same way
before it is matched against the admitted program.
When a named owned spawn reached an existing process, the host rolled the
ownership registration back and returned the existing PID, and the manager
bound it to the owner anyway; if the owner ended meanwhile the manager
terminated that unrelated process. Bind now reports ErrOwnedNotStarted for a
rolled-back registration and the spawn fails with the name-taken error,
leaving the existing process alone.
…nt definition

A no-argument process.upgrade reloads the process's current definition, but
the permission check saw an empty resource, so a policy denying upgrades of
that definition was bypassed by omitting it.
…owning them

ContinueFrameContext refused execution values that are Closers, so upgrading
a process attached to a terminal (tty port and binding) failed and the
process was terminated. A continuation frame now refers to the execution's
values and leaves closing them to the execution's frame. The execution's
frame is sealed first, so a continuation is always a fork of it.
program_lifecycle waited for the first spawn's EXIT and discarded every other
event, so when the second spawn exited first its EXIT was lost and the test
timed out.
A frame fork copies the grants from the goroutine that forks it, such as an
asynchronous function call, while the process keeps acquiring grants.
A process rejected at Init or lifecycle start released its frame without
completing it, so owned children it had already spawned outlived it.
…reused

A lease released or read after its store was closed and pooled acted on the
store's next user.
A restricted process accumulated a grant for every process it spawned; an
exited process is never addressable again, so its grant is released when the
exit event is delivered.
wolfy-j added 22 commits October 3, 2026 17:57
Completion was reported on the execution's root frame, so after a
cross-source upgrade lifecycle observers saw the original definition (the
otel span was named after the source the process upgraded away from). The
scheduler reports completion on the current incarnation's context again, and
hosts complete and release the frame they created through
ctxapi.ExecutionFrame, which a continuation frame records.
Admitted processes start from a clean frame so they inherit no caller
authority, which also cut them off from the caller's trace. The admitter now
adds ctxapi.PropagatorPairs: the caller's values that implement Propagator,
the values designed to cross into another process, in their cross-process
form.
A process started by a process with a child limit did not carry the limit,
so a child of an eval could start any number of processes of its own.
The limit now follows the tree: descendants draw on their ancestor's slots.
A detached eval is not owned, so what it spawned was not owned either and
outlived the eval and its bounded lifetime. A detached eval now owns every
process it starts.
The metrics module wrote to the host collector without consulting the
security scope, so an eval admitted with it recorded arbitrary series
though its policy allows no such action. Recording is now refused when the
scope denies metrics.record; scopes without an opinion are unaffected.
…n of privileged modules

Eval code reaches no dynamic loading, debug or host access, cannot alter
tables shared across processes or leak state into other evals, finds no
imports or bindings changed by another run, keeps explicit send grants
against lookups, starts children that stay under its policy though their
entries configure allow-all, and still ends with its owner when it traps
links and retries upgrades.
process.registry.lookup resolved any registered name for an eval and
granted it the PID, so a capability-mode eval could discover and message
any named process. The lookup now passes the security scope, which denies
it for evals unless the policy allows the lookup command. Processes whose
scope has no opinion are unaffected.
…f running evals

Spawning a compiled program with a method option ignored it and ran the
compiled method. The spawn method now overrides it.

The program cache evicted by recency alone, so a program whose evals still
ran could be dropped and its handle fail on the next spawn. Running evals
now pin their program and capacity trimming skips pinned programs. Evicting
a program that is not cached reports ErrEvalProgramNotFound, and Evict
returns only an error.
Commands are command IDs: numeric IDs and the process control command names
parse, spawn and lookup are admitted, and every other command, including a
granted upgrade, fails admission with the plain eval policy unsupported
error. compile="jit" parses and fails admission with that error too.
Limits require integer values; a float such as 2.0 is rejected.
…to end

The test app configures a 2s lua.eval.detached_lifetime; a detached eval
spinning under the largest step limits ends when it elapses.
@wolfy-j
wolfy-j force-pushed the feature/lua-eval-host branch from 182e8d4 to 15e6795 Compare October 3, 2026 22:08
Context keys gain a Process flag independent of Inherit. The scheduler upgrade
path copies every process pair (pid, lifecycle options, terminal port) onto the
replacement frame instead of special-casing the pid.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants