Skip to content

boot: use embedded Lua cache in place, faster registry restore - #892

Merged
wolfy-j merged 6 commits into
mainfrom
perf/boot-startup
Oct 2, 2026
Merged

wolfy-j merged 6 commits into
mainfrom
perf/boot-startup

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

First launch after a binary change extracts, verifies and installs every embedded Lua cache entry, while registry transitions copy the entire working map after each accepted operation. This reads the binary's authenticated Lua cache from memory and updates only transaction-owned registry state.

Changes:

  • Authenticate the embedded archive's SHA-256 digest, cache schema and linked Lua toolchain once, index its immutable bytes in memory, and decode metadata/artifacts on demand. No seed-entry writes, markers, individual payload rehashing, or seed-time disk pruning. Source/dependency/typecheck fingerprints and artifact decoding still gate use; a rejected seed or entry falls back to persistent cache or compilation. New compiled entries remain in the bounded writable disk store, and cache modes/stage switches still apply. Update-time disk-cache merging adopts individually verified entry directories with rename and prunes once.
  • Reuse committed dependency declarations during overlay validation, while deriving group/namespace membership from the candidate state and re-extracting changed entries. Failed candidate validation never mutates the committed index.
  • Fix stale dependency indexes after registering new extraction patterns: serialize registration with mutation and invalidate the index only after successful registration. TestOverlayValidationRefreshesAfterDependencyPatternRegistration loads a durable entry referencing live:admission through data.late, registers that path later, then proves the forbidden durable-to-overlay dependency rejects admission immediately.
  • Apply accepted operations directly to the runner's private working map, keeping initial/rollback snapshots separate. Validation and component acceptance still precede mutation, public ApplyOperation still returns isolated map snapshots, and other builders retain the existing fallback. Tests cover rejected operations, kind changes, branch isolation, listener refusal, compensating effects and rollback.
  • Add optional structured pre-event-bus host boot phases, mesh start phases and independent owner-service start phases. Tests cover omitted/nil host loggers and successful/failed mesh and owner starts. README documents the implemented cache and diagnostic contracts.

Exported Go boundaries are intentional:

  • app.BootLogger and its BootLogger() *zap.Logger method let standalone native hosts supply their structured diagnostic logger before runtime logging/event wiring exists.
  • (*topology.DepIndex).VisitDependencies lets the registry validate candidate overlays against committed declarations across the package boundary without re-decoding unchanged entries.
  • (*topology.StateBuilder).ApplyOperationToPrivateState lets the runner opt into mutation of its unshared transaction map across the package boundary; callers must retain separate rollback snapshots and publish only committed state.
  • code.Config.EmbeddedCache, cache.Reader, and cache.NewLayeredStore connect the authenticated read-only seed to the existing Lua manager and writable cache. lua.cache.embedded accepts a typed native-host override, not YAML authorization.

Measurements (before = origin/main 728b75942, after = 0cac2f848; Go 1.27.0, Linux/amd64, Ryzen 7950X3D, GOMAXPROCS=32; medians of three runs, three iterations per run):

Benchmark Before median After median Allocated bytes before → after
Real Bee cache adoption, 1,657 verified entries (BenchmarkLuaSeedDirectory) 5,100.74 ms 116.31 ms 104.89 MB → 58.94 MB
Synthetic cache adoption, 906 entries (BenchmarkLuaSeed906) 1,689.08 ms 9.37 ms 20.28 MB → 3.86 MB
Actual registry runner transition, 1,000 entries (BenchmarkBootTransition1000) 455.56 ms 1.11 ms 285.33 MB → 0.952 MB
Map application only, 1,000 entries (BenchmarkPrivateBootStateOperations, snapshots → private) 215.63 ms 0.129 ms 285.42 MB → 0.344 MB

The first three comparisons use identical benchmark inputs on baseline/final worktrees. The last isolates old snapshot semantics and the new private-map path in one binary. Benchmarks use go test -run '^$' -bench NAME -benchtime=3x -count=3; real-cache adoption also sets WIPPY_LUA_SEED_BENCH_DIR to the verified snapshot root.

The real Bee snapshot contains 1,657 verified compile/typecheck cache entries in a 5,852,111-byte compressed archive; one corrupt retained metadata entry was excluded before making the identical benchmark input. Archive construction and state cleanup are excluded from adoption timing. The synthetic archive has 906 entries with 1 KiB proto payloads. The real transaction-runner benchmark restores 1,000 entries without component listeners; history IO and Lua compilation are excluded. The host was busy, so timings are diagnostics rather than an end-to-end Bee startup claim. Immutable bytes stay resident for the runtime invocation, bounded by the existing archive-size limit.

Validation passed:

  • make lint (0 issues).
  • make test-application (fts5/sqlite_vec/treesitter/sqlite_preupdate_hook).
  • go test and go vet on cmd/app, runtime/lua/code, runtime/lua/code/cache, boot/components/runtime/lua, system/registry/..., system/supervisor, and boot/components/system.
  • go test -race ./system/registry/... ./cmd/app, plus race on the Lua cache/manager and touched boot/supervisor packages.
  • New tests prove zero seed-entry disk writes, compile/typecheck hits, source/bytecode mismatch recompilation, immutable-read isolation, rejection of digest/schema/toolchain mismatches and unsafe/duplicate/truncated/oversized archives, and cache mode behavior.

The four local boot-runtime commits are preserved on a fresh perf/boot-startup branch, followed by the direct embedded-cache change and verification/benchmarks. No merge requested.

@wolfy-j
wolfy-j requested a review from skhaz October 2, 2026 16:45
@wolfy-j

wolfy-j commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Fixed the Windows CI failure in bd22713. The failing TestMergeLuaCacheMovesVerifiedEntriesAndReplacesCorruptDerivedData used os.Stat before rename; Windows resolves its identity lazily from the now-removed source path in os.SameFile. The test now captures identity via File.Stat on an open handle, closes that handle before the move, and keeps the identity/source-removal/content assertions. No production behavior changed. Linux race tests pass for cmd/app, Lua code/cache/lint, registry, and touched boot/supervisor packages; relevant go vet and formatting/diff checks pass. Fresh full CI is running, including the native Windows execution of this test. Rodrigo's review requested. No merge or release.

@wolfy-j

wolfy-j commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Final head bd22713 is green: all 13 GitHub checks pass, including the full native Windows test job and the previously failing cache-rename identity test. Rodrigo approved this exact head. Local race/vet/lint checks passed; the rename test also passed 30 repetitions and registry race suites 5 repetitions. A temporary, unpublished integration tree combining runtime #887/#889/#890/#891/#892 merged without manual conflict resolution and passed the combined targeted race suites, make test-application, and make lint (0 issues). That was a local compatibility check only: no GitHub PR has been merged and no runtime or module has been released.

@wolfy-j
wolfy-j merged commit 5eb9901 into main Oct 2, 2026
13 checks passed
@wolfy-j
wolfy-j deleted the perf/boot-startup branch October 2, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants