Skip to content

fix: stabilize microVM clocks and concurrent actor startup - #51

Merged
EItanya merged 2 commits into
mainfrom
fix/microvm-upstream-restore
Sep 29, 2026
Merged

EItanya merged 2 commits into
mainfrom
fix/microvm-upstream-restore

Conversation

@EItanya

@EItanya EItanya commented Sep 29, 2026 •

Copy link
Copy Markdown

Cloud Hypervisor v53 corrects the guest's kvm-clock across restore, but our amd64 guests selected tsc and resumed 3–14 seconds behind the host. Backport agent-substrate#1945 to select clocksource=kvm-clock for amd64 guests. ARM64 keeps its existing clock configuration; v53 handles its clock correction through cloud-hypervisor/cloud-hypervisor#8343. Existing amd64 golden snapshots must be recreated to adopt the boot argument.

The RNG fix from agent-substrate#1524, including mandatory reseeding and reuse of one Kata connection, is already in this fork's main.

Also fix the shared sysctl race exposed by this PR's gVisor E2E run. Concurrent actor starts use different network namespaces but share the /proc/sys mount. One call can remount it read-only while another is writing, causing actor startup to fail with EROFS; overlapping writes can also prevent the read-only remount. Serialize the entire write/remount sequence. The regression uses real network namespaces and a private mount namespace so it leaves the host untouched.

Validation:

  • The sysctl regression failed all 10 runs against the original helper and passed 20 runs with the fix and race detection.
  • Networking race tests, the full root-gated suite with race detection, and make verify passed.
  • Earlier local kagent validation of the RNG and clock fixes on amd64 passed the full microVM suite (185 cases, 10 existing skips), followed by 24 repeated checkpoint/fork and timeout cases after the RNG review fixes. Clock differences with kvm-clock stayed within about 0.21 seconds. These runs used the previous fork base; they have not been rerun on the newly synced base.

Related: kagent-dev/kagent#3004. CI still needs a Substrate release containing the fixes.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

(cherry picked from commit e63120e)
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
(cherry picked from commit b510de0)
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
@EItanya
EItanya force-pushed the fix/microvm-upstream-restore branch from ebc7c1b to 4feb48b Compare September 29, 2026 21:56
@EItanya EItanya changed the title fix: backport upstream microVM restore fixes fix: select kvm-clock for amd64 microVM guests Sep 29, 2026
@EItanya
EItanya marked this pull request as ready for review September 29, 2026 21:58
Actor setup changes network namespaces while retaining a shared mount namespace. Serialize writes and temporary remounts so one setup cannot restore /proc/sys to read-only while another is still writing. Exercise concurrent writes in isolated mount and network namespaces.

Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
@EItanya EItanya changed the title fix: select kvm-clock for amd64 microVM guests fix: stabilize microVM clocks and concurrent actor startup Sep 29, 2026
@EItanya
EItanya merged commit 228790e into main Sep 29, 2026
5 checks passed
EItanya added a commit that referenced this pull request Sep 30, 2026
* Select kvm-clock for amd64 micro-VM guests

(cherry picked from commit e63120e)
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
(cherry picked from commit b510de0)
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>

* fix: serialize sysctl writes across network namespaces

Actor setup changes network namespaces while retaining a shared mount namespace. Serialize writes and temporary remounts so one setup cannot restore /proc/sys to read-only while another is still writing. Exercise concurrent writes in isolated mount and network namespaces.

Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>

---------

Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Co-authored-by: Adhita Selvaraj <adhita.selvaraj@gmail.com>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants