Repository navigation
feat(ci): add a Windows guest image for a persistent self-hosted runner - #42
Virgile-pct wants to merge 3 commits into
Conversation
A Windows Server 2025 guest (dockur/windows, pinned by digest) run with Docker Compose on a Linux host with KVM. At first start it installs the mesh-llm Windows build prerequisites unattended (MSVC Build Tools, Rust, Node and pnpm, CMake, Ninja, sccache, just, CUDA 12.6, the Vulkan SDK and the HIP SDK through mesh-llm's own install-windows-sdk.ps1), and register-runner.ps1 turns it into a GitHub Actions runner service. It is a CI image, not a distribution channel: it contains no mesh-llm and is documented as an exception to the package-first image flow.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
|
|
||
| Write-Step 'Ninja 1.13.2, sccache 0.19.0, just 1.59.0' | ||
| Expand-ToolZip (Get-Download 'ninja-win.zip' 'https://github.com/ninja-build/ninja/releases/download/v1.13.2/ninja-win.zip' '07fc8261b42b20e71d1720b39068c2e14ffcee6396b76fb7a795fb460b78dc65') 'ninja.exe' | ||
| Expand-ToolZip (Get-Download 'sccache-v0.19.0-x86_64-pc-windows-msvc.zip' 'https://github.com/mozilla/sccache/releases/download/v0.19.0/sccache-v0.19.0-x86_64-pc-windows-msvc.zip' '28be8c61f6b2c89b9eb2869f6af7f78e6f992e78a8abae0ce855fcc129cbeac1') 'sccache.exe' |
There was a problem hiding this comment.
sccache 0.19.0 + stable Rust 1.100 cannot compile mesh-llm on Windows: since Rust 1.100 made Cargo's per-unit build-dir layout the default, sccache >= 0.17 expands rustc's -L dependency= argfile into a single command line, which fails with The filename or extension is too long. (os error 206) on this build graph. Every Windows lane on mesh-llm main and on same-window PRs has failed on exactly that since ~20:30Z today, and mesh-llm's fix (#3047) pins sccache to 0.16.0. Either pin 0.16.0 here to match, or document that workflows install their own sccache (sccache-action) and this PATH copy is for ad-hoc use only — as provisioned, any build in this guest that uses PATH sccache fails.
i386
left a comment
There was a problem hiding this comment.
Reviewed at head 27d724a89740c2ae7aaeb38d365f4d32950e19d1 — requesting one change, detailed inline.
Overall this is well built: the compose service pins dockur/windows by digest, binds all three ports (web console, RDP, SSH) to the host's loopback only, requires GUEST_PASSWORD explicitly, and the provisioning story is honest about licensing (180-day evaluation), GPU-less limits, and the CI-image-not-distribution-channel boundary, which keeps it on the right side of AGENTS.md's package rules. provision.ps1 is idempotent (Test-Installed per tool), verifies SHA-256 where publishers publish one, records digests for the rest in versions.txt, uses the correct administrators_authorized_keys + icacls pattern for OpenSSH key auth, and pulls the HIP SDK through mesh-llm's own install-windows-sdk.ps1 at a commit-pinned ref (I checked: the ref resolves on raw.githubusercontent and the script exists at that commit). register-runner.ps1 checksums the runner archive, runs --unattended --runasservice --replace, and never persists the token.
The one change needed: sccache 0.19.0 is the version currently breaking every mesh-llm Windows build. Since stable Rust moved to 1.100, sccache ≥ 0.17 expands rustc's per-unit -L dependency= argfile into a single command line, which fails with os error 206 on this build graph — every Windows lane on mesh-llm main and same-window PRs has been red on exactly that since ~20:30Z today, and mesh-llm's fix (#3047) pins sccache to 0.16.0. A guest provisioned with 0.19.0 on PATH reproduces the failure for any build that uses it. Either pin 0.16.0 here to match #3047, or state in the README that workflows install their own sccache (sccache-action) and the provisioned copy is for ad-hoc use only — but as written, the image ships a toolchain combination that cannot compile mesh-llm on Windows today.
CI: the only check on this head is CodeRabbit (skipped), so there's no automated signal; for a compose + provisioning change with no in-repo test surface that's acceptable — the verification story is booting the guest, as the README documents.
sccache 0.17 and later expand Cargo's rustc argfile, and with Rust 1.100's build-dir layout that command line passes the Windows limit (os error 206, mozilla/sccache#2787), so a guest with 0.19.0 cannot build mesh-llm. 0.16.0 is the version mesh-llm's Windows CI (#3047) and developer bootstrap pin.
|
good catch, pinned sccache to 0.16.0 in ae3862b to match Mesh-LLM/mesh-llm#3047. my build in the guest ran yesterday on Rust 1.99.1, before the 1.100 switch, that's why it didn't hit it |
…and a VM that stays up
|
Found why the guest stopped booting on my test host: under WSL2, the image's default Hyper-V enlightenments left an installed guest stuck at the boot logo with the CPU idle, and WSL2 also shut its VM down whenever no terminal was attached, killing the guest. With |
Why
A persistent Windows runner on lab hardware needs a Windows machine that comes up with every
mesh-llmbuild prerequisite, reproducibly. This adds one as a Docker Compose service: a Windows Server 2025 guest (dockur/windows, pinned by digest) on a Linux host with KVM, provisioned unattended at first start.It is a CI image, not a distribution channel: it never contains
mesh-llmand does not follow theproduct bundle -> native package -> runtime imageflow. The README says so.What
docker/windows-guest/compose.yml: the guest, ports bound to the host's loopback only, size and credentials from the environment.oem/provision.ps1: runs once at the end of setup. Installs OpenSSH, PowerShell 7.7.0, Git 2.56.1, CMake 4.4.5, Ninja 1.13.2, sccache 0.16.0 (the version mesh-llm's Windows CI pins since fix(ci): pin Windows sccache to 0.16.0 so Rust 1.100 builds fit the command line mesh-llm#3047), just 1.59.0, VS 2022 Build Tools (VC tools, Windows SDK 26100), Rust (rustup 1.29.3, stable), Node 26.13.1 with pnpm 10.34.5, CUDA 12.6.3, the Vulkan SDK and runtime 1.4.328.1, and the HIP SDK through mesh-llm's ownscripts/install-windows-sdk.ps1at a pinned commit, so the guest and hosted CI get the same SDK. Checksums are verified where the vendor publishes one and recorded otherwise. Each step skips what is already installed, so the script also repairs a guest.oem/register-runner.ps1: registers the guest as a GitHub Actions runner service (actions/runner 2.338.2, checksum verified), labelswindows,x64,mesh-windows-guest. The registration token is never written to disk.README.md,.gitignore(guest disk and generated files),.gitattributes(CRLF for the.batthat cmd.exe runs).Validation
On a Linux host (WSL2 Ubuntu with
/dev/kvm, Docker 29.1.3; guest at 8 GB RAM, 6 cores, 128 GB disk):Unattended install and provisioning completed;
C:\OEM\versions.txtlists every tool at its pinned version (MSVC 17.14, rustc 1.99.1, nvcc 12.6,VULKAN_SDK1.4.328.1,HIP_PATHROCm 6.4).A provisioned guest uses about 29 GB inside Windows and about 35 GB on the host, including the cached 8 GB ISO.
mesh-llmmain (e506096c5) built inside the guest, with Rust 1.99.1 and the sccache 0.19.0 the guest had then (before Rust 1.100; the pin to 0.16.0 came after review and was not rebuilt in the guest) with the repository'sscripts/build-windows.ps1: CUDA (-CudaArch 89), Vulkan and ROCm (-RocmArch gfx1100) all exit 0 (about 60, 11 and 23 minutes; the CUDA run resumed a build cut by the stop described below and includes the Rust host).mesh-llm --versionruns, andtarget/debug/native-runtimesholds the cuda12, vulkan and rocm runtimes withggml-cuda.dll,ggml-vulkan.dllandggml-hip.dll.install-windows-sdk.ps1expects GitHub Actions variables (RUNNER_TEMP);provision.ps1sets it to its download cache. Its Vulkan path uses Chocolatey, so the guest installs the Vulkan SDK and runtime itself and only takes the HIP part from that script.The precheck's source-build policy (
git clone|cargo build|pnpm.*buildunderdocker/) finds nothing in these files: the guest clones and builds nothing itself; jobs do. No.shfile is added.Limits