Skip to content

fix: unbreak mise run apply on the GPU builds - #410

Merged
odsod merged 3 commits into
mainfrom
fix/apply-failures
Nov 19, 2026
Merged

odsod merged 3 commits into
mainfrom
fix/apply-failures

Conversation

@odsod

@odsod odsod commented Nov 19, 2026

Copy link
Copy Markdown
Owner

Why

mise run apply failed in llama:setup and logged a Herdr warning on every run.

The llama failure was not a broken toolchain. Tarball builds have no git
history, so llama.cpp resolved BUILD_NUMBER 0, asked the Hugging Face bucket
for UI assets tagged b0 (which does not exist), and fell back to latest.
Those assets come from a newer UI with no workbox-*.js, so the release's own
asset validator aborted the build before llama-server linked.

The Herdr warning was an offline Tailscale peer. hc machine add needs ssh, so
the sync timed out on odsod-x1-carbon (last seen 10 days ago) and reported a
standing failure.

What

  • fix(llama): pass the tag's bNNNN as LLAMA_BUILD_NUMBER so the bucket
    serves the assets built for that release. Tags without a build number, such
    as v0.9.0, skip the flag.
  • fix(herdr): only sync peers with .Online == true. Offline machines are
    added by the first apply after they reappear.
  • chore(bootstrap): install ccache. Both GPU builds warn when it is
    missing, and both rebuild from source on version bumps.

Test plan

  • mise run apply exits 0
  • mise bootstrap --dry-run reports no pending work
  • llama and whisper /health return 200; llama UI at :8179 returns 200
    with gzip
  • chat completion returns OK with system_fingerprint: b12359-unknown,
    which confirms the build number reaches the binary
  • mise run check passes
  • herdr:sync-machines runs without a warning

Verified with llama_version = "1.0.0-b12359-b1b770d", bumped separately. On
0.9.0 the new flag is skipped, so this change is inert until that bump lands.

- Tarball builds have no git history, so llama.cpp resolves BUILD_NUMBER 0
  and asks the HF bucket for UI assets tagged "b0".
- That bucket does not exist, so it falls back to "latest", whose asset set
  lacks workbox-*.js and fails the release's own asset validation.
- Pass the tag's bNNNN as LLAMA_BUILD_NUMBER so the matching assets are
  fetched.
- Omit the flag for tags without a build number, such as v0.9.0.
- The Tailscale peer list included offline machines.
- `hc machine add` needs ssh, so it timed out and logged a warning on every
  apply.
- Filter on .Online so only reachable peers are attempted.
- Offline machines are added by the first apply after they reappear.
- llama.cpp and whisper.cpp configure warn when ccache is missing.
- Both services rebuild from source on version bumps, so the cache pays off.
@odsod
odsod merged commit 73d88e8 into main Nov 19, 2026
1 check passed
@odsod
odsod deleted the fix/apply-failures branch November 19, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant