[bborbe]

Problem

DiskRootOutOfSpace fires on nuke-k3s-prod-worker-0 when / free space drops below 20%. The kubelet's image GC starts at image-gc-high-threshold=80, which is 80% used — i.e. exactly 20% free. The GC trigger and the alert threshold are the same line, so the node sits on the alert line by design and undershoots it at every collection trough.

Measurement (2026-10-23, read-only)

Signal	Value
kubelet_image_garbage_collected_total{reason="space"} over 24h	144 (one every ~10 min)
Image pulls over 24h	2,568 (387 of them in the 1–5 GB bucket)
Free fraction over 24h	10.5% – 42.2%, below 20% in 32/97 samples (33%)
Pearson r(completed-pod count, free fraction)	0.222
Free fraction with only 8–13 completed pods resident	still fell to 13.7%

Each GC cycle collects from 80% used down to 70% used — about 7.6 GiB on a 76.45 GiB rootfs. At 144 cycles/day that is roughly 1 TiB/day of image data churning through the store.

Change

image-gc-high-threshold 80 → 70, image-gc-low-threshold 70 → 60, on this node only.

GC now starts at 70% used (30% free) and collects to 60% used (40% free), keeping the trough 10 points above the alert line instead of sitting on it.

Scope

nuke-k3s-prod-worker-0 only — it is the node carrying the alert and the backtest job churn. hm.nuke-k3s-prod-master-0 and hm.nuke-k3s-dev-worker-0 keep 80/70; they run no comparable job volume.

Risk and rollback

Raising GC aggressiveness means images are evicted earlier, so pods using mutable :prod tags may re-pull sooner. It does not change eviction-hard=imagefs.available<10%. Rollback is reverting this commit and re-applying — the kubelet restarts with the previous args.

This does not reduce the underlying pull churn (imagePullPolicy: Always across ~20 workloads with mutable :prod tags); that is tracked separately.

Validation

• make precommit (bw test --ignore-missing-faults) passes; hm.nuke-k3s-prod-worker-0 has no metadata conflicts.
• Applying to the node is a separate, operator-run step.

[ben-s-pull-request-reviewer]

{
  "verdict": "approve",
  "summary": "Config-only change raising kubelet image-GC headroom on one prod worker: high/low 70/60 is valid percent-used semantics (70 > 60) and GC now starts 10 points before the 20%-free alert and well before the 10%-free eviction floor. Mechanical funnel returned 0 findings; selector mode matched 8 candidate rules, 1 applicable (git-commit/imperative-mood — commit subject uses imperative 'start'), no violations; precommit skipped (selector mode) — CI covers lint+test; no .reviewignore exclusions.",
  "comments": [
    {
      "file": "nodes/hm.nuke-k3s-prod-worker-0.py",
      "line": 48,
      "severity": "nit",
      "blocking": false,
      "blocking_reason": "",
      "message": "Nice to Have (Optional): the new comment explains the alert-vs-GC relationship generically, while the 14 other node files still set 80/70. Consider stating in the comment that this override is node-specific (the 144 GCs / 2,568 pulls / 10.5%-42.2% free-space measurement is from this worker) so a future reader does not assume the other nodes are misconfigured. Non-blocking."
    }
  ],
  "concerns_addressed": [
    {
      "concern": "Verify kubelet threshold semantics: both values are percent used, so 70/60 leaves 30%/40% free. Confirm high > low is enforced and that 60 does not push GC into continuous collection against `eviction-hard=imagefs.available<10%`.",
      "disposition": "not-an-issue",
      "detail": "kubelet image-gc-high/low-threshold are percent of imagefs used; 70 > 60 satisfies kubelet's high>low constraint. GC at 70% used (30% free) still runs 20 points before eviction-hard imagefs.available<10% (90% used), so the two do not collide and low=60 targets 40% free, not continuous collection."
    },
    {
      "concern": "Only this node diverges to 70/60 while hm.nuke-k3s-prod-master-0 and hm.nuke-k3s-dev-worker-0 stay at 80/70. Check whether a shared default or per-node override mechanism exists and whether the divergence is intentional and documented.",
      "disposition": "not-an-issue",
      "detail": "No shared default exists — each nodes/*.py hardcodes its own kubelet-arg list; 14 other node files use 80/70. The divergence is intentional and documented in the commit body (node-specific 24h measurement of GCs, pulls, and free-space oscillation) plus the new inline comment."
    },
    {
      "concern": "More aggressive GC evicts images earlier. With imagePullPolicy Always and mutable :prod tags, this can increase re-pulls beyond the measured 2,568/day. Check whether the claimed reduction in churn is actually supported or whether the change only moves the alert line while adding registry/network load.",
      "disposition": "not-an-issue",
      "detail": "The PR does not claim a pull-count reduction; it trades earlier GC for headroom above the 20%-free alert. Pull churn may rise somewhat and is worth watching post-rollout, but that is an operational tradeoff, not a defect in the change; no pull-policy or registry setting is touched by the diff."
    },
    {
      "concern": "The DiskRootOutOfSpace alert (fires below 20% free) is not part of this diff. Check whether the alert threshold itself was considered as an alternative or complement, and whether other nodes carrying the same alert need the same change.",
      "disposition": "not-an-issue",
      "detail": "The alert rule is not defined anywhere in this repo (grep for DiskRootOutOfSpace/imagefs found only the kubelet args), so it is managed externally. Changing GC to keep free space above the alert is the complement rather than the alternative; no other node has a measured pressure signal in this PR, so widening the change would be speculative."
    },
    {
      "concern": "No automated validation of kubelet args is visible. Confirm whether the repo has golden/snapshot tests or config validation for node definitions beyond the metadata-conflict check in make precommit, and whether those need updating.",
      "disposition": "not-an-issue",
      "detail": "Config-only node definition; the repo has no golden/snapshot tests for kubelet args — make precommit runs `bw test --ignore-missing-faults` (live bundle test against hosts). precommit skipped in selector mode; CI exercises it."
    },
    {
      "concern": "Merging does not apply the change; rollout is a separate operator step and rollback is manual. Check that the PR documents this operationally and that no automation expects the live node to change on merge.",
      "disposition": "not-an-issue",
      "detail": "bundlewrap applies node config via `bw apply`; merge alone does not touch hosts, and no automation in this repo expects the live node to change on merge. Rollout and rollback remain the standard manual operator steps used for every other node change."
    },
    {
      "concern": "Low priority: confirm the change does not weaken image integrity guarantees by evicting images in a way that interacts with pull policy or image verification on this node.",
      "disposition": "not-an-issue",
      "detail": "The diff only changes image-GC percentages; it does not touch imagePullPolicy, registries, or any image-verification configuration. Evicting unused images earlier does not alter integrity guarantees — re-pulled images go through the same pull path."
    }
  ]
}