Skip to content

UCI rigs cannot queue behind a contended device — DeviceLock stops extending on the fourth holder handoff #212

Description

@JC-000

boot_check.py, rig_https_local.py and rig_https_banner.py hardcode their DeviceLock acquire timeout (60 s / 120 s / 120 s) with no environment override.

The mechanism is not an insufficient literal

DeviceLock.acquire is queue-aware. With the default progress_window=60.0, while the holder's PID is alive and its lockfile mtime is fresh, the deadline is re-armed on every poll and timeout is never consulted. Measured hardware-free:

queue-aware (progress_window=60): acquire(timeout=2.0) behind an 8.0s live holder -> True after 8.0s
legacy      (progress_window=None):                                               -> False after 2.0s

So the hardcoded 120 s would have outlasted an 80-minute run behind a single healthy holder.

What defeats it is the handoff chain. _MAX_HOLDER_HANDOFFS = 3 (device_lock.py:166) is the number of holder-identity changes tolerated; on the fourth, extension stops permanently for that acquire and the literal becomes the entire budget. With several lanes cycling one device, that is reached in seconds.

The fix is therefore an env override for the acquire timeout and preserving progress_window — a larger literal alone does not address the handoff chain.

Upstream

  • JC-000/c64-test-harness#232 — the docstring off-by-one ("after 3 handoffs" vs "on the fourth"). Filed 2026-09-07. Note that the ambiguous phrasing propagated into five files before it was caught; this issue's own title was corrected for the same reason.
  • JC-000/c64-test-harness#233 — env default plus a progress callback. Filed 2026-09-07.

Observed

2026-09-07 on U64E 10.43.23.81. A peer's tools/test_ip65_rrnet_hw.py held the lock for a run of the expected 40-80 minute duration. The rigs could not be launched to queue behind it; the workaround was polling device_lock_holder() read-only every 30 s and launching into the gap, which races any newcomer. Later in the same session the device was held by another lane with DeviceLock.peek_queue_depth() reporting 3 waiters, so the multi-lane contention that triggers the handoff chain is routine, not hypothetical.

Second, related: device config does not survive between lanes

A peer took the lock between two of my runs and left the device at CPU Speed 1 MHz with the REU Disabled. The next comb run's boot precompute (~45 s at 48 MHz, ~36 min at 1 MHz) could not finish inside the rig's 120 s boot budget, and the rig failed at boot with the screen stuck on comb precompute - 45s at 48mhz.

That is a device-config failure that reads exactly like a code defect. It was diagnosed only because the re-prep printed its before-state: REU before: enabled=False, turbo before: Manual/1. Project memory records agents repeatedly misdiagnosing this class as "U64E wedged / firmware corruption" and taking unnecessary destructive recovery action.

Every comb run needs a prep immediately before it, not once per session. The prep must read back and report its before-state, or the next occurrence will be misdiagnosed the same way.

Fix status

A fix exists on local branch fix/uci-devicelock-timeout (not pushed to origin at the time of writing).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions