Skip to content

fix: keep Spotlight from failing smoke cleanup - #756

Draft
sudoHG wants to merge 3 commits into
mainfrom
sudoHG/shadow-spotlight-cleanup
Draft

sudoHG wants to merge 3 commits into
mainfrom
sudoHG/shadow-spotlight-cleanup

Conversation

@sudoHG

@sudoHG sudoHG commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Prevent transient Spotlight file handles from turning a passing smoke-shadow group red during cleanup. Shadow 43803713182, controlled by main 2936b4b, tested the paired nightly 43774339730 source 9805b95. All four iOS groups reached flows with setup_failure_step: null, confirming #753's setup fix. At 2026-11-16T05:16:27Z, iphone-debug job 133095989044 reported cleanup-failed: step=lsof-holders and cleanup-holder: family=spotlight.

That group had already timed out with runner exit 124; this fix does not repair its flow failure. The other five groups and both producers cleaned up successfully. Historical cleanup failures remain unproven individually.

What changed

  • Rename only the consumer root to $RUNNER_TEMP/smoke-execution.noindex, following the Spotlight exclusion convention.
  • For either consumer or producer cleanup, allow exactly one re-probe only after lsof returns 0, nonempty stdout, empty stderr and exactly the spotlight family. A plain if emits cleanup-retry: family=spotlight attempt=1, sleeps 5 seconds and repeats the same command with the same 30-second timeout. Attribution comes from the last probe; nonempty stdout with exit 0 or 1 remains lsof-holders with family lines even alongside stderr, as in fix: prepare smoke archive consumers before identity checks #753. Stderr still prevents a re-probe.
  • Extend existing tests and document the cleanup contract in docs/CI_NIGHTLY.md.

The deletion condition is identical (returncode == 1, empty stdout and empty stderr). A final probe reporting any holder retains the root. The added delay lets transient Spotlight holds clear. The .noindex suffix is conventionally skipped by Spotlight; its hosted effect is NOT_VERIFIED, and the single re-probe is a fallback.

What deliberately did not change

The producer root smoke-producer, freshness and ownership checks, smoke-records, uploaded contents, shutdown/delete order, check=True for simulator deletion, inherited delete stderr, 60/60/30-second caps and fail-closed retention stay intact. No workflow, workflow policy, runner source, App behavior, flow budget, verdict, retry/yellow rule or acceptance threshold changed. The approved 23-flow migration and its 5–7 shadow / 10 actually-executed-night gates remain unchanged.

Repository-wide path search found only the consumer implementation and its existing tests, plus scripts/test_check_workflow_policy.py:158. That last reference deliberately tests a forbidden upload path and remains unchanged; there were no existing documentation or diagnostic readers of the old path.

Time budget

The coordinator accepts the full cleanup tail: one re-probe adds at most 5 + 30 = 35 seconds. The complete cleanup worst case is 60 + 60 + 30 + 5 + 30 = 185 seconds for flow groups (previously 150) and 30 + 5 + 30 = 65 seconds for producers (previously 30). No timeout changes.

Jobs Inner budget Job timeout Cleanup worst case, before → after
iPhone / iPad Debug and Release groups 27 min 30 min 150 → 185 s
TV Debug and Release groups 22 min 25 min 150 → 185 s
ios-debug / ios-release producers 27 min 30 min 30 → 65 s

Flow job timeouts come from matrix.timeout_minutes, generated as the group budget plus the unchanged three-minute reserve. Producer job timeouts remain 30 minutes. For a flow group, reaching this worst-case cleanup tail against the reserve requires the group to have already used its whole inner budget; that group is already red with exit 124. A passing group finishes well inside its budget. The reserve also covers finalization and uploads, so worst-case cleanup can hit the job timeout. If that timeout kills cleanup, the job stays red and the root is retained; nothing can turn green. The coordinator explicitly accepts this tail without changing any timeout.

Verification

Final commit: 561af8f1e54784fdcf20d7a21bb994e269b521c6, based on 2936b4b4b43871f0545f8d6ea647d496f1e198ad.

Command / check Result
Nine static entries in scripts/run_host_checks.py HOST_CHECKS (excluding the separate Python-suite entry) PASS, all exit 0; workflow policy: 17 workflows, 0 violations
python3 -B -m unittest discover -s scripts -p test_ci_smoke_shadow.py PASS, 56 tests, exit 0
python3 -B -m unittest discover -s scripts -p test_check_workflow_policy.py PASS, 52 tests, exit 0
python3 -B scripts/git_privacy_gate.py --range origin/main..HEAD PASS, 6 blobs / 2 metadata records, exit 0
git diff --check PASS, exit 0
iOS / tvOS build and Swift tests NOT_RUN: Python cleanup and documentation only; no Swift or target membership changes
Full scripts/check_all.sh / full Python population NOT_RUN: ran the requested nine static checks and the two related Python suites

Round 2 changed the existing stderr sub-case first: the previous implementation failed on both roots (exit 1, two assertion failures), and the corrected implementation passed. Coverage retains clear/persistent Spotlight, mixed/non-Spotlight families, stderr, other return codes, re-probe launch errors, re-probe timeout and last-probe family attribution on both roots. Deleted the duplicate empty-stdout test and the duplicate initial-launch-error sub-case; existing lsof-empty and lsof-unavailable cases still cover those behaviors. The nonempty-stdout trigger remains as a defensive check.

Mutation proofs

Each mutation was compiled in memory against the final source without editing tracked files. The named existing test methods ran against the mutant and again immediately after restoring the original function code.

retry = SmokeShadowTests.test_cleanup_reprobes_only_transient_spotlight_holders_once_before_deleting.

Mutation Detecting tests Mutant exit → restored exit
Zero re-probes (if False and ...) retry 1 → 0
Two re-probes (duplicate the guarded probe block) retry: held consumer and producer cases both fail because the decoy third probe returns clear and cleanup incorrectly succeeds 1 → 0
Remove return-code trigger retry 1 → 0
Remove nonempty-stdout trigger behavior-equivalent (families of empty stdout is ['other']) 0 → 0; not a killed mutation
Remove empty-stderr trigger retry 1 → 0
Remove Spotlight-only family trigger retry 1 → 0
Remove sleep retry 1 → 0
Change sleep from 5 to 4 seconds retry 1 → 0
Require empty stderr for holder attribution retry: stderr consumer and producer cases both fail 1 → 0
Revert .noindex rename archive-group expectation, ownership/foreign-input cleanup and cleanup-failure attribution tests 1 → 0 (all 3 restored tests pass)

Local Spotlight and cleanup evidence

The round 1 $TMPDIR experiment was nondiscriminating and is superseded. Round 2 created identical unique random content in a.noindex/proof.txt and b/proof.txt under /Users/hutong/Coding/immichSlides/worktrees/spotlight-proof-ftu8ua80/, outside any git checkout. Queried mdfind -onlyin <parent> <token> every 10 seconds for 180 seconds, including the initial and final polls (19 queries, all exit 0, final elapsed 180.1 seconds). Neither file was ever found: b_ever_found=false, a_ever_found=false. Because b was never found, the exclusion proof is NOT_VERIFIED. lsof +D <parent> returned 1 with zero stdout/stderr bytes; the scratch parent was then deleted and its absence read back. No Spotlight settings changed.

Round 1 (f557222) ran the real ci_smoke_shadow.py cleanup CLI after real ci_build_archive.pack_products / extract_products of a synthetic, secret-free Products archive into smoke-execution.noindex/derived/Build: exit 0, zero retry lines, root removed. No cleanup subprocess or filesystem operation was mocked. This was a synthetic archive, not a compiled App or hosted archive, and it was not repeated in round 2. The scratch parent was also removed after an empty holder probe.

Not verified and risks

  • Hosted proof: NOT_RUN; observe the first smoke-shadow after merge. No workflow dispatch, CI approval, ready-for-review flip or merge was performed.
  • Local Spotlight exclusion proof is NOT_VERIFIED because its ordinary-folder control never appeared in the index.
  • One re-probe can still leave persistent holders; that remains an intentional red failure with retained inputs. This fix does not resolve the observed runner exit 124 or prove the cause of historical cleanup failures.
  • No App/UI/localization changes; device UI, Swift builds/tests and screenshots are NOT_RUN for this script-only change.

Cross-model review

Claude Opus 5.5 (xhigh) reviewed head f557222 (read-only): MERGEABLE AFTER FIXES. Fail-closed confirmed: no path removes the root after any probe reported a holder or stderr; the only exit 0 is a final lsof exit 1 with both streams empty. Coordinator rulings, all implemented in 561af8f:

Finding Ruling Resolution
M1: the 145 s margin counted only the re-probe; the full cleanup worst case is 185 s (flow) / 65 s (producer) Accept the tail: it is reachable only after a group has exhausted its inner budget and is already red (exit 124); a job-timeout kill stays red and retains the root Docs and body now state the full worst case; the undefined "two-minute minimum" wording was removed; no timeout changed
L2: lsof-holders attribution newly required empty stderr (caused by a contradictory spec line) Option (b): restore #753's rule Line restored to its origin/main form; the stderr sub-case now expects no retry, lsof-holders, family=spotlight
L3: the empty-stdout test guarded a behavior-equivalent mutation; the new unavailable sub-case duplicated lsof-unavailable Accept Both removed; the non-empty-stdout condition is kept as defensive and marked behavior-equivalent
L4: docs stated the .noindex effect as fact Accept Reworded: conventional Spotlight skip, hosted effect NOT_VERIFIED, re-probe is a fallback
Nit: for attempt in range(1, 2) Accept Replaced with a single if; the held cases' decoy third probe kills a second-re-probe mutation

The coordinator self-verified the round-2 delta (f557222..561af8f, 3 files, +20/-31) against these rulings. The local Spotlight proof stayed NOT_VERIFIED in an indexed-candidate location as well; the hosted effect will be read from the first post-merge shadow's cleanup lines.

Round 3: test isolation

Head: 1a8ac5d6ec2e827fac0b08a57afb262ad3161577.

The archive-mode tests imported run_strict_ci_tracer under dependency patches, caching three MagicMock attributes: file_hash, workspace_preflight and prepare_simulator. The support module's preparation function was restored afterward, but the tracer's alias was not. Restoring only that alias eliminated all five tracer subtest failures.

Only scripts/test_ci_smoke_shadow.py changes: import the tracer before dependency patches and patch its run_runner directly. After both tests, tracer mock attributes are empty and its preparation function matches the support module. Unmocked production phase processes are unaffected; assertions, order, skips, runners, workflows and timeouts are unchanged.

Both culprit pairs use the existing tracer test StrictCITracerTests.test_submitted_boot_settles_during_cold_build_and_gates_every_warm_case after one of:

  • SmokeShadowTests.test_archive_group_verifies_planned_expectations_before_slow_simulator_creation
  • SmokeShadowTests.test_setup_failures_name_the_step_and_keep_subprocess_timeouts_distinct_from_the_group_budget

Each pair passes (2 tests, exit 0). Reverting the fix in memory makes each fail with the same five subtest failures (exit 1); restoring it passes again (exit 0). The original controller/tests from origin/main 2936b4b4b43871f0545f8d6ea647d496f1e198ad fail both pairs identically (exit 1), proving this predates the PR.

Final-head verification Result
python3 -B -m unittest test_ci_smoke_shadow test_run_strict_e2e 318 passed in one process, exit 0
python3 -B scripts/run_host_checks.py --timeout-seconds 1200 One Python process: 1,848 passed / 5 existing calibration skips; all nine static checks pass; exit 0
Same entry point with --host-platform linux --python-shards 2 --timeout-seconds 1200 1,847 passed / same 5 skips, exit 0; both related modules together in shard 1
Workflow policy / privacy range / both diff whitespace checks 17 workflows, 0 violations / 7 blobs, 3 metadata records / clean; all exit 0

Both host runs have no failures/errors or infrastructure findings; these supersede the earlier full-suite NOT_RUN entry. The five unchanged ReviewedScreenshotCalibrationTests need the external screenshot dataset. Linux selection/sharding was exercised locally on macOS. Local App builds, offline Swift unit/UI suites, simulator runs, screenshots and post-push hosted verification remain NOT_RUN; repository behavior docs require no change.

Round 3 (561af8f..1a8ac5d, test-only, +4/-2): the coordinator found the hosted host-linux failure (CI run 43814331354), reproduced it in-process on both this head and origin/main 2936b4b, and bisected it to the two #753 run_group archive-mode tests. The worker identified the leak: patch("run_strict_ci_tracer.run_runner") first-imported the tracer while strict_e2e_runner_support.prepare_simulator (and two other dependencies) were patched, so the tracer kept the mocks as import-time aliases. Coordinator self-verified the fix (import the tracer before patching, patch run_runner on the module object) and re-ran the failing pair locally: OK.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant