Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prevent transient Spotlight file handles from turning a passing smoke-shadow group red during cleanup. Shadow 43803713182, controlled by main
2936b4b, tested the paired nightly 43774339730 source9805b95. All four iOS groups reached flows withsetup_failure_step: null, confirming #753's setup fix. At2026-11-16T05:16:27Z, iphone-debug job 133095989044 reportedcleanup-failed: step=lsof-holdersandcleanup-holder: family=spotlight.That group had already timed out with runner exit 124; this fix does not repair its flow failure. The other five groups and both producers cleaned up successfully. Historical cleanup failures remain unproven individually.
What changed
$RUNNER_TEMP/smoke-execution.noindex, following the Spotlight exclusion convention.spotlightfamily. A plainifemitscleanup-retry: family=spotlight attempt=1, sleeps 5 seconds and repeats the same command with the same 30-second timeout. Attribution comes from the last probe; nonempty stdout with exit 0 or 1 remainslsof-holderswith family lines even alongside stderr, as in fix: prepare smoke archive consumers before identity checks #753. Stderr still prevents a re-probe.docs/CI_NIGHTLY.md.The deletion condition is identical (
returncode == 1, empty stdout and empty stderr). A final probe reporting any holder retains the root. The added delay lets transient Spotlight holds clear. The.noindexsuffix is conventionally skipped by Spotlight; its hosted effect is NOT_VERIFIED, and the single re-probe is a fallback.What deliberately did not change
The producer root
smoke-producer, freshness and ownership checks,smoke-records, uploaded contents, shutdown/delete order,check=Truefor simulator deletion, inherited delete stderr, 60/60/30-second caps and fail-closed retention stay intact. No workflow, workflow policy, runner source, App behavior, flow budget, verdict, retry/yellow rule or acceptance threshold changed. The approved 23-flow migration and its 5–7 shadow / 10 actually-executed-night gates remain unchanged.Repository-wide path search found only the consumer implementation and its existing tests, plus
scripts/test_check_workflow_policy.py:158. That last reference deliberately tests a forbidden upload path and remains unchanged; there were no existing documentation or diagnostic readers of the old path.Time budget
The coordinator accepts the full cleanup tail: one re-probe adds at most
5 + 30 = 35seconds. The complete cleanup worst case is60 + 60 + 30 + 5 + 30 = 185seconds for flow groups (previously 150) and30 + 5 + 30 = 65seconds for producers (previously 30). No timeout changes.ios-debug/ios-releaseproducersFlow job timeouts come from
matrix.timeout_minutes, generated as the group budget plus the unchanged three-minute reserve. Producer job timeouts remain 30 minutes. For a flow group, reaching this worst-case cleanup tail against the reserve requires the group to have already used its whole inner budget; that group is already red with exit 124. A passing group finishes well inside its budget. The reserve also covers finalization and uploads, so worst-case cleanup can hit the job timeout. If that timeout kills cleanup, the job stays red and the root is retained; nothing can turn green. The coordinator explicitly accepts this tail without changing any timeout.Verification
Final commit:
561af8f1e54784fdcf20d7a21bb994e269b521c6, based on2936b4b4b43871f0545f8d6ea647d496f1e198ad.scripts/run_host_checks.py HOST_CHECKS(excluding the separate Python-suite entry)python3 -B -m unittest discover -s scripts -p test_ci_smoke_shadow.pypython3 -B -m unittest discover -s scripts -p test_check_workflow_policy.pypython3 -B scripts/git_privacy_gate.py --range origin/main..HEADgit diff --checkscripts/check_all.sh/ full Python populationRound 2 changed the existing stderr sub-case first: the previous implementation failed on both roots (exit 1, two assertion failures), and the corrected implementation passed. Coverage retains clear/persistent Spotlight, mixed/non-Spotlight families, stderr, other return codes, re-probe launch errors, re-probe timeout and last-probe family attribution on both roots. Deleted the duplicate empty-stdout test and the duplicate initial-launch-error sub-case; existing
lsof-emptyandlsof-unavailablecases still cover those behaviors. The nonempty-stdout trigger remains as a defensive check.Mutation proofs
Each mutation was compiled in memory against the final source without editing tracked files. The named existing test methods ran against the mutant and again immediately after restoring the original function code.
retry=SmokeShadowTests.test_cleanup_reprobes_only_transient_spotlight_holders_once_before_deleting.if False and ...).noindexrenameLocal Spotlight and cleanup evidence
The round 1
$TMPDIRexperiment was nondiscriminating and is superseded. Round 2 created identical unique random content ina.noindex/proof.txtandb/proof.txtunder/Users/hutong/Coding/immichSlides/worktrees/spotlight-proof-ftu8ua80/, outside any git checkout. Queriedmdfind -onlyin <parent> <token>every 10 seconds for 180 seconds, including the initial and final polls (19 queries, all exit 0, final elapsed 180.1 seconds). Neither file was ever found:b_ever_found=false,a_ever_found=false. Because b was never found, the exclusion proof is NOT_VERIFIED.lsof +D <parent>returned 1 with zero stdout/stderr bytes; the scratch parent was then deleted and its absence read back. No Spotlight settings changed.Round 1 (
f557222) ran the realci_smoke_shadow.py cleanupCLI after realci_build_archive.pack_products/extract_productsof a synthetic, secret-free Products archive intosmoke-execution.noindex/derived/Build: exit 0, zero retry lines, root removed. No cleanup subprocess or filesystem operation was mocked. This was a synthetic archive, not a compiled App or hosted archive, and it was not repeated in round 2. The scratch parent was also removed after an empty holder probe.Not verified and risks
Cross-model review
Claude Opus 5.5 (xhigh) reviewed head
f557222(read-only): MERGEABLE AFTER FIXES. Fail-closed confirmed: no path removes the root after any probe reported a holder or stderr; the only exit 0 is a final lsof exit 1 with both streams empty. Coordinator rulings, all implemented in561af8f:lsof-holdersattribution newly required empty stderr (caused by a contradictory spec line)origin/mainform; the stderr sub-case now expects no retry,lsof-holders,family=spotlightunavailablesub-case duplicatedlsof-unavailable.noindexeffect as factfor attempt in range(1, 2)if; the held cases' decoy third probe kills a second-re-probe mutationThe coordinator self-verified the round-2 delta (
f557222..561af8f, 3 files, +20/-31) against these rulings. The local Spotlight proof stayed NOT_VERIFIED in an indexed-candidate location as well; the hosted effect will be read from the first post-merge shadow's cleanup lines.Round 3: test isolation
Head:
1a8ac5d6ec2e827fac0b08a57afb262ad3161577.The archive-mode tests imported
run_strict_ci_tracerunder dependency patches, caching threeMagicMockattributes:file_hash,workspace_preflightandprepare_simulator. The support module's preparation function was restored afterward, but the tracer's alias was not. Restoring only that alias eliminated all five tracer subtest failures.Only
scripts/test_ci_smoke_shadow.pychanges: import the tracer before dependency patches and patch itsrun_runnerdirectly. After both tests, tracer mock attributes are empty and its preparation function matches the support module. Unmocked production phase processes are unaffected; assertions, order, skips, runners, workflows and timeouts are unchanged.Both culprit pairs use the existing tracer test
StrictCITracerTests.test_submitted_boot_settles_during_cold_build_and_gates_every_warm_caseafter one of:SmokeShadowTests.test_archive_group_verifies_planned_expectations_before_slow_simulator_creationSmokeShadowTests.test_setup_failures_name_the_step_and_keep_subprocess_timeouts_distinct_from_the_group_budgetEach pair passes (2 tests, exit 0). Reverting the fix in memory makes each fail with the same five subtest failures (exit 1); restoring it passes again (exit 0). The original controller/tests from
origin/main2936b4b4b43871f0545f8d6ea647d496f1e198adfail both pairs identically (exit 1), proving this predates the PR.python3 -B -m unittest test_ci_smoke_shadow test_run_strict_e2epython3 -B scripts/run_host_checks.py --timeout-seconds 1200--host-platform linux --python-shards 2 --timeout-seconds 1200Both host runs have no failures/errors or infrastructure findings; these supersede the earlier full-suite NOT_RUN entry. The five unchanged
ReviewedScreenshotCalibrationTestsneed the external screenshot dataset. Linux selection/sharding was exercised locally on macOS. Local App builds, offline Swift unit/UI suites, simulator runs, screenshots and post-push hosted verification remain NOT_RUN; repository behavior docs require no change.Round 3 (
561af8f..1a8ac5d, test-only, +4/-2): the coordinator found the hostedhost-linuxfailure (CI run 43814331354), reproduced it in-process on both this head andorigin/main2936b4b, and bisected it to the two #753run_grouparchive-mode tests. The worker identified the leak:patch("run_strict_ci_tracer.run_runner")first-imported the tracer whilestrict_e2e_runner_support.prepare_simulator(and two other dependencies) were patched, so the tracer kept the mocks as import-time aliases. Coordinator self-verified the fix (import the tracer before patching, patchrun_runneron the module object) and re-ran the failing pair locally: OK.