Repository navigation
build(scripts): share one resolved Swift package checkout across local non-release builds - #679
Open
JasonYeYuhe wants to merge 2 commits into
Open
JasonYeYuhe wants to merge 2 commits into
JasonYeYuhe wants to merge 2 commits into
Conversation
…l non-release builds About 2.4 GB of every ~3 GB derived-data root was sentry-cocoa's SourcePackages (five xcframeworks). Agents that built into a fresh root per step and trashed it afterwards unpacked it again each time: about 119 GB went to the Trash on 2026-10-14 (trash-inflow investigation, P2). - scripts/lib/build_env.sh (new, sourced or run): spm_cache_args resolves once per package set into ~/Library/Caches/CLIPulse-build/SourcePackages-<h8> (key: the project's Package.resolved, its package references and CLIPulseCore's .package lines), under a mkdir lock with stale takeover, with -onlyUsePackageVersionsFromResolvedFile; builds then pass -clonedSourcePackagesDirPath <dir> -disableAutomaticPackageResolution. Skipped on CI=true, for a caller-supplied dir, with CLIPULSE_SPM_DIR=off, or for a dir inside iCloud Documents/Desktop; a failed resolve falls back to per-root resolution. CLIPULSE_SPM_ROOT moves the keyed dirs (worktrees that build one after another can share them). Run as `scripts/lib/build_env.sh xcodebuild ...` it also adds COMPILER_INDEX_STORE_ENABLE=NO. Adapted from a reviewed helper of the same shape in another project. - capture_ios_screenshots.sh builds with the shared packages and COMPILER_INDEX_STORE_ENABLE=NO; its throwaway derived data stays as it was. - render_macos_qa_views.sh header, docs/qa/macos-offscreen-renders.md and AGENTS.md show the QA build through the helper; AGENTS.md gains "Swift packages for local builds". - Untouched on purpose: build-appstore.sh, build_signed_app.sh, build-release.sh, build_devid_dmg.sh, build-tmux-universal.sh, the helper pkg scripts and e2e_helper_local.sh (signed or release builds stay cold); check_release_surfaces.sh builds nothing. Verified (Xcode 27.0): a resolve with -onlyUsePackageVersionsFromResolvedFile (2.4 GB, ~30 s) and two `CLIPulse QA` builds, generic/platform=macOS, from two different worktrees into their own derived data on that one directory: both exit 0 (82 s, 84 s), 1.3 GB derived data each (both archs), no second resolve or download, Package.resolved untouched. Offline harness with a fake xcodebuild: skip cases, fallback, key changes, 5 concurrent callers -> one resolve, stale and live locks, run mode, set -euo pipefail callers (19/19). scripts/test_appstore_screenshots.py 190/190. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e cases, sharing stated plainly - Run mode takes the project from -project (or the current directory, as xcodebuild does) and adds the shared packages only for its own checkout's project; another checkout's gets a warning and resolves into its derived data. Before, one checkout's helper handed another checkout's project its own key's directory, so different pins could land under the wrong marker. - build_lock_take records the pid of the shell that takes the lock, not $$ (in a subshell that is the parent's), so a subshell killed mid-resolve leaves a lock its waiters take over rather than one that looks held for 40 minutes. It returns 2 at once when the lock cannot be created at all; spm_cache_args then falls back to the derived data, as it now also does when CLIPULSE_SPM_ROOT cannot be created. - A relative CLIPULSE_SPM_DIR or CLIPULSE_SPM_ROOT is made absolute against the current directory (the capture script builds from another one), and a directory under $TMPDIR is refused like one in iCloud. - AGENTS.md and the header say plainly that every checkout shares the directory by default, also when building at the same time; that each build rewrites its workspace-state.json and two worktrees building at once is unmeasured; and that concurrent work sets CLIPULSE_SPM_ROOT per checkout. Verified: bash -n (shellcheck is not installed). Offline harness with a fake xcodebuild and swift: 40/40; the pre-fix helper fails exactly the 16 new cases. Four waiters of the workflow wrapper and this helper on a lock left by a dead holder, 20 rounds: no round resolved twice (pre-fix wrapper: 7 of 20). One CLIPulse QA build, generic/platform=macOS, through the workflow wrapper into this helper under the heavy-build lock: resolve + build 95 s, exit 0, 2.4 GB of packages in the shared directory and 1.3 GB of derived data with no SourcePackages of its own; Package.resolved untouched. The second worktree (base commit, no helper) computes the same key and resolves nothing. scripts/test_appstore_screenshots.py 190/190; repo hygiene checks pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
About 2.4 GB of every ~3 GB derived-data root is sentry-cocoa's SourcePackages (five xcframeworks). Agents built into a fresh root per step, trashed it afterwards, and unpacked the packages again every time: about 119 GB went to the Trash on 2026-10-14.
What
scripts/lib/build_env.sh(new, sourced or run).spm_cache_argsresolves once per package set into~/Library/Caches/CLIPulse-build/SourcePackages-<h8>, under a mkdir lock with stale takeover. The key coversPackage.resolved, the package references and CLIPulseCore's.packagelines. Builds then pass-clonedSourcePackagesDirPath <dir> -disableAutomaticPackageResolution. Run asscripts/lib/build_env.sh xcodebuild …, it also addsCOMPILER_INDEX_STORE_ENABLE=NO. Adapted from the reviewed helper in Stride.capture_ios_screenshots.shbuilds with the shared packages. Therender_macos_qa_views.shheader,docs/qa/macos-offscreen-renders.mdandAGENTS.mdshow the QA build going through the helper.AGENTS.mdgains a "Swift packages for local builds" section.CI=true, for a caller-supplied dir, withCLIPULSE_SPM_DIR=off, and for a dir inside iCloud Documents/Desktop. A failed resolve falls back to per-root resolution.Untouched on purpose (signed and release builds stay cold):
build-appstore.sh,build_signed_app.sh,build-release.sh,build_devid_dmg.sh,build-tmux-universal.sh, the helper pkg scripts ande2e_helper_local.sh.Measured
~/clipulse-wt/_bin/build-env.sh, outside the repo): a second worktree building into the same SwiftPM scratch recompiled with no error and added +0 bytes (2.9 GB before and after), where each step used to make a new 2.95 GB root.Tests
heavy.shand the helper: BUILD SUCCEEDED.scripts/test_appstore_screenshots.py: 204/204.check_no_internal_docs.sh,check_copyright.shandcheck_no_tra*all pass.--helpline range incapture_ios_screenshots.sh, now2,74pto match the merged header.🤖 Generated with Claude Code