Repository navigation
ci: publish releases to trunk.io/releases/analytics-cli as well as GitHub #1216
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| #!/usr/bin/env bash | ||
| # Publish one release's assets to releases/analytics-cli/prod/<version>/ in the trunk-releases | ||
| # bucket (served at https://trunk.io/releases/analytics-cli/prod/). Never moves channel.json: | ||
| # a published version is a prerelease until promote_release.yml points latest at it. | ||
| # | ||
| # Usage: publish-release-to-s3.sh [--skip-complete] <version> <asset-dir> | ||
| # | ||
| # Every *.tar.gz / *.zip in <asset-dir> is uploaded under its own filename. Objects are | ||
| # immutable: an asset already in the bucket is skipped if its recorded sha256 matches and is an | ||
| # error if it differs. manifest.json goes up last, so its presence means the version is | ||
| # complete. With --skip-complete (backfills), a version whose manifest already exists is a | ||
| # no-op; without it, that is an error. | ||
| set -euo pipefail | ||
|
|
||
| bucket="${BUCKET:-trunk-releases}" | ||
| prefix="${PREFIX:-releases/analytics-cli/prod}" | ||
|
|
||
| skip_complete=false | ||
| if [[ ${1-} == "--skip-complete" ]]; then | ||
| skip_complete=true | ||
| shift | ||
| fi | ||
| version="${1:?usage: publish-release-to-s3.sh [--skip-complete] <version> <asset-dir>}" | ||
| asset_dir="${2:?usage: publish-release-to-s3.sh [--skip-complete] <version> <asset-dir>}" | ||
|
|
||
| if [[ ! ${version} =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]]; then | ||
| echo "::error::'${version}' is not a semver version" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Prints the object's recorded sha256, or nothing when the key does not exist. Any failure other | ||
| # than a 404 aborts: without s3:ListBucket a missing key reads as 403, which must not be taken | ||
| # for "absent". | ||
| recorded_sha256() { | ||
| local key="$1" output | ||
| if output="$(aws s3api head-object --bucket "${bucket}" --key "${key}" \ | ||
| --query 'Metadata.sha256' --output text 2>&1)"; then | ||
| echo "${output}" | ||
| elif [[ ${output} != *"Not Found"* && ${output} != *"404"* ]]; then | ||
| echo "::error::head-object s3://${bucket}/${key} failed: ${output}" >&2 | ||
| return 1 | ||
| fi | ||
| } | ||
|
|
||
| version_prefix="${prefix}/${version}" | ||
| manifest_key="${version_prefix}/manifest.json" | ||
|
|
||
| manifest_sha="$(recorded_sha256 "${manifest_key}")" | ||
| if [[ -n ${manifest_sha} ]]; then | ||
| if [[ ${skip_complete} == true ]]; then | ||
| echo "${version} is already published (s3://${bucket}/${manifest_key}); nothing to do" | ||
| exit 0 | ||
| fi | ||
| echo "::error::${version} is already published (s3://${bucket}/${manifest_key}); released versions are immutable" | ||
| exit 1 | ||
| fi | ||
|
|
||
| shopt -s nullglob | ||
| assets=("${asset_dir}"/*.tar.gz "${asset_dir}"/*.zip) | ||
| if [[ ${#assets[@]} -eq 0 ]]; then | ||
| echo "::error::no *.tar.gz or *.zip assets in ${asset_dir}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| manifest="$(jq -n --arg v "${version}" '{version: $v, artifacts: {}}')" | ||
| for asset in "${assets[@]}"; do | ||
| name="$(basename "${asset}")" | ||
| key="${version_prefix}/${name}" | ||
| sha="$(sha256sum "${asset}" | cut -d' ' -f1)" | ||
| case "${name}" in | ||
| *.zip) content_type="application/zip" ;; | ||
| *) content_type="application/gzip" ;; | ||
| esac | ||
|
|
||
| existing="$(recorded_sha256 "${key}")" | ||
| if [[ -z ${existing} ]]; then | ||
| aws s3 cp "${asset}" "s3://${bucket}/${key}" --only-show-errors \ | ||
| --content-type "${content_type}" \ | ||
| --metadata "sha256=${sha},release-version=${version}" | ||
| echo "uploaded ${key}" | ||
| elif [[ ${existing} == "${sha}" ]]; then | ||
| echo "already uploaded ${key}" | ||
| else | ||
| echo "::error::s3://${bucket}/${key} already exists with sha256 ${existing}, not ${sha}; released artifacts are immutable" | ||
| exit 1 | ||
| fi | ||
|
|
||
| manifest="$(jq --arg n "${name}" --arg s "${sha}" '.artifacts[$n] = {sha256: $s}' <<<"${manifest}")" | ||
| done | ||
|
|
||
| manifest_file="$(mktemp)" | ||
| jq . <<<"${manifest}" >"${manifest_file}" | ||
| aws s3 cp "${manifest_file}" "s3://${bucket}/${manifest_key}" --only-show-errors \ | ||
| --content-type "application/json" \ | ||
| --metadata "sha256=$(sha256sum "${manifest_file}" | cut -d' ' -f1),release-version=${version}" | ||
| rm -f "${manifest_file}" | ||
| echo "published ${version} to https://trunk.io/${version_prefix}/ (${#assets[@]} artifacts)" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| #!/usr/bin/env bash | ||
| # Point releases/analytics-cli/prod/channel.json at an already-published version. This is the | ||
| # only mutable object in the prefix; consumers read `latest` from it. | ||
| # | ||
| # Usage: set-latest-in-s3.sh <version> | ||
| set -euo pipefail | ||
|
|
||
| bucket="${BUCKET:-trunk-releases}" | ||
| prefix="${PREFIX:-releases/analytics-cli/prod}" | ||
| version="${1:?usage: set-latest-in-s3.sh <version>}" | ||
|
|
||
| if [[ ! ${version} =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]]; then | ||
| echo "::error::'${version}' is not a semver version" | ||
| exit 1 | ||
| fi | ||
|
|
||
| manifest_key="${prefix}/${version}/manifest.json" | ||
| if ! output="$(aws s3api head-object --bucket "${bucket}" --key "${manifest_key}" 2>&1)"; then | ||
| if [[ ${output} == *"Not Found"* || ${output} == *"404"* ]]; then | ||
| echo "::error::no s3://${bucket}/${manifest_key}; ${version} was never published (or only partially). Publish it first (mirror_release_to_s3.yml)." | ||
| else | ||
| echo "::error::head-object s3://${bucket}/${manifest_key} failed: ${output}" | ||
| fi | ||
| exit 1 | ||
| fi | ||
|
|
||
| jq -nc --arg v "${version}" '{latest: $v}' | | ||
| aws s3 cp - "s3://${bucket}/${prefix}/channel.json" --only-show-errors \ | ||
| --content-type "application/json" --cache-control "max-age=60" | ||
| echo "https://trunk.io/${prefix}/channel.json now points at ${version}" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| #!/usr/bin/env bash | ||
| # GitHub records asset digests only since June 2025 (0.10.0-beta.0 on); older assets pass unchecked. | ||
| # Usage: verify-github-digests.sh <tag> <asset-dir> | ||
| set -euo pipefail | ||
|
|
||
| tag="${1:?usage: verify-github-digests.sh <tag> <asset-dir>}" | ||
| asset_dir="${2:?usage: verify-github-digests.sh <tag> <asset-dir>}" | ||
|
|
||
| declare -A digests | ||
| while IFS=$'\t' read -r name digest; do | ||
| digests["${name}"]="${digest}" | ||
| done < <(gh api "repos/{owner}/{repo}/releases/tags/${tag}" \ | ||
| --jq '.assets[] | [.name, (.digest // "")] | @tsv') | ||
|
|
||
| verified=0 | ||
| unverified=0 | ||
| shopt -s nullglob | ||
| for asset in "${asset_dir}"/*.tar.gz "${asset_dir}"/*.zip; do | ||
| name="$(basename "${asset}")" | ||
| expected="${digests[${name}]-}" | ||
| if [[ -z ${expected} ]]; then | ||
| unverified=$((unverified + 1)) | ||
| continue | ||
| fi | ||
| actual="sha256:$(sha256sum "${asset}" | cut -d' ' -f1)" | ||
| if [[ ${actual} != "${expected}" ]]; then | ||
| echo "::error::${tag}/${name}: downloaded ${actual}, GitHub recorded ${expected}" | ||
| exit 1 | ||
| fi | ||
| verified=$((verified + 1)) | ||
| done | ||
|
|
||
| echo "${tag}: ${verified} asset(s) match GitHub's digest, ${unverified} have no GitHub digest" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| name: Mirror Releases to S3 | ||
| run-name: Mirror ${{ inputs.tags }} to S3 | ||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| # trunk-ignore(checkov/CKV_GHA_7) | ||
| tags: | ||
| type: string | ||
| description: Space-separated release tags to copy from GitHub to trunk.io/releases/analytics-cli/prod/, or `all` | ||
| required: true | ||
| set-latest-to-github-latest: | ||
| type: boolean | ||
| description: Afterwards, point channel.json at GitHub's current latest release | ||
| default: false | ||
| concurrency: ${{ github.workflow }} | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| mirror: | ||
| name: Mirror ${{ inputs.tags }} | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 120 | ||
| permissions: | ||
| contents: read | ||
| id-token: write | ||
| env: | ||
| TAGS: ${{ inputs.tags }} | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Configure AWS credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| with: | ||
| role-to-assume: ${{ vars.TRUNK_RELEASES_PUBLISH_ROLE_ARN }} | ||
| role-skip-session-tagging: true | ||
| aws-region: us-west-2 | ||
|
|
||
| # --skip-complete makes a re-run cheap: versions already published are skipped, and a | ||
| # version interrupted mid-upload is finished. | ||
| - name: Mirror | ||
| run: | | ||
| if [[ ${TAGS} == all ]]; then | ||
| TAGS="$(gh release list --limit 1000 --exclude-drafts --json tagName --jq '.[].tagName' | tr '\n' ' ')" | ||
| fi | ||
| for tag in ${TAGS}; do | ||
| dir="$(mktemp -d)" | ||
| gh release download "${tag}" --dir "${dir}" --pattern '*.tar.gz' --pattern '*.zip' | ||
| .github/scripts/verify-github-digests.sh "${tag}" "${dir}" | ||
| .github/scripts/publish-release-to-s3.sh --skip-complete "${tag}" "${dir}" | ||
| rm -rf "${dir}" | ||
| done | ||
|
|
||
| - name: Point channel.json at GitHub's latest release | ||
| if: inputs.set-latest-to-github-latest | ||
| run: .github/scripts/set-latest-in-s3.sh "$(gh release view --json tagName --jq .tagName)" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| name: Promote Release | ||
| run-name: Promote ${{ inputs.version }} to latest | ||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| # trunk-ignore(checkov/CKV_GHA_7) | ||
| version: | ||
| type: string | ||
| description: Already-released version to make latest (on GitHub and at trunk.io/releases/analytics-cli/prod/channel.json) | ||
| required: true | ||
| concurrency: ${{ github.workflow }} | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| promote: | ||
| name: Promote ${{ inputs.version }} | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: write | ||
| id-token: write | ||
| env: | ||
| VERSION: ${{ inputs.version }} | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Check the GitHub release | ||
| run: | | ||
| if [[ ${VERSION} == *-* ]]; then | ||
| echo "::error::${VERSION} is a prerelease version; only a stable version can be latest" | ||
| exit 1 | ||
| fi | ||
| gh release view "${VERSION}" --json tagName,isDraft --jq 'if .isDraft then error("release is a draft") else .tagName end' | ||
|
|
||
| - name: Configure AWS credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| with: | ||
| role-to-assume: ${{ vars.TRUNK_RELEASES_PUBLISH_ROLE_ARN }} | ||
| role-skip-session-tagging: true | ||
| aws-region: us-west-2 | ||
|
|
||
| # S3 moves first: its preconditions (the role, a complete manifest) are the ones that fail in | ||
|
Collaborator
There was a problem hiding this comment. precondition: I would like to eventually do a
Member
Author
There was a problem hiding this comment. The launcher can't pin a version yet (
Member
Author
There was a problem hiding this comment. Update: I moved this check to #1217, which is stacked on this PR, and reverted the URL-rewrite version here. #1217 runs |
||
| # practice, and when they do GitHub has not moved either. Both steps are idempotent, so if the | ||
| # GitHub edit fails afterwards, re-running this workflow converges the two. | ||
| - name: Point channel.json at ${{ inputs.version }} | ||
| run: .github/scripts/set-latest-in-s3.sh "${VERSION}" | ||
|
|
||
| - name: Mark ${{ inputs.version }} latest on GitHub | ||
| run: gh release edit "${VERSION}" --prerelease=false --latest | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noting the potential for accidentally overwriting other things here. Acceptable for now, just noting the one potential hole