Release action: opt in to publishing on register (DUPLOAI-3746)

.github/workflows/release.yml

@@ -33,6 +33,11 @@ on:
   push:
     branches: [main]
   workflow_dispatch:
+    inputs:
+      publish:
+        description: Publish each allowlisted extension's current version, including one an earlier run registered or a platform owner unpublished. Allowlist entries with "publish" true publish new builds regardless.
+        type: boolean
+        default: false
 
 permissions:
   contents: write        # create GitHub Releases
@@ -109,6 +114,7 @@ jobs:
           EXTENSION_SIGNING_KEY: ${{ secrets.CONSOLE_SIGNING_KEY }}
           EXTENSION_SIGNING_CERT: ${{ vars.CONSOLE_SIGNING_CERT }}
           CONSOLE_API_KEY: ${{ secrets.CONSOLE_API_KEY }}
+          EXTENSION_PUBLISH: ${{ inputs.publish }}
           CONSOLE_URL: ${{ vars.CONSOLE_URL }}
           BUNDLE_BUCKET: ${{ vars.BUNDLE_BUCKET }}
         run: ./scripts/release-extensions.sh

docs/releasing-extensions.md

@@ -27,8 +27,8 @@ frontend bundle ships without the file that Native Federation hosts need.
 
 In `duplocloud` and `duplocloud-internal`, the release also carries `extension.zip.sig`, and, for an id the
 allowlist maps to this repository, the build is uploaded to the channels bucket and registered as a new,
-unpublished version with the license server. Nothing in this workflow ever publishes a version. A platform
-owner does that separately, once the build has been verified.
+unpublished version with the license server. A version stays unpublished, for a platform owner to publish
+once the build has been verified, unless the extension opts in to publishing (see "Publishing on register").
 
 Outside those two organizations, a release ships the zip alone. A customer's copy of this workflow holds no
 DuploCloud signing credential, so it signs nothing, uploads nothing, and registers nothing.
@@ -90,9 +90,10 @@ A platform owner sets up each of these once in the console. Each one then serves
 
 1. **The API key behind `CONSOLE_API_KEY`.** In the platform-owner team, create a custom role holding
    **View extension versions** and **Manage extension versions**, create a service account with that role,
-   and store its API key as the organization secret. The release job reads and creates only versions and
-   artifacts, and the allowlist supplies each extension's uuid, so the key needs no other capability. The
-   console's own guide covers the service-account workflow in `docs/platform_owner/extension-versions.md`.
+   and store its API key as the organization secret. The release job reads and creates versions and artifacts,
+   and publishes a version when opted in. Manage extension versions covers publishing, and the allowlist
+   supplies each extension's uuid, so the key needs no other capability. The console's own guide covers the
+   service-account workflow in `docs/platform_owner/extension-versions.md`.
 2. **The signing key behind `CONSOLE_SIGNING_KEY` and `CONSOLE_SIGNING_CERT`.** The publishing team needs a
    namespace covering its manifest ids, and a signing key created under that team. The console returns the
    key's private PEM once, at creation, and that goes into the secret. The key's certificate goes into the
@@ -106,7 +107,7 @@ A platform owner sets up each of these once in the console. Each one then serves
 ### Joining the allowlist
 
 `.github/extension-publishers.json` ships empty in devkit. Each entry maps one manifest id to one
-repository and one console extension:
+repository and one console extension, with an optional `"publish": true` (see "Publishing on register"):
 
 ```json
 {
@@ -128,8 +129,10 @@ pinned devkit commit, so joining the allowlist takes a PR in both repositories:
 repository added to the role's trust list there. Either alone leaves the id unable to publish, refused by
 AWS on one side or skipped by the allowlist check on the other.
 
-The design calls for two approvals from the `devkit-maintainers` team on changes to this file, `release.yml`,
-`release-extensions.sh` and the signer, though the ruleset enforcing that is not configured yet.
+Changes to this file, `release.yml`, `release-extensions.sh`, `_publish.sh`, `_publishers.sh`, the signer and
+its `extension-sign.requirements.txt` are meant to take two approvals from the `devkit-maintainers` team,
+through a path-scoped rule in devkit's ruleset. Together those files decide what gets signed, uploaded and
+registered, and the requirements file pins the libraries the signer loads the key into.
 
 ## The channels bucket and the license server
 
@@ -141,7 +144,7 @@ default to the production bucket and the production console.
 A build that is already finished stops before any download of the zip or any S3 call. The script reads the
 Release's digest for `extension.zip`, downloads only `extension.zip.sig`, and looks up the version and its
 artifact for this SDK in the license server. An artifact whose path, hash and signature match is reported
-as already published. Registration only ever follows both uploads, so a matching artifact means both objects
+as already uploaded and registered. Registration only ever follows both uploads, so a matching artifact means both objects
 are already in the bucket. A failed lookup fails the job. Anything else takes the full path below.
 
 Otherwise, before uploading, the script downloads both assets from the GitHub Release itself, never
@@ -158,8 +161,28 @@ Registering the version with the license server follows the same shape. The scri
 its number before creating it, and an error saying the record already exists just means a concurrent run
 created it first, so it looks up again rather than failing. It registers the artifact for this SDK the same
 way, by looking it up before creating it. An existing artifact whose path, hash and signature all match
-counts as already done. One that differs fails the job. The script never marks a version published. Every
-version it creates starts unpublished, and stays that way until a platform owner publishes it.
+counts as already done. One that differs fails the job. Every version it creates starts unpublished.
+
+### Publishing on register
+
+Publishing a version is what ships its build to every entitled install, so it is opt-in, in either of two ways.
+
+- **Per extension.** Set `"publish": true` on the extension's allowlist entry. A push to `main` then publishes a
+  version when that run registers its first build. A push that resumes a release an earlier run registered, or
+  adds another SDK's build to a version that already has one, never publishes, so a version a platform owner
+  unpublished stays unpublished.
+- **Per run.** Start the Extension Release workflow by hand with its `publish` input checked. That run
+  publishes every version it registers, and also a version an earlier run registered, since a person chose that
+  run. It is also how to publish a version whose publish failed after registration.
+
+Either way, the version is published only after its build is uploaded and its artifact is registered and
+matches the release. A version that is already published is left alone. A repository builds against one SDK at
+a time, so a version with builds for several SDKs is published once its first build is registered.
+
+The release job reads the extension repository's own copy of the allowlist, so `"publish": true` is gated by
+that repository's branch protection, not by devkit's review. Whoever can merge to the repository's `main` can turn
+it on, much as they could change the release scripts the job runs. The console key that registers versions can
+also publish them, since one capability covers both.
 
 ## Certificate reissue and key rotation
 

scripts/_publish.sh

@@ -1,7 +1,9 @@
 # shellcheck shell=bash
 # Sourced by scripts/release-extensions.sh. Signing is mandatory in Duplo's two organizations and skipped anywhere
 # else, so a customer copy of the release workflow behaves as before with no Duplo credential. Upload and registration
-# run only in Duplo's organizations, for a manifest id the allowlist maps to this repository.
+# run only in Duplo's organizations, for a manifest id the allowlist maps to this repository. A registered version
+# stays unpublished unless the caller sets publish_version (scripts/release-extensions.sh, from the allowlist entry's
+# "publish" or a manual run's publish input).
 
 # duplo_org: exit 0 when this run belongs to one of Duplo's organizations.
 duplo_org() { case "${GITHUB_REPOSITORY_OWNER:-}" in duplocloud|duplocloud-internal) return 0 ;; *) return 1 ;; esac; }
@@ -47,13 +49,35 @@ artifact_matches() {
     <<<"$1" >/dev/null
 }
 
+# maybe_publish <tag> <version uuid> <new|existing>: mark the version published when publish_version allows it and it
+# is not already. publish_version=new publishes only the version's first build, newly registered by this call, so a
+# push that resumes an existing tag or adds another SDK's build never publishes and a deliberate unpublish sticks.
+# publish_version=any, a manual run's choice, also publishes a build an earlier run registered. Runs only after the
+# build's artifact is registered and matches the release, so a version is never published ahead of the build it ships.
+maybe_publish() {
+  case "${publish_version:-}" in
+    any) ;;
+    new) [ "$3" = new ] || return 0 ;;
+    *) return 0 ;;
+  esac
+  local tag=$1 vuuid=$2 out
+  # A push re-run finds the build registered and does not publish it, so a manual run is what retries a failed publish.
+  local retry="Start Extension Release by hand with its publish input checked to retry"
+  out="$(console GET "/api/extensions/$uuid/versions/$vuuid/")" \
+    || { echo "::error::$tag — console lookup of the version to publish failed: $out. $retry"; return 1; }
+  if jq -e '.is_published == true' <<<"$out" >/dev/null; then echo "==> $tag's version is already published"; return 0; fi
+  out="$(console PATCH "/api/extensions/$uuid/versions/$vuuid/" '{"is_published": true}')" \
+    || { echo "::error::$tag — console refused to publish the version: $out. $retry"; return 1; }
+  echo "==> published $tag in the license server"
+}
+
 # publish_build <tag> <id> <version> <sdk> <dir>: upload the Release's zip and .sig, then register both. Reads the
 # assets from the Release so a re-run reuses the bytes already signed and released, with no rebuild.
 #
 # A finished build stops early. Registration runs only after both uploads, so a registered artifact matching the
 # Release's digest and .sig means both objects are already in the bucket, and the run needs neither the zip nor S3.
 publish_build() {
-  local tag=$1 id=$2 version=$3 sdk=$4 v dl localsha sha key vuuid out art
+  local tag=$1 id=$2 version=$3 sdk=$4 v dl localsha sha key vuuid out art first
   for v in "$id" "$version" "$sdk"; do
     [[ "$v" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] || { echo "::error::$tag — $v cannot form a bucket key."; return 1; }
   done
@@ -74,7 +98,7 @@ publish_build() {
       out="$(console GET "/api/extensions/$uuid/versions/$vuuid/artifacts/?sdk_version=$sdk")" \
         || { echo "::error::$tag — console lookup of the artifact failed: $out"; rm -rf "$dl"; return 1; }
       if artifact_matches "$out" "$(artifact_body "$key" "$sdk" "$sha" "$dl/extension.zip.sig")"; then
-        echo "==> $tag already published"; rm -rf "$dl"; return 0
+        echo "==> $tag already uploaded and registered"; rm -rf "$dl"; maybe_publish "$tag" "$vuuid" existing; return
       fi
     fi
   fi
@@ -108,11 +132,18 @@ publish_build() {
     || { echo "::error::$tag — console lookup of the artifact failed: $out"; rm -rf "$dl"; return 1; }
   if [ "$(jq 'length' <<<"$out")" -gt 0 ]; then
     if artifact_matches "$out" "$art"; then
-      echo "==> $tag already registered"; rm -rf "$dl"; return 0
+      echo "==> $tag already registered"; rm -rf "$dl"; maybe_publish "$tag" "$vuuid" existing; return
     fi
     echo "::error::$tag — the console already holds a build for SDK $sdk that differs from this release."; rm -rf "$dl"; return 1
   fi
+  # A version's first build is new to it. A later SDK's build of a version that already has one is not, so a version a
+  # platform owner unpublished stays unpublished when an unchanged extension is rebuilt against a new SDK.
+  out="$(console GET "/api/extensions/$uuid/versions/$vuuid/artifacts/")" \
+    || { echo "::error::$tag — console lookup of the version's builds failed: $out"; rm -rf "$dl"; return 1; }
+  first=existing
+  [ "$(jq 'length' <<<"$out")" -gt 0 ] || first=new
   out="$(console POST "/api/extensions/$uuid/versions/$vuuid/artifacts/" "$art")" \
     || { echo "::error::$tag — console refused the artifact: $out"; rm -rf "$dl"; return 1; }
-  echo "==> registered $tag in the license server (unpublished)"; rm -rf "$dl"
+  echo "==> registered $tag in the license server"; rm -rf "$dl"
+  maybe_publish "$tag" "$vuuid" "$first"
 }

scripts/_publishers.sh

@@ -20,3 +20,12 @@ publisher_extension_uuid() {
   [ -n "$uuid" ] || return 1
   printf '%s\n' "$uuid"
 }
+
+# publisher_auto_publish <manifest-id> <org/repo>: exit 0 when that repository's entry for the id sets "publish": true,
+# so a build the release job newly registers for it is also published. The job reads the extension repository's own
+# copy of this file, so the flag is gated by that repository's branch protection, not by devkit's review.
+publisher_auto_publish() {
+  [ -f "$publishers_file" ] || return 1
+  jq -e --arg id "$1" --arg repo "$2" \
+    'any(.publishers[]; .manifestId == $id and .repository == $repo and .publish == true)' "$publishers_file" >/dev/null 2>&1
+}

scripts/release-extensions.sh

@@ -21,7 +21,9 @@
 # signing credential. Within a Duplo organization, the manifest id also has to be in the extension publisher
 # allowlist (scripts/_publishers.sh) for the repository, or the release still ships signed but scripts/_publish.sh's
 # publish_build is never called for it. An allowlisted id with no CONSOLE_API_KEY fails before it is signed or
-# released, since that build could be released but never registered.
+# released, since that build could be released but never registered. A registered version stays unpublished unless
+# its allowlist entry sets "publish": true, which publishes only a build this run newly registers, or
+# EXTENSION_PUBLISH=true (a manual run's publish input) asks to publish it.
 #
 # Resuming: a tag that already has a release is never rebuilt, re-signed or re-released. Its own assets, read back
 # with gh release view, decide what happens next: a signed release calls publish_build again so a run that stopped
@@ -95,6 +97,14 @@ for m in extensions/*/manifest.json extension/*/manifest.json extension/manifest
       echo "::notice::$dir — $id is not in the extension publisher allowlist for $repo, so it is released but not published to the license server."
     fi
   fi
+  # Publishing is opt-in. An allowlist entry's "publish" publishes only a build this run newly registers, and a manual
+  # run's publish input (EXTENSION_PUBLISH) also publishes one an earlier run registered. publish_build reads
+  # publish_version, and _publish.sh's maybe_publish has the rule.
+  publish_version=""
+  if [ -n "$uuid" ]; then
+    if [ "${EXTENSION_PUBLISH:-}" = true ]; then publish_version=any
+    elif publisher_auto_publish "$id" "$repo"; then publish_version=new; fi
+  fi
   # Registration needs the console key, so a missing one fails here, before a release that could never be registered.
   if [ -n "$uuid" ] && [ -z "${CONSOLE_API_KEY:-}" ]; then
     echo "::error::$dir — $id is allowlisted for $repo, but CONSOLE_API_KEY is not set, so it cannot be registered."

tests/_publish_stubs.sh

@@ -66,8 +66,20 @@ case "$method $url" in
     [ -n "${RACE:-}" ] && [ ! -f "$CONSOLE/raced" ] && { touch "$CONSOLE/raced"; echo '[{"uuid":"v-1"}]' > "$CONSOLE/versions.json"; echo '{"version":["1.0.0 already exists for this extension."]}'; exit 22; }
     [ -n "${VERSION_VANISHES:-}" ] && { echo '{"uuid":"v-1"}'; exit 0; }
     echo '[{"uuid":"v-1"}]' > "$CONSOLE/versions.json"; echo '{"uuid":"v-1"}' ;;
-  "GET "*"/artifacts/?sdk_version="*) cat "$CONSOLE/artifacts.json" 2>/dev/null || echo '[]' ;;
-  "POST "*"/artifacts/") echo "[$data]" > "$CONSOLE/artifacts.json"; echo "$data" ;;
+  "GET "*"/versions/v-1/")
+    if [ -f "$CONSOLE/published" ]; then echo '{"uuid":"v-1","is_published":true}'; else echo '{"uuid":"v-1","is_published":false}'; fi ;;
+  "PATCH "*"/versions/v-1/")
+    # PATCH_FAIL refuses the publish alone, so the version GET before it still answers.
+    [ -n "${PATCH_FAIL:-}" ] && { echo '{"detail":"refused"}'; exit 22; }
+    echo "$data" > "$CONSOLE/published"; echo '{"uuid":"v-1","is_published":true}' ;;
+  # Artifacts for every SDK sit in one list, filtered by sdk_version when the URL names one, so a version can hold
+  # builds for several SDKs.
+  "GET "*"/artifacts/?sdk_version="*)
+    jq -c --arg s "${url##*sdk_version=}" '[.[] | select(.sdk_version == $s)]' "$CONSOLE/artifacts.json" 2>/dev/null || echo '[]' ;;
+  "GET "*"/artifacts/") cat "$CONSOLE/artifacts.json" 2>/dev/null || echo '[]' ;;
+  "POST "*"/artifacts/")
+    jq -c --argjson a "$data" '. + [$a]' "$CONSOLE/artifacts.json" 2>/dev/null > "$CONSOLE/a.tmp" || echo "[$data]" > "$CONSOLE/a.tmp"
+    mv "$CONSOLE/a.tmp" "$CONSOLE/artifacts.json"; echo "$data" ;;
 esac
 EOF
 

tests/test-extension-publishers.sh

@@ -23,6 +23,16 @@ for F in "$REAL" "$FIXTURE"; do
     "$F" >/dev/null && ok || bad "shape"
 done
 
+t "publish, where an entry sets it, is a boolean"
+jq -e 'all(.publishers[]; (has("publish") | not) or (.publish | type == "boolean"))' "$REAL" >/dev/null && ok || bad "publish is not a boolean"
+
+t "publisher_auto_publish is true only for an entry that sets publish true"
+jq '.publishers[0].publish = true' "$FIXTURE" > "$TMP/auto.json"
+id="$(jq -r '.publishers[0].manifestId' "$FIXTURE")"; repo="$(jq -r '.publishers[0].repository' "$FIXTURE")"
+id2="$(jq -r '.publishers[1].manifestId' "$FIXTURE")"; repo2="$(jq -r '.publishers[1].repository' "$FIXTURE")"
+( publishers_file="$TMP/auto.json"; publisher_auto_publish "$id" "$repo" && ! publisher_auto_publish "$id2" "$repo2" \
+  && ! publisher_auto_publish "$id" "other/repo" ) && ok || bad "auto-publish lookup"
+
 t "the real file is empty or every entry passes the shape check"
 jq -e '(.publishers | length == 0) or
   all(.publishers[]; (.repository | test("^[^/]+/[^/]+$")) and (.consoleExtension | test("^[0-9a-f-]{36}$")))' \