Skip to content

Adds guidance on separate Cloud Run operator(user) and Worker API keys - #30

Merged
harish-narayanappa-10 merged 4 commits into
mainfrom
cloud-run-credential-separation
Oct 27, 2026
Merged

harish-narayanappa-10 merged 4 commits into
mainfrom
cloud-run-credential-separation

Conversation

@harish-narayanappa-10

@harish-narayanappa-10 harish-narayanappa-10 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to the Cloud Run skill testing

One API key was written to both the engineer's CLI profile and the Worker's
secret, so rotating or revoking it for one broke the other.

  • Separate hand-offs: the Worker's key goes only to the secret; each engineer
    sets up their own CLI profile.
  • Narrower rules for reusing an existing profile or secret.
  • Add Key rotation guidance

Split the single API-key hand-off into a Worker hand-off (secret only)
and an operator hand-off (CLI profile only), so each key can be rotated
or revoked without breaking the other. Narrow profile and secret reuse,
and add rotation runbooks for both keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@harish-narayanappa-10
harish-narayanappa-10 requested a review from a team as a code owner October 12, 2026 06:26

@campoy campoy left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this guidance. A few suggestions to make cleanup and migration safer for existing deployments.

Comment thread references/gcp-cloud-run/setup.md Outdated
```

Never read back `api_key` with `temporal config get`, and never run `gcloud secrets versions access`. The `worker deployment list` call proves the profile authenticates to the Namespace. Expect exactly one `ENABLED` secret version. If the hand-off ran more than once, keep the newest version, confirm the pool uses `latest`, and disable the older ones with `gcloud secrets versions disable <VERSION> --secret <SECRET_NAME> --project <YOUR_GCP_PROJECT>`. Secret versions are immutable: replace a bad one by adding a correct version, never by editing it.
Never read back `api_key` with `temporal config get`, and never run `gcloud secrets versions access`. The `worker deployment list` call proves the profile authenticates to the Namespace. Expect exactly one `ENABLED` secret version. If the Worker hand-off ran more than once, keep the newest version and disable the older ones with `gcloud secrets versions disable <VERSION> --secret <SECRET_NAME> --project <YOUR_GCP_PROJECT>`. Secret versions are immutable: replace a bad one by adding a correct version, never by editing it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we remove the instruction to disable older secret versions immediately after the handoff? A retained pool may still reference one. Keeping them enabled until the rotation checks confirm they’re unused would make this consistent with the new rotation guidance.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, did as suggested - removed the instruction to disable older secret versions after the hand-off. The check is now that the newest version is enabled, so a retained pool that still references an older version should not be affected

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this reads well now. Since the pool mounts :latest, checking that the newest version is enabled is enough.

**If the user already has a CLI profile and a secret**, for example from an earlier deployment, reuse them instead of running the hand-off. Confirm the profile with the read-back commands above, always passing `--namespace`: a profile's stored Namespace can differ from the target. Confirm the secret has an `ENABLED` version with `gcloud secrets versions list`, then ask the user to confirm that it holds the same key as the profile; you must not read either value. Record a reused secret as shared, so teardown keeps it.
#### Reuse what already works

**An existing CLI profile.** If the operator already has a profile for this Namespace, for example from an earlier deployment, skip the operator hand-off, but only after the user confirms that the profile's key is not also stored in a secret. A profile written together with a secret from a single key shares that key with every pool that mounts the secret, so rotating or revoking it breaks them. If the user cannot confirm, run the operator hand-off with a new key; you must not read either value to compare them. Confirm the profile with the read-back commands above, always passing `--namespace`: a profile's stored Namespace can differ from the target.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would help to add a short migration example for deployments created with the previous shared-key setup: configure a new operator key, verify CLI access, and keep the existing Worker key valid until its own migration is complete. That gives existing users a safe path to the new arrangement.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, added a migration path for an existing shared profile/secret key - create and verify a separate operator key and CLI profile

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the new paragraph covers it.

Comment thread references/gcp-cloud-run/setup.md Outdated
Rotate each key on its own schedule, by overlap: the user creates the new key, its consumer switches to it, you confirm the consumer works, and only then is the old key disabled, with the user's approval, and later deleted. Never revoke the old key first. <!-- https://docs.temporal.io/cloud/api-keys -->

- **The operator's key.** Run the [operator hand-off](#hand-off-the-temporal-api-keys) with the new key, then confirm with `temporal --profile <PROFILE> worker deployment list --namespace <NAMESPACE>`.
- **The Worker's key.** Run the [Worker hand-off](#hand-off-the-temporal-api-keys), which adds a new secret version. Cloud Run reads the secret when an instance starts, so running instances keep the old key. Before the old key is disabled, list every pool that mounts the secret, in every region the project uses, and confirm from each pool's logs that every instance started before the new version has shut down. A requested count of `0` is not that evidence. Never disable the newest secret version. <!-- https://docs.cloud.google.com/run/docs/configuring/workerpools/secrets -->

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we spell out how to verify that replacement Workers authenticate successfully and the old instances have stopped, including idle pools? If shutdown evidence is unavailable, keep the old key valid and report rotation as incomplete. Please also distinguish disabling a Secret Manager version from revoking a Temporal API key.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To address this introduced three checks before disabling the old keys - list every pool using the secret, confirm the new key works, confirm old instances have shut down. If any of these can't be confirmed, keep the old key and report the rotation as incomplete

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the three checks and the 'report it as incomplete' fallback are what I had in mind. I left two small follow-ups on the new lines.

Comment thread references/gcp-cloud-run/setup.md Outdated

Keep any binding or runner the inventory does not attribute to this run.
7. Delete the container image from Artifact Registry, and any Secret Manager secret this run created that no other deployment uses; keep a reused or shared secret. Delete the Artifact Registry repository too if this run created it. Ask before revoking a Temporal Cloud API key: it is account-scoped, not deployment-scoped. Once nothing else uses the local CLI profile, remove it with `temporal config delete-profile --profile <PROFILE>`.
7. Delete the container image from Artifact Registry, and any Secret Manager secret this run created that no other deployment uses; keep a reused or shared secret. Delete the Artifact Registry repository too if this run created it. Ask before revoking either Temporal Cloud API key: a key is account-scoped, not deployment-scoped, and the Worker's key may also be mounted by another pool. List those pools as in [Rotate the Temporal API keys](#rotate-the-temporal-api-keys) first. Once nothing else uses the local CLI profile, and only if this run wrote it, remove it with `temporal config delete-profile --profile <PROFILE>`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small ownership detail: “only if this run wrote it” also covers updating an existing profile during rotation. Could we track whether the profile was created or modified, and only delete profiles created for this deployment? That would preserve the operator’s existing configuration during teardown.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated - wow, its distinguishable if a profile created by this run from a pre-existing profile. Teardown deletes only profiles created by the run and retains modified/reused profiles

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, teardown now keeps modified profiles. One follow-up inline on how the run tells the two apart.

@campoy

campoy commented Oct 12, 2026

Copy link
Copy Markdown

Good improvements, I left a couple nits I'd like to consider.

…verless into cloud-run-credential-separation
@harish-narayanappa-10
harish-narayanappa-10 force-pushed the cloud-run-credential-separation branch from 66fb608 to 4fd43e0 Compare October 26, 2026 20:44
Comment thread references/gcp-cloud-run/setup.md Outdated
- **The operator's key.** Run the [operator hand-off](#hand-off-the-temporal-api-keys) with the new key, then confirm with `temporal --profile <PROFILE> worker deployment list --namespace <NAMESPACE>`.
- **The Worker's key.** Run the [Worker hand-off](#hand-off-the-temporal-api-keys), which adds a new secret version. Cloud Run reads the secret when an instance starts, so running instances keep the old key. Before the old key is disabled: <!-- https://docs.cloud.google.com/run/docs/configuring/workerpools/secrets -->
- List every pool that mounts the secret, in every region the project uses, with the command below.
- Confirm the new key works: start a Workflow on this deployment's Task Queue and check each listed pool's logs for authentication errors. If a listed pool connects to a different Namespace (its `TEMPORAL_NAMESPACE`), confirm with that deployment's owner that the new key works there.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Old instances still hold the old key and will pick up this Workflow, so no auth errors doesn't prove the new key works. Could this check run after the next bullet, once they've stopped?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, missed the ordering earlier. new key test now runs after confirming old workers have stopped

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the order works now.

Comment thread references/gcp-cloud-run/setup.md Outdated
- **The Worker's key.** Run the [Worker hand-off](#hand-off-the-temporal-api-keys), which adds a new secret version. Cloud Run reads the secret when an instance starts, so running instances keep the old key. Before the old key is disabled: <!-- https://docs.cloud.google.com/run/docs/configuring/workerpools/secrets -->
- List every pool that mounts the secret, in every region the project uses, with the command below.
- Confirm the new key works: start a Workflow on this deployment's Task Queue and check each listed pool's logs for authentication errors. If a listed pool connects to a different Namespace (its `TEMPORAL_NAMESPACE`), confirm with that deployment's owner that the new key works there.
- Confirm from each pool's logs, with the [pool log query](diagnostics.md#read-the-pool-logs), that every instance started before the new version has shut down. A requested count of `0` is not that evidence. A pool with no instance started before the new version needs no action.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The linked log query only covers the last hour (--freshness=1h), so it misses older instances. Could we say to widen it past the new version's createTime, and treat a missing startup line as unconfirmed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed the query limits, expanded log coverage. Missing/unconfirmed evidence keeps the old key in place

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good.

if [ "${#dots}" -ne 2 ]; then
printf 'Expected a JWT-shaped key with two dots; nothing was changed\n' >&2; exit 1
fi
temporal --profile <PROFILE> config set --prop address --value "<ENDPOINT>" &&

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

config set overwrites an existing profile silently, so the run can't tell created from modified. Could the agent check temporal config list first and pick a new name if <PROFILE> exists?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, makes sense - added a name check, rotation preserves existing profile settings

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this covers it.

```

CLI v1.8.2 spells the field `BuildID`, and `worker deployment list` returns `versionSummaries` as `null`, so read versions from `describe`. Then confirm the version can take traffic. Take its pool, region, image digest, and Task Queue from the recorded deployment inventory or committed release inputs; Temporal does not return the pool a version targets. Run the [Task Queue binding check](diagnostics.md#start-here-did-the-expected-worker-bind), and confirm the pool still exists, its instance split contains exactly one revision, and that revision serves the recorded digest as in [audit items 2 and 3](diagnostics.md#audit-an-existing-pool). If there is no record, or the pool is missing or serves another digest, stop before changing routing and tell the user.
CLI v1.8.2 spells the field `BuildID`, and `worker deployment list` returns `versionSummaries` as `null`, so read versions from `describe`. Then confirm the version can take traffic. Take its pool, region, image digest, and Task Queue from the recorded deployment inventory or committed release inputs; Temporal does not return the pool a version targets. Run the [Task Queue binding check](diagnostics.md#start-here-did-the-expected-worker-bind), and confirm the pool still exists, its instance split contains exactly one revision, and that revision serves the recorded digest as in [audit items 2 and 3](diagnostics.md#audit-an-existing-pool). If the record is missing or any check fails or cannot be confirmed, stop before changing routing and tell the user.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is follow up to comment in #31 - (#31 (comment))

The rollback now stops if any check fails or cannot be confirmed

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up.

@harish-narayanappa-10 harish-narayanappa-10 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed a comment made earlier on another PR - #31 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants