Repository navigation
Adds guidance on separate Cloud Run operator(user) and Worker API keys - #30
Conversation
Split the single API-key hand-off into a Worker hand-off (secret only) and an operator hand-off (CLI profile only), so each key can be rotated or revoked without breaking the other. Narrow profile and secret reuse, and add rotation runbooks for both keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
campoy
left a comment
There was a problem hiding this comment.
Thanks for adding this guidance. A few suggestions to make cleanup and migration safer for existing deployments.
| ``` | ||
|
|
||
| Never read back `api_key` with `temporal config get`, and never run `gcloud secrets versions access`. The `worker deployment list` call proves the profile authenticates to the Namespace. Expect exactly one `ENABLED` secret version. If the hand-off ran more than once, keep the newest version, confirm the pool uses `latest`, and disable the older ones with `gcloud secrets versions disable <VERSION> --secret <SECRET_NAME> --project <YOUR_GCP_PROJECT>`. Secret versions are immutable: replace a bad one by adding a correct version, never by editing it. | ||
| Never read back `api_key` with `temporal config get`, and never run `gcloud secrets versions access`. The `worker deployment list` call proves the profile authenticates to the Namespace. Expect exactly one `ENABLED` secret version. If the Worker hand-off ran more than once, keep the newest version and disable the older ones with `gcloud secrets versions disable <VERSION> --secret <SECRET_NAME> --project <YOUR_GCP_PROJECT>`. Secret versions are immutable: replace a bad one by adding a correct version, never by editing it. |
There was a problem hiding this comment.
Could we remove the instruction to disable older secret versions immediately after the handoff? A retained pool may still reference one. Keeping them enabled until the rotation checks confirm they’re unused would make this consistent with the new rotation guidance.
There was a problem hiding this comment.
Ack, did as suggested - removed the instruction to disable older secret versions after the hand-off. The check is now that the newest version is enabled, so a retained pool that still references an older version should not be affected
There was a problem hiding this comment.
Thanks, this reads well now. Since the pool mounts :latest, checking that the newest version is enabled is enough.
| **If the user already has a CLI profile and a secret**, for example from an earlier deployment, reuse them instead of running the hand-off. Confirm the profile with the read-back commands above, always passing `--namespace`: a profile's stored Namespace can differ from the target. Confirm the secret has an `ENABLED` version with `gcloud secrets versions list`, then ask the user to confirm that it holds the same key as the profile; you must not read either value. Record a reused secret as shared, so teardown keeps it. | ||
| #### Reuse what already works | ||
|
|
||
| **An existing CLI profile.** If the operator already has a profile for this Namespace, for example from an earlier deployment, skip the operator hand-off, but only after the user confirms that the profile's key is not also stored in a secret. A profile written together with a secret from a single key shares that key with every pool that mounts the secret, so rotating or revoking it breaks them. If the user cannot confirm, run the operator hand-off with a new key; you must not read either value to compare them. Confirm the profile with the read-back commands above, always passing `--namespace`: a profile's stored Namespace can differ from the target. |
There was a problem hiding this comment.
It would help to add a short migration example for deployments created with the previous shared-key setup: configure a new operator key, verify CLI access, and keep the existing Worker key valid until its own migration is complete. That gives existing users a safe path to the new arrangement.
There was a problem hiding this comment.
Done, added a migration path for an existing shared profile/secret key - create and verify a separate operator key and CLI profile
| Rotate each key on its own schedule, by overlap: the user creates the new key, its consumer switches to it, you confirm the consumer works, and only then is the old key disabled, with the user's approval, and later deleted. Never revoke the old key first. <!-- https://docs.temporal.io/cloud/api-keys --> | ||
|
|
||
| - **The operator's key.** Run the [operator hand-off](#hand-off-the-temporal-api-keys) with the new key, then confirm with `temporal --profile <PROFILE> worker deployment list --namespace <NAMESPACE>`. | ||
| - **The Worker's key.** Run the [Worker hand-off](#hand-off-the-temporal-api-keys), which adds a new secret version. Cloud Run reads the secret when an instance starts, so running instances keep the old key. Before the old key is disabled, list every pool that mounts the secret, in every region the project uses, and confirm from each pool's logs that every instance started before the new version has shut down. A requested count of `0` is not that evidence. Never disable the newest secret version. <!-- https://docs.cloud.google.com/run/docs/configuring/workerpools/secrets --> |
There was a problem hiding this comment.
Could we spell out how to verify that replacement Workers authenticate successfully and the old instances have stopped, including idle pools? If shutdown evidence is unavailable, keep the old key valid and report rotation as incomplete. Please also distinguish disabling a Secret Manager version from revoking a Temporal API key.
There was a problem hiding this comment.
To address this introduced three checks before disabling the old keys - list every pool using the secret, confirm the new key works, confirm old instances have shut down. If any of these can't be confirmed, keep the old key and report the rotation as incomplete
There was a problem hiding this comment.
Thanks, the three checks and the 'report it as incomplete' fallback are what I had in mind. I left two small follow-ups on the new lines.
|
|
||
| Keep any binding or runner the inventory does not attribute to this run. | ||
| 7. Delete the container image from Artifact Registry, and any Secret Manager secret this run created that no other deployment uses; keep a reused or shared secret. Delete the Artifact Registry repository too if this run created it. Ask before revoking a Temporal Cloud API key: it is account-scoped, not deployment-scoped. Once nothing else uses the local CLI profile, remove it with `temporal config delete-profile --profile <PROFILE>`. | ||
| 7. Delete the container image from Artifact Registry, and any Secret Manager secret this run created that no other deployment uses; keep a reused or shared secret. Delete the Artifact Registry repository too if this run created it. Ask before revoking either Temporal Cloud API key: a key is account-scoped, not deployment-scoped, and the Worker's key may also be mounted by another pool. List those pools as in [Rotate the Temporal API keys](#rotate-the-temporal-api-keys) first. Once nothing else uses the local CLI profile, and only if this run wrote it, remove it with `temporal config delete-profile --profile <PROFILE>`. |
There was a problem hiding this comment.
Small ownership detail: “only if this run wrote it” also covers updating an existing profile during rotation. Could we track whether the profile was created or modified, and only delete profiles created for this deployment? That would preserve the operator’s existing configuration during teardown.
There was a problem hiding this comment.
Updated - wow, its distinguishable if a profile created by this run from a pre-existing profile. Teardown deletes only profiles created by the run and retains modified/reused profiles
There was a problem hiding this comment.
Thanks, teardown now keeps modified profiles. One follow-up inline on how the run tells the two apart.
|
Good improvements, I left a couple nits I'd like to consider. |
…verless into cloud-run-credential-separation
66fb608 to
4fd43e0
Compare
| - **The operator's key.** Run the [operator hand-off](#hand-off-the-temporal-api-keys) with the new key, then confirm with `temporal --profile <PROFILE> worker deployment list --namespace <NAMESPACE>`. | ||
| - **The Worker's key.** Run the [Worker hand-off](#hand-off-the-temporal-api-keys), which adds a new secret version. Cloud Run reads the secret when an instance starts, so running instances keep the old key. Before the old key is disabled: <!-- https://docs.cloud.google.com/run/docs/configuring/workerpools/secrets --> | ||
| - List every pool that mounts the secret, in every region the project uses, with the command below. | ||
| - Confirm the new key works: start a Workflow on this deployment's Task Queue and check each listed pool's logs for authentication errors. If a listed pool connects to a different Namespace (its `TEMPORAL_NAMESPACE`), confirm with that deployment's owner that the new key works there. |
There was a problem hiding this comment.
Old instances still hold the old key and will pick up this Workflow, so no auth errors doesn't prove the new key works. Could this check run after the next bullet, once they've stopped?
There was a problem hiding this comment.
Ack, missed the ordering earlier. new key test now runs after confirming old workers have stopped
| - **The Worker's key.** Run the [Worker hand-off](#hand-off-the-temporal-api-keys), which adds a new secret version. Cloud Run reads the secret when an instance starts, so running instances keep the old key. Before the old key is disabled: <!-- https://docs.cloud.google.com/run/docs/configuring/workerpools/secrets --> | ||
| - List every pool that mounts the secret, in every region the project uses, with the command below. | ||
| - Confirm the new key works: start a Workflow on this deployment's Task Queue and check each listed pool's logs for authentication errors. If a listed pool connects to a different Namespace (its `TEMPORAL_NAMESPACE`), confirm with that deployment's owner that the new key works there. | ||
| - Confirm from each pool's logs, with the [pool log query](diagnostics.md#read-the-pool-logs), that every instance started before the new version has shut down. A requested count of `0` is not that evidence. A pool with no instance started before the new version needs no action. |
There was a problem hiding this comment.
The linked log query only covers the last hour (--freshness=1h), so it misses older instances. Could we say to widen it past the new version's createTime, and treat a missing startup line as unconfirmed?
There was a problem hiding this comment.
Missed the query limits, expanded log coverage. Missing/unconfirmed evidence keeps the old key in place
| if [ "${#dots}" -ne 2 ]; then | ||
| printf 'Expected a JWT-shaped key with two dots; nothing was changed\n' >&2; exit 1 | ||
| fi | ||
| temporal --profile <PROFILE> config set --prop address --value "<ENDPOINT>" && |
There was a problem hiding this comment.
config set overwrites an existing profile silently, so the run can't tell created from modified. Could the agent check temporal config list first and pick a new name if <PROFILE> exists?
There was a problem hiding this comment.
Ack, makes sense - added a name check, rotation preserves existing profile settings
…verless into cloud-run-credential-separation
| ``` | ||
|
|
||
| CLI v1.8.2 spells the field `BuildID`, and `worker deployment list` returns `versionSummaries` as `null`, so read versions from `describe`. Then confirm the version can take traffic. Take its pool, region, image digest, and Task Queue from the recorded deployment inventory or committed release inputs; Temporal does not return the pool a version targets. Run the [Task Queue binding check](diagnostics.md#start-here-did-the-expected-worker-bind), and confirm the pool still exists, its instance split contains exactly one revision, and that revision serves the recorded digest as in [audit items 2 and 3](diagnostics.md#audit-an-existing-pool). If there is no record, or the pool is missing or serves another digest, stop before changing routing and tell the user. | ||
| CLI v1.8.2 spells the field `BuildID`, and `worker deployment list` returns `versionSummaries` as `null`, so read versions from `describe`. Then confirm the version can take traffic. Take its pool, region, image digest, and Task Queue from the recorded deployment inventory or committed release inputs; Temporal does not return the pool a version targets. Run the [Task Queue binding check](diagnostics.md#start-here-did-the-expected-worker-bind), and confirm the pool still exists, its instance split contains exactly one revision, and that revision serves the recorded digest as in [audit items 2 and 3](diagnostics.md#audit-an-existing-pool). If the record is missing or any check fails or cannot be confirmed, stop before changing routing and tell the user. |
There was a problem hiding this comment.
This change is follow up to comment in #31 - (#31 (comment))
The rollback now stops if any check fails or cannot be confirmed
harish-narayanappa-10
left a comment
There was a problem hiding this comment.
Addressed a comment made earlier on another PR - #31 (comment)
Follow-up to the Cloud Run skill testing
One API key was written to both the engineer's CLI profile and the Worker's
secret, so rotating or revoking it for one broke the other.
sets up their own CLI profile.