Skip to content

Provide clarity on Cloud Run Terraform scope, add a pool audit and rollback checks - #31

Merged
harish-narayanappa-10 merged 3 commits into
mainfrom
cloud-run-infrastructure-as-code
Oct 27, 2026
Merged

harish-narayanappa-10 merged 3 commits into
mainfrom
cloud-run-infrastructure-as-code

Conversation

@harish-narayanappa-10

@harish-narayanappa-10 harish-narayanappa-10 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to the Cloud Run skill testing

The skill assumed one person deploying once. When asked to put pools in shared
Terraform, an agent kept state on one machine, and a new release could replace
or change an older build's pool.

  • The Terraform module covers IAM only; pools are created with gcloud unless
    the user asks for Terraform.
  • Short rules for Terraform-managed pools: one pinned pool per build, never
    updated; instance count left to Temporal; locked remote state.
  • Pool audit: a checklist for reviewing an existing Worker Pool, marking
    anything it can't confirm as unverified.
  • Rollback checks: confirm the previous version still exists before rolling
    back, and treat a missing or null Task Queue binding the same way.

@harish-narayanappa-10
harish-narayanappa-10 requested a review from a team as a code owner October 12, 2026 06:29

@campoy campoy left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing these findings. A few suggestions to strengthen shared ownership, recovery, and validation.

- **Leave the instance count to Temporal.** Temporal writes only `scaling.manual_instance_count`. Create the pool with manual scaling at `0` and set `ignore_changes = [scaling[0].manual_instance_count]`.
- **Keep pools in their own root.** The invoker module requires the Google provider `~> 4.0` (`iam.md`); use `>= 7.18.0` for `google_cloud_run_v2_worker_pool`. One root resolves one provider version.
- **Keep registration out of Terraform.** Register versions and change routing with the `temporal` CLI as in [Rolling out a new build](#rolling-out-a-new-build).
- **Treat the pools as shared infrastructure.** Use remote state with locking, and apply only from reviewed, committed inputs that list every build still in use.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we apply the remote-state and locking requirement to the IAM root as well? The invoker and its grants are shared infrastructure too, so losing that state with a worktree would leave part of the original ownership problem unresolved.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done - shared invokers need remote state with locking.
A one-time local setup can keep local state.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that covers it.

## Rollback

Set the previous version current again. Its existing pool scales up on the next Task without rebuilding or reverting an image. If you reused the same pool, setting the previous Temporal version current is not enough because both versions address mutable pool state. Restore the known-good Cloud Run revision split as described above; if that revision was deleted, redeploy the old image deliberately.
Before rolling back, confirm the previous version still exists in the deployment:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking that the previous version exists is a useful start. Could we also confirm its pool still exists, serves the expected image digest, and has the expected Task Queue bindings before changing routing? That would catch a retained version whose infrastructure has drifted or disappeared.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added checks for the old pool, digest, and Task Queue binding. Stop before routing if the recorded information is missing or does not match.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is what I had in mind. One small follow-up inline.

2. **Is the image pinned by digest?** The container image should end in `@sha256:`. The pool's output does not include the deployed digest; read it from the revision named in `status.latestReadyRevisionName` with `gcloud run worker-pools revisions describe <REVISION> --region <REGION> --project <YOUR_GCP_PROJECT> --format='value(status.imageDigest)'`, and check it matches the recorded artifact, as in [`setup.md`](setup.md#step-4-create-the-worker-pool) Step 4. A tag can move under a live version.
3. **Does the pool currently allocate instances to exactly one revision?** Check that `status.instanceSplits` lists one revision. The pool name should carry the build ID, and `TEMPORAL_BUILD_ID` and `TEMPORAL_DEPLOYMENT_NAME` should match the version. Whether the pool was ever reused across builds, and which build each revision ran, cannot be read from Cloud Run; report it as unverified unless trusted evidence such as the team's Terraform or release history shows it. → [one pool per build](versioning.md#one-worker-pool-per-build-id).
4. **Are the identities separate?** The pool runs as the runner, never the invoker, and the API key comes from a Secret Manager reference, not a plain environment-variable value. → `iam.md`.
5. **Does Temporal own the count?** Pass or fail on `scalingMode` being `manual`. Report `lastModifier` as information, never as a failure: any later update by a person replaces it on a healthy pool. → [Read the pool's annotations](#read-the-pools-annotations).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new audit guidance on lastModifier is helpful. Could we put the same clarification beside the registration checkpoint in setup.md, and label startup-log checks as first-registration checks? That would help prevent agents from turning them into permanent recovery requirements.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified that startup logs and lastModifier are only for first registration, existing versions should use the binding check

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the note beside the checkpoint reads well.


Create pools with `gcloud` as in `setup.md` Step 4. Declare them in Terraform only when the user asks, and then:

- **Give each build its own pool and never update it.** Pin each pool's image by digest. Any change to a pool's template, including a value shared by every pool such as an environment variable, creates a new revision and replaces that pool's running instances: the [redeploy hazard](#the-hazard-redeploying-into-a-live-pool). A release plan should only create the new build's pool; stop if it updates or replaces a pool a version still points at.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be useful to validate these rules with a small A → B → A example from a fresh checkout using shared state: adding B leaves A’s pool unchanged, and rollback restores routing to A. That would give us confidence that the guidance covers the failure we observed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested A → B → A with shared Terraform state. Adding B from a fresh checkout created only B and did not change A. Both A and B ran successfully. Switching back to A worked, and A still used its original pool and image. All temporary resources were deleted after the test.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for running it end to end.

@campoy

campoy commented Oct 12, 2026

Copy link
Copy Markdown

Thanks for this!

The clearer Terraform scope, shared-state guidance, and protection for retained pools address several of the main findings from our evaluation. The audit checklist and explicit handling of null bindings are helpful additions too.

I left a few inline suggestions around IAM state, rollback readiness, startup-check scope, and validating the release lifecycle.

| jq -e --arg b '<PREVIOUS_BUILD_ID>' '[.versionSummaries[]?.BuildID] | index($b) != null'
```

CLI v1.8.2 spells the field `BuildID`, and `worker deployment list` returns `versionSummaries` as `null`, so read versions from `describe`. Then confirm the version can take traffic. Take its pool, region, image digest, and Task Queue from the recorded deployment inventory or committed release inputs; Temporal does not return the pool a version targets. Run the [Task Queue binding check](diagnostics.md#start-here-did-the-expected-worker-bind), and confirm the pool still exists, its instance split contains exactly one revision, and that revision serves the recorded digest as in [audit items 2 and 3](diagnostics.md#audit-an-existing-pool). If there is no record, or the pool is missing or serves another digest, stop before changing routing and tell the user.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small gap: the stop list covers a missing record, pool or digest, but not a failed binding check or a split with more than one revision. Could we stop on those too?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will look into this and cover this as part of PR #30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants