Skip to content

feat(gcp): set up a connection with Terraform, and sign log pushes instead of a URL secret - #1499

Closed
JeremyFunk wants to merge 6 commits into
feat/gcp-promql-mcpfrom
feat/gcp-terraform
Closed

JeremyFunk wants to merge 6 commits into
feat/gcp-promql-mcpfrom
feat/gcp-terraform

Conversation

@JeremyFunk

@JeremyFunk JeremyFunk commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Google Cloud admins can now set a connection up with Terraform instead of pasting the setup script, and log pushes no longer carry a secret.

What changes

  • Terraform module deploy/terraform-google-maple: the resources the script creates, under the same names, so either can take over from the other. Additive *_iam_member grants only. Mock-provider tofu test covers project/org scopes, logs off, and a secret-bearing endpoint being refused; a new workflow runs fmt/validate/test.
  • Signed pushes (OIDC): the push subscription signs each request as a new push service account maple-push-<19 hex> that holds no role. The script and the module both set it up. The ingest gateway verifies the Google ID token (RS256 against Google's JWKS, issuer, expiry, verified email), then checks the email is the connector's push account in its host project and that the audience names the connector. Subscriptions an earlier script made keep working with the URL secret until the script runs again.
  • Script: creates the push account, lets Pub/Sub's service agent sign as it, sets --push-auth-service-account/--push-auth-token-audience, and deletes the account when log forwarding is turned off or cleaned up. The secret now only authenticates the script's own reports (REPORT_ENDPOINT); Google Cloud never gets it.
  • API: setup_scripts also returns terraform (the module block). New POST /v2/integrations/gcp/connectors/{id}/check_access reads the scope as the read-only account and records it as in place, since Terraform sends no report.
  • Confirmation without a report: the first signed push to a connector nothing has reported on records log forwarding as set up (once; the cached identity stops asking before the write).
  • Web: the apply stage has a Cloud Shell script / Terraform switch. The Terraform view shows the block, the apply step, and Maple's confirmation (first log entry, Check access). Rendered in /lab/gcp.
  • Docs: "Set up with Terraform" with import blocks for taking over a script-made connection, updated permissions and disconnect.

Verified

  • Vitest: setup script (60, real bash against fake gcloud), Terraform renderer, connector service, PromQL service incl. checkAccess, v2 route test, domain contract tests.
  • Rust: full ingest suite (346), incl. verifier unit tests and signed-push receiver tests.
  • tofu validate/tofu test on the module; a rendered block validated against the module with OpenTofu.

Not verified: a real terraform apply and a real signed push against Google Cloud. Worth one run on maple-development-project before release.

Before release

  • Push the tag terraform-google-maple-v1.0.0 after merge: the generated block pins ?ref= to it.
  • Optional: mirror the module to a terraform-google-maple repo to publish it on the Terraform Registry.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

The setup script and a new Terraform module create the same resources. The push
subscription now signs each request as a dedicated push service account that holds no
role, so neither the subscription nor Terraform state carries the connection's secret.
setup_scripts now also returns the Terraform module block. check_access signs in as the
connection's read-only account and records it as in place, which a Terraform run cannot
report the way the setup script does.
…h account

A push carrying a Google ID token is accepted when Google signed it for the
connector's push service account in its host project and for an audience naming the
connector. The first such push to a connector nothing has reported on (Terraform
sends no report) records log forwarding as set up. Pushes with the URL secret keep
working for subscriptions an earlier script created.
The apply stage offers the Terraform module beside the Cloud Shell script, with a
check that confirms the read-only access, since a Terraform run reports nothing.
Documents setting a connection up with Terraform, taking over a script-made one by
import, and the push service account. The module's mock-provider tests run in CI.
@maple-review-bot

maple-review-bot Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 8/10 · likely safe to merge
Push auth agrees across the script, the module and the ingest, but no real Google apply or signed push was ever run.
quality 98/100 · 1 note · tests covered · risk medium · 2/3 new units observable

Google Cloud connections can now be set up with a Terraform module, and Pub/Sub pushes are OIDC-signed instead of carrying a URL secret; the ingest verifies the ID token, and a new check_access endpoint confirms read-only access without a report. The change is safe to merge.

  • setup_scripts also returns a secret-free terraform module block
  • Ingest verifies signed Pub/Sub pushes against Google's JWKS in preference to the URL secret
  • New POST .../check_access reads the scope as the read-only account and records it
  • Setup script creates and deletes the maple-push-* service account

Before merge

Findings

🔵 Note · F1 · The JWKS fetch has no Client span, so ingest→Google is absent from the service map

observability · MAP-01 · apps/ingest/src/gcp_push_auth.rs:268-272

fetch_keys sends its request through the raw HttpClient (a plain reqwest::Client outside --features hotpath), so a key fetch appears only as untraced time inside the gcp_logpush receiver span, and www.googleapis.com never shows as a peer of the ingest. A refused or slow fetch is then indistinguishable from a slow push until the retry. Add a client span around it, the way clickhouse_export_span covers the ClickHouse export (telemetry.rs:3088), with peer.service = www.googleapis.com and the outcome recorded on it.

🤖 Prompt to fix this finding with an AI agent
Findings from an automated review of commit 741679f4e2635b4223fc08ab9e7241bb7c0cea6a. Verify each one against the current code before changing anything, fix only those that still apply, and keep each fix to the lines it names.

---

F1 · Note · observability · MAP-01 · apps/ingest/src/gcp_push_auth.rs:268-272
The JWKS fetch has no Client span, so ingest→Google is absent from the service map
`fetch_keys` sends its request through the raw `HttpClient` (a plain `reqwest::Client` outside `--features hotpath`), so a key fetch appears only as untraced time inside the `gcp_logpush` receiver span, and `www.googleapis.com` never shows as a peer of the ingest. A refused or slow fetch is then indistinguishable from a slow push until the retry. Add a client span around it, the way `clickhouse_export_span` covers the ClickHouse export (telemetry.rs:3088), with `peer.service = www.googleapis.com` and the outcome recorded on it.

Production impact

Telemetry this change adds and removes (1)
  • ➕ span name GcpPromQlService.checkAccess · packages/backend/src/services/integrations/GcpPromQlService.ts:622
What was checked
  • Tenant scoping: the resolver keys on connector id and OrgId throughout (main.rs:6318, GcpPromQlService.ts:631)
  • A metrics: None report cannot wipe read-only state (COALESCE, main.rs:7114)
  • Push account name, audience and roles agree across the script, module and ingest (logs.tf:67, variables.tf:36), so old URL-secret subscriptions keep working
Observability coverage: 2 of 3 changes observable
Change Kind Observable Evidence
POST /v1/logpush/gcp/{connector_id} signed-push verification and connector resolution inbound entrypoint yes Server span gcp_logpush with maple.gcp.push_auth and maple.gcp.connector_id (main.rs:4878, 4991); metrics::gcp_auth_failure() on refusal (main.rs:4952)
Google JWKS fetch for push tokens outbound call no raw HttpClient get with no Client span (gcp_push_auth.rs:268); only the parent receiver span covers it
POST /v2/integrations/gcp/connectors/{id}/check_access inbound entrypoint yes Effect.fn("GcpPromQlService.checkAccess") span annotated with orgId and maple.gcp.connector_id (GcpPromQlService.ts:622)

741679f · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one. Check ids refer to Maple's instrumentation audit.

@coderabbitai

coderabbitai Bot commented Oct 15, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ec7a016d-439d-4e53-bb51-bc750daa64f3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@JeremyFunk

Copy link
Copy Markdown
Collaborator Author

Folded into the stack: ingest verification into #1320, everything else into #1477, merged up to #1478. Closing.

@JeremyFunk JeremyFunk closed this Oct 15, 2026
@JeremyFunk
JeremyFunk deleted the feat/gcp-terraform branch October 15, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant