Repository navigation
feat(ingest): receive Cloud Logging entries from GCP Pub/Sub push - #1320
Confidence 3/10 · 3 issues to address
🔴 Confidence 3/10 · risky as written
The signed-push path still cannot authenticate (F3, maple-push- vs the script's maple- account), so GCP OIDC pushes 401 in production while the tests pass.
quality 70/100 · 3 warnings · tests covered · risk high · 3/4 new units observable
Adds the GCP Pub/Sub log-push receiver: a new authenticated route, connector resolver with caches and throttled health writes, a LogEntry-to-OTLP mapper and four counters, with thorough Rust tests. Not safe to merge yet: the signed-push account name still disagrees with what the setup script creates (F3), so OIDC push subscriptions cannot authenticate.
handle_gcp_logpushacks LogEntry pushes and drops poison payloads at 200GcpConnectorResolverroutes by secret or Pub/Sub ID token, with setup-report health writesgcp_logging::parse_log_entrymaps Cloud Logging entries to OTLP log records- Four
ingest_gcp_*counters plus a per-org GCP in-flight limiter
Before merge
- Env var · Set
INGEST_GCP_ORG_MAX_IN_FLIGHTin every environment, or confirm its default ·apps/ingest/alchemy.run.ts:619 - Infra · Check every environment has what this config now expects ·
apps/ingest/alchemy.run.ts - Manual · Merge #1321 first: this PR selects
gcp_connectors.project_id,logs_enabledandapplied_logs_enabled, which that PR creates - Manual · Fix the push account name mismatch before enabling OIDC push subscriptions in the setup script
Findings
🟠 Warning · F5 · A failed signing-key fetch is retried by every later push, under the write lock
performance · apps/ingest/src/gcp_push_auth.rs:261-263
When fetch_keys fails, fetched_at keeps its old value, so due (gcp_push_auth.rs:236) stays true and every later push calls Google's /oauth2/v3/certs again while holding cached's write lock; the REFETCH_INTERVAL guard only covers the unknown_key case. Once the keys are older than KEYS_TTL, a certs-endpoint outage makes each GCP push wait out one in-flight fetch and then issue its own, so pushes serialise behind a fetch that keeps failing.
Record the attempt on failure too — keep a `last_attempt: Instant` that `due` also checks, or on `Err` push `fetched_at` back to `Instant::now() - (KEYS_TTL - REFETCH_INTERVAL)` — so a failing fetch is retried at most once per `REFETCH_INTERVAL` while the keys still in hand keep verifying.
Still open from earlier reviews
- 🟠 Warning · F3 ·
gcp_push_service_accountbuilds an account name the repo never creates ·apps/ingest/src/main.rs:5170 - 🟠 Warning · F4 · Google's signing-key fetch has no Client span ·
apps/ingest/src/gcp_push_auth.rs:267
Production impact
Production traffic of the changed files (last 7 days)
| File | Calls/day | Busiest operations |
|---|---|---|
apps/ingest/src/main.rs |
176k | maple 175k/day, 94.6% errPOST 906/day, 3.6% errPOST 140/day, 0.5% err |
apps/ingest/src/gcp_logging.rs |
809 | GET 662/day, 2.6% errGET 147/day, 0.1% err |
What was checked
- Poison payload is data loss:
rejection_loses_data("decode")marks the span Error (otel.rs:229) - OIDC audience is pinned to
/v1/logpush/gcp/{connector_id}and the email to the connector's project (main.rs:6305) - Setup reports are recognised only after auth and are rate-limited per connector (main.rs:5052, 6237)
Observability coverage: 3 of 4 changes observable
| Change | Kind | Observable | Evidence |
|---|---|---|---|
| POST /v1/logpush/gcp/{connector_id} inbound endpoint | server | yes | server span with otel.name/http.route/maple.gcp.* at apps/ingest/src/main.rs:4862 |
| Google signing-key fetch (oidc push auth) | client | no | gcp_push_auth.rs:267 calls the shared client with no span (open finding F4) |
| gcp_connectors Postgres lookups and health/setup writes | client | yes | postgres_client_span around the queries, e.g. apps/ingest/src/main.rs:6882 |
| GCP push counters and per-org data-loss counter | metrics | yes | ingest_gcp_entries_total / auth, parse, disabled counters in apps/ingest/src/metrics.rs:114 |
13399a9 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.
Annotations
Check warning on line 263 in apps/ingest/src/gcp_push_auth.rs
maple-review-bot / Maple / review
performance: A failed signing-key fetch is retried by every later push, under the write lock
When `fetch_keys` fails, `fetched_at` keeps its old value, so `due` (`gcp_push_auth.rs:236`) stays true and every later push calls Google's `/oauth2/v3/certs` again while holding `cached`'s write lock; the `REFETCH_INTERVAL` guard only covers the `unknown_key` case. Once the keys are older than `KEYS_TTL`, a certs-endpoint outage makes each GCP push wait out one in-flight fetch and then issue its own, so pushes serialise behind a fetch that keeps failing.