Skip to content

feat(setup-audit): warn at 0.1% and flag 1% of traces without a root span - #1564

Merged
JeremyFunk merged 12 commits into
mainfrom
feat/setup-health-rootless-traces
Oct 20, 2026
Merged

JeremyFunk merged 12 commits into
mainfrom
feat/setup-health-rootless-traces

Merge remote-tracking branch 'origin/main' into feat/setup-health-roo…

d4df1a1
Select commit
Loading
Failed to load commit list.
Maple Review Bot / Maple / review succeeded Oct 20, 2026 in 9m 1s

Confidence 10/10 · No issues found

🟢 Confidence 10/10 · safe to merge
Tier boundaries, the 50-trace floor and the sampler exclusion are pinned by tests; only the window-total-before-LIMIT behaviour rests on the author's prod run rather than a local one.
no findings · tests covered · risk medium · 1/1 new units observable

TRC-02 becomes an org-wide rootless-trace rate with info/warn/critical tiers, read from traces instead of service_map_children, with org totals from window functions so the row cap cannot inflate the rate. Thresholds, sampler exclusion and the degraded-join behaviour are tested; safe to merge.

  • auditRootlessTracesSQL scans traces and left-joins root spans, excluding sampler-kept fragments and traces that started before the window
  • TRC-02 reports info, warn or critical per tier via the new AuditEvaluation.severity
  • SetupAuditService computes AuditSamplingOverrides from the ingest policy and runs the two trace joins independently
  • TRC-02's fix hint and button now point at the new missing-root-spans guide

Production impact

Production traffic of the changed files (last 7 days)
File Calls/day Busiest operations
packages/backend/src/services/org/SetupAuditService.ts 9 SetupAuditService.fetchWarehouseInputs 9/day, 0.0% err
What was checked
  • Sampler exclusion mirrors ingest's per-span keep rules in should_keep_trace (apps/ingest/src/telemetry.rs:4043-4079), including the floor(ns/1e6) >= ms slow-span equivalence
  • org* totals are window functions over the grouped rows, emitted before LIMIT 1000 (setup-audit.ts:652-668), so the cap cannot lower the rate
  • Exactly 0.1% and exactly 1% land in info and warn respectively, per the domain tests (setup-audit.test.ts:1497-1590)
Observability coverage: 1 of 1 changes observable
Change Kind Observable Evidence
Rootless trace scan (SetupAuditService.fetchWarehouseInputs → auditRootlessTracesSQL) outbound DB query yes Runs through warehouse.compiledQuery with its own context, so it keeps the existing Client span and db.duration_ms; a failure logs a warning with the cause before degrading to []

d4df1a1 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.