[Olorunfemi20]

Summary

#1473 Prometheus alert metrics and health telemetry

• New src/lib/log-retention-telemetry.js, with its own registry merged into GET /metrics. Series:
  • log_retention_purge_runs_total{outcome} (success, failure, skipped)
  • log_retention_purge_duration_seconds{outcome}
  • log_retention_purge_deleted_rows_total{table} and log_retention_purge_batches_total{table}
  • log_retention_purge_duration_exceeded_total
  • log_retention_purge_retries_total{result}
  • log_retention_purge_lock_total{result}
  • gauges: health state, in progress, consecutive failures, last success time, last run time
• Health is derived from run history:
  • idle: no run in this process yet
  • healthy
  • degraded: last run failed, overran its time budget, or a run is overdue
  • unhealthy: failure streak at the limit, or no success within the staleness limit
• Endpoints: new GET /health/log-retention (503 only when unhealthy). GET /health lists services.log_retention for information only; it does not change ok.
• The cron script pushes its metrics to a Pushgateway when LOG_RETENTION_PUSHGATEWAY_URL is set, because the process exits before any scrape could reach it.
• Alert rules: docs/alerts/log-retention.rules.yml (unhealthy, failing, stale, duration exceeded, retries exhausted, lock errors).
• Tests: src/lib/log-retention-telemetry.test.js covers every health transition, label sanitising, gauges refreshed at scrape time, and the Pushgateway success, refusal and failure paths.

#1474 Automated retry with exponential backoff

• New src/lib/log-retention-retry.js. Every purge query runs through withLogRetentionRetry(): index check, FK check, each delete batch, lock connect and lock acquire.
• Retried: SQLSTATE 40001, 40P01, 55P03, 57014, 57P01-57P03, 53300, class 08, and socket errors.
• Never retried: safety-check failures (new LogRetentionSafetyError, retryable: false) and any other SQLSTATE.
• Delay: full jitter, random(0, min(max, base * 2^n)).
• Settings: LOG_PURGE_RETRY_MAX_ATTEMPTS / _BASE_DELAY_MS / _MAX_DELAY_MS, clamped to hard ceilings (8 attempts, 60 s). Retrying stops once the run passes LOG_PURGE_MAX_DURATION_MS.
• Tests: src/lib/log-retention-retry.test.js (classification, option clamping, backoff math, recovery, exhaustion, abort, log fields) and new cases in src/lib/log-retention.test.js.

#1475 Distributed concurrency control and locking

• New src/lib/log-retention-lock.js:
  • an in-process single flight per pool;
  • a Postgres session advisory lock, pg_try_advisory_lock(hashtext(<fixed key>)), on a dedicated pooled connection. It never waits; contended means skipped, not failed.
  • Postgres drops the lock if the holder dies. If unlocking fails, the connection is destroyed instead of returned to the pool, which also drops the lock.
• Delete batches now use FOR UPDATE SKIP LOCKED, so even unlocked callers never block on, or double-delete, each other's rows.
• New runLogRetentionPurge() wraps purgeWebhookLogs() in the lock and returns { skipped: true, reason: "locked" } when another run holds it.
• scripts/purge-webhook-logs.js now uses it.
• Optional in-API schedule LOG_RETENTION_PURGE_CRON (off by default), safe on every instance.
• Tests: src/lib/log-retention-lock.test.js (acquire, contend, in-process skip, per-pool instances, acquire errors, retry, release failures, single client) and runLogRetentionPurge cases in src/lib/log-retention.test.js.

#1476 Integration and stress test suite

• tests/helpers/fake-log-retention-db.js: an in-memory Postgres double implementing the safety checks, DELETE ... FOR UPDATE SKIP LOCKED (rejects a delete without it), and session advisory locks. Several pools sharing one database simulate several instances. It records deleted ids, double deletes, peak lock holders, open connections and held locks.
• tests/integration/log-retention.test.js covers the full lock, retry, purge and telemetry pipeline:
  • batching, and only expired rows deleted;
  • skip while another host holds the lock;
  • one of two concurrent instances runs; a same-process double trigger is skipped;
  • recovery from transient lock, check and delete failures;
  • a clean failure after exhaustion; the missing-index refusal; dropping the lock when unlock fails; overlapping unlocked callers;
  • over HTTP through the real app: /health/log-retention (idle, healthy, 503), /health and /metrics.
• load-tests/log-retention-stress.test.js:
  • 50 racing instances: exactly one purges 5,000 rows;
  • 10 waves with 20% injected transient failures and random latency (seeded PRNG): converge with no leaks;
  • 16 unlocked callers: every row deleted exactly once;
  • a 100-trigger burst on one instance: only one reaches the database lock;
  • a 50,000-row backlog throughput check.

Test output

Security notes

• Table, column and lock key are server-side constants. Only retention days and batch size reach SQL, and only as bind parameters.
• Metric labels come from fixed sets: no row ids, log contents or error text. Unknown values map to other / failure / error.
• Health bodies contain counts and timestamps only.
• The Pushgateway URL must be http(s) and is never logged, because it may carry credentials.
• Retry ceilings prevent a failing purge from becoming load on the primary. Safety checks still fail closed and are never retried.
• The lock never waits and cannot get stuck past the holder's connection.

Notes

• Reading of the issues: "Log Retention & Archival Service" is src/lib/log-retention.js and its cron script, the webhook delivery log purge. Payment archival in maintenance.js is untouched.
• Backwards compatibility:
  • purgeWebhookLogs() keeps its signature and return value, and still takes no lock. It gains an optional retry option.
  • The existing console log and alert lines are unchanged.
  • Schedulers should use runLogRetentionPurge().
• Behaviour changes:
  • The delete batch query adds FOR UPDATE SKIP LOCKED.
  • Transient query failures are now retried instead of failing the run immediately.
  • The cron script exits successfully with a "Run skipped" line when another host holds the lock.
• Health is per process. API instances report idle unless LOG_RETENTION_PURGE_CRON is set there; for the cron script, use the Pushgateway metrics.
• New metrics also register on prom-client's default registry, the same as the existing per-module registries.
• No existing tests or fixtures were changed. The original two tests in log-retention.test.js are untouched; new cases were appended.

closes #1473
closes #1474
closes #1475
closes #1476

[drips-wave]

@Olorunfemi20 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[vercel]

@Olorunfemi20 is attempting to deploy a commit to the Emmanuel's projects Team on Vercel.

A member of the Team first needs to authorize it.

[gitguardian]

⚠️ GitGuardian has uncovered 4 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
38822392	Triggered	JSON Web Token	a15452c	backend/src/middleware/x402.test.js	View secret
38865302	Triggered	Generic High Entropy Secret	a15452c	backend/src/lib/sep0001-generator.sanitization.test.js	View secret
38865486	Triggered	Generic High Entropy Secret	a15452c	backend/src/lib/websocket-relay-retry.test.js	View secret
38864685	Triggered	Generic High Entropy Secret	a15452c	backend/src/lib/exchange-rate-oracle-sanitizer.test.js	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}