Repository navigation
feat(webhooks): durable outbox for account- and caller-anchored notifications, not only task webhooks #1290
Description
Summary
On 8.0.0, durable webhook delivery covers only two cases. The first is terminal task webhooks: PgTaskWebhookOutbox.enqueue_terminal, or send_mcp on a supervisor. The second is the reporting.* notifications, through the reporting outbox. Every other AdCP 3.2 notification type has, at best, a one-shot signed WebhookSender.send_* call that returns a single attempt's result. A seller that has to deliver account- or catalog-anchored notifications reliably must therefore build its own queue, retry, lease and signing-scope handling. PgTaskWebhookOutbox already implements all of that, but only for terminal task webhooks.
Details (adcp 8.0.0)
# run with: uv run --no-project --with 'adcp[pg]==8.0.0' python r02.py
import inspect
from adcp.webhook_sender import WebhookSender
from adcp.webhook_supervisor import InMemoryWebhookDeliverySupervisor, WebhookDeliverySupervisor
from adcp.webhook_supervisor_pg import PgWebhookDeliverySupervisor
def public_async(cls):
return sorted(n for n, f in vars(cls).items() if not n.startswith("_") and inspect.iscoroutinefunction(f))
print("WebhookDeliverySupervisor:", public_async(WebhookDeliverySupervisor))
print("InMemoryWebhookDeliverySupervisor:", public_async(InMemoryWebhookDeliverySupervisor))
print("PgWebhookDeliverySupervisor:", public_async(PgWebhookDeliverySupervisor))
print("WebhookSender (one-shot):", [n for n in public_async(WebhookSender) if n.startswith("send")])WebhookDeliverySupervisor: ['send_mcp']
InMemoryWebhookDeliverySupervisor: ['send_mcp']
PgWebhookDeliverySupervisor: ['create_schema', 'run_worker', 'send_mcp']
WebhookSender (one-shot): ['send_artifact_webhook', 'send_collection_list_changed', 'send_mcp', 'send_prepared', 'send_property_list_changed', 'send_raw', 'send_revocation_notification', 'send_webhook_challenge', 'send_wholesale_feed', 'send_wholesale_feed_to_subscription']
adcp/webhook_supervisor.py:190-221: the supervisor protocol issend_mcp(task_id, status, task_type, ...)only.webhook_supervisor_pg.py:344-425: the queue table is shaped around a task (task_id TEXT NOT NULL,status_str,task_type,result_json), and the worker always replays the row throughWebhookSender.send_mcp.adcp/decisioning/pg/task_webhook_outbox.py:279-300:enqueue_terminal(conn, ...)writes on the caller's transaction, but it rejects any status other thancompleted/failedand needs atask_id.adcp/webhook_sender.py:216-228, 719:PreparedWebhookis the right durable unit, butprepare_mcpis the only way to build one.
These enums/notification-type.json types have no SDK durable path:
creative.status_changed,creative.assignment_changed,creative.purged,indicators.changedaccount.status_changed,account.change_recordedproduct.created,product.updated,product.priced,product.removedsignal.created,signal.updated,signal.priced,signal.removedwholesale_feed.bulk_change(these and the product/signal types have one-shotsend_wholesale_feedonly)capabilities.changed,principal.changed
Why an adopter hits this
Embedded Sales Agent delivers account.status_changed, product/signal catalog changes and wholesale_feed.bulk_change to buyer notification_configs. It does so through src/services/protocol_change_webhooks.py (744 LOC) and ProtocolWebhookService.send_notification (src/services/protocol_webhook_service.py, 725 LOC). Signing is per buyer agent, using HMAC or RFC 9421 credentials loaded by src/services/webhook_signing.py (587 LOC). Delivery is three in-process attempts on a background task or thread, so a pod restart loses any notification still pending. (ProtocolWebhookService also sends task push notifications and delivery reports. That part overlaps with SDK send_mcp / PgTaskWebhookOutbox, and ESA has not migrated it.)
ESA also runs a durable Postgres outbox for its own management-API events (src/services/lifecycle_webhook_outbox.py, src/management_api/services/webhook_delivery.py). That envelope is ESA-specific. It shows the same claim/lease/retry machinery being rebuilt a second time.
Ask
- Make the outbox take arbitrary prepared notifications. For example,
PgTaskWebhookOutbox.enqueue_prepared(conn, prepared: PreparedWebhook, *, account_id, notification_type, signing_scope_id=None), or a siblingPgNotificationOutboxthat shares the worker, encryption and retry horizon. - Add
prepare_*counterparts to the one-shot senders (prepare_wholesale_feed,prepare_raw, and typed builders for creative, account, capabilities and principal webhooks). Callers could then bind the body and idempotency key inside their business transaction and leave delivery to the worker.
Activity
bokelley commentedon Oct 2, 2026
Triage
Classification: Feature request
Bucket(s): client (webhook sender / Pg outbox layer)
Status: ready-to-implement
Milestone: omitted
What the experts said:
PgTaskWebhookOutboxhas three hard blockers for reuse: (1)task_id TEXT NOT NULL UNIQUE— no valid sentinel for non-task rows; (2)_envelope_aadbindstask_id/task_type/status/operation_id— non-task rows would produce unsafe AAD that defeats the authenticated-encryption guarantee; (3)_validate_stored_bodycross-checks those same task fields and would permanently quarantine every non-task row at worker-replay time. A separatePgNotificationOutboxclass with its own table is the only clean path. Also flagged:WebhookDeliverySupervisorProtocol only exposessend_mcp; the issue should decide whether to extend it withsend_preparedor leave the new outbox as a standalone object.PreparedWebhookis the right durable unit for all notification types — signing scope lives in the outbox row's AAD, not in the serialized bytes, sosend_preparedroutes correctly at delivery time regardless of notification category. ConfirmedPgNotificationOutboxis the only non-breaking path. Critical: any new outbox must mint a distinct AAD version string (e.g."notification-outbox-v1") and commitnotification_typein the AAD to prevent type-confusion attacks where a storage compromise swaps an encryptedcreative.status_changedenvelope into a row whosenotification_typecolumn readsaccount.status_changedand decryption still succeeds. Prior art:reporting_notification_deliveriesschema (src/adcp/reporting/ledger/reporting_notification_outbox.sql) already demonstrates the right shape —notification_type,signing_scope_id,idempotency_key UNIQUE, and the standard worker states.My take: Both experts agree and the code confirms: a new
PgNotificationOutboxclass with its own table is the right and only non-breaking approach. The request is well-specified and the AAD type-confusion note from the protocol expert is the key security constraint to get right.Implementation scope:
src/adcp/webhook_sender.py— addprepare_*factory methods alongside each one-shotsend_*:prepare_wholesale_feed,prepare_raw,prepare_collection_list_changed,prepare_property_list_changed,prepare_artifact_webhook,prepare_revocation_notification. Each serializes payload + generates idempotency key + returnsPreparedWebhook(url, idempotency_key, body). Validation checks from the correspondingsend_*method (e.g.notification_type == event.event_typeinsend_wholesale_feed) must run inside theprepare_*method, not deferred — an invalid envelope stored durably has no recourse at delivery time.src/adcp/notification_outbox_pg.py(new file) —PgNotificationOutboxclass:adcp_notification_outbox— mirrorreporting_notification_deliveriesshape:notification_type TEXT NOT NULL,account_id TEXT NOT NULL,idempotency_key TEXT UNIQUE,signing_scope_id TEXT,encrypted_body BYTEA,envelope_nonce BYTEA,state IN ('pending','in_flight','delivered','expired','invalid'), standard lease/retry/purge columns. Droptask_id,terminal_status,task_type.enqueue_prepared(conn, prepared: PreparedWebhook, *, notification_type: str, account_id: str, signing_scope_id: str | None = None) -> int— write on the caller's open transaction (same atomic-outbox contract asenqueue_terminal). AAD must include["notification-outbox-v1", notification_type, account_id, url, idempotency_key (, signing_scope_id)].run_worker()/process_one()— AES-256-GCM decrypt,_validate_stored_bodyequivalent bindingnotification_type+idempotency_key, thensender.send_prepared(prepared). Quarantine onInvalidTagor body mismatch.src/adcp/decisioning/pg/__init__.pyand/orsrc/adcp/__init__.py— exportPgNotificationOutbox.WebhookDeliverySupervisorProtocol gains asend_preparedabstract method orPgNotificationOutboxstays standalone. The issue doesn't specify; either is workable.Non-breaking rationale: All additions. No changes to
PgTaskWebhookOutbox,WebhookSender,PgWebhookDeliverySupervisor, or their public APIs. New table is opt-in viacreate_schema().Required checks before PR:
pytest tests/(targeted subset — new outbox unit tests covering success/retry/quarantine/AAD-mismatch paths),mypy src/,ruff check .Triaged by Claude Code. Session: https://claude.ai/code/session_01TF3MYaArTY2gSo3jwx7fLF
bokelley commentedon Oct 4, 2026
Completed by #1335 (validated non-task notification preparation) and #1338 (transactional PostgreSQL notification outbox, squash a0fbee3).