[Ankur Gupta (ankur-pattern)]

Summary

Adds max_throttle_retries to the http task: extra attempts for 429 responses that carry a Retry-After header. Each one sleeps for Retry-After and doesn't count against max_retries. The default is 0, so existing pipelines behave exactly as before.

Also fixes a small bug: a 429 whose Retry-After couldn't be parsed used to retry with no sleep at all. It now falls back to the existing 2^attempt backoff.

Why

The SP-API pipelines in data-airflow (catalog search and competitive pricing) call Amazon through rate-gate. Rate-gate keeps a token bucket per seller account (2 calls a second for catalog, 0.5 for pricing). When the bucket is empty it answers 429 with Retry-After: 1 or 2 and X-Intercepted-By: rate-gate. Those 429s share the max_retries: 5 budget with every other error, so a busy account fails its chunks within seconds instead of making them wait.

On 2026-10-28 the SERP catalog run had 517 of 548 chunks running at once, 8 to 12 per account. 246 chunks needed an Airflow retry, and 14 reached try 4 or 5; try 5 failing would have blocked publish. Raising max_retries itself isn't the fix, because it also stretches Amazon's own 429 backoff (2^attempt, uncapped). Amazon's 429s don't send Retry-After, so they keep using max_retries.

With this change a pipeline can set, for example, max_throttle_retries: 60. Oversubscribing an account then slows its chunks down instead of failing them.

Changes

• http/http.go: new MaxThrottleRetries field, copied in newFromInput. In call, a throttled 429 within budget sleeps Retry-After and retries without using an attempt. Each throttle wait is capped at 60s (maxThrottleDelay); a longer Retry-After takes the normal max_retries path, so the budget can't stall a record for hours. Each throttle retry is logged.
• http/retry.go: retryAfter(response) parses Retry-After (seconds) for 429s. handleBackoff uses it and falls back to 2^attempt when it's missing or unparseable.
• http/README.md: documents the field.
• http/retry_test.go: tests against an httptest server covering throttles beyond max_retries, no budget, both budgets used up, a missing Retry-After, an unparseable Retry-After, and a Retry-After above the cap.

Testing

• go vet -copylocks=false ./... and go test -race ./... pass locally, the same as CI.

Rollout

ECS ingest tasks run the image caterpillar:manual-latest (heimdall-config/configs/cp_task_def.json.tmpl), so that image needs a rebuild after this merges. YAML parsing isn't strict, so data-airflow can add max_throttle_retries before then; it's ignored until the new binary runs.

[argus-pattern]

The throttle path sleeps for whatever Retry-After the server sends, with no upper bound, and up to max_throttle_retries times. With the suggested max_throttle_retries: 120, a single upstream (or a proxy in front of it) answering Retry-After: 300 would hang this record for about 10 hours before failing, and time.Sleep cannot be interrupted. Before this change the same uncapped sleep could only happen max_retries times. Nothing is logged either, so a stalled task gives no sign that it is being throttled.

Fix: cap the per-throttle delay (a constant, or a max_throttle_delay option), or fall through to the normal max_retries path when delay is above the cap. A debug or info log per throttle retry would also help.

[Ankur Gupta (ankur-pattern)]

Fixed in b0898d0. Each throttle wait is now capped at 60s (maxThrottleDelay). A Retry-After above that skips the throttle budget and takes the normal max_retries path, so the worst case per call is max_throttle_retries × 60s instead of unbounded. Every throttle retry logs http: 429 throttle retry n/N in <delay>. The URL is left out of the log line so query params don't end up in task logs. Added a test case for a Retry-After above the cap, and the README documents the 60s limit. The PR body example now uses 60, the value data-airflow sets, not 120.

[Ankur Gupta (ankur-pattern)]

Withdrawn: we don't need max_throttle_retries.