Skip to content

feat(sdk,core,webapp,run-engine): runtime override for the combined queue concurrency limit - #4829

Open
matt-aitken wants to merge 13 commits into
feat/queue-gates-contractfrom
feat/queue-concurrency-overrides
Open

feat(sdk,core,webapp,run-engine): runtime override for the combined queue concurrency limit#4829
matt-aitken wants to merge 13 commits into
feat/queue-gates-contractfrom
feat/queue-concurrency-overrides

Conversation

@matt-aitken

@matt-aitken matt-aitken commented Aug 29, 2026

Copy link
Copy Markdown
Member

Summary

Adjust a queue's combined concurrency limit at runtime without a deploy. queues.overrideCombinedConcurrencyLimit raises or lowers the cap on concurrent runs across all of a queue's concurrencyKey values, and queues.resetCombinedConcurrencyLimit reverts to the declared configuration. Stacked on #4827.

await queues.overrideCombinedConcurrencyLimit("my-queue", 100);
await queues.resetCombinedConcurrencyLimit("my-queue");

Design

Combined overrides follow the existing override bookkeeping exactly: validated against the environment limit, the declared base captured on first override, reset restoring it, and deploys preserving an active override instead of clobbering it from the code manifest (the declared value lands in the base). The read path clamps to the environment limit at admit time.

Per-key runtime overrides were built in earlier revisions of this PR and then deliberately removed: composing a per-key override hash with pause semantics and keeping it convergent with durable rows across deploys, resets, and races needs its own design, so that feature is deferred to a future PR. Declared per-key behavior is unchanged (concurrencyLimit applies per key as before).

@changeset-bot

changeset-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 43c724e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 27 packages
Name Type
@trigger.dev/sdk Patch
@trigger.dev/core Patch
@trigger.dev/python Patch
@internal/dashboard-agent Patch
@trigger.dev/build Patch
trigger.dev Patch
@trigger.dev/redis-worker Patch
@trigger.dev/schema-to-json Patch
@internal/clickhouse Patch
@internal/llm-model-catalog Patch
@internal/metrics-pipeline Patch
@trigger.dev/rbac Patch
@internal/redis Patch
@internal/replication Patch
@internal/run-engine Patch
@internal/run-store Patch
@internal/schedule-engine Patch
@internal/tracing Patch
@internal/webhook-engine Patch
@internal/webhook-sources Patch
@internal/cache Patch
@trigger.dev/react-hooks Patch
@trigger.dev/rsc Patch
@trigger.dev/database Patch
@trigger.dev/otlp-importer Patch
@trigger.dev/sso Patch
@internal/testcontainers Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ad3b3fa8-6657-41f1-a54e-aae53b0ee8d3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Observability map

As of 43c724e.

19/100 over 456 measured of 474 entry points (base 20, down 1)

What this PR changed

route base head now failing
/api/v1/queues/:queueParam/concurrency/combined/override new 0 request-context
/api/v1/queues/:queueParam/concurrency/combined/reset new 0 request-context

FIX FIRST

  • /api/v1/projects/:projectRef/envvars (sensitive) - auth-boundary, request-context
  • /auth/sso (sensitive) - auth-boundary, request-context
  • /_app/orgs/:organizationSlug/settings/team (sensitive) - error-classification, auth-scope, request-context

AUDIT 3 of 50 sensitive mutations record an actor. 47 without one.
CONTEXT 23 of 456 entry points name a tenant on a failure path. 354 appear only here, 39 of them sensitive, in the JSON rather than the fix list.

What the score is made of
CHECKS
  error-classification  184 applicable, 106 pass,   0 sole, global without it 11
  auth-boundary          62 applicable,  57 pass,   0 sole, global without it 16
  auth-scope             19 applicable,  17 pass,   0 sole, global without it 19
  request-context       456 applicable,  23 pass, 250 sole, global without it 65
  audit-trail            50 applicable,   3 pass,   0 sole, not in the score

The score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@pkg-pr-new

pkg-pr-new Bot commented Aug 29, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@43c724e

trigger.dev

npm i https://pkg.pr.new/trigger.dev@43c724e

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@43c724e

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@43c724e

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@43c724e

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@43c724e

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@43c724e

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@43c724e

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@43c724e

commit: 43c724e

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-overrides branch from 5110870 to fe1f3a4 Compare August 29, 2026 15:33
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-overrides branch from 2e8db3a to eba7d2a Compare August 29, 2026 18:11
@matt-aitken matt-aitken changed the title feat(sdk,core,webapp,run-engine): runtime overrides for total and per-key queue limits feat(sdk,core,webapp,run-engine): runtime overrides for combined and per-key queue limits Aug 29, 2026
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-overrides branch from eba7d2a to 0f83715 Compare August 29, 2026 18:18
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-overrides branch from 0f83715 to 0bef7b3 Compare August 29, 2026 18:21
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-overrides branch from 0bef7b3 to f103a5e Compare August 31, 2026 12:02
@matt-aitken matt-aitken changed the title feat(sdk,core,webapp,run-engine): runtime overrides for combined and per-key queue limits feat(sdk,core,webapp,run-engine): runtime override for the combined queue concurrency limit Aug 31, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-overrides branch from 8ba16f9 to 9800b6a Compare August 31, 2026 12:15
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-overrides branch 4 times, most recently from da92e21 to 6b14574 Compare August 31, 2026 14:18
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-overrides branch from 6b14574 to c227170 Compare August 31, 2026 15:26
Sparse ckLimits HASH at the base queue whose fields are the exact ck-variant
queue names, plus engine methods to set (atomic cardinality cap, default
1000 per queue), remove, and read the overrides. The admit-path gate wiring
follows.
The ck dequeue admit and both enqueue fast paths read the queue's ckLimits
HASH for the variant being admitted and use the env-clamped override in
place of the queue's per-key limit, behind the totalConcurrencyEnabled
flag. Covered by tests for lowered and raised keys, removal, the
cardinality cap, and flag-off behavior.
Three nullable TaskQueue columns record when, by whom, and from what
declared base the total concurrency limit was overridden, and a new
TaskQueueConcurrencyKeyOverride child table stores per-key limit overrides,
unique per queue and key and cascading with the queue.
queues.overrideConcurrencyLimit accepts a named concurrencyKey to adjust one
key's limit independently, and new overrideTotalConcurrencyLimit and
resetTotalConcurrencyLimit calls adjust the cap across all keys. Four API
routes back them; the concurrency system validates against the environment
limit, captures the declared base on first override, persists per-key
overrides in the child table alongside the engine hash, and deploys keep an
overridden total instead of clobbering it from the manifest.
…ked keys

A variant blocked at its per-key limit or by a gate now backs off in the
candidate index instead of pinning the bounded window, so zero-limit keys can
never starve runnable keys behind them; acks and nacks rebalance the score
back the moment capacity frees. Override writes persist before enforcing and
resets enforce before clearing, so a failure on either side leaves a state a
retry converges from, with the cap-rejected row compensated away. Deploys
restore per-key overrides from their durable rows into the engine, and an
empty concurrencyKey in the SDK no longer falls through to the queue-wide
endpoint.
…en races

Gate capacity now reads the gate queue's ckLimits hash, so an override on a
key applies whether runs meet it as their own queue or as a gate. A reset
deletes only the exact row generation it read, so a concurrent override's
newer record survives, and a cap rejection deletes its row unconditionally
since the cap can only reject keys absent from the engine hash.
…fe cap cleanup

Gate admission reads a per-key override only when concurrency limit
enforcement is enabled, matching the primary admit paths (the flag now
threads through the unkeyed enqueue and dequeue scripts too). The
cap-rejection cleanup deletes only the exact row generation the rejected
request wrote, so a concurrent request that succeeded after capacity freed
keeps its durable record.
The client build's export analyzer cannot statically resolve destructured
route exports, so it treats the module's exports as depending on server-only
code and the build fails. Also export the builder's loader so non-POST
methods get a 405, matching the other concurrency routes.
The override and reset endpoints, client methods and SDK functions now say
combinedConcurrencyLimit, matching the queue option.
…limit overrides

Per-key runtime overrides are deferred: composing a per-key hash with pause
semantics and keeping it convergent with durable rows across deploys and
resets needs its own design. Combined-limit overrides stay. Declared per-key
behavior is unchanged (concurrencyLimit applies per key as before).
Removes the per-key override endpoints, the override-aware admit and gauge
reads, and the per-key limit column, following the removal of runtime
per-key overrides from this stack.
…remain

The per-key admit reads are removed in the PR above alongside the rest of
the engine-internal plumbing; the key builders they reference stay until
then so every level of the stack compiles.
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-overrides branch from c227170 to 43c724e Compare August 31, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant