Repository navigation
Prioritize sticky workflow polls during autoscaling - #3208
Quinn-With-Two-Ns wants to merge 3 commits into
Conversation
Balance sticky and normal admission using backlog, the sticky scaling\ntarget, and slot capacity instead of requiring both pools to have polls.\nWake admission waiters on changes and add regression coverage.
There was a problem hiding this comment.
🟡 Changes recommended
The changelog inaccurately claims unconditional Core parity, omitting that Java still applies backlog and target checks when no maximum capacity is advertised, unlike Core.
1 open issue
What changed in this PR
This PR addresses #3205 by adding workflow-specific admission balancing to the Java SDK’s autoscaling pollers, prioritizing sticky backlog while preserving independent scaling targets.
Changes:
- Replace generic queue balancing with backlog-, target-, and capacity-aware workflow admission.
- Connect admission checks to poll lifecycle, target updates, and sticky draining.
- Add regression tests, public behavior documentation, and release notes.
| File | Description |
|---|---|
temporal-sdk/src/test/java/io/temporal/internal/worker/WorkflowPollerBalancerTest.java |
Tests admission rules, capacity limits, and waiter notifications. |
temporal-sdk/src/test/java/io/temporal/internal/worker/AsyncWorkflowPollerBalancingTest.java |
Tests backlog updates and sticky slot reservations. |
temporal-sdk/src/main/java/io/temporal/worker/tuning/PollerBehaviorAutoscaling.java |
Documents workflow admission behavior. |
temporal-sdk/src/main/java/io/temporal/internal/worker/WorkflowWorker.java |
Installs the shared workflow balancer. |
temporal-sdk/src/main/java/io/temporal/internal/worker/WorkflowPollerBalancer.java |
Implements workflow-specific admission rules. |
temporal-sdk/src/main/java/io/temporal/internal/worker/AsyncWorkflowPollTask.java |
Reports backlog, poll lifecycle, targets, and drain state. |
temporal-sdk/src/main/java/io/temporal/internal/worker/AsyncPoller.java |
Replaces generic balancing with task-specific admission hooks. |
CHANGELOG.md |
Adds release notes for autoscaling workflow balancing. |
Review effort: Balanced
|
|
||
| ### Fixed | ||
| - Autoscaling workflow pollers now prioritize meaningful sticky backlog up to the sticky pool's | ||
| target and use slot capacity to balance sticky and non-sticky admission, matching Core's policy. |
There was a problem hiding this comment.
Hmm, looks like all three SDKs differ here when the slot supplier reports no maximum. Go blocks additional polls of one queue type until the other has an active poll. Rust uses the workflow cache size as a fallback bound and preserves room for the other type’s first poll. Java treats capacity as effectively unlimited, so it doesn’t preserve that room. Can Java ensure there's always room for the other type's first poll?
There was a problem hiding this comment.
@yuandrew Addressed in 7be3028. When the slot supplier advertises no maximum, additional polls of one queue type now wait while the other type has no active poll. Once both have an active poll, backlog and sticky-target rules continue to govern admission without imposing an arbitrary pool-size bound. This uses the conservative Go-style fallback rather than a cache-size fallback, since Java cache size does not constrain slot acquisition. The changelog and API docs now describe this explicitly instead of claiming unconditional Core parity.
Added a poller-level regression using a supplier with two real slots but no advertised maximum, exercising both startup orders. It verifies the first queue cannot consume both slots before the other starts. Temporarily removing the fallback caused that test to fail as expected. There is also a recovery regression showing sticky grows from one to three polls as two normal polls release slots after sticky backlog is reported.
All 217 selected tests pass, including internal worker, sticky-drain shutdown, heartbeat-target, and autoscaling-startup coverage. The broader slot-admission race caveat still applies: active poll counts do not include executing tasks or pending reservations, so this is not a strict guarantee of an outstanding poll of each type at all times.
yuandrew
left a comment
There was a problem hiding this comment.
Should we add a regression test for the scenario from the incident?
Have normal polls occupy most of a small shared slot pool, introduce sticky backlog with room below the sticky target, then complete normal polls and verify sticky polling grows instead of normal polls reclaiming the released slots. The current tests cover the admission rules, but don't exercise that recovery path.
Wait for the other workflow queue's first poll when its slot capacity is unknown, rather than treating the limit as effectively unbounded. Cover both startup orders with an unadvertised two-slot supplier and add the incident recovery regression. Document the fallback explicitly.

What changed?
Why?
Fixes #3205. The previous policy neither prioritized sticky backlog nor considered the sticky scaling target or available slot capacity. This aligns the known-capacity admission rules with Core without introducing a fixed sticky/non-sticky ratio or changing independent scaling targets. With no advertised maximum, Java uses a conservative first-poll fallback rather than Core's workflow-cache capacity bound, because cache size does not constrain Java slot acquisition. Active counts still exclude executing tasks and pending reservations, so this policy does not guarantee an outstanding poll of each type through every slot-admission race.
Validation
Passed formatting:
Passed 217 tests across 38 classes, with no failures or skips:
Both new poller-level regressions were also checked by temporarily disabling their corresponding admission guards and observing the expected failures; all production guards were restored.
git diff --checkpassed. The full repository test suite andclean buildwere not run.Breaking changes?
None. No public API signatures changed. Activity, Nexus, and fixed-size workflow polling retain their existing policies.
Note
Medium Risk
Changes poll admission and slot sharing between sticky and normal workflow queues in autoscaling workers—core worker throughput behavior with race-sensitive synchronization, but well-tested and no public API changes.
Overview
Autoscaling workflow workers replace the generic
PollQueueBalancerinAsyncPollerwith a workflow-specificWorkflowPollerBalancerwired through newPollTaskAsynchooks (awaitPollAdmission, poll lifecycle, sticky target updates). Admission runs before slot reservation and favors sticky polling when backlog is meaningful and the sticky pool is below its scaling target, while still allowing each queue’s first poll and honoring advertised slot limits.AsyncWorkflowPollTaskfeeds sticky backlog hints into the balancer and participates in shutdown when normal polling is disabled.PollerBehaviorAutoscalingJavaDoc and the changelog document the behavior, including the conservative rule when no maximum slot count is advertised (unlike Core, workflow cache size is not used as a fallback). Regression coverage is added inWorkflowPollerBalancerTestandAsyncWorkflowPollerBalancingTest.Reviewed by Cursor Bugbot for commit 778ee31. Bugbot is set up for automated code reviews on this repo. Configure here.