Skip to content

Prioritize sticky workflow polls during autoscaling - #3208

Open
Quinn-With-Two-Ns wants to merge 3 commits into
mainfrom
fix/autoscaling-sticky-poller-balancing
Open

Quinn-With-Two-Ns wants to merge 3 commits into
mainfrom
fix/autoscaling-sticky-poller-balancing

Conversation

@Quinn-With-Two-Ns

@Quinn-With-Two-Ns Quinn-With-Two-Ns commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

What changed?

  • Replace the autoscaling poller's generic queue balancing with a workflow-specific admission balancer.
  • Prioritize sticky admission when sticky backlog is greater than one, exceeds active sticky polls, and sticky is below its independent scaling target. Preserve first-poll admission for both queue types.
  • Prevent additional sticky slot reservations at the sticky target and use slot capacity to preserve room for the other queue's first poll without unnecessarily blocking spare capacity.
  • Wake admission waiters when backlog, active poll counts, or the sticky target changes, and preserve sticky draining when normal polling is disabled.
  • When a slot supplier does not advertise a maximum, block additional polls of one queue type until the other has an active poll. Once both are active, continue applying backlog and target rules without imposing an arbitrary capacity bound.
  • Add a recovery regression starting with three normal polls and one sticky poll in four shared slots; after sticky backlog is reported, completing two normal polls grows sticky polling to its target without replacement normal polls reclaiming those slots.
  • Test both startup orders with a supplier that has two real slots but does not advertise its maximum, plus unknown-capacity wakeups, rechecks, and sticky draining.
  • Add regression tests, public behavior documentation, and a changelog entry.

Why?

Fixes #3205. The previous policy neither prioritized sticky backlog nor considered the sticky scaling target or available slot capacity. This aligns the known-capacity admission rules with Core without introducing a fixed sticky/non-sticky ratio or changing independent scaling targets. With no advertised maximum, Java uses a conservative first-poll fallback rather than Core's workflow-cache capacity bound, because cache size does not constrain Java slot acquisition. Active counts still exclude executing tasks and pending reservations, so this policy does not guarantee an outstanding poll of each type through every slot-admission race.

Validation

Passed formatting:

./gradlew --offline spotlessApply --console=plain

Passed 217 tests across 38 classes, with no failures or skips:

./gradlew :temporal-sdk:test --offline --console=plain   --tests 'io.temporal.internal.worker.*'   --tests 'io.temporal.worker.shutdown.StickyWorkflowDrainShutdownTest'   --tests 'io.temporal.worker.WorkerHeartbeatPollerTargetTest'   --tests 'io.temporal.worker.WorkerPollerAutoEnrollStartupTest'

Both new poller-level regressions were also checked by temporarily disabling their corresponding admission guards and observing the expected failures; all production guards were restored.

git diff --check passed. The full repository test suite and clean build were not run.

Breaking changes?

None. No public API signatures changed. Activity, Nexus, and fixed-size workflow polling retain their existing policies.


Note

Medium Risk
Changes poll admission and slot sharing between sticky and normal workflow queues in autoscaling workers—core worker throughput behavior with race-sensitive synchronization, but well-tested and no public API changes.

Overview
Autoscaling workflow workers replace the generic PollQueueBalancer in AsyncPoller with a workflow-specific WorkflowPollerBalancer wired through new PollTaskAsync hooks (awaitPollAdmission, poll lifecycle, sticky target updates). Admission runs before slot reservation and favors sticky polling when backlog is meaningful and the sticky pool is below its scaling target, while still allowing each queue’s first poll and honoring advertised slot limits.

AsyncWorkflowPollTask feeds sticky backlog hints into the balancer and participates in shutdown when normal polling is disabled. PollerBehaviorAutoscaling JavaDoc and the changelog document the behavior, including the conservative rule when no maximum slot count is advertised (unlike Core, workflow cache size is not used as a fallback). Regression coverage is added in WorkflowPollerBalancerTest and AsyncWorkflowPollerBalancingTest.

Reviewed by Cursor Bugbot for commit 778ee31. Bugbot is set up for automated code reviews on this repo. Configure here.

Balance sticky and normal admission using backlog, the sticky scaling\ntarget, and slot capacity instead of requiring both pools to have polls.\nWake admission waiters on changes and add regression coverage.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The changelog inaccurately claims unconditional Core parity, omitting that Java still applies backlog and target checks when no maximum capacity is advertised, unlike Core.

1 open issue
What changed in this PR

This PR addresses #3205 by adding workflow-specific admission balancing to the Java SDK’s autoscaling pollers, prioritizing sticky backlog while preserving independent scaling targets.

Changes:

  • Replace generic queue balancing with backlog-, target-, and capacity-aware workflow admission.
  • Connect admission checks to poll lifecycle, target updates, and sticky draining.
  • Add regression tests, public behavior documentation, and release notes.
File Description
temporal-sdk/​src/​test/​java/​io/​temporal/​internal/​worker/​WorkflowPollerBalancerTest.java Tests admission rules, capacity limits, and waiter notifications.
temporal-sdk/​src/​test/​java/​io/​temporal/​internal/​worker/​AsyncWorkflowPollerBalancingTest.java Tests backlog updates and sticky slot reservations.
temporal-sdk/​src/​main/​java/​io/​temporal/​worker/​tuning/​PollerBehaviorAutoscaling.java Documents workflow admission behavior.
temporal-sdk/​src/​main/​java/​io/​temporal/​internal/​worker/​WorkflowWorker.java Installs the shared workflow balancer.
temporal-sdk/​src/​main/​java/​io/​temporal/​internal/​worker/​WorkflowPollerBalancer.java Implements workflow-specific admission rules.
temporal-sdk/​src/​main/​java/​io/​temporal/​internal/​worker/​AsyncWorkflowPollTask.java Reports backlog, poll lifecycle, targets, and drain state.
temporal-sdk/​src/​main/​java/​io/​temporal/​internal/​worker/​AsyncPoller.java Replaces generic balancing with task-specific admission hooks.
CHANGELOG.md Adds release notes for autoscaling workflow balancing.

Review effort: Balanced

Comment thread CHANGELOG.md Outdated

### Fixed
- Autoscaling workflow pollers now prioritize meaningful sticky backlog up to the sticky pool's
target and use slot capacity to balance sticky and non-sticky admission, matching Core's policy.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, looks like all three SDKs differ here when the slot supplier reports no maximum. Go blocks additional polls of one queue type until the other has an active poll. Rust uses the workflow cache size as a fallback bound and preserves room for the other type’s first poll. Java treats capacity as effectively unlimited, so it doesn’t preserve that room. Can Java ensure there's always room for the other type's first poll?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yuandrew Addressed in 7be3028. When the slot supplier advertises no maximum, additional polls of one queue type now wait while the other type has no active poll. Once both have an active poll, backlog and sticky-target rules continue to govern admission without imposing an arbitrary pool-size bound. This uses the conservative Go-style fallback rather than a cache-size fallback, since Java cache size does not constrain slot acquisition. The changelog and API docs now describe this explicitly instead of claiming unconditional Core parity.

Added a poller-level regression using a supplier with two real slots but no advertised maximum, exercising both startup orders. It verifies the first queue cannot consume both slots before the other starts. Temporarily removing the fallback caused that test to fail as expected. There is also a recovery regression showing sticky grows from one to three polls as two normal polls release slots after sticky backlog is reported.

All 217 selected tests pass, including internal worker, sticky-drain shutdown, heartbeat-target, and autoscaling-startup coverage. The broader slot-admission race caveat still applies: active poll counts do not include executing tasks or pending reservations, so this is not a strict guarantee of an outstanding poll of each type at all times.

@Quinn-With-Two-Ns
Quinn-With-Two-Ns marked this pull request as ready for review November 16, 2026 05:27
@Quinn-With-Two-Ns
Quinn-With-Two-Ns requested a review from a team as a code owner November 16, 2026 05:27
@yuandrew yuandrew self-assigned this Nov 17, 2026

@yuandrew yuandrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a regression test for the scenario from the incident?

Have normal polls occupy most of a small shared slot pool, introduce sticky backlog with room below the sticky target, then complete normal polls and verify sticky polling grows instead of normal polls reclaiming the released slots. The current tests cover the admission rules, but don't exercise that recovery path.

Wait for the other workflow queue's first poll when its slot capacity
is unknown, rather than treating the limit as effectively unbounded.

Cover both startup orders with an unadvertised two-slot supplier and
add the incident recovery regression. Document the fallback explicitly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Align autoscaling workflow poller balancing with Core

3 participants