Repository navigation
reverse_tunnel: rebalance new sockets by idle count, not sockets ever accepted - #49410
Open
skpulipaka26 wants to merge 1 commit into
Open
skpulipaka26 wants to merge 1 commit into
skpulipaka26 wants to merge 1 commit into
Conversation
… accepted Signed-off-by: Sai Krishna Pulipaka <skpulipaka@gmail.com>
skpulipaka26
requested review from
agrawroh,
botengyao and
yanavlasov
as code owners
November 18, 2026 19:58
skpulipaka26
requested a deployment
to
external-contributors
November 18, 2026 19:58 — with
GitHub Actions
Waiting
|
Hi @skpulipaka26, welcome and thank you for your contribution. We will try to review your Pull Request as quickly as possible. In the meantime, please take a look at the contribution guidelines if you have not done so already. |
Author
|
@basundhara-c @agrawroh thanks for reviewing #49367. This is its sibling on the accept side: the responder's per-worker socket count only ever went up, so under tunnel churn a worker can end up with no idle socket for a node. Could you take a look when you have a moment? |
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commit Message: reverse_tunnel: rebalance new sockets by idle count, not sockets ever accepted
When the responder accepts a reverse tunnel socket,
UpstreamSocketManager::pickLeastLoadedSocketManagerhands it to the worker with the smallestnode_to_conn_count_map_[node], but that count is only ever incremented, so it counts sockets ever accepted rather than sockets the worker holds. A request can only take an idle socket from its own worker (getConnectionSocket), so as sockets are used or die and the initiator redials, a worker can be left with no idle socket for a node while others hold several, and requests on it fail withfallback_no_reverse_socket. The count now tracks the worker's idle sockets: it is decremented, undersocket_manager_lock, when a socket is popped for a request or an idle socket dies, so a replacement goes back to the worker that lost one. The empty node/cluster ID check moves before the rebalancing pick, so a rejected socket no longer reserves a slot on the target worker.Additional Description: Seen with responders running 16 workers and 64 tunnels per initiator: after tunnel churn the per-worker spread drifted far from even. Related to #49367, which picks a node with an idle socket when routing by cluster ID; both make the responder's choices follow idle sockets.
Risk Level: Low
Testing: Unit test
TestUpstreamSocketManagerRebalancing.RebalanceToWorkerWithFewestIdleSockets: fails on main (worker 0 left with 0 idle sockets and worker 1 with 2), passes with the fix; the rest ofupstream_socket_interfacepasses unchanged. End to end with v1.40.2 initiators and a 16-worker responder, 3 initiator pods of 64 tunnels each, killing a random quarter of each pod's tunnels every 5s for 60 rounds (same seed): the fewest tunnels any (worker, node) pair held after a round was 0 in 6 rounds, 1 in 34 and 2 in 20 on stock Envoy; with the fix 0 in 1 round (the kill itself taking all of one pair's sockets before the redial), 1 in 10, 2 in 13 and 3 in 36.Docs Changes: N/A
Release Notes: changelogs/current/bug_fixes/reverse_tunnel__rebalance-by-idle-sockets.rst
Platform Specific Features: N/A