Skip to content

reverse_tunnel: rebalance new sockets by idle count, not sockets ever accepted - #49410

Open
skpulipaka26 wants to merge 1 commit into
envoyproxy:mainfrom
skpulipaka26:reverse-tunnel-rebalance-by-idle-sockets
Open

skpulipaka26 wants to merge 1 commit into
envoyproxy:mainfrom
skpulipaka26:reverse-tunnel-rebalance-by-idle-sockets

Conversation

@skpulipaka26

Copy link
Copy Markdown

Commit Message: reverse_tunnel: rebalance new sockets by idle count, not sockets ever accepted

When the responder accepts a reverse tunnel socket, UpstreamSocketManager::pickLeastLoadedSocketManager hands it to the worker with the smallest node_to_conn_count_map_[node], but that count is only ever incremented, so it counts sockets ever accepted rather than sockets the worker holds. A request can only take an idle socket from its own worker (getConnectionSocket), so as sockets are used or die and the initiator redials, a worker can be left with no idle socket for a node while others hold several, and requests on it fail with fallback_no_reverse_socket. The count now tracks the worker's idle sockets: it is decremented, under socket_manager_lock, when a socket is popped for a request or an idle socket dies, so a replacement goes back to the worker that lost one. The empty node/cluster ID check moves before the rebalancing pick, so a rejected socket no longer reserves a slot on the target worker.

Additional Description: Seen with responders running 16 workers and 64 tunnels per initiator: after tunnel churn the per-worker spread drifted far from even. Related to #49367, which picks a node with an idle socket when routing by cluster ID; both make the responder's choices follow idle sockets.

Risk Level: Low

Testing: Unit test TestUpstreamSocketManagerRebalancing.RebalanceToWorkerWithFewestIdleSockets: fails on main (worker 0 left with 0 idle sockets and worker 1 with 2), passes with the fix; the rest of upstream_socket_interface passes unchanged. End to end with v1.40.2 initiators and a 16-worker responder, 3 initiator pods of 64 tunnels each, killing a random quarter of each pod's tunnels every 5s for 60 rounds (same seed): the fewest tunnels any (worker, node) pair held after a round was 0 in 6 rounds, 1 in 34 and 2 in 20 on stock Envoy; with the fix 0 in 1 round (the kill itself taking all of one pair's sockets before the redial), 1 in 10, 2 in 13 and 3 in 36.

Docs Changes: N/A
Release Notes: changelogs/current/bug_fixes/reverse_tunnel__rebalance-by-idle-sockets.rst
Platform Specific Features: N/A

… accepted

Signed-off-by: Sai Krishna Pulipaka <skpulipaka@gmail.com>
@repokitteh-read-only

Copy link
Copy Markdown

Hi @skpulipaka26, welcome and thank you for your contribution.

We will try to review your Pull Request as quickly as possible.

In the meantime, please take a look at the contribution guidelines if you have not done so already.

🐱

Caused by: #49410 was opened by skpulipaka26.

see: more, trace.

@skpulipaka26

Copy link
Copy Markdown
Author

@basundhara-c @agrawroh thanks for reviewing #49367. This is its sibling on the accept side: the responder's per-worker socket count only ever went up, so under tunnel churn a worker can end up with no idle socket for a node. Could you take a look when you have a moment?

This branch is waiting to be deployed

1 waiting deployment
external-contributors — 236e1e2b Waiting Nov 18, 2026 by skpulipaka26 via authorize #62776
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant