Skip to content

alts: talk to the handshaker service without blocking the worker - #48680

Open
NetanelPersikGoogle wants to merge 4 commits into
envoyproxy:mainfrom
NetanelPersikGoogle:alts-async-handshaker
Open

NetanelPersikGoogle wants to merge 4 commits into
envoyproxy:mainfrom
NetanelPersikGoogle:alts-async-handshaker

Conversation

@NetanelPersikGoogle

@NetanelPersikGoogle NetanelPersikGoogle commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Commit Message: alts: talk to the handshaker service without blocking the worker

Additional Description:

Problem. The ALTS transport socket talks to the ALTS handshaker service through AltsProxy, which makes blocking gRPC calls (Write()/Read() on a synchronous bidi stream) on the worker thread, with a 30 s deadline. While a call waits, the worker serves nothing else: no other handshakes, no requests on established connections, no timers. A slow, overloaded or unreachable handshaker service therefore stalls every connection on that worker.

This was raised when the transport socket moved to public gRPC APIs in #29803:

  • @KBaichoo noted that it "can easily bring the entire proxy to a halt if the grpc service is unavailable / experiencing high latency" (comment).
  • @botengyao suggested an async stream with a response callback, like Envoy's async gRPC client, and called it the "top priority TODO" (1, 2).

It merged as is because the extension had no real users yet, which left TODO(matthewstevenson88): Investigate using Envoy's async gRPC client in alts_proxy.cc. This PR resolves that TODO for server-side handshakes.

We hit this in production. An Envoy-based proxy on GKE terminates ALTS and uses the GKE metadata server as its handshaker service. Held handshaker calls used up gRPC's per-connection call limit (gRFC A105; fixed on the Envoy side in #48487). After that, every new handshake call waited out the full 30 s deadline:

  • Each of the proxy's 32 workers sat in back-to-back 30 s waits: about 64 handshake failures per minute and no successful handshakes.
  • Peers kept reconnecting, and the proxy piled up more than 8,000 open connections. In a reproduction, requests in flight on established connections hung too.
  • It did not recover until it was restarted.

#48487 removed that trigger, but the failure mode is generic: anything that slows down the handshaker service freezes the workers the same way. With async calls, those handshakes would still have failed, but the workers would have kept serving their established connections.

Change.

  • New AltsAsyncProxy drives the same grpc.gcp.HandshakerService/DoHandshake stream through Envoy's async gRPC client (Grpc::AsyncClient<HandshakerReq, HandshakerResp>).

    • At most one request is outstanding. Its callback runs exactly once, on the worker's dispatcher, with the response or an error.
    • The stream keeps the existing 30 s deadline (AltsClientContextDeadline, moved to alts_proxy.h).
    • Destroying the proxy drops the pending callback, half-closes the stream and leaves it to waitForRemoteCloseAndDelete(), so no callback runs afterwards.
  • AltsProxy now shares its request builders and response check with AltsAsyncProxy. Its behavior is unchanged.

  • AltsTsiHandshaker::createAsyncForServer() (and createAsyncForClient()) create a handshaker whose next() returns right away; the result arrives later through the same OnNextDone callback. The handshaker copies the received bytes, because TsiHandshaker drains the caller's buffer as soon as next() returns and the handshake result needs those bytes to compute the unused bytes.

  • TsiHandshaker::deferredDelete() now cancels a call that is still waiting for the handshaker service. A connection that closes mid-handshake frees its handshaker and stream right away, instead of after the response or the 30 s deadline. A result that was already posted is handled as before (delete_on_done_).

  • TsiSocket already handled a pending next() (handshaker_next_calling_), and TsiHandshaker already delivered results through dispatcher.post(). What is new is that the socket can be read while a call is pending: with blocking calls, the posted result ran in the same event-loop iteration (scheduleCallbackCurrentIteration()), before any new read. TsiSocket keeps reading during a call, so it notices when a peer resets the connection or, without half-close, closes it, and the call is cancelled. It now also handles what it reads in the meantime:

    • The bytes read during a call follow the bytes that the handshake leaves unused, so prepending those keeps the order. doHandshakeNextDone() no longer asserts that raw_read_buffer_ is empty there.
    • It stops reading once default_max_frame_size_ (16 KiB) or more is buffered, so that a peer can't make it buffer without bound, and it resumes when the call is done.
    • With half-close, an end of stream that arrives on its own while a call waits no longer fails the handshake: the call may still complete it, and the data and the end of stream are then delivered. Without half-close, the connection is still closed right away, which cancels the call.
  • Bug fix for all ALTS handshakes, blocking ones included. An end of stream that arrived together with the peer's last handshake message was dropped: the handshake result is posted, so it is processed after doRead() returns, and nothing asked for another read to report the end of stream. doHandshakeNextDone() now also asks for a read when end_stream_read_ is set. Release note: changelogs/current/bug_fixes/alts__end-of-stream-with-last-handshake-message.rst.

  • For downstream sockets, the config builds a google_grpc GrpcService for handshaker_service:

    • Target URI is the address and the stat prefix is alts_handshaker.
    • Channel args are those of AltsChannelPool (local subchannel pool, plus the optional keepalive env var), with no credentials, as before.

    When the factory is created, it checks once that Envoy's async gRPC client can serve this config (that needs Google gRPC); otherwise it logs a warning and keeps the blocking path. Each worker then takes its client from the cluster manager's async client cache (getOrCreateRawAsyncClientWithHashKey()). If that unexpectedly fails, the handshake falls back to the blocking path.

  • Runtime guard envoy.reloadable_features.alts_async_handshaker, true by default. Setting it to false restores the blocking path (AltsChannelPool + AltsProxy), which is unchanged apart from the end-of-stream fix.

Why only server-side (downstream) handshakes for now. Upstream connections live in the cluster manager's thread-local connection pools.

  • When a worker shuts down, its thread-local slots are destroyed in reverse allocation order. The async gRPC client manager's thread-local state (GoogleAsyncClientThreadLocal) is allocated after the cluster manager's own slot, so it goes first. It resets every Google gRPC stream and puts it on the deferred-delete list without callbacks.
  • Only then are the connection pools destroyed, and each pool's destructAllConnections() clears the deferred-delete list.
  • An upstream connection still mid-handshake at that point would touch a freed stream when it closes.

Downstream connections are closed earlier (handler_.reset() in WorkerImpl::threadRoutine), so they are safe. Making upstream handshakes async needs a way to detach pending handshakes before that teardown; it's left as a TODO for a follow-up. The production case above is server-side.

Worth a reviewer's look.

  • Concurrency. The blocking design capped in-flight handshaker calls at one per worker. A worker can now have many in flight, so the handshaker service sees the real handshake concurrency during bursts.
    • Listener connection limits and the overload manager still apply.
    • Each worker has its own channel (local subchannel pool), so a per-connection stream limit on the handshaker service (100 on the GKE metadata server) applies per worker.
    • With alts: release the handshaker once the handshake completes #48487, a stream lives only as long as its handshake.
  • Reading while a call waits. TsiSocket keeps reading during a call, rather than leaving the peer's bytes in the kernel until the call returns. That way it notices peers that give up and cancels their calls right away, which matters most when the handshaker service is slow. The cap is checked before each read, so a peer can make it buffer at most about 16 KiB plus one read in the meantime.
  • The end-of-stream fix is not runtime guarded. It only changes connections whose peer sends its end of stream with its last handshake message, which TsiSocket used to drop. Happy to put it behind a runtime guard if you prefer.
  • Connections to the handshaker service. There is one channel per worker instead of the shared pool of 10.
    • The cache evicts a worker's client after max_cached_entry_idle_duration (default 50 s) without new handshakes on that worker; handshakes in progress keep their client alive.
    • AltsChannelPool is still created for upstream handshakes and the fallback; its channels connect lazily.
  • Google gRPC rather than Envoy gRPC. handshaker_service is an address, not a cluster, so Envoy gRPC would need a new config field and a cluster. That could come in a follow-up; this PR needs no API change.
  • Threading. Callbacks run on the worker that owns the connection, and destroying the proxy guarantees no later callbacks. This avoids the cross-thread lifetime problems of the earlier gRPC-core async TSI handshaker that alts: Major rewrite of the ALTS transport socket to fix concurrent connection establishment bugs (version 2). #29803 replaced.
  • Observability. The handshaker streams now show up in Envoy's gRPC client stats under grpc.alts_handshaker. (streams_total, and streams_closed_<code>, e.g. streams_closed_4 for deadline exceeded). These stats are shared by all ALTS transport sockets that use the same handshaker service.
  • Deadline. The 30 s deadline and its TODO stay as they are; making it configurable can be a separate change.

Risk Level: Medium. This changes how every server-side ALTS handshake talks to the handshaker service. It is runtime guarded (default on), and the blocking path is kept, unchanged apart from the end-of-stream fix, both behind the guard and as a fallback.

Testing:

  • New unit tests use a fake async handshaker client and stream (fake_async_handshaker_lib):
    • alts_async_proxy_test (14 tests): stream start and deadline, requests and responses, handshaker service errors, stream failures, a second outstanding request, unexpected messages, destruction.

    • AltsTsiHandshakerAsyncTest (9 tests): client and server handshakes, unused bytes after the caller reuses its buffer, cancel, failures.

    • TsiHandshakerAsyncTest (3 tests): the result is posted only when the response arrives; deferredDelete() while waiting cancels and deletes; deferredDelete() after the result is posted waits for it.

    • TsiSocketTest (6 new tests):

      • HandshakeWithEndOfStreamAfterClientFinished: with a blocking handshaker, and posted callbacks run after doRead() returns as on a real dispatcher, an end of stream that arrives with the ClientFinished is reported once the handshake completes.
      • The other five use a server-side handshake that waits for the handshaker service:
        • AsyncHandshakeWithDataReceivedWhileWaiting: the ClientFinished arrives together with the start of the client's application data, and more application data arrives while the server waits. Once the handshake completes, the server reads all of it, in order.
        • AsyncHandshakeStopsReadingWhileWaitingWithFullBuffer: the server stops reading once 16 KiB or more is buffered, and reads the rest, in order, once the handshake completes.
        • AsyncHandshakeWithEndOfStreamAfterClientFinished: an end of stream that arrives with the ClientFinished is reported once the handshake completes.
        • AsyncHandshakeEndOfStreamWhileWaitingCloses: without half-close, an end of stream on its own closes the connection, and closing it cancels the call.
        • AsyncHandshakeEndOfStreamWhileWaitingWithHalfClose: with half-close, the connection stays open, and the data and the end of stream are delivered once the handshake completes.

      Without the TsiSocket changes, AsyncHandshakeWithDataReceivedWhileWaiting aborts on the removed assert (in builds with assertions enabled), and the full-buffer, half-close and both end-of-stream-after-ClientFinished tests fail.

    • config_test (5 new tests), checking the handshakers the config creates:

      • a downstream handshake uses the async client, configured with the right GrpcService;
      • it blocks without Google gRPC, when the client is unavailable, and with the guard off;
      • an upstream handshake always blocks.
  • Integration:
    • Envoy's server-side handshakes in the existing ALTS integration tests now run the async path end to end, using Envoy's Google gRPC client against the fake handshaker service.
    • New AltsIntegrationTestSlowHandshaker.WaitingHandshakeDoesNotBlockWorker runs Envoy with one worker. It holds one connection's handshake at the handshaker service, checks that another connection still gets its request served, then releases the handshake and checks that it completes. With the guard off, it fails ("Timed out waiting for new stream"), so it does catch the blocking behavior.
    • New AltsIntegrationTestBlockingHandshaker covers the guard-off path.
  • All ALTS unit and integration tests pass in Google's internal build of Envoy, where the sources are identical apart from include paths. check_format and the pedantic spelling check pass locally; the local clang-format is not the version upstream pins.

Docs Changes: N/A
Release Notes:

  • changelogs/current/minor_behavior_changes/alts__async-handshaker-service-calls.rst
  • changelogs/current/bug_fixes/alts__end-of-stream-with-last-handshake-message.rst

Platform Specific Features: N/A
Runtime guard: envoy.reloadable_features.alts_async_handshaker

@repokitteh-read-only

Copy link
Copy Markdown

CC @envoyproxy/runtime-guard-changes: FYI only for changes made to (source/common/runtime/runtime_features.cc).

🐱

Caused by: #48680 was opened by NetanelPersikGoogle.

see: more, trace.

AltsProxy makes blocking gRPC calls to the ALTS handshaker service on the
worker thread, with a 30 second deadline. While a call waits, the worker
serves nothing else, so a slow or unreachable handshaker service stalls every
connection on that worker. This was a known limitation when the transport
socket moved to public gRPC APIs (envoyproxy#29803), recorded as a TODO in
alts_proxy.cc.

Add AltsAsyncProxy, which drives the same DoHandshake stream through Envoy's
async gRPC client, and use it for server-side (downstream) handshakes:
AltsTsiHandshaker::createAsyncForServer() creates a handshaker whose next()
returns right away, and the response is delivered later on the worker's
dispatcher, where TsiHandshaker already posts its results. If the connection
closes while a call waits, TsiHandshaker::deferredDelete() cancels the call,
which half-closes the stream and stops its callbacks.

TsiSocket keeps reading from the network while a call waits, so it notices
when a peer resets the connection or, without half-close, closes it, and the
call is cancelled. The bytes that it reads meanwhile follow the bytes that the
handshake leaves unused, so it no longer asserts that its read buffer is empty
when it prepends those. It stops reading once default_max_frame_size_ bytes
are buffered, and resumes when the call is done. With half-close, an end of
stream that arrives on its own while a call waits no longer fails the
handshake, since the call may still complete it.

This also fixes a bug that affects all ALTS handshakes, blocking ones
included: an end of stream that arrived together with the peer's last
handshake message was dropped, because the handshake result is processed
after doRead() returns and nothing asked for another read. TsiSocket now asks
for a read when a handshake step completes after it has read an end of stream.

The ALTS config builds a google_grpc GrpcService for handshaker_service, with
the channel args of AltsChannelPool, and gets each worker's client from the
cluster manager's async client cache. If Envoy's async gRPC client is not
available (for example, when Envoy is built without Google gRPC), handshakes
keep making blocking calls.

Client-side (upstream) handshakes still make blocking calls for now: upstream
connections live in the cluster manager's thread-local connection pools, which
a worker destroys after the async gRPC client's thread-local state, so a
pending upstream handshake could outlive its stream.

The new behavior is on by default, behind the runtime guard
envoy.reloadable_features.alts_async_handshaker.

Signed-off-by: Netanel Persik <netanelpersik@google.com>
Signed-off-by: Netanel Persik <netanelpersik@google.com>
Signed-off-by: Netanel Persik <netanelpersik@google.com>
@NetanelPersikGoogle

Copy link
Copy Markdown
Contributor Author

/retest

@RyanTheOptimist

Copy link
Copy Markdown
Contributor

@botengyao friendly maintainer on-call ping

@botengyao botengyao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is very nice, thanks for addressing this, LGTM module some questions

/wait

}

void TsiHandshaker::deferredDelete() {
if (calling_ && !result_posted_) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can it be limited to only calling_?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. result_posted_ separates "still waiting for the handshaker service" (safe to cancel: nothing refers to the handshaker yet) from "the result is already posted" (a queued closure holds a raw pointer to the handshaker, so deleting now would be a use-after-free, and it would trip ASSERT(calling_)?.

Blocking calls are always in the second state, because they post before next() returns, so testing only calling_ would break the blocking path too - DeferredDeleteAfterResultIsPostedWaitsForIt covers it

@NetanelPersikGoogle

Copy link
Copy Markdown
Contributor Author

cc @matthewstevenson88 @rockspore

Signed-off-by: Netanel Persik <netanelpersik@google.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants