Repository navigation
alts: talk to the handshaker service without blocking the worker - #48680
NetanelPersikGoogle wants to merge 4 commits into
Conversation
AltsProxy makes blocking gRPC calls to the ALTS handshaker service on the worker thread, with a 30 second deadline. While a call waits, the worker serves nothing else, so a slow or unreachable handshaker service stalls every connection on that worker. This was a known limitation when the transport socket moved to public gRPC APIs (envoyproxy#29803), recorded as a TODO in alts_proxy.cc. Add AltsAsyncProxy, which drives the same DoHandshake stream through Envoy's async gRPC client, and use it for server-side (downstream) handshakes: AltsTsiHandshaker::createAsyncForServer() creates a handshaker whose next() returns right away, and the response is delivered later on the worker's dispatcher, where TsiHandshaker already posts its results. If the connection closes while a call waits, TsiHandshaker::deferredDelete() cancels the call, which half-closes the stream and stops its callbacks. TsiSocket keeps reading from the network while a call waits, so it notices when a peer resets the connection or, without half-close, closes it, and the call is cancelled. The bytes that it reads meanwhile follow the bytes that the handshake leaves unused, so it no longer asserts that its read buffer is empty when it prepends those. It stops reading once default_max_frame_size_ bytes are buffered, and resumes when the call is done. With half-close, an end of stream that arrives on its own while a call waits no longer fails the handshake, since the call may still complete it. This also fixes a bug that affects all ALTS handshakes, blocking ones included: an end of stream that arrived together with the peer's last handshake message was dropped, because the handshake result is processed after doRead() returns and nothing asked for another read. TsiSocket now asks for a read when a handshake step completes after it has read an end of stream. The ALTS config builds a google_grpc GrpcService for handshaker_service, with the channel args of AltsChannelPool, and gets each worker's client from the cluster manager's async client cache. If Envoy's async gRPC client is not available (for example, when Envoy is built without Google gRPC), handshakes keep making blocking calls. Client-side (upstream) handshakes still make blocking calls for now: upstream connections live in the cluster manager's thread-local connection pools, which a worker destroys after the async gRPC client's thread-local state, so a pending upstream handshake could outlive its stream. The new behavior is on by default, behind the runtime guard envoy.reloadable_features.alts_async_handshaker. Signed-off-by: Netanel Persik <netanelpersik@google.com>
1b741ab to
4a11f37
Compare
Signed-off-by: Netanel Persik <netanelpersik@google.com>
Signed-off-by: Netanel Persik <netanelpersik@google.com>
|
/retest |
|
@botengyao friendly maintainer on-call ping |
botengyao
left a comment
There was a problem hiding this comment.
this is very nice, thanks for addressing this, LGTM module some questions
/wait
| } | ||
|
|
||
| void TsiHandshaker::deferredDelete() { | ||
| if (calling_ && !result_posted_) { |
There was a problem hiding this comment.
can it be limited to only calling_?
There was a problem hiding this comment.
I don't think so. result_posted_ separates "still waiting for the handshaker service" (safe to cancel: nothing refers to the handshaker yet) from "the result is already posted" (a queued closure holds a raw pointer to the handshaker, so deleting now would be a use-after-free, and it would trip ASSERT(calling_)?.
Blocking calls are always in the second state, because they post before next() returns, so testing only calling_ would break the blocking path too - DeferredDeleteAfterResultIsPostedWaitsForIt covers it
Signed-off-by: Netanel Persik <netanelpersik@google.com>
Commit Message: alts: talk to the handshaker service without blocking the worker
Additional Description:
Problem. The ALTS transport socket talks to the ALTS handshaker service through
AltsProxy, which makes blocking gRPC calls (Write()/Read()on a synchronous bidi stream) on the worker thread, with a 30 s deadline. While a call waits, the worker serves nothing else: no other handshakes, no requests on established connections, no timers. A slow, overloaded or unreachable handshaker service therefore stalls every connection on that worker.This was raised when the transport socket moved to public gRPC APIs in #29803:
It merged as is because the extension had no real users yet, which left
TODO(matthewstevenson88): Investigate using Envoy's async gRPC clientinalts_proxy.cc. This PR resolves that TODO for server-side handshakes.We hit this in production. An Envoy-based proxy on GKE terminates ALTS and uses the GKE metadata server as its handshaker service. Held handshaker calls used up gRPC's per-connection call limit (gRFC A105; fixed on the Envoy side in #48487). After that, every new handshake call waited out the full 30 s deadline:
#48487 removed that trigger, but the failure mode is generic: anything that slows down the handshaker service freezes the workers the same way. With async calls, those handshakes would still have failed, but the workers would have kept serving their established connections.
Change.
New
AltsAsyncProxydrives the samegrpc.gcp.HandshakerService/DoHandshakestream through Envoy's async gRPC client (Grpc::AsyncClient<HandshakerReq, HandshakerResp>).AltsClientContextDeadline, moved toalts_proxy.h).waitForRemoteCloseAndDelete(), so no callback runs afterwards.AltsProxynow shares its request builders and response check withAltsAsyncProxy. Its behavior is unchanged.AltsTsiHandshaker::createAsyncForServer()(andcreateAsyncForClient()) create a handshaker whosenext()returns right away; the result arrives later through the sameOnNextDonecallback. The handshaker copies the received bytes, becauseTsiHandshakerdrains the caller's buffer as soon asnext()returns and the handshake result needs those bytes to compute the unused bytes.TsiHandshaker::deferredDelete()now cancels a call that is still waiting for the handshaker service. A connection that closes mid-handshake frees its handshaker and stream right away, instead of after the response or the 30 s deadline. A result that was already posted is handled as before (delete_on_done_).TsiSocketalready handled a pendingnext()(handshaker_next_calling_), andTsiHandshakeralready delivered results throughdispatcher.post(). What is new is that the socket can be read while a call is pending: with blocking calls, the posted result ran in the same event-loop iteration (scheduleCallbackCurrentIteration()), before any new read.TsiSocketkeeps reading during a call, so it notices when a peer resets the connection or, without half-close, closes it, and the call is cancelled. It now also handles what it reads in the meantime:doHandshakeNextDone()no longer asserts thatraw_read_buffer_is empty there.default_max_frame_size_(16 KiB) or more is buffered, so that a peer can't make it buffer without bound, and it resumes when the call is done.Bug fix for all ALTS handshakes, blocking ones included. An end of stream that arrived together with the peer's last handshake message was dropped: the handshake result is posted, so it is processed after
doRead()returns, and nothing asked for another read to report the end of stream.doHandshakeNextDone()now also asks for a read whenend_stream_read_is set. Release note:changelogs/current/bug_fixes/alts__end-of-stream-with-last-handshake-message.rst.For downstream sockets, the config builds a
google_grpcGrpcServiceforhandshaker_service:alts_handshaker.AltsChannelPool(local subchannel pool, plus the optional keepalive env var), with no credentials, as before.When the factory is created, it checks once that Envoy's async gRPC client can serve this config (that needs Google gRPC); otherwise it logs a warning and keeps the blocking path. Each worker then takes its client from the cluster manager's async client cache (
getOrCreateRawAsyncClientWithHashKey()). If that unexpectedly fails, the handshake falls back to the blocking path.Runtime guard
envoy.reloadable_features.alts_async_handshaker, true by default. Setting it to false restores the blocking path (AltsChannelPool+AltsProxy), which is unchanged apart from the end-of-stream fix.Why only server-side (downstream) handshakes for now. Upstream connections live in the cluster manager's thread-local connection pools.
GoogleAsyncClientThreadLocal) is allocated after the cluster manager's own slot, so it goes first. It resets every Google gRPC stream and puts it on the deferred-delete list without callbacks.destructAllConnections()clears the deferred-delete list.Downstream connections are closed earlier (
handler_.reset()inWorkerImpl::threadRoutine), so they are safe. Making upstream handshakes async needs a way to detach pending handshakes before that teardown; it's left as a TODO for a follow-up. The production case above is server-side.Worth a reviewer's look.
TsiSocketkeeps reading during a call, rather than leaving the peer's bytes in the kernel until the call returns. That way it notices peers that give up and cancels their calls right away, which matters most when the handshaker service is slow. The cap is checked before each read, so a peer can make it buffer at most about 16 KiB plus one read in the meantime.max_cached_entry_idle_duration(default 50 s) without new handshakes on that worker; handshakes in progress keep their client alive.AltsChannelPoolis still created for upstream handshakes and the fallback; its channels connect lazily.handshaker_serviceis an address, not a cluster, so Envoy gRPC would need a new config field and a cluster. That could come in a follow-up; this PR needs no API change.grpc.alts_handshaker.(streams_total, andstreams_closed_<code>, e.g.streams_closed_4for deadline exceeded). These stats are shared by all ALTS transport sockets that use the same handshaker service.Risk Level: Medium. This changes how every server-side ALTS handshake talks to the handshaker service. It is runtime guarded (default on), and the blocking path is kept, unchanged apart from the end-of-stream fix, both behind the guard and as a fallback.
Testing:
fake_async_handshaker_lib):alts_async_proxy_test(14 tests): stream start and deadline, requests and responses, handshaker service errors, stream failures, a second outstanding request, unexpected messages, destruction.AltsTsiHandshakerAsyncTest(9 tests): client and server handshakes, unused bytes after the caller reuses its buffer, cancel, failures.TsiHandshakerAsyncTest(3 tests): the result is posted only when the response arrives;deferredDelete()while waiting cancels and deletes;deferredDelete()after the result is posted waits for it.TsiSocketTest(6 new tests):HandshakeWithEndOfStreamAfterClientFinished: with a blocking handshaker, and posted callbacks run afterdoRead()returns as on a real dispatcher, an end of stream that arrives with the ClientFinished is reported once the handshake completes.AsyncHandshakeWithDataReceivedWhileWaiting: the ClientFinished arrives together with the start of the client's application data, and more application data arrives while the server waits. Once the handshake completes, the server reads all of it, in order.AsyncHandshakeStopsReadingWhileWaitingWithFullBuffer: the server stops reading once 16 KiB or more is buffered, and reads the rest, in order, once the handshake completes.AsyncHandshakeWithEndOfStreamAfterClientFinished: an end of stream that arrives with the ClientFinished is reported once the handshake completes.AsyncHandshakeEndOfStreamWhileWaitingCloses: without half-close, an end of stream on its own closes the connection, and closing it cancels the call.AsyncHandshakeEndOfStreamWhileWaitingWithHalfClose: with half-close, the connection stays open, and the data and the end of stream are delivered once the handshake completes.Without the
TsiSocketchanges,AsyncHandshakeWithDataReceivedWhileWaitingaborts on the removed assert (in builds with assertions enabled), and the full-buffer, half-close and both end-of-stream-after-ClientFinished tests fail.config_test(5 new tests), checking the handshakers the config creates:GrpcService;AltsIntegrationTestSlowHandshaker.WaitingHandshakeDoesNotBlockWorkerruns Envoy with one worker. It holds one connection's handshake at the handshaker service, checks that another connection still gets its request served, then releases the handshake and checks that it completes. With the guard off, it fails ("Timed out waiting for new stream"), so it does catch the blocking behavior.AltsIntegrationTestBlockingHandshakercovers the guard-off path.check_formatand the pedantic spelling check pass locally; the local clang-format is not the version upstream pins.Docs Changes: N/A
Release Notes:
Platform Specific Features: N/A
Runtime guard: envoy.reloadable_features.alts_async_handshaker