Repository navigation
feat(server): drain connections on SIGTERM before exiting - #131
Merged
Merged
Conversation
The server installed no signal handler. As PID 1 in its container it ignored SIGTERM, so every stop waited out Docker's 10 s timeout and was then killed mid-request, leaving shared terminal sessions marked live in the database with their key bytes still stored. On SIGTERM or Ctrl-C the server now stops accepting connections, lets in-flight requests finish, ends open sync streams (clients already reconnect on their own), and tells every shared-terminal socket the session ended with reason "server_restart" before closing it with 1012. The host's normal disconnect cleanup then marks the session ended. Upgraded sockets are tracked separately because hyper stops tracking a connection once it upgrades. Whatever is still open after SHUTDOWN_GRACE_SECS (default 8) is dropped and the process exits. Refs VoltiusApp/voltius#800
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #133. Useful on a single instance today; also a prerequisite for rolling deploys later.
Problem
No signal handler was installed. As PID 1 in the container the server ignored SIGTERM, so every
docker stop/ recreate waited the full 10 s and then SIGKILLed it:ended_at IS NULLand itssession_key_bytes, because the host's disconnect cleanup never ranChange
On SIGTERM or Ctrl-C:
with_graceful_shutdown)./v1/sync/streamresponses end. The client's SSE loop already reconnects with backoff.{"type":"session_ended","reason":"server_restart"}and a 1012 close. Current clients handlesession_endedalready. The host's socket then runs its normal cleanup, so the session is marked ended in the DB and its key bytes are cleared.TaskTracker, because hyper stops tracking a connection once it upgrades.SHUTDOWN_GRACE_SECS(default 8, below Docker's 10 s) is dropped and the process exits.The "send end frame, then close" sequence is now one helper,
end_socket, shared with the access-revoked path.This doesn't keep anything alive across a restart. Shared sessions still end on every redeploy; they now end cleanly.
Tests
shutdown::tests: waits for tracked sockets, gives up after the grace period,stopping()resolves on begindraining_ends_the_session_for_everyone_attached: host and guest both get theserver_restartframe and a 1012 close, and the session row is ended-D warningsclean.