Fix the flaky overload check in bin/conformance and the devcell tests - #68
Merged
Merged
Conversation
flavorjones
changed the title
flavorjones
force-pushed
the
overload-flake-547
branch
2 times, most recently
from
September 28, 2026 19:16
e52abbb to
2f3c0b0
Compare
flavorjones
changed the title
The battery is the list of checks that `bin/conformance` and `rake test:devcell` run against a live cell. Its overload check fills every worker and queue place with sleep requests, then expects another request to be refused with `capacity`. It failed intermittently against a healthy cell: the worker that served the previous check was still cleaning up and held its place, so the cell refused a sleep request and never filled. Wait for the cell to report nothing running or queued before filling it. [Fix #66]
flavorjones
force-pushed
the
overload-flake-547
branch
from
September 28, 2026 19:30
3588938 to
7257288
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The battery (
examples/lib/battery.rb) is the list of checks thatrake test:devcellandbin/conformancerun against a live cell. Its overload check fills every worker and every queue place with 2-second sleep requests, then sends another request and expects the cell to refuse it withcapacity.On ruby head, this check failed intermittently in CI. The cell refused some of the sleep requests as well, so it never filled, and the check gave up. An operator running
bin/conformanceagainst a healthy image could see the same failure.The supervisor counts a worker as busy until the worker reports idle. The worker reports idle only after it has answered its caller and cleaned up (
worker.rb). In the failing runs, the worker that served the previous check was still cleaning up when the overload check started. The cell refused a sleep request because that place was taken. The place freed a moment later, and the cell stayed a place short of full.Fixes #66
Details
examples/gateruns the overload check against a fake cell that holds a leftover request from an earlier check. Without the wait, the check fails the same way it did in CI.bin/conformanceand the devcell tests are affected. The cell's behavior is unchanged.