fix(evaluation): reject parallelism=0 in EvaluateConfig/InferenceConfig - #7000
Open
gaurav-gandhi-2411 wants to merge 1 commit into
Open
fix(evaluation): reject parallelism=0 in EvaluateConfig/InferenceConfig#7000gaurav-gandhi-2411 wants to merge 1 commit into
gaurav-gandhi-2411 wants to merge 1 commit into
Conversation
parallelism is forwarded unclamped to asyncio.Semaphore(value=parallelism) (local_eval_service.py:186,216). A negative value fails loud (Semaphore itself raises ValueError); 0 fails silent -- the semaphore constructs fine and every subsequent .acquire() hangs forever, since Semaphore(0) can never be released. Add ge=1, matching JudgeModelOptions.parallelism_limit, ToolThreadPoolConfig.max_workers, and ContextCacheConfig.cache_intervals, which already carry the same constraint. Gap first raised on google#6951. Claude-Session: https://claude.ai/code/session_019SUgxvPkwsx4AasNXyU8V2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related
#6951 is where this gap was publicly established: a repo-wide sweep for
floor-less count/limit parameters in
src/google/adk/evaluation/*.py,posted as a comment on that issue, first flagged
EvaluateConfig.parallelism/InferenceConfig.parallelismalongside severalother candidates (a collaborator, @surajksharma07, engaged on the thread and
confirmed the related
evaluate_eval_setempty-collection defect). Afollow-up comment on the same issue narrowed the sweep after two of the
other candidates turned out weaker than first stated, and reaffirmed this
one —
EvaluateConfig.parallelism/InferenceConfig.parallelismat0— asthe confirmed, evidence-backed gap this PR fixes.
🔴 Required Information
Describe the Bug:
EvaluateConfig.parallelismandInferenceConfig.parallelism(
base_eval_service.py:46,72) have no lower-bound constraint. Both valuesare forwarded, unclamped, straight to
asyncio.Semaphore(value=parallelism)(
local_eval_service.py:186,216). A negative value fails loud —asyncio.Semaphoreitself raisesValueErrorimmediately on construction.A zero value fails silent: the semaphore constructs successfully, and every
subsequent
.acquire()blocks forever, since aSemaphore(0)can never bereleased by anything. In
LocalEvalService.perform_inference/evaluatethis means the CLI (or any caller) hangs indefinitely with no exception, no
error message, and no evaluation ever completing.
Steps to Reproduce:
Expected Behavior:
Same as the existing convention for count/limit-like fields elsewhere in
this package —
pydantic.ValidationErrorat construction time for adegenerate value, not a silent hang at call time.
Observed Behavior:
EvaluateConfig(eval_metrics=[...], parallelism=0)andInferenceConfig(parallelism=0)both construct successfully today, thenhang the first evaluation/inference run indefinitely.
Changes
src/google/adk/evaluation/base_eval_service.py: addge=1toEvaluateConfig.parallelismandInferenceConfig.parallelism, matchingthe existing convention for count-like fields in this package —
JudgeModelOptions.parallelism_limit(eval_metrics.py),ToolThreadPoolConfig.max_workers(run_config.py), andContextCacheConfig.cache_intervals(context_cache_config.py) are allalready
ge=1.tests/unittests/evaluation/test_base_eval_service.py(new file): addtest_evaluate_config_rejects_zero_parallelismandtest_inference_config_rejects_zero_parallelism, each assertingparallelism=0raisespydantic.ValidationError.Testing
No regressions in either run.
Risk & rollback
Additive validation only — narrows accepted input, does not change behavior
for any value ever actually used (the default is 4; no caller or test in the
repo constructs either config with
parallelism<=0). Revert is a two-lineremoval of
ge=1.