Repository navigation
Let the filesystem benchmark suite set the S3 client options (GH-1325) - #1407
Conversation
The [filesystem] clients tables name sets of botocore Config options, such as the checksum settings that #1325 measures, which the S3 clients of every API take: through the connection's s3_config for PyAthena and config_kwargs for s3fs. Each set is a case, and a name other than "default", the only set by default, ends the case IDs. The objects suite passes its concurrency as the batch_size of every API's batch operations, since the synchronous filesystem runs them concurrently (GH-1322), and the README no longer describes its sequential loops, the one-block reads of get_file() (GH-1323), or aio writes left running by a failed batch (GH-1337). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| return None | ||
|
|
||
|
|
||
| def client_options(settings: Settings, case: Case) -> dict[str, Any]: |
There was a problem hiding this comment.
Self-review round one (implementation behavior): base 13031a5daf4d2afd66eba33c50089024ec48314e, head 1a1dec13333f753ef98950b6f907af7883cb6716 (initial pass at 6613c402).
Covered: Settings.filesystem_clients and its validation (name, table, Config(**options), max_pool_connections); Case.client (default, ID suffix, Case(**payload) of older records without it); the filesystem matrix; client_options() for the connection's s3_config (merged over config by Connection) and s3fs's config_kwargs (AioConfig, which takes s3); the untimed checks, which use the same connection client; the objects suite's batch_size for every API and its equivalence with the synchronous default; the fleet path of the settings (jobs group the client sets into the job of their operation and API; the worker reads the queue's config.toml); README.
FINDINGS, repaired: a run reads the client sets from its own configuration, and a fleet worker from the queue's config.toml, so jobs made with a configuration that the queue does not publish would measure only the default client without an error. The README now says to give jobs and queue the same configuration (1a1dec1).
| The `[filesystem]` section of `config.toml` sets the swept values, 8 and 40 workers and pools of 10 and 40 by default, and a `block_size` of 8 MiB. | ||
| Its `clients` tables name sets of other botocore `Config` options of the S3 clients, passed the same way with every API; each is a case, and a name other than `default`, the only one by default, ends the case IDs, such as `filesystem-s3-sync-put-w8-p10-required`. | ||
| The clients may not set `max_pool_connections`, which `pool_connections` sweeps. | ||
| A run takes them from its `--config`, and a fleet worker from the queue's `config.toml`, so give `jobs` and `queue` the same configuration. |
There was a problem hiding this comment.
Self-review round two (claims, compatibility, operations): base 13031a5daf4d2afd66eba33c50089024ec48314e, head 1a1dec13333f753ef98950b6f907af7883cb6716.
Claims checked:
- "Existing IDs and runs are unchanged": the default configuration has only the
defaultset, whose IDs carry no suffix; the existing ID tests pass unchanged. - "Since Run the batch operations of S3FileSystem concurrently (GH-1322) #1344 the synchronous filesystem runs these operations concurrently with
max_workersas its default batch size":S3FileSystem._batch_limit()at master; so the objects suite measured concurrent synchronous batches already, and passingbatch_sizechanges no measured value. - "A concurrency of 1 measures the latency of one call per object": each API then runs one call at a time (a one-thread pool, or batches of one coroutine), so the time per object is one call's latency plus the batch's scheduling.
- README: the three removed statements contradict Download in parallel ranges in get_file() #1323 (waves of ranges), Run the batch operations of S3FileSystem concurrently (GH-1322) #1344 (concurrent synchronous batches), and Align AioS3FileSystem with S3FileSystem for cancellation and arguments (GH-1337) #1395 (a failed aio batch drains its calls); the
getdescription matchesS3File._download().
Compatibility: older result records without client load with the default; a configuration without clients behaves as before. Operator: none at runtime; the clients only change what a configured run measures.
Evidence: just benchmark test in the three environments; mutants per change. No AWS run. FINDINGS: none beyond round one.
- The options of a named client apply to the filesystem suite only; the objects suite, which also creates its s3fs filesystem with filesystem(), raised KeyError for a configuration without "default". - A set with options is checked by creating an S3 client, which sends no request, so that a value such as a misspelled checksum setting fails when the configuration is loaded instead of in every trial. - The tests check the s3_config that the PyAthena connection gets and the config_kwargs of s3fs separately. - The README limits the one-call latency at concurrency 1 to the operations that transfer objects; rm deletes up to 1,000 keys per request at any concurrency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| The case's connection pool size, with the options of its client in the | ||
| filesystem suite; the objects suite's clients take no other option. | ||
| """ | ||
| options = settings.filesystem_clients[case.client] if case.suite == "filesystem" else {} |
There was a problem hiding this comment.
Independent review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5d4-8562-7683-a161-b6406bbc206c), static review only, base 13031a5daf4d2afd66eba33c50089024ec48314e, head 1a1dec13333f753ef98950b6f907af7883cb6716, detached snapshot unchanged. Prompt: the diff, the intended behavior, conventions and docs/testing.md; no PR text or prior findings.
Covered: configuration loading and validation; case IDs and matrix defaults; result serialization, reporting, and resume keys; fleet jobs, queues, and worker configuration; Connection.s3_config merging; s3fs's AioConfig; objects batch dispatch; README claims against the filesystem. Default IDs and matrix order are unchanged, client names survive serialization and distinguish resume records, and workers rebuild the cases from the queued configuration. Result: FINDINGS (4).
1. P2 — Filesystem client settings leak into objects-suite s3fs runs (reviewed at filesystem.py:250). run_objects() also calls this factory, but objects cases always use client="default", so a configuration with only [filesystem.clients.required] makes objects-suite s3fs setup raise KeyError: 'default'; options in default would affect only objects-suite s3fs.
There was a problem hiding this comment.
Verified and fixed in a75cdf5: client_options() applies the named options to filesystem-suite cases only, and an objects case gets its pool alone. test_objects_clients_take_no_filesystem_client_options (a configuration without default) is killed when the scoping is removed.
| try: | ||
| # The values, such as the checksum settings, are checked when a client | ||
| # is created, which sends no request. | ||
| botocore.session.get_session().create_client( |
There was a problem hiding this comment.
Independent review 2. P2 — Invalid checksum values pass configuration loading (reviewed at config.py:157). request_checksum_calculation = "when_requred" passes Settings.load(): botocore's Config accepts it and validates on client creation (InvalidChecksumConfigError), so planning and queueing accept a configuration whose trials fail during setup.
There was a problem hiding this comment.
Verified and fixed in a75cdf5: a set with options is checked by creating an S3 client with dummy credentials, which sends no request (about 4 to 40 ms), and botocore's errors become ValueError when the configuration is loaded. An empty set is not checked, so the default configuration loads as before. A misspelled request_checksum_calculation was added to test_invalid_measurement_settings, and is killed without the check.
|
|
||
|
|
||
| @pytest.mark.parametrize("api", ["sync", "aio"]) | ||
| def test_pyathena_clients_take_the_options_of_the_case(api, monkeypatch, tmp_path): |
There was a problem hiding this comment.
Independent review 3. P3 — The sync/aio test cases do not protect client-option propagation (reviewed at test_filesystem.py:297). Both parameters only repeat the helper assertion; reverting run_filesystem() to s3_config=Config(max_pool_connections=case.pool_connections) leaves them passing.
There was a problem hiding this comment.
Verified and fixed in a75cdf5. test_pyathena_clients_take_the_options_of_the_case (sync, aio) captures the s3_config that run_filesystem() gives connect(): its pool, checksum setting, and s3 options. It is killed by the revert you describe. The s3fs check is its own test.
| Each case sets one concurrency from `[objects] concurrency`, 8 and 40 by default: the filesystem's `max_workers` and connection pool, as in the filesystem suite, and the `batch_size` of the `aio` and s3fs operations, which is the number of files they transfer at the same time. | ||
| The synchronous `S3FileSystem` inherits fsspec's sequential loops for `cat`, `get`, `put`, `pipe`, and `copy`, so its concurrency only sizes its executor and pool; its `rm` and the other two APIs' delete in DeleteObjects requests of 1,000 keys, which no batch size affects; and `find` pages through ListObjectsV2 with every API. | ||
| Each case sets one concurrency from `[objects] concurrency`, 8 and 40 by default: the filesystem's `max_workers` and connection pool, as in the filesystem suite, and the `batch_size` of the `cat`, `get`, `put`, `pipe`, and `copy` operations of every API, which is the number of files they transfer at the same time; the synchronous `S3FileSystem` runs them concurrently since #1322. | ||
| With a concurrency of 1, `cat`, `get`, `put`, `pipe`, and `copy` transfer one object at a time, so their time per object is the latency of one call. |
There was a problem hiding this comment.
Independent review 4. P3 — The concurrency-one latency statement includes bulk operations (reviewed at README.md:315). rm of 1,000 objects is one bulk request at any concurrency, and find measures paginated listings; also, PyAthena's rm/_rm send their DeleteObjects requests in parallel, up to max_workers by default.
There was a problem hiding this comment.
Verified and fixed in a75cdf5: the README limits the one-call latency at concurrency 1 to cat, get, put, pipe, and copy. It now says that rm deletes up to 1,000 keys per request with every API, PyAthena's up to max_workers requests at once, so a set of up to 1,000 objects is one request at any concurrency, and that find pages through ListObjectsV2.
The follow-up review found that creating a client to check the options read the AWS configuration of the environment, such as AWS_PROFILE, and could query instance metadata, and that botocore changes nested options such as retries while it creates a client. The names of the options are checked with Config(), which changes nothing, and the values of the two checksum settings against those botocore accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| try: | ||
| # Checks the names of the options; creating a client, which also checks | ||
| # values, would read the AWS configuration of the environment. | ||
| Config(**options) |
There was a problem hiding this comment.
Independent follow-up review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5dc-b363-76c0-87db-b3d84c9fe1b2), static review only, repair 1a1dec13333f753ef98950b6f907af7883cb6716..a75cdf517176fcb55153ec3d1e8b03dbcc644212 (same base 13031a5daf4d2afd66eba33c50089024ec48314e), snapshot unchanged.
Covered: the four repairs, the filesystem and objects call paths, the connection's s3_config, README batching claims, the CLI and worker construction of Settings, botocore's client creation. The objects isolation and README repairs are complete, the checksum settings are validated, and the rewritten tests detect missing forwarding. Result: FINDINGS (2).
1. P2 — Configuration validation depends on AWS configuration and can access the network (reviewed at config.py:163). The session still reads the AWS configuration: AWS_PROFILE=missing makes valid options raise, also in offline plan and the tests; Settings.profile is ignored there; malformed endpoint overrides fail unrelated validation; AWS_DEFAULTS_MODE=auto can query EC2 metadata; and each nonempty set loads the service model again in every worker before the baseline.
There was a problem hiding this comment.
Verified and fixed in bd7cd85: loading no longer creates a client. Config(**options) checks the option names, which reads neither the environment nor the network. The values of request_checksum_calculation and response_checksum_validation, the settings that botocore checks only at client creation, are checked against when_supported/when_required. Other values are left to botocore at the trial. test_invalid_measurement_settings now also covers an invalid response_checksum_validation; removing the value check fails both checksum cases.
| # botocore checks the values of the checksum settings only when it creates | ||
| # a client, which a trial would fail at. | ||
| for key in CHECKSUM_SETTINGS: | ||
| if key in options and options[key] not in CHECKSUM_VALUES: |
There was a problem hiding this comment.
Independent follow-up 1, finding 2. P2 — Validation mutates stored options (reviewed at config.py:168). Client creation mutates nested dictionaries such as retries (max_attempts → total_max_attempts, the resolved mode), so a validation run under another retry mode writes its values into settings.filesystem_clients, which the real client then uses.
There was a problem hiding this comment.
Verified and fixed in bd7cd85: the client creation that changed nested options in place is gone. Config(**options) leaves them unchanged (checked with retries = {max_attempts = 2} and s3 = {...}), so settings.filesystem_clients keeps the configured values.
botocore lowercases the checksum settings before it checks them, so "WHEN_REQUIRED" is valid, and Config() itself raises BotoCoreError subclasses, such as InvalidRetryModeError for an unknown retry mode, which become a ValueError of the client again. A test loads a set with AWS_PROFILE naming a missing profile, an uppercase checksum value, and nested retries, which must be kept as configured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| for key in CHECKSUM_SETTINGS: | ||
| value = options.get(key, CHECKSUM_VALUES[0]) | ||
| # botocore lowercases the value before it checks it. | ||
| if not isinstance(value, str) or value.lower() not in CHECKSUM_VALUES: |
There was a problem hiding this comment.
Independent second follow-up review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5e3-77e7-70e1-a0bf-fd7c8678b4f3), static review only, repair a75cdf517176fcb55153ec3d1e8b03dbcc644212..bd7cd858393cba1b6dfa7e7632a014394eb106e4 (same base 13031a5daf4d2afd66eba33c50089024ec48314e), snapshot unchanged.
Covered: the repair diff, settings loading, the locked botocore 1.43.102's validation, the filesystem consumers, the changed and neighboring tests. Both side effects of the previous validation are removed. Result: FINDINGS (2).
1. P2 — Valid checksum values are rejected (reviewed at config.py:170). botocore lowercases the strings before validating them, so request_checksum_calculation = "WHEN_REQUIRED" is valid, but the new comparison rejects it; both checksum settings are affected.
There was a problem hiding this comment.
Verified and fixed in 0cfe188: the value is compared in lowercase, as botocore compares it, and the stored option keeps its case. A value that is not a string is rejected. test_filesystem_clients_are_checked_without_the_aws_configuration loads "WHEN_REQUIRED"; it fails without the lowercasing.
| # Checks the names of the options; creating a client, which also checks | ||
| # values, would read the AWS configuration of the environment. | ||
| Config(**options) | ||
| except (TypeError, BotoCoreError) as e: |
There was a problem hiding this comment.
Independent second follow-up, finding 2. P3 — Constructor validation exceptions escape normalization (reviewed at config.py:165). retries = { mode = "standrad" } raises InvalidRetryModeError, a BotoCoreError, from Config itself, which the narrowed handler misses: it bypasses the CLI's error handler with a traceback.
There was a problem hiding this comment.
Verified and fixed in 0cfe188: BotoCoreError from Config() becomes the ValueError of the client again. retries = { mode = "standrad" } was added to test_invalid_measurement_settings, which fails without the wrapping.
| {"filesystem_clients": {"required": "when_required"}}, | ||
| {"filesystem_clients": {"required": {"checksum": "when_required"}}}, | ||
| {"filesystem_clients": {"required": {"request_checksum_calculation": "when_requred"}}}, | ||
| {"filesystem_clients": {"required": {"response_checksum_validation": "never"}}}, |
There was a problem hiding this comment.
Independent second follow-up, test note. The added response_checksum_validation = "never" case is deterministic, but would also pass without the repair: the previous client construction rejected it too. It covers the checksum validation, not the environment access or the in-place change of nested options.
There was a problem hiding this comment.
Agreed; addressed in 0cfe188 with test_filesystem_clients_are_checked_without_the_aws_configuration. With AWS_PROFILE naming a missing profile, it loads an uppercase checksum value and nested retries, and asserts that the stored options are unchanged. Validating by creating a client fails it (mutant killed), as do the missing lowercasing and the missing wrapping (each killed).
| for key in CHECKSUM_SETTINGS: | ||
| value = options.get(key, CHECKSUM_VALUES[0]) | ||
| # botocore lowercases the value before it checks it. | ||
| if not isinstance(value, str) or value.lower() not in CHECKSUM_VALUES: |
There was a problem hiding this comment.
Independent third follow-up review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5e8-51d4-74e0-a03e-9720a0691bc5), static review only, repair bd7cd858393cba1b6dfa7e7632a014394eb106e4..0cfe188dff7abd02a80ef6a1601e94c46be029ae (same base 13031a5daf4d2afd66eba33c50089024ec48314e), snapshot unchanged.
Covered: the repair diff, the surrounding configuration and client plumbing, the locked botocore 1.43.102 source, the changed tests. Checksum casing matches botocore's normalization without changing the supplied options; BotoCoreError subclasses raised by Config, such as InvalidRetryModeError, are wrapped as ValueError; both added cases fail against the previous snapshot; the nested-dictionary assertion detects botocore's in-place retry normalization; validation neither reads the AWS configuration nor creates clients, so the tests are deterministic.
Result: CLEAN.
WHAT
Benchmark harness changes for the re-measurement of step 5 of #1332, which covers #1322, #1323, #1325, and #1324.
S3 client options of the filesystem suite (Document and measure the default upload checksums of the S3 client #1325 plan item 1). New
[filesystem] clientstables name sets of botocoreConfigoptions, which the S3 clients of every API take: PyAthena's through the connection'ss3_config, and s3fs's throughconfig_kwargs(client_options()infilesystem.py).default, the only set by default, ends the case ID, for examplefilesystem-s3-sync-put-w8-p10-required. Existing IDs and runs are unchanged.Config.request_checksum_calculationandresponse_checksum_validationmust bewhen_supportedorwhen_requiredin any case, since botocore checks them only when it creates a client. A set may not containmax_pool_connections, sincepool_connectionssweeps it.Configoption, as the #1325 comment asks for its third configuration (s3 = { payload_signing_enabled = false }).Objects suite. The suite passes its concurrency as
batch_sizeto thecat,get,put,pipe, andcopyof every API, the synchronous filesystem included.max_workers, the same value, as its default batch size. Passing the value makes the cases independent of that default.README. It describes the client sets with the three configurations of Document and measure the default upload checksums of the S3 client #1325, and drops three statements that Run the batch operations of S3FileSystem concurrently #1322, Download in parallel ranges in get_file() #1323, and Align AioS3FileSystem with S3FileSystem for cancellation, recursive put, expand_path, and arguments #1337 made untrue:
get_file();It also states that
rmdeletes up to 1,000 keys per request at any concurrency, and thatfindpages through ListObjectsV2.The free-threaded 3.15t environment of #1324 is #1409.
WHY
Refs #1325, #1322, #1332. Prepares the re-measurement that step 5 of #1332 plans, with the scope agreed for it: #1322 and #1323 with the question 4 probe, #1325, and #1324, with
aioand s3fs measured on the same host.TEST
Tested at 0cfe188.
just benchmark lint,just docs lint.just benchmark test→ 353 passed, 2 skipped (default and polars2 environments); 354 passed, 1 skipped (s3fs environment).test_filesystem_clients_add_named_cases: loads three client sets from TOML, and checks the matrix, the IDs, and their uniqueness.test_invalid_measurement_settings: eight new invalid client sets, including misspelled checksum values and an unknown retry mode.test_filesystem_clients_are_checked_without_the_aws_configuration: loads a set withAWS_PROFILEnaming a missing profile, an uppercase checksum value, and nestedretries, which stay as configured.test_pyathena_clients_take_the_options_of_the_case(sync, aio): thes3_configthat the connection gets.test_s3fs_client_takes_the_options_of_the_case: theconfig_kwargsof s3fs.test_objects_clients_take_no_filesystem_client_options: an objects case with a configuration that has nodefaultset.🤖 Generated with Claude Code