Skip to content

Let the filesystem benchmark suite set the S3 client options (GH-1325) - #1407

Merged
laughingman7743 merged 5 commits into
masterfrom
feat/1325-benchmark-client-profiles
Nov 15, 2026
Merged

laughingman7743 merged 5 commits into
masterfrom
feat/1325-benchmark-client-profiles

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

WHAT

Benchmark harness changes for the re-measurement of step 5 of #1332, which covers #1322, #1323, #1325, and #1324.

The free-threaded 3.15t environment of #1324 is #1409.

WHY

Refs #1325, #1322, #1332. Prepares the re-measurement that step 5 of #1332 plans, with the scope agreed for it: #1322 and #1323 with the question 4 probe, #1325, and #1324, with aio and s3fs measured on the same host.

TEST

Tested at 0cfe188.

  • just benchmark lint, just docs lint.
  • just benchmark test → 353 passed, 2 skipped (default and polars2 environments); 354 passed, 1 skipped (s3fs environment).
  • New and changed tests:
    • test_filesystem_clients_add_named_cases: loads three client sets from TOML, and checks the matrix, the IDs, and their uniqueness.
    • test_invalid_measurement_settings: eight new invalid client sets, including misspelled checksum values and an unknown retry mode.
    • test_filesystem_clients_are_checked_without_the_aws_configuration: loads a set with AWS_PROFILE naming a missing profile, an uppercase checksum value, and nested retries, which stay as configured.
    • test_pyathena_clients_take_the_options_of_the_case (sync, aio): the s3_config that the connection gets.
    • test_s3fs_client_takes_the_options_of_the_case: the config_kwargs of s3fs.
    • test_objects_clients_take_no_filesystem_client_options: an objects case with a configuration that has no default set.
    • The batch-size assertions of the objects tests now include the synchronous API.
  • Mutation check: 12 mutants, each killed. They cover the ID suffix, the matrix loop, the options of the clients, the synchronous batch size, the option validation, the options of the PyAthena connection, the scope of the options to the filesystem suite, the checksum values, their lowercasing, the wrapping of botocore's errors, and validation by creating a client.
  • Reviews: two self-review rounds; Codex independent review (4 findings) and follow-ups (2, 2, CLEAN), recorded inline.
  • No AWS run: the harness only changes what a run measures. The re-measurement itself follows.

🤖 Generated with Claude Code

laughingman7743 and others added 2 commits November 14, 2026 01:52
The [filesystem] clients tables name sets of botocore Config options,
such as the checksum settings that #1325 measures, which the S3 clients
of every API take: through the connection's s3_config for PyAthena and
config_kwargs for s3fs. Each set is a case, and a name other than
"default", the only set by default, ends the case IDs.

The objects suite passes its concurrency as the batch_size of every
API's batch operations, since the synchronous filesystem runs them
concurrently (GH-1322), and the README no longer describes its
sequential loops, the one-block reads of get_file() (GH-1323), or aio
writes left running by a failed batch (GH-1337).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
return None


def client_options(settings: Settings, case: Case) -> dict[str, Any]:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (implementation behavior): base 13031a5daf4d2afd66eba33c50089024ec48314e, head 1a1dec13333f753ef98950b6f907af7883cb6716 (initial pass at 6613c402).

Covered: Settings.filesystem_clients and its validation (name, table, Config(**options), max_pool_connections); Case.client (default, ID suffix, Case(**payload) of older records without it); the filesystem matrix; client_options() for the connection's s3_config (merged over config by Connection) and s3fs's config_kwargs (AioConfig, which takes s3); the untimed checks, which use the same connection client; the objects suite's batch_size for every API and its equivalence with the synchronous default; the fleet path of the settings (jobs group the client sets into the job of their operation and API; the worker reads the queue's config.toml); README.

FINDINGS, repaired: a run reads the client sets from its own configuration, and a fleet worker from the queue's config.toml, so jobs made with a configuration that the queue does not publish would measure only the default client without an error. The README now says to give jobs and queue the same configuration (1a1dec1).

Comment thread benchmarks/README.md
The `[filesystem]` section of `config.toml` sets the swept values, 8 and 40 workers and pools of 10 and 40 by default, and a `block_size` of 8 MiB.
Its `clients` tables name sets of other botocore `Config` options of the S3 clients, passed the same way with every API; each is a case, and a name other than `default`, the only one by default, ends the case IDs, such as `filesystem-s3-sync-put-w8-p10-required`.
The clients may not set `max_pool_connections`, which `pool_connections` sweeps.
A run takes them from its `--config`, and a fleet worker from the queue's `config.toml`, so give `jobs` and `queue` the same configuration.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (claims, compatibility, operations): base 13031a5daf4d2afd66eba33c50089024ec48314e, head 1a1dec13333f753ef98950b6f907af7883cb6716.

Claims checked:

Compatibility: older result records without client load with the default; a configuration without clients behaves as before. Operator: none at runtime; the clients only change what a configured run measures.

Evidence: just benchmark test in the three environments; mutants per change. No AWS run. FINDINGS: none beyond round one.

- The options of a named client apply to the filesystem suite only; the
  objects suite, which also creates its s3fs filesystem with
  filesystem(), raised KeyError for a configuration without "default".
- A set with options is checked by creating an S3 client, which sends no
  request, so that a value such as a misspelled checksum setting fails
  when the configuration is loaded instead of in every trial.
- The tests check the s3_config that the PyAthena connection gets and
  the config_kwargs of s3fs separately.
- The README limits the one-call latency at concurrency 1 to the
  operations that transfer objects; rm deletes up to 1,000 keys per
  request at any concurrency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The case's connection pool size, with the options of its client in the
filesystem suite; the objects suite's clients take no other option.
"""
options = settings.filesystem_clients[case.client] if case.suite == "filesystem" else {}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5d4-8562-7683-a161-b6406bbc206c), static review only, base 13031a5daf4d2afd66eba33c50089024ec48314e, head 1a1dec13333f753ef98950b6f907af7883cb6716, detached snapshot unchanged. Prompt: the diff, the intended behavior, conventions and docs/testing.md; no PR text or prior findings.

Covered: configuration loading and validation; case IDs and matrix defaults; result serialization, reporting, and resume keys; fleet jobs, queues, and worker configuration; Connection.s3_config merging; s3fs's AioConfig; objects batch dispatch; README claims against the filesystem. Default IDs and matrix order are unchanged, client names survive serialization and distinguish resume records, and workers rebuild the cases from the queued configuration. Result: FINDINGS (4).

1. P2 — Filesystem client settings leak into objects-suite s3fs runs (reviewed at filesystem.py:250). run_objects() also calls this factory, but objects cases always use client="default", so a configuration with only [filesystem.clients.required] makes objects-suite s3fs setup raise KeyError: 'default'; options in default would affect only objects-suite s3fs.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in a75cdf5: client_options() applies the named options to filesystem-suite cases only, and an objects case gets its pool alone. test_objects_clients_take_no_filesystem_client_options (a configuration without default) is killed when the scoping is removed.

Comment thread benchmarks/pyathena_bench/config.py Outdated
try:
# The values, such as the checksum settings, are checked when a client
# is created, which sends no request.
botocore.session.get_session().create_client(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review 2. P2 — Invalid checksum values pass configuration loading (reviewed at config.py:157). request_checksum_calculation = "when_requred" passes Settings.load(): botocore's Config accepts it and validates on client creation (InvalidChecksumConfigError), so planning and queueing accept a configuration whose trials fail during setup.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in a75cdf5: a set with options is checked by creating an S3 client with dummy credentials, which sends no request (about 4 to 40 ms), and botocore's errors become ValueError when the configuration is loaded. An empty set is not checked, so the default configuration loads as before. A misspelled request_checksum_calculation was added to test_invalid_measurement_settings, and is killed without the check.



@pytest.mark.parametrize("api", ["sync", "aio"])
def test_pyathena_clients_take_the_options_of_the_case(api, monkeypatch, tmp_path):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review 3. P3 — The sync/aio test cases do not protect client-option propagation (reviewed at test_filesystem.py:297). Both parameters only repeat the helper assertion; reverting run_filesystem() to s3_config=Config(max_pool_connections=case.pool_connections) leaves them passing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in a75cdf5. test_pyathena_clients_take_the_options_of_the_case (sync, aio) captures the s3_config that run_filesystem() gives connect(): its pool, checksum setting, and s3 options. It is killed by the revert you describe. The s3fs check is its own test.

Comment thread benchmarks/README.md
Each case sets one concurrency from `[objects] concurrency`, 8 and 40 by default: the filesystem's `max_workers` and connection pool, as in the filesystem suite, and the `batch_size` of the `aio` and s3fs operations, which is the number of files they transfer at the same time.
The synchronous `S3FileSystem` inherits fsspec's sequential loops for `cat`, `get`, `put`, `pipe`, and `copy`, so its concurrency only sizes its executor and pool; its `rm` and the other two APIs' delete in DeleteObjects requests of 1,000 keys, which no batch size affects; and `find` pages through ListObjectsV2 with every API.
Each case sets one concurrency from `[objects] concurrency`, 8 and 40 by default: the filesystem's `max_workers` and connection pool, as in the filesystem suite, and the `batch_size` of the `cat`, `get`, `put`, `pipe`, and `copy` operations of every API, which is the number of files they transfer at the same time; the synchronous `S3FileSystem` runs them concurrently since #1322.
With a concurrency of 1, `cat`, `get`, `put`, `pipe`, and `copy` transfer one object at a time, so their time per object is the latency of one call.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review 4. P3 — The concurrency-one latency statement includes bulk operations (reviewed at README.md:315). rm of 1,000 objects is one bulk request at any concurrency, and find measures paginated listings; also, PyAthena's rm/_rm send their DeleteObjects requests in parallel, up to max_workers by default.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in a75cdf5: the README limits the one-call latency at concurrency 1 to cat, get, put, pipe, and copy. It now says that rm deletes up to 1,000 keys per request with every API, PyAthena's up to max_workers requests at once, so a set of up to 1,000 objects is one request at any concurrency, and that find pages through ListObjectsV2.

The follow-up review found that creating a client to check the options
read the AWS configuration of the environment, such as AWS_PROFILE, and
could query instance metadata, and that botocore changes nested options
such as retries while it creates a client. The names of the options are
checked with Config(), which changes nothing, and the values of the two
checksum settings against those botocore accepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
try:
# Checks the names of the options; creating a client, which also checks
# values, would read the AWS configuration of the environment.
Config(**options)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5dc-b363-76c0-87db-b3d84c9fe1b2), static review only, repair 1a1dec13333f753ef98950b6f907af7883cb6716..a75cdf517176fcb55153ec3d1e8b03dbcc644212 (same base 13031a5daf4d2afd66eba33c50089024ec48314e), snapshot unchanged.

Covered: the four repairs, the filesystem and objects call paths, the connection's s3_config, README batching claims, the CLI and worker construction of Settings, botocore's client creation. The objects isolation and README repairs are complete, the checksum settings are validated, and the rewritten tests detect missing forwarding. Result: FINDINGS (2).

1. P2 — Configuration validation depends on AWS configuration and can access the network (reviewed at config.py:163). The session still reads the AWS configuration: AWS_PROFILE=missing makes valid options raise, also in offline plan and the tests; Settings.profile is ignored there; malformed endpoint overrides fail unrelated validation; AWS_DEFAULTS_MODE=auto can query EC2 metadata; and each nonempty set loads the service model again in every worker before the baseline.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in bd7cd85: loading no longer creates a client. Config(**options) checks the option names, which reads neither the environment nor the network. The values of request_checksum_calculation and response_checksum_validation, the settings that botocore checks only at client creation, are checked against when_supported/when_required. Other values are left to botocore at the trial. test_invalid_measurement_settings now also covers an invalid response_checksum_validation; removing the value check fails both checksum cases.

Comment thread benchmarks/pyathena_bench/config.py Outdated
# botocore checks the values of the checksum settings only when it creates
# a client, which a trial would fail at.
for key in CHECKSUM_SETTINGS:
if key in options and options[key] not in CHECKSUM_VALUES:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up 1, finding 2. P2 — Validation mutates stored options (reviewed at config.py:168). Client creation mutates nested dictionaries such as retries (max_attempts → total_max_attempts, the resolved mode), so a validation run under another retry mode writes its values into settings.filesystem_clients, which the real client then uses.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in bd7cd85: the client creation that changed nested options in place is gone. Config(**options) leaves them unchanged (checked with retries = {max_attempts = 2} and s3 = {...}), so settings.filesystem_clients keeps the configured values.

botocore lowercases the checksum settings before it checks them, so
"WHEN_REQUIRED" is valid, and Config() itself raises BotoCoreError
subclasses, such as InvalidRetryModeError for an unknown retry mode,
which become a ValueError of the client again. A test loads a set with
AWS_PROFILE naming a missing profile, an uppercase checksum value, and
nested retries, which must be kept as configured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
for key in CHECKSUM_SETTINGS:
value = options.get(key, CHECKSUM_VALUES[0])
# botocore lowercases the value before it checks it.
if not isinstance(value, str) or value.lower() not in CHECKSUM_VALUES:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent second follow-up review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5e3-77e7-70e1-a0bf-fd7c8678b4f3), static review only, repair a75cdf517176fcb55153ec3d1e8b03dbcc644212..bd7cd858393cba1b6dfa7e7632a014394eb106e4 (same base 13031a5daf4d2afd66eba33c50089024ec48314e), snapshot unchanged.

Covered: the repair diff, settings loading, the locked botocore 1.43.102's validation, the filesystem consumers, the changed and neighboring tests. Both side effects of the previous validation are removed. Result: FINDINGS (2).

1. P2 — Valid checksum values are rejected (reviewed at config.py:170). botocore lowercases the strings before validating them, so request_checksum_calculation = "WHEN_REQUIRED" is valid, but the new comparison rejects it; both checksum settings are affected.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in 0cfe188: the value is compared in lowercase, as botocore compares it, and the stored option keeps its case. A value that is not a string is rejected. test_filesystem_clients_are_checked_without_the_aws_configuration loads "WHEN_REQUIRED"; it fails without the lowercasing.

# Checks the names of the options; creating a client, which also checks
# values, would read the AWS configuration of the environment.
Config(**options)
except (TypeError, BotoCoreError) as e:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent second follow-up, finding 2. P3 — Constructor validation exceptions escape normalization (reviewed at config.py:165). retries = { mode = "standrad" } raises InvalidRetryModeError, a BotoCoreError, from Config itself, which the narrowed handler misses: it bypasses the CLI's error handler with a traceback.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in 0cfe188: BotoCoreError from Config() becomes the ValueError of the client again. retries = { mode = "standrad" } was added to test_invalid_measurement_settings, which fails without the wrapping.

{"filesystem_clients": {"required": "when_required"}},
{"filesystem_clients": {"required": {"checksum": "when_required"}}},
{"filesystem_clients": {"required": {"request_checksum_calculation": "when_requred"}}},
{"filesystem_clients": {"required": {"response_checksum_validation": "never"}}},

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent second follow-up, test note. The added response_checksum_validation = "never" case is deterministic, but would also pass without the repair: the previous client construction rejected it too. It covers the checksum validation, not the environment access or the in-place change of nested options.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; addressed in 0cfe188 with test_filesystem_clients_are_checked_without_the_aws_configuration. With AWS_PROFILE naming a missing profile, it loads an uppercase checksum value and nested retries, and asserts that the stored options are unchanged. Validating by creating a client fails it (mutant killed), as do the missing lowercasing and the missing wrapping (each killed).

for key in CHECKSUM_SETTINGS:
value = options.get(key, CHECKSUM_VALUES[0])
# botocore lowercases the value before it checks it.
if not isinstance(value, str) or value.lower() not in CHECKSUM_VALUES:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent third follow-up review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5e8-51d4-74e0-a03e-9720a0691bc5), static review only, repair bd7cd858393cba1b6dfa7e7632a014394eb106e4..0cfe188dff7abd02a80ef6a1601e94c46be029ae (same base 13031a5daf4d2afd66eba33c50089024ec48314e), snapshot unchanged.

Covered: the repair diff, the surrounding configuration and client plumbing, the locked botocore 1.43.102 source, the changed tests. Checksum casing matches botocore's normalization without changing the supplied options; BotoCoreError subclasses raised by Config, such as InvalidRetryModeError, are wrapped as ValueError; both added cases fail against the previous snapshot; the nested-dictionary assertion detects botocore's in-place retry normalization; validation neither reads the AWS configuration nor creates clients, so the tests are deterministic.

Result: CLEAN.

@laughingman7743
laughingman7743 marked this pull request as ready for review November 13, 2026 17:20
@laughingman7743
laughingman7743 merged commit 31e5260 into master Nov 15, 2026
14 checks passed
@laughingman7743
laughingman7743 deleted the feat/1325-benchmark-client-profiles branch November 15, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant