Skip to content

Record the filesystem re-measurement of 2026-11-15 (GH-1332) - #1420

Merged
laughingman7743 merged 4 commits into
masterfrom
docs/1332-filesystem-remeasurement
Nov 16, 2026
Merged

laughingman7743 merged 4 commits into
masterfrom
docs/1332-filesystem-remeasurement

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

WHAT

Records the re-measurement of #1332's last step, run at 1d9e5039 on one c8gn.2xlarge Spot host.

Findings, in short:

  1. The synchronous S3FileSystem ran the batch operations at 1.01 to 1.08 times AioS3FileSystem. The previous entry measured 6.5 to 20.3 objects/s when they ran one object at a time; this run measured 97 to 508.
  2. get_file() downloaded at 232 to 613 MiB/s, with s3fs at 0.77 to 1.06 times PyAthena. The previous entry measured 54 to 73 MiB/s.
  3. One request at a time took 68 to 84 ms per 1 MiB object with every API: 1.5 to 2.6 times a request of a lane at concurrency 8. Question 4's gap is not specific to the synchronous filesystem.
  4. With 40 workers, PyAthena's 1 GiB and 6 GiB uploads were 1.17 to 1.45 times faster with when_required and 1.38 to 1.66 times faster when payload signing was also disabled. The default frames each part as an aws-chunked stream with a CRC32 trailer, while when_required sends it unframed; both are read in 128 KiB blocks. This was read from botocore's source, not profiled.
  5. On free-threaded 3.15t, PyAthena's transfers at concurrency 40 ran at 1.16 to 2.70 times their throughput on 3.15, and the 1 GiB read and get at 1.46 to 2.20 times. s3fs ran at 0.81 to 0.99 times at concurrency 40.

WHY

Closes #1324 (plan items 2 and 3). Closes #1325 (plan item 2, with the measured cost in the Checksums subsection, as its comment proposed). Closes #1332 (the re-measurement is its last step).

Refs #1315, #1322, and #1323. The entry's "Status for #1315" lists the measured differences that remain; the decision on #1315 stays open.

TEST

Tested at 2145a2f.

  • just docs lint and rumdl check on the three changed files.
  • just docs build: no new warnings. The four warnings in docs/usage.md are missing cross-reference targets in other sections, which are also present on master.
  • Every number in the findings was recomputed from the recovered job-summary.csv files, the 561 cases of the three queues, and checked against the tables.
    • The generator scripts and rows.json are kept outside the repository.
    • The recovered results are in the maintainer's checkout: benchmarks/results/recovered-step5/, 7,300 files.
  • The environment table comes from the trials' environment.json files and the stack's recorded parameters and instances.
    • The lockfile hash in environment.json equals sha256sum of the commit's uv.lock.
  • The aws-chunked mechanism in finding 4 was checked against the locked botocore 1.43.102 and urllib3 2.8.0 sources: AwsChunkedWrapper._DEFAULT_CHUNK_SIZE, botocore's BUFFER_SIZE = 1024 * 128, which it passes to the pool as blocksize, and convert_body_to_file_like_object, which wraps a bytes body in BytesIO. It was not profiled.
  • Reviews: two self-review rounds; Codex independent review (5 findings) and follow-up (1 finding), all repaired and recorded inline.
  • No AWS run: the entry records a finished run. The stack was deleted after recovery, and no instance, group, scratch bucket, or scratch database remained.

🤖 Generated with Claude Code

laughingman7743 and others added 2 commits November 17, 2026 00:50
The history entry records the filesystem and objects suites at
1d9e503 on one c8gn.2xlarge Spot host: the concurrent batch operations
of the synchronous S3FileSystem (#1322) and the parallel range downloads
of get_file() (#1323) against AioS3FileSystem and s3fs, the time of one
request at a time (question 4 of #1322), the upload checksum settings
(#1325), and Python 3.15 against its free-threaded build (#1324).

docs/usage.md states the measured throughput and CPU time of the
checksum settings in its Checksums subsection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host then stayed idle until 14:29 UTC on 2026-11-16, when a script on it found every job done without a failure and retired it.
After recovering the results, `cleanup --execute` removed the source objects and sets, and the stack was deleted.

## Findings

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (content against the data): base fb22bdc5edc589b02a6d0dccea39800eb82f7d2c, head 22bfa95b6cdbe271c844b78a7490e35eea300a01.

Covered:

  • Every number in the five findings and in "Status for Benchmark the S3 filesystem against s3fs and decide on a standalone package #1315", recomputed from the 561 rows of the recovered job-summary.csv files.
  • The generated tables against the same rows.
  • The selection, against the queued jobs.json files (153/30/3) and the queue configurations.
  • The run record, against the done markers (first start and last end per queue) and hosts/*.json.
  • The environment table, against environment.json (Python, dependency versions, lock_sha256, gil_enabled, no awscrt), stack.json, and instances.json.

Repairs before posting:

  • "Deletes and listings within 0.92 to 1.08 of each other" holds only for s3fs against sync (aio's listing differs by up to 1.13), so it was narrowed.
  • "About twice as long per request" fails for s3fs (1.46), so it became 1.5 to 2.6 times.
  • The CPU reduction was 34 to 39%, not 35 to 39%.
  • The status line claimed single-object copies, which this run did not measure.
  • An RSS explanation ("up to 40 ranges in flight") was not supported by the numbers, so it was removed.

Result: CLEAN after the repairs. Limits: five trials on one host; the 64 MiB free-threaded ratios are noisy and stated as such.

Comment thread docs/usage.md
Over HTTPS, the client leaves the hash out of a request that sets `ContentMD5`, which S3 checks instead, unless its `s3` options set `payload_signing_enabled` to `True`.
It also leaves the hash out if its `s3` options set `payload_signing_enabled` to `False` or if it sends unsigned requests, such as `S3FileSystem(anon=True)`.

In the [benchmark run of 2026-11-15](https://github.com/pyathena-dev/PyAthena/blob/master/benchmarks/history/2026-11-15-1d9e5039-filesystem.md), on one `c8gn.2xlarge` host with 40 workers and 8 MiB parts, `S3FileSystem` wrote a 1 GiB object at 836 MiB/s with the defaults, at 1,002 MiB/s with `request_checksum_calculation="when_required"`, and at 1,216 MiB/s when its `s3` options also set `payload_signing_enabled` to `False`.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (claims, docs, issue closure): base fb22bdc5edc589b02a6d0dccea39800eb82f7d2c, head 22bfa95b6cdbe271c844b78a7490e35eea300a01.

Claims checked:

Docs reader: the paragraph states numbers and settings only, without recommendation.

Result: CLEAN.

The independent review found that botocore reads request bodies in
blocks of 128 KiB (httpsession BUFFER_SIZE), not urllib3's 16 KiB; that
a third 64 MiB aio case fell below the eight-worker range; that the s3fs
ranges and thread peaks held only for some operations; and that the
data supports no blanket threshold of variation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`write` and `put` of the 1 GiB object ran at 1.17 to 1.28 times the default throughput with `required`, and at 1.38 to 1.46 times with `unsigned`; of the 6 GiB object, at 1.41 to 1.45 and 1.64 to 1.66 times, up to 1,623 and 1,840 MiB/s.
For the 256 MiB object, the other configurations ran within 0.96 to 1.16 times of the default.
`default` and `required` took the same CPU time, 10.8 to 11.7 seconds for the 6 GiB object, and `unsigned` 34 to 39% less.
With the default, botocore 1.43.102 sends each part as an `aws-chunked` stream of 1 MiB chunks with a CRC32 trailer (`AwsChunkedWrapper`), which the HTTP connection reads in blocks of 128 KiB, the block size that botocore sets for its connection pools; with `when_required`, it sends the part as one buffer and signs its SHA-256 hash.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed): Codex CLI codex exec -s read-only with read access to the recovered data and the locked botocore/urllib3 sources (model gpt-6-astra, session 01a1e50e-d468-7103-9a19-f1f6cf4f7407), static review with read-only recomputation, base fb22bdc5edc589b02a6d0dccea39800eb82f7d2c, head 22bfa95b6cdbe271c844b78a7490e35eea300a01, detached snapshot unchanged. Prompt: the diff, the data layout, and the closure scope of #1324/#1325/#1332; no PR text or prior findings.

Covered: the three changed files; all seven tables against the 561 summary rows and 2,805 raw trials (all values match); queue selections, environments, manifests, completion and host records; the previous entry; benchmark code; botocore/urllib3 sources. The closure requirements are covered subject to the findings. Stack deletion is not evidenced by the data snapshot. Result: FINDINGS (5).

1. P2 — the wrapper is read in 128 KiB blocks, not 16 KiB (reviewed at line 76). botocore's httpsession.py:93 sets BUFFER_SIZE = 1024 * 128 and passes it as blocksize (line 357) when urllib3's PoolKey has key_blocksize, which the installed urllib3 has.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in 7faf74a: BUFFER_SIZE = 1024 * 128 (botocore 1.43.102 httpsession.py:93) is set because urllib3 2.8.0 has PoolKey.key_blocksize, so the entry now says the HTTP connection reads the wrapper in blocks of 128 KiB, the block size that botocore sets for its connection pools. The round-two self-review record cited urllib3's 16 KiB default, which botocore overrides; the PR body is corrected too. The mechanism stays labeled as read from source and not profiled.

5. **On free-threaded Python 3.15t, PyAthena's transfers ran up to 2.7 times faster with 40 threads, and s3fs's did not.**
With concurrency 40, both PyAthena APIs catted, downloaded, uploaded, piped, and copied the medium set at 1.16 to 2.70 times their throughput on Python 3.15, and ran `info()` at 2.22 to 2.24 times: 1.91 to 2.12 times for `cat` and 2.57 to 2.70 times for downloads, up to 930 and 680 objects per second.
With 40 workers, they read the 1 GiB object at 1.96 to 2.20 times, downloaded it at 1.46 to 1.98 times, and wrote and put it at 1.37 to 1.47 times, up to 1,217, 1,048, and 1,212 MiB/s.
With concurrency 8, the PyAthena APIs ran within 0.94 to 1.09 times of Python 3.15, and with 8 workers within 0.96 to 1.26 times, except three 64 MiB cases of the aio API at 0.54 to 0.81.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review 2. P2 — a third eight-worker exception (reviewed at line 83): aio put, 64 MiB, pool 10, 143.17 / 176.45 = 0.81.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in 7faf74a: the eight-worker ratios below 0.96 are aio read (0.56), get (0.54), and put (0.81) of the 64 MiB object with pool 10. The entry now says "except three 64 MiB cases of the aio API at 0.54 to 0.81". The maximum stays 1.26.

With concurrency 40, both PyAthena APIs catted, downloaded, uploaded, piped, and copied the medium set at 1.16 to 2.70 times their throughput on Python 3.15, and ran `info()` at 2.22 to 2.24 times: 1.91 to 2.12 times for `cat` and 2.57 to 2.70 times for downloads, up to 930 and 680 objects per second.
With 40 workers, they read the 1 GiB object at 1.96 to 2.20 times, downloaded it at 1.46 to 1.98 times, and wrote and put it at 1.37 to 1.47 times, up to 1,217, 1,048, and 1,212 MiB/s.
With concurrency 8, the PyAthena APIs ran within 0.94 to 1.09 times of Python 3.15, and with 8 workers within 0.96 to 1.26 times, except three 64 MiB cases of the aio API at 0.54 to 0.81.
s3fs, which runs its requests on one event loop thread, ran at 0.81 to 0.94 times its throughput on Python 3.15 in every operation but `rm` (0.99) with concurrency 40, at 0.91 to 1.01 times with concurrency 8, and at 0.83 to 1.15 times for one object.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review 3. P3 — the s3fs ranges hold only for transfers and info() (reviewed at line 84): over every operation, 0.91 to 1.01 at concurrency 8 (find 0.906) and 0.81 to 0.99 at concurrency 40 (rm 0.992).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in 7faf74a: per operation, s3fs on 3.15t against 3.15 is 0.906 (find) to 1.008 at concurrency 8, and 0.805 to 0.942 with rm at 0.992 at concurrency 40. The entry now says 0.81 to 0.94 "in every operation but rm (0.99)" at concurrency 40, and 0.91 to 1.01 at concurrency 8.

Against s3fs, the transfers split by operation: s3fs uploaded, piped, and copied at 0.95 to 1.15 times the synchronous throughput, catted at 0.65 to 0.69 times, and downloaded at 1.40 to 1.42 times.
s3fs deleted and listed at 0.92 to 1.08 times the synchronous throughput, and ran `info()` at 1.04 times at concurrency 8 and 1.31 times at concurrency 40.
In the transfers and `info()`, the synchronous processes peaked at 19 threads at concurrency 8 and 51 at concurrency 40, and at 27 and 91 for downloads, whose files read their ranges in the request pool; s3fs's peaked at 13.
In `rm`, every API's peaked at 43 or 44 threads, with the 32 threads that the harness uses to prepare the copies to delete.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review 4. P3 — thread peaks are unqualified (reviewed at line 61): rm peaked at 44 threads for s3fs and 43 for sync, with the harness's preparation threads.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in 7faf74a: the sentence is scoped to the transfers and info(). A new sentence states that in rm every API peaked at 43 or 44 threads, with the 32 threads that the harness uses to prepare the copies to delete (SETUP_WORKERS = 32 in objects.py), as the previous entry also stated.


## Limits

- One host and five trials per case in one run; the entry does not estimate the variation between runs, so small differences between cases are not attributed to their settings.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review 5. P2 — no blanket 10% variation threshold (reviewed at line 96): the data establishes none; for example, s3fs's batch get at concurrency 40 fell 9.3% on 3.15t with non-overlapping trials (2.611 to 2.695 s against 2.907 to 3.022 s).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed and fixed in 7faf74a: the limit now says that the run is one host and five trials per case, that the entry does not estimate the variation between runs, and that small differences between cases are therefore not attributed to their settings. The 10% threshold is gone.

)

botocore converts a bytes Body of PutObject and UploadPart to BytesIO,
which the HTTP connection also reads in 128 KiB blocks; the difference
is the aws-chunked framing, not one buffer against blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`write` and `put` of the 1 GiB object ran at 1.17 to 1.28 times the default throughput with `required`, and at 1.38 to 1.46 times with `unsigned`; of the 6 GiB object, at 1.41 to 1.45 and 1.64 to 1.66 times, up to 1,623 and 1,840 MiB/s.
For the 256 MiB object, the other configurations ran within 0.96 to 1.16 times of the default.
`default` and `required` took the same CPU time, 10.8 to 11.7 seconds for the 6 GiB object, and `unsigned` 34 to 39% less.
With the default, botocore 1.43.102 sends each part as an `aws-chunked` stream of 1 MiB chunks with a CRC32 trailer (`AwsChunkedWrapper`); with `when_required`, it sends the part as a `BytesIO` without that framing and signs its SHA-256 hash.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review (relayed): Codex CLI codex exec -s read-only with read access to the recovered data and sources (model gpt-6-astra, session 01a1e518-ad03-7bb3-b4a6-270aeb095405), static review with read-only recomputation, repair 22bfa95b6cdbe271c844b78a7490e35eea300a01..7faf74a47b8f2cc70ccc52c3b7f52dd84dd47e78 (same base fb22bdc5edc589b02a6d0dccea39800eb82f7d2c), snapshot unchanged.

Covered: the repair hunks and surrounding text; raw trial medians and thread peaks; botocore/urllib3 upload handling; fsspec's event loop; the benchmark's preparation and sampling; the revised variation caveat. Confirmed: the 128 KiB blocks of the default wrapper; exactly three eight-worker exceptions (aio 64 MiB pool 10: get 0.537, read 0.556, put 0.811); s3fs 0.81 to 0.94 except rm 0.99 at concurrency 40 and 0.91 to 1.01 at 8; rm peaks of 43/44 with the 32 preparation threads; the removal of the 10% threshold. Result: FINDINGS (1).

1. P2 — "with when_required, it sends the part as one buffer" is wrong (reviewed at line 77): botocore's convert_body_to_file_like_object (handlers.py:897, registered for UploadPart and PutObject at 1513) turns the bytes into BytesIO, which urllib3 reads with body.read(blocksize) (util/request.py:269): 64 blocks of 128 KiB for an 8 MiB part. The SHA-256 signing clause is correct; the inaccuracy predates the repair.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in 2145a2f: convert_body_to_file_like_object wraps a bytes Body of PutObject and UploadPart in BytesIO, and urllib3's chunk_readable() reads any file-like body with body.read(blocksize). The entry now says that with when_required botocore sends the part as a BytesIO without the aws-chunked framing and signs its SHA-256 hash, and that the HTTP connection reads either body in blocks of 128 KiB. So the difference stated is the framing and the CRC32 trailer, not one buffer against blocks. It is still marked as read from source and not profiled. The #1325 comment draft is corrected the same way.

For the 256 MiB object, the other configurations ran within 0.96 to 1.16 times of the default.
`default` and `required` took the same CPU time, 10.8 to 11.7 seconds for the 6 GiB object, and `unsigned` 34 to 39% less.
With the default, botocore 1.43.102 sends each part as an `aws-chunked` stream of 1 MiB chunks with a CRC32 trailer (`AwsChunkedWrapper`); with `when_required`, it sends the part as a `BytesIO` without that framing and signs its SHA-256 hash.
The HTTP connection reads either body in blocks of 128 KiB, the block size that botocore sets for its connection pools.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent second follow-up review (relayed): Codex CLI codex exec -s read-only with read access to the sources (model gpt-6-astra, session 01a1e51f-30fc-7db1-bd96-cd43a796b790), static review only, repair 7faf74a47b8f2cc70ccc52c3b7f52dd84dd47e78..2145a2f9414ed86bcdb888ba6213a21789f6195f (same base fb22bdc5edc589b02a6d0dccea39800eb82f7d2c), snapshot unchanged.

Covered: finding 4 (lines 77 and 78), PyAthena's multipart body handoff, botocore 1.43.102's checksum, body, signing, and connection paths, and urllib3 2.8.0's body reading. The default's AwsChunkedWrapper produces 1 MiB chunks with a CRC32 trailer. when_required converts the part's bytes to BytesIO, omits the aws-chunked framing, and signs the payload's SHA-256 hash. Both bodies are read with read(131072).

Result: CLEAN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant