Repository navigation
Add benchmark environments for Python 3.15 and 3.15t (GH-1324) - #1409
Conversation
The new filesystem dependency group has only PyAthena, s3fs, and psutil, and is synced alone (--only-group) into .venv-fs315 with Python 3.15 and into .venv-fs315t with its free-threaded build, which lacks wheels of the DataFrame backends. The template installs both interpreters, gcc for psutil, which has no 3.15t wheel on Linux, and the two environments. jobs --environment fs315 or fs315t runs the filesystem and objects suites there, s3fs included, and rejects the Athena suites. The runner imports the adapters, and with them the DataFrame backends, only where they are installed; heartbeat() moves to pyathena_bench.sampling, which the filesystem suites import instead. Each trial and the run's environment record whether the GIL is enabled, so that a free-threaded run shows whether an import re-enabled it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| # environment, for interpreters that lack their wheels, has none of them and runs | ||
| # only the filesystem and objects suites. | ||
| try: | ||
| from pyathena_bench import adapters |
There was a problem hiding this comment.
Self-review round one (implementation behavior): base 13031a5daf4d2afd66eba33c50089024ec48314e, head 9a9935063f3a08ff318a9555025f2f67a0b08d26.
Covered: the import chain of every harness module (only runner -> adapters imports the DataFrame backends; cases imports pyathena.filesystem.s3_core, aws boto3, config the standard library); the optional import of adapters (ModuleNotFoundError only, so another import error of the default environment still fails) and the Athena suites' guard in the trial; heartbeat() in sampling; environment() and the trial result's gil_enabled; the fleet's environment selection (suite restriction, s3fs routing, suffixes, interpreter(), the worker's check of the queued environments); the filesystem group with --only-group (PyAthena from the workspace source, no project dependencies); the lock entries of the 3.15t dependencies for Linux aarch64 (aiohttp, multidict, yarl, propcache, and wrapt have cp315t wheels; frozenlist a pure wheel; psutil only an abi3 wheel, so 3.15t builds it from the sdist, 3.15 uses the abi3 wheel); the template (gcc, the interpreters, the syncs); README.
Result: CLEAN for the implementation. Limits: the trial-level gil_enabled is not covered by a unit test (the environment record is); the template is not deployed here.
| git -C /opt/pyathena checkout --detach ${GitCommit} | ||
| export UV_PYTHON_INSTALL_DIR=/opt/uv-python | ||
| uv python install ${PythonVersion} | ||
| uv python install ${PythonVersion} ${FilesystemPythonVersion} ${FilesystemPythonVersion}t |
There was a problem hiding this comment.
Self-review round two (claims, compatibility, operations): base 13031a5daf4d2afd66eba33c50089024ec48314e, head 9a9935063f3a08ff318a9555025f2f67a0b08d26.
Claims checked:
- "psutil 7.2.2 has cp313t and cp314t wheels for Linux aarch64 but no cp315t one": PyPI file list of psutil 7.2.2; the lock entry has the abi3 wheel and the sdist only.
- "The 3.15t sync builds it from source": locally, the installed psutil's
WHEELreadsGenerator: setuptools,Tag: cp315-cp315t-linux_x86_64. - "imported no DataFrame backend / without the GIL": the smoke test in the synced 3.15t environment (
sys._is_gil_enabled()False after importing s3fs, aiohttp, psutil, and the suites), and the new subprocess test in the default environment. - "The relock re-resolved beautifulsoup4/soupsieve, dependencies of furo": the lock diff; furo is the docs theme, not in any benchmark environment.
Existing callers: the default, polars2, and s3fs environments still import the adapters before the baseline, so their measurements are unchanged; jobs without --environment is unchanged.
Operator: the bootstrap installs gcc and two more interpreters and environments, which lengthens host setup; the psutil build needs gcc. The fs315 and fs315t trials have a smaller RSS baseline (no DataFrame backend), which the README says to compare only among themselves.
FINDINGS: none.
The independent review found that uv 0.12.5, the template's default, knows no Python 3.15 release (its newest is 3.15.0rc1), so a fresh host would fail to install the interpreters before any environment is synced. uv 0.13.7 lists 3.15.2 and its free-threaded build for Linux aarch64. The trial of a filesystem or objects suite is tested through the worker without the adapters: it records the GIL after its measurement, which a test measurement enables, and an Athena suite fails with the reason. The subprocess test keeps the import checks; its GIL assertion compared the helper with itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| Type: String | ||
| Default: '0.12.5' | ||
| # A release that knows the Python versions below; 0.12.5 knows no 3.15 release. | ||
| Default: '0.13.7' |
There was a problem hiding this comment.
Independent review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5de-dc04-7b93-87bc-248698869787), static review only, base 13031a5daf4d2afd66eba33c50089024ec48314e, head 9a9935063f3a08ff318a9555025f2f67a0b08d26, detached snapshot unchanged. Prompt: the diff without uv.lock, the intended behavior, conventions and docs/testing.md; no PR text or prior findings.
Covered: the filesystem and objects imports through the CLI, report, fleet worker, prepare/cleanup, and trial execution; workspace dependencies, the lockfile and conflicts; the AL2023/aarch64 bootstrap; environment routing and the existing environments; GIL records; README; the added tests against docs/testing.md. Result: FINDINGS (2).
1. P1 — Bootstrap retains a uv version too old for Python 3.15.2 (reviewed at benchmark.yaml:370, the uv python install line). UvVersion defaulted to 0.12.5 (released August 14), before Python 3.15.2 (October 25), and uv's interpreter catalog is fixed per release, so on a fresh host the unconditional uv python install fails before any environment sync, the existing environments included.
There was a problem hiding this comment.
Verified and fixed in 91c8bfa: uv python list --all-platforms --all-arches --only-downloads with uv 0.12.5 lists only cpython-3.15.0rc1 for Linux aarch64, while 0.13.7 lists 3.15.2 and cpython-3.15.2+freethreaded-linux-aarch64-gnu. The default is now 0.13.7, the uv version that produced this uv.lock and the local 3.15t smoke test. A comment on the parameter states the requirement. No offline test can check the catalog of a uv release, so this is not tested.
|
|
||
|
|
||
| @pytest.mark.parametrize("suite", ["filesystem", "objects"]) | ||
| def test_trial_of_a_filesystem_suite_needs_no_adapters(suite, monkeypatch): |
There was a problem hiding this comment.
Independent review 2. P2 — The backend-free test never executes a trial or verifies its GIL record (reviewed at test_runner.py:538). The subprocess only imports modules and calls environment(): a DataFrame import inside a measurement function, or removing the worker's gil_enabled assignment, leaves it passing, and comparing the environment value with runner.gil_enabled() accepts a hard-coded result.
There was a problem hiding this comment.
Verified; fixed in 91c8bfa for the worker and the GIL record. test_trial_of_a_filesystem_suite_needs_no_adapters (filesystem, objects) runs worker() with adapters set to None, a stub session, and a stub measurement. The stub enables a fake sys._is_gil_enabled (initially False). The test asserts that the environment record is False and that the trial records True, the state after the measurement. Removing the trial's assignment fails it, and so does reading the GIL before the measurement. test_trial_of_an_athena_suite_needs_the_adapters covers the guard. The tautological assertion was removed from the subprocess test, which keeps the import checks. Not covered: an import that a measurement function makes at run time. Repository rules forbid runtime imports (CLAUDE.md), the measurement functions have their own tests with fakes, and on AWS such an import would fail every fs315/fs315t trial with ModuleNotFoundError, visibly.
| Type: String | ||
| Default: '0.12.5' | ||
| # A release that knows the Python versions below; 0.12.5 knows no 3.15 release. | ||
| Default: '0.13.7' |
There was a problem hiding this comment.
Independent follow-up review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5ec-08b4-7d73-aa1c-1dae211ba0a0), static review only, repair 9a9935063f3a08ff318a9555025f2f67a0b08d26..91c8bfa7058942aa59fcc1e1741cdccc971c2f99 (same base 13031a5daf4d2afd66eba33c50089024ec48314e), snapshot unchanged.
Covered: the repair diff; interpreter installation; all five bootstrap environments; workspace groups and the lockfile; --no-sync execution and the fleet's interpreter selection; the worker's and environment's GIL records; test distinctness, regression sensitivity, and determinism. uv 0.13.7 addresses the interpreter catalog and matches the lockfile producer; the existing environments keep Python 3.12 and their groups; --no-sync still preserves the environment. The new tests run both worker branches, assert the environment and post-measurement GIL values independently, and check the Athena suites' rejection; the omitted runtime-import test is not required under the repository rules.
Result: CLEAN.
WHAT
Plan item 1 of #1324: benchmark environments for Python 3.15 and its free-threaded build, 3.15t, which run only the filesystem and objects suites, without the DataFrame backends that lack free-threaded wheels.
filesystemdependency group inbenchmarks/pyproject.tomlhas onlyPyAthena,psutil, ands3fs.--only-group filesystem, which leaves out the benchmark's own dependencies: pandas, Polars, pyarrow, awswrangler, and SQLAlchemy.pyproject.toml, since s3fs's aiobotocore pins botocore, as for thes3fsgroup.beautifulsoup4/soupsieve, dependencies of the docs theme (furo), to one version, under the new conflict markers; no benchmark environment changes.FilesystemPythonVersion, default3.15.2.${FilesystemPythonVersion}and${FilesystemPythonVersion}t), and syncs/opt/pyathena/.venv-fs315and/opt/pyathena/.venv-fs315t.UvVersiondefaults to 0.13.7 instead of 0.12.5, whose interpreter catalog has no 3.15 release after 3.15.0rc1. 0.13.7 is the version that produceduv.lock, and it listscpython-3.15.2andcpython-3.15.2+freethreadedfor linux-aarch64-gnu. The existing environments keep their Python version and groups.jobs --environment fs315and--environment fs315trun jobs of the filesystem and objects suites in those environments, the s3fs API included, whose jobs carry the environment's suffix there.pyathena_bench.adapters, and with it the DataFrame backends, only when they are installed (ModuleNotFoundError, asfilesystem.pyimports s3fs), and an Athena suite without them raisesRuntimeErrorin its trial.heartbeat()moves to the newpyathena_bench.sampling, which the filesystem and objects suites import instead ofadapters.gil_enabled(sys._is_gil_enabled()) after its measurement, for plan item 3: whether an import re-enabled the GIL. The run's environment records it too.WHY
Refs #1324. Part of the re-measurement of step 5 of #1332, whose agreed scope includes #1324.
TEST
Tested at 91c8bfa.
just benchmark lint,just docs lint,uv lock --check(via--lockedin the benchmark recipes).just benchmark test→ 348 passed, 2 skipped (default and polars2 environments); 349 passed, 1 skipped (s3fs environment).cpython-3.15.2+freethreadedon x86_64.uv sync --directory benchmarks --locked --only-group filesystem --python 3.15.2tsynced the environment, building psutil from source (WHEEL:Generator: setuptools,Tag: cp315-cp315t-linux_x86_64). In it,python -m pyathena_bench --helpand the imports of the filesystem and objects suites ran without the GIL (sys._is_gil_enabled()is False), imported no DataFrame backend, and expanded filesystem and objects jobs forfs315t.uv python list --all-platforms --all-arches --only-downloads: uv 0.12.5 lists onlycpython-3.15.0rc1-linux-aarch64-gnu; 0.13.7 lists 3.15.2 and 3.15.2+freethreaded for it.test_filesystem_environments_run_every_api_of_their_suitesandtest_filesystem_environments_reject_the_athena_suites(fleet);test_trial_of_a_filesystem_suite_needs_no_adapters(filesystem, objects): the worker runs without the adapters and records the GIL after the measurement, which the test measurement enables;test_trial_of_an_athena_suite_needs_the_adapters;test_filesystem_suites_run_without_the_dataframe_backends: a child process whose imports of the backends fail imports the harness, gets no adapters, and records the environment without them;🤖 Generated with Claude Code