Skip to content

Add benchmark environments for Python 3.15 and 3.15t (GH-1324) - #1409

Merged
laughingman7743 merged 2 commits into
masterfrom
feat/1324-free-threaded-benchmark-env
Nov 15, 2026
Merged

laughingman7743 merged 2 commits into
masterfrom
feat/1324-free-threaded-benchmark-env

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

WHAT

Plan item 1 of #1324: benchmark environments for Python 3.15 and its free-threaded build, 3.15t, which run only the filesystem and objects suites, without the DataFrame backends that lack free-threaded wheels.

  • Dependencies. A new filesystem dependency group in benchmarks/pyproject.toml has only PyAthena, psutil, and s3fs.
    • It is synced alone, with --only-group filesystem, which leaves out the benchmark's own dependencies: pandas, Polars, pyarrow, awswrangler, and SQLAlchemy.
    • It joins the conflict set of the root pyproject.toml, since s3fs's aiobotocore pins botocore, as for the s3fs group.
    • The relock also re-resolved beautifulsoup4/soupsieve, dependencies of the docs theme (furo), to one version, under the new conflict markers; no benchmark environment changes.
  • Template.
    • New parameter FilesystemPythonVersion, default 3.15.2.
    • Bootstrap installs gcc, both interpreters (${FilesystemPythonVersion} and ${FilesystemPythonVersion}t), and syncs /opt/pyathena/.venv-fs315 and /opt/pyathena/.venv-fs315t.
    • UvVersion defaults to 0.13.7 instead of 0.12.5, whose interpreter catalog has no 3.15 release after 3.15.0rc1. 0.13.7 is the version that produced uv.lock, and it lists cpython-3.15.2 and cpython-3.15.2+freethreaded for linux-aarch64-gnu. The existing environments keep their Python version and groups.
    • psutil 7.2.2 has cp313t and cp314t wheels for Linux aarch64 but no cp315t one, so the 3.15t sync builds it from source. The other compiled packages (aiohttp, multidict, yarl, propcache, and wrapt) have cp315t wheels, and frozenlist has a pure wheel.
  • Fleet.
    • jobs --environment fs315 and --environment fs315t run jobs of the filesystem and objects suites in those environments, the s3fs API included, whose jobs carry the environment's suffix there.
    • Any other suite is rejected for them.
  • Runner.
    • The runner imports pyathena_bench.adapters, and with it the DataFrame backends, only when they are installed (ModuleNotFoundError, as filesystem.py imports s3fs), and an Athena suite without them raises RuntimeError in its trial.
    • heartbeat() moves to the new pyathena_bench.sampling, which the filesystem and objects suites import instead of adapters.
    • Each trial result records gil_enabled (sys._is_gil_enabled()) after its measurement, for plan item 3: whether an import re-enabled the GIL. The run's environment records it too.
  • README. It describes the environments, the job option, the GIL record, and the different RSS baseline: these environments import no DataFrame backend.

WHY

Refs #1324. Part of the re-measurement of step 5 of #1332, whose agreed scope includes #1324.

TEST

Tested at 91c8bfa.

  • just benchmark lint, just docs lint, uv lock --check (via --locked in the benchmark recipes).
  • just benchmark test → 348 passed, 2 skipped (default and polars2 environments); 349 passed, 1 skipped (s3fs environment).
  • Local smoke test with uv 0.13.7's cpython-3.15.2+freethreaded on x86_64. uv sync --directory benchmarks --locked --only-group filesystem --python 3.15.2t synced the environment, building psutil from source (WHEEL: Generator: setuptools, Tag: cp315-cp315t-linux_x86_64). In it, python -m pyathena_bench --help and the imports of the filesystem and objects suites ran without the GIL (sys._is_gil_enabled() is False), imported no DataFrame backend, and expanded filesystem and objects jobs for fs315t.
  • uv python list --all-platforms --all-arches --only-downloads: uv 0.12.5 lists only cpython-3.15.0rc1-linux-aarch64-gnu; 0.13.7 lists 3.15.2 and 3.15.2+freethreaded for it.
  • New tests:
    • test_filesystem_environments_run_every_api_of_their_suites and test_filesystem_environments_reject_the_athena_suites (fleet);
    • test_trial_of_a_filesystem_suite_needs_no_adapters (filesystem, objects): the worker runs without the adapters and records the GIL after the measurement, which the test measurement enables;
    • test_trial_of_an_athena_suite_needs_the_adapters;
    • test_filesystem_suites_run_without_the_dataframe_backends: a child process whose imports of the backends fail imports the harness, gets no adapters, and records the environment without them;
    • the template test asserts gcc, the interpreters, and the two syncs.
  • Mutation check: 9 mutants, each killed. They cover the suite restriction, the s3fs routing, the optional adapters, the environment's GIL record, the 3.15t sync, the trial's GIL record, the GIL read before the measurement, and the Athena suites' guard.
  • Reviews: two self-review rounds; Codex independent review (2 findings) and follow-up (CLEAN), recorded inline.
  • Not run here: the template on AWS. The Linux aarch64 build of psutil happens at the next deployment.

🤖 Generated with Claude Code

The new filesystem dependency group has only PyAthena, s3fs, and psutil,
and is synced alone (--only-group) into .venv-fs315 with Python 3.15 and
into .venv-fs315t with its free-threaded build, which lacks wheels of
the DataFrame backends. The template installs both interpreters, gcc for
psutil, which has no 3.15t wheel on Linux, and the two environments.

jobs --environment fs315 or fs315t runs the filesystem and objects
suites there, s3fs included, and rejects the Athena suites. The runner
imports the adapters, and with them the DataFrame backends, only where
they are installed; heartbeat() moves to pyathena_bench.sampling, which
the filesystem suites import instead. Each trial and the run's
environment record whether the GIL is enabled, so that a free-threaded
run shows whether an import re-enabled it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# environment, for interpreters that lack their wheels, has none of them and runs
# only the filesystem and objects suites.
try:
from pyathena_bench import adapters

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (implementation behavior): base 13031a5daf4d2afd66eba33c50089024ec48314e, head 9a9935063f3a08ff318a9555025f2f67a0b08d26.

Covered: the import chain of every harness module (only runner -> adapters imports the DataFrame backends; cases imports pyathena.filesystem.s3_core, aws boto3, config the standard library); the optional import of adapters (ModuleNotFoundError only, so another import error of the default environment still fails) and the Athena suites' guard in the trial; heartbeat() in sampling; environment() and the trial result's gil_enabled; the fleet's environment selection (suite restriction, s3fs routing, suffixes, interpreter(), the worker's check of the queued environments); the filesystem group with --only-group (PyAthena from the workspace source, no project dependencies); the lock entries of the 3.15t dependencies for Linux aarch64 (aiohttp, multidict, yarl, propcache, and wrapt have cp315t wheels; frozenlist a pure wheel; psutil only an abi3 wheel, so 3.15t builds it from the sdist, 3.15 uses the abi3 wheel); the template (gcc, the interpreters, the syncs); README.

Result: CLEAN for the implementation. Limits: the trial-level gil_enabled is not covered by a unit test (the environment record is); the template is not deployed here.

git -C /opt/pyathena checkout --detach ${GitCommit}
export UV_PYTHON_INSTALL_DIR=/opt/uv-python
uv python install ${PythonVersion}
uv python install ${PythonVersion} ${FilesystemPythonVersion} ${FilesystemPythonVersion}t

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (claims, compatibility, operations): base 13031a5daf4d2afd66eba33c50089024ec48314e, head 9a9935063f3a08ff318a9555025f2f67a0b08d26.

Claims checked:

  • "psutil 7.2.2 has cp313t and cp314t wheels for Linux aarch64 but no cp315t one": PyPI file list of psutil 7.2.2; the lock entry has the abi3 wheel and the sdist only.
  • "The 3.15t sync builds it from source": locally, the installed psutil's WHEEL reads Generator: setuptools, Tag: cp315-cp315t-linux_x86_64.
  • "imported no DataFrame backend / without the GIL": the smoke test in the synced 3.15t environment (sys._is_gil_enabled() False after importing s3fs, aiohttp, psutil, and the suites), and the new subprocess test in the default environment.
  • "The relock re-resolved beautifulsoup4/soupsieve, dependencies of furo": the lock diff; furo is the docs theme, not in any benchmark environment.

Existing callers: the default, polars2, and s3fs environments still import the adapters before the baseline, so their measurements are unchanged; jobs without --environment is unchanged.

Operator: the bootstrap installs gcc and two more interpreters and environments, which lengthens host setup; the psutil build needs gcc. The fs315 and fs315t trials have a smaller RSS baseline (no DataFrame backend), which the README says to compare only among themselves.

FINDINGS: none.

The independent review found that uv 0.12.5, the template's default,
knows no Python 3.15 release (its newest is 3.15.0rc1), so a fresh host
would fail to install the interpreters before any environment is
synced. uv 0.13.7 lists 3.15.2 and its free-threaded build for Linux
aarch64.

The trial of a filesystem or objects suite is tested through the worker
without the adapters: it records the GIL after its measurement, which a
test measurement enables, and an Athena suite fails with the reason.
The subprocess test keeps the import checks; its GIL assertion compared
the helper with itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type: String
Default: '0.12.5'
# A release that knows the Python versions below; 0.12.5 knows no 3.15 release.
Default: '0.13.7'

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5de-dc04-7b93-87bc-248698869787), static review only, base 13031a5daf4d2afd66eba33c50089024ec48314e, head 9a9935063f3a08ff318a9555025f2f67a0b08d26, detached snapshot unchanged. Prompt: the diff without uv.lock, the intended behavior, conventions and docs/testing.md; no PR text or prior findings.

Covered: the filesystem and objects imports through the CLI, report, fleet worker, prepare/cleanup, and trial execution; workspace dependencies, the lockfile and conflicts; the AL2023/aarch64 bootstrap; environment routing and the existing environments; GIL records; README; the added tests against docs/testing.md. Result: FINDINGS (2).

1. P1 — Bootstrap retains a uv version too old for Python 3.15.2 (reviewed at benchmark.yaml:370, the uv python install line). UvVersion defaulted to 0.12.5 (released August 14), before Python 3.15.2 (October 25), and uv's interpreter catalog is fixed per release, so on a fresh host the unconditional uv python install fails before any environment sync, the existing environments included.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and fixed in 91c8bfa: uv python list --all-platforms --all-arches --only-downloads with uv 0.12.5 lists only cpython-3.15.0rc1 for Linux aarch64, while 0.13.7 lists 3.15.2 and cpython-3.15.2+freethreaded-linux-aarch64-gnu. The default is now 0.13.7, the uv version that produced this uv.lock and the local 3.15t smoke test. A comment on the parameter states the requirement. No offline test can check the catalog of a uv release, so this is not tested.



@pytest.mark.parametrize("suite", ["filesystem", "objects"])
def test_trial_of_a_filesystem_suite_needs_no_adapters(suite, monkeypatch):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review 2. P2 — The backend-free test never executes a trial or verifies its GIL record (reviewed at test_runner.py:538). The subprocess only imports modules and calls environment(): a DataFrame import inside a measurement function, or removing the worker's gil_enabled assignment, leaves it passing, and comparing the environment value with runner.gil_enabled() accepts a hard-coded result.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified; fixed in 91c8bfa for the worker and the GIL record. test_trial_of_a_filesystem_suite_needs_no_adapters (filesystem, objects) runs worker() with adapters set to None, a stub session, and a stub measurement. The stub enables a fake sys._is_gil_enabled (initially False). The test asserts that the environment record is False and that the trial records True, the state after the measurement. Removing the trial's assignment fails it, and so does reading the GIL before the measurement. test_trial_of_an_athena_suite_needs_the_adapters covers the guard. The tautological assertion was removed from the subprocess test, which keeps the import checks. Not covered: an import that a measurement function makes at run time. Repository rules forbid runtime imports (CLAUDE.md), the measurement functions have their own tests with fakes, and on AWS such an import would fail every fs315/fs315t trial with ModuleNotFoundError, visibly.

Type: String
Default: '0.12.5'
# A release that knows the Python versions below; 0.12.5 knows no 3.15 release.
Default: '0.13.7'

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review (relayed): Codex CLI codex exec -s read-only (model gpt-6-astra, session 01a1d5ec-08b4-7d73-aa1c-1dae211ba0a0), static review only, repair 9a9935063f3a08ff318a9555025f2f67a0b08d26..91c8bfa7058942aa59fcc1e1741cdccc971c2f99 (same base 13031a5daf4d2afd66eba33c50089024ec48314e), snapshot unchanged.

Covered: the repair diff; interpreter installation; all five bootstrap environments; workspace groups and the lockfile; --no-sync execution and the fleet's interpreter selection; the worker's and environment's GIL records; test distinctness, regression sensitivity, and determinism. uv 0.13.7 addresses the interpreter catalog and matches the lockfile producer; the existing environments keep Python 3.12 and their groups; --no-sync still preserves the environment. The new tests run both worker branches, assert the environment and post-measurement GIL values independently, and check the Athena suites' rejection; the omitted runtime-import test is not required under the repository rules.

Result: CLEAN.

@laughingman7743
laughingman7743 marked this pull request as ready for review November 13, 2026 17:37
@laughingman7743
laughingman7743 merged commit 1d9e503 into master Nov 15, 2026
15 checks passed
@laughingman7743
laughingman7743 deleted the feat/1324-free-threaded-benchmark-env branch November 15, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant