diff --git a/.github/workflows/wheels.yml b/.github/workflows/wheels.yml index 0ae9597..14f47dc 100644 --- a/.github/workflows/wheels.yml +++ b/.github/workflows/wheels.yml @@ -15,7 +15,24 @@ on: workflow_dispatch: jobs: + # A release tag must name the version that CMakeLists.txt sets (project()), + # which the wheels, the sdist and numpy2x.__version__ all take. + version: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Tag matches the version in CMakeLists.txt + if: startsWith(github.ref, 'refs/tags/v') + shell: bash + run: | + v=$(sed -nE 's/^project\(numpy2x VERSION ([^ ]+).*/\1/p' CMakeLists.txt) + if [ "v$v" != "$GITHUB_REF_NAME" ]; then + echo "::error::tag $GITHUB_REF_NAME does not match version $v in CMakeLists.txt" + exit 1 + fi + wheels: + needs: version strategy: fail-fast: false matrix: @@ -36,6 +53,7 @@ jobs: path: wheelhouse/*.whl sdist: + needs: version runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 diff --git a/docs/development.md b/docs/development.md index ac19136..a79213b 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,5 +1,40 @@ # Development +How numpy2x is built, tested, measured and released. For using numpy2x, see the +[README](../README.md) and [testing.md](testing.md). + +## How it works + +numpy2x intercepts NumPy operations (through NumPy's array protocols and, with +`python -m numpy2x`, its own module) and issues each one as a task. The Python thread only +issues tasks and runs ahead; two worker threads run each task as soon as its inputs are +computed. Every task calls the inner loops of the installed NumPy (ufunc, reduction, gufunc and +sort loops), exactly as NumPy calls them, so results are bit-identical to NumPy. Whatever numpy2x +does not run itself falls back to NumPy, after waiting for the data it reads. + +## Repository layout + +``` +include/numpy2x/numpy2x.h C interface of the runtime; the Python bridge uses only this +src/runtime/ runtime: arrays, tasks, worker pool, element memory pool, + copy-on-write, layouts; no Python dependency + (workers.cpp: the number of worker threads) +src/kernels/ adapters that call NumPy's loops (element-wise, reductions, + matmul, gufuncs, sort) +src/capi/capi.cpp implementation of the C interface +python/numpy2x/ Python package: _numpy2x.c (C extension, the per-operation hot + path), _ndarray.py (array class, fallbacks), _ufuncs.py, _linalg.py, + _fft.py, _join.py, _indexing.py, _compose.py, _sort.py +python/_numpy2x_hook.py `import numpy` gives numpy2x in user code (python -m numpy2x, + NUMPY2X=1 through numpy2x.pth) +python/tests/ pytest suite: every test compares with NumPy +python/benchmarks/ README examples, benchmark suite, shape maps +tests/ C/C++ tests (CTest) +docker/ Linux, Windows and Wine images +tools/dev.ps1, dev.cmd, dev.sh one-command build and test +.github/workflows/ ci.yml (build + tests), wheels.yml (wheels, releases, PyPI) +``` + ## Build and test `dev.cmd` (Windows, also by double-click) and `./dev.sh` (Linux) pull, build into `.venv` and @@ -8,6 +43,10 @@ run the tests; run programs with `.venv\Scripts\python -m numpy2x prog.py` (Linu `dev.cmd -NoPull -NoTest -Python `, `./dev.sh --no-pull --no-test`, `PYTHON=python3.13 ./dev.sh`. +Requirements: Python >= 3.12 (64 bit), NumPy >= 2.5, CMake >= 3.24 and a C++20 compiler +(Visual Studio 2022 with the C++ workload on Windows, GCC or Clang on Linux); pip fetches CMake +and Ninja itself. + With CMake directly: ```bash @@ -20,8 +59,29 @@ PYTHONPATH=build/python python3 -m pytest -q python/tests `pip wheel .` builds a wheel (`numpy2x-*.whl`) to install on other machines with the same platform, Python version and architecture. -Every test compares numpy2x with NumPy running the same code: results must be bit-identical -(values, dtypes, shapes, strides, warnings, errors). +## Rules for changes + +- **Bit-identical to NumPy.** Every result must equal plain NumPy's: values, dtypes, shapes, + strides, warnings and errors. Never trade this for speed: no own kernels, no reordered sums. +- **Test against NumPy.** Every new operation needs a test that runs the same code with numpy2x + and with NumPy and compares the results, and a test that it actually runs asynchronously + (`python/tests/async_path.py`: `assert_async`). +- **Measure performance changes.** Changes to kernels, the issue path or the per-operation code + are performance changes: compare with `main` (benchmark suite, cases the change targets) + before merging. +- **Linux and Windows behave the same.** Platform-specific code stays behind small functions + with one implementation per platform; CI tests both. +- **Two worker threads, single-stage execution.** The number of worker threads + (`src/runtime/workers.cpp`) and the execution model (a task starts when its inputs are + completely computed) are conditions of the patent license ([PATENTS](../PATENTS)). Changes to + either are not covered by that license; see "Licenses" below. +- Formatting: `.clang-format` (C/C++). Code, comments and commit messages in English. + +## Continuous integration + +`ci.yml` runs on every push to `main` and on pull requests: pip build and the pytest suite on +Linux (GCC) and Windows (MSVC) with Python 3.12 and 3.13, and `tools/dev.ps1` as started from +Visual Studio's developer prompt. Pull requests are merged only with green CI. ## Benchmarks @@ -31,7 +91,9 @@ PYTHONPATH=build/python python3 python/benchmarks/suite.py [--quick] # ~100 pr PYTHONPATH=build/python python3 python/benchmarks/array_pipelining.py ``` -Use a Release build. Reports go to `./reports` (untracked). +Use a Release build and an otherwise idle machine. Reports go to `./reports` (untracked). +`readme_examples.py` replaces the committed results in `python/benchmarks/results/readme` and +the charts in `docs/images`; commit them only from a run whose results are all bit-identical. ## Docker @@ -43,3 +105,53 @@ docker build -f docker/linux/Dockerfile -t numpy2x:linux . docker run --rm numpy2x:linux # all test suites docker run --rm -v "$PWD:/work" numpy2x:linux python prog.py # run a program with numpy2x ``` + +## Releasing + +The version is set in one place: `project(numpy2x VERSION x.y.z ...)` in the top-level +`CMakeLists.txt`. The package version (pyproject reads it), `numpy2x.__version__` and the version +test all take it from there. + +1. On `main`, change the version in `CMakeLists.txt` and merge it (commit message + `numpy2x x.y.z`). CI must be green. Versions follow semantic versioning: x.y.(z+1) for fixes, + x.(y+1).0 for new features, (x+1).0.0 for incompatible changes. +2. Tag that commit with the same version and push the tag: + + ```bash + git checkout main + git pull origin main + git tag -a vx.y.z -m "numpy2x x.y.z" + git push origin vx.y.z + ``` + +3. The tag starts `wheels.yml`. It first checks that the tag matches the version in + `CMakeLists.txt` and stops otherwise; then it builds the wheels (CPython 3.12 and 3.13, Windows + x64 and Linux x86_64 manylinux) and the sdist, runs the full test suite on each wheel, creates + the GitHub Release `vx.y.z` with these files and uploads them to PyPI. +4. Add the title and release notes to the GitHub Release. +5. Check: `pip install numpy2x==x.y.z` in a fresh environment, `numpy2x.__version__`, and the + project page on https://pypi.org/project/numpy2x/. + +Notes: + +- PyPI accepts each version only once. A broken release is fixed with a new version, not by + uploading the same one again. +- The PyPI upload uses trusted publishing: the GitHub environment `pypi` and the repository + variable `PYPI_PUBLISH = true` (a repository variable, not a secret or an environment + variable) must exist; the PyPI project trusts `ILNumerics/numpy2x`, workflow `wheels.yml`, + environment `pypi`. +- If the upload was skipped or failed, rerun it for the existing tag: Actions → wheels → + Run workflow → "Use workflow from": Tags → the tag. A run on `main` builds and tests but + neither releases nor uploads. +- New Python versions need an entry in `[tool.cibuildwheel] build` in `pyproject.toml` (and + in the matrix of `ci.yml`); until then, users on them build from the sdist. +- Every tagged release is a "Released Version" under the patent license; branches and commits + between releases are not. + +## Licenses + +The code is under the Clear BSD License ([LICENSE](../LICENSE)). numpy2x implements methods +patented by ILNumerics; their use is licensed by [PATENTS](../PATENTS). Versions that run more than +two worker threads per process, perform staged execution or are adapted to other languages, array +libraries or runtimes are not covered by that license, also as forks. Third-party notices: +[NOTICE](../NOTICE). Patents: https://ilnumerics.net/patents.html. diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index a5800d9..6bbdcdb 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1,5 +1,7 @@ add_executable(test_version test_version.c) target_link_libraries(test_version PRIVATE numpy2x) +# The version is set once, in project() of the top-level CMakeLists.txt. +target_compile_definitions(test_version PRIVATE NX_EXPECTED_VERSION="${numpy2x_VERSION}") add_test(NAME version COMMAND test_version) # Verifies that NumPy's inner loops can be called from worker threads without the GIL. diff --git a/tests/test_version.c b/tests/test_version.c index a39dc02..f66807d 100644 --- a/tests/test_version.c +++ b/tests/test_version.c @@ -5,7 +5,7 @@ int main(void) { const char* v = nx_version(); - if (v == NULL || strcmp(v, "1.0.0") != 0) { + if (v == NULL || strcmp(v, NX_EXPECTED_VERSION) != 0) { fprintf(stderr, "unexpected version: %s\n", v ? v : "(null)"); return 1; }