Automate PyPI publishing and document the contribution and release process

.github/workflows/publish.yml

@@ -0,0 +1,119 @@
+name: Publish
+
+# Builds the distributions for a GitHub release and uploads them to PyPI using
+# Trusted Publishing. Merging to main never publishes anything; only publishing a
+# release on GitHub (which creates or points at a v* tag) does, and the upload
+# step waits for approval in the protected "pypi" environment. See CONTRIBUTING.md.
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:  # build and verify only, without publishing
+
+permissions:
+  contents: read
+
+# Never run two publishes at once, and never cancel one that is in progress
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: false
+
+jobs:
+  build:
+    name: Build and verify distributions
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
+        with:
+          python-version: "3.14"
+
+      - name: Build distributions
+        run: |
+          python -m pip install build twine
+          python -m build
+          python -m twine check --strict dist/*
+          ls -l dist
+
+      - name: Install the wheel and run the tests against it
+        run: |
+          python -m pip install "$(ls dist/*.whl)[test]"
+          python -m pip check
+          # Work from outside the checkout so that "orca" resolves to the
+          # installed wheel rather than the source tree
+          cd "$RUNNER_TEMP"
+          python -c "from importlib.metadata import version; print('installed version', version('orca'))"
+          # The tests live inside the package but are left out of the wheel, so
+          # copy the test folders into the installed package and run them from
+          # there, exercising the wheel rather than the source tree
+          installed="$(python -c 'import os, orca; print(os.path.dirname(orca.__file__))')"
+          echo "installed package at $installed"
+          (cd "$GITHUB_WORKSPACE/orca" && find . -type d -name tests -print0 | tar -cf - --null -T -) \
+            | tar -C "$installed" -xf -
+          python -m pytest --pyargs orca -q -p no:cacheprovider
+
+      - name: Check that the package version matches the release
+        if: github.event_name == 'release'
+        env:
+          PRERELEASE: ${{ github.event.release.prerelease }}
+        run: |
+          # Read the version from the installed wheel's metadata, from outside the
+          # checkout, so the check binds the tag to the artifact being uploaded rather
+          # than to the source tree (or the egg-info that the build leaves behind)
+          cd "$RUNNER_TEMP"
+          expected="${GITHUB_REF_NAME#v}"
+          actual="$(python -c 'from importlib.metadata import version; print(version("orca"))')"
+          echo "tag $GITHUB_REF_NAME expects version $expected; installed wheel reports $actual"
+          if [ "$expected" != "$actual" ]; then
+            echo "::error::the package version does not match the release tag"
+            exit 1
+          fi
+          if [[ "$actual" =~ (a|b|rc|dev)[0-9]*$ ]]; then
+            if [ "$PRERELEASE" != "true" ]; then
+              echo "::error::$actual is a pre-release version, but the GitHub release is not marked as a pre-release"
+              exit 1
+            fi
+          elif [ "$PRERELEASE" = "true" ]; then
+            echo "::error::$actual is a final version, but the GitHub release is marked as a pre-release"
+            exit 1
+          fi
+
+      - name: Save distributions
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        with:
+          name: dist
+          path: dist/
+          if-no-files-found: error
+
+  publish:
+    name: Publish to PyPI
+    needs: build
+    if: github.event_name == 'release'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    environment:
+      name: pypi
+      url: https://pypi.org/project/orca/
+    permissions:
+      id-token: write  # for Trusted Publishing; no other access is needed
+
+    steps:
+      - name: Download distributions
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+        with:
+          name: dist
+          path: dist/
+
+      - name: Upload to PyPI
+        uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
+        with:
+          # Files upload one at a time and PyPI refuses re-uploads of a filename, so
+          # let a re-run after a partial failure finish the release instead of aborting
+          skip-existing: true

CONTRIBUTING.md

@@ -1,41 +1,101 @@
-Contributing to Orca
-====================
+Thanks for using Orca! 
 
-Style
------
+This is an open source project that's part of the Urban Data Science Toolkit. Development and maintenance is a collaboration between UrbanSim Inc and other contributors.
 
-- Python code should follow the [PEP 8 Style Guide][pep8].
-- Python docstrings should follow the [NumPy documentation format][numpydoc].
+You can contact Sam Maurer, the lead maintainer, at `maurer@urbansim.com`.
 
-### Imports
 
-Imports should be one per line.
-Imports should be grouped into standard library, third-party,
-and intra-library imports. `from` import should follow "regular" `imports`.
-Within each group the imports should be alphabetized.
-Here's an example:
+## If you have a problem:
 
-```python
-import sys
-from glob import glob
+- Take a look at the [open issues](https://github.com/UDST/orca/issues) and [closed issues](https://github.com/UDST/orca/issues?q=is%3Aissue+is%3Aclosed) to see if there's already a related discussion
 
-import numpy as np
+- Open a new issue describing the problem -- if possible, include any error messages, the operating system and version of python you're using, and versions of any libraries that may be relevant
 
-import package.module as module
-from package.othermod import useful_func
-```
 
-Imports of scientific Python libraries should follow these conventions:
+## Feature proposals:
 
-```python
-import matplotlib.pyplot as plt
-import numpy as np
-import pandas as pd
-import scipy as sp
-```
+- Take a look at the [open issues](https://github.com/UDST/orca/issues) and [closed issues](https://github.com/UDST/orca/issues?q=is%3Aissue+is%3Aclosed) to see if there's already a related discussion
 
+- Post your proposal as a new issue, so we can discuss it (some proposals may not be a good fit for the project; see the project scope in the README)
 
-Thanks!
 
-[pep8]: http://legacy.python.org/dev/peps/pep-0008/
-[numpydoc]: https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
+## Contributing code:
+
+- Create a new branch of `UDST/orca`, or fork the repository to your own account
+
+- Make your changes, following the existing styles for code and inline documentation: [PEP 8](https://peps.python.org/pep-0008/) for code (checked with Ruff in CI) and the [NumPy format](https://numpydoc.readthedocs.io/en/latest/format.html) for docstrings
+
+- Add tests if possible! They live in [`orca/tests`](https://github.com/UDST/orca/tree/main/orca/tests) and [`orca/utils/tests`](https://github.com/UDST/orca/tree/main/orca/utils/tests), and run with `pytest` from the repository root
+
+- Open a pull request to the `UDST/orca` main branch, including a writeup of your changes -- take a look at some of the closed PR's for examples
+
+- Automated checks run on every pull request: the test suite against the oldest and newest supported Python and dependency versions (on Linux, macOS, and Windows), a code quality check, a package build, and a documentation build
+
+- Current maintainers will review the code, suggest changes, and hopefully merge it!
+
+
+## Updating the version number:
+
+- Each pull request that changes substantive code should increment the development version number, e.g. from `1.9.dev0` to `1.9.dev1`, so that users know exactly which version they're running
+
+- It works best to do this just before merging (in case other PR's are merged first, and so you know the release date for the changelog and documentation)
+
+- The version number lives in `orca/__init__.py` (`pyproject.toml` reads it from there); `docs/source/conf.py` repeats the latest production release
+
+- Please also add a section to `HISTORY.rst` describing the changes!
+
+
+## Updating the documentation: 
+
+- See instructions in `docs/README.md`
+
+
+## Preparing a release:
+
+- Make a new branch for release prep
+
+- Update the version number and changelog
+  - `HISTORY.rst`
+  - `orca/__init__.py`
+  - `docs/source/conf.py`
+
+- Make sure all the tests are passing, and check if updates are needed to `README.rst` or to the documentation
+
+- Open a pull request to the main branch, and merge it once the checks pass
+
+- Publish the release on GitHub: create a tag on the merge commit named with a `v` prefix (e.g. `v1.9` for version `1.9`), and use the changelog text as the release notes. Publishing the release starts the `Publish` workflow described below
+
+- For anything more than a trivial release, do a dry run first with a release candidate: set the version to e.g. `1.9rc1`, tag it `v1.9rc1`, and mark the GitHub release as a pre-release. Pip ignores pre-releases unless asked for them (`pip install --pre orca==1.9rc1`), and the Conda Forge bots ignore them too, so this is a safe way to test the whole process. There's no need to delete the release candidate from PyPI afterward. Then repeat with the final version number
+
+- After the release, rebuild and publish the documentation (see `docs/README.md`)
+
+
+## Distributing a release on PyPI (for pip installation):
+
+- Publishing is automated by the `Publish` GitHub Actions workflow (`.github/workflows/publish.yml`), which runs when a release is published on GitHub. It builds the source distribution and wheel, checks them with `twine check --strict`, installs the wheel and runs the test suite against it, confirms that the package version matches the release tag, and then uploads the files to PyPI using [Trusted Publishing](https://docs.pypi.org/trusted-publishers/), so no PyPI credentials are stored on GitHub
+
+- The upload step runs in the repository's `pypi` deployment environment, which requires approval from a maintainer: once the build job succeeds, the workflow pauses until a reviewer approves the deployment from the workflow run page. Merging to `main` never publishes anything
+
+- Check https://pypi.org/project/orca/ for the new version, and try `pip install orca` in a fresh environment
+
+- One-time setup, in case it needs to be repeated: a PyPI owner of the project registers the trusted publisher at https://pypi.org/manage/project/orca/settings/publishing/ with owner `UDST`, repository `orca`, workflow `publish.yml`, and environment `pypi`; and a repository admin creates the `pypi` environment at https://github.com/UDST/orca/settings/environments with required reviewers
+
+- Manual fallback, if the workflow can't be used: register an account at https://pypi.org with two-factor authentication enabled, ask one of the current maintainers to add you to the project, and create an API token scoped to the Orca project. Then `pip install build twine`, delete any old files in `dist/`, and run `python -m build`, `twine check --strict dist/*`, and `twine upload dist/*`, entering `__token__` as the username and the token as the password
+
+
+## Distributing a release on Conda Forge (for conda installation):
+
+- The [conda-forge/orca-feedstock](https://github.com/conda-forge/orca-feedstock) repository controls the Conda Forge release, including which GitHub users have maintainer status for the feedstock
+
+- Conda Forge bots usually detect new releases on PyPI within a few hours and open a pull request to update the feedstock, which a current feedstock maintainer needs to review and merge
+
+- Before merging, check that the run requirements and the Python version floor in `recipe/meta.yaml` still match `pyproject.toml`; the bot only updates the version and hash. Additional changes can be pushed to the bot's branch, for example to update the requirements or the list of maintainers
+
+- You can also fork the feedstock and open a pull request manually, updating the version number and pasting the new hash of the `.tar.gz` file uploaded to PyPI (available on the pypi.org project page). It seems like this must be done from a personal account (not a group account like UDST) so that the bots can be granted permission for automated cleanup
+
+- Check https://anaconda.org/conda-forge/orca for the new version (may take a few minutes for it to appear)
+
+
+## Branch policy
+
+The `main` branch is the default integration and release branch. All new pull requests should target `main`. (Before v1.9, development happened on a `dev` branch that was periodically merged into `main` for releases; as of the v1.9 release cycle, `dev` is retired.)

HISTORY.rst

@@ -11,6 +11,8 @@ Next release
 * Modernize Python packaging and continuous integration: tests run on
   Linux, macOS, and Windows, and every pull request also builds and
   installs the package and builds the documentation.
+* Automate publishing to PyPI from GitHub releases using Trusted
+  Publishing, and document the contribution and release process.
 
 v1.8
 ====

orca/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "1.9.dev1"
+__version__ = "1.9.dev2"
 
 from .orca import *