The spack extension of C2SM and MCH

Spack is the package manager used by C2SM and MeteoSwiss to install and deploy software on supercomputers, local machines and the cloud.

Documentations

Infos about c2sm-supported software and machines (legacy versions)

• spack-c2sm v0.22.2.3

Note

Docs for other spack-c2sm versions are available at https://C2SM.github.io/spack-c2sm/VERSION_NUMBER, e.g. v0.20.1.4.

General infos about spack

• Official spack v1.1.1 (version currently used by spack-c2sm)

The first 3 numbers of every spack-c2sm version match with the version of spack it uses as a submodule.

Workflow for Users

We suggest local/individual spack instances and the use of spack environments.

Clone the repository

git clone --depth 1 --recurse-submodules --shallow-submodules https://github.com/C2SM/spack-c2sm.git

Tip

Use -b <tag> (e.g. -b v1.1.1-0.0) to target a specific version/branch instead of the default branch.

Setup the shell environment and optionally specify an upstream, where spack will look for installed software, i.e.

. spack-c2sm/setup-env.sh # Generic
. spack-c2sm/setup-env.sh euler # For Euler
. spack-c2sm/setup-env.sh /user-environment # For Santis
. spack-c2sm/setup-env.sh /mch-environment/v8 # For Balfrin / MeteoSwiss

Sourcing this file will put the spack command in your PATH, set up your MODULEPATH to use Spack’s packages, and add other useful shell integration for certain commands, environments, and modules. For bash, it also sets up tab completion. (source: spack docu)

Optionally activate a spack environment

spack env activate <path_to_env>

and starts exploring

spack info <package>
spack spec <spec>

and installing

spack install <spec>
spack dev-build <spec>

packages.

Updating spack-c2sm is in the hands of the user.

git pull
git submodule update --recursive

Before an update we advice to clean your instance

spack uninstall -a
spack clean -a
rm -rf ~/.spack

After an update we advice to rebuild packages, preferably in a new shell so that no outdated shell variables are retained.

Workflow for Local Spack Development (Linting & Testing)

Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate

Install pinned dev tools

pip install --upgrade pip
pip install -r requirements/dev.txt
pip install -r requirements/test.txt

Install pre-commit hooks locally (optional)

pre-commit install

Hooks run automatically on git commit

Run all hooks manually (recommended before push)

pre-commit run --all-files

• Lints and auto-fixes safe issues (like unused imports)
• Checks YAML, formatting, and other configured hooks
• Shows errors/warnings for anything that cannot be auto-fixed

Testing strategy

For more information on the different types of tests see the README in the test directory.

PR tests

To merge PRs into the main branch we require short (taking ~5 minutes) unit and integration tests to run successfully. These tests are ran via a GitHub Action that runs automatically after any push to a PR branch.

Nightly tests

Each day at 3 a.m. system tests run on Säntis (via CSCS CI) and Balfrin (via Jenkins) to test the installation of different packages with Spack on the systems. Since these tests may take longer to complete we decided to not require them to merge PRs, however special comments can still be used on PRs to trigger them nonetheless:

• For Säntis: type cscs-ci run in a PR comment.
• For Balfrin: type launch jenkins in a PR comment.

Releases/Tags

Release tags are created by the Spack-Admin GitHub Team as needed or upon request. The creation of a new release tag is coordinated within the admin team.

When creating a new release tag please follow the naming convention below:

v${SPACK_VERSION}-${SPACK_C2SM_VERSION}

Where SPACK_VERSION corresponds to the upstream Spack version this repo is based on, e.g. SPACK_VERSION=1.1.0. SPACK_C2SM_VERSION is ${MAJOR}(.${PATCH}):

• MAJOR increments with every regular spack-c2sm release based on a specific SPACK_VERSION, starting from 0. 0 marks the transitional release that adapts spack-c2sm to a new upstream Spack version, and MAJOR/PATCH both reset to 0 whenever SPACK_VERSION changes.
• PATCH is optional and only appears once a fix needs to be backported onto a release line that is no longer at the tip of main (see below), starting at 1 and incrementing from there for each new MAJOR. It's omitted entirely for ordinary releases off main.

So, for example, v1.1.0-0 is the transitional release for Spack v1.1.0, v1.1.0-1 is the next regular release, and v1.1.0-0.1 is a backported patch on top of the 1.1.0-0 line.

Patching an older release

If a fix needs to be backported onto a MAJOR line that's already in production (i.e. superseded on main), create (or reuse) a branch named:

release/v${SPACK_VERSION}-${MAJOR}

e.g. release/v1.1.0-1, branched from the v1.1.0-1 tag. Merge the fix(es) into that branch, then tag the result v${SPACK_VERSION}-${MAJOR}.${PATCH} (starting PATCH at 1 and incrementing from there), e.g. v1.1.0-1.1, then v1.1.0-1.2, etc. The release branch and PATCH component only come into play once a backport is actually needed — ordinary releases off main never carry a PATCH.

Note

Tags created before this scheme was introduced use the older vX.Y.Z.W format (a single incrementing number, e.g. v1.1.1.1, v0.22.2.5), occasionally with an extra .N hotfix suffix (e.g. v0.22.2.5.1). The vX.Y.Z-M.P scheme described above applies from the next release onward.

Command cheat sheet

	Command
Clone	git clone --depth 1 --recurse-submodules --shallow-submodules -b <branch/tag> https://github.com/C2SM/spack-c2sm.git
Update	git pull git submodule update --recursive
Load	. spack-c2sm/setup-env.sh to run without an upstream or . spack-c2sm/setup-env.sh /user-environment to use /user-environment as an upstream spack compiler find autodetects compilers spack external find --all autodetects externally installed packages
Clean	spack uninstall -a uninstalls all packages spack clean -a cleans all misc caches

Spec syntax: <package>@<version>%<compiler>+<variant> ~<variant>^<sub-package> +<sub-package-variant><compiler flags>

	Command
Find	spack find lists all installed packages. spack find <spec> lists all installed packages that match the spec.
Info	spack info <package>
Spec	spack spec <spec> concretizes abstract spec (unspecfied variant = any) Spack is not required to use the default of an unspecified variant. The default value is only a tiebreaker for the concretizer.
Install	spack install <spec>
Locate	spack location --install-dir <spec> prints location of all installs that satisfy the spec
Load env	spack load <spec> loads run environment
Activate env	spack env activate <env_name>
Deactivate env	spack deactivate