From 9d54aea94cd98d651c76fc2ddeeecbbe169421e6 Mon Sep 17 00:00:00 2001 From: Mike Langmayr Date: Thu, 5 Nov 2026 15:08:28 -0800 Subject: [PATCH 1/2] Install the checkout as it is and drop release tags --- README.md | 7 ++- docs/install.md | 94 ++++++++++---------------------------- docs/operations/systemd.md | 16 +++---- systemd/install.sh | 92 ++++--------------------------------- 4 files changed, 45 insertions(+), 164 deletions(-) diff --git a/README.md b/README.md index 9c0fce8..f8098f9 100644 --- a/README.md +++ b/README.md @@ -67,15 +67,14 @@ workspace members, and a plain `uv sync` installs only this package's own dependencies, leaving theirs out. `uv.lock` is committed: it pins the exact commit of libby, instrumentctl and -every other git dependency, and a release tag pins the lock. `uv sync` installs -what it records. To move the dependencies forward, update it in a PR: +every other git dependency. `uv sync` installs what it records. To move the dependencies forward, update it in a PR: ```bash uv lock --upgrade && uv sync --all-packages ``` -Hosts run releases rather than `main`; see [docs/install.md](docs/install.md) for how to cut -one. +Hosts run `main` and `install.sh` installs the checkout as it is; see +[docs/install.md](docs/install.md). ## Submodules diff --git a/docs/install.md b/docs/install.md index 4ef3003..306288f 100644 --- a/docs/install.md +++ b/docs/install.md @@ -99,59 +99,47 @@ activating anything. ## What install.sh does -It moves the checkout to the newest release, bootstraps a pinned `uv`, runs -`uv sync --locked --all-packages` into `/opt/hispec/venv`, then hands over to -`hispec install`. That command comes from +It installs `/opt/hispec/app` as it is, edits included, and runs no git command +that changes it. It bootstraps a pinned `uv`, runs +`uv sync --locked --all-packages` into `/opt/hispec/venv` from the checkout's +`uv.lock`, then hands over to `hispec install`. That command comes from [instrumentctl](https://github.com/CaltechOpticalObservatories/instrumentctl#host-setup), whose README describes every step. It is shared with other instruments, so HISPEC only supplies `src/hispec/cli/instrument.toml`. -## Releases +## Updating a host -Hosts run a release, never whatever `main` holds. A release is an annotated -`hispec-X.Y` tag on `main`, and its message is the release notes. The tag -pins everything: hispec's own code, its submodules, and through the committed -`uv.lock` the exact commit of libby, instrumentctl and every other git -dependency. A change merged into libby therefore reaches a host only when a -hispec release locks it in. - -To cut one: - -1. If dependencies should move, open a PR that updates them and commits - `uv.lock`, so the new commits are reviewed like any change. libby is pinned - to a release tag in `pyproject.toml`, so moving it means changing that tag; - `uv lock --upgrade` moves the others. -2. Once it is on `main`, tag and push: - - ```bash - git tag -a hispec-0.2 -m "What changed, for the release notes" - git push origin hispec-0.2 - ``` - -Pushing the tag runs the Release workflow, which creates the GitHub release -from the tag message and posts it to Slack. It refuses a lightweight tag, so -always tag with `-a`. To announce a tag again, run the workflow by hand from -the Actions tab. - -`install.sh` picks the newest `hispec-*` tag. `--ref` installs something else -on purpose, a branch or an older tag, e.g. to try `main` on one host: +Hosts run `main`. Pulling is a separate step from installing: ```bash -sudo /opt/hispec/app/systemd/install.sh --ref main +git -C /opt/hispec/app pull +git -C /opt/hispec/app submodule update --init --recursive ``` -It refuses to move a checkout that has uncommitted edits; save them with -`hispec hotfix` first. +`hispec update` does the same, and keeps a driver that runs an unmerged hotfix +on its branch. hispec and every driver are installed editable, so the pulled +code takes effect at the next `hispec restart `. Re-run `install.sh` +only after `pyproject.toml` or `uv.lock` changes. + +An edit made on the host, in hispec or in a driver, is installed as it is. +Keep it with `hispec hotfix "what you fixed"`, which sends a driver edit to +the driver's own repository; see +[instrumentctl](https://github.com/CaltechOpticalObservatories/instrumentctl#hotfixes). + +Dependencies move only through a PR that updates `uv.lock`, so the new commits +are reviewed like any change. libby is pinned to a release tag in +`pyproject.toml`, so moving it means changing that tag; `uv lock --upgrade` +moves the others. ## Re-running it `install.sh` is idempotent and reports only what it changes. Re-run it: -- to move the host to a new release; +- after a pull that changed `pyproject.toml` or `uv.lock`; - to enrol another operator: `sudo /opt/hispec/app/systemd/install.sh carol`; - to repair anything `hispec doctor` flags. -Adding a daemon that the running release already contains does not need it: +Adding a daemon that the checkout already contains does not need it: `hispec deploy` covers that, with no `sudo`. ## Changing an extra's code on a host @@ -186,40 +174,6 @@ request from the clone; pushing needs a GitHub account with access, since the deploy key only reads. To go back to the GitHub build, remove the clone and re-run `install.sh`. -## Running a developer-managed checkout - -Sometimes a host has to run code that is not in a release yet, edited and -pulled by a developer working on that host. `--local` makes `install.sh` leave -the hispec checkout alone and install it as it is: - -```bash -sudo /opt/hispec/app/systemd/install.sh --local -``` - -In this mode `install.sh` never pulls. It runs no `git fetch`, `pull`, -`checkout` or `submodule update` in `/opt/hispec/app`, and it accepts -uncommitted edits. Bringing in new commits is up to the developer: - -```bash -git -C /opt/hispec/app pull -git -C /opt/hispec/app submodule update --init --recursive -``` - -Dependencies are still installed exactly: the venv is synced with -`uv sync --locked` from that checkout's `uv.lock`. hispec itself is installed -editable, so an edit under `src/hispec` takes effect at the next -`hispec restart `. Re-run `install.sh` only after `pyproject.toml` or -`uv.lock` changes. - -The choice is recorded in `/etc/hispec/source`, so later runs, such as -enrolling another operator, stay local without the flag. To go back to -releases, commit or discard the edits, then pass a ref, which also clears the -record: - -```bash -sudo /opt/hispec/app/systemd/install.sh --ref hispec-X.Y -``` - ## Development checkouts Your own clone for development is separate from all of this. Build it with diff --git a/docs/operations/systemd.md b/docs/operations/systemd.md index 6307798..fbfc8a2 100644 --- a/docs/operations/systemd.md +++ b/docs/operations/systemd.md @@ -248,11 +248,10 @@ doctor` reports anything deployed somewhere it does not belong. A host learns its own name from `/etc/hispec/host`, which the installer writes. Also add the row to the table in [Deployed instances](#deployed-instances) and -to the inventory in {doc}`../architecture/daemons`. Merge it, cut a release -(see {doc}`../install`), then on the host: +to the inventory in {doc}`../architecture/daemons`. Merge it, then on the host: ```bash -sudo /opt/hispec/app/systemd/install.sh # moves the host to the new release +git -C /opt/hispec/app pull hispec deploy ``` @@ -417,17 +416,18 @@ The config is read at startup. Edit `/etc/hispec/.yaml`, then ### Picking up code changes -Hosts run releases, so a change reaches a host once it is in a release (see +Hosts run `main`, installed editable, so pulling is enough (see {doc}`../install`): ```bash -sudo /opt/hispec/app/systemd/install.sh # moves the host to the newest release +git -C /opt/hispec/app pull +git -C /opt/hispec/app submodule update --init --recursive hispec restart # or: hispec restart fei -hispec deploy --new # if the release added instances +hispec deploy --new # if the pull added instances ``` -`install.sh --ref main` runs `main` instead, to try a change before it is -released. +Re-run `sudo /opt/hispec/app/systemd/install.sh` when the pull changed +`pyproject.toml` or `uv.lock`. ## Deployed instances diff --git a/systemd/install.sh b/systemd/install.sh index b0a0d72..38e61ac 100755 --- a/systemd/install.sh +++ b/systemd/install.sh @@ -1,29 +1,21 @@ #!/usr/bin/env bash # Host setup for running the hispec daemons under systemd. For admins / IT. -# Run as root. Safe to re-run: it converges the host onto the latest release. +# Run as root. Safe to re-run. # # sudo ./systemd/install.sh # set the host up # sudo ./systemd/install.sh alice bob # ... and enrol two operators # sudo HISPEC_OPS_USERS="alice bob" ./systemd/install.sh # same thing # sudo ./systemd/install.sh --extra scripts # ... with an optional extra -# sudo ./systemd/install.sh --ref main # install a branch or tag instead -# sudo ./systemd/install.sh --local # install the checkout as it is # -# A release is an annotated hispec-X.Y tag. The checkout moves to the newest -# one, and the venv is synced from that release's uv.lock, so every git -# dependency is the exact commit the release pinned. +# It installs /opt/hispec/app as it is, edits included, and never changes it: +# pulling is up to whoever runs it. The venv is synced from the checkout's +# uv.lock, so every git dependency is the exact commit the lock records. # # Extras are recorded in /etc/hispec/extras, and every later run syncs them # again. Some follow the host role: feiserver always gets atcam, the tracking # camera module, which any other role is refused; feiserver and hispecserver # always get atlas, the viewer. # -# --local leaves the hispec checkout to whoever develops on this host: no -# fetch, pull, checkout or submodule update, and edits are allowed. It installs -# whatever is checked out, dependencies from that checkout's uv.lock. It is -# recorded in /etc/hispec/source, so later runs stay local until one passes -# --ref. See docs/install.md. -# # A clone in /opt/hispec/extra/ replaces the GitHub build of its extra, for # changing that code on this host; see docs/install.md. Remove the clone and # re-run to go back. @@ -41,13 +33,11 @@ # instrumentctl and is shared with other instruments. set -euo pipefail -ORIGINAL_ARGS=("$@") REPO_DIR="${HISPEC_REPO_DIR:-/opt/hispec/app}" VENV_DIR="${HISPEC_VENV_DIR:-/opt/hispec/venv}" EXTRAS_FILE="${HISPEC_ETC_DIR:-/etc/hispec}/extras" HOST_FILE="${HISPEC_ETC_DIR:-/etc/hispec}/host" -SOURCE_FILE="${HISPEC_ETC_DIR:-/etc/hispec}/source" CAMERA_EXTRA=atcam CAMERA_ROLE=feiserver VIEWER_EXTRA=atlas @@ -90,36 +80,15 @@ if [[ -f "$EXTRAS_FILE" ]]; then mapfile -t EXTRAS <"$EXTRAS_FILE" fi ROLE_ARG="" -REF="" -LOCAL_ARG="" while (($#)); do case $1 in --extra) EXTRAS+=("$2"); shift 2 ;; - --local) LOCAL_ARG=1; shift ;; - --ref) REF=$2; shift 2 ;; - --ref=*) REF=${1#*=}; shift ;; --host-role) ROLE_ARG=$2; INSTALL_ARGS+=("$1" "$2"); shift 2 ;; --host-role=*) ROLE_ARG=${1#*=}; INSTALL_ARGS+=("$1"); shift ;; *) INSTALL_ARGS+=("$1"); shift ;; esac done -# Local source: asked for now, or recorded by an earlier --local run. An -# explicit --ref goes back to installing releases. -if [[ -n ${HISPEC_LOCAL_SOURCE:-} ]]; then - LOCAL_ARG=1 -fi -if [[ -n $LOCAL_ARG && -n $REF ]]; then - echo "error: --local installs the checkout as it is; it cannot be combined with --ref." >&2 - exit 1 -fi -LOCAL="" -if [[ -n $LOCAL_ARG ]]; then - LOCAL=1 -elif [[ -z $REF && -f $SOURCE_FILE && $(tr -d '[:space:]' <"$SOURCE_FILE") == local ]]; then - LOCAL=1 -fi - # Same precedence as `hispec install`, which never overwrites a recorded role ROLE="" if [[ -f "$HOST_FILE" ]]; then @@ -162,50 +131,17 @@ for extra in "${EXTRAS[@]}"; do EXTRA_FLAGS+=(--extra "$extra") done -# Fetching the release and syncing the lock reach the private camera repo +# Syncing the lock reaches the private camera repo if [[ -f "$DEPLOY_KEY" ]]; then export GIT_SSH_COMMAND="ssh -i $DEPLOY_KEY -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new" fi -if [[ -n $LOCAL ]]; then - # The developer owns the checkout: no git command here changes it - LOCAL_DESC=$(git -C "$REPO_DIR" symbolic-ref --short -q HEAD || - git -C "$REPO_DIR" rev-parse --short HEAD) - if [[ -n $(git -C "$REPO_DIR" status --porcelain --untracked-files=no) ]]; then - LOCAL_DESC+=", modified" - fi - echo "installing local checkout $REPO_DIR ($LOCAL_DESC); not pulling, that is up to you" -else - # Refuse to move a checkout with edits in it: checking out the release would - # carry them along or stop halfway. `hispec hotfix` saves them first. - if [[ -n $(git -C "$REPO_DIR" status --porcelain --untracked-files=no) ]]; then - echo "error: $REPO_DIR has uncommitted edits:" >&2 - git -C "$REPO_DIR" status --short --untracked-files=no >&2 - echo 'Save them with: hispec hotfix "what you fixed" (or discard: hispec hotfix --undo)' >&2 - exit 1 - fi - git -C "$REPO_DIR" fetch --quiet --tags --force origin - if [[ -z $REF ]]; then - REF=$(git -C "$REPO_DIR" tag --list 'hispec-*' --sort=-version:refname | head -n1) - if [[ -z $REF ]]; then - echo "error: no hispec-* release tag yet; pass --ref main to install the branch." >&2 - exit 1 - fi - fi - if git -C "$REPO_DIR" show-ref --verify --quiet "refs/remotes/origin/$REF"; then - # A branch: follow its tip on origin, refusing rather than dropping local commits - git -C "$REPO_DIR" checkout --quiet "$REF" - git -C "$REPO_DIR" merge --quiet --ff-only "origin/$REF" - else - git -C "$REPO_DIR" checkout --quiet --detach "$REF" - fi - git -C "$REPO_DIR" submodule update --quiet --init --recursive - # The checkout may have replaced this very script; carry on in the release's copy - if [[ -z ${HISPEC_INSTALL_CHECKED_OUT:-} ]]; then - HISPEC_INSTALL_CHECKED_OUT=1 exec "$REPO_DIR/systemd/install.sh" "${ORIGINAL_ARGS[@]}" --ref "$REF" - fi - echo "installing $REF ($(git -C "$REPO_DIR" rev-parse --short HEAD))" +CHECKOUT_DESC=$(git -C "$REPO_DIR" symbolic-ref --short -q HEAD || + git -C "$REPO_DIR" rev-parse --short HEAD) +if [[ -n $(git -C "$REPO_DIR" status --porcelain --untracked-files=no) ]]; then + CHECKOUT_DESC+=", modified" fi +echo "installing $REPO_DIR ($CHECKOUT_DESC) as it is; pulling is up to you" # Pinned, so re-running the installer cannot resolve with a different uv if [[ "$("$UV_BIN" --version 2>/dev/null)" != "uv $UV_VERSION"* ]]; then @@ -245,12 +181,4 @@ if ((${#EXTRAS[@]})); then mkdir -p "$(dirname "$EXTRAS_FILE")" printf '%s\n' "${EXTRAS[@]}" >"$EXTRAS_FILE" fi -# Remembered so a later plain run (enrolling an operator, say) stays local -if [[ -n $LOCAL ]]; then - mkdir -p "$(dirname "$SOURCE_FILE")" - echo local >"$SOURCE_FILE" -else - rm -f "$SOURCE_FILE" -fi - exec "$VENV_DIR/bin/hispec" install "${INSTALL_ARGS[@]}" From f5e76cdde9552a163fa8559f52f93b9935743445 Mon Sep 17 00:00:00 2001 From: Mike Langmayr Date: Thu, 5 Nov 2026 17:42:27 -0800 Subject: [PATCH 2/2] Add install.sh --checkout to fetch and check out a ref before installing --- docs/install.md | 11 +++++++++-- systemd/install.sh | 37 ++++++++++++++++++++++++++++++++----- 2 files changed, 41 insertions(+), 7 deletions(-) diff --git a/docs/install.md b/docs/install.md index 306288f..5407068 100644 --- a/docs/install.md +++ b/docs/install.md @@ -99,8 +99,8 @@ activating anything. ## What install.sh does -It installs `/opt/hispec/app` as it is, edits included, and runs no git command -that changes it. It bootstraps a pinned `uv`, runs +By default it installs `/opt/hispec/app` as it is, edits included, and runs no +git command that changes it. It bootstraps a pinned `uv`, runs `uv sync --locked --all-packages` into `/opt/hispec/venv` from the checkout's `uv.lock`, then hands over to `hispec install`. That command comes from [instrumentctl](https://github.com/CaltechOpticalObservatories/instrumentctl#host-setup), @@ -121,6 +121,13 @@ on its branch. hispec and every driver are installed editable, so the pulled code takes effect at the next `hispec restart `. Re-run `install.sh` only after `pyproject.toml` or `uv.lock` changes. +`install.sh --checkout ` fetches and checks out a branch or tag, with its +submodules, before installing, and refuses a checkout with uncommitted edits: + +```bash +sudo /opt/hispec/app/systemd/install.sh --checkout main +``` + An edit made on the host, in hispec or in a driver, is installed as it is. Keep it with `hispec hotfix "what you fixed"`, which sends a driver edit to the driver's own repository; see diff --git a/systemd/install.sh b/systemd/install.sh index 38e61ac..c071d98 100755 --- a/systemd/install.sh +++ b/systemd/install.sh @@ -6,10 +6,12 @@ # sudo ./systemd/install.sh alice bob # ... and enrol two operators # sudo HISPEC_OPS_USERS="alice bob" ./systemd/install.sh # same thing # sudo ./systemd/install.sh --extra scripts # ... with an optional extra +# sudo ./systemd/install.sh --checkout main # fetch and check out a branch or tag first # -# It installs /opt/hispec/app as it is, edits included, and never changes it: -# pulling is up to whoever runs it. The venv is synced from the checkout's -# uv.lock, so every git dependency is the exact commit the lock records. +# By default it installs /opt/hispec/app as it is, edits included, and runs no +# git command that changes it. --checkout fetches and checks out the ref, with +# its submodules, refusing a checkout with edits. The venv is synced from the +# checkout's uv.lock, so every git dependency is the exact commit it records. # # Extras are recorded in /etc/hispec/extras, and every later run syncs them # again. Some follow the host role: feiserver always gets atcam, the tracking @@ -33,6 +35,7 @@ # instrumentctl and is shared with other instruments. set -euo pipefail +ORIGINAL_ARGS=("$@") REPO_DIR="${HISPEC_REPO_DIR:-/opt/hispec/app}" VENV_DIR="${HISPEC_VENV_DIR:-/opt/hispec/venv}" @@ -80,9 +83,12 @@ if [[ -f "$EXTRAS_FILE" ]]; then mapfile -t EXTRAS <"$EXTRAS_FILE" fi ROLE_ARG="" +CHECKOUT_REF="" while (($#)); do case $1 in --extra) EXTRAS+=("$2"); shift 2 ;; + --checkout) CHECKOUT_REF=$2; shift 2 ;; + --checkout=*) CHECKOUT_REF=${1#*=}; shift ;; --host-role) ROLE_ARG=$2; INSTALL_ARGS+=("$1" "$2"); shift 2 ;; --host-role=*) ROLE_ARG=${1#*=}; INSTALL_ARGS+=("$1"); shift ;; *) INSTALL_ARGS+=("$1"); shift ;; @@ -131,17 +137,38 @@ for extra in "${EXTRAS[@]}"; do EXTRA_FLAGS+=(--extra "$extra") done -# Syncing the lock reaches the private camera repo +# Fetching with --checkout and syncing the lock reach the private camera repo if [[ -f "$DEPLOY_KEY" ]]; then export GIT_SSH_COMMAND="ssh -i $DEPLOY_KEY -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new" fi +if [[ -n $CHECKOUT_REF && -z ${HISPEC_INSTALL_CHECKED_OUT:-} ]]; then + # Checking out over edits would carry them along or stop halfway + if [[ -n $(git -C "$REPO_DIR" status --porcelain --untracked-files=no) ]]; then + echo "error: $REPO_DIR has uncommitted edits, so --checkout will not move it:" >&2 + git -C "$REPO_DIR" status --short --untracked-files=no >&2 + echo 'Save them with: hispec hotfix "what you fixed" (or discard: hispec hotfix --undo)' >&2 + exit 1 + fi + git -C "$REPO_DIR" fetch --quiet --tags --force origin + if git -C "$REPO_DIR" show-ref --verify --quiet "refs/remotes/origin/$CHECKOUT_REF"; then + # A branch: follow its tip on origin, refusing rather than dropping local commits + git -C "$REPO_DIR" checkout --quiet "$CHECKOUT_REF" + git -C "$REPO_DIR" merge --quiet --ff-only "origin/$CHECKOUT_REF" + else + git -C "$REPO_DIR" checkout --quiet --detach "$CHECKOUT_REF" + fi + git -C "$REPO_DIR" submodule update --quiet --init --recursive + # The checkout may have replaced this very script; carry on in its copy + HISPEC_INSTALL_CHECKED_OUT=1 exec "$REPO_DIR/systemd/install.sh" "${ORIGINAL_ARGS[@]}" +fi + CHECKOUT_DESC=$(git -C "$REPO_DIR" symbolic-ref --short -q HEAD || git -C "$REPO_DIR" rev-parse --short HEAD) if [[ -n $(git -C "$REPO_DIR" status --porcelain --untracked-files=no) ]]; then CHECKOUT_DESC+=", modified" fi -echo "installing $REPO_DIR ($CHECKOUT_DESC) as it is; pulling is up to you" +echo "installing $REPO_DIR ($CHECKOUT_DESC)" # Pinned, so re-running the installer cannot resolve with a different uv if [[ "$("$UV_BIN" --version 2>/dev/null)" != "uv $UV_VERSION"* ]]; then