Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,15 +67,14 @@ workspace members, and a plain `uv sync` installs only this package's own
dependencies, leaving theirs out.

`uv.lock` is committed: it pins the exact commit of libby, instrumentctl and
every other git dependency, and a release tag pins the lock. `uv sync` installs
what it records. To move the dependencies forward, update it in a PR:
every other git dependency. `uv sync` installs what it records. To move the dependencies forward, update it in a PR:

```bash
uv lock --upgrade && uv sync --all-packages
```

Hosts run releases rather than `main`; see [docs/install.md](docs/install.md) for how to cut
one.
Hosts run `main` and `install.sh` installs the checkout as it is; see
[docs/install.md](docs/install.md).

## Submodules

Expand Down
95 changes: 28 additions & 67 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,59 +99,54 @@ activating anything.

## What install.sh does

It moves the checkout to the newest release, bootstraps a pinned `uv`, runs
`uv sync --locked --all-packages` into `/opt/hispec/venv`, then hands over to
`hispec install`. That command comes from
By default it installs `/opt/hispec/app` as it is, edits included, and runs no
git command that changes it. It bootstraps a pinned `uv`, runs
`uv sync --locked --all-packages` into `/opt/hispec/venv` from the checkout's
`uv.lock`, then hands over to `hispec install`. That command comes from
[instrumentctl](https://github.com/CaltechOpticalObservatories/instrumentctl#host-setup),
whose README describes every step. It is shared with other instruments, so
HISPEC only supplies `src/hispec/cli/instrument.toml`.

## Releases
## Updating a host

Hosts run a release, never whatever `main` holds. A release is an annotated
`hispec-X.Y` tag on `main`, and its message is the release notes. The tag
pins everything: hispec's own code, its submodules, and through the committed
`uv.lock` the exact commit of libby, instrumentctl and every other git
dependency. A change merged into libby therefore reaches a host only when a
hispec release locks it in.
Hosts run `main`. Pulling is a separate step from installing:

To cut one:

1. If dependencies should move, open a PR that updates them and commits
`uv.lock`, so the new commits are reviewed like any change. libby is pinned
to a release tag in `pyproject.toml`, so moving it means changing that tag;
`uv lock --upgrade` moves the others.
2. Once it is on `main`, tag and push:

```bash
git tag -a hispec-0.2 -m "What changed, for the release notes"
git push origin hispec-0.2
```
```bash
git -C /opt/hispec/app pull
git -C /opt/hispec/app submodule update --init --recursive
```

Pushing the tag runs the Release workflow, which creates the GitHub release
from the tag message and posts it to Slack. It refuses a lightweight tag, so
always tag with `-a`. To announce a tag again, run the workflow by hand from
the Actions tab.
`hispec update` does the same, and keeps a driver that runs an unmerged hotfix
on its branch. hispec and every driver are installed editable, so the pulled
code takes effect at the next `hispec restart <instance>`. Re-run `install.sh`
only after `pyproject.toml` or `uv.lock` changes.

`install.sh` picks the newest `hispec-*` tag. `--ref` installs something else
on purpose, a branch or an older tag, e.g. to try `main` on one host:
`install.sh --checkout <ref>` fetches and checks out a branch or tag, with its
submodules, before installing, and refuses a checkout with uncommitted edits:

```bash
sudo /opt/hispec/app/systemd/install.sh --ref main
sudo /opt/hispec/app/systemd/install.sh --checkout main
```

It refuses to move a checkout that has uncommitted edits; save them with
`hispec hotfix` first.
An edit made on the host, in hispec or in a driver, is installed as it is.
Keep it with `hispec hotfix "what you fixed"`, which sends a driver edit to
the driver's own repository; see
[instrumentctl](https://github.com/CaltechOpticalObservatories/instrumentctl#hotfixes).

Dependencies move only through a PR that updates `uv.lock`, so the new commits
are reviewed like any change. libby is pinned to a release tag in
`pyproject.toml`, so moving it means changing that tag; `uv lock --upgrade`
moves the others.

## Re-running it

`install.sh` is idempotent and reports only what it changes. Re-run it:

- to move the host to a new release;
- after a pull that changed `pyproject.toml` or `uv.lock`;
- to enrol another operator: `sudo /opt/hispec/app/systemd/install.sh carol`;
- to repair anything `hispec doctor` flags.

Adding a daemon that the running release already contains does not need it:
Adding a daemon that the checkout already contains does not need it:
`hispec deploy` covers that, with no `sudo`.

## Changing an extra's code on a host
Expand Down Expand Up @@ -186,40 +181,6 @@ request from the clone; pushing needs a GitHub account with access, since the
deploy key only reads. To go back to the GitHub build, remove the clone and
re-run `install.sh`.

## Running a developer-managed checkout

Sometimes a host has to run code that is not in a release yet, edited and
pulled by a developer working on that host. `--local` makes `install.sh` leave
the hispec checkout alone and install it as it is:

```bash
sudo /opt/hispec/app/systemd/install.sh --local
```

In this mode `install.sh` never pulls. It runs no `git fetch`, `pull`,
`checkout` or `submodule update` in `/opt/hispec/app`, and it accepts
uncommitted edits. Bringing in new commits is up to the developer:

```bash
git -C /opt/hispec/app pull
git -C /opt/hispec/app submodule update --init --recursive
```

Dependencies are still installed exactly: the venv is synced with
`uv sync --locked` from that checkout's `uv.lock`. hispec itself is installed
editable, so an edit under `src/hispec` takes effect at the next
`hispec restart <instance>`. Re-run `install.sh` only after `pyproject.toml` or
`uv.lock` changes.

The choice is recorded in `/etc/hispec/source`, so later runs, such as
enrolling another operator, stay local without the flag. To go back to
releases, commit or discard the edits, then pass a ref, which also clears the
record:

```bash
sudo /opt/hispec/app/systemd/install.sh --ref hispec-X.Y
```

## Development checkouts

Your own clone for development is separate from all of this. Build it with
Expand Down
16 changes: 8 additions & 8 deletions docs/operations/systemd.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,11 +248,10 @@ doctor` reports anything deployed somewhere it does not belong. A host learns
its own name from `/etc/hispec/host`, which the installer writes.

Also add the row to the table in [Deployed instances](#deployed-instances) and
to the inventory in {doc}`../architecture/daemons`. Merge it, cut a release
(see {doc}`../install`), then on the host:
to the inventory in {doc}`../architecture/daemons`. Merge it, then on the host:

```bash
sudo /opt/hispec/app/systemd/install.sh # moves the host to the new release
git -C /opt/hispec/app pull
hispec deploy <name>
```

Expand Down Expand Up @@ -417,17 +416,18 @@ The config is read at startup. Edit `/etc/hispec/<name>.yaml`, then

### Picking up code changes

Hosts run releases, so a change reaches a host once it is in a release (see
Hosts run `main`, installed editable, so pulling is enough (see
{doc}`../install`):

```bash
sudo /opt/hispec/app/systemd/install.sh # moves the host to the newest release
git -C /opt/hispec/app pull
git -C /opt/hispec/app submodule update --init --recursive
hispec restart <name> # or: hispec restart fei
hispec deploy --new # if the release added instances
hispec deploy --new # if the pull added instances
```

`install.sh --ref main` runs `main` instead, to try a change before it is
released.
Re-run `sudo /opt/hispec/app/systemd/install.sh` when the pull changed
`pyproject.toml` or `uv.lock`.

## Deployed instances

Expand Down
97 changes: 26 additions & 71 deletions systemd/install.sh
Original file line number Diff line number Diff line change
@@ -1,29 +1,23 @@
#!/usr/bin/env bash
# Host setup for running the hispec daemons under systemd. For admins / IT.
# Run as root. Safe to re-run: it converges the host onto the latest release.
# Run as root. Safe to re-run.
#
# sudo ./systemd/install.sh # set the host up
# sudo ./systemd/install.sh alice bob # ... and enrol two operators
# sudo HISPEC_OPS_USERS="alice bob" ./systemd/install.sh # same thing
# sudo ./systemd/install.sh --extra scripts # ... with an optional extra
# sudo ./systemd/install.sh --ref main # install a branch or tag instead
# sudo ./systemd/install.sh --local # install the checkout as it is
# sudo ./systemd/install.sh --checkout main # fetch and check out a branch or tag first
#
# A release is an annotated hispec-X.Y tag. The checkout moves to the newest
# one, and the venv is synced from that release's uv.lock, so every git
# dependency is the exact commit the release pinned.
# By default it installs /opt/hispec/app as it is, edits included, and runs no
# git command that changes it. --checkout fetches and checks out the ref, with
# its submodules, refusing a checkout with edits. The venv is synced from the
# checkout's uv.lock, so every git dependency is the exact commit it records.
#
# Extras are recorded in /etc/hispec/extras, and every later run syncs them
# again. Some follow the host role: feiserver always gets atcam, the tracking
# camera module, which any other role is refused; feiserver and hispecserver
# always get atlas, the viewer.
#
# --local leaves the hispec checkout to whoever develops on this host: no
# fetch, pull, checkout or submodule update, and edits are allowed. It installs
# whatever is checked out, dependencies from that checkout's uv.lock. It is
# recorded in /etc/hispec/source, so later runs stay local until one passes
# --ref. See docs/install.md.
#
# A clone in /opt/hispec/extra/ replaces the GitHub build of its extra, for
# changing that code on this host; see docs/install.md. Remove the clone and
# re-run to go back.
Expand All @@ -47,7 +41,6 @@ REPO_DIR="${HISPEC_REPO_DIR:-/opt/hispec/app}"
VENV_DIR="${HISPEC_VENV_DIR:-/opt/hispec/venv}"
EXTRAS_FILE="${HISPEC_ETC_DIR:-/etc/hispec}/extras"
HOST_FILE="${HISPEC_ETC_DIR:-/etc/hispec}/host"
SOURCE_FILE="${HISPEC_ETC_DIR:-/etc/hispec}/source"
CAMERA_EXTRA=atcam
CAMERA_ROLE=feiserver
VIEWER_EXTRA=atlas
Expand Down Expand Up @@ -90,36 +83,18 @@ if [[ -f "$EXTRAS_FILE" ]]; then
mapfile -t EXTRAS <"$EXTRAS_FILE"
fi
ROLE_ARG=""
REF=""
LOCAL_ARG=""
CHECKOUT_REF=""
while (($#)); do
case $1 in
--extra) EXTRAS+=("$2"); shift 2 ;;
--local) LOCAL_ARG=1; shift ;;
--ref) REF=$2; shift 2 ;;
--ref=*) REF=${1#*=}; shift ;;
--checkout) CHECKOUT_REF=$2; shift 2 ;;
--checkout=*) CHECKOUT_REF=${1#*=}; shift ;;
--host-role) ROLE_ARG=$2; INSTALL_ARGS+=("$1" "$2"); shift 2 ;;
--host-role=*) ROLE_ARG=${1#*=}; INSTALL_ARGS+=("$1"); shift ;;
*) INSTALL_ARGS+=("$1"); shift ;;
esac
done

# Local source: asked for now, or recorded by an earlier --local run. An
# explicit --ref goes back to installing releases.
if [[ -n ${HISPEC_LOCAL_SOURCE:-} ]]; then
LOCAL_ARG=1
fi
if [[ -n $LOCAL_ARG && -n $REF ]]; then
echo "error: --local installs the checkout as it is; it cannot be combined with --ref." >&2
exit 1
fi
LOCAL=""
if [[ -n $LOCAL_ARG ]]; then
LOCAL=1
elif [[ -z $REF && -f $SOURCE_FILE && $(tr -d '[:space:]' <"$SOURCE_FILE") == local ]]; then
LOCAL=1
fi

# Same precedence as `hispec install`, which never overwrites a recorded role
ROLE=""
if [[ -f "$HOST_FILE" ]]; then
Expand Down Expand Up @@ -162,51 +137,39 @@ for extra in "${EXTRAS[@]}"; do
EXTRA_FLAGS+=(--extra "$extra")
done

# Fetching the release and syncing the lock reach the private camera repo
# Fetching with --checkout and syncing the lock reach the private camera repo
if [[ -f "$DEPLOY_KEY" ]]; then
export GIT_SSH_COMMAND="ssh -i $DEPLOY_KEY -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new"
fi

if [[ -n $LOCAL ]]; then
# The developer owns the checkout: no git command here changes it
LOCAL_DESC=$(git -C "$REPO_DIR" symbolic-ref --short -q HEAD ||
git -C "$REPO_DIR" rev-parse --short HEAD)
if [[ -n $(git -C "$REPO_DIR" status --porcelain --untracked-files=no) ]]; then
LOCAL_DESC+=", modified"
fi
echo "installing local checkout $REPO_DIR ($LOCAL_DESC); not pulling, that is up to you"
else
# Refuse to move a checkout with edits in it: checking out the release would
# carry them along or stop halfway. `hispec hotfix` saves them first.
if [[ -n $CHECKOUT_REF && -z ${HISPEC_INSTALL_CHECKED_OUT:-} ]]; then
# Checking out over edits would carry them along or stop halfway
if [[ -n $(git -C "$REPO_DIR" status --porcelain --untracked-files=no) ]]; then
echo "error: $REPO_DIR has uncommitted edits:" >&2
echo "error: $REPO_DIR has uncommitted edits, so --checkout will not move it:" >&2
git -C "$REPO_DIR" status --short --untracked-files=no >&2
echo 'Save them with: hispec hotfix "what you fixed" (or discard: hispec hotfix --undo)' >&2
exit 1
fi
git -C "$REPO_DIR" fetch --quiet --tags --force origin
if [[ -z $REF ]]; then
REF=$(git -C "$REPO_DIR" tag --list 'hispec-*' --sort=-version:refname | head -n1)
if [[ -z $REF ]]; then
echo "error: no hispec-* release tag yet; pass --ref main to install the branch." >&2
exit 1
fi
fi
if git -C "$REPO_DIR" show-ref --verify --quiet "refs/remotes/origin/$REF"; then
if git -C "$REPO_DIR" show-ref --verify --quiet "refs/remotes/origin/$CHECKOUT_REF"; then
# A branch: follow its tip on origin, refusing rather than dropping local commits
git -C "$REPO_DIR" checkout --quiet "$REF"
git -C "$REPO_DIR" merge --quiet --ff-only "origin/$REF"
git -C "$REPO_DIR" checkout --quiet "$CHECKOUT_REF"
git -C "$REPO_DIR" merge --quiet --ff-only "origin/$CHECKOUT_REF"
else
git -C "$REPO_DIR" checkout --quiet --detach "$REF"
git -C "$REPO_DIR" checkout --quiet --detach "$CHECKOUT_REF"
fi
git -C "$REPO_DIR" submodule update --quiet --init --recursive
# The checkout may have replaced this very script; carry on in the release's copy
if [[ -z ${HISPEC_INSTALL_CHECKED_OUT:-} ]]; then
HISPEC_INSTALL_CHECKED_OUT=1 exec "$REPO_DIR/systemd/install.sh" "${ORIGINAL_ARGS[@]}" --ref "$REF"
fi
echo "installing $REF ($(git -C "$REPO_DIR" rev-parse --short HEAD))"
# The checkout may have replaced this very script; carry on in its copy
HISPEC_INSTALL_CHECKED_OUT=1 exec "$REPO_DIR/systemd/install.sh" "${ORIGINAL_ARGS[@]}"
fi

CHECKOUT_DESC=$(git -C "$REPO_DIR" symbolic-ref --short -q HEAD ||
git -C "$REPO_DIR" rev-parse --short HEAD)
if [[ -n $(git -C "$REPO_DIR" status --porcelain --untracked-files=no) ]]; then
CHECKOUT_DESC+=", modified"
fi
echo "installing $REPO_DIR ($CHECKOUT_DESC)"

# Pinned, so re-running the installer cannot resolve with a different uv
if [[ "$("$UV_BIN" --version 2>/dev/null)" != "uv $UV_VERSION"* ]]; then
curl -LsSf "https://astral.sh/uv/$UV_VERSION/install.sh" |
Expand Down Expand Up @@ -245,12 +208,4 @@ if ((${#EXTRAS[@]})); then
mkdir -p "$(dirname "$EXTRAS_FILE")"
printf '%s\n' "${EXTRAS[@]}" >"$EXTRAS_FILE"
fi
# Remembered so a later plain run (enrolling an operator, say) stays local
if [[ -n $LOCAL ]]; then
mkdir -p "$(dirname "$SOURCE_FILE")"
echo local >"$SOURCE_FILE"
else
rm -f "$SOURCE_FILE"
fi

exec "$VENV_DIR/bin/hispec" install "${INSTALL_ARGS[@]}"
Loading