Status: accepted for version 0.1 on 2026-11-11.
pullgate lets a Linux host change only through its own updates. The host pulls a git repo by itself. It applies a new commit only after its owner taps Apply on a Telegram question that names that exact commit.
Read THREAT-MODEL.md next. It says what this design protects, and where it stops.
- instance: one repo that one host follows, with one config file
/etc/pullgate/NAME.toml, one state directory and one lock. A host can have several instances.NAMEmatches[a-z0-9][a-z0-9-]{0,31}. - tip: the commit that the configured branch points to after a fetch.
- applied SHA: the last commit that root applied with success. It is empty before the first apply.
- held SHA: the last SHA whose apply failed, while no newer apply has worked (section 10).
- question: one Telegram message that names one full SHA and has two buttons, Apply and Skip.
- code: a random one-time code of 128 bits (22 characters, base64url). The buttons carry it instead of the SHA.
- approval: a small file that the listener writes when the owner taps a button. It holds the code and the button (apply or skip). Nothing else.
- tree: a clean, read-only copy of the files of one commit, written by pullgate itself from git objects.
- check request: a small file that the listener writes when the owner
sends
/checkor/check NAME. It holds no data from the message. It starts a normal run now, instead of at the next timer (sections 8 and 9). - pause file:
/var/lib/pullgate/NAME/paused. While it exists, the instance does nothing: no run, no apply. - update question: a question that names one release of pullgate itself: its version and the SHA-256 of its tarball (section 15).
These rules come from issue #1. Each one has tests (section 14).
- The gate lives on the host, never in the repo. What runs, who may
approve and which paths get a warning come only from the host config:
/etc/pullgate/NAME.tomland/etc/pullgate/listener.toml. pullgate never reads any setting from the repo. A commit cannot change the gate. - One exact commit. The question shows the full SHA once, on its last line. Apply applies only that SHA, even when a newer commit arrives in between. The tap is bound to the question (its code), not to the text.
- No code from the new commit runs before Apply. Before the tap,
pullgate only fetches the commit and reads it with git. The optional
checkandsummarycommands may read the tree as data. They run as an unprivileged user (check_user), never as root, on a read-only tree. - Only one process holds the bot token: the listener. It runs as its own
unprivileged user. It only connects out (
getUpdateslong-poll), it opens no port. It accepts a tap only from the configured owner user ID, in the configured chat, on an open question. All it can do is write an approval for that open question. The approval carries only the code. Root looks up the SHA in its own state. The only other thing it can write is a check request for/checkof the owner. That request holds no data, and it only starts a normal run sooner (sections 8 and 9). - Root does the apply. It acts only on an approval whose code matches a
question that root recorded itself, and only within the time limit of that
question. By default it moves forward only: the new SHA must descend from
the applied SHA. Going to an older or unrelated commit needs the explicit
setting
allow_older = true. Before the first apply there is no applied SHA, so any tip may be asked. - Root never runs git in a directory another user can write. Before every git command, root checks the git directory and every directory above it (section 6). git runs with hooks off and without system or global config. pullgate writes the tree itself from git objects: no filters, no submodules, no LFS.
| process | user | started by | holds |
|---|---|---|---|
pullgate run NAME |
root | a timer (pullgate-run@NAME.timer), or a path unit (pullgate-check@NAME.path) when a check request exists |
the clone, the state, the deploy key |
pullgate apply NAME |
root | a path unit (pullgate-apply@NAME.path) when the inbox is not empty, or the auto approval exists |
the state |
pullgate listen |
pullgate (own user) |
a service (pullgate-listen.service) |
the bot token, its list of open questions |
check, summary |
check_user (default pullgate-check) |
pullgate run |
nothing |
apply, health |
root | pullgate apply |
what the owner's command needs |
pullgate status NAME |
root | the owner | nothing |
pullgate update |
root | a timer (pullgate-update.timer, daily), or a path unit (pullgate-update.path) for /update and a tap |
the binary, the units, its own state (section 15) |
Root and the listener never talk directly. They use a spool directory
(section 8). Both units start the same pullgate-run@NAME.service, so a
run on /check has the same sandbox, the same lock and the same rules as a
run on the timer. systemd never starts one unit twice at once.
The run unit is sandboxed (read-only system, private /tmp, no new
privileges). The apply unit runs the owner's apply command as root, so it
can be less strict; each example sets what its apply needs.
| path | owner, mode | what |
|---|---|---|
/etc/pullgate/NAME.toml |
root, 0600 or 0644 | the instance config |
/etc/pullgate/listener.toml |
root, 0640 group listener, or 0644 | the listener config |
| the deploy key | root, 0600 | read-only SSH deploy key |
| the known hosts file | root, 0644 | the pinned SSH host key of the git server |
| the bot token file | the listener user, 0600 (or a systemd credential) | the bot token |
/var/lib/pullgate/NAME/ |
root, 0711 | the state directory |
/var/lib/pullgate/NAME/repo.git |
root, 0700 | the bare clone |
/var/lib/pullgate/NAME/state.json |
root, 0600 | the state (section 9) |
/var/lib/pullgate/NAME/lock |
root, 0600 | the lock (flock) |
/var/lib/pullgate/NAME/paused |
root | the pause file |
/var/lib/pullgate/NAME/auto-approval |
root, 0600 | the approval that auto mode writes (section 13) |
/var/lib/pullgate/NAME/trees/ |
root, 0711 | the trees of one process (section 6) |
/var/spool/pullgate/ and /var/spool/pullgate/NAME/ |
root, 0755 | the spool |
/var/spool/pullgate/NAME/outbox |
root, group listener, 0770 (no setgid) | messages from root to the listener |
/var/spool/pullgate/NAME/outbox/*.json |
root, group listener, 0640 | one message |
/var/spool/pullgate/NAME/inbox |
listener, 0700 | approvals from the listener to root |
/var/spool/pullgate/NAME/check |
listener, 0700 | check requests from the listener to root |
/var/lib/pullgate-listener/state.json |
listener, 0600 | the listener's open questions |
/var/lib/pullgate/_update/ and .../tmp/ |
root, 0700 | the state, the lock and the downloads of pullgate update (section 15) |
/var/spool/pullgate/_update/ |
as NAME/ above |
the outbox, inbox and check request of the updates |
/usr/local/bin/.pullgate.prev |
root | the old binary during an update (a hard link) |
The paths can be changed with --config-dir, --state-dir and
--spool-dir, for tests.
Checks of the gate files. pullgate refuses to start when the instance
config, the deploy key, the known hosts file, or the program (the first
word) of a command is a symlink, is not owned by root, or can be written by
group or others; or when a directory above one of them fails the check of
section 6. A symlink on the way to a program (such as /bin -> usr/bin)
is allowed only when it is owned by root and the directory that holds it
passes the check too. The deploy key must also not be readable by group or others. The
listener checks listener.toml and its token file the same way (the token
file must be owned by the listener user). In tests, "root" means the user
that runs the test.
Files in directories that the listener can write. Root never follows a
path that the listener can change. It opens the spool directories with
os.OpenRoot (no symlinks out of the directory), fixes their owner and mode
through the open directory, creates files in them with
O_CREAT|O_EXCL|O_NOFOLLOW under a random temporary name and renames them
in the same directory. /var/spool/pullgate and NAME/ are owned by root
and pass the check of section 6, so the listener cannot swap outbox,
inbox or check for a symlink.
No setgid bit on the outbox. Root gives each outbox file the group of
the listener user itself: it creates the file, then calls fchown on the
open file (group only) and fchmod to 0640, all before the rename. So the
listener never sees an outbox file with another group, and no directory
needs the setgid bit. This matters because pullgate-run@.service sets
RestrictSUIDSGID=yes, which refuses every chmod that sets a setuid or
setgid bit. Version 0.1.0 made the outbox 2770, and its first run failed
under the unit (issue #19). The other choice was to remove
RestrictSUIDSGID=yes from the run unit. We keep it: the run unit runs as
root with CAP_FOWNER, and it must never make a setuid or setgid file.
Ensure also takes the setgid bit off an outbox made by 0.1.0. Removing a
bit is allowed under RestrictSUIDSGID=yes. Because root is not in the
listener's group, the fchown needs CAP_CHOWN: every unit that runs
pullgate run or pullgate apply must keep it. Without it, root cannot
write to the outbox, and the owner gets no question and no reply.
mode = "ask" # "ask" (default), "auto" or "off"
on_failure = "rollback" # "rollback" or "pause"; the code default is "pause"
host = "example-host" # shown in each message; default: the host name
[repo]
url = "ssh://git@git.example.com/owner/config-repo.git"
branch = "main"
deploy_key = "/etc/pullgate/NAME.key"
known_hosts = "/etc/pullgate/NAME.known_hosts"
[commands]
# argv lists, no shell. The first word is an absolute path.
check = ["/usr/local/bin/check-config", "--strict"] # optional
summary = ["/usr/local/bin/config-diff"] # optional
apply = ["/usr/local/sbin/install-config"] # required
health = ["/usr/local/sbin/health-check"] # optional
check_user = "pullgate-check"
check_timeout = "5m"
apply_timeout = "30m"
[watch]
# A warning in the question for each changed path that matches.
# "*" matches inside one path part, "**" matches any number of parts.
paths = ["policy/**", "*.toml"]
[limits]
questions_per_day = 10 # default 10
approval_ttl = "12h" # default 12h, at most 7 days
max_failed_tries = 1 # default 1; counts only with on_failure = "rollback"
max_files = 100000 # default 100000 files in one tree
max_tree_bytes = 1073741824 # default 1 GiB in one tree
allow_older = false # default false (rule 5)
refuse_symlinks = false # default false; true refuses a commit with a symlink that points out of the tree
[listener]
user = "pullgate" # the user of the listener; owns the inboxUnknown keys are an error. An unknown mode or on_failure is an error. A
missing apply is an error. Every error names the file, the key and the fix.
There is no default for on_failure in the example configs: each example
sets it. The code default is pause, because it is the safe choice.
The repo URL. ssh://… and user@host:path use SSH (section 6). A path
that starts with /, or a file:// URL, is a local repo. root runs git in
it too (git-upload-pack), so a local repo must pass the same check as the
clone (section 6): owned by root, nobody else can write it. Other URLs
(https://, git://, relative paths) are refused. The key and known hosts
paths may hold only A-Za-z0-9._/-, because git gives GIT_SSH_COMMAND to
a shell.
Before each git command, root checks the git directory:
- The git directory and each directory above it up to
/must be owned by root (the running user) and not writable by group or others. - One exception for a directory above: it may be writable by others when it
is owned by root and has the sticky bit (like
/tmp). Then no other user can rename or remove our directory in it. - Every file and directory inside the git directory must be owned by root, must not be writable by group or others, and must be a regular file or a directory (no symlink, no special file).
If a check fails, pullgate stops with an error that names the path, and runs no git.
Each git command runs:
- with an empty environment, plus
PATH,HOMEset to the state directory,GIT_CONFIG_NOSYSTEM=1,GIT_CONFIG_GLOBAL=/dev/null,GIT_TERMINAL_PROMPT=0,GIT_ASKPASS=,LC_ALL=CandGIT_SSH_COMMAND(below); - with these settings before the subcommand:
core.hooksPath=/dev/null,core.fsmonitor=false,protocol.allow=never,protocol.ssh.allow=always,protocol.file.allow=always(for local repos, see section 5),submodule.recurse=false,fetch.recurseSubmodules=false,core.attributesFile=/dev/null,transfer.fsckObjects=true,gc.auto=0andmaintenance.auto=false; - with
--git-dirset, never in a work tree; - in its own process group, with a timeout that kills the group, and a cap on its output.
The fetch is:
git fetch --no-tags --no-recurse-submodules --no-write-fetch-head \
--no-auto-gc -- URL +refs/heads/BRANCH:refs/pullgate/tip
GIT_SSH_COMMAND is ssh -F /dev/null -i KEY -o IdentitiesOnly=yes -o IdentityAgent=none -o UserKnownHostsFile=KNOWN_HOSTS -o GlobalKnownHostsFile=/dev/null -o StrictHostKeyChecking=yes -o BatchMode=yes. The URL comes from the host config. It must not start
with -, so git cannot read it as an option. The branch must be a plain
name ([A-Za-z0-9][A-Za-z0-9._/-]*, no ..).
Then git rev-parse --verify refs/pullgate/tip^{commit} gives the tip.
pullgate keeps refs/pullgate/applied on the applied SHA, so git never
loses it after a force push. No gc runs.
The other reads are git merge-base --is-ancestor, git rev-list --count,
git log -z --no-mailmap --format=%s, git diff-tree -r -z --no-renames --name-only and git ls-tree -r -z.
The tree. pullgate never uses git checkout or git archive, because
both apply .gitattributes (filters, export-subst, export-ignore). It
reads git ls-tree -r -l -z --full-tree SHA and then each blob with
git cat-file --batch. It writes the files itself:
- Each path part must not be empty,
.,..or.git(in any case), and must not hold a NUL. A bad path stops the export. - Regular files are written 0444 (0555 when the git mode is 100755), with
O_CREAT|O_EXCL|O_NOFOLLOW, inside directories that pullgate made in this export. A repeated name, or a path below a file or a symlink of the tree, stops the export. - Symlinks (mode 120000) are written as symlinks. pullgate never follows a
symlink of a tree. A symlink whose target is absolute or holds
..gets a warning in the question; withrefuse_symlinks = trueit stops the export. A command that reads the tree must not follow symlinks blindly. - Submodules (mode 160000) are skipped and listed in the question as a warning.
- The export stops when the tree has more than
max_filesentries or more thanmax_tree_bytesbytes. - The export goes to
trees/.tmp-RANDOM. Only when it is complete does it get its final name. Then each directory gets mode 0555, and the top directory gets 0750 with groupcheck_user.
Fresh trees. A tree belongs to one process. pullgate run exports the
trees for check and summary; pullgate apply exports fresh trees for
apply, health and a rollback. Each process removes its trees when it
ends, and removes trees that an earlier process left (after a crash) when it
starts. So a tree that a command changed, or a half-written tree, is never
used again.
- Read the config. Take the lock (
flock, no wait). If another run or apply holds it, log "busy" and exit 0; but when a check request waits, wait for the lock instead (section 8). If the pause file exists, log "paused" and exit 0. Ifmode = "off", log "off" and exit 0. When a check request waits, take it first (section 8); then paused and off send a reply ("NAME is paused" or "NAME is off"), and do not fetch. - If the state shows an apply in progress (section 10), write the pause file, report, and exit.
- Make sure the clone and the spool directories exist with the right owners and modes (section 4). Fetch (section 6).
- Expire: each question in the state whose time is up is removed. If it was open, root asks the listener to edit it ("expired") and marks its SHA expired.
- Decide:
- tip = applied SHA: nothing to do. If a question is open, close it ("the branch is back to the applied commit").
- tip = the SHA of the open question: nothing to do.
- tip is the held SHA, or was asked before and the question expired or
was skipped, or its check failed, or it was refused, or it failed
max_failed_triestimes: nothing to do.pullgate status NAME --ask-againclears this mark, and also closes an open question for the tip (section 13). - tip is not a descendant of the applied SHA (an older commit, a force
push, a rewritten branch) and
allow_olderis false: refuse. Send one notice for this SHA and mark it refused. Whenallow_olderis true, ask, with a warning line at the top: "This commit is not a descendant of the applied commit." With no applied SHA, every tip is allowed. - the questions in the last 24 hours reached
questions_per_day: log it, send one notice per day, and do not ask.
- Export the tree of the tip, and the tree of the applied SHA
(
PULLGATE_OLD_TREE). - Run
checkascheck_user(section 11). If it fails, mark the SHA "check failed", send a notice with the end of its output, and do not ask. The same SHA is not checked again. In that output the tree paths (PULLGATE_TREE,PULLGATE_OLD_TREE) becomeNEWandOLD. The notice keeps the last lines, at most 10 and at most 1500 characters, so the reason (most tools print it last) is shown. A line longer than 300 characters keeps its start and its end, with…between. - Run
summaryascheck_user, and build the message (section 12). - If a question is open, close it: root marks it superseded in its state, and asks the listener to edit it ("a newer commit replaced this question"). A tap that the listener took before the edit still counts for the old SHA (rule 2); section 9 says how.
ask: make a new code, record the question in the state (synced to disk) and write the question to the outbox.auto: record the question in the state the same way, but instead of a message, write the code to the auto approval file (section 13).
Each instance has /var/spool/pullgate/NAME/outbox and .../inbox.
Outbox (root writes, the listener reads and removes). One JSON file per
message, named SEQ.json with a 20-digit sequence number (nanoseconds),
written to a random .tmp-* name and renamed. The listener handles them in
name order and skips names that start with .. Kinds:
ask:question(an id, unique per instance),code,expires(time),text(HTML, ready to send);edit:question,text: replace the text of that question and remove its buttons. The listener forgets the code of the question;notice:text: a new message;reply:question,text: a reply to that question, or a new message when the listener does not know the question.
A file the listener cannot read or parse is removed and logged. The owner
also gets one notice per round for such files: "an outbox message of
instance NAME could not be read and was removed", with the command
pullgate status NAME --ask-again. The notice shows no part of the file,
because it may be a question that the owner never saw. If Telegram refuses
a message, the listener tries again on the next round, and drops it after
one day with a log line.
Inbox (the listener writes, root reads and removes). One file per tap,
SEQ.json, written to a .tmp-* name and renamed, at most 1 KiB, with
{"code": "...", "action": "apply" | "skip"}. Unknown fields are an error.
Root opens each file through os.OpenRoot with O_NOFOLLOW, needs a
regular file owned by the listener user, reads at most 1 KiB, and removes the
file before it acts. A bad file is logged and removed. Root removes .tmp-*
files older than one minute.
pullgate-apply@NAME.path watches the inbox (DirectoryNotEmpty=) and the
auto approval (PathExists=), and starts pullgate apply NAME. The apply
service has a start limit, so a stuck file cannot start it in a loop.
Check requests (the listener writes, root reads and removes). The
listener writes /var/spool/pullgate/NAME/check/request for /check
(section 9): a .tmp-* file renamed to that one fixed name, so many
requests before root reads them are one file. The content is always the
six bytes check\n. It holds nothing from the message: not the text, not
a SHA, not a code. A check request can only start a normal run. It is not
an approval, and pullgate apply never reads this directory.
pullgate-check@NAME.path watches the file check/request
(PathExists=; another file there starts nothing) and starts pullgate-run@NAME.service, the unit of the timer. pullgate run,
with the lock held (a run that had to wait for the lock, and then finds
no request, stops: another run took the request):
- Reads and removes every file in the directory, with the checks of an
approval: through
os.OpenRootwithO_NOFOLLOW, a regular file owned by the listener user, at most 64 bytes. It must also have the namerequestand exactly the contentcheck\n. Root removes each file before it acts. A bad file is logged and removed; it is not a request. A run that finds only bad files goes on as a normal (timer) run, with no reply and no mark against the limit. - The limit, kept by root in its own
state.json(check_at), not by the listener: at most one run per minute (CheckInterval) starts from a check request. When the last one is less than a minute away (before or after, so a clock jump back of less than a minute does not pass it), the run removes the request and exits without a fetch. It sends one reply per minute: "A check ran less than a minute ago, so this /check was dropped." (check_note). - Paused or off: reply so, do not fetch.
- Else the normal run (section 7). After it, a reply says what it found,
unless the question or a notice says it already: "No new commit. The
applied commit is SHA.", "the question for SHA is still open", "SHA was
handled before (mark)", "the last apply of SHA failed and it is held",
or, in
automode, "New commit SHA … it is applied now, without a tap" (section 13). A run that fails sends "The check failed. See the journal of the host", without the error text.
A request while the lock is held. A run started by the timer exits
"busy" when another process holds the lock. A run that sees a check request
waits for the lock instead (flock with a wait), so the request is not
lost and the path unit does not start the run again and again. The lock
makes sure that two runs, or a run and an apply, never run at once. The
run unit's TimeoutStartSec caps the wait.
The start limit. Each check request starts the run unit. The path unit
has a trigger limit (TriggerLimitBurst=10 in 60 seconds), which stops a
loop when a run fails before it removes the request. If the limit is hit,
systemd stops the path unit until the owner runs systemctl reset-failed pullgate-check@NAME.path and starts it again. The run unit has no start
limit (StartLimitIntervalSec=0), so many requests never stop the timer.
So that /check alone never hits the limit, the listener also writes at
most one request per instance in 10 seconds (CheckGap, kept in
memory). Root does not trust this: its own limit is the one above.
Root keeps its questions in state.json:
{ "id": 7, "code": "…", "sha": "…", "base": "…", "asked": time,
"expires": time, "open": true }
base is the applied SHA when the question was asked. At most one
question is open. A superseded question stays in the list ("open": false)
until it expires, so that a tap that the listener took before the close
still counts (rule 2). The listener forgets a code when it handles the
edit that closes the question, so it cannot take a tap for it after that.
Every list entry is removed when it is answered or expires.
The listener calls deleteWebhook at start and then getUpdates with
allowed_updates = ["callback_query", "message"]. It keeps the last 200
update_ids that it handled in its state file. An update with one of them
was handled before: it is ignored, a tap and a message alike. A lower ID
that is not in the list still counts, because Telegram may start the IDs
again at a random number after a quiet week. It accepts a tap only when all of
these hold, else it answers "This question is closed." and writes nothing:
from.idisowner_id, andmessage.chat.idischat_id;- the data is
a:CODEors:CODE, and CODE is a code that it holds; - the message ID is the message ID of that question;
- the message was not edited (no
edit_date), and its text is the text that Telegram gave back when the listener sent it (a SHA-256 of it is kept). So a token thief who edits the text above real buttons gets no tap counted (THREAT-MODEL.md T1); - the question has not expired by the listener's clock;
- the question has no tap yet. After the first tap the listener forgets the code, so a second tap does nothing.
Then it writes the approval, edits the question ("Apply tapped. Waiting for the host." or "Skipped."), and answers the tap.
/check. The listener writes a check request (section 8) only when all
of these hold, else it writes nothing:
from.idisowner_id, andchat.idischat_id;- it is a new message, not an edited one (no
edited_message, noedit_date); - the text is exactly
/check(a request for every instance ininstances) or exactly/check NAME, with one space and a NAME that is ininstances. Nothing before, after or between: no second word, no new line, no@. So/check@otherbotdoes nothing. The listener does not know its own bot name, so it refuses/check@for its own name too (in a private chat Telegram sends/checkwithout it).
Then it replies "Checking NAME…" to the message. A second /check for the
same instance within 10 seconds writes nothing and gets "a check was asked
a few seconds ago" (section 8, the start limit). When a request cannot be
written (for example the host never ran pullgate for NAME, so the directory
does not exist), it replies that the request was not saved. To the owner in
the chat, a text that starts with /check but does not match gets the
usage text, which lists the instances and never repeats the message.
/help. When the text is exactly /help, from owner_id, in
chat_id, not edited, the listener replies with a fixed list: the
commands (/check, /check NAME, /update, /help), the instance names from
instances, what Apply and Skip do, and the root commands for the host
(pullgate status NAME, --pause, --resume, --ask-again), at most
once per minute. The
listener does not know the mode or the pause file of an instance (root's
files), so the list points to pullgate status NAME for them. Any other
text of the owner in the chat that is not a command (does not start with
/check) gets one reply, "Not a command. Send /help for the list.", at
most once per minute (in memory), so a stuck client or a long paste cannot
fill the chat. A message of another user, or in another chat, or an
edited message, gets no answer, so the bot does not tell strangers that it
exists. The listener
never logs the text of a message: only why it was ignored and the sender
ID.
Root (pullgate apply NAME) takes the lock (with a wait). If the pause
file exists or mode = "off", it removes the approvals, replies "ignored:
the instance is paused" (or off) to each question it knows, and exits.
Else it reads the approvals in name order. For each one:
- find the question with that code in its own state, with a constant-time compare. No match: log and ignore.
- check
asked <= now < expiresby its own clock. Else: drop the question, reply "expired", ignore. - remove the question from its state (synced), so the code works once.
skip: mark the SHA skipped, reply "skipped", next approval.apply: take the SHA from its own state, never from the approval. Check again that the SHA descends from the applied SHA (unlessallow_older, or there is no applied SHA). Else reply "refused" and ignore. Then apply it (section 10).
pullgate apply does at most one apply. Approvals after it stay in the
inbox for the next start. After an apply, every open question whose base
is no longer the applied SHA is closed ("the applied commit changed"), and
its SHA mark is cleared, so the next run asks again with a correct summary.
The steps, with the state synced to disk at each arrow:
- Write
in_progress = {phase: "apply", sha, old_sha}→ - Export fresh trees (section 6). Run
applyas root in the tree of the SHA (environment in section 11). Thenhealth, if it is set. - Both succeed: the applied SHA becomes this SHA,
refs/pullgate/appliedmoves,in_progressand the held SHA are cleared, all SHA marks are dropped (a SHA that is not a descendant is refused again anyway) → report "applied". - One fails: count a failed try for this SHA →. Then by
on_failure:rollback, with an applied SHA: writein_progress = {phase: "rollback", sha, old_sha}→. Runapplywith the last good SHA, thenhealth. If that works, clearin_progress→ and report "failed, rolled back to OLD". If it fails too, write the pause file, clearin_progress→ and report "failed, the rollback failed too, the instance is paused". With no applied SHA there is nothing to roll back to: hold the SHA, as withpause.pause: do not roll back and do not write the pause file. The SHA becomes the held SHA. Clearin_progress→. It is not tried again, every report andpullgate statussay that the last apply failed, and a newer commit is still asked, with the line "The last apply (SHA) failed." at the top. So the owner can fix it forward with a new commit. The applied SHA does not change after a failure. State moves forward only after success.
Tries. Every try needs a new question and a new tap (in auto mode, a
new run). With rollback, a SHA with fewer than max_failed_tries failed
tries is asked again on the next run; with max_failed_tries failed tries
it is not asked again until a new commit (or --ask-again). With pause,
a failed SHA is held at once and the count does not matter.
A crash leaves in_progress in the state. The next run or apply cannot
know what happened on the host. It writes the pause file, marks the SHA
failed and reports "the apply (or rollback) of SHA was cut off; check the
host, then run pullgate status NAME --resume".
Changes to the gate. An approved commit may change the gate itself: the
host config, the units, or the pullgate binary, when the apply command
installs them. pullgate reads its config once when a process starts, so a
rollback uses the commands of the process that did the apply. Put the gate
files of the repo in the watch list, so the question warns about them.
Commands are argv lists. The first word must be an absolute path. No shell. They run with an empty environment plus:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binPULLGATE_INSTANCE: the instance namePULLGATE_SHA: the full SHA of the commitPULLGATE_OLD_SHA: the applied SHA, empty on the first applyPULLGATE_TREE: the tree ofPULLGATE_SHA, read-onlyPULLGATE_OLD_TREE: the tree ofPULLGATE_OLD_SHA, read-only, empty on the first applyPULLGATE_CHANGED_FILES: the path of a read-only file with the paths that differ between the two SHAs, each one ended by a NUL byte (all paths on the first apply). Read it withxargs -0. A file, because a list of paths can be too long for the environment and a path can hold a newline.PULLGATE_FAILED_SHA: the held SHA, if there is one. The host may be part way to it, not atPULLGATE_OLD_SHA.PULLGATE_ROLLBACK:1during a rollback, else empty.
In a rollback, PULLGATE_SHA and PULLGATE_TREE are the last good SHA
and its tree; PULLGATE_OLD_SHA and PULLGATE_OLD_TREE are the SHA that
just failed and its tree; PULLGATE_CHANGED_FILES lists the paths that
differ between them. So apply must be safe to run again on an older tree.
Their working directory is PULLGATE_TREE. Their umask is 022. Each has a timeout
(check_timeout, apply_timeout); on timeout the whole process group is
killed and the command counts as failed. pullgate keeps at most 64 KiB of
their output. The output of apply and health goes to the journal; the
report shows only the exit status. The output of check (its last lines)
and of summary goes into Telegram messages, after the cleaning of
section 12.
check and summary (rule 3) run as check_user, with its own group and
no other groups. When pullgate runs as root, check_user must not be root
(uid 0), and must not be the listener user or share its group, because a
check that is taken over could then write approvals: pullgate refuses such
a config. When pullgate does not run as root
(tests, a trial), it cannot change user, so check_user must be the user
that runs it. A process of check_user could leave its process group and
live on. So after each check and summary, root runs kill -KILL -1 as
check_user, which stops every process of that user; and when the run unit
ends, systemd stops everything left in its cgroup. Use one check_user per
instance. The run unit's sandbox (read-only system, private /tmp, no new
privileges) also applies to them.
The question is HTML (parse_mode = HTML), with link previews off:
pullgate on <b>HOST</b>, instance <b>NAME</b>
⚠ The last apply (SHA) failed. (only when a SHA is held)
⚠ This commit is not a descendant of … (only with allow_older)
Apply this commit?
<code>SHORT SHA</code>
First apply: there is no applied commit yet. All N files are listed.
or: Since <code>SHORT APPLIED SHA</code>:
⚠ Watched path changed: <code>path</code>
⚠ Symlink out of the tree: <code>path</code> → <code>target</code>
⚠ Submodule skipped: <code>path</code>
Commits (N):
• <code>subject</code>
Changed files (N):
• <code>path</code>
Summary:
<pre>output of the summary command</pre>
Full: <code>FULL SHA</code>
SHAs. Every message shows a SHA as its first 12 characters, in
<code>. The question, as it is sent, shows the full SHA once, on its last line
("Full:"), so it can be copied and compared. That line stays after a cut.
(After a tap, the listener adds its note below it.) Logs,
the state and pullgate status keep the full SHA. Root applies the full
SHA from its own state, never a SHA from the text.
Text from the repo is only shown as code. Text from the repo, the
commit, or a command that ran on it (paths, subjects, symlink targets,
submodule paths, the output of summary and check, and errors that may
hold any of them) is put into a message only by message.Code (one line in
<code>) or message.Output (lines in <pre>). Telegram makes no link,
mention, hashtag or bot command of text inside <code> or <pre>, so a
file named README.md or login.example.com stays plain text. The escape
function is not exported, so other code cannot put repo text into a message
another way; a test also checks the gate code for its own HTML. Before that,
the text is cleaned: control and invisible characters (also right-to-left
marks) become ? (newlines stay only in the summary output), each line is
cut to 200 characters, and then it is HTML-escaped. At most 50 commit
subjects are read. Every message and every edit is sent with link previews
off.
The message must fit one Telegram message (4096 characters). pullgate cuts
it to 3800 characters, counted on the escaped text in UTF-16 units: the head
(host, instance, SHA, the warnings) always stays, and when lines are left
out the last line says how many ("… 120 more lines left out"). The cut is on
whole lines, so it never splits an HTML tag or entity. The <pre> block is
closed after a cut.
ask(the default): as above.auto: no question.pullgate runrecords the question in its state and writes its code to/var/lib/pullgate/NAME/auto-approval(root-owned, 0600). The path unit startspullgate apply, which handles it like an approval from the inbox, then removes it, and reports. The apply needs the same checks (code, time, forward only). For development hosts./checkon anautoinstance starts the run as the timer does, so a new commit is applied at once, without a tap. This is a choice: an auto instance applies every new commit within one timer period anyway, so/checkonly makes it sooner. It cannot apply a commit that the timer would not apply. The reply says "it is applied now, without a tap". Who does not want this keeps the instance inaskmode.off:pullgate runandpullgate applydo nothing.
pullgate status NAME prints the mode, the applied SHA, the open question,
the held SHA, the pause file and an apply in progress. Flags:
--ask-again: clear the marks of the last fetched tip (expired, skipped, check failed, refused, failed, held), so the next run asks again. If a question for the tip is open (for example its message was lost), root removes it from its state, then asks the listener to close its message. The next run asks again with a new question ID and a new code. An approval for the old code finds no question and does nothing, even one that the listener wrote before the close;--pause: write the pause file;--resume: remove the pause file and clear an apply in progress.
The listener (pullgate listen --config /etc/pullgate/listener.toml):
token_file = "/etc/pullgate/bot-token" # or $CREDENTIALS_DIRECTORY/bot-token
owner_id = 123456789 # the Telegram user ID of the owner
chat_id = 123456789 # the chat (the private chat with the owner)
instances = ["config-repo", "playbooks"]
spool_dir = "/var/spool/pullgate" # default
state_file = "/var/lib/pullgate-listener/state.json"
api_url = "https://api.telegram.org" # only tests change itSeveral instances share one listener when they share one bot: two
getUpdates loops on one token conflict.
All processes log with log/slog to stderr; the units set
SyslogIdentifier=pullgate, so each step is in the journal. The bot token,
the deploy key and the code never appear in a log line or an error.
Each rule and each limit has a test. The tests use a fake Telegram server
(net/http/httptest) and a local bare git repo as the remote. They need
git. On CI, a missing git is a failure, not a skip.
The end-to-end test (test/e2e, build tag e2e) builds the binary, makes a
bare repo with two commits, runs pullgate run, the listener against the
fake Telegram server, a tap, pullgate apply, and checks that the second
commit was applied. Then it sends /check and checks the request, the run
and the reply "No new commit".
The tests of install.sh (install_linux_test.go, at the root) run the
script with /bin/sh against a fake release: a fake curl, uname, gh
and pullgate. They never use the network. They show that the script
installs nothing when a check fails (section 16), and that it installs when
all checks pass. CI also runs a pinned ShellCheck on the script.
test/units/first-run.sh runs on CI as root, on the throwaway runner. It
installs the binary and the units of contrib/systemd/ as the README
says, with no drop-in. It runs the first pullgate run under the real
pullgate-run@.service, checks the owners and modes of section 4, then
runs pullgate listen under the real pullgate-listen.service against a
fake Telegram on 127.0.0.1, and checks that the listener sent the
question and removed it from the outbox. At the end it runs
systemctl enable on the four units of the README (a list in the
script), and checks that each is enabled. It also follows the README's
update from 0.1.0 for the listener: the old unit with add-wants, then the
new unit and reenable. So a unit setting that breaks a run or the listener, or a unit
that cannot be enabled, fails CI. Without root, CI also runs
systemd-analyze verify on every unit, and test/units/install-sections.sh
checks that each unit the README tells you to enable has an [Install]
section with WantedBy=.
internal/selfupdate tests pullgate update with the real built-in
install.sh against a fake release (a fake curl, gh and uname, and
fake binaries): the question, the install, a wrong SHA-256, a failed
attestation, a wrong version (each before the question and on Apply), a
tarball that changed after the question, an older version, a newer release
while a question is open, Skip, an expired question, a listener that does
not start (rollback), a crash at each step, the daily notice (once, again
each day for a security release, off with check = false), and the unit
update (a changed unit, a new unit, a masked unit, a disabled unit, a
drop-in).
test/units/full-loop.sh (CI job install-loop, on each pull request
and on main) is the path a user takes, as root on the throwaway runner:
- It runs the install commands from README.md itself: each install
block has a comment line
<!-- install-test:NAME -->(binary,setup,start) before itsshblock, and the script runs the block as written, withNAMEset toconfig-repo.test/units/readme-block.sh --checkfails when a block is missing, or when an install command (useradd,groupadd,install -X,chown,chmod,cp contrib/systemd,systemctl enable) is in an unmarked fenced block. So the README cannot drift from what CI runs. - A local bare repo is the remote. A fake Telegram on
127.0.0.1(test/units/faketg.py --tap) records each message and taps Apply on each question as the owner. - A good commit:
pullgate rununderpullgate-run@.serviceasks (withcheckandsummaryas the check user), the listener (userpullgate) sends the question and writes the approval,pullgate-apply@.serviceapplies it, andpullgate statusshows it as applied. - It restarts the listener and checks that it polls again.
- A commit whose apply fails: the rollback runs, and its notice reaches the fake Telegram.
- With
--update-to(the CI job builds this commit twice, as 900.0.0 and 900.0.1): a fake GitHub (a fake/usr/bin/curlthat serves local files, and a fake gh in[updates] gh, because the build under test has no attestation), a changed unit and a drop-in, then/updateon the fake Telegram. The fake taps Apply; root installs 900.0.1 with the built-ininstall.sh, puts back the shipped unit, lists the drop-in, restarts the listener, and the new listener sends the report. - Each unit of the README
startblock isenabledandactive.
After each loop, the outbox must be empty and the listener must have logged no error and no warning. So a message that root writes and the listener cannot read fails the job.
The workflow Install a release (install-release.yml) runs the same
script against a published release: it installs the release with the
tag's own install.sh --version X.Y.Z, then runs
full-loop.sh --installed X.Y.Z with the README and units of the tag.
The Release workflow calls it after it publishes; it also runs on
release: published and by hand.
The owner updates pullgate from Telegram. It is the same gate as for a commit: one question that names one exact release, and a tap. What it trusts: THREAT-MODEL.md, T20.
The parts. pullgate update runs as root, from
pullgate-update.service. Two units start it: pullgate-update.timer
(once a day) and pullgate-update.path (the check request or an approval
of the updates). It reads /etc/pullgate/listener.toml (it needs no
token), and uses its own state and spool, named _update
(/var/lib/pullgate/_update/, /var/spool/pullgate/_update/). An instance
name cannot start with _, so no instance can use them. The binary carries
two things of its own version: a copy of install.sh (section 16), and the
units of contrib/systemd/ (pullgate units prints them as JSON, with the
list of units that the README enables).
The listener handles /update as /check: only exactly /update,
from owner_id, in chat_id, not edited. It writes the check request
_update/check/request (the six bytes check\n, nothing from the
message), at most one in 10 seconds, and replies "Looking for a new release
of pullgate…". A tap on an update question writes an approval to
_update/inbox, as for an instance (section 9). It sends what root puts
in _update/outbox.
Looking for a release. Root runs the built-in install.sh --check-only with TMPDIR=/var/lib/pullgate/_update/tmp. The script does
the checks of section 16 on the newest release (the SHA-256 and the
attestation; it does not run the new binary before the tap), installs
nothing, and
prints lines that start with @pullgate for root: version X.Y.Z security|normal, up-to-date X.Y.Z, fetched X.Y.Z, checked X.Y.Z SHA256 security|normal. A release is a security release when its notes
have the section ### Security (release-please makes it from security:
pull requests, docs/releasing.md). Root itself checks again that the
version is newer than its own. Then:
- Up to date: nothing. For
/update, a notice says so. - A newer release that passed every check: for
/update, root asks. From the timer, it sends one notice per release: the version, the link to the release notes and "send /update to install it". A security release says so on the first line, and its notice comes again each day (23 hours) until it is installed. A notice has no buttons and needs no tap.[updates] check = falseinlistener.tomlstops the daily look and its notices;/updatestill works. - A release that failed a check (after its files were downloaded):
one warning notice per release ("failed a check, so it is not
offered"), with the reason from
install.sh. It never gets a question or the "/update" hint. For/update, the warning comes each time. - No answer from GitHub (before the download): a log line; for
/update, a notice. A build from the source (pullgate dev) does nothing and says so: it cannot know which release is newer.
At most one /update per minute is acted on (check_at in root's state).
The question. Root makes a new code, writes the question to its state (synced), then to the outbox, as in section 7:
pullgate on <b>HOST</b>, update of pullgate
⚠ Security release. It fixes a security problem. (only for one)
Update pullgate to <code>X.Y.Z</code>?
Installed now: <code>OLD</code>
Release notes: https://github.com/code6x/pullgate/releases/tag/vX.Y.Z
File: <code>pullgate_X.Y.Z_linux_ARCH.tar.gz</code>
SHA-256: <code>…64 hex digits…</code>
It can be answered for 12 hours. At most one update question is open.
When the timer or /update finds a newer checked release while a question
is open, root closes the old one ("a newer question replaced this one")
and asks again. When the installed version is the newest again (installed
by hand), it closes the question.
Root's state, /var/lib/pullgate/_update/state.json (root, 0600):
question (id, code, version, sha256, asked, expires), notified and
warned (release → time of its last notice), check_at, check_note,
and in_progress (version, from, started) while an update runs.
Apply. pullgate update takes the lock (_update/lock), reads the
approvals as in section 9 (code with a constant-time compare, time, once),
and takes the version and the SHA-256 from its own state. Skip: a reply,
nothing else. Apply, forward only (the version must be newer than its
own; never an older one from Telegram):
- Write
in_progress(synced). Give the old binary a second name,DIR/.pullgate.prev(a hard link), and sync the directory. - Run the built-in
install.sh --version X.Y.Z --sha256 SHA256 --forward-only --bin-dir DIR(and--ghfrom[updates] gh). It downloads the files again and does every check again; the tarball must also have the SHA-256 of the question. It renames the new file overDIR/pullgateas its last step. A failure here installs nothing: root removes.pullgate.prev, clearsin_progressand reports why. - Run
DIR/pullgate version: it must printpullgate X.Y.Z. - Run
DIR/pullgate units(the new binary). Check the list: only namespullgate-*.service,.timer,.path, at most 32 files of 64 KiB. Write each unit that differs from/etc/systemd/system/NAME(a temporary file, sync, rename), and keep the old content. A unit there that is a link (masked) is left alone. Drop-ins (/etc/systemd/system/pullgate-*.d/*) are never changed; they are listed in the report. systemctl daemon-reload. For each unit that the README enables (for each instance ininstances): when it is enabled, or new in this version,systemctl reenableandsystemctl restartit. A unit that the owner disabled stays disabled. The listener is re-enabled here, but restarted last.systemctl restart pullgate-listen.service, wait 5 seconds, and checksystemctl is-active pullgate-listen.service.- Remove
.pullgate.prev, clearin_progress, and write the report: "pullgate is updated: OLD → NEW", the units it installed, the links it left alone, and every drop-in. The new listener sends it.
When step 3, 4, 5 or 6 fails: rename .pullgate.prev back over
DIR/pullgate, write back the old unit files (and remove the new ones),
daemon-reload, restart the units and the listener, clear in_progress,
and report "the update failed: REASON. pullgate OLD is back". If the
rollback has problems too, the report says which, and how to fix it by
hand with install.sh.
A crash leaves in_progress. The binary is the old file or the new
one: each change of the name pullgate is a rename in one directory, after
a sync (THREAT-MODEL.md, T10). The next start reports "the update was cut
off" with what pullgate version says, and how to check the host. It does
not guess, and leaves .pullgate.prev for the owner.
The config (in listener.toml, every key optional):
[updates]
check = true # false: no daily look, no notices; /update works
listener_user = "pullgate" # the listener user: owns the inbox of the updates
gh = "/usr/bin/gh" # the gh that checks the attestation, as install.sh --ghThere is no key to skip a check.
Not yet: signed commits (git verify-commit), a web UI.
install.sh, at the root of the repo, installs a release, or updates one.
It is a POSIX sh script. Root runs it on the host itself:
sh install.sh [--version X.Y.Z] [--bin-dir DIR] [--gh PATH] [--reinstall]pullgate update runs the copy of this script that is built into the
binary, with three more flags (section 15): --check-only (every check,
install nothing), --sha256 HEX (with --version: install only a tarball
with this SHA-256) and --forward-only (never the same or an older
version). It reads the lines that start with @pullgate on the standard
output.
All the work is in functions. Only the last line starts main, so a
download that stops in the middle runs nothing. The steps:
- Clean the environment: remove
TAR_OPTIONS,GZIP,CURL_HOME, theSSL_CERT_*,OPENSSL_*andLD_*variables,GCONV_PATH,GLIBC_TUNABLES,GODEBUG,CDPATH,IFSand more (the list and the reasons are inclean_envof the script). SetLC_ALL=C. As root, setPATH=/usr/sbin:/usr/bin:/sbin:/bin. pullgate and gh run with a time limit (timeout): 60 seconds, and 300 seconds forgh attestation verify. curl has its own limit. - Read the arguments.
--versionmust beX.Y.Z. A wrong argument exits with 64. - Check the platform: Linux on amd64 or arm64 only.
- Check the paths: only root may change the bin directory
(default
/usr/local/bin), each directory above it, the old binary and the--ghfile. The old binaryDIR/pullgatemust be a regular file or missing, never a link. A directory that others can write may pass only above the checked path, and only with the sticky bit (as/tmp). - Make a temporary directory with
mktemp -dinTMPDIR(default/tmp). CheckTMPDIR, each directory above it, and the new directory. - Ask the GitHub API for the newest release, or for the release of
--version. Its tag must bevX.Y.Zof that version. - Run the old binary with
version. The same version: nothing to do, exit 0, unless--reinstall. A newer installed version: nothing to do, unless--versionasks for the older one; then it says that this is a downgrade. An unknown version (pullgate dev, no output, or it does not run): fail, unless--reinstall. - Download
pullgate_<version>_linux_<arch>.tar.gz,SHA256SUMSandpullgate_<version>.sigstore.json. Every download usescurl -q --proto '=https' --tlsv1.2and has a size limit: 1 MiB for the API answer,SHA256SUMSand the bundle; 64 MiB for a tarball. SHA256SUMSmust have exactly one line for the tarball, and the SHA-256 must match.- Get gh: the
--ghfile (gh 2.93.0 or newer), or gh 2.102.0, downloaded and checked against the SHA-256 for this CPU that is written in the script. gh runs with no token and an empty config. - Run
gh attestation verifyon the tarball with the bundle,--repo code6x/pullgate,--cert-identity https://github.com/code6x/pullgate/.github/workflows/release.yml@refs/heads/mainand--deny-self-hosted-runners. - Take only
pullgateout of the tarball. Runpullgate versionfrom the temporary directory. It must printpullgate X.Y.Zof the version asked for, and nothing else. - Copy it into the bin directory under a new name, mode 0755, owner
root. Run
pullgate versionthere again. Write it to the disk (sync), then rename it over the old binary. The rename is atomic: a new process gets the old file or the new one. - Print what to restart. The script restarts nothing.
pullgate-listen.serviceruns the old binary until it is restarted. The timers and path units start a new process each time, so they get the new binary at their next start.
A trap removes the temporary directory and a new file that was not renamed yet, at every exit. A failure exits with 1, and its last line says that nothing was installed. What the script trusts: THREAT-MODEL.md, T17.