walgit

Walgit is a WAL-first Git server in Go, based on Cursor's Git at any scale.

Object storage is authoritative. Each node keeps a disposable bare-repository cache for Git's native tools, and rebuilds that cache from the write-ahead log when it falls behind or becomes damaged. A push becomes visible at one conditional index write; every fetch checks that index before serving data.

This repository implements that storage and consistency layer. It is not a complete Git hosting product: there is no UI, organization model, global router, or general-purpose authorization system.

Quick start

Walgit requires Go 1.26, Git with proc-receive support, and macOS or Linux. Its locking and path-safety checks use Unix primitives.

mkdir -p ./bin
go build -o ./bin/walgit ./cmd/walgit

./bin/walgit init-node --config ./walgit-data/node.json
./bin/walgit create --config ./walgit-data/node.json org/example
./bin/walgit serve \
  --config ./walgit-data/node.json \
  --listen 127.0.0.1:8080 \
  --allow-anonymous-push

--allow-anonymous-push is only sensible for local development. In another terminal:

git clone http://127.0.0.1:8080/git/org/example.git
cd example
git config user.name "Example User"
git config user.email "user@example.invalid"
echo hello > README.md
git add README.md
git commit -m "initial commit"
git push origin HEAD:main

For authenticated pushes, omit --allow-anonymous-push and set a token:

WALGIT_PUSH_TOKEN=replace-me \
  ./bin/walgit serve --config ./walgit-data/node.json

Git may use any non-empty username; the token is the password. Fetches remain public. Terminate TLS at a reverse proxy or pass --tls-cert and --tls-key.

How it works

• Each accepted push produces an immutable WAL entry containing a complete, non-thin replay pack and the exact atomic ref transaction.
• One ETag-protected index serializes repository changes. Publishing its new value is the point at which a push becomes visible.
• A node materializes the current snapshot and uncompacted WAL tail into a local bare repository. That cache may be deleted at any time and rebuilt.
• Git still speaks Git: Walgit delegates the wire protocol, pack validation, and ref plumbing to a pinned Git installation.

Repositories may use SHA-1 or SHA-256. Compaction publishes a full-pack snapshot without holding the write lease during its expensive upload. The complete transaction model and crash guarantees live in docs/design.md.

Repository commands

./bin/walgit create --config ./walgit-data/node.json org/example
./bin/walgit compact --config ./walgit-data/node.json org/example
./bin/walgit history --config ./walgit-data/node.json org/example

create and compact need exclusive ownership of the node's cache, so they fail while serve owns it. history reads object-store metadata directly and can run alongside the server.

When the server is running, the optional admin API can create and compact repositories without taking offline ownership:

WALGIT_ADMIN_TOKEN=replace-me \
WALGIT_PUSH_TOKEN=replace-me-too \
  ./bin/walgit serve --config ./walgit-data/node.json

curl -X PUT \
  -H 'Authorization: Bearer replace-me' \
  'http://127.0.0.1:8080/v1/repositories/org/example?object-format=sha256'

curl -X POST \
  -H 'Authorization: Bearer replace-me' \
  'http://127.0.0.1:8080/v1/compactions/org/example'

Object storage

The default backend stores objects on the local filesystem. For shared storage, initialize a node with an S3-compatible backend:

./bin/walgit init-node \
  --config ./node-a/node.json \
  --backend s3 \
  --s3-bucket my-git-wal \
  --s3-region us-west-2 \
  --s3-prefix production

AWS_PROFILE=production WALGIT_PUSH_TOKEN=replace-me \
  ./bin/walgit serve --config ./node-a/node.json

Credentials stay out of the node config and come from the AWS SDK credential chain. A custom --s3-endpoint must be an origin only: no credentials, path, query, or fragment. HTTPS is required except for an explicitly enabled, IP-literal loopback endpoint. A local MinIO setup therefore looks like this:

./bin/walgit init-node \
  --config ./minio-node/node.json \
  --backend s3 \
  --s3-bucket walgit \
  --s3-region us-east-1 \
  --s3-endpoint http://127.0.0.1:9000 \
  --s3-path-style \
  --allow-insecure-s3-http

That exception is for local development. Without TLS, requests, authentication headers, and repository data travel in plaintext. The backend must correctly implement conditional If-Match and If-None-Match writes and If-None-Match reads.

Multiple nodes

Give every node its own cache and temporary directories, then point them at the same bucket, store prefix, and WAL prefix. Optional UDP gossip hints can warm a peer's cache after a push:

./bin/walgit init-node \
  --config ./node-a/node.json \
  --backend s3 --s3-bucket my-git-wal --s3-region us-west-2 \
  --node-id node-a --gossip-listen 0.0.0.0:9400 \
  --gossip-peer node-b.internal:9400

Gossip never participates in correctness or push acknowledgement. Hints may be dropped, duplicated, reordered, or forged; every request still checks the object-store index. Peer names resolve once at startup, and incoming packets are admitted by source IP and rate-limited before decoding. Run gossip on a private, anti-spoofed network—the source-IP check is not authentication.

Operations

Git executable

create, compact, and serve resolve the first git on PATH by default. They canonicalize it and reject an executable or parent directory that another local account could replace. The server pins both that executable and its validated helper directory; generated hooks receive the same canonical path.

Use --git-executable /absolute/path/to/git when the default is rejected. This flag selects runtime tooling only; it is not stored in the node config. Before opening its HTTP listener, serve exercises proc-receive against a temporary repository and refuses to start if the selected Git cannot route the hook.

Local trust

• Cache, temporary, and filesystem-store roots must be owned by the service user and must not overlap, including through case-insensitive filesystem aliases.
• Existing path components and symlink targets must also be protected from replacement by another account. Walgit resolves those paths once and binds each generated hook to a fingerprint of the same effective configuration; changing a root or retargeting a link makes pushes fail until restart.
• Cache and store roots cannot be writable by group or other users. The temporary root is stricter because it holds uncommitted packs: it cannot be a symlink or accessible to group or other users.
• The node config must be a non-symlink regular file outside those roots. It may be owned by root or the service user, but neither it nor its ancestry may be replaceable by another account.
• One process owns a cache root at a time. Git descendants inherit that ownership fence, so a canceled request cannot release the cache while a surviving subprocess still uses it.

These checks use Unix ownership and mode bits. Keep node configs and data roots free of extended ACLs that grant another account write or delete access.

Request timeouts

Flag	Default	Limit
--io-timeout	2m	Idle request reads and response writes
--backend-timeout	30m	A Git process tree with no wire progress
--request-timeout	2h	Total request lifetime, even while bytes move

Raise these only for a legitimate large-repository workload.

Deliberate limits

Walgit publishes each push with its own index compare-and-swap. It does not yet include group commit, a global router, membership and health management, automatic compaction scheduling, cache eviction, orphan-WAL collection, or production-grade rate and concurrency limits. Those are throughput and operations work; they do not weaken the durability or read-freshness contract implemented here.

Verification

go test ./...
go test -race ./...
go vet ./...
go build ./...

The integration suite performs real SHA-1 and SHA-256 HTTP pushes, removes the entire local repository, and clones it again from object storage.