Skip to content

docs(readme): streamline documentation and refresh dashboard screenshots #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
simonvanlierde merged 2 commits into from
Nov 1, 2026
Merged

docs(readme): streamline documentation and refresh dashboard screenshots #68

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b859293
docs(readme): streamline documentation and refresh dashboard screenshots
simonvanlierde Nov 1, 2026
7cda33d
docs(screenshots): refresh dashboard captures after 5-minute load soak
simonvanlierde Nov 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ Notable changes to this stack. Format follows

### Changed

- Streamlined repository documentation and onboarding guides around researcher workflows, extracting host deployment guards and S3 storage migration into `docs/RUNBOOK.md` and refreshing dashboard screenshots.
- `bootstrap.sh` creates only a `watchdog` healthchecks.io check per
project/environment, since that check reports the other scheduled jobs.
The `backup` and `restore-check` checks it used to create were never
Expand Down
225 changes: 50 additions & 175 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,54 +6,36 @@
[![Grafana LGTM](https://img.shields.io/badge/Grafana-Loki%20%7C%20Tempo%20%7C%20Prometheus-f46800?logo=grafana&logoColor=white)](compose.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)

Central monitoring for CML's research software. One host runs Grafana, Loki,
Tempo, Prometheus, and an OpenTelemetry Collector. Projects send logs, traces,
and metrics here over OTLP (the OpenTelemetry protocol), and Grafana shows
them side by side.
Central monitoring for CML research software. A single host runs Grafana, Loki,
Tempo, Prometheus, and an OpenTelemetry Collector. Research applications send
metrics, traces, and logs over standard OTLP (OpenTelemetry protocol), and Grafana
displays them side by side.

## Try it in one command

You need Docker with the [Compose plugin](https://docs.docker.com/compose/install/)
and [`just`](https://github.com/casey/just#installation). Everything else runs
in containers.

```sh
cp .env.example .env
just demo
```

This starts the stack plus a small FastAPI service under constant load
(`compose.demo.yml`). The service is auto-instrumented and fails one request
in ten, so the error panels and the error-rate alert have something to show.
Give it a minute, then open Grafana at <http://localhost:3002>
(admin / change-me):
## Who reports here

- **Dashboards → Service Health (RED)**: request rate, error rate, and
latency. The dots on the latency panel are exemplars. Click one to open the
trace behind that measurement.
- **Dashboards → Logs**: log volume by service and level, an error feed, and
a live tail.
- **Alerting → Alert rules**: the rules Grafana evaluates. `HighErrorRate`
trips on the demo service after five minutes.
| Project | Environment | Telemetry | Host | Spoke overlay |
| --- | --- | --- | --- | --- |
| [R9lab](https://github.com/CMLPlatform/relab) | `r9lab/prod` | Logs, Traces, Metrics | CML VM | `templates/compose.telemetry.yml` |
| [R9lab](https://github.com/CMLPlatform/relab) | `r9lab/staging` | Logs, Traces, Metrics | CML VM | `templates/compose.telemetry.yml` |
| [CML Gallery](https://github.com/CMLPlatform/cml-gallery) | Planned | Probes, RUM | GitHub Pages | `templates/web/rum.html` |

![Service Health (RED) dashboard](docs/img/service-health.png)
The alert marker in `config/grafana/alerting/projects.yaml` guards active reporting.
The `ProjectsUncovered` rule fires if an unlisted spoke begins transmitting.

`just demo-down` removes the demo services. `just demo-destroy` removes the
whole demo stack.
## Connect your project

The demo runs under its own compose project on its own port, so it never
touches a stack already running on the host. The same holds for `just smoke`
on :3001.
Connecting a service takes about five minutes without application code changes.
See [docs/ONBOARDING.md](docs/ONBOARDING.md) for step-by-step recipes:

## How it works
1. **Web services (FastAPI/Python):** Add standard OTel environment variables to emit traces, RED metrics, and correlated logs.
2. **Static websites and portals:** Register an outside-in uptime probe (`just probe-add`) or drop in the browser Real User Monitoring snippet ([`templates/web/rum.html`](templates/web/rum.html)) for Core Web Vitals.
3. **Project hosts and VMs:** Run `./bootstrap.sh <project> <env>` to deploy a Grafana Alloy agent for container logs and host resources.

Everything enters through one gateway, the OpenTelemetry Collector. A project
configures a single endpoint, and a storage backend can be swapped later
without touching any application.
## Architecture

```mermaid
flowchart LR
apps["Project apps"] -->|"OTLP over HTTPS<br/>(gRPC/HTTP direct on private nets)"| cf["Cloudflare Tunnel<br/>(production, optional)"]
apps["Project apps & sites"] -->|"OTLP over HTTPS<br/>(HTTP/protobuf)"| cf["Cloudflare Tunnel<br/>(production)"]
user["Browser"] -->|HTTPS| cf
cf --> otel
cf --> grafana
Expand All @@ -67,160 +49,53 @@ flowchart LR
end
```

Solid arrows write telemetry. Dotted arrows are Grafana reading at query
time. Locally there is no tunnel, and Grafana is at `localhost:3000` for
`just up` or `localhost:3002` for `just demo`.
The stack operates on a hub-and-spoke model:

The stack runs on a single host. [ADR 0001](docs/adr/0001-observability-stack.md)
records why, and the alternatives. [ADR 0002](docs/adr/0002-hub-and-spoke-observability.md)
describes the hub-and-spoke design that serves multiple projects.
- **Single gateway:** Everything enters through the OpenTelemetry Collector on one port.
- **Unified alerting:** Grafana evaluates all alert rules across both metrics and logs. No separate Alertmanager daemon.
- **Outside-in detection:** External probers and dead-man's switches are the only alarms whose default state is firing. Silence indicates failure, not health.

### Who reports here
Architecture decisions and trade-offs are documented in [ADR 0001 (Stack Selection)](docs/adr/0001-observability-stack.md) and [ADR 0002 (Hub-and-Spoke Design)](docs/adr/0002-hub-and-spoke-observability.md).

| Project | Environment | Telemetry | Host | Spoke overlay |
| --- | --- | --- | --- | --- |
| [R9lab](https://github.com/CMLPlatform/relab) | `r9lab/prod` | Logs, Traces, Metrics | CML VM | `templates/compose.telemetry.yml` |
| [R9lab](https://github.com/CMLPlatform/relab) | `r9lab/staging` | Logs, Traces, Metrics | CML VM | `templates/compose.telemetry.yml` |
| [CML Gallery](https://github.com/CMLPlatform/cml-gallery) | Planned | Logs, Metrics | CML Host | `templates/compose.telemetry.yml` |

The public alert marker in `config/grafana/alerting/projects.yaml` guards active
reporting; `ProjectsUncovered` trips if an unlisted spoke begins transmitting.
## Try it in one command

## Run it for real
Prerequisites: Docker with [Compose plugin](https://docs.docker.com/compose/install/) and [`just`](https://github.com/casey/just#installation).

```sh
cp .env.example .env # set GRAFANA_ADMIN_PASSWORD
just up
just check # validate every config in the repo
cp .env.example .env
just demo
```

Grafana: <http://localhost:3000>.
This launches the core monitoring stack and a demo FastAPI service under simulated load on port 3002:

`COMPOSE_FILE` in `.env` names the overlays a host runs. In production, set
`COMPOSE_FILE=compose.yml:compose.tunnel.yml`. `just up`, `just restart`, `just backup`,
`just restore`, and plain `docker compose logs` or `ps` then act on that set.
- Open Grafana at <http://localhost:3002> (`admin` / `change-me`).
- View **Dashboards → Service Health (RED)** for request rates, error rates, and latency traces.
- Stop and clean up: `just demo-destroy`.

With the tunnel overlay active, `just up` refuses to start until five settings
are real: a generated `OTLP_AUTH_TOKEN`, a changed `GRAFANA_ADMIN_PASSWORD`,
`GRAFANA_ROOT_URL` pointing at the tunnel hostname, `GRAFANA_COOKIE_SECURE=true`,
and a non-empty `ALERT_WEBHOOK_URL`. It also refuses `GRAFANA_JWT_AUTH=true`
without `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD`, and a `.env`,
`infra/terraform.tfvars` or OpenTofu state file that its group or other users
can read.
An empty `HEARTBEAT_URL` only warns.
![Service Health dashboard](docs/img/service-health.png)

The Cloudflare edge is code too. The tunnel, its hostnames, DNS, and the
Access rule that puts an email one-time PIN in front of Grafana live in
`infra/` as an OpenTofu configuration. Applying it produces the
`CLOUDFLARE_TUNNEL_TOKEN` the overlay needs. The bootstrap steps are at the
top of [infra/main.tf](infra/main.tf).
## Operating the stack

### Checks
Host deployment, secret rotation, pre-flight environment checks, alert matrices, and backups are documented in [docs/RUNBOOK.md](docs/RUNBOOK.md).

| Recipe | What it does |
| --- | --- |
| `just lint` | Static checks: compose files, rendered alert templates, YAML, workflows, shell, Python, OpenTofu formatting, dashboard JSON, git history for secrets |
| `just validate` | Runs each stack config through the exact image the stack uses |
| `just check` | `lint` plus `validate` |
| `just smoke` | Boots an isolated copy of the stack and asserts it works end to end |
| `just restore-check` | Rehearses backup and restore on the smoke stack |
| `just hooks` | Installs the git hooks: gitleaks, a CITATION.cff schema check and citesync at commit, a Conventional Commits check on the message, `check` at push, `infra-validate` at push when `infra/` changed |

Every `just` check runs in a pinned container. Nothing is installed on the host.
The citation hooks are the exception: prek builds their Python environments itself.

`just smoke` covers what has no offline validator. It boots the stack in
production shape with JWT auth on, then asserts that every dashboard and
alert rule provisioned, that the contact points carry the URLs from the
environment, that Grafana refuses a forged token, that every scrape target is
up, and that a metric and a log posted through the collector come back out of
Prometheus and Loki with their labels. `SMOKE_ALERTS=1 just smoke` also stops
a scrape target and waits for `TargetDown` to fire, about three minutes more.
The assertions are in [scripts/smoke.sh](scripts/smoke.sh).

CI runs `lint` on one job and `validate` plus `smoke` (with the alert round
trip) on another, on every pull request.

## Sending telemetry from a project

You need the OTLP endpoint, the bearer token, and a few naming conventions.
[docs/ONBOARDING.md](docs/ONBOARDING.md) has copy-paste templates for
applications. [templates/README.md](templates/README.md) covers the host
agent that ships container logs and host metrics.

> [!WARNING]
> Never publish ports 4317/4318 to the internet. The compose file binds them to
> `127.0.0.1`. The tunnel is the way in.

## Alerting

Grafana evaluates and delivers the rules in `config/grafana/alerting/`:
telemetry silent per project, a project sending telemetry with no rule file,
container crash-looping or OOM-killed, scrape target down, OTel export
failures, alert delivery failing, error rate above 5%, p99 latency above 2s,
host memory above 90%, disk above 80%, disk projected full within 3 days,
Prometheus active series above 30k, and 5,000 new series in 30 minutes. There is no Alertmanager. Grafana rules can query
Loki as well as Prometheus, and one engine means one answer to "who gets
told".

Notifications go to the webhook in `ALERT_WEBHOOK_URL` (ntfy, Slack, and so
on). One rule, `Watchdog`, fires permanently and posts to `HEARTBEAT_URL`
every five minutes. Point that at a dead man's switch such as healthchecks.io,
which alerts when the pings stop. That is the only way to notice the host
itself dying.

Set both variables. An unset `ALERT_WEBHOOK_URL` drops every alert while the
heartbeat keeps reporting healthy. With the tunnel overlay, `just up` refuses
to start without it.

## Storage

Everything persists to local Docker volumes, captured by `just backup`. The
one exception is `otel_queue`, the collector's on-disk export queue. It holds
seconds of in-flight telemetry and is not worth restoring.

When local disk stops fitting, Loki and Tempo can move to any S3-compatible
object store. The appendix of ADR 0001 documents that change.

## Layout
In production, run:

```sh
compose.yml # core services
compose.tunnel.yml # production overlay: Cloudflare Tunnel
compose.demo.yml # demo overlay: sample telemetry source
compose.sandbox.yml # isolation overlay for `just demo` and `just smoke`
demo/ # the demo FastAPI service
scripts/smoke.sh # what `just smoke` asserts against the booted stack
config/
otel-collector.yaml # ingestion gateway (OTLP in → Loki/Tempo/Prometheus out)
loki.yaml # logs
tempo.yaml # traces
prometheus.yaml # metrics
grafana/ # provisioned datasources, dashboards, and alerting
dashboards/ # JSON dashboards; Grafana loads them from here
docs/ # runbook, onboarding, ADRs, screenshots
infra/ # OpenTofu: Cloudflare tunnel, ingress routes, DNS
templates/ # files a project host vendors: agent config, overlays
# Set COMPOSE_FILE=compose.yml:compose.tunnel.yml in .env
just up
just check
```

`dashboards/*.json` is the source of truth. The directory is mounted read-only
and UI saves are disabled, so a change made in the browser lasts until the
page reloads. Edit the JSON and provisioning picks it up within 30 seconds. To
keep something built interactively, export it as JSON and paste it into the
file.

## Documentation

| Document | What it covers |
| Document | Purpose |
| --- | --- |
| [docs/ONBOARDING.md](docs/ONBOARDING.md) | Connecting an application: endpoint, token, copy-paste templates |
| [templates/README.md](templates/README.md) | Onboarding a project host: the agent, GPU hosts, removing a project |
| [docs/RUNBOOK.md](docs/RUNBOOK.md) | Operations: rotating secrets, disk pressure, backup and restore |
| [docs/adr/](docs/adr/) | Why the stack looks like this |
| [CHANGELOG.md](CHANGELOG.md) | Release history |
| [CONTRIBUTING.md](.github/CONTRIBUTING.md) | Issues, pull requests, checks, commit style |
| [SECURITY.md](.github/SECURITY.md) | Reporting a vulnerability, and what is in scope |
| [Code of Conduct](.github/CODE_OF_CONDUCT.md) | How we treat each other, and how to report a problem |

Licensed under the [MIT License](LICENSE). To cite the stack, use GitHub's
"Cite this repository" button, which reads [CITATION.cff](CITATION.cff).
| [docs/ONBOARDING.md](docs/ONBOARDING.md) | Connect applications, static sites, and browser frontends |
| [templates/README.md](templates/README.md) | Deploy host agent (Grafana Alloy) for host metrics and container logs |
| [docs/RUNBOOK.md](docs/RUNBOOK.md) | Host operations, pre-flight guards, alerts, and troubleshooting |
| [docs/adr/](docs/adr/) | Architectural Decision Records (ADR 0001, ADR 0002) |
| [CHANGELOG.md](CHANGELOG.md) | Release history and upgrade notes |
| [CONTRIBUTING.md](.github/CONTRIBUTING.md) | Contribution workflow, conventions, and checks |
| [SECURITY.md](.github/SECURITY.md) | Security policy and vulnerability disclosure |

Licensed under the [MIT License](LICENSE). To cite the stack, see [CITATION.cff](CITATION.cff).
Loading
Loading