Repository navigation
Conversation
lib/monitoring.sh gives the monitoring modules one way to bind to 127.0.0.1 or the WireGuard address, open ports to the VPN only, publish below the admin port, register Prometheus targets and Grafana data sources, install checksum-verified upstream releases and check listening sockets. fw_vpn_address names the server's WireGuard address, flat apt repositories need no component, and the web servers can proxy with the prefix stripped. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
wireguard sets up wg0 with keys in the lss secrets and peers from the settings; dns-resolver runs a validating Unbound on localhost and points the server at it, restoring the previous resolver on removal; ntp-server opens the chrony of the time module to the WireGuard network or everyone, rate-limited. time now writes a confdir line so ntp-server can add its own file without owning chrony.conf. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…d modules Debian and Ubuntu get Prometheus, Alertmanager and the node exporter from the distribution, the RHEL family the upstream releases pinned by sha256. Grafana, Loki and Alloy come from Grafana's repository, Netdata from its own, both with pinned key fingerprints. Alloy replaces Promtail, which reached its end of life on 2026-03-02. Every interface listens on 127.0.0.1 or the WireGuard address and runs with a sandbox drop-in. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
restic backs up to a local path, SFTP or S3 on a systemd timer, applies the retention, checks the repository and exports its state for Prometheus; the password, the S3 keys and the SSH key live in the lss secrets. db-dumps dumps every database engine lss installed before each backup. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
With restic installed, guard reports a last successful backup older than LSS_GUARD_BACKUP_MAX_AGE hours and runs the restore test once a week. Its findings go to the node exporter's textfile directory, where an alert rule picks them up. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
devops is minimal-secure, Docker and the monitoring stack. A comment marks where Forgejo joins once its module exists. The integration matrix tests the backup, monitoring and network modules and the devops profile; the haproxy test port moves off 9100, which the node exporter takes. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Module groups, addresses, settings and the guard's backup checks. The resource figures follow from the integration runs. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
How a module with a web interface binds, opens its ports, publishes below the admin port and registers with Prometheus and Grafana. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
The finding names the restore-test command, and finding also logs it, so the check matched its own message. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…ecret The network tests use the public keys of the wg(8) examples as peers. .gitleaksignore lists these two findings by fingerprint. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…nar findings mon_url builds the URL of a module's interface and states once why plain HTTP is safe there (127.0.0.1 or the encrypted WireGuard tunnel). Positional parameters go to named locals, and Loki's retention hours are computed before the heredoc. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…ansion GF_*__FILE variables are resolved by the entry script of Grafana's container image only; the packaged grafana-server ignored the variable and created the admin with the default password, so the login check failed. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
NoNewPrivileges forbids the switch to ifconfig_t when wg-quick runs ip, so wg0 never came up on AlmaLinux 10 with SELinux enforcing. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…ainers Debian starts chronyd with '!' as root although the unit says User=_chrony; the replacement command line lost the prefix and chronyd stopped with 'Not superuser'. The drop-in now takes the vendor line and adds -x, and also lifts the unit's condition on running outside a container. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Priming the root servers and keys can take longer than one 5 s probe. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…t output Packages that add a system user (grafana) print 'Not creating home directory', a line that starts with Not. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
wg reads its settings from a pipe through /dev/fd; in the private /dev of PrivateDevices SELinux denied that on AlmaLinux 10 (fopen: Permission denied). Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…switch Netdata 2 removed the setting that turned its Cloud off; /api/v1/info reports cloud-enabled as true on every agent. An agent connects only once claimed, so the verification now requires agent-claimed and aclk-available to be false, and lss no longer writes the ignored cloud.conf. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
file_remove kept the checksum, so a drop-in a module stopped writing, or an admin path the removed web server took along, was reported as missing on every verify. mon_web_admin forgets the admin path when no web server is left; the integration test removes prometheus before nginx. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
The package's installation script restarts Loki before mod_configure creates /var/lib/loki; with ReadWritePaths=- the directory stayed read-only in the sandbox and Loki hit its start limit (seen on the Debian 13 VM in the devops profile). StateDirectory creates it for the user loki, and a failed start is reset before lss enables the unit. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…ners chrony 4.8 on Ubuntu 26.04 stops with 'Could not send notification to $NOTIFY_SOCKET' inside a container. Where the unit runs chronyd in the foreground, the container drop-in makes it Type=simple without a notification socket; the forking unit of the RHEL family stays as is. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
prepare exported AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and RESTIC_PASSWORD_FILE before the database dumps ran, so the dump helper and its children (pg_dumpall as postgres) inherited them. The credentials now go into the environment of each restic process only (a subshell in run_restic), the dumps run before prepare, and the dump programs start with an empty environment; pg_dumpall as postgres gets nothing of root's. A unit test fails if the dump helper sees any AWS_ or RESTIC_ variable. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
The pass-through to restic still called it directly and lost the repository and password once they stopped being exported, so the restic module's verification failed. It now goes through restic_env like every other restic call. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…h start unbound.service of the RHEL family runs unbound-anchor as ExecStartPre and fails when the root servers are out of reach (Rocky 9 in CI). Unbound keeps the anchor current itself; DISABLE_UNBOUND_ANCHOR=yes is the unit's own switch. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
The parsed port, allow and confdir lines and chronyd's sockets go to the log before the verification fails, to find why AlmaLinux 10 does not serve on port 123. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Taken from the RESOURCES lines of the integration run on all systems: the largest value, rounded up, and the resident size of each service. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
… answer chronyd -p listed port 0 from the main file but none of the lines of the confdir file on Rocky 9 and AlmaLinux 10; the full output, the confdir line, the directory and the running command line tell why. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…no DNS On the CI machines outgoing DNS to the root servers is blocked. In the profile all job dns-resolver still pointed systemd-resolved at Unbound, and every later package download failed on name resolution. The server now switches to Unbound only after the signed root zone resolved; otherwise it keeps its previous resolvers and lss warns, naming LSS_DNS_RESOLVER_FORWARD. Verify skips the system check in that case. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
The EXIT trap read the local variable of mon_install_stopped. When
pkg_install died, bash on Debian 13 ran the trap with that variable gone
("policy: unbound variable"), so policy-rc.d stayed and forbade every
later service start. The trap now uses the global MON_POLICY_RC. The test
runs the failure path with set -u; Git Bash does not reproduce the unbound
case, CI's bash did.
Refs #6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…not answer Rocky 9 and AlmaLinux 10 list the confdir line and the file in it, yet chronyd -p shows none of its lines. The version, the end of the main file and parse runs of the file and of the confdir alone narrow it down. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
…rnel A RHEL-family container on a Debian or Ubuntu host runs chronyd under the host's AppArmor profile for /usr/sbin/chronyd, which allows /etc/chrony/** and /etc/chrony.* only. chronyd could not open /etc/chrony.d/lss-ntp-server.conf (Permission denied) and never opened port 123. The time module and ntp-server now share one rule for the directory. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
The encryption controller can report later than the API on a busy host; the profile run on AlmaLinux 10 failed now and then right after start. The check now waits up to 60 seconds and logs the status (key names and hash state, no key material) when it still fails. Refs #6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com>
Signed-off-by: VectoDE <system.bot.discord.utility@gmail.com> # Conflicts: # .github/workflows/integration.yml
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 4 of lss 2.0 (#6): backup, monitoring, network, the
devopsprofile and guard's backup checks.Modules
restic(local path, SFTP or S3; password, S3 keys and SSH key in the lss secrets; systemd timer, retention with forget/prune,checkafter each run) anddb-dumps(MariaDB, MySQL, PostgreSQL, MongoDB, Redis and Valkey, dumped before each backup).node-exporter,prometheus(alert rules for down targets, disks, memory, guard findings, failed or old backups and failed restore tests),alertmanager,grafana,loki,alloy,netdataandsmartd.LSS_<MODULE>_ACCESS=vpn. WithLSS_<MODULE>_WEB_ADMIN=trueit is published below the web server's admin port.wireguard(server, peers from the settings, keys in the secrets, firewall),dns-resolver(Unbound with DNSSEC on localhost) andntp-server(the chrony oftimeserves the WireGuard network or everyone, rate-limited).devops: minimal-secure, Docker and the monitoring stack. A comment marks where Forgejo will join.Package sources
data/checksums.txtEvery service gets a sandbox drop-in.
Tests
tests/unit/{backup,monitoring,network}.bats, plus additions to the guard, web and system tests.backup,backup-alt,monitoringandnetwork;wireguardin the host and VM kernel runs;profile devopson Debian 13 and AlmaLinux 10.Refs #6
🤖 Generated with Claude Code