Skip to content

Fix v19 first-boot races, live lockouts, and startup delay - #380

Open
lirazsiri wants to merge 3 commits into
19.xfrom
fix/mysql-activation-race
Open

Fix v19 first-boot races, live lockouts, and startup delay#380
lirazsiri wants to merge 3 commits into
19.xfrom
fix/mysql-activation-race

Conversation

@lirazsiri

@lirazsiri lirazsiri commented Sep 2, 2026

Copy link
Copy Markdown
Member

This v19 release branch fixes three defects reproduced on the release path:

  • preserve MariaDB while systemd reports concurrent activation or reload states
  • disable fail2ban for live and casper boots, while relaxing installed SSH protection to 10 retries over 10 minutes
  • apply the upstream inithooks wait-once correction so late firstboot hooks do not each add a ten-second startup wait

The startup defect was measured on a Hub-launched WordPress image: inithooks.service took 3 minutes 14 seconds and logged 150 artificial wait seconds, while 95secupdates itself took 3 seconds.

Verification:

  • 6 focused unit tests pass
  • shell syntax and git diff --check pass
  • the final WordPress ISO squashfs contains the MariaDB and fail2ban policies
  • a mocked runner with multiple late hooks observes a single ten-call startup wait

The v19 MySQL helper treated systemd activating state as an inactive service. A first-boot hook racing MariaDB startup would therefore mark the service as self-started and stop it when the helper exited, causing later application hooks such as WordPress initialization to fail.

Record the unit state before waiting for startup and only stop MariaDB when the helper actually initiated an inactive service. Build-time temporary startup behavior remains unchanged.

Verified with focused active, activating, and inactive state tests, plus a live AWS boot-race reproduction that left MariaDB active and allowed the WordPress hook to complete.
Skip fail2ban when the kernel command line identifies a live ISO session, where
a ban cannot provide durable protection and can lock an evaluator out of the
appliance. Installed systems continue to run fail2ban, but now allow ten
retries within ten minutes and limit a ban to ten minutes.

Add focused policy tests for both the live-boot conditions and the installed
system thresholds. The fail2ban and MariaDB service-state tests pass together.
@lirazsiri lirazsiri changed the title Preserve MariaDB during concurrent boot activation Fix v19 first-boot service races and live lockouts Sep 2, 2026
Inithooks 2.3.6 waits up to ten seconds before every firstboot hook numbered 30 or later. Because the inithooks service keeps systemd in the starting state, a Hub-launched WordPress appliance spent 150 seconds in repeated waits while its security update hook needed only three seconds.

Overlay the upstream wait-once correction into v19 appliance roots until the fixed inithooks package reaches the archive. This applies the acceleration to ISO boots and to cloud images built from those ISOs.
@lirazsiri lirazsiri changed the title Fix v19 first-boot service races and live lockouts Fix v19 first-boot races, live lockouts, and startup delay Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant