Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -438,10 +438,18 @@ now; `genco tree` has each group's `limits` (`null` is the default, its
`defaults` what an empty one means).

**All groups together.** The groups' slices are in `genco.slice`, which
genco caps at the RAM without a reserve for the system: a quarter of the
RAM, at least 768 MB, at most half (3.9 GB of RAM: 2.9 GB for the
groups). So the groups' own limits may add up to far more than the RAM,
groups rarely peak together; when they do, the kernel takes memory back
genco caps at the RAM without the reserve for the system: by default a
quarter of the RAM, at least 768 MB, at most half (3.9 GB of RAM: 2.9 GB
for the groups). Settings → Memory and `genco memory` show the split and
what the groups and the system use; the reserve is a setting there
(`genco memory reserve 1.5G`, `default`), 512 MB to half the RAM, and
applies at once. genco's agent moves it too, in the daily health check
or on an alert, when the system runs short while the groups have room or
the other way round, and says why. A reserve that leaves the groups
less than they use now without their page cache is refused. A new group
limit is at most the cap; one a group already has stays, the cap holds
it anyway. So the groups' own limits may add up to far more than the
RAM, groups rarely peak together; when they do, the kernel takes memory back
from them and kills within the groups, never the databases, nginx, mail
or genco. The shared services (not genco, whose children such as imports
and the agent's commands would inherit it) also get a low OOM score
Expand Down
4 changes: 4 additions & 0 deletions docs/PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,10 @@ terminal.
the RAM without a reserve (genco.slice), shared services with a
low OOM score, alerts on memory pressure and OOM kills instead of
the sum of limits, the group's usage and 24 h peak beside its limit
- [x] the reserve for the system as a setting (operator 2026-11-12):
Settings → Memory, `genco memory [reserve <size|default>]`,
512 MB to half the RAM, moved by the agent when the split doesn't
fit what the groups and the system use
- [x] drop-ins that override genco's units reported by apply, the minute
check and the config view; `genco group adopt-limits`
- [x] ImageMagick limits for groups with PHP, from the group's memory
Expand Down
1 change: 1 addition & 0 deletions server/src/activity.rs
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ fn describe(method: &Method, path: &str) -> Option<String> {
("POST", "/settings/mail/hosting", "Change mail hosting"),
("POST", "/settings/notify", "Change notifications"),
("POST", "/settings/model", "Change the agent's model"),
("POST", "/settings/memory", "Change the memory reserve"),
("POST", "/domains/*/env", "Change environment variables"),
("POST", "/groups/*/workers", "Add worker"),
("POST", "/workers/*", "Change worker"),
Expand Down
10 changes: 9 additions & 1 deletion server/src/agent/instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,10 +269,18 @@ field in the panel.
`genco-*.service.d/`) or `systemctl set-property`: they override genco and are reported as a
problem; `genco group adopt-limits <group>` moves a slice drop-in's
limits into the settings. All groups together are capped at the RAM
without a reserve for the system (`genco.slice`), a group's limit at
without the reserve for the system (`genco.slice`), a group's limit at
most that; their limits may add up to more. Base a limit on what the
group really uses (Usage, `memory.peak` of its slice), alerts come on
memory pressure and OOM kills.
- The reserve for the system: `genco memory` shows the split and what
the groups and the system use, `genco memory reserve <size|default>`
changes it within its bounds and applies. Move it yourself when the
split doesn't fit: raise it when the system (databases, mail, genco)
runs short or swaps while the groups have room, lower it when groups
get OOM kills at the cap while the system leaves much of its reserve
unused. Small steps (256-512 MB), and say what you changed and why
with the numbers.
- Craft with Imagick: transforms of large originals take 500-700 MB.
genco limits ImageMagick per process for groups with PHP
(MAGICK_MEMORY_LIMIT a quarter of the soft limit, at most 256 MiB, then
Expand Down
3 changes: 3 additions & 0 deletions server/src/agent/routines/health.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ admin, and write a short report:
- Certificates: any domain without one, or expiring in under 30 days
(`certbot certificates`)?
- Mail: anything stuck in the queue (`postqueue -p`)?
- Memory: `genco memory`. If the split between the groups and the
system doesn't fit what they use, move the reserve (see your
instructions) and say so.
- Updates are the weekly update round's; only say if the box needs a
reboot (`/var/run/reboot-required`).
- Open warnings: `genco attention list` shows the routines and alerts
Expand Down
16 changes: 9 additions & 7 deletions server/src/apply/memory.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
//! Memory of the whole server: all groups share `genco.slice`, capped at
//! the RAM without a reserve for the system (limits.rs), so groups whose
//! the RAM without the reserve for the system (memory.rs), so groups whose
//! own limits add up to more than the RAM run out among themselves. The
//! shared services get a low OOM score: when the kernel has to kill
//! outside the groups after all, it picks something else than the
Expand Down Expand Up @@ -28,11 +28,10 @@ const SYSTEM_SLICE: &str = "/sys/fs/cgroup/system.slice";

/// genco.slice and a drop-in with the OOM score for each protected service
/// that runs here.
pub(super) fn memory_files() -> Vec<File> {
let (ram, _) = crate::limits::server_memory();
pub(super) fn memory_files(cap_mb: u32) -> Vec<File> {
let mut files = vec![File {
path: render::unit_path(GROUPS_SLICE),
content: crate::limits::groups_slice_unit(ram),
content: crate::limits::groups_slice_unit(cap_mb),
private_to: None,
}];
for (unit, _) in protected_units() {
Expand Down Expand Up @@ -86,10 +85,13 @@ fn protected(unit: &str) -> bool {
/// The cap on the running slice, and the OOM score of the protected
/// services' processes right away (the drop-in counts from their next
/// start).
pub(super) async fn memory(r: &mut Report, changed: &HashSet<PathBuf>) {
pub(super) async fn memory(
r: &mut Report,
changed: &HashSet<PathBuf>,
cap_mb: u32,
) {
if changed.contains(&render::unit_path(GROUPS_SLICE)) {
let (ram, _) = crate::limits::server_memory();
let cap = format!("MemoryMax={}M", crate::limits::groups_cap(ram));
let cap = format!("MemoryMax={cap_mb}M");
if r.run(&[
"systemctl",
"set-property",
Expand Down
10 changes: 6 additions & 4 deletions server/src/apply/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ pub async fn apply(cfg: &Config, db: &Db) -> Report {
return r;
};
let blocked = blocked(&mut r, db);
let cap = crate::memory::groups_cap(&db.conn());
let old_names = crate::panel::old_names(&db.conn());
relay_sasl(&mut r, db).await;

Expand All @@ -161,7 +162,7 @@ pub async fn apply(cfg: &Config, db: &Db) -> Report {
deploy_key(&mut r, g).await;
}
}
sync(&mut r, cfg, &groups, &blocked, &old_names).await;
sync(&mut r, cfg, &groups, &blocked, &old_names, cap).await;
databases(&mut r, &groups).await;

// with a new certificate the sites get https, the apps the mail host
Expand All @@ -170,7 +171,7 @@ pub async fn apply(cfg: &Config, db: &Db) -> Report {
for g in &mut groups {
g.mail_host = host.clone();
}
sync(&mut r, cfg, &groups, &blocked, &old_names).await;
sync(&mut r, cfg, &groups, &blocked, &old_names, cap).await;
}

// genco doesn't touch them, an admin moves them into the group's
Expand All @@ -192,14 +193,15 @@ async fn sync(
groups: &[Group],
blocked: &[String],
old_names: &[String],
cap_mb: u32,
) {
let certs = host::certs(groups);
let mut files: Vec<File> = groups
.iter()
.flat_map(|g| render::group_files(g, &certs))
.collect();
files.extend(panel_files(cfg, old_names));
files.extend(memory_files());
files.extend(memory_files(cap_mb));
files.push(firewall_file(cfg, groups, blocked));
files.push(File {
path: render::RESERVED_PORTS.into(),
Expand All @@ -225,7 +227,7 @@ async fn sync(
r.run(&["systemctl", "daemon-reload"]).await;
}

memory(r, &changed).await;
memory(r, &changed, cap_mb).await;
for g in groups {
set_limits(r, g, &changed).await;
start_units(r, g, &changed).await;
Expand Down
2 changes: 1 addition & 1 deletion server/src/hosting.rs
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,7 @@ pub fn set_limits(conn: &Connection, id: i64, limits: &Limits) -> Result<()> {
)
.optional()?
.flatten();
limits::check_here(limits, kept)?;
limits::check_here(limits, crate::memory::groups_cap(conn), kept)?;
conn.execute(
"UPDATE domain_groups SET memory_max_mb = ?1, memory_high_mb = ?2,
cpu_percent = ?3, tasks_max = ?4 WHERE id = ?5",
Expand Down
82 changes: 45 additions & 37 deletions server/src/limits.rs
Original file line number Diff line number Diff line change
Expand Up @@ -77,27 +77,24 @@ impl Limits {
}
}

/// Within what the server has: `ram_mb` and `cores`. A group may
/// have at most what all groups together may use (`groups_cap`); a
/// memory limit it already has (`kept_mb`) stays allowed, the cap of
/// all groups holds it anyway.
/// Within what the server has: `cap_mb`, what all groups may use
/// together (`groups_cap`), and `cores`. A memory limit the group
/// already has (`kept_mb`) stays allowed, the cap of all groups holds
/// it anyway.
pub fn check(
&self,
ram_mb: u32,
cap_mb: u32,
cores: u32,
kept_mb: Option<u32>,
) -> Result<()> {
let max = self.memory_max();
let cap = groups_cap(ram_mb);
if self.memory_max_mb.is_some_and(|m| Some(m) != kept_mb)
&& !(128..=cap).contains(&max)
&& !(128..=cap_mb).contains(&max)
{
return Err(Error::Invalid(format!(
"the memory limit is 128 MB to {}, what all groups may use \
together (the server's {} without {} for the system)",
size(cap),
size(ram_mb),
size(reserve(ram_mb))
together (`genco memory`)",
size(cap_mb)
)));
}
if !(64..=max).contains(&self.memory_high()) {
Expand Down Expand Up @@ -154,18 +151,14 @@ pub fn with_defaults<S: serde::Serializer>(
s: S,
) -> std::result::Result<S::Ok, S::Error> {
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
struct Shown<'a> {
#[serde(flatten)]
limits: &'a Limits,
defaults: Defaults,
/// what all groups may use together, the most one may have
groups_cap_mb: u32,
}
Shown {
limits,
defaults: limits.defaults(),
groups_cap_mb: groups_cap(server_memory().0),
}
.serialize(s)
}
Expand Down Expand Up @@ -252,37 +245,50 @@ fn cores() -> u32 {
std::thread::available_parallelism().map_or(1, |n| n.get() as u32)
}

/// Checks the limits against this server, `kept_mb` is the memory limit
/// the group has now.
pub fn check_here(limits: &Limits, kept_mb: Option<u32>) -> Result<()> {
limits.check(server_memory().0, cores(), kept_mb)
/// Checks the limits against this server with what all groups may use
/// (`cap_mb`), `kept_mb` is the memory limit the group has now.
pub fn check_here(
limits: &Limits,
cap_mb: u32,
kept_mb: Option<u32>,
) -> Result<()> {
limits.check(cap_mb, cores(), kept_mb)
}

/// The least an admin or the agent may keep for the system.
pub const MIN_RESERVE_MB: u32 = 512;

/// The memory genco keeps for the system (the databases, nginx, mail,
/// genco itself): a quarter of the RAM, at least 768 MB, at most half.
pub fn reserve(ram_mb: u32) -> u32 {
/// genco itself) unless set (memory.rs): a quarter of the RAM, at least
/// 768 MB, at most half.
pub fn default_reserve(ram_mb: u32) -> u32 {
(ram_mb / 4).max(768).min(ram_mb / 2)
}

/// What a reserve may be: `MIN_RESERVE_MB` to half the RAM.
pub fn reserve_bounds(ram_mb: u32) -> (u32, u32) {
(MIN_RESERVE_MB.min(ram_mb / 2), ram_mb / 2)
}

/// What all groups together may use (`genco.slice`): the RAM without the
/// `reserve`. Their own limits may add up to much more, they rarely peak
/// reserve. Their own limits may add up to much more, they rarely peak
/// together; when they do, a group runs out, never the system.
pub fn groups_cap(ram_mb: u32) -> u32 {
ram_mb - reserve(ram_mb)
pub fn groups_cap(ram_mb: u32, reserve_mb: u32) -> u32 {
ram_mb.saturating_sub(reserve_mb)
}

/// The unit of the slice all groups' slices are in.
pub fn groups_slice_unit(ram_mb: u32) -> String {
pub fn groups_slice_unit(cap_mb: u32) -> String {
format!(
"{}: genco sets it from the RAM
"{}: genco memory reserve changes it
[Unit]
Description=All groups of genco

[Slice]
MemoryMax={}
",
crate::render::HEADER,
unit_size(groups_cap(ram_mb))
unit_size(cap_mb)
)
}

Expand Down Expand Up @@ -533,7 +539,7 @@ mod tests {

#[test]
fn bounds_follow_the_server() {
let ok = |l: Limits| l.check(4000, 2, None).is_ok();
let ok = |l: Limits| l.check(3000, 2, None).is_ok();
assert!(ok(Limits::default()));
let mem = |mb| Limits {
memory_max_mb: Some(mb),
Expand All @@ -545,8 +551,8 @@ mod tests {
assert!(ok(mem(3000)));
assert!(!ok(mem(3001)));
// one it had before the cap stays
assert!(mem(3500).check(4000, 2, Some(3500)).is_ok());
assert!(mem(3600).check(4000, 2, Some(3500)).is_err());
assert!(mem(3500).check(3000, 2, Some(3500)).is_ok());
assert!(mem(3600).check(3000, 2, Some(3500)).is_err());
// a soft limit above the max
assert!(!ok(Limits {
memory_high_mb: Some(2048),
Expand All @@ -570,13 +576,15 @@ mod tests {

#[test]
fn the_system_keeps_a_reserve() {
assert_eq!(reserve(4000), 1000);
assert_eq!(groups_cap(4000), 3000);
assert_eq!(reserve(2048), 768);
assert_eq!(reserve(1024), 512);
assert_eq!(reserve(16384), 4096);
assert!(groups_slice_unit(4000).contains("MemoryMax=3000M\n"));
assert!(groups_slice_unit(4096).contains("MemoryMax=3G\n"));
assert_eq!(default_reserve(4000), 1000);
assert_eq!(groups_cap(4000, 1000), 3000);
assert_eq!(default_reserve(2048), 768);
assert_eq!(default_reserve(1024), 512);
assert_eq!(default_reserve(16384), 4096);
assert_eq!(reserve_bounds(4000), (512, 2000));
assert_eq!(reserve_bounds(800), (400, 400));
assert!(groups_slice_unit(3000).contains("MemoryMax=3000M\n"));
assert!(groups_slice_unit(3072).contains("MemoryMax=3G\n"));
}

#[test]
Expand Down
25 changes: 25 additions & 0 deletions server/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ mod llm;
mod mail;
mod mailbox;
mod mailfilter;
mod memory;
mod notify;
mod panel;
mod peers;
Expand Down Expand Up @@ -167,6 +168,12 @@ pub(crate) enum Cmd {
/// Block addresses from the sites, mail and workers (never ssh)
#[clap(subcommand)]
Firewall(firewall::FirewallCmd),
/// How the RAM is split between the groups and the system; change the
/// reserve for the system
Memory {
#[clap(subcommand)]
cmd: Option<memory::MemoryCmd>,
},
/// The routines and alerts that still need attention: list them,
/// resolve one that is taken care of
#[clap(subcommand)]
Expand Down Expand Up @@ -219,6 +226,7 @@ impl Cmd {
| Cmd::Setup { .. }
| Cmd::Apply | Cmd::Deploy { .. }
| Cmd::Backup { .. }
| Cmd::Memory { cmd: Some(_) }
| Cmd::Restic { .. }
| Cmd::Update { .. }
| Cmd::Panel(_)
Expand Down Expand Up @@ -248,6 +256,7 @@ impl Cmd {
| Cmd::Firewall(firewall::FirewallCmd::List)
| Cmd::Activity { .. }
| Cmd::Attention(conversations::AttentionCmd::List)
| Cmd::Memory { cmd: None }
| Cmd::Update { .. }
| Cmd::Runners(RunnersCmd::List)
| Cmd::Group(cli::GroupCmd::DeployKey { .. })
Expand Down Expand Up @@ -403,6 +412,22 @@ async fn run(args: Args) -> Result<(), String> {
}
Cmd::Docs { .. } => unreachable!("read before the config"),
Cmd::User(cmd) => user(cfg, cmd),
Cmd::Memory { cmd: None } => {
let db = Db::open(&cfg.db_path())?;
let s = memory::status(&db.conn()).map_err(|e| e.to_string())?;
println!("{}", s.text());
Ok(())
}
Cmd::Memory {
cmd: Some(memory::MemoryCmd::Reserve { size }),
} => {
let reserve =
limits::parse_size(&size).map_err(|e| e.to_string())?;
let db = Db::open(&cfg.db_path())?;
memory::set_reserve(&db.conn(), reserve)
.map_err(|e| e.to_string())?;
apply_cmd(cfg).await
}
Cmd::Attention(cmd) => {
edit(cfg, |_, conn| conversations::attention(conn, cmd))
}
Expand Down
Loading