From b4a2b10d94181ff8b58070fe98f5df492998ea8a Mon Sep 17 00:00:00 2001 From: Tom Sommer Date: Sat, 19 Sep 2026 15:35:47 +0200 Subject: [PATCH 1/3] Document request body buffering, temp-file spooling and their performance impact README gains a "Performance considerations" section covering request body buffering in the ACCESS phase, the effect on proxy_request_buffering, the client_body_buffer_size spill to a temp file that libmodsecurity re-reads by name, response body copying and sendfile, and rules loading. --- README.md | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/README.md b/README.md index a93e5df0..1ee15db0 100644 --- a/README.md +++ b/README.md @@ -185,6 +185,60 @@ modsecurity_use_error_log Turns on or off ModSecurity error log functionality. +# Performance considerations + +## Request body + +When `modsecurity` is on, the connector reads the **whole** request body in +nginx's ACCESS phase, before the content handler (for example `proxy_pass`) +runs. Consequences: + +* `proxy_request_buffering off` has no effect in such locations: the body has + already been buffered by the time the proxy module sees it. +* Bodies larger than `client_body_buffer_size` (twice the system page size, 8k + on most platforms) are written to a temporary file under + `client_body_temp_path`. libmodsecurity then re-opens that file and reads it + back into memory in full, because ModSecurity v3 has no + `SecRequestBodyInMemoryLimit`: the directive was removed and the rule parser + rejects it as a configuration error. The spill therefore costs a write and a + read and saves no memory. Raising `client_body_buffer_size` avoids the spill, + but nginx allocates that buffer for every request that carries a body (it is + trimmed to the body size only when the Content-Length is known and smaller), + so a large value multiplies worker memory under concurrency. Size it against + the bodies you actually expect, not against `client_max_body_size`, which + only decides which requests are accepted, nor against `SecRequestBodyLimit`, + which is applied after the connector has already read the body. A + tmpfs-backed `client_body_temp_path` keeps the spill off disk, at the price + of holding it in RAM. +* libmodsecurity copies the body into its own buffer even when + `SecRequestBodyAccess Off` is set; the copy is made before libmodsecurity + returns early from phase 2, so the buffering above is not avoided either. + +## Response body + +The connector feeds every response buffer to libmodsecurity regardless of +`SecResponseBodyAccess`. libmodsecurity skips the copy only for responses whose +Content-Type is not listed in `SecResponseBodyMimeType` (with no such list +every response is copied), and applies `SecResponseBodyLimit` together with +`SecResponseBodyLimitAction` once the copy grows past the limit. The connector +also asks nginx to keep every response body in memory +(`filter_need_in_memory`), again regardless of `SecResponseBodyAccess`. In +locations where `modsecurity` is on, a file-backed response (a static file, or +a proxied response buffered to disk) is therefore read into memory and copied +through the output chain before it is sent, even when no rule ever looks at +it. `sendfile` is still used to transmit it, so the cost is the extra read and +copy rather than the loss of `sendfile`. Each buffer chain is fed to +ModSecurity before it is forwarded, and phase 4 runs on the chain that carries +the end of the response, before that chain is passed on. A phase 4 rule can +therefore still act on the buffers of that last chain, but not on anything the +preceding calls have already sent to the client. + +## Rules loading + +Each `modsecurity_rules_file` directive parses and compiles the referenced +file again. Load a large rule set such as the OWASP Core Rule Set once at the +`http` level and only toggle `modsecurity on|off` per server or location. + # Contributing As an open source project we invite (and encourage) anyone from the community to contribute to our project. This may take the form of: new From eb10fe63eaaea846b66ced82ab36d69c908ed8d2 Mon Sep 17 00:00:00 2001 From: Tom Sommer Date: Fri, 13 Nov 2026 07:36:03 +0100 Subject: [PATCH 2/3] README: correct the performance section after review Request body: - Only "modsecurity off" avoids the buffering and the copy. The access handler reads the body whenever the module is enabled, and Transaction::appendRequestBody/appendResponseBody do not check the rule engine state; only the process* phases do. - Spill threshold: ngx_http_read_client_request_body uses size = client_body_buffer_size + (client_body_buffer_size >> 2) and allocates rest (+ preread, request_body_in_single_buf) when a non-chunked rest is below that, so HTTP/1.x bodies with a known length stay in memory up to ~1.25x. Chunked bodies get client_body_buffer_size; ngx_http_v2_read_request_body allocates min(Content-Length + 1, client_body_buffer_size). A body whose rest fits in r->header_in uses that buffer and allocates nothing. - Transaction::requestBodyFromFile reads the whole file into a local std::string before appendRequestBody copies it into m_requestBody, so the body is held twice while loading and the file read is not bounded by SecRequestBodyLimit (which only bounds m_requestBody). - request_body_in_persistent_file keeps the temp file named (ngx_open_tempfile unlinks only non-persistent files); it is opened 0600 and removed by ngx_pool_delete_file because the connector sets request_body_in_clean_file. request_body_file_group_access (0660) is set only by the dav module, after the connector has already read the body. ngx_create_paths creates a missing temp path 0700 but does not tighten an existing one. Covers what #381 documented. - Transaction::processRequestBody runs the multipart/urlencoded/JSON/XML processors before the SecRequestBodyAccess early return; the DisabledRuleEngine return comes before them. Response body: - ngx_output_chain reads in_file buffers into at most output_buffers (default 2 32k) temp buffers and keeps in_file set for sendfile. - SecResponseBodyMimeTypesClear sets m_set with an empty set: appendResponseBody only skips when the set is non-empty, processResponseBody skips when m_set and no match. - Phase 4 runs after the header filter has passed the headers on; with r->header_sent the intervention returns -1 and the body filter calls ngx_http_filter_finalize_request, so the response is aborted (tests/modsecurity.t expects '' for /phase4). Rules loading: - ngx_http_modsecurity_create_conf calls msc_create_rules_set for every location conf and merge_conf always calls msc_rules_merge, which appends shared_ptr references per phase after a duplicate id check. --- README.md | 100 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 65 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index 1ee15db0..d927bbfe 100644 --- a/README.md +++ b/README.md @@ -191,53 +191,83 @@ Turns on or off ModSecurity error log functionality. When `modsecurity` is on, the connector reads the **whole** request body in nginx's ACCESS phase, before the content handler (for example `proxy_pass`) -runs. Consequences: +runs, and libmodsecurity keeps its own copy of it. Only `modsecurity off` +avoids this: `SecRuleEngine Off` or `DetectionOnly` and +`SecRequestBodyAccess Off` change what libmodsecurity does with the copy, not +whether the body is buffered and copied. Consequences: * `proxy_request_buffering off` has no effect in such locations: the body has already been buffered by the time the proxy module sees it. -* Bodies larger than `client_body_buffer_size` (twice the system page size, 8k - on most platforms) are written to a temporary file under - `client_body_temp_path`. libmodsecurity then re-opens that file and reads it - back into memory in full, because ModSecurity v3 has no - `SecRequestBodyInMemoryLimit`: the directive was removed and the rule parser - rejects it as a configuration error. The spill therefore costs a write and a - read and saves no memory. Raising `client_body_buffer_size` avoids the spill, - but nginx allocates that buffer for every request that carries a body (it is - trimmed to the body size only when the Content-Length is known and smaller), - so a large value multiplies worker memory under concurrency. Size it against - the bodies you actually expect, not against `client_max_body_size`, which - only decides which requests are accepted, nor against `SecRequestBodyLimit`, - which is applied after the connector has already read the body. A - tmpfs-backed `client_body_temp_path` keeps the spill off disk, at the price - of holding it in RAM. -* libmodsecurity copies the body into its own buffer even when - `SecRequestBodyAccess Off` is set; the copy is made before libmodsecurity - returns early from phase 2, so the buffering above is not avoided either. +* A body that does not fit in nginx's body buffer is written to a temporary + file under `client_body_temp_path`. Over HTTP/1.x with a Content-Length, + bodies up to about 1.25 times `client_body_buffer_size` (twice the system + page size, 8k on most platforms) stay in memory; chunked and HTTP/2 bodies + spill once they exceed `client_body_buffer_size`. +* libmodsecurity reopens a spilled file by name and reads it back in full, + because ModSecurity v3 has no `SecRequestBodyInMemoryLimit` (the rule + parser rejects it). It reads the whole file into a temporary string and + then copies that into its own buffer, so the body is held twice while it is + loaded, and the file is read in full even when it exceeds + `SecRequestBodyLimit`, which only caps the second copy. The spill costs a + write and a read and saves no memory. +* Raising `client_body_buffer_size` avoids the spill, but every body + without a Content-Length, and every body that spills, gets a buffer of the + full size (a body that stays in memory gets one of its own size, and one + that fits in the rest of the request header buffer needs none), so a large + value multiplies worker memory under concurrency. Size it against the + bodies you actually expect, not against `client_max_body_size`, which only + decides which requests are accepted, nor against `SecRequestBodyLimit`, + which is applied after the connector has already read the body. +* The spilled file keeps its name under `client_body_temp_path` for the + duration of the request, because libmodsecurity reopens it by path; it is + created with mode 0600 and deleted when the request ends (unless + `client_body_in_file_only on` asks nginx to keep it). nginx creates a + missing `client_body_temp_path` with mode 0700, but leaves the permissions + of an existing directory alone: keep that directory private to the nginx + user. A tmpfs-backed `client_body_temp_path` keeps the spill off disk, at + the price of holding it in RAM. +* With `SecRequestBodyAccess Off`, libmodsecurity still runs the request + body processor (multipart, urlencoded, or JSON/XML when selected with + `ctl:requestBodyProcessor`) over its copy in phase 2, and only then returns + before evaluating the phase 2 rules. `SecRuleEngine Off` skips the parsing + as well, but not the buffering or the copy. ## Response body -The connector feeds every response buffer to libmodsecurity regardless of -`SecResponseBodyAccess`. libmodsecurity skips the copy only for responses whose -Content-Type is not listed in `SecResponseBodyMimeType` (with no such list -every response is copied), and applies `SecResponseBodyLimit` together with -`SecResponseBodyLimitAction` once the copy grows past the limit. The connector -also asks nginx to keep every response body in memory -(`filter_need_in_memory`), again regardless of `SecResponseBodyAccess`. In -locations where `modsecurity` is on, a file-backed response (a static file, or -a proxied response buffered to disk) is therefore read into memory and copied -through the output chain before it is sent, even when no rule ever looks at -it. `sendfile` is still used to transmit it, so the cost is the extra read and -copy rather than the loss of `sendfile`. Each buffer chain is fed to -ModSecurity before it is forwarded, and phase 4 runs on the chain that carries -the end of the response, before that chain is passed on. A phase 4 rule can -therefore still act on the buffers of that last chain, but not on anything the -preceding calls have already sent to the client. +The connector asks nginx to keep every response body in memory +(`filter_need_in_memory`) and feeds every response buffer to libmodsecurity, +regardless of `SecResponseBodyAccess` and `SecRuleEngine`; again only +`modsecurity off` avoids this. + +* A file-backed response (a static file, or a proxied response buffered to + disk) is read into memory in `output_buffers`-sized pieces (default + `2 32k`) before it is passed on, so nginx's own extra memory per request + stays bounded by `output_buffers`. `sendfile` is still used to transmit it, + so the cost is the extra read and copy rather than the loss of `sendfile`. +* libmodsecurity accumulates its own copy of the whole response. It skips + the copy only for responses whose Content-Type is not in a non-empty + `SecResponseBodyMimeType` list, and caps it at `SecResponseBodyLimit` + according to `SecResponseBodyLimitAction`. To limit the copy, list only the + types you inspect. +* Do not use `SecResponseBodyMimeTypesClear` to turn response inspection off. + If it leaves the list empty, libmodsecurity copies every response, but + phase 4 inspects none of them. +* Phase 4 runs on the buffer that ends the response, after the response + headers have been committed and earlier buffers have been passed on. A + disruptive phase 4 action can therefore neither send its status nor + redirect: the connector aborts the response, and the client receives + nothing or a truncated response. ## Rules loading Each `modsecurity_rules_file` directive parses and compiles the referenced file again. Load a large rule set such as the OWASP Core Rule Set once at the `http` level and only toggle `modsecurity on|off` per server or location. +The rule objects are then shared, but every `server` and `location` block, +including those with `modsecurity off` or without any modsecurity directive, +still gets its own rules set into which the enclosing level's rules are +merged at configuration load, so configuration load time and memory grow +with the number of locations times the number of rules. # Contributing From 0ad6ff2be3898b6a1983e5c907a9721eb347cf87 Mon Sep 17 00:00:00 2001 From: Tom Sommer Date: Fri, 13 Nov 2026 07:39:38 +0100 Subject: [PATCH 3/3] README: HTTP/3 body buffer, temp path permissions, NoFilesLimit ngx_http_v3_read_request_body allocates client_body_buffer_size unconditionally, and the header-buffer shortcut is in the HTTP/1.x path only. ngx_create_paths chowns an existing temp path and adds owner rwx but never clears group/other bits. Transaction::processRequestBody skips the urlencoded, JSON and XML processors when the body exceeds SecRequestBodyNoFilesLimit; multipart always runs. --- README.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index d927bbfe..44b82ccb 100644 --- a/README.md +++ b/README.md @@ -211,10 +211,11 @@ whether the body is buffered and copied. Consequences: `SecRequestBodyLimit`, which only caps the second copy. The spill costs a write and a read and saves no memory. * Raising `client_body_buffer_size` avoids the spill, but every body - without a Content-Length, and every body that spills, gets a buffer of the - full size (a body that stays in memory gets one of its own size, and one - that fits in the rest of the request header buffer needs none), so a large - value multiplies worker memory under concurrency. Size it against the + without a Content-Length, every body that spills and every HTTP/3 body gets + a buffer of the full size (otherwise a body that stays in memory gets one + of its own size, and an HTTP/1.x body that fits in the rest of the request + header buffer needs none), so a large value multiplies worker memory under + concurrency. Size it against the bodies you actually expect, not against `client_max_body_size`, which only decides which requests are accepted, nor against `SecRequestBodyLimit`, which is applied after the connector has already read the body. @@ -222,14 +223,16 @@ whether the body is buffered and copied. Consequences: duration of the request, because libmodsecurity reopens it by path; it is created with mode 0600 and deleted when the request ends (unless `client_body_in_file_only on` asks nginx to keep it). nginx creates a - missing `client_body_temp_path` with mode 0700, but leaves the permissions - of an existing directory alone: keep that directory private to the nginx - user. A tmpfs-backed `client_body_temp_path` keeps the spill off disk, at + missing `client_body_temp_path` with mode 0700, but never removes group or + other access from an existing directory: keep that directory private to + the nginx user. A tmpfs-backed `client_body_temp_path` keeps the spill off disk, at the price of holding it in RAM. * With `SecRequestBodyAccess Off`, libmodsecurity still runs the request body processor (multipart, urlencoded, or JSON/XML when selected with `ctl:requestBodyProcessor`) over its copy in phase 2, and only then returns - before evaluating the phase 2 rules. `SecRuleEngine Off` skips the parsing + before evaluating the phase 2 rules. Only a body over + `SecRequestBodyNoFilesLimit` skips the urlencoded, JSON and XML parsers; + multipart is always parsed. `SecRuleEngine Off` skips the parsing as well, but not the buffering or the copy. ## Response body