From 22b3d7519bff8c08c1d15ac587191588f32c6bc0 Mon Sep 17 00:00:00 2001 From: Tom Sommer Date: Sat, 19 Sep 2026 15:45:10 +0200 Subject: [PATCH 1/6] Add modsecurity_log_thread_pool to run phase 5 and audit logging in a thread msc_process_logging() evaluates the LOGGING phase and writes the audit log synchronously on the worker's event loop. With SecAuditEngine On, or RelevantOnly under attack traffic, that is per-request file I/O inside the worker. The new opt-in directive posts that work to an nginx thread pool: the transaction is detached from the request pool, processed in the thread, and cleaned up from the completion handler. The libmodsecurity log callback now receives a small holder struct instead of the connection log so its destination can be redirected to the thread pool's log once the request is gone. The early logging path used by interventions stays synchronous. Requires --with-threads and PCRE2; PCRE1 builds refuse the directive because the connector's pcre_malloc swap is process-global. --- README.md | 64 +++++++++++- src/ngx_http_modsecurity_common.h | 23 ++++- src/ngx_http_modsecurity_log.c | 150 ++++++++++++++++++++++++--- src/ngx_http_modsecurity_module.c | 111 +++++++++++++++++--- tests/modsecurity-log-thread-pool.t | 151 ++++++++++++++++++++++++++++ 5 files changed, 472 insertions(+), 27 deletions(-) create mode 100644 tests/modsecurity-log-thread-pool.t diff --git a/README.md b/README.md index a93e5df0..e3a0a1e5 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ Further information about nginx third-party add-ons support are available [here] # Usage ModSecurity for nginx extends your nginx configuration directives. -It adds four new directives and they are: +It adds the following directives: modsecurity ----------- @@ -185,6 +185,68 @@ modsecurity_use_error_log Turns on or off ModSecurity error log functionality. +modsecurity_log_thread_pool +--------------------------- +**syntax:** *modsecurity_log_thread_pool <name> | off* + +**context:** *http, server, location* + +**default:** *off* + +Runs the LOGGING phase (phase 5) rules and the audit log write in the named +nginx `thread_pool` instead of on the worker's event loop, so that audit log +I/O no longer stalls the other connections handled by the worker. The +transaction is detached from the request when the task is posted and is +cleaned up once the task completes. + +This directive requires nginx built with `--with-threads` and with PCRE2; +builds using the original PCRE library refuse it at configuration time, +because there the connector has to swap the process-global `pcre_malloc` and +`pcre_free` pointers around every libmodsecurity call, which is not +thread-safe. + +Rule messages emitted during phase 5 are written using the thread pool's log, +which is the main context `error_log` at its configured level, rather than the +`error_log` of the server or location the request was handled by, and they lack +the usual client and request context. As they are emitted at the `info` level, +the main `error_log` has to be set to `info` or lower for them to appear, and it +has to be a file: an `error_log` that writes to `syslog:` or `memory:` must not +be used with this directive, because those writers are not thread-safe. Any +`SecDebugLog` output produced during phase 5 is likewise written from the +thread. + +The logging performed early by a disruptive intervention stays synchronous, as +does everything up to phase 4 and the logging phase of subrequests. + +The feature is experimental, and it needs a libmodsecurity that supports +processing transactions concurrently -- the way the threaded Apache MPMs use +it. Two caveats are worth knowing about: + +* concurrent writes of serial audit log entries rely on the atomicity of a + single `fwrite()`; the `fcntl()` lock taken by the library does not exclude + other threads of the same process. `SecAuditLogType Concurrent`, or a thread + pool with a single thread, avoids the question; +* the LMDB collection backend (`--with-lmdb`) has not been reviewed for use + from several threads of one process. + +Logging tasks that are still queued when a worker shuts down are still run -- +the pool's exit task queues behind them -- so their audit log entries are +written, but their completion handler does not run, and the transaction, the +log holder and the task pool are released only by the exiting process. Because +the worker waits for the whole queue to drain, a thread blocked on a contended +audit log can delay shutdown beyond `worker_shutdown_timeout`, which does not +apply to thread pools. + +```nginx +thread_pool modsec threads=4; + +http { + modsecurity on; + modsecurity_rules_file /etc/modsecurity/main.conf; + modsecurity_log_thread_pool modsec; +} +``` + # Contributing As an open source project we invite (and encourage) anyone from the community to contribute to our project. This may take the form of: new diff --git a/src/ngx_http_modsecurity_common.h b/src/ngx_http_modsecurity_common.h index 9f79b5ba..ab908040 100644 --- a/src/ngx_http_modsecurity_common.h +++ b/src/ngx_http_modsecurity_common.h @@ -22,6 +22,10 @@ #include #include +#if (NGX_THREADS) && (NGX_PCRE2) +#include +#endif + #include #include @@ -77,10 +81,22 @@ typedef struct { } ngx_http_modsecurity_header_t; +/* + * Holder for the log destination given to libmodsecurity as the log callback + * data. The indirection allows the destination to be redirected to the + * thread pool's log once the request -- and thus its connection log -- is + * gone. + */ +typedef struct { + ngx_log_t *log; +} ngx_http_modsecurity_msclog_t; + + typedef struct { ngx_http_request_t *r; Transaction *modsec_transaction; ModSecurityIntervention *delayed_intervention; + ngx_http_modsecurity_msclog_t *msclog; #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) /* @@ -100,6 +116,7 @@ typedef struct { unsigned logged:1; unsigned intervention_triggered:1; unsigned request_body_processed:1; + unsigned msclog_heap:1; } ngx_http_modsecurity_ctx_t; @@ -124,6 +141,9 @@ typedef struct { #endif ngx_http_complex_value_t *transaction_id; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_thread_pool_t *log_thread_pool; +#endif } ngx_http_modsecurity_conf_t; @@ -163,8 +183,9 @@ int ngx_http_modsecurity_store_ctx_header(ngx_http_request_t *r, ngx_str_t *name #endif /* ngx_http_modsecurity_log.c */ -void ngx_http_modsecurity_log(void *log, const void* data); +void ngx_http_modsecurity_log(void *data, const void* msg); ngx_int_t ngx_http_modsecurity_log_handler(ngx_http_request_t *r); +ngx_int_t ngx_http_modsecurity_log_phase_handler(ngx_http_request_t *r); /* ngx_http_modsecurity_access.c */ ngx_int_t ngx_http_modsecurity_access_handler(ngx_http_request_t *r); diff --git a/src/ngx_http_modsecurity_log.c b/src/ngx_http_modsecurity_log.c index 094685c5..69621792 100644 --- a/src/ngx_http_modsecurity_log.c +++ b/src/ngx_http_modsecurity_log.c @@ -24,22 +24,153 @@ void -ngx_http_modsecurity_log(void *log, const void* data) +ngx_http_modsecurity_log(void *data, const void* msg) { - const char *msg; - if (log == NULL) { + ngx_http_modsecurity_msclog_t *msclog = data; + + if (msclog == NULL || msclog->log == NULL) { return; } - msg = (const char *) data; - ngx_log_error(NGX_LOG_INFO, (ngx_log_t *)log, 0, "%s", msg); + ngx_log_error(NGX_LOG_INFO, msclog->log, 0, "%s", (const char *) msg); +} + + +static ngx_int_t +ngx_http_modsecurity_log_transaction(ngx_http_request_t *r, + ngx_http_modsecurity_ctx_t *ctx) +{ + ngx_pool_t *old_pool; + + dd("calling msc_process_logging for %p", ctx); + old_pool = ngx_http_modsecurity_pcre_malloc_init(r->pool); + msc_process_logging(ctx->modsec_transaction); + ngx_http_modsecurity_pcre_malloc_done(old_pool); + + return NGX_OK; +} + + +#if (NGX_THREADS) && (NGX_PCRE2) + +typedef struct { + ngx_pool_t *pool; + Transaction *transaction; + ngx_http_modsecurity_msclog_t *msclog; +} ngx_http_modsecurity_log_task_t; + + +static void +ngx_http_modsecurity_log_thread_handler(void *data, ngx_log_t *log) +{ + ngx_http_modsecurity_log_task_t *lt = data; + + /* the request and its connection log may be gone by now */ + lt->msclog->log = log; + + msc_process_logging(lt->transaction); +} + + +static void +ngx_http_modsecurity_log_thread_event_handler(ngx_event_t *ev) +{ + ngx_http_modsecurity_log_task_t *lt = ev->data; + + msc_transaction_cleanup(lt->transaction); + ngx_free(lt->msclog); + ngx_destroy_pool(lt->pool); +} + + +static ngx_int_t +ngx_http_modsecurity_log_offload(ngx_http_request_t *r, + ngx_http_modsecurity_ctx_t *ctx, ngx_thread_pool_t *tp) +{ + ngx_pool_t *pool; + ngx_thread_task_t *task; + ngx_http_modsecurity_log_task_t *lt; + + pool = ngx_create_pool(512, ngx_cycle->log); + if (pool == NULL) { + return NGX_ERROR; + } + + task = ngx_thread_task_alloc(pool, sizeof(ngx_http_modsecurity_log_task_t)); + if (task == NULL) { + ngx_destroy_pool(pool); + return NGX_ERROR; + } + + lt = task->ctx; + lt->pool = pool; + lt->transaction = ctx->modsec_transaction; + lt->msclog = ctx->msclog; + + task->handler = ngx_http_modsecurity_log_thread_handler; + task->event.data = lt; + task->event.handler = ngx_http_modsecurity_log_thread_event_handler; + + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "ModSecurity: posting logging to the thread pool"); + + if (ngx_thread_task_post(tp, task) != NGX_OK) { + ngx_destroy_pool(pool); + return NGX_ERROR; + } + + /* the task now owns the transaction and the log holder */ + ctx->modsec_transaction = NULL; + ctx->msclog = NULL; + + return NGX_OK; +} + +#endif + + +ngx_int_t +ngx_http_modsecurity_log_phase_handler(ngx_http_request_t *r) +{ + ngx_http_modsecurity_ctx_t *ctx; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_conf_t *mcf; +#endif + + ctx = ngx_http_modsecurity_get_module_ctx(r); + if (ctx == NULL || ctx->logged || ctx->modsec_transaction == NULL) { + return NGX_OK; + } + +#if (NGX_THREADS) && (NGX_PCRE2) + /* + * Subrequests do not own a context of their own -- the access phase is + * skipped for them -- so their log phase, which runs while the main + * request is still being processed, must not detach the main request's + * transaction. + */ + mcf = ngx_http_get_module_loc_conf(r, ngx_http_modsecurity_module); + + if (r == r->main && mcf->log_thread_pool != NULL && ctx->msclog_heap) { + if (ngx_http_modsecurity_log_offload(r, ctx, mcf->log_thread_pool) + == NGX_OK) + { + return NGX_OK; + } + + ngx_log_error(NGX_LOG_WARN, r->connection->log, 0, + "ModSecurity: could not post logging to thread pool, " + "processing inline"); + } +#endif + + return ngx_http_modsecurity_log_transaction(r, ctx); } ngx_int_t ngx_http_modsecurity_log_handler(ngx_http_request_t *r) { - ngx_pool_t *old_pool; ngx_http_modsecurity_ctx_t *ctx; dd("catching a new _log_ phase handler"); @@ -66,10 +197,5 @@ ngx_http_modsecurity_log_handler(ngx_http_request_t *r) return NGX_OK; } - dd("calling msc_process_logging for %p", ctx); - old_pool = ngx_http_modsecurity_pcre_malloc_init(r->pool); - msc_process_logging(ctx->modsec_transaction); - ngx_http_modsecurity_pcre_malloc_done(old_pool); - - return NGX_OK; + return ngx_http_modsecurity_log_transaction(r, ctx); } diff --git a/src/ngx_http_modsecurity_module.c b/src/ngx_http_modsecurity_module.c index d3d9624d..d2499475 100644 --- a/src/ngx_http_modsecurity_module.c +++ b/src/ngx_http_modsecurity_module.c @@ -255,7 +255,15 @@ ngx_http_modsecurity_cleanup(void *data) ctx = (ngx_http_modsecurity_ctx_t *) data; - msc_transaction_cleanup(ctx->modsec_transaction); + if (ctx->modsec_transaction != NULL) { + msc_transaction_cleanup(ctx->modsec_transaction); + ctx->modsec_transaction = NULL; + } + + if (ctx->msclog_heap && ctx->msclog != NULL) { + ngx_free(ctx->msclog); + ctx->msclog = NULL; + } #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) /* @@ -288,29 +296,64 @@ ngx_http_modsecurity_create_ctx(ngx_http_request_t *r) dd("creating transaction with the following rules: '%p' -- ms: '%p'", mcf->rules_set, mmcf->modsec); + /* + * Take the cleanup slot before anything that has to be released: from + * here on every error return leaves the transaction and the log holder + * to ngx_http_modsecurity_cleanup(), which copes with either being NULL. + */ + cln = ngx_pool_cleanup_add(r->pool, sizeof(ngx_http_modsecurity_ctx_t)); + if (cln == NULL) + { + dd("failed to create the ModSecurity context cleanup"); + return NGX_CONF_ERROR; + } + cln->handler = ngx_http_modsecurity_cleanup; + cln->data = ctx; + +#if (NGX_THREADS) && (NGX_PCRE2) + if (mcf->log_thread_pool != NULL) { + /* + * the holder must outlive the request pool: the logging phase is + * processed in a thread after the request is gone + */ + ctx->msclog = ngx_alloc(sizeof(ngx_http_modsecurity_msclog_t), + r->connection->log); + ctx->msclog_heap = 1; + + } else +#endif + { + ctx->msclog = ngx_palloc(r->pool, + sizeof(ngx_http_modsecurity_msclog_t)); + } + + if (ctx->msclog == NULL) { + dd("failed to allocate memory for the log holder."); + /* + * nothing has to be released yet; disarm the cleanup so that the + * half-built context stays invisible to + * ngx_http_modsecurity_get_module_ctx() + */ + cln->handler = NULL; + return NULL; + } + + ctx->msclog->log = r->connection->log; + if (mcf->transaction_id) { if (ngx_http_complex_value(r, mcf->transaction_id, &s) != NGX_OK) { return NGX_CONF_ERROR; } - ctx->modsec_transaction = msc_new_transaction_with_id(mmcf->modsec, mcf->rules_set, (char *) s.data, r->connection->log); + ctx->modsec_transaction = msc_new_transaction_with_id(mmcf->modsec, mcf->rules_set, (char *) s.data, ctx->msclog); } else { - ctx->modsec_transaction = msc_new_transaction(mmcf->modsec, mcf->rules_set, r->connection->log); + ctx->modsec_transaction = msc_new_transaction(mmcf->modsec, mcf->rules_set, ctx->msclog); } dd("transaction created"); ngx_http_set_ctx(r, ctx, ngx_http_modsecurity_module); - cln = ngx_pool_cleanup_add(r->pool, sizeof(ngx_http_modsecurity_ctx_t)); - if (cln == NULL) - { - dd("failed to create the ModSecurity context cleanup"); - return NGX_CONF_ERROR; - } - cln->handler = ngx_http_modsecurity_cleanup; - cln->data = ctx; - #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) ctx->sanity_headers_out = ngx_array_create(r->pool, 12, sizeof(ngx_http_modsecurity_header_t)); if (ctx->sanity_headers_out == NULL) { @@ -480,6 +523,34 @@ char *ngx_conf_set_transaction_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf } +static char * +ngx_conf_set_log_thread_pool(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) +{ +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_conf_t *mcf = conf; + ngx_str_t *value = cf->args->elts; + + if (mcf->log_thread_pool != NGX_CONF_UNSET_PTR) { + return "is duplicate"; + } + + if (ngx_strcmp(value[1].data, "off") == 0) { + mcf->log_thread_pool = NULL; + return NGX_CONF_OK; + } + + mcf->log_thread_pool = ngx_thread_pool_add(cf, &value[1]); + if (mcf->log_thread_pool == NULL) { + return NGX_CONF_ERROR; + } + + return NGX_CONF_OK; +#else + return "requires nginx built with --with-threads and PCRE2"; +#endif +} + + static ngx_command_t ngx_http_modsecurity_commands[] = { { ngx_string("modsecurity"), @@ -521,6 +592,14 @@ static ngx_command_t ngx_http_modsecurity_commands[] = { 0, NULL }, + { + ngx_string("modsecurity_log_thread_pool"), + NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, + ngx_conf_set_log_thread_pool, + NGX_HTTP_LOC_CONF_OFFSET, + 0, + NULL + }, { ngx_string("modsecurity_use_error_log"), NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_FLAG, @@ -607,7 +686,7 @@ ngx_http_modsecurity_init(ngx_conf_t *cf) dd("Not able to create a new NGX_HTTP_LOG_PHASE handle"); return NGX_ERROR; } - *h_log = ngx_http_modsecurity_log_handler; + *h_log = ngx_http_modsecurity_log_phase_handler; rc = ngx_http_modsecurity_header_filter_init(); @@ -724,6 +803,9 @@ ngx_http_modsecurity_create_conf(ngx_conf_t *cf) conf->pool = cf->pool; conf->transaction_id = NGX_CONF_UNSET_PTR; conf->use_error_log = NGX_CONF_UNSET; +#if (NGX_THREADS) && (NGX_PCRE2) + conf->log_thread_pool = NGX_CONF_UNSET_PTR; +#endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) conf->sanity_checks_enabled = NGX_CONF_UNSET; #endif @@ -764,6 +846,9 @@ ngx_http_modsecurity_merge_conf(ngx_conf_t *cf, void *parent, void *child) ngx_conf_merge_value(c->enable, p->enable, 0); ngx_conf_merge_ptr_value(c->transaction_id, p->transaction_id, NULL); ngx_conf_merge_value(c->use_error_log, p->use_error_log, 1); +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_conf_merge_ptr_value(c->log_thread_pool, p->log_thread_pool, NULL); +#endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) ngx_conf_merge_value(c->sanity_checks_enabled, p->sanity_checks_enabled, 0); #endif diff --git a/tests/modsecurity-log-thread-pool.t b/tests/modsecurity-log-thread-pool.t new file mode 100644 index 00000000..f810d680 --- /dev/null +++ b/tests/modsecurity-log-thread-pool.t @@ -0,0 +1,151 @@ +#!/usr/bin/perl + +# +# ModSecurity, http://www.modsecurity.org/ +# Copyright (c) 2015 Trustwave Holdings, Inc. (http://www.trustwave.com/) +# +# You may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# If any of the files related to licensing are missing or if you have any +# other questions related to licensing please contact Trustwave Holdings, Inc. +# directly using the email address security@modsecurity.org. +# + + +# Tests for ModSecurity module, phase 5 and audit logging in a thread pool. + +############################################################################### + +use warnings; +use strict; + +use Test::More; + +BEGIN { use FindBin; chdir($FindBin::Bin); } + +use lib 'lib'; +use Test::Nginx; + +############################################################################### + +select STDERR; $| = 1; +select STDOUT; $| = 1; + +my $t = Test::Nginx->new()->has(qw/http threads/); + +plan(skip_all => 'nginx built with --without-pcre2') + unless $t->has_module('(?s)^(?!.*--without-pcre2)'); + +$t->write_file_expand('nginx.conf', <<'EOF'); + +%%TEST_GLOBALS%% + +daemon off; + +thread_pool modsec threads=2; + +events { +} + +http { + %%TEST_GLOBALS_HTTP%% + + server { + listen 127.0.0.1:8080; + server_name localhost; + + modsecurity on; + modsecurity_log_thread_pool modsec; + modsecurity_rules ' + SecRuleEngine On + SecRule ARGS:what "@streq block" "id:71,phase:1,deny,status:403,log,auditlog" + SecRule ARGS:what "@streq p5" "id:72,phase:5,log,auditlog,pass" + '; + + # libmodsecurity merges a non-empty parent SecAuditLog over the + # child's one, so the audit log is configured per location here. + + location / { + modsecurity_rules ' + SecAuditEngine RelevantOnly + SecAuditLogParts ABH + SecAuditLog %%TESTDIR%%/audit.txt + SecAuditLogType Serial + SecAuditLogStorageDir %%TESTDIR%%/ + '; + } + + location /sync/ { + modsecurity_log_thread_pool off; + modsecurity_rules ' + SecAuditEngine RelevantOnly + SecAuditLogParts ABH + SecAuditLog %%TESTDIR%%/audit-sync.txt + SecAuditLogType Serial + SecAuditLogStorageDir %%TESTDIR%%/ + '; + } + } +} + +EOF + +$t->write_file('index.html', 'INDEX'); +mkdir($t->testdir() . '/sync'); +$t->write_file('/sync/index.html', 'SYNC INDEX'); +$t->run(); +$t->plan(11); + +############################################################################### + +like(http_get('/index.html?what=block'), qr/^HTTP.*403/, 'phase 1 deny still blocks'); +like(http_get('/index.html?what=p5'), qr/INDEX/, 'request with phase 5 rule passes'); +like(http_get('/index.html?what=none'), qr/INDEX/, 'plain request passes'); +like(http_get('/sync/index.html?what=p5'), qr/SYNC INDEX/, 'thread pool off in location'); + +my $audit = wait_for_file($t->testdir() . '/audit.txt', qr/\[id "72"\]/); +like($audit, qr/\[id "72"\]/, 'phase 5 rule logged from thread'); +like($audit, qr/what=block/, 'blocked request audited'); +like($audit, qr/\[id "71"\]/, 'phase 1 deny audited'); + +my $sync = wait_for_file($t->testdir() . '/audit-sync.txt', qr/\[id "72"\]/); +like($sync, qr/\[id "72"\]/, 'synchronous location still audits'); + +# The thread pool's log is the main error_log, which has no handler, so a rule +# message written from the thread carries no ", client: ..." suffix, while one +# written on the request's connection log does. This tells the offloaded path +# apart from the (deliberately silent) synchronous fallback. + +my $errlog = read_file($t->testdir() . '/error.log'); +unlike($errlog, qr/could not post logging/, 'logging was not refused by the pool'); +like($errlog, qr/^(?:(?!client:).)*\[id "72"\](?:(?!client:).)*$/m, + 'phase 5 rule message written from the thread pool log'); +like($errlog, qr/^.*\[id "72"\].*client:.*$/m, + 'phase 5 rule message of the synchronous location keeps the request context'); + +############################################################################### + +sub wait_for_file { + my ($path, $re) = @_; + my $data = ''; + for (1 .. 50) { + $data = read_file($path); + last if $data =~ $re; + select undef, undef, undef, 0.1; + } + return $data; +} + +sub read_file { + my ($path) = @_; + open my $fh, '<', $path or return ''; + local $/; + my $data = <$fh>; + close $fh; + return defined $data ? $data : ''; +} + +############################################################################### From fa8d9ce1bb8f6fe34849b3c9fb5a551a99095e8a Mon Sep 17 00:00:00 2001 From: Tom Sommer Date: Fri, 13 Nov 2026 07:38:55 +0100 Subject: [PATCH 2/6] Log thread pool: bound pending tasks, fix ctx lifetime and subrequests Rework after review: - create_ctx: take the cleanup slot unarmed and arm it, and publish the ctx, only once the transaction exists (the pattern #384 uses), so a failed create (transaction id evaluation, allocation, sanity array) leaves nothing that ngx_http_modsecurity_get_module_ctx() can find with a NULL transaction. The sanity header array is created before the slot as well, since the cleanup destroys it. - One allocation per request: the log holder, the ngx_thread_task_t and the transaction pointer live in one ngx_calloc'd struct, created in create_ctx and freed either by the request cleanup or, once posted, by the task's completion handler. No pool per task any more. - The offload decision is taken at log time only, from the location the request is logged in. The struct is allocated for every request when any location names a pool (a main-conf flag), so a request whose ctx was created in a location without the directive and that is then internally redirected into one with it is offloaded too. Configurations not using the directive keep the pool-allocated holder and pay nothing. - New max_pending=N parameter (default 128): a per-worker count of posted and not yet completed tasks. At the limit the request is logged inline, which restores back-pressure instead of queueing transactions with their buffered bodies up to the pool's max_queue (65536 by default). The fallback warning is written at most once every 10 seconds per worker; nginx itself already logs every ngx_thread_task_post() queue overflow at the error level. - The LOG phase handler returns early for subrequests: they share the main request's pool, so the cleanup scan returned the main ctx and with log_subrequest on ran phase 5 and the audit write a first time, synchronously and before the response was complete. - "modsecurity_log_thread_pool off" is accepted on builds without threads or PCRE2; only a pool name is refused there. - Test: try_run() instead of the --without-pcre2 guard, so any build that refuses the directive skips; new cases for an SSI subrequest with log_subrequest on (one audit record), an internal redirect into a location with the directive (offloaded), and max_pending reached (inline fallback and warning), the latter using an @inspectFile script that keeps a single-thread pool busy for a second. --- README.md | 43 ++++++--- src/ngx_http_modsecurity_common.h | 25 ++++- src/ngx_http_modsecurity_log.c | 80 ++++++++-------- src/ngx_http_modsecurity_module.c | 143 ++++++++++++++++++---------- tests/modsecurity-log-thread-pool.t | 68 +++++++++++-- 5 files changed, 247 insertions(+), 112 deletions(-) diff --git a/README.md b/README.md index e3a0a1e5..52a22484 100644 --- a/README.md +++ b/README.md @@ -187,7 +187,7 @@ Turns on or off ModSecurity error log functionality. modsecurity_log_thread_pool --------------------------- -**syntax:** *modsecurity_log_thread_pool <name> | off* +**syntax:** *modsecurity_log_thread_pool <name> [max_pending=<number>] | off* **context:** *http, server, location* @@ -197,13 +197,26 @@ Runs the LOGGING phase (phase 5) rules and the audit log write in the named nginx `thread_pool` instead of on the worker's event loop, so that audit log I/O no longer stalls the other connections handled by the worker. The transaction is detached from the request when the task is posted and is -cleaned up once the task completes. +cleaned up once the task completes. Whether a request is offloaded is decided +by the configuration of the location the request is finally logged in. + +Each posted task keeps the complete transaction in memory until it has run, +including the buffered request and response bodies, up to +`SecRequestBodyLimit` and `SecResponseBodyLimit`. The optional `max_pending` +parameter limits the number of logging tasks a worker has posted, to any pool, +and not yet completed (default 128); a request that finds the limit reached is +logged on the event loop, as without this directive, and a warning is written +at most once every 10 seconds. Size it so that `max_pending` times the per-request +memory fits the worker's budget, and keep the thread pool's `max_queue` at or +above it, since nginx logs a `queue overflow` error for every task the pool +refuses. This directive requires nginx built with `--with-threads` and with PCRE2; -builds using the original PCRE library refuse it at configuration time, -because there the connector has to swap the process-global `pcre_malloc` and -`pcre_free` pointers around every libmodsecurity call, which is not -thread-safe. +other builds refuse it at configuration time, except for +`modsecurity_log_thread_pool off`, which is accepted everywhere. Builds using +the original PCRE library are excluded because there the connector has to swap +the process-global `pcre_malloc` and `pcre_free` pointers around every +libmodsecurity call, which is not thread-safe. Rule messages emitted during phase 5 are written using the thread pool's log, which is the main context `error_log` at its configured level, rather than the @@ -216,7 +229,9 @@ be used with this directive, because those writers are not thread-safe. Any thread. The logging performed early by a disruptive intervention stays synchronous, as -does everything up to phase 4 and the logging phase of subrequests. +does everything up to phase 4. The logging phase runs once per transaction, for +the main request; with `log_subrequest on`, the log phase of a subrequest does +not process it. The feature is experimental, and it needs a libmodsecurity that supports processing transactions concurrently -- the way the threaded Apache MPMs use @@ -231,19 +246,19 @@ it. Two caveats are worth knowing about: Logging tasks that are still queued when a worker shuts down are still run -- the pool's exit task queues behind them -- so their audit log entries are -written, but their completion handler does not run, and the transaction, the -log holder and the task pool are released only by the exiting process. Because -the worker waits for the whole queue to drain, a thread blocked on a contended -audit log can delay shutdown beyond `worker_shutdown_timeout`, which does not -apply to thread pools. +written, but their completion handler does not run, and the transaction and +the task are released only by the exiting process. Because the worker waits +for the whole queue to drain, a thread blocked on a contended audit log can +delay shutdown beyond `worker_shutdown_timeout`, which does not apply to +thread pools. ```nginx -thread_pool modsec threads=4; +thread_pool modsec threads=4 max_queue=128; http { modsecurity on; modsecurity_rules_file /etc/modsecurity/main.conf; - modsecurity_log_thread_pool modsec; + modsecurity_log_thread_pool modsec max_pending=128; } ``` diff --git a/src/ngx_http_modsecurity_common.h b/src/ngx_http_modsecurity_common.h index ab908040..6c88d26e 100644 --- a/src/ngx_http_modsecurity_common.h +++ b/src/ngx_http_modsecurity_common.h @@ -92,11 +92,29 @@ typedef struct { } ngx_http_modsecurity_msclog_t; +#if (NGX_THREADS) && (NGX_PCRE2) + +/* + * Heap-allocated so that it can outlive the request: owned by the request's + * context until the logging task is posted, by the task afterwards. + */ +typedef struct { + ngx_thread_task_t task; + Transaction *transaction; + ngx_http_modsecurity_msclog_t msclog; +} ngx_http_modsecurity_log_task_t; + +#endif + + typedef struct { ngx_http_request_t *r; Transaction *modsec_transaction; ModSecurityIntervention *delayed_intervention; - ngx_http_modsecurity_msclog_t *msclog; + ngx_http_modsecurity_msclog_t msclog; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_log_task_t *log_task; +#endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) /* @@ -116,7 +134,6 @@ typedef struct { unsigned logged:1; unsigned intervention_triggered:1; unsigned request_body_processed:1; - unsigned msclog_heap:1; } ngx_http_modsecurity_ctx_t; @@ -126,6 +143,9 @@ typedef struct { ngx_uint_t rules_inline; ngx_uint_t rules_file; ngx_uint_t rules_remote; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_flag_t log_thread_pool_used; +#endif } ngx_http_modsecurity_main_conf_t; @@ -143,6 +163,7 @@ typedef struct { ngx_http_complex_value_t *transaction_id; #if (NGX_THREADS) && (NGX_PCRE2) ngx_thread_pool_t *log_thread_pool; + ngx_uint_t log_max_pending; #endif } ngx_http_modsecurity_conf_t; diff --git a/src/ngx_http_modsecurity_log.c b/src/ngx_http_modsecurity_log.c index 69621792..d8f97e29 100644 --- a/src/ngx_http_modsecurity_log.c +++ b/src/ngx_http_modsecurity_log.c @@ -53,11 +53,9 @@ ngx_http_modsecurity_log_transaction(ngx_http_request_t *r, #if (NGX_THREADS) && (NGX_PCRE2) -typedef struct { - ngx_pool_t *pool; - Transaction *transaction; - ngx_http_modsecurity_msclog_t *msclog; -} ngx_http_modsecurity_log_task_t; +/* logging tasks of this worker posted and not yet completed */ +static ngx_uint_t ngx_http_modsecurity_log_pending; +static time_t ngx_http_modsecurity_log_inline_warned; static void @@ -66,7 +64,7 @@ ngx_http_modsecurity_log_thread_handler(void *data, ngx_log_t *log) ngx_http_modsecurity_log_task_t *lt = data; /* the request and its connection log may be gone by now */ - lt->msclog->log = log; + lt->msclog.log = log; msc_process_logging(lt->transaction); } @@ -78,50 +76,43 @@ ngx_http_modsecurity_log_thread_event_handler(ngx_event_t *ev) ngx_http_modsecurity_log_task_t *lt = ev->data; msc_transaction_cleanup(lt->transaction); - ngx_free(lt->msclog); - ngx_destroy_pool(lt->pool); + ngx_free(lt); + + ngx_http_modsecurity_log_pending--; } static ngx_int_t ngx_http_modsecurity_log_offload(ngx_http_request_t *r, - ngx_http_modsecurity_ctx_t *ctx, ngx_thread_pool_t *tp) + ngx_http_modsecurity_ctx_t *ctx, ngx_http_modsecurity_conf_t *mcf) { - ngx_pool_t *pool; - ngx_thread_task_t *task; ngx_http_modsecurity_log_task_t *lt; - pool = ngx_create_pool(512, ngx_cycle->log); - if (pool == NULL) { - return NGX_ERROR; + if (ngx_http_modsecurity_log_pending >= mcf->log_max_pending) { + return NGX_DECLINED; } - task = ngx_thread_task_alloc(pool, sizeof(ngx_http_modsecurity_log_task_t)); - if (task == NULL) { - ngx_destroy_pool(pool); - return NGX_ERROR; - } + lt = ctx->log_task; - lt = task->ctx; - lt->pool = pool; lt->transaction = ctx->modsec_transaction; - lt->msclog = ctx->msclog; - task->handler = ngx_http_modsecurity_log_thread_handler; - task->event.data = lt; - task->event.handler = ngx_http_modsecurity_log_thread_event_handler; + lt->task.ctx = lt; + lt->task.handler = ngx_http_modsecurity_log_thread_handler; + lt->task.event.data = lt; + lt->task.event.handler = ngx_http_modsecurity_log_thread_event_handler; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "ModSecurity: posting logging to the thread pool"); - if (ngx_thread_task_post(tp, task) != NGX_OK) { - ngx_destroy_pool(pool); + if (ngx_thread_task_post(mcf->log_thread_pool, <->task) != NGX_OK) { return NGX_ERROR; } - /* the task now owns the transaction and the log holder */ + /* the task now owns the transaction and itself */ ctx->modsec_transaction = NULL; - ctx->msclog = NULL; + ctx->log_task = NULL; + + ngx_http_modsecurity_log_pending++; return NGX_OK; } @@ -137,30 +128,35 @@ ngx_http_modsecurity_log_phase_handler(ngx_http_request_t *r) ngx_http_modsecurity_conf_t *mcf; #endif + /* + * subrequests share the main request's pool, so the cleanup scan in + * ngx_http_modsecurity_get_module_ctx() would return its context + */ + if (r != r->main) { + return NGX_OK; + } + ctx = ngx_http_modsecurity_get_module_ctx(r); if (ctx == NULL || ctx->logged || ctx->modsec_transaction == NULL) { return NGX_OK; } #if (NGX_THREADS) && (NGX_PCRE2) - /* - * Subrequests do not own a context of their own -- the access phase is - * skipped for them -- so their log phase, which runs while the main - * request is still being processed, must not detach the main request's - * transaction. - */ mcf = ngx_http_get_module_loc_conf(r, ngx_http_modsecurity_module); - if (r == r->main && mcf->log_thread_pool != NULL && ctx->msclog_heap) { - if (ngx_http_modsecurity_log_offload(r, ctx, mcf->log_thread_pool) - == NGX_OK) - { + if (mcf->log_thread_pool != NULL && ctx->log_task != NULL) { + if (ngx_http_modsecurity_log_offload(r, ctx, mcf) == NGX_OK) { return NGX_OK; } - ngx_log_error(NGX_LOG_WARN, r->connection->log, 0, - "ModSecurity: could not post logging to thread pool, " - "processing inline"); + if (ngx_time() - ngx_http_modsecurity_log_inline_warned >= 10) { + ngx_http_modsecurity_log_inline_warned = ngx_time(); + + ngx_log_error(NGX_LOG_WARN, r->connection->log, 0, + "ModSecurity: could not post logging to thread " + "pool, %ui tasks pending, processing inline", + ngx_http_modsecurity_log_pending); + } } #endif diff --git a/src/ngx_http_modsecurity_module.c b/src/ngx_http_modsecurity_module.c index 58cbc4c0..c33100f1 100644 --- a/src/ngx_http_modsecurity_module.c +++ b/src/ngx_http_modsecurity_module.c @@ -29,6 +29,10 @@ #define strdup _strdup #endif +#if (NGX_THREADS) && (NGX_PCRE2) +#define NGX_HTTP_MODSECURITY_LOG_MAX_PENDING 128 +#endif + static ngx_int_t ngx_http_modsecurity_init(ngx_conf_t *cf); static void *ngx_http_modsecurity_create_main_conf(ngx_conf_t *cf); static char *ngx_http_modsecurity_init_main_conf(ngx_conf_t *cf, void *conf); @@ -266,10 +270,12 @@ ngx_http_modsecurity_cleanup(void *data) ctx->modsec_transaction = NULL; } - if (ctx->msclog_heap && ctx->msclog != NULL) { - ngx_free(ctx->msclog); - ctx->msclog = NULL; +#if (NGX_THREADS) && (NGX_PCRE2) + if (ctx->log_task != NULL) { + ngx_free(ctx->log_task); + ctx->log_task = NULL; } +#endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) /* @@ -288,7 +294,11 @@ ngx_http_modsecurity_create_ctx(ngx_http_request_t *r) ngx_pool_cleanup_t *cln; ngx_http_modsecurity_ctx_t *ctx; ngx_http_modsecurity_conf_t *mcf; + ngx_http_modsecurity_msclog_t *msclog; ngx_http_modsecurity_main_conf_t *mmcf; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_log_task_t *lt; +#endif ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_modsecurity_ctx_t)); if (ctx == NULL) @@ -300,12 +310,16 @@ ngx_http_modsecurity_create_ctx(ngx_http_request_t *r) mmcf = ngx_http_get_module_main_conf(r, ngx_http_modsecurity_module); mcf = ngx_http_get_module_loc_conf(r, ngx_http_modsecurity_module); - dd("creating transaction with the following rules: '%p' -- ms: '%p'", mcf->rules_set, mmcf->modsec); +#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) + ctx->sanity_headers_out = ngx_array_create(r->pool, 12, sizeof(ngx_http_modsecurity_header_t)); + if (ctx->sanity_headers_out == NULL) { + return NGX_CONF_ERROR; + } +#endif /* - * Take the cleanup slot before anything that has to be released: from - * here on every error return leaves the transaction and the log holder - * to ngx_http_modsecurity_cleanup(), which copes with either being NULL. + * The cleanup is armed, and the ctx published, only once the + * transaction exists. */ cln = ngx_pool_cleanup_add(r->pool, sizeof(ngx_http_modsecurity_ctx_t)); if (cln == NULL) @@ -313,59 +327,45 @@ ngx_http_modsecurity_create_ctx(ngx_http_request_t *r) dd("failed to create the ModSecurity context cleanup"); return NGX_CONF_ERROR; } - cln->handler = ngx_http_modsecurity_cleanup; - cln->data = ctx; + + if (mcf->transaction_id) { + if (ngx_http_complex_value(r, mcf->transaction_id, &s) != NGX_OK) { + return NGX_CONF_ERROR; + } + } + + msclog = &ctx->msclog; #if (NGX_THREADS) && (NGX_PCRE2) - if (mcf->log_thread_pool != NULL) { - /* - * the holder must outlive the request pool: the logging phase is - * processed in a thread after the request is gone - */ - ctx->msclog = ngx_alloc(sizeof(ngx_http_modsecurity_msclog_t), - r->connection->log); - ctx->msclog_heap = 1; + if (mmcf->log_thread_pool_used) { + lt = ngx_calloc(sizeof(ngx_http_modsecurity_log_task_t), + r->connection->log); + if (lt == NULL) { + return NULL; + } - } else -#endif - { - ctx->msclog = ngx_palloc(r->pool, - sizeof(ngx_http_modsecurity_msclog_t)); + ctx->log_task = lt; + msclog = <->msclog; } +#endif - if (ctx->msclog == NULL) { - dd("failed to allocate memory for the log holder."); - /* - * nothing has to be released yet; disarm the cleanup so that the - * half-built context stays invisible to - * ngx_http_modsecurity_get_module_ctx() - */ - cln->handler = NULL; - return NULL; - } + msclog->log = r->connection->log; - ctx->msclog->log = r->connection->log; + dd("creating transaction with the following rules: '%p' -- ms: '%p'", mcf->rules_set, mmcf->modsec); if (mcf->transaction_id) { - if (ngx_http_complex_value(r, mcf->transaction_id, &s) != NGX_OK) { - return NGX_CONF_ERROR; - } - ctx->modsec_transaction = msc_new_transaction_with_id(mmcf->modsec, mcf->rules_set, (char *) s.data, ctx->msclog); + ctx->modsec_transaction = msc_new_transaction_with_id(mmcf->modsec, mcf->rules_set, (char *) s.data, msclog); } else { - ctx->modsec_transaction = msc_new_transaction(mmcf->modsec, mcf->rules_set, ctx->msclog); + ctx->modsec_transaction = msc_new_transaction(mmcf->modsec, mcf->rules_set, msclog); } dd("transaction created"); - ngx_http_set_ctx(r, ctx, ngx_http_modsecurity_module); + cln->handler = ngx_http_modsecurity_cleanup; + cln->data = ctx; -#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) - ctx->sanity_headers_out = ngx_array_create(r->pool, 12, sizeof(ngx_http_modsecurity_header_t)); - if (ctx->sanity_headers_out == NULL) { - return NGX_CONF_ERROR; - } -#endif + ngx_http_set_ctx(r, ctx, ngx_http_modsecurity_module); return ctx; } @@ -532,27 +532,72 @@ char *ngx_conf_set_transaction_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf static char * ngx_conf_set_log_thread_pool(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { + ngx_str_t *value; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_int_t n; + ngx_http_modsecurity_conf_t *mcf = conf; + ngx_http_modsecurity_main_conf_t *mmcf; +#endif + + value = cf->args->elts; + #if (NGX_THREADS) && (NGX_PCRE2) - ngx_http_modsecurity_conf_t *mcf = conf; - ngx_str_t *value = cf->args->elts; if (mcf->log_thread_pool != NGX_CONF_UNSET_PTR) { return "is duplicate"; } if (ngx_strcmp(value[1].data, "off") == 0) { + if (cf->args->nelts > 2) { + goto invalid; + } + mcf->log_thread_pool = NULL; + mcf->log_max_pending = 0; return NGX_CONF_OK; } + mcf->log_max_pending = NGX_HTTP_MODSECURITY_LOG_MAX_PENDING; + + if (cf->args->nelts > 2) { + if (ngx_strncmp(value[2].data, "max_pending=", 12) != 0) { + goto invalid; + } + + n = ngx_atoi(value[2].data + 12, value[2].len - 12); + if (n == NGX_ERROR || n == 0) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid max_pending value \"%V\"", &value[2]); + return NGX_CONF_ERROR; + } + + mcf->log_max_pending = n; + } + mcf->log_thread_pool = ngx_thread_pool_add(cf, &value[1]); if (mcf->log_thread_pool == NULL) { return NGX_CONF_ERROR; } + mmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_modsecurity_module); + mmcf->log_thread_pool_used = 1; + return NGX_CONF_OK; + +invalid: + + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid parameter \"%V\"", &value[2]); + return NGX_CONF_ERROR; + #else + + if (cf->args->nelts == 2 && ngx_strcmp(value[1].data, "off") == 0) { + return NGX_CONF_OK; + } + return "requires nginx built with --with-threads and PCRE2"; + #endif } @@ -600,7 +645,7 @@ static ngx_command_t ngx_http_modsecurity_commands[] = { }, { ngx_string("modsecurity_log_thread_pool"), - NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, + NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE12, ngx_conf_set_log_thread_pool, NGX_HTTP_LOC_CONF_OFFSET, 0, @@ -811,6 +856,7 @@ ngx_http_modsecurity_create_conf(ngx_conf_t *cf) conf->use_error_log = NGX_CONF_UNSET; #if (NGX_THREADS) && (NGX_PCRE2) conf->log_thread_pool = NGX_CONF_UNSET_PTR; + conf->log_max_pending = NGX_CONF_UNSET_UINT; #endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) conf->sanity_checks_enabled = NGX_CONF_UNSET; @@ -854,6 +900,7 @@ ngx_http_modsecurity_merge_conf(ngx_conf_t *cf, void *parent, void *child) ngx_conf_merge_value(c->use_error_log, p->use_error_log, 1); #if (NGX_THREADS) && (NGX_PCRE2) ngx_conf_merge_ptr_value(c->log_thread_pool, p->log_thread_pool, NULL); + ngx_conf_merge_uint_value(c->log_max_pending, p->log_max_pending, 0); #endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) ngx_conf_merge_value(c->sanity_checks_enabled, p->sanity_checks_enabled, 0); diff --git a/tests/modsecurity-log-thread-pool.t b/tests/modsecurity-log-thread-pool.t index f810d680..00e588da 100644 --- a/tests/modsecurity-log-thread-pool.t +++ b/tests/modsecurity-log-thread-pool.t @@ -34,10 +34,7 @@ use Test::Nginx; select STDERR; $| = 1; select STDOUT; $| = 1; -my $t = Test::Nginx->new()->has(qw/http threads/); - -plan(skip_all => 'nginx built with --without-pcre2') - unless $t->has_module('(?s)^(?!.*--without-pcre2)'); +my $t = Test::Nginx->new()->has(qw/http threads ssi/); $t->write_file_expand('nginx.conf', <<'EOF'); @@ -46,6 +43,7 @@ $t->write_file_expand('nginx.conf', <<'EOF'); daemon off; thread_pool modsec threads=2; +thread_pool slow threads=1; events { } @@ -63,8 +61,11 @@ http { SecRuleEngine On SecRule ARGS:what "@streq block" "id:71,phase:1,deny,status:403,log,auditlog" SecRule ARGS:what "@streq p5" "id:72,phase:5,log,auditlog,pass" + SecRule ARGS:what "@streq p5redir" "id:73,phase:5,log,pass" '; + log_subrequest on; + # libmodsecurity merges a non-empty parent SecAuditLog over the # child's one, so the audit log is configured per location here. @@ -88,6 +89,29 @@ http { SecAuditLogStorageDir %%TESTDIR%%/ '; } + + location /ssi/ { + ssi on; + modsecurity_rules ' + SecAuditEngine RelevantOnly + SecAuditLogParts ABH + SecAuditLog %%TESTDIR%%/audit-ssi.txt + SecAuditLogType Serial + SecAuditLogStorageDir %%TESTDIR%%/ + '; + } + + location /busy/ { + modsecurity_log_thread_pool slow max_pending=1; + modsecurity_rules ' + SecRule REQUEST_URI "@inspectFile %%TESTDIR%%/sleep.sh" "id:74,phase:5,pass,nolog" + '; + } + + location /redir/ { + modsecurity_log_thread_pool off; + try_files /nonexistent /index.html; + } } } @@ -96,8 +120,13 @@ EOF $t->write_file('index.html', 'INDEX'); mkdir($t->testdir() . '/sync'); $t->write_file('/sync/index.html', 'SYNC INDEX'); -$t->run(); -$t->plan(11); +mkdir($t->testdir() . '/ssi'); +$t->write_file('/ssi/index.html', 'SSI '); +mkdir($t->testdir() . '/busy'); +$t->write_file('/busy/index.html', 'BUSY INDEX'); +$t->write_file('sleep.sh', "#!/bin/sh\nsleep 1\n"); +chmod(0755, $t->testdir() . '/sleep.sh'); +$t->try_run('no modsecurity_log_thread_pool')->plan(18); ############################################################################### @@ -105,6 +134,8 @@ like(http_get('/index.html?what=block'), qr/^HTTP.*403/, 'phase 1 deny still blo like(http_get('/index.html?what=p5'), qr/INDEX/, 'request with phase 5 rule passes'); like(http_get('/index.html?what=none'), qr/INDEX/, 'plain request passes'); like(http_get('/sync/index.html?what=p5'), qr/SYNC INDEX/, 'thread pool off in location'); +like(http_get('/ssi/index.html?what=p5'), qr/SSI INDEX/, 'request with subrequest passes'); +like(http_get('/redir/x?what=p5redir'), qr/INDEX/, 'internally redirected request passes'); my $audit = wait_for_file($t->testdir() . '/audit.txt', qr/\[id "72"\]/); like($audit, qr/\[id "72"\]/, 'phase 5 rule logged from thread'); @@ -119,12 +150,37 @@ like($sync, qr/\[id "72"\]/, 'synchronous location still audits'); # written on the request's connection log does. This tells the offloaded path # apart from the (deliberately silent) synchronous fallback. +# With log_subrequest on, the log phase of the SSI subrequest must not run +# phase 5 of the main request's transaction. Each serial audit log record +# starts with a part-A section header line, so counting those counts the +# records; the second run would come from the thread, hence the extra wait. + +my $ssi = wait_for_file($t->testdir() . '/audit-ssi.txt', qr/\[id "72"\]/); +select undef, undef, undef, 0.5; +$ssi = read_file($t->testdir() . '/audit-ssi.txt'); +my $records = () = $ssi =~ /^---\w+---A--$/mg; +is($records, 1, 'phase 5 ran once for a request with a subrequest'); + +wait_for_file($t->testdir() . '/error.log', qr/\[id "73"\]/); + my $errlog = read_file($t->testdir() . '/error.log'); unlike($errlog, qr/could not post logging/, 'logging was not refused by the pool'); like($errlog, qr/^(?:(?!client:).)*\[id "72"\](?:(?!client:).)*$/m, 'phase 5 rule message written from the thread pool log'); like($errlog, qr/^.*\[id "72"\].*client:.*$/m, 'phase 5 rule message of the synchronous location keeps the request context'); +like($errlog, qr/^(?:(?!client:).)*\[id "73"\](?:(?!client:).)*$/m, + 'offload decided by the location the request ended in'); + +# The phase 5 rule of /busy/ keeps the only thread of its pool busy for a +# second, so the second request finds max_pending reached and is logged +# inline. + +like(http_get('/busy/index.html'), qr/BUSY INDEX/, 'first request to busy pool'); +like(http_get('/busy/index.html'), qr/BUSY INDEX/, 'second request to busy pool'); +like(read_file($t->testdir() . '/error.log'), + qr/could not post logging to thread pool, 1 tasks pending/, + 'max_pending reached falls back to inline logging'); ############################################################################### From 096c4a60a2cf6e3c3d92b67c06d5e6bc834d1062 Mon Sep 17 00:00:00 2001 From: Tom Sommer Date: Fri, 13 Nov 2026 07:44:14 +0100 Subject: [PATCH 3/6] README: recommend a dedicated pool for modsecurity_log_thread_pool max_pending only keeps the pool from refusing tasks if nothing else, such as aio threads, posts to the same pool. --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 52a22484..327da60c 100644 --- a/README.md +++ b/README.md @@ -207,9 +207,9 @@ parameter limits the number of logging tasks a worker has posted, to any pool, and not yet completed (default 128); a request that finds the limit reached is logged on the event loop, as without this directive, and a warning is written at most once every 10 seconds. Size it so that `max_pending` times the per-request -memory fits the worker's budget, and keep the thread pool's `max_queue` at or -above it, since nginx logs a `queue overflow` error for every task the pool -refuses. +memory fits the worker's budget, and use a pool dedicated to this directive +with `max_queue` at or above it, since nginx logs a `queue overflow` error for +every task the pool refuses. This directive requires nginx built with `--with-threads` and with PCRE2; other builds refuse it at configuration time, except for From 3634b2f52396079e6fc791746691be415ca74beb Mon Sep 17 00:00:00 2001 From: Tom Sommer Date: Fri, 13 Nov 2026 08:30:33 +0100 Subject: [PATCH 4/6] Initialise the transaction id string in create_ctx The id is now computed before the log holder is allocated and used further down under a second mcf->transaction_id test. MSVC cannot see that both tests agree and fails the -W4 -WX Windows build with C4701 (potentially uninitialized local variable 's'). --- src/ngx_http_modsecurity_module.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ngx_http_modsecurity_module.c b/src/ngx_http_modsecurity_module.c index c33100f1..9151ff48 100644 --- a/src/ngx_http_modsecurity_module.c +++ b/src/ngx_http_modsecurity_module.c @@ -290,7 +290,7 @@ ngx_http_modsecurity_cleanup(void *data) ngx_http_modsecurity_ctx_t * ngx_http_modsecurity_create_ctx(ngx_http_request_t *r) { - ngx_str_t s; + ngx_str_t s = ngx_null_string; ngx_pool_cleanup_t *cln; ngx_http_modsecurity_ctx_t *ctx; ngx_http_modsecurity_conf_t *mcf; From 8565f83b1a7b29e4c8c05b7e500a0fbe25e723f0 Mon Sep 17 00:00:00 2001 From: Tom Sommer Date: Fri, 13 Nov 2026 10:46:27 +0100 Subject: [PATCH 5/6] Log thread pool: serialise logging, destroy in thread, drain at exit - Serial audit writes: libmodsecurity's SharedFiles::write() only takes an fcntl() F_WRLCK, which belongs to the process. Two threads of one worker both get it, and the first F_UNLCK releases it while the other thread is still in fwrite(), so another worker can write at the same time and large records interleave. A per-worker ngx_thread_mutex_t, created in init_process when any location names a pool, is now held around every msc_process_logging() call: in the thread handler and in ngx_http_modsecurity_log_transaction(), which covers the inline fallback, locations with the directive off and the early logging of ngx_http_modsecurity_process_intervention(). The event loop may wait for a thread's write, never longer than the write it would otherwise have done itself; more than one thread gives no logging parallelism. - The thread now destroys the transaction right after logging it, so its temporary files (saved uploads, unlinked by the Transaction destructor) are removed even when the worker exits before the completion handler runs. The destructor only frees memory, unlinks those files and writes debug log lines. The completion handler only frees the task and decrements the pending count. - Graceful shutdown: ngx_worker_process_cycle() exits once only cancelable timers are left, so a non-cancelable timer is kept armed while tasks are pending and deleted when the last completion handler runs. The worker therefore stays up until all tasks have completed, and the pool's exit task always finds an empty queue. Fast shutdown still exits at once; ngx_thread_pool_destroy() returns without joining when its exit task does not fit into the queue, so the README now asks for max_queue greater than max_pending on a dedicated pool, and its shutdown section states what is actually guaranteed. - New test modsecurity-log-thread-pool-exit.t: graceful stop with one task running and one queued on a threads=1 max_queue=1 pool (both audit records written; 0 without the drain timer), and a worker terminated while a task with a saved upload runs (temporary file removed; still there without the in-thread cleanup). --- README.md | 45 ++++-- src/ngx_http_modsecurity_common.h | 3 + src/ngx_http_modsecurity_log.c | 109 +++++++++++++- src/ngx_http_modsecurity_module.c | 4 + tests/modsecurity-log-thread-pool-exit.t | 174 +++++++++++++++++++++++ 5 files changed, 314 insertions(+), 21 deletions(-) create mode 100644 tests/modsecurity-log-thread-pool-exit.t diff --git a/README.md b/README.md index 327da60c..feeafcab 100644 --- a/README.md +++ b/README.md @@ -208,8 +208,20 @@ and not yet completed (default 128); a request that finds the limit reached is logged on the event loop, as without this directive, and a warning is written at most once every 10 seconds. Size it so that `max_pending` times the per-request memory fits the worker's budget, and use a pool dedicated to this directive -with `max_queue` at or above it, since nginx logs a `queue overflow` error for -every task the pool refuses. +with `max_queue` greater than `max_pending`. A pool that refuses a task makes +nginx log a `queue overflow` error, and at worker exit a full queue leaves no +room for the pool's own exit task (see below). + +Within a worker, phase 5 and the audit log write are serialised: the thread, +the inline fallback and the early logging of a disruptive intervention all +hold one per-worker mutex around them. The serial audit log writer of +libmodsecurity only takes an `fcntl()` lock, which belongs to the process, so +two threads of one worker do not exclude each other, and one thread releasing +the lock lets another process write while the other thread is still writing. +As a consequence a thread pool with more than one thread gives no additional +logging parallelism, and the event loop can wait for the mutex while a thread +writes; that wait is never longer than the write it would have done itself +without this directive. `threads=1` is enough. This directive requires nginx built with `--with-threads` and with PCRE2; other builds refuse it at configuration time, except for @@ -235,25 +247,28 @@ not process it. The feature is experimental, and it needs a libmodsecurity that supports processing transactions concurrently -- the way the threaded Apache MPMs use -it. Two caveats are worth knowing about: +it: phases 1 to 4 of one transaction still run on the event loop while phase 5 +of another runs in the thread. Two caveats are worth knowing about: -* concurrent writes of serial audit log entries rely on the atomicity of a - single `fwrite()`; the `fcntl()` lock taken by the library does not exclude - other threads of the same process. `SecAuditLogType Concurrent`, or a thread - pool with a single thread, avoids the question; +* `SecDebugLog` writes made during phases 1 to 4 are not covered by the mutex + and can interleave with debug log writes from the thread; * the LMDB collection backend (`--with-lmdb`) has not been reviewed for use from several threads of one process. -Logging tasks that are still queued when a worker shuts down are still run -- -the pool's exit task queues behind them -- so their audit log entries are -written, but their completion handler does not run, and the transaction and -the task are released only by the exiting process. Because the worker waits -for the whole queue to drain, a thread blocked on a contended audit log can -delay shutdown beyond `worker_shutdown_timeout`, which does not apply to -thread pools. +The thread destroys the transaction once it has been logged, so temporary +files of the transaction, such as saved uploads, are removed even if the +worker exits before the task's completion handler runs. + +On graceful shutdown the worker does not exit while logging tasks are +pending: it keeps a timer armed until their completion handlers have run. On +fast shutdown the worker exits at once, and the thread pool runs the tasks +still queued before its threads exit, provided the pool's exit task fits into +the queue -- hence `max_queue` greater than `max_pending` on a dedicated pool. +In both cases a thread blocked on a contended audit log delays the worker's +exit, and `worker_shutdown_timeout` does not shorten that wait. ```nginx -thread_pool modsec threads=4 max_queue=128; +thread_pool modsec threads=1 max_queue=256; http { modsecurity on; diff --git a/src/ngx_http_modsecurity_common.h b/src/ngx_http_modsecurity_common.h index 6c88d26e..3175aa76 100644 --- a/src/ngx_http_modsecurity_common.h +++ b/src/ngx_http_modsecurity_common.h @@ -207,6 +207,9 @@ int ngx_http_modsecurity_store_ctx_header(ngx_http_request_t *r, ngx_str_t *name void ngx_http_modsecurity_log(void *data, const void* msg); ngx_int_t ngx_http_modsecurity_log_handler(ngx_http_request_t *r); ngx_int_t ngx_http_modsecurity_log_phase_handler(ngx_http_request_t *r); +#if (NGX_THREADS) && (NGX_PCRE2) +ngx_int_t ngx_http_modsecurity_log_init_process(ngx_cycle_t *cycle); +#endif /* ngx_http_modsecurity_access.c */ ngx_int_t ngx_http_modsecurity_access_handler(ngx_http_request_t *r); diff --git a/src/ngx_http_modsecurity_log.c b/src/ngx_http_modsecurity_log.c index d8f97e29..d3e1b1b7 100644 --- a/src/ngx_http_modsecurity_log.c +++ b/src/ngx_http_modsecurity_log.c @@ -36,37 +36,110 @@ ngx_http_modsecurity_log(void *data, const void* msg) } +#if (NGX_THREADS) && (NGX_PCRE2) + +/* logging tasks of this worker posted and not yet completed */ +static ngx_uint_t ngx_http_modsecurity_log_pending; +static time_t ngx_http_modsecurity_log_inline_warned; + +/* keeps a gracefully exiting worker alive while tasks are pending */ +static ngx_event_t ngx_http_modsecurity_log_drain; +static ngx_connection_t ngx_http_modsecurity_log_drain_dumb; + +/* serialises msc_process_logging() within the worker */ +static ngx_thread_mutex_t ngx_http_modsecurity_log_mutex; +static ngx_flag_t ngx_http_modsecurity_log_mutex_created; + + +ngx_int_t +ngx_http_modsecurity_log_init_process(ngx_cycle_t *cycle) +{ + ngx_http_modsecurity_main_conf_t *mmcf; + + mmcf = ngx_http_cycle_get_module_main_conf(cycle, + ngx_http_modsecurity_module); + + if (mmcf == NULL || !mmcf->log_thread_pool_used) { + return NGX_OK; + } + + if (ngx_thread_mutex_create(&ngx_http_modsecurity_log_mutex, cycle->log) + != NGX_OK) + { + return NGX_ERROR; + } + + ngx_http_modsecurity_log_mutex_created = 1; + + return NGX_OK; +} + + +static ngx_flag_t +ngx_http_modsecurity_log_lock(ngx_log_t *log) +{ + if (!ngx_http_modsecurity_log_mutex_created) { + return 0; + } + + return ngx_thread_mutex_lock(&ngx_http_modsecurity_log_mutex, log) + == NGX_OK; +} + + +static void +ngx_http_modsecurity_log_unlock(ngx_flag_t locked, ngx_log_t *log) +{ + if (locked) { + (void) ngx_thread_mutex_unlock(&ngx_http_modsecurity_log_mutex, log); + } +} + +#endif + + static ngx_int_t ngx_http_modsecurity_log_transaction(ngx_http_request_t *r, ngx_http_modsecurity_ctx_t *ctx) { ngx_pool_t *old_pool; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_flag_t locked; + + locked = ngx_http_modsecurity_log_lock(r->connection->log); +#endif dd("calling msc_process_logging for %p", ctx); old_pool = ngx_http_modsecurity_pcre_malloc_init(r->pool); msc_process_logging(ctx->modsec_transaction); ngx_http_modsecurity_pcre_malloc_done(old_pool); +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_log_unlock(locked, r->connection->log); +#endif + return NGX_OK; } #if (NGX_THREADS) && (NGX_PCRE2) -/* logging tasks of this worker posted and not yet completed */ -static ngx_uint_t ngx_http_modsecurity_log_pending; -static time_t ngx_http_modsecurity_log_inline_warned; - - static void ngx_http_modsecurity_log_thread_handler(void *data, ngx_log_t *log) { + ngx_flag_t locked; ngx_http_modsecurity_log_task_t *lt = data; /* the request and its connection log may be gone by now */ lt->msclog.log = log; + locked = ngx_http_modsecurity_log_lock(log); msc_process_logging(lt->transaction); + ngx_http_modsecurity_log_unlock(locked, log); + + /* the completion handler does not run if the worker exits first */ + msc_transaction_cleanup(lt->transaction); + lt->transaction = NULL; } @@ -75,10 +148,24 @@ ngx_http_modsecurity_log_thread_event_handler(ngx_event_t *ev) { ngx_http_modsecurity_log_task_t *lt = ev->data; - msc_transaction_cleanup(lt->transaction); ngx_free(lt); ngx_http_modsecurity_log_pending--; + + if (ngx_http_modsecurity_log_pending == 0 + && ngx_http_modsecurity_log_drain.timer_set) + { + ngx_del_timer(&ngx_http_modsecurity_log_drain); + } +} + + +static void +ngx_http_modsecurity_log_drain_handler(ngx_event_t *ev) +{ + if (ngx_http_modsecurity_log_pending) { + ngx_add_timer(ev, 1000); + } } @@ -114,6 +201,16 @@ ngx_http_modsecurity_log_offload(ngx_http_request_t *r, ngx_http_modsecurity_log_pending++; + if (!ngx_http_modsecurity_log_drain.timer_set) { + ngx_http_modsecurity_log_drain.handler = + ngx_http_modsecurity_log_drain_handler; + ngx_http_modsecurity_log_drain.log = ngx_cycle->log; + ngx_http_modsecurity_log_drain.data = + &ngx_http_modsecurity_log_drain_dumb; + ngx_http_modsecurity_log_drain_dumb.fd = (ngx_socket_t) -1; + ngx_add_timer(&ngx_http_modsecurity_log_drain, 1000); + } + return NGX_OK; } diff --git a/src/ngx_http_modsecurity_module.c b/src/ngx_http_modsecurity_module.c index 9151ff48..3b6ea342 100644 --- a/src/ngx_http_modsecurity_module.c +++ b/src/ngx_http_modsecurity_module.c @@ -685,7 +685,11 @@ ngx_module_t ngx_http_modsecurity_module = { NGX_HTTP_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_log_init_process, /* init process */ +#else NULL, /* init process */ +#endif NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ diff --git a/tests/modsecurity-log-thread-pool-exit.t b/tests/modsecurity-log-thread-pool-exit.t new file mode 100644 index 00000000..791dbf0b --- /dev/null +++ b/tests/modsecurity-log-thread-pool-exit.t @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +# +# ModSecurity, http://www.modsecurity.org/ +# Copyright (c) 2015 Trustwave Holdings, Inc. (http://www.trustwave.com/) +# +# You may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# If any of the files related to licensing are missing or if you have any +# other questions related to licensing please contact Trustwave Holdings, Inc. +# directly using the email address security@modsecurity.org. +# + + +# Tests for ModSecurity module, logging tasks in a thread pool at worker exit. + +############################################################################### + +use warnings; +use strict; + +use Test::More; + +BEGIN { use FindBin; chdir($FindBin::Bin); } + +use lib 'lib'; +use Test::Nginx; + +############################################################################### + +select STDERR; $| = 1; +select STDOUT; $| = 1; + +my $t = Test::Nginx->new()->has(qw/http threads/); + +$t->write_file_expand('nginx.conf', <<'EOF'); + +%%TEST_GLOBALS%% + +daemon off; + +thread_pool tiny threads=1 max_queue=1; +thread_pool slow threads=1; + +events { +} + +http { + %%TEST_GLOBALS_HTTP%% + + server { + listen 127.0.0.1:8080; + server_name localhost; + + modsecurity on; + modsecurity_rules ' + SecRuleEngine On + SecRequestBodyAccess On + SecRule REQUEST_URI "@inspectFile %%TESTDIR%%/sleep.sh" "id:81,phase:5,pass,nolog" + '; + + # max_pending=2 with max_queue=1 leaves no room for the pool's exit + # task while one task runs and one waits. + + location /drain/ { + modsecurity_log_thread_pool tiny max_pending=2; + modsecurity_rules ' + SecAuditEngine On + SecAuditLogParts ABZ + SecAuditLog %%TESTDIR%%/audit.txt + SecAuditLogType Serial + SecAuditLogStorageDir %%TESTDIR%%/ + '; + } + + location = /pid { + modsecurity off; + add_header X-Pid $pid; + return 204; + } + + location /upload/ { + modsecurity_log_thread_pool slow; + modsecurity_rules ' + SecUploadDir %%TESTDIR%%/upload + SecTmpSaveUploadedFiles On + SecUploadKeepFiles Off + '; + } + } +} + +EOF + +mkdir($t->testdir() . '/drain'); +$t->write_file('/drain/index.html', 'DRAIN INDEX'); +mkdir($t->testdir() . '/upload'); +$t->write_file('sleep.sh', "#!/bin/sh\nsleep 1\n"); +chmod(0755, $t->testdir() . '/sleep.sh'); +$t->try_run('no modsecurity_log_thread_pool')->plan(5); + +############################################################################### + +# Graceful shutdown waits for the logging tasks, even when the pool's queue +# has no room for its exit task. + +like(http_get('/drain/index.html'), qr/DRAIN INDEX/, 'first request'); +select undef, undef, undef, 0.2; +like(http_get('/drain/index.html'), qr/DRAIN INDEX/, 'second request'); + +$t->stop(); + +my $records = () = read_file($t->testdir() . '/audit.txt') =~ /^---\w+---Z--$/mg; +is($records, 2, 'audit records written before the worker exited'); + +# Fast shutdown of the worker: the pool still runs the task, which also destroys the +# transaction and so removes its temporary upload file. + +$t->run(); + +my $body = <<"EOF"; +--XyZ\r +Content-Disposition: form-data; name="f"; filename="a.txt"\r +Content-Type: text/plain\r +\r +UPLOADED\r +--XyZ--\r +EOF + +http(<<"EOF" . $body); +POST /upload/index.html HTTP/1.0\r +Host: localhost\r +Content-Type: multipart/form-data; boundary=XyZ\r +Content-Length: @{[ length $body ]}\r +\r +EOF + +select undef, undef, undef, 0.2; +ok(upload_files($t->testdir()) > 0, 'upload saved to a temporary file'); + +http_get('/pid') =~ qr/X-Pid: (\d+)/; +my $worker = $1; +kill 'TERM', $worker; + +for (1 .. 50) { + last unless kill 0, $worker; + select undef, undef, undef, 0.1; +} + +is(upload_files($t->testdir()), 0, 'temporary upload file removed at fast shutdown'); + +############################################################################### + +sub upload_files { + my ($dir) = @_; + opendir(my $dh, $dir . '/upload') or return -1; + my @files = grep { !/^\./ } readdir($dh); + closedir($dh); + return scalar @files; +} + +sub read_file { + my ($path) = @_; + open my $fh, '<', $path or return ''; + local $/; + my $data = <$fh>; + close $fh; + return defined $data ? $data : ''; +} + +############################################################################### From a1419f9d935aa877a1d683cabd11d4323c6a0e60 Mon Sep 17 00:00:00 2001 From: Tom Sommer Date: Mon, 16 Nov 2026 13:24:55 +0100 Subject: [PATCH 6/6] README: unbounded mutex wait, SIGKILL on fast shutdown The logging mutex also covers other transactions' phase 5 and audit writes, so an inline or early log can wait longer than its own write. nginx -s stop makes the master SIGKILL workers that are still running after its termination delay (50 ms doubling, past 1000 ms), so queued or running logging tasks can be lost whatever the queue size. Only graceful shutdown drains; say so, and keep the max_queue advice for a worker that receives SIGTERM directly. --- README.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index ea8f30c8..248bb360 100644 --- a/README.md +++ b/README.md @@ -245,9 +245,9 @@ libmodsecurity only takes an `fcntl()` lock, which belongs to the process, so two threads of one worker do not exclude each other, and one thread releasing the lock lets another process write while the other thread is still writing. As a consequence a thread pool with more than one thread gives no additional -logging parallelism, and the event loop can wait for the mutex while a thread -writes; that wait is never longer than the write it would have done itself -without this directive. `threads=1` is enough. +logging parallelism, and inline or early logging on the event loop can wait +for the mutex while another transaction's phase 5 and audit write run, so that +wait is not bounded by the request's own logging time. `threads=1` is enough. This directive requires nginx built with `--with-threads` and with PCRE2; other builds refuse it at configuration time, except for @@ -285,13 +285,17 @@ The thread destroys the transaction once it has been logged, so temporary files of the transaction, such as saved uploads, are removed even if the worker exits before the task's completion handler runs. -On graceful shutdown the worker does not exit while logging tasks are -pending: it keeps a timer armed until their completion handlers have run. On -fast shutdown the worker exits at once, and the thread pool runs the tasks -still queued before its threads exit, provided the pool's exit task fits into -the queue -- hence `max_queue` greater than `max_pending` on a dedicated pool. -In both cases a thread blocked on a contended audit log delays the worker's -exit, and `worker_shutdown_timeout` does not shorten that wait. +On graceful shutdown (`nginx -s quit`) the worker does not exit while logging +tasks are pending: it keeps a timer armed until their completion handlers have +run. Use graceful shutdown when queued audit records must be written. A worker +that receives `SIGTERM` exits at once, and the thread pool runs the tasks still +queued before its threads exit, provided the pool's exit task fits into the +queue -- hence `max_queue` greater than `max_pending` on a dedicated pool. On +fast shutdown (`nginx -s stop`), though, the master sends `SIGKILL` to workers +that have not exited after about 1.5 seconds, so queued and running logging tasks +can be lost and their temporary files left behind, whatever the queue size. A +thread blocked on a contended audit log delays a graceful exit, and +`worker_shutdown_timeout` does not shorten that wait. ```nginx thread_pool modsec threads=1 max_queue=256;