diff --git a/README.md b/README.md index 36b859c2..248bb360 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ Further information about building nginx third-party modules is available [here] # Usage ModSecurity for nginx extends your nginx configuration directives. -It adds six new directives and they are: +It adds the following directives: modsecurity ----------- @@ -211,6 +211,102 @@ modsecurity_use_error_log Turns on or off ModSecurity error log functionality. +modsecurity_log_thread_pool +--------------------------- +**syntax:** *modsecurity_log_thread_pool <name> [max_pending=<number>] | off* + +**context:** *http, server, location* + +**default:** *off* + +Runs the LOGGING phase (phase 5) rules and the audit log write in the named +nginx `thread_pool` instead of on the worker's event loop, so that audit log +I/O no longer stalls the other connections handled by the worker. The +transaction is detached from the request when the task is posted and is +cleaned up once the task completes. Whether a request is offloaded is decided +by the configuration of the location the request is finally logged in. + +Each posted task keeps the complete transaction in memory until it has run, +including the buffered request and response bodies, up to +`SecRequestBodyLimit` and `SecResponseBodyLimit`. The optional `max_pending` +parameter limits the number of logging tasks a worker has posted, to any pool, +and not yet completed (default 128); a request that finds the limit reached is +logged on the event loop, as without this directive, and a warning is written +at most once every 10 seconds. Size it so that `max_pending` times the per-request +memory fits the worker's budget, and use a pool dedicated to this directive +with `max_queue` greater than `max_pending`. A pool that refuses a task makes +nginx log a `queue overflow` error, and at worker exit a full queue leaves no +room for the pool's own exit task (see below). + +Within a worker, phase 5 and the audit log write are serialised: the thread, +the inline fallback and the early logging of a disruptive intervention all +hold one per-worker mutex around them. The serial audit log writer of +libmodsecurity only takes an `fcntl()` lock, which belongs to the process, so +two threads of one worker do not exclude each other, and one thread releasing +the lock lets another process write while the other thread is still writing. +As a consequence a thread pool with more than one thread gives no additional +logging parallelism, and inline or early logging on the event loop can wait +for the mutex while another transaction's phase 5 and audit write run, so that +wait is not bounded by the request's own logging time. `threads=1` is enough. + +This directive requires nginx built with `--with-threads` and with PCRE2; +other builds refuse it at configuration time, except for +`modsecurity_log_thread_pool off`, which is accepted everywhere. Builds using +the original PCRE library are excluded because there the connector has to swap +the process-global `pcre_malloc` and `pcre_free` pointers around every +libmodsecurity call, which is not thread-safe. + +Rule messages emitted during phase 5 are written using the thread pool's log, +which is the main context `error_log` at its configured level, rather than the +`error_log` of the server or location the request was handled by, and they lack +the usual client and request context. As they are emitted at the `info` level, +the main `error_log` has to be set to `info` or lower for them to appear, and it +has to be a file: an `error_log` that writes to `syslog:` or `memory:` must not +be used with this directive, because those writers are not thread-safe. Any +`SecDebugLog` output produced during phase 5 is likewise written from the +thread. + +The logging performed early by a disruptive intervention stays synchronous, as +does everything up to phase 4. The logging phase runs once per transaction, for +the main request; with `log_subrequest on`, the log phase of a subrequest does +not process it. + +The feature is experimental, and it needs a libmodsecurity that supports +processing transactions concurrently -- the way the threaded Apache MPMs use +it: phases 1 to 4 of one transaction still run on the event loop while phase 5 +of another runs in the thread. Two caveats are worth knowing about: + +* `SecDebugLog` writes made during phases 1 to 4 are not covered by the mutex + and can interleave with debug log writes from the thread; +* the LMDB collection backend (`--with-lmdb`) has not been reviewed for use + from several threads of one process. + +The thread destroys the transaction once it has been logged, so temporary +files of the transaction, such as saved uploads, are removed even if the +worker exits before the task's completion handler runs. + +On graceful shutdown (`nginx -s quit`) the worker does not exit while logging +tasks are pending: it keeps a timer armed until their completion handlers have +run. Use graceful shutdown when queued audit records must be written. A worker +that receives `SIGTERM` exits at once, and the thread pool runs the tasks still +queued before its threads exit, provided the pool's exit task fits into the +queue -- hence `max_queue` greater than `max_pending` on a dedicated pool. On +fast shutdown (`nginx -s stop`), though, the master sends `SIGKILL` to workers +that have not exited after about 1.5 seconds, so queued and running logging tasks +can be lost and their temporary files left behind, whatever the queue size. A +thread blocked on a contended audit log delays a graceful exit, and +`worker_shutdown_timeout` does not shorten that wait. + +```nginx +thread_pool modsec threads=1 max_queue=256; + +http { + modsecurity on; + modsecurity_rules_file /etc/modsecurity/main.conf; + modsecurity_log_thread_pool modsec max_pending=128; +} +``` + # Contributing As an open source project we invite (and encourage) anyone from the community to contribute to our project. This may take the form of: new diff --git a/src/ngx_http_modsecurity_common.h b/src/ngx_http_modsecurity_common.h index 9f79b5ba..3175aa76 100644 --- a/src/ngx_http_modsecurity_common.h +++ b/src/ngx_http_modsecurity_common.h @@ -22,6 +22,10 @@ #include #include +#if (NGX_THREADS) && (NGX_PCRE2) +#include +#endif + #include #include @@ -77,10 +81,40 @@ typedef struct { } ngx_http_modsecurity_header_t; +/* + * Holder for the log destination given to libmodsecurity as the log callback + * data. The indirection allows the destination to be redirected to the + * thread pool's log once the request -- and thus its connection log -- is + * gone. + */ +typedef struct { + ngx_log_t *log; +} ngx_http_modsecurity_msclog_t; + + +#if (NGX_THREADS) && (NGX_PCRE2) + +/* + * Heap-allocated so that it can outlive the request: owned by the request's + * context until the logging task is posted, by the task afterwards. + */ +typedef struct { + ngx_thread_task_t task; + Transaction *transaction; + ngx_http_modsecurity_msclog_t msclog; +} ngx_http_modsecurity_log_task_t; + +#endif + + typedef struct { ngx_http_request_t *r; Transaction *modsec_transaction; ModSecurityIntervention *delayed_intervention; + ngx_http_modsecurity_msclog_t msclog; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_log_task_t *log_task; +#endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) /* @@ -109,6 +143,9 @@ typedef struct { ngx_uint_t rules_inline; ngx_uint_t rules_file; ngx_uint_t rules_remote; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_flag_t log_thread_pool_used; +#endif } ngx_http_modsecurity_main_conf_t; @@ -124,6 +161,10 @@ typedef struct { #endif ngx_http_complex_value_t *transaction_id; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_thread_pool_t *log_thread_pool; + ngx_uint_t log_max_pending; +#endif } ngx_http_modsecurity_conf_t; @@ -163,8 +204,12 @@ int ngx_http_modsecurity_store_ctx_header(ngx_http_request_t *r, ngx_str_t *name #endif /* ngx_http_modsecurity_log.c */ -void ngx_http_modsecurity_log(void *log, const void* data); +void ngx_http_modsecurity_log(void *data, const void* msg); ngx_int_t ngx_http_modsecurity_log_handler(ngx_http_request_t *r); +ngx_int_t ngx_http_modsecurity_log_phase_handler(ngx_http_request_t *r); +#if (NGX_THREADS) && (NGX_PCRE2) +ngx_int_t ngx_http_modsecurity_log_init_process(ngx_cycle_t *cycle); +#endif /* ngx_http_modsecurity_access.c */ ngx_int_t ngx_http_modsecurity_access_handler(ngx_http_request_t *r); diff --git a/src/ngx_http_modsecurity_log.c b/src/ngx_http_modsecurity_log.c index 094685c5..d3e1b1b7 100644 --- a/src/ngx_http_modsecurity_log.c +++ b/src/ngx_http_modsecurity_log.c @@ -24,22 +24,246 @@ void -ngx_http_modsecurity_log(void *log, const void* data) +ngx_http_modsecurity_log(void *data, const void* msg) { - const char *msg; - if (log == NULL) { + ngx_http_modsecurity_msclog_t *msclog = data; + + if (msclog == NULL || msclog->log == NULL) { return; } - msg = (const char *) data; - ngx_log_error(NGX_LOG_INFO, (ngx_log_t *)log, 0, "%s", msg); + ngx_log_error(NGX_LOG_INFO, msclog->log, 0, "%s", (const char *) msg); +} + + +#if (NGX_THREADS) && (NGX_PCRE2) + +/* logging tasks of this worker posted and not yet completed */ +static ngx_uint_t ngx_http_modsecurity_log_pending; +static time_t ngx_http_modsecurity_log_inline_warned; + +/* keeps a gracefully exiting worker alive while tasks are pending */ +static ngx_event_t ngx_http_modsecurity_log_drain; +static ngx_connection_t ngx_http_modsecurity_log_drain_dumb; + +/* serialises msc_process_logging() within the worker */ +static ngx_thread_mutex_t ngx_http_modsecurity_log_mutex; +static ngx_flag_t ngx_http_modsecurity_log_mutex_created; + + +ngx_int_t +ngx_http_modsecurity_log_init_process(ngx_cycle_t *cycle) +{ + ngx_http_modsecurity_main_conf_t *mmcf; + + mmcf = ngx_http_cycle_get_module_main_conf(cycle, + ngx_http_modsecurity_module); + + if (mmcf == NULL || !mmcf->log_thread_pool_used) { + return NGX_OK; + } + + if (ngx_thread_mutex_create(&ngx_http_modsecurity_log_mutex, cycle->log) + != NGX_OK) + { + return NGX_ERROR; + } + + ngx_http_modsecurity_log_mutex_created = 1; + + return NGX_OK; +} + + +static ngx_flag_t +ngx_http_modsecurity_log_lock(ngx_log_t *log) +{ + if (!ngx_http_modsecurity_log_mutex_created) { + return 0; + } + + return ngx_thread_mutex_lock(&ngx_http_modsecurity_log_mutex, log) + == NGX_OK; +} + + +static void +ngx_http_modsecurity_log_unlock(ngx_flag_t locked, ngx_log_t *log) +{ + if (locked) { + (void) ngx_thread_mutex_unlock(&ngx_http_modsecurity_log_mutex, log); + } +} + +#endif + + +static ngx_int_t +ngx_http_modsecurity_log_transaction(ngx_http_request_t *r, + ngx_http_modsecurity_ctx_t *ctx) +{ + ngx_pool_t *old_pool; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_flag_t locked; + + locked = ngx_http_modsecurity_log_lock(r->connection->log); +#endif + + dd("calling msc_process_logging for %p", ctx); + old_pool = ngx_http_modsecurity_pcre_malloc_init(r->pool); + msc_process_logging(ctx->modsec_transaction); + ngx_http_modsecurity_pcre_malloc_done(old_pool); + +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_log_unlock(locked, r->connection->log); +#endif + + return NGX_OK; +} + + +#if (NGX_THREADS) && (NGX_PCRE2) + +static void +ngx_http_modsecurity_log_thread_handler(void *data, ngx_log_t *log) +{ + ngx_flag_t locked; + ngx_http_modsecurity_log_task_t *lt = data; + + /* the request and its connection log may be gone by now */ + lt->msclog.log = log; + + locked = ngx_http_modsecurity_log_lock(log); + msc_process_logging(lt->transaction); + ngx_http_modsecurity_log_unlock(locked, log); + + /* the completion handler does not run if the worker exits first */ + msc_transaction_cleanup(lt->transaction); + lt->transaction = NULL; +} + + +static void +ngx_http_modsecurity_log_thread_event_handler(ngx_event_t *ev) +{ + ngx_http_modsecurity_log_task_t *lt = ev->data; + + ngx_free(lt); + + ngx_http_modsecurity_log_pending--; + + if (ngx_http_modsecurity_log_pending == 0 + && ngx_http_modsecurity_log_drain.timer_set) + { + ngx_del_timer(&ngx_http_modsecurity_log_drain); + } +} + + +static void +ngx_http_modsecurity_log_drain_handler(ngx_event_t *ev) +{ + if (ngx_http_modsecurity_log_pending) { + ngx_add_timer(ev, 1000); + } +} + + +static ngx_int_t +ngx_http_modsecurity_log_offload(ngx_http_request_t *r, + ngx_http_modsecurity_ctx_t *ctx, ngx_http_modsecurity_conf_t *mcf) +{ + ngx_http_modsecurity_log_task_t *lt; + + if (ngx_http_modsecurity_log_pending >= mcf->log_max_pending) { + return NGX_DECLINED; + } + + lt = ctx->log_task; + + lt->transaction = ctx->modsec_transaction; + + lt->task.ctx = lt; + lt->task.handler = ngx_http_modsecurity_log_thread_handler; + lt->task.event.data = lt; + lt->task.event.handler = ngx_http_modsecurity_log_thread_event_handler; + + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "ModSecurity: posting logging to the thread pool"); + + if (ngx_thread_task_post(mcf->log_thread_pool, <->task) != NGX_OK) { + return NGX_ERROR; + } + + /* the task now owns the transaction and itself */ + ctx->modsec_transaction = NULL; + ctx->log_task = NULL; + + ngx_http_modsecurity_log_pending++; + + if (!ngx_http_modsecurity_log_drain.timer_set) { + ngx_http_modsecurity_log_drain.handler = + ngx_http_modsecurity_log_drain_handler; + ngx_http_modsecurity_log_drain.log = ngx_cycle->log; + ngx_http_modsecurity_log_drain.data = + &ngx_http_modsecurity_log_drain_dumb; + ngx_http_modsecurity_log_drain_dumb.fd = (ngx_socket_t) -1; + ngx_add_timer(&ngx_http_modsecurity_log_drain, 1000); + } + + return NGX_OK; +} + +#endif + + +ngx_int_t +ngx_http_modsecurity_log_phase_handler(ngx_http_request_t *r) +{ + ngx_http_modsecurity_ctx_t *ctx; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_conf_t *mcf; +#endif + + /* + * subrequests share the main request's pool, so the cleanup scan in + * ngx_http_modsecurity_get_module_ctx() would return its context + */ + if (r != r->main) { + return NGX_OK; + } + + ctx = ngx_http_modsecurity_get_module_ctx(r); + if (ctx == NULL || ctx->logged || ctx->modsec_transaction == NULL) { + return NGX_OK; + } + +#if (NGX_THREADS) && (NGX_PCRE2) + mcf = ngx_http_get_module_loc_conf(r, ngx_http_modsecurity_module); + + if (mcf->log_thread_pool != NULL && ctx->log_task != NULL) { + if (ngx_http_modsecurity_log_offload(r, ctx, mcf) == NGX_OK) { + return NGX_OK; + } + + if (ngx_time() - ngx_http_modsecurity_log_inline_warned >= 10) { + ngx_http_modsecurity_log_inline_warned = ngx_time(); + + ngx_log_error(NGX_LOG_WARN, r->connection->log, 0, + "ModSecurity: could not post logging to thread " + "pool, %ui tasks pending, processing inline", + ngx_http_modsecurity_log_pending); + } + } +#endif + + return ngx_http_modsecurity_log_transaction(r, ctx); } ngx_int_t ngx_http_modsecurity_log_handler(ngx_http_request_t *r) { - ngx_pool_t *old_pool; ngx_http_modsecurity_ctx_t *ctx; dd("catching a new _log_ phase handler"); @@ -66,10 +290,5 @@ ngx_http_modsecurity_log_handler(ngx_http_request_t *r) return NGX_OK; } - dd("calling msc_process_logging for %p", ctx); - old_pool = ngx_http_modsecurity_pcre_malloc_init(r->pool); - msc_process_logging(ctx->modsec_transaction); - ngx_http_modsecurity_pcre_malloc_done(old_pool); - - return NGX_OK; + return ngx_http_modsecurity_log_transaction(r, ctx); } diff --git a/src/ngx_http_modsecurity_module.c b/src/ngx_http_modsecurity_module.c index 1e55ade0..3b6ea342 100644 --- a/src/ngx_http_modsecurity_module.c +++ b/src/ngx_http_modsecurity_module.c @@ -29,6 +29,10 @@ #define strdup _strdup #endif +#if (NGX_THREADS) && (NGX_PCRE2) +#define NGX_HTTP_MODSECURITY_LOG_MAX_PENDING 128 +#endif + static ngx_int_t ngx_http_modsecurity_init(ngx_conf_t *cf); static void *ngx_http_modsecurity_create_main_conf(ngx_conf_t *cf); static char *ngx_http_modsecurity_init_main_conf(ngx_conf_t *cf, void *conf); @@ -261,7 +265,17 @@ ngx_http_modsecurity_cleanup(void *data) ctx = (ngx_http_modsecurity_ctx_t *) data; - msc_transaction_cleanup(ctx->modsec_transaction); + if (ctx->modsec_transaction != NULL) { + msc_transaction_cleanup(ctx->modsec_transaction); + ctx->modsec_transaction = NULL; + } + +#if (NGX_THREADS) && (NGX_PCRE2) + if (ctx->log_task != NULL) { + ngx_free(ctx->log_task); + ctx->log_task = NULL; + } +#endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) /* @@ -276,11 +290,15 @@ ngx_http_modsecurity_cleanup(void *data) ngx_http_modsecurity_ctx_t * ngx_http_modsecurity_create_ctx(ngx_http_request_t *r) { - ngx_str_t s; + ngx_str_t s = ngx_null_string; ngx_pool_cleanup_t *cln; ngx_http_modsecurity_ctx_t *ctx; ngx_http_modsecurity_conf_t *mcf; + ngx_http_modsecurity_msclog_t *msclog; ngx_http_modsecurity_main_conf_t *mmcf; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_log_task_t *lt; +#endif ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_modsecurity_ctx_t)); if (ctx == NULL) @@ -292,37 +310,62 @@ ngx_http_modsecurity_create_ctx(ngx_http_request_t *r) mmcf = ngx_http_get_module_main_conf(r, ngx_http_modsecurity_module); mcf = ngx_http_get_module_loc_conf(r, ngx_http_modsecurity_module); - dd("creating transaction with the following rules: '%p' -- ms: '%p'", mcf->rules_set, mmcf->modsec); +#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) + ctx->sanity_headers_out = ngx_array_create(r->pool, 12, sizeof(ngx_http_modsecurity_header_t)); + if (ctx->sanity_headers_out == NULL) { + return NGX_CONF_ERROR; + } +#endif + + /* + * The cleanup is armed, and the ctx published, only once the + * transaction exists. + */ + cln = ngx_pool_cleanup_add(r->pool, sizeof(ngx_http_modsecurity_ctx_t)); + if (cln == NULL) + { + dd("failed to create the ModSecurity context cleanup"); + return NGX_CONF_ERROR; + } if (mcf->transaction_id) { if (ngx_http_complex_value(r, mcf->transaction_id, &s) != NGX_OK) { return NGX_CONF_ERROR; } - ctx->modsec_transaction = msc_new_transaction_with_id(mmcf->modsec, mcf->rules_set, (char *) s.data, r->connection->log); + } - } else { - ctx->modsec_transaction = msc_new_transaction(mmcf->modsec, mcf->rules_set, r->connection->log); + msclog = &ctx->msclog; + +#if (NGX_THREADS) && (NGX_PCRE2) + if (mmcf->log_thread_pool_used) { + lt = ngx_calloc(sizeof(ngx_http_modsecurity_log_task_t), + r->connection->log); + if (lt == NULL) { + return NULL; + } + + ctx->log_task = lt; + msclog = <->msclog; } +#endif - dd("transaction created"); + msclog->log = r->connection->log; - ngx_http_set_ctx(r, ctx, ngx_http_modsecurity_module); + dd("creating transaction with the following rules: '%p' -- ms: '%p'", mcf->rules_set, mmcf->modsec); - cln = ngx_pool_cleanup_add(r->pool, sizeof(ngx_http_modsecurity_ctx_t)); - if (cln == NULL) - { - dd("failed to create the ModSecurity context cleanup"); - return NGX_CONF_ERROR; + if (mcf->transaction_id) { + ctx->modsec_transaction = msc_new_transaction_with_id(mmcf->modsec, mcf->rules_set, (char *) s.data, msclog); + + } else { + ctx->modsec_transaction = msc_new_transaction(mmcf->modsec, mcf->rules_set, msclog); } + + dd("transaction created"); + cln->handler = ngx_http_modsecurity_cleanup; cln->data = ctx; -#if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) - ctx->sanity_headers_out = ngx_array_create(r->pool, 12, sizeof(ngx_http_modsecurity_header_t)); - if (ctx->sanity_headers_out == NULL) { - return NGX_CONF_ERROR; - } -#endif + ngx_http_set_ctx(r, ctx, ngx_http_modsecurity_module); return ctx; } @@ -486,6 +529,79 @@ char *ngx_conf_set_transaction_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf } +static char * +ngx_conf_set_log_thread_pool(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) +{ + ngx_str_t *value; +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_int_t n; + ngx_http_modsecurity_conf_t *mcf = conf; + ngx_http_modsecurity_main_conf_t *mmcf; +#endif + + value = cf->args->elts; + +#if (NGX_THREADS) && (NGX_PCRE2) + + if (mcf->log_thread_pool != NGX_CONF_UNSET_PTR) { + return "is duplicate"; + } + + if (ngx_strcmp(value[1].data, "off") == 0) { + if (cf->args->nelts > 2) { + goto invalid; + } + + mcf->log_thread_pool = NULL; + mcf->log_max_pending = 0; + return NGX_CONF_OK; + } + + mcf->log_max_pending = NGX_HTTP_MODSECURITY_LOG_MAX_PENDING; + + if (cf->args->nelts > 2) { + if (ngx_strncmp(value[2].data, "max_pending=", 12) != 0) { + goto invalid; + } + + n = ngx_atoi(value[2].data + 12, value[2].len - 12); + if (n == NGX_ERROR || n == 0) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid max_pending value \"%V\"", &value[2]); + return NGX_CONF_ERROR; + } + + mcf->log_max_pending = n; + } + + mcf->log_thread_pool = ngx_thread_pool_add(cf, &value[1]); + if (mcf->log_thread_pool == NULL) { + return NGX_CONF_ERROR; + } + + mmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_modsecurity_module); + mmcf->log_thread_pool_used = 1; + + return NGX_CONF_OK; + +invalid: + + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid parameter \"%V\"", &value[2]); + return NGX_CONF_ERROR; + +#else + + if (cf->args->nelts == 2 && ngx_strcmp(value[1].data, "off") == 0) { + return NGX_CONF_OK; + } + + return "requires nginx built with --with-threads and PCRE2"; + +#endif +} + + static ngx_command_t ngx_http_modsecurity_commands[] = { { ngx_string("modsecurity"), @@ -527,6 +643,14 @@ static ngx_command_t ngx_http_modsecurity_commands[] = { 0, NULL }, + { + ngx_string("modsecurity_log_thread_pool"), + NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE12, + ngx_conf_set_log_thread_pool, + NGX_HTTP_LOC_CONF_OFFSET, + 0, + NULL + }, { ngx_string("modsecurity_use_error_log"), NGX_HTTP_LOC_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_MAIN_CONF|NGX_CONF_FLAG, @@ -561,7 +685,11 @@ ngx_module_t ngx_http_modsecurity_module = { NGX_HTTP_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_http_modsecurity_log_init_process, /* init process */ +#else NULL, /* init process */ +#endif NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ @@ -613,7 +741,7 @@ ngx_http_modsecurity_init(ngx_conf_t *cf) dd("Not able to create a new NGX_HTTP_LOG_PHASE handle"); return NGX_ERROR; } - *h_log = ngx_http_modsecurity_log_handler; + *h_log = ngx_http_modsecurity_log_phase_handler; rc = ngx_http_modsecurity_header_filter_init(); @@ -730,6 +858,10 @@ ngx_http_modsecurity_create_conf(ngx_conf_t *cf) conf->pool = cf->pool; conf->transaction_id = NGX_CONF_UNSET_PTR; conf->use_error_log = NGX_CONF_UNSET; +#if (NGX_THREADS) && (NGX_PCRE2) + conf->log_thread_pool = NGX_CONF_UNSET_PTR; + conf->log_max_pending = NGX_CONF_UNSET_UINT; +#endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) conf->sanity_checks_enabled = NGX_CONF_UNSET; #endif @@ -770,6 +902,10 @@ ngx_http_modsecurity_merge_conf(ngx_conf_t *cf, void *parent, void *child) ngx_conf_merge_value(c->enable, p->enable, 0); ngx_conf_merge_ptr_value(c->transaction_id, p->transaction_id, NULL); ngx_conf_merge_value(c->use_error_log, p->use_error_log, 1); +#if (NGX_THREADS) && (NGX_PCRE2) + ngx_conf_merge_ptr_value(c->log_thread_pool, p->log_thread_pool, NULL); + ngx_conf_merge_uint_value(c->log_max_pending, p->log_max_pending, 0); +#endif #if defined(MODSECURITY_SANITY_CHECKS) && (MODSECURITY_SANITY_CHECKS) ngx_conf_merge_value(c->sanity_checks_enabled, p->sanity_checks_enabled, 0); #endif diff --git a/tests/modsecurity-log-thread-pool-exit.t b/tests/modsecurity-log-thread-pool-exit.t new file mode 100644 index 00000000..791dbf0b --- /dev/null +++ b/tests/modsecurity-log-thread-pool-exit.t @@ -0,0 +1,174 @@ +#!/usr/bin/perl + +# +# ModSecurity, http://www.modsecurity.org/ +# Copyright (c) 2015 Trustwave Holdings, Inc. (http://www.trustwave.com/) +# +# You may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# If any of the files related to licensing are missing or if you have any +# other questions related to licensing please contact Trustwave Holdings, Inc. +# directly using the email address security@modsecurity.org. +# + + +# Tests for ModSecurity module, logging tasks in a thread pool at worker exit. + +############################################################################### + +use warnings; +use strict; + +use Test::More; + +BEGIN { use FindBin; chdir($FindBin::Bin); } + +use lib 'lib'; +use Test::Nginx; + +############################################################################### + +select STDERR; $| = 1; +select STDOUT; $| = 1; + +my $t = Test::Nginx->new()->has(qw/http threads/); + +$t->write_file_expand('nginx.conf', <<'EOF'); + +%%TEST_GLOBALS%% + +daemon off; + +thread_pool tiny threads=1 max_queue=1; +thread_pool slow threads=1; + +events { +} + +http { + %%TEST_GLOBALS_HTTP%% + + server { + listen 127.0.0.1:8080; + server_name localhost; + + modsecurity on; + modsecurity_rules ' + SecRuleEngine On + SecRequestBodyAccess On + SecRule REQUEST_URI "@inspectFile %%TESTDIR%%/sleep.sh" "id:81,phase:5,pass,nolog" + '; + + # max_pending=2 with max_queue=1 leaves no room for the pool's exit + # task while one task runs and one waits. + + location /drain/ { + modsecurity_log_thread_pool tiny max_pending=2; + modsecurity_rules ' + SecAuditEngine On + SecAuditLogParts ABZ + SecAuditLog %%TESTDIR%%/audit.txt + SecAuditLogType Serial + SecAuditLogStorageDir %%TESTDIR%%/ + '; + } + + location = /pid { + modsecurity off; + add_header X-Pid $pid; + return 204; + } + + location /upload/ { + modsecurity_log_thread_pool slow; + modsecurity_rules ' + SecUploadDir %%TESTDIR%%/upload + SecTmpSaveUploadedFiles On + SecUploadKeepFiles Off + '; + } + } +} + +EOF + +mkdir($t->testdir() . '/drain'); +$t->write_file('/drain/index.html', 'DRAIN INDEX'); +mkdir($t->testdir() . '/upload'); +$t->write_file('sleep.sh', "#!/bin/sh\nsleep 1\n"); +chmod(0755, $t->testdir() . '/sleep.sh'); +$t->try_run('no modsecurity_log_thread_pool')->plan(5); + +############################################################################### + +# Graceful shutdown waits for the logging tasks, even when the pool's queue +# has no room for its exit task. + +like(http_get('/drain/index.html'), qr/DRAIN INDEX/, 'first request'); +select undef, undef, undef, 0.2; +like(http_get('/drain/index.html'), qr/DRAIN INDEX/, 'second request'); + +$t->stop(); + +my $records = () = read_file($t->testdir() . '/audit.txt') =~ /^---\w+---Z--$/mg; +is($records, 2, 'audit records written before the worker exited'); + +# Fast shutdown of the worker: the pool still runs the task, which also destroys the +# transaction and so removes its temporary upload file. + +$t->run(); + +my $body = <<"EOF"; +--XyZ\r +Content-Disposition: form-data; name="f"; filename="a.txt"\r +Content-Type: text/plain\r +\r +UPLOADED\r +--XyZ--\r +EOF + +http(<<"EOF" . $body); +POST /upload/index.html HTTP/1.0\r +Host: localhost\r +Content-Type: multipart/form-data; boundary=XyZ\r +Content-Length: @{[ length $body ]}\r +\r +EOF + +select undef, undef, undef, 0.2; +ok(upload_files($t->testdir()) > 0, 'upload saved to a temporary file'); + +http_get('/pid') =~ qr/X-Pid: (\d+)/; +my $worker = $1; +kill 'TERM', $worker; + +for (1 .. 50) { + last unless kill 0, $worker; + select undef, undef, undef, 0.1; +} + +is(upload_files($t->testdir()), 0, 'temporary upload file removed at fast shutdown'); + +############################################################################### + +sub upload_files { + my ($dir) = @_; + opendir(my $dh, $dir . '/upload') or return -1; + my @files = grep { !/^\./ } readdir($dh); + closedir($dh); + return scalar @files; +} + +sub read_file { + my ($path) = @_; + open my $fh, '<', $path or return ''; + local $/; + my $data = <$fh>; + close $fh; + return defined $data ? $data : ''; +} + +############################################################################### diff --git a/tests/modsecurity-log-thread-pool.t b/tests/modsecurity-log-thread-pool.t new file mode 100644 index 00000000..00e588da --- /dev/null +++ b/tests/modsecurity-log-thread-pool.t @@ -0,0 +1,207 @@ +#!/usr/bin/perl + +# +# ModSecurity, http://www.modsecurity.org/ +# Copyright (c) 2015 Trustwave Holdings, Inc. (http://www.trustwave.com/) +# +# You may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# If any of the files related to licensing are missing or if you have any +# other questions related to licensing please contact Trustwave Holdings, Inc. +# directly using the email address security@modsecurity.org. +# + + +# Tests for ModSecurity module, phase 5 and audit logging in a thread pool. + +############################################################################### + +use warnings; +use strict; + +use Test::More; + +BEGIN { use FindBin; chdir($FindBin::Bin); } + +use lib 'lib'; +use Test::Nginx; + +############################################################################### + +select STDERR; $| = 1; +select STDOUT; $| = 1; + +my $t = Test::Nginx->new()->has(qw/http threads ssi/); + +$t->write_file_expand('nginx.conf', <<'EOF'); + +%%TEST_GLOBALS%% + +daemon off; + +thread_pool modsec threads=2; +thread_pool slow threads=1; + +events { +} + +http { + %%TEST_GLOBALS_HTTP%% + + server { + listen 127.0.0.1:8080; + server_name localhost; + + modsecurity on; + modsecurity_log_thread_pool modsec; + modsecurity_rules ' + SecRuleEngine On + SecRule ARGS:what "@streq block" "id:71,phase:1,deny,status:403,log,auditlog" + SecRule ARGS:what "@streq p5" "id:72,phase:5,log,auditlog,pass" + SecRule ARGS:what "@streq p5redir" "id:73,phase:5,log,pass" + '; + + log_subrequest on; + + # libmodsecurity merges a non-empty parent SecAuditLog over the + # child's one, so the audit log is configured per location here. + + location / { + modsecurity_rules ' + SecAuditEngine RelevantOnly + SecAuditLogParts ABH + SecAuditLog %%TESTDIR%%/audit.txt + SecAuditLogType Serial + SecAuditLogStorageDir %%TESTDIR%%/ + '; + } + + location /sync/ { + modsecurity_log_thread_pool off; + modsecurity_rules ' + SecAuditEngine RelevantOnly + SecAuditLogParts ABH + SecAuditLog %%TESTDIR%%/audit-sync.txt + SecAuditLogType Serial + SecAuditLogStorageDir %%TESTDIR%%/ + '; + } + + location /ssi/ { + ssi on; + modsecurity_rules ' + SecAuditEngine RelevantOnly + SecAuditLogParts ABH + SecAuditLog %%TESTDIR%%/audit-ssi.txt + SecAuditLogType Serial + SecAuditLogStorageDir %%TESTDIR%%/ + '; + } + + location /busy/ { + modsecurity_log_thread_pool slow max_pending=1; + modsecurity_rules ' + SecRule REQUEST_URI "@inspectFile %%TESTDIR%%/sleep.sh" "id:74,phase:5,pass,nolog" + '; + } + + location /redir/ { + modsecurity_log_thread_pool off; + try_files /nonexistent /index.html; + } + } +} + +EOF + +$t->write_file('index.html', 'INDEX'); +mkdir($t->testdir() . '/sync'); +$t->write_file('/sync/index.html', 'SYNC INDEX'); +mkdir($t->testdir() . '/ssi'); +$t->write_file('/ssi/index.html', 'SSI '); +mkdir($t->testdir() . '/busy'); +$t->write_file('/busy/index.html', 'BUSY INDEX'); +$t->write_file('sleep.sh', "#!/bin/sh\nsleep 1\n"); +chmod(0755, $t->testdir() . '/sleep.sh'); +$t->try_run('no modsecurity_log_thread_pool')->plan(18); + +############################################################################### + +like(http_get('/index.html?what=block'), qr/^HTTP.*403/, 'phase 1 deny still blocks'); +like(http_get('/index.html?what=p5'), qr/INDEX/, 'request with phase 5 rule passes'); +like(http_get('/index.html?what=none'), qr/INDEX/, 'plain request passes'); +like(http_get('/sync/index.html?what=p5'), qr/SYNC INDEX/, 'thread pool off in location'); +like(http_get('/ssi/index.html?what=p5'), qr/SSI INDEX/, 'request with subrequest passes'); +like(http_get('/redir/x?what=p5redir'), qr/INDEX/, 'internally redirected request passes'); + +my $audit = wait_for_file($t->testdir() . '/audit.txt', qr/\[id "72"\]/); +like($audit, qr/\[id "72"\]/, 'phase 5 rule logged from thread'); +like($audit, qr/what=block/, 'blocked request audited'); +like($audit, qr/\[id "71"\]/, 'phase 1 deny audited'); + +my $sync = wait_for_file($t->testdir() . '/audit-sync.txt', qr/\[id "72"\]/); +like($sync, qr/\[id "72"\]/, 'synchronous location still audits'); + +# The thread pool's log is the main error_log, which has no handler, so a rule +# message written from the thread carries no ", client: ..." suffix, while one +# written on the request's connection log does. This tells the offloaded path +# apart from the (deliberately silent) synchronous fallback. + +# With log_subrequest on, the log phase of the SSI subrequest must not run +# phase 5 of the main request's transaction. Each serial audit log record +# starts with a part-A section header line, so counting those counts the +# records; the second run would come from the thread, hence the extra wait. + +my $ssi = wait_for_file($t->testdir() . '/audit-ssi.txt', qr/\[id "72"\]/); +select undef, undef, undef, 0.5; +$ssi = read_file($t->testdir() . '/audit-ssi.txt'); +my $records = () = $ssi =~ /^---\w+---A--$/mg; +is($records, 1, 'phase 5 ran once for a request with a subrequest'); + +wait_for_file($t->testdir() . '/error.log', qr/\[id "73"\]/); + +my $errlog = read_file($t->testdir() . '/error.log'); +unlike($errlog, qr/could not post logging/, 'logging was not refused by the pool'); +like($errlog, qr/^(?:(?!client:).)*\[id "72"\](?:(?!client:).)*$/m, + 'phase 5 rule message written from the thread pool log'); +like($errlog, qr/^.*\[id "72"\].*client:.*$/m, + 'phase 5 rule message of the synchronous location keeps the request context'); +like($errlog, qr/^(?:(?!client:).)*\[id "73"\](?:(?!client:).)*$/m, + 'offload decided by the location the request ended in'); + +# The phase 5 rule of /busy/ keeps the only thread of its pool busy for a +# second, so the second request finds max_pending reached and is logged +# inline. + +like(http_get('/busy/index.html'), qr/BUSY INDEX/, 'first request to busy pool'); +like(http_get('/busy/index.html'), qr/BUSY INDEX/, 'second request to busy pool'); +like(read_file($t->testdir() . '/error.log'), + qr/could not post logging to thread pool, 1 tasks pending/, + 'max_pending reached falls back to inline logging'); + +############################################################################### + +sub wait_for_file { + my ($path, $re) = @_; + my $data = ''; + for (1 .. 50) { + $data = read_file($path); + last if $data =~ $re; + select undef, undef, undef, 0.1; + } + return $data; +} + +sub read_file { + my ($path) = @_; + open my $fh, '<', $path or return ''; + local $/; + my $data = <$fh>; + close $fh; + return defined $data ? $data : ''; +} + +###############################################################################