Repository navigation
Conversation
Subrequests share the parent's pool, so the cleanup-list fallback in ngx_http_modsecurity_get_module_ctx() returns the main request's transaction for them. With auth_request the auth subrequest's response headers were fed to phase 3 as the "response headers" of the main request and ctx->processed was set, so the real response headers were never inspected. With log_subrequest on, the subrequest also runs the LOG phase, so msc_process_logging() was called on the main request's transaction once for the subrequest and once for the main request: phase 5 and the audit log ran twice for a single transaction. Skip subrequests in the header filter, in the body filter and in the log handler; the main request's own headers, body and logging phase are still processed. The body filter cannot currently see a subrequest, because the module is ordered after the postpone filter and that one forwards subrequest output as r->main, but the check is kept so the module stays correct if the filter order ever changes. No check is needed in the access handler: ngx_http_core_access_phase() already skips the whole ACCESS phase for subrequests. Behaviour change: subrequests are no longer inspected at all, even when the parent location has modsecurity off and the subrequest's location has it on. A subrequest never produces a response of its own, so there is nothing for the WAF to protect there.
|
No actionable comments were generated in the recent review. Walkthrough
Pre-merge checks:
|
|
| * content, not to the response the client sees; only inspect the main | ||
| * request. | ||
| */ | ||
| if (r != r->main) { |
There was a problem hiding this comment.
Subrequest bodies can skip response-body inspection when served from files. This was already true before this PR, and this early return is the natural place to fix it.
SSI/addition subrequest output does reach the body filter, through the postpone filter as r->main. But filter_need_in_memory is only ever set on the main request (L467), so the subrequest's copy filter does not have to load file buffers into memory. With sendfile on, an <!--# include virtual="/static.txt" --> (or add_after_body) of a static file arrives as an in_file buf with an empty pos..last. msc_append_response_body() then gets 0 bytes, and the included content reaches the client without phase-4 inspection.
On master the subrequest also returned before L467 (ctx->processed was already set), so this is not a regression. Since this branch now handles subrequests explicitly, setting r->filter_need_in_memory = 1 here (when the main request has a ctx) would close the gap.
There was a problem hiding this comment.
Done in 3ab4c22. The subrequest branch of the header filter now sets r->filter_need_in_memory = 1 when ngx_http_modsecurity_get_module_ctx(r->main) is non-NULL, then passes the subrequest on. That is the flag ngx_http_copy_filter() reads for the subrequest (r->main_filter_need_in_memory || r->filter_need_in_memory). I set it per subrequest rather than setting main_filter_need_in_memory on the main request, because ngx_http_subrequest() only copies that into subrequests created after the main header filter has run.
New test in modsecurity-subrequest.t: sendfile on, an SSI <!--# include virtual="/static.txt" --> of a text/plain file, and a phase-4 RESPONSE_BODY rule matching a marker that is only present in the included file. It fails against master and against a build of this branch with only the filter_need_in_memory line removed (the audit record has rule 500 but not 503), and it passes with the change.
| return ngx_http_next_body_filter(r, in); | ||
| } | ||
|
|
||
| if (r != r->main) { |
There was a problem hiding this comment.
As the PR description says, this guard cannot be reached with the enforced filter order: postpone runs before this filter and forwards subrequest output as r->main. As a "safeguard", though, it fails open. If the order ever changed (or a build ended up with modsecurity ahead of postpone), SSI/addition-included content would silently skip response-body inspection. Before this PR it would have been inspected against the main transaction.
Either drop it as dead code, or make it fail closed, e.g. look up the ctx via r->main instead of skipping. A security filter's fallback should not turn into an inspection bypass.
There was a problem hiding this comment.
Dropped in 6efc199, together with the other per-hook guards. The subrequest check now sits in ngx_http_modsecurity_get_module_ctx() (see the log.c thread).
I did not turn it into a lookup via r->main. Subrequest output can only reach this filter through ngx_http_postpone_filter(), which calls the next filter with r->main. On that path r == r->main, so the included content is inspected against the main transaction, as before. For that to change, the module would have to sit above postpone:
- postpone is always compiled in (
auto/modules:if :;block). configputs this module below it in both link modes:ngx_module_orderfor dynamic builds, and theHTTP_FILTER_MODULESrepositioning for static builds, which exits with an error if it cannot place the module.- The body inspection already depends on that same placement to see the body before gzip.
A fail-closed branch would also have needed process_intervention() and ngx_http_filter_finalize_request() to be split between r and r->main, all for a path the build cannot produce. Deleting it seemed the better choice.
| log_subrequest on; | ||
| modsecurity_rules ' | ||
| SecRuleEngine On | ||
| SecRule REQUEST_URI "@streq /index.html" "id:500,phase:2,log,auditlog,pass" |
There was a problem hiding this comment.
The phase-5 double-run assertion only works because modsecurity's ACCESS handler runs before auth_request's, so rule 500 has already made the transaction relevant when the subrequest is logged. That order comes from module registration order (handlers registered later run first). If auth_request ran first, then on master the premature phase 5 would see a non-relevant transaction and write no record. You would get 1 record with or without the fix, and the test would pass on master. A phase:1 rule (or one matching a request header) would make the transaction relevant before any ACCESS-phase handler order matters.
There was a problem hiding this comment.
Agreed that the test depended on handler order. A phase:1 rule would not have fixed that, though: phases 1 and 2 both run inside this module's own ACCESS handler (ngx_http_modsecurity_access_handler(): msc_process_request_headers() at access.c:275, msc_process_request_body() at access.c:444). If auth_request ran first, the subrequest would finish and be logged before any rule ran, whatever the phase.
Changed in 6efc199. The record count now uses an SSI include virtual with log_subrequest on. That subrequest is created while the main body is being sent, after every ACCESS-phase handler has run, so the request-header rule (now phase:1, matching X-Test) has already made the transaction relevant. Against a master build it counts 2 records and fails; with this branch it counts 1. The auth_request part of the file now only checks the phase-3 header behaviour. That part has the same order dependence: if auth_request's handler ran first, no ctx would exist yet when the subrequest's header filter runs, and tests 3 and 4 would pass on master too. I kept it because the phase-3 bug needs a subrequest whose headers go out before the main response's, and auth_request is that case. The order is fixed by module registration: ngx_http_init_phase_handlers() walks each phase's handlers in reverse, and this filter module registers after the core auth_request module. Against a master build both assertions fail.
| * share the parent's pool and therefore its transaction; processing the | ||
| * logging phase there would run phase 5 and the audit log twice. | ||
| */ | ||
| if (r != r->main) { |
There was a problem hiding this comment.
Root cause / altitude: all three new guards (here, header filter L432, body filter L53) work around one thing, the pool-cleanup fallback in ngx_http_modsecurity_get_module_ctx(). Subrequests share the parent's pool, so that fallback hands them the parent's transaction. Returning NULL from get_module_ctx() when r != r->main (before the cleanup walk) would fix this in one place. It would also cover the other callers (ngx_http_modsecurity_process_intervention()) and any future ones, without copying the same check and comment into every hook. Each hook already handles a NULL ctx by passing through.
There was a problem hiding this comment.
Done in 6efc199. ngx_http_modsecurity_get_module_ctx() returns NULL for r != r->main before the cleanup walk, and the guards in the log handler, header filter and body filter are gone.
I checked every caller:
- Log handler, body filter, header filter: pass through on NULL.
- Access handler and
ngx_http_modsecurity_request_read(): only run for the main request (ngx_http_core_access_phase()skips the ACCESS phase for subrequests, and the read-body post handler belongs to the main request). ngx_http_modsecurity_process_intervention(): every call site (access.c, header filter, body filter) runs only after that hook holds a non-NULL ctx.- The header resolvers and
store_ctx_header(): only reached from the main-request path of the header filter.
Because the access handler is the only place a ctx is created, subrequests never had one of their own. The cleanup walk was the only way they got a transaction.
The header filter still has an r != r->main branch, but now it only sets filter_need_in_memory for the L432 issue (3ab4c22). It does not look up the subrequest's own ctx.
|
|
||
| ############################################################################### | ||
|
|
||
| sub read_file { |
There was a problem hiding this comment.
Reuse: Test::Nginx already provides $t->read_file($name), which reads a file from the test dir. my $audit = $t->read_file('audit.txt'); would replace this local helper. (It dies on a missing file rather than returning '', which gives a clearer failure here anyway.)
There was a problem hiding this comment.
Done in 6efc199: both audit files are read with $t->read_file(), and the local helper is removed.
| @@ -0,0 +1,109 @@ | |||
| #!/usr/bin/perl | |||
|
|
|||
| # (C) Andrei Belov | |||
There was a problem hiding this comment.
Nit: this new file credits (C) Andrei Belov, apparently copied from the other test headers, but he did not write it. Use the actual author, or no copyright line.
The three r != r->main guards added to the log handler, the header filter and the body filter all worked around one thing: subrequests share the parent's pool (ngx_http_subrequest() sets sr->pool = r->pool), so the pool-cleanup fallback in ngx_http_modsecurity_get_module_ctx() handed them the main request's transaction. The connector never creates a context for a subrequest (ngx_http_core_access_phase() skips the ACCESS phase for them), so that fallback was the only way a subrequest got one. Return NULL from ngx_http_modsecurity_get_module_ctx() for subrequests before the cleanup walk and drop the per-hook guards. Every caller already treats a NULL ctx as "not ours" and passes through, and ngx_http_modsecurity_process_intervention() and the header resolvers are only reached from the main-request path. The body-filter guard is dropped rather than turned into a lookup via r->main: subrequest output reaches this filter only through the postpone filter, which forwards it as r->main (ngx_http_postpone_filter()), so the included content is inspected against the main transaction. postpone is always compiled in (auto/modules) and config places this module below it for both static and dynamic builds, the same ordering the body inspection already relies on to see the body before gzip. Test: the phase-5 double-run check now uses an SSI include, which is created while the main body is sent, after every ACCESS-phase handler. The auth_request variant depended on this module's ACCESS handler running before auth_request's; a phase:1 rule would not have changed that, since phases 1 and 2 both run inside that same handler. The record count is 2 on master and 1 with this change. Also use Test::Nginx's read_file() and drop the copied copyright line.
SSI and addition subrequest output reaches the body filter through the postpone filter as part of the main response, but the copy filter decides whether to read file buffers into memory per request (ngx_http_copy_filter(): r->main_filter_need_in_memory || r->filter_need_in_memory, both of the subrequest). The connector only set filter_need_in_memory on the main request, so with sendfile on an included static file arrived as an in_file buffer with an empty pos..last, msc_append_response_body() got 0 bytes and the included content reached the client without phase-4 inspection. This predates the subrequest changes: on master the subrequest returned at ctx->processed before the flag was set. In the header filter, set filter_need_in_memory on a subrequest whose main request has a modsecurity context, then pass it through. This is done per subrequest rather than via main_filter_need_in_memory on the main request, because that flag is only copied into subrequests created afterwards. Test: SSI include of a static text file with sendfile on and a phase-4 rule matching content only present in the included file; it fails without this change and passes with it.
Review follow-up. The r->internal / r->filter_finalize guard assumed only
nginx core clears r->ctx. lua-nginx-module's ngx.exec("@name") zeroes
r->ctx itself (ngx_http_lua_handle_exec) and then calls
ngx_http_named_location(), which sets r->internal only once it has found
the location. On "could not find named location" or the redirect-cycle
500 the request is finalized with the context gone, internal = 0 and
filter_finalize = 0, so the guard skipped the walk, phase 3 and
msc_process_logging() never ran and the audit record was lost.
Drop the guard: walk whenever the slot is empty, as before this branch.
When the walk finds the transaction, store it back with
ngx_http_set_ctx(), so the following calls for the same request (header
filter, every body filter call, log handler, and the access handler of a
redirect target) are a plain array read again until the next reset. A
request without a transaction still walks on every call, as on master.
Replace the stale realip comment and drop the claim that subrequests
recovering the main transaction is correct; that is fixed separately in
owasp-modsecurity#394.
Test changes:
- prove recovery by counting a logged phase-1 rule in the error log: a
request that lost its context gets a second transaction at the
try_files target, which runs phase 1 again. Request data cannot tell
the two apart: the new transaction is fed r->unparsed_uri, which an
internal redirect does not change, so ARGS (and any TX state derived
from them) are the same in both.
- replace the deny-before-try_files case with a phase-1 deny followed by
an error_page redirect; without recovery the error page request gets
a second transaction and the deny is audited twice.
- drop SecAuditLogStorageDir, which only applies to Concurrent.
With get_module_ctx() reduced to the r->ctx lookup, tests 5-8 fail
(phase-1 count 3/2, phase-3 count 1/2, deny records 2/1, no record for
the 412).
|
Reworked after review, in two commits on top of the previous head:
Tests:
Related PRs:
|
|
|
Follow-up on the review:
New issues: none found. CI is green. Overall: all six points are addressed, and this looks good to merge from my side. Please keep the rebase notes for #392/#395/#396 in mind, whichever lands second. |



what
ngx_http_modsecurity_get_module_ctx()returns NULL for subrequests (r != r->main), before the pool-cleanup walk. The header filter, the body filter and the LOG-phase handler already pass through on a NULL ctx, andngx_http_modsecurity_process_intervention()is only called once a hook holds a non-NULL ctx, so no per-hook guard is needed. No change to the access handler: nginx core already skips the ACCESS phase for subrequests.filter_need_in_memoryon a subrequest whose main request has a modsecurity context, so file-backed subrequest output (SSIinclude virtualof a static file withsendfile on;add_before_body/add_after_bodytake the same path) is read into memory and inspected in phase 4 as part of the main response.tests/modsecurity-subrequest.t:auth_request: the main response's headers are inspected in phase 3, the auth subrequest's response is not.log_subrequest on: exactly one audit record per request.sendfile on: a phase-4 rule matching content only present in the included file fires.why
ngx_http_subrequest():sr->pool = r->pool), andngx_http_modsecurity_get_module_ctx()falls back to walking the pool cleanup list when a request has no context of its own. A subrequest therefore got the main request's transaction. Fixing it in that function covers every caller, present and future.auth_request, the auth subrequest's response headers were fed to phase 3 as the main request's response headers andctx->processedwas set, so the real response headers were never inspected (the new test fails on master: the rule for the main response header is absent, the rule for the subrequest's status is present).log_subrequest on, the LOG phase also runs for subrequests, somsc_process_logging()ran more than once on one transaction: phase 5 and the audit log executed twice (the new test counts two audit records on master, one after the fix).filter_need_in_memorywas only ever set on the main request, and the copy filter decides per request (ngx_http_copy_filter()reads the subrequest's ownmain_filter_need_in_memory || filter_need_in_memory). Withsendfile on, included static content arrived as anin_filebuffer with no in-memory data and skipped phase-4 inspection. This predates this PR (on master the subrequest returned atctx->processedbefore the flag was set); the new test fails on master and without the header-filter change.modsecurity onwhile the parent has it off. Their output that reaches the client goes through the postpone filter asr->mainand is inspected as part of the main response.references
ngx_http_modsecurity_get_module_ctx().Origin: this change comes from a performance review of the connector done with Claude Fable 5.1 (Anthropic). After review it was reworked to fix the subrequest case in
ngx_http_modsecurity_get_module_ctx()and to keep subrequest output in memory. Verified by building the module against nginx 1.31.9 with libmodsecurity v3/master and running the fulltests/modsecurity*.tsuite (16 files, 258 tests, all passing); the new tests were also run against a master build and against a build without the header-filter change to confirm they fail there.Summary by CodeRabbit
Bug Fixes
Tests