Repository navigation
Conversation
The header and body filters run for every response on the server, including locations where modsecurity is off, and each call without a module context walked r->pool->cleanup looking for one. r->ctx is only discarded by ngx_http_internal_redirect(), which zeroes it in ngx_http_core_module.c:2652 and sets r->internal at :2663, by ngx_http_named_location() (:2726, r->internal at :2720) and by ngx_http_filter_finalize_request(), which zeroes it in ngx_http_special_response.c:551 and sets r->filter_finalize at :557. Subrequests get a fresh context array (ngx_http_core_module.c:2471) and have internal set too (:2577). The walk is now limited to those requests, matching ngx_http_realip_get_module_ctx() (ngx_http_realip_module.c:552), the function this fallback is modelled on. The filter finalize case is reachable from an ordinary, non internal request: the not modified filter calls ngx_http_filter_finalize_request() with a NULL module on a failed If-Match or If-Unmodified-Since precondition (ngx_http_not_modified_filter_module.c:67 and :74), which discards every module context. tests/modsecurity-internal-redirect.t covers it along with try_files and named location redirects.
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
|
The header and body filters run for every response on the server, including locations where modsecurity is off, and each call without a module context walked r->pool->cleanup looking for one. r->ctx is only discarded by ngx_http_internal_redirect(), which zeroes it in ngx_http_core_module.c:2652 and sets r->internal at :2663, by ngx_http_named_location() (:2726, r->internal at :2720) and by ngx_http_filter_finalize_request(), which zeroes it in ngx_http_special_response.c:551 and sets r->filter_finalize at :557. Subrequests get a fresh context array (ngx_http_core_module.c:2471) and have internal set too (:2577). The walk is now limited to those requests, matching ngx_http_realip_get_module_ctx() (ngx_http_realip_module.c:552), the function this fallback is modelled on. The filter finalize case is reachable from an ordinary, non internal request: the not modified filter calls ngx_http_filter_finalize_request() with a NULL module on a failed If-Match or If-Unmodified-Since precondition (ngx_http_not_modified_filter_module.c:67 and :74), which discards every module context. tests/modsecurity-internal-redirect.t covers it along with try_files and named location redirects. Author: Tom Sommer owasp-modsecurity#392
| ngx_http_modsecurity_ctx_t *ctx; | ||
| ctx = ngx_http_get_module_ctx(r, ngx_http_modsecurity_module); | ||
| if (ctx == NULL) { | ||
| if (ctx == NULL && (r->internal || r->filter_finalize)) { |
There was a problem hiding this comment.
The guard assumes nginx core is the only thing that clears r->ctx, but third-party modules do it too. lua-nginx-module's ngx.exec("@name") zeroes r->ctx itself (ngx_http_lua_util.c, "clear the modules contexts") and then calls ngx_http_named_location(). That function only sets r->internal = 1 once it finds the location. If the location is missing, or the request hits the "rewrite or internal redirection cycle" path, it finalizes with 500 and leaves internal at 0 and filter_finalize unset.
In that case the 500 response's header filter and the log handler now get ctx == NULL and skip the walk. Phase 3 is skipped and msc_process_logging() never runs, so the audit record for a request that already went through phases 1 and 2 is lost. Before this change the walk recovered it.
That is an edge case, but the comment's claim that "every other request still has its context" only holds for nginx core.
There was a problem hiding this comment.
Confirmed. In lua-nginx-module, ngx_http_lua_handle_exec() zeroes r->ctx and then calls ngx_http_named_location(). That function sets r->internal = 1 only in the branch where it finds the location. The "could not find named location" and redirect-cycle paths finalize with 500 while internal and filter_finalize are both still 0. The empty-URI path in ngx_http_named_location() behaves the same way. (ngx.exec(uri) is not affected: that branch does not zero r->ctx itself, and ngx_http_internal_redirect() returns on the cycle check before its own ngx_memzero().)
Done in b85a200: I removed the guard. The walk now runs whenever the slot is empty, as on master, and the recovered context is written back (see the efficiency thread). get_module_ctx() no longer depends on who cleared r->ctx. This repo's test build has no lua module, so I could not test the lua path directly.
| <$fh>; | ||
| }; | ||
|
|
||
| like($audit, qr/\[id "92"\]/, 'phase 3 ran after internal redirect (ctx recovered)'); |
There was a problem hiding this comment.
These assertions pass whether or not the context is recovered after the redirect. If get_module_ctx() returned NULL at /fallback or @named, the access handler there would just create a new transaction. Rule 92 would match on that new transaction and be audit-logged. The original transaction has no matches, so under RelevantOnly it never writes a record. The log still has exactly two [id "92"] records, so the is(scalar @p3, 2, ...) check passes too.
To actually prove recovery, carry state from before the redirect into phase 3. For example, add SecRule ARGS:what "@streq ok" "id:90,phase:1,pass,nolog,setvar:tx.pre=1" and make rule 92 a chain that also requires TX:pre "@eq 1". A fresh transaction would not have tx.pre set.
There was a problem hiding this comment.
Agreed, the assertions did not prove recovery. The tx.pre chain doesn't prove it for the try_files case either. I checked this with a build where get_module_ctx() only reads r->ctx. The new transaction at /fallback is fed r->unparsed_uri (ngx_http_modsecurity_access_handler() -> msc_process_uri()), and ngx_http_internal_redirect() does not change unparsed_uri. So ARGS:what=ok and tx.pre=1 are set again, and the chained rule 92 still matched. For @named it does discriminate, but only because return runs before the access phase, so no new transaction is created.
Done in b85a200: the test now counts a logged phase-1 rule (id 90) in the error log. A lost context means a second transaction that runs phase 1 again, so the count is 3 instead of 2. With the no-recovery build, tests 5-8 fail:
- phase-1 count 3, expected 2 (try_files)
- phase-3 count 1, expected 2 (named location)
- deny records 2, expected 1 (error_page)
- no record for the 412 (filter finalization)
All of them pass with the fix.
| * r->ctx is only discarded by ngx_http_internal_redirect() and | ||
| * ngx_http_named_location(), which set r->internal, and by | ||
| * ngx_http_filter_finalize_request(), which sets r->filter_finalize; | ||
| * subrequests start with a fresh context array and have internal |
There was a problem hiding this comment.
This is existing behaviour, but the new comment presents it as intended, so it is worth a closer look. Subrequests share the parent's pool (sr->pool = r->pool, ngx_http_core_module.c:2500), and the access phase is skipped for subrequests (ngx_http_core_access_phase: if (r != r->main)). A subrequest therefore never gets its own transaction. This walk hands it the main request's transaction.
Example: an auth_request subrequest's response headers go through ngx_http_modsecurity_header_filter() with the main transaction. The filter runs phase 3 on the auth backend's headers and status and sets ctx->processed = 1. When the real response comes through later, the filter returns early on processed, so phase 3 never sees the actual response. It may be worth handling r != r->main explicitly here, or at least not describing subrequest recovery as correct.
There was a problem hiding this comment.
Agreed. Subrequests picking up the main transaction is a bug, and the comment should not describe it as intended. Done in b85a200: the comment now only says that a reset context can still be found in the cleanup handler, and the subrequest sentence is gone.
The subrequest fix itself is in #394 (r != r->main). I left it out of this PR so the two don't overlap. Until #394 is merged, the write-back stores in sr->ctx the same main transaction the walk already returned on every call, so this PR does not change subrequest behaviour.
| * the server, including locations where modsecurity is off. | ||
| */ | ||
| ngx_pool_cleanup_t *cln; | ||
| for (cln = r->pool->cleanup; cln; cln = cln->next) { |
There was a problem hiding this comment.
Efficiency follow-up: after a successful walk, the recovered pointer is not written back. Every later call for the same request walks the whole cleanup list again: the header filter, each body-filter chain (one per output buffer chain), and the log handler. try_files $uri /index.php is a very common pattern and sets r->internal on nearly every request, so those requests still take the slow path once per chain.
Calling ngx_http_set_ctx(r, ctx, ngx_http_modsecurity_module) once ctx is found would make every later lookup an O(1) array read. Most of the walks this PR is meant to avoid would then disappear, including the redirected case.
There was a problem hiding this comment.
Done in b85a200: get_module_ctx() now calls ngx_http_set_ctx() once the walk finds the context. Later calls for the same request (the header filter, each body filter call, the log handler, the access handler of a redirect target) are then an array read until the next reset.
With the write-back in place I also dropped the r->internal || r->filter_finalize guard (see the lua thread). A request with a transaction now walks once per context reset.
One correction: requests that never had a transaction, such as locations with modsecurity off, have nothing to write back. They still walk on every filter call, as on master. Skipping that walk safely would need a negative marker in our r->ctx slot, one that any reset also clears. I left that out because the walk only covers the cleanups registered on r->pool.
| ############################################################################### | ||
|
|
||
| like(http_get('/missing?what=ok'), qr/FALLBACK/, 'try_files fallback served'); | ||
| like(http_get('/missing?what=bad'), qr/^HTTP.*403/, 'phase 1 deny before try_files'); |
There was a problem hiding this comment.
Minor: this case does not exercise any redirect or context recovery. Rule 91 denies in the access phase at location /, before try_files (precontent phase) runs, so the 403 and the [id "91"] audit check behave the same with or without this change. A phase-1 deny inside the redirect target, for example a /fallback?what=bad reached through try_files, would actually test the recovered context.
There was a problem hiding this comment.
Agreed, the old case tested nothing. The suggested replacement doesn't discriminate either. With try_files $uri /fallback?what=bad, a request for /b/missing?what=ok returned 200 with the fix and with the no-recovery build:
- With recovery, phase 1 does not run again: the access handler runs phase 1 only when it creates the transaction.
- Without recovery, the new transaction still gets the client's
r->unparsed_uri, not the args of the redirect target.
So a phase-1 rule can't see the redirect target's args whether or not the context is recovered.
Done in b85a200: I replaced the case with a phase-1 deny followed by error_page 403 /denied.html. With recovery, the error-page request reuses the transaction (already intervened and logged), and the deny is audited once. Without recovery, the error page gets a second transaction that matches rule 91 again and writes a second audit record. The count is 2 instead of 1, and the test fails.
| SecAuditLogParts ABH | ||
| SecAuditLog %%TESTDIR%%/audit.txt | ||
| SecAuditLogType Serial | ||
| SecAuditLogStorageDir %%TESTDIR%%/ |
There was a problem hiding this comment.
Nit: SecAuditLogStorageDir only applies to SecAuditLogType Concurrent. With Serial it is unused and can be removed.
Review follow-up. The r->internal / r->filter_finalize guard assumed only
nginx core clears r->ctx. lua-nginx-module's ngx.exec("@name") zeroes
r->ctx itself (ngx_http_lua_handle_exec) and then calls
ngx_http_named_location(), which sets r->internal only once it has found
the location. On "could not find named location" or the redirect-cycle
500 the request is finalized with the context gone, internal = 0 and
filter_finalize = 0, so the guard skipped the walk, phase 3 and
msc_process_logging() never ran and the audit record was lost.
Drop the guard: walk whenever the slot is empty, as before this branch.
When the walk finds the transaction, store it back with
ngx_http_set_ctx(), so the following calls for the same request (header
filter, every body filter call, log handler, and the access handler of a
redirect target) are a plain array read again until the next reset. A
request without a transaction still walks on every call, as on master.
Replace the stale realip comment and drop the claim that subrequests
recovering the main transaction is correct; that is fixed separately in
owasp-modsecurity#394.
Test changes:
- prove recovery by counting a logged phase-1 rule in the error log: a
request that lost its context gets a second transaction at the
try_files target, which runs phase 1 again. Request data cannot tell
the two apart: the new transaction is fed r->unparsed_uri, which an
internal redirect does not change, so ARGS (and any TX state derived
from them) are the same in both.
- replace the deny-before-try_files case with a phase-1 deny followed by
an error_page redirect; without recovery the error page request gets
a second transaction and the deny is audited twice.
- drop SecAuditLogStorageDir, which only applies to Concurrent.
With get_module_ctx() reduced to the r->ctx lookup, tests 5-8 fail
(phase-1 count 3/2, phase-3 count 1/2, deny records 2/1, no record for
the 412).
|
Reworked in b85a200 after the review. Code. The Test. Results. Full suite: Scope. This PR is now smaller than it was:
The test is the more useful part: master had no coverage for context recovery after redirects or filter finalization. If you think the write-back alone isn't worth a PR, I'm fine closing this and moving the test elsewhere. I've retitled the PR to "Cache the context recovered from the pool cleanup in get_module_ctx" to match. |
|
|
Follow-up on the review:
I also checked whether the write-back could attach a transaction to the wrong request. It can't. Every lookup goes through No new issues found. CI is green. From my side this looks ready. |



what
ngx_http_modsecurity_get_module_ctx()recovers the transaction by walkingr->pool->cleanup(becauser->ctxwas reset), it now writes the pointer back withngx_http_set_ctx(). Later lookups for the same request are an array read again until the next reset.r->internalorr->filter_finalizewas set. Review showed that third-party modules also clearr->ctx: lua-nginx-module'sngx.exec("@name")zeroes it before callingngx_http_named_location(). If the location is missing, the URI is empty or a redirect cycle is detected, that function finalizes the request without settingr->internal. That guard was dropped.tests/modsecurity-internal-redirect.tcovers atry_filesfallback, a named location, a phase-1 deny followed by anerror_pageredirect, and a failedIf-Matchprecondition (412 throughngx_http_filter_finalize_request()). Each checks that the original transaction is recovered rather than replaced by a second one: phase 1 runs once per request, phase 3 runs, the deny is audited once, and the logging phase runs after filter finalization.why
r->ctxis cleared by internal redirects, named locations, filter finalization and some third-party modules. After a reset, every later lookup for the request (the header filter, each body filter call, the log handler, the access handler of the redirect target) walked the cleanup list again.try_files $uri /index.phptakes this path on most requests.modsecurity offlocations) still walk on every call, as on master.get_module_ctx()reduced to the plainr->ctxread, 4 of the new file's 8 tests fail.references
sizeof(ctx)cleanup allocation inngx_http_modsecurity_create_ctx()alone because fix: correctness gaps in response header/intervention handling #385 already removes it.Origin: this change comes from a performance review of the connector done with Claude Fable 5.1 (Anthropic). The patch and its test were verified by building the module against nginx master with libmodsecurity v3/master, running the full
tests/modsecurity*.tsuite (16 files, 259 tests, all passing), and running the new test against a build without context recovery (4 of 8 fail).Summary by CodeRabbit