Skip to content

Cache the context recovered from the pool cleanup in get_module_ctx - #392

Open
tomsommer wants to merge 3 commits into
owasp-modsecurity:masterfrom
tomsommer:perf/ctx-lookup-fast-path
Open

tomsommer wants to merge 3 commits into
owasp-modsecurity:masterfrom
tomsommer:perf/ctx-lookup-fast-path

Conversation

@tomsommer

@tomsommer tomsommer commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

what

  • When ngx_http_modsecurity_get_module_ctx() recovers the transaction by walking r->pool->cleanup (because r->ctx was reset), it now writes the pointer back with ngx_http_set_ctx(). Later lookups for the same request are an array read again until the next reset.
  • The walk still runs whenever the module's slot is empty, as on master. An earlier revision skipped it unless r->internal or r->filter_finalize was set. Review showed that third-party modules also clear r->ctx: lua-nginx-module's ngx.exec("@name") zeroes it before calling ngx_http_named_location(). If the location is missing, the URI is empty or a redirect cycle is detected, that function finalizes the request without setting r->internal. That guard was dropped.
  • New test tests/modsecurity-internal-redirect.t covers a try_files fallback, a named location, a phase-1 deny followed by an error_page redirect, and a failed If-Match precondition (412 through ngx_http_filter_finalize_request()). Each checks that the original transaction is recovered rather than replaced by a second one: phase 1 runs once per request, phase 3 runs, the deny is audited once, and the logging phase runs after filter finalization.

why

  • r->ctx is cleared by internal redirects, named locations, filter finalization and some third-party modules. After a reset, every later lookup for the request (the header filter, each body filter call, the log handler, the access handler of the redirect target) walked the cleanup list again. try_files $uri /index.php takes this path on most requests.
  • The gain has not been measured and is expected to be modest, since the walk only covers the request pool's cleanups. Requests that never had a transaction (modsecurity off locations) still walk on every call, as on master.
  • Before this PR, nothing tested context recovery. With get_module_ctx() reduced to the plain r->ctx read, 4 of the new file's 8 tests fail.

references


Origin: this change comes from a performance review of the connector done with Claude Fable 5.1 (Anthropic). The patch and its test were verified by building the module against nginx master with libmodsecurity v3/master, running the full tests/modsecurity*.t suite (16 files, 259 tests, all passing), and running the new test against a build without context recovery (4 of 8 fail).

Summary by CodeRabbit

  • Tests
    • Added integration coverage for ModSecurity context recovery across fallback and named-location redirects, error-page redirects after phase 1 denials, and filter finalization after failed preconditions.
    • Verified response statuses, rule counts, audit logging, and precondition logging across these scenarios.

The header and body filters run for every response on the server,
including locations where modsecurity is off, and each call without a
module context walked r->pool->cleanup looking for one.

r->ctx is only discarded by ngx_http_internal_redirect(), which zeroes
it in ngx_http_core_module.c:2652 and sets r->internal at :2663, by
ngx_http_named_location() (:2726, r->internal at :2720) and by
ngx_http_filter_finalize_request(), which zeroes it in
ngx_http_special_response.c:551 and sets r->filter_finalize at :557.
Subrequests get a fresh context array (ngx_http_core_module.c:2471) and
have internal set too (:2577).  The walk is now limited to those
requests, matching ngx_http_realip_get_module_ctx()
(ngx_http_realip_module.c:552), the function this fallback is modelled
on.

The filter finalize case is reachable from an ordinary, non internal
request: the not modified filter calls ngx_http_filter_finalize_request()
with a NULL module on a failed If-Match or If-Unmodified-Since
precondition (ngx_http_not_modified_filter_module.c:67 and :74), which
discards every module context.  tests/modsecurity-internal-redirect.t
covers it along with try_files and named location redirects.
@coderabbitai

coderabbitai Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change updates a comment about the context recovered from a request-pool cleanup handler. It also adds an Nginx integration test for context recovery across internal redirects and filter finalization.

Changes: ModSecurity context recovery

Layer / File(s) Summary
Context description and integration coverage
src/ngx_http_modsecurity_module.c, tests/modsecurity-internal-redirect.t
The comment identifies the recovered context as the ModSecurity transaction. The integration test checks response bodies and status codes, phase 1 and phase 3 rule counts, denial audit output, and logging after a failed precondition.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Refactor

Suggested reviewers: airween

Merge Risk: Minimal · up to b85a2

This change updates a comment and adds integration tests for context recovery across redirects and filter finalization. No runtime behavior change or concrete risk was identified, so it looks safe to merge after normal CI.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: the context recovered from the pool cleanup list is now cached in get_module_ctx. The PR objectives confirm that when the cleanup-list walk finds a transaction, get_mo…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title names the main change: the context recovered from the pool cleanup list is now cached in get_module_ctx. The PR objectives confirm that when the cleanup-list walk finds a transaction, get_module_ctx writes it back with ngx_http_set_ctx(). The title does not mention the removed guard or the new tests, which is acceptable because it need not cover every detail.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3e8a6571-c987-42e5-b1b4-fa2da838a84d
Commits

Reviewing files that changed from the base of the PR and between 2b54ecd and b85a200.

Files selected for processing: 2

  • src/ngx_http_modsecurity_module.c
  • tests/modsecurity-internal-redirect.t

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 1 remain after this review.

@sonarqubecloud

Copy link
Copy Markdown

JakubOnderka added a commit to JakubOnderka/ModSecurity-nginx that referenced this pull request Nov 4, 2026
The header and body filters run for every response on the server,
including locations where modsecurity is off, and each call without a
module context walked r->pool->cleanup looking for one.

r->ctx is only discarded by ngx_http_internal_redirect(), which zeroes
it in ngx_http_core_module.c:2652 and sets r->internal at :2663, by
ngx_http_named_location() (:2726, r->internal at :2720) and by
ngx_http_filter_finalize_request(), which zeroes it in
ngx_http_special_response.c:551 and sets r->filter_finalize at :557.
Subrequests get a fresh context array (ngx_http_core_module.c:2471) and
have internal set too (:2577).  The walk is now limited to those
requests, matching ngx_http_realip_get_module_ctx()
(ngx_http_realip_module.c:552), the function this fallback is modelled
on.

The filter finalize case is reachable from an ordinary, non internal
request: the not modified filter calls ngx_http_filter_finalize_request()
with a NULL module on a failed If-Match or If-Unmodified-Since
precondition (ngx_http_not_modified_filter_module.c:67 and :74), which
discards every module context.  tests/modsecurity-internal-redirect.t
covers it along with try_files and named location redirects.

Author: Tom Sommer
owasp-modsecurity#392
Comment thread src/ngx_http_modsecurity_module.c Outdated
ngx_http_modsecurity_ctx_t *ctx;
ctx = ngx_http_get_module_ctx(r, ngx_http_modsecurity_module);
if (ctx == NULL) {
if (ctx == NULL && (r->internal || r->filter_finalize)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guard assumes nginx core is the only thing that clears r->ctx, but third-party modules do it too. lua-nginx-module's ngx.exec("@name") zeroes r->ctx itself (ngx_http_lua_util.c, "clear the modules contexts") and then calls ngx_http_named_location(). That function only sets r->internal = 1 once it finds the location. If the location is missing, or the request hits the "rewrite or internal redirection cycle" path, it finalizes with 500 and leaves internal at 0 and filter_finalize unset.

In that case the 500 response's header filter and the log handler now get ctx == NULL and skip the walk. Phase 3 is skipped and msc_process_logging() never runs, so the audit record for a request that already went through phases 1 and 2 is lost. Before this change the walk recovered it.

That is an edge case, but the comment's claim that "every other request still has its context" only holds for nginx core.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. In lua-nginx-module, ngx_http_lua_handle_exec() zeroes r->ctx and then calls ngx_http_named_location(). That function sets r->internal = 1 only in the branch where it finds the location. The "could not find named location" and redirect-cycle paths finalize with 500 while internal and filter_finalize are both still 0. The empty-URI path in ngx_http_named_location() behaves the same way. (ngx.exec(uri) is not affected: that branch does not zero r->ctx itself, and ngx_http_internal_redirect() returns on the cycle check before its own ngx_memzero().)

Done in b85a200: I removed the guard. The walk now runs whenever the slot is empty, as on master, and the recovered context is written back (see the efficiency thread). get_module_ctx() no longer depends on who cleared r->ctx. This repo's test build has no lua module, so I could not test the lua path directly.

Comment thread tests/modsecurity-internal-redirect.t Outdated
<$fh>;
};

like($audit, qr/\[id "92"\]/, 'phase 3 ran after internal redirect (ctx recovered)');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These assertions pass whether or not the context is recovered after the redirect. If get_module_ctx() returned NULL at /fallback or @named, the access handler there would just create a new transaction. Rule 92 would match on that new transaction and be audit-logged. The original transaction has no matches, so under RelevantOnly it never writes a record. The log still has exactly two [id "92"] records, so the is(scalar @p3, 2, ...) check passes too.

To actually prove recovery, carry state from before the redirect into phase 3. For example, add SecRule ARGS:what "@streq ok" "id:90,phase:1,pass,nolog,setvar:tx.pre=1" and make rule 92 a chain that also requires TX:pre "@eq 1". A fresh transaction would not have tx.pre set.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the assertions did not prove recovery. The tx.pre chain doesn't prove it for the try_files case either. I checked this with a build where get_module_ctx() only reads r->ctx. The new transaction at /fallback is fed r->unparsed_uri (ngx_http_modsecurity_access_handler() -> msc_process_uri()), and ngx_http_internal_redirect() does not change unparsed_uri. So ARGS:what=ok and tx.pre=1 are set again, and the chained rule 92 still matched. For @named it does discriminate, but only because return runs before the access phase, so no new transaction is created.

Done in b85a200: the test now counts a logged phase-1 rule (id 90) in the error log. A lost context means a second transaction that runs phase 1 again, so the count is 3 instead of 2. With the no-recovery build, tests 5-8 fail:

  • phase-1 count 3, expected 2 (try_files)
  • phase-3 count 1, expected 2 (named location)
  • deny records 2, expected 1 (error_page)
  • no record for the 412 (filter finalization)

All of them pass with the fix.

Comment thread src/ngx_http_modsecurity_module.c Outdated
* r->ctx is only discarded by ngx_http_internal_redirect() and
* ngx_http_named_location(), which set r->internal, and by
* ngx_http_filter_finalize_request(), which sets r->filter_finalize;
* subrequests start with a fresh context array and have internal

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is existing behaviour, but the new comment presents it as intended, so it is worth a closer look. Subrequests share the parent's pool (sr->pool = r->pool, ngx_http_core_module.c:2500), and the access phase is skipped for subrequests (ngx_http_core_access_phase: if (r != r->main)). A subrequest therefore never gets its own transaction. This walk hands it the main request's transaction.

Example: an auth_request subrequest's response headers go through ngx_http_modsecurity_header_filter() with the main transaction. The filter runs phase 3 on the auth backend's headers and status and sets ctx->processed = 1. When the real response comes through later, the filter returns early on processed, so phase 3 never sees the actual response. It may be worth handling r != r->main explicitly here, or at least not describing subrequest recovery as correct.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Subrequests picking up the main transaction is a bug, and the comment should not describe it as intended. Done in b85a200: the comment now only says that a reset context can still be found in the cleanup handler, and the subrequest sentence is gone.

The subrequest fix itself is in #394 (r != r->main). I left it out of this PR so the two don't overlap. Until #394 is merged, the write-back stores in sr->ctx the same main transaction the walk already returned on every call, so this PR does not change subrequest behaviour.

* the server, including locations where modsecurity is off.
*/
ngx_pool_cleanup_t *cln;
for (cln = r->pool->cleanup; cln; cln = cln->next) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Efficiency follow-up: after a successful walk, the recovered pointer is not written back. Every later call for the same request walks the whole cleanup list again: the header filter, each body-filter chain (one per output buffer chain), and the log handler. try_files $uri /index.php is a very common pattern and sets r->internal on nearly every request, so those requests still take the slow path once per chain.

Calling ngx_http_set_ctx(r, ctx, ngx_http_modsecurity_module) once ctx is found would make every later lookup an O(1) array read. Most of the walks this PR is meant to avoid would then disappear, including the redirected case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in b85a200: get_module_ctx() now calls ngx_http_set_ctx() once the walk finds the context. Later calls for the same request (the header filter, each body filter call, the log handler, the access handler of a redirect target) are then an array read until the next reset.

With the write-back in place I also dropped the r->internal || r->filter_finalize guard (see the lua thread). A request with a transaction now walks once per context reset.

One correction: requests that never had a transaction, such as locations with modsecurity off, have nothing to write back. They still walk on every filter call, as on master. Skipping that walk safely would need a negative marker in our r->ctx slot, one that any reset also clears. I left that out because the walk only covers the cleanups registered on r->pool.

Comment thread tests/modsecurity-internal-redirect.t Outdated
###############################################################################

like(http_get('/missing?what=ok'), qr/FALLBACK/, 'try_files fallback served');
like(http_get('/missing?what=bad'), qr/^HTTP.*403/, 'phase 1 deny before try_files');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: this case does not exercise any redirect or context recovery. Rule 91 denies in the access phase at location /, before try_files (precontent phase) runs, so the 403 and the [id "91"] audit check behave the same with or without this change. A phase-1 deny inside the redirect target, for example a /fallback?what=bad reached through try_files, would actually test the recovered context.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the old case tested nothing. The suggested replacement doesn't discriminate either. With try_files $uri /fallback?what=bad, a request for /b/missing?what=ok returned 200 with the fix and with the no-recovery build:

  • With recovery, phase 1 does not run again: the access handler runs phase 1 only when it creates the transaction.
  • Without recovery, the new transaction still gets the client's r->unparsed_uri, not the args of the redirect target.

So a phase-1 rule can't see the redirect target's args whether or not the context is recovered.

Done in b85a200: I replaced the case with a phase-1 deny followed by error_page 403 /denied.html. With recovery, the error-page request reuses the transaction (already intervened and logged), and the deny is audited once. Without recovery, the error page gets a second transaction that matches rule 91 again and writes a second audit record. The count is 2 instead of 1, and the test fails.

Comment thread tests/modsecurity-internal-redirect.t Outdated
SecAuditLogParts ABH
SecAuditLog %%TESTDIR%%/audit.txt
SecAuditLogType Serial
SecAuditLogStorageDir %%TESTDIR%%/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: SecAuditLogStorageDir only applies to SecAuditLogType Concurrent. With Serial it is unused and can be removed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in b85a200: removed.

Tom Sommer added 2 commits November 13, 2026 07:22
Review follow-up. The r->internal / r->filter_finalize guard assumed only
nginx core clears r->ctx. lua-nginx-module's ngx.exec("@name") zeroes
r->ctx itself (ngx_http_lua_handle_exec) and then calls
ngx_http_named_location(), which sets r->internal only once it has found
the location. On "could not find named location" or the redirect-cycle
500 the request is finalized with the context gone, internal = 0 and
filter_finalize = 0, so the guard skipped the walk, phase 3 and
msc_process_logging() never ran and the audit record was lost.

Drop the guard: walk whenever the slot is empty, as before this branch.
When the walk finds the transaction, store it back with
ngx_http_set_ctx(), so the following calls for the same request (header
filter, every body filter call, log handler, and the access handler of a
redirect target) are a plain array read again until the next reset. A
request without a transaction still walks on every call, as on master.

Replace the stale realip comment and drop the claim that subrequests
recovering the main transaction is correct; that is fixed separately in
owasp-modsecurity#394.

Test changes:
- prove recovery by counting a logged phase-1 rule in the error log: a
  request that lost its context gets a second transaction at the
  try_files target, which runs phase 1 again. Request data cannot tell
  the two apart: the new transaction is fed r->unparsed_uri, which an
  internal redirect does not change, so ARGS (and any TX state derived
  from them) are the same in both.
- replace the deny-before-try_files case with a phase-1 deny followed by
  an error_page redirect; without recovery the error page request gets
  a second transaction and the deny is audited twice.
- drop SecAuditLogStorageDir, which only applies to Concurrent.

With get_module_ctx() reduced to the r->ctx lookup, tests 5-8 fail
(phase-1 count 3/2, phase-3 count 1/2, deny records 2/1, no record for
the 412).
@tomsommer

Copy link
Copy Markdown
Contributor Author

Reworked in b85a200 after the review.

Code. The r->internal || r->filter_finalize guard is gone. It assumed only nginx core clears r->ctx, which isn't true: if lua-nginx-module's ngx.exec("@name") fails to redirect, the audit record is lost. get_module_ctx() now walks the pool cleanup list whenever the slot is empty, as on master. When the walk finds the transaction, the context is written back with ngx_http_set_ctx(), so the rest of the request is O(1) until the next reset. The comment no longer presents subrequest recovery as correct; that fix is in #394.

Test. tests/modsecurity-internal-redirect.t now proves that the context is recovered. It counts phase-1 executions in the error log, because a lost context means a second transaction. It also covers a phase-1 deny followed by an error_page redirect, and keeps the 412 filter-finalization case. With get_module_ctx() reduced to the plain r->ctx read, 4 of the 8 tests fail. All pass with the fix.

Results. Full suite: Files=16, Tests=259, all tests successful (nginx master, libmodsecurity v3/master).

Scope. This PR is now smaller than it was:

  • What's left is the write-back. It removes repeated walks for requests whose context was reset, such as try_files ... /index.php, where every body filter call used to walk the list again.
  • Requests with no transaction at all (modsecurity off locations) still walk on every filter call, exactly as on master. That was the original motivation, and I don't have a correct and simple way to skip it.
  • I have not measured the gain. The walk covers only r->pool's own cleanups, so I expect it to be small.

The test is the more useful part: master had no coverage for context recovery after redirects or filter finalization. If you think the write-back alone isn't worth a PR, I'm fine closing this and moving the test elsewhere. I've retitled the PR to "Cache the context recovered from the pool cleanup in get_module_ctx" to match.

@tomsommer tomsommer changed the title Skip the pool cleanup walk in get_module_ctx for ordinary requests Cache the context recovered from the pool cleanup in get_module_ctx Nov 13, 2026
@sonarqubecloud

Copy link
Copy Markdown

@fzipi

fzipi commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Follow-up on the review:

  • Guard assumed only nginx core clears r->ctx (lua ngx.exec("@name") case) → fixed. The guard is gone and the walk runs whenever the slot is empty, as on master.
  • Test assertions passed without recovery → fixed. Thanks for checking the tx.pre suggestion: you're right that it doesn't discriminate for try_files, because the new transaction is fed the unchanged r->unparsed_uri. Counting the logged phase-1 rule (id 90) is a better signal.
  • Comment described subrequests picking up the main transaction as intended → fixed. The comment is neutral now, and the behaviour fix is left to Do not run the connector on subrequests #394.
  • Write back the recovered ctx → fixed with ngx_http_set_ctx() after the walk. Your correction about modsecurity off locations still walking every time is right.
  • Phase-1 deny case didn't exercise a redirect → fixed with the error_page 403 case. You're also right that my /fallback?what=bad suggestion wouldn't have discriminated.
  • Unused SecAuditLogStorageDir → fixed.

I also checked whether the write-back could attach a transaction to the wrong request. It can't. Every lookup goes through ngx_http_modsecurity_get_module_ctx(), so the cached value is always the pointer the walk would have returned anyway. create_ctx() only runs when that lookup returns NULL, so a pool never gets a second transaction that a cached pointer could hide. After an internal redirect, master already reused the transaction through the walk and never created a new one. For subrequests (sr->pool = r->pool, fresh sr->ctx), the cached value is the main transaction that the walk returned before, so behaviour doesn't change. With #394's r != r->main early return in place, the write-back is never reached for subrequests, so the two PRs combine cleanly in either merge order.

No new issues found. CI is green. From my side this looks ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants