Skip to content

Implement Phase 4 handling for ModSecurity-nginx - #377

Open
Easton97-Jens wants to merge 50 commits into
owasp-modsecurity:masterfrom
Easton97-Jens:master-phase4
Open

Easton97-Jens wants to merge 50 commits into
owasp-modsecurity:masterfrom
Easton97-Jens:master-phase4

Conversation

@Easton97-Jens

@Easton97-Jens Easton97-Jens commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

What changed

This PR introduces and stabilizes dedicated phase:4 handling for the nginx ModSecurity connector. It also adapts the relevant native-result and lifecycle corrections from ModSecurity-conector d420a12, preserving this standalone connector and its JSON event schema.

Main additions include:

  • Support for configurable phase4 handling modes:
    • off (default)
    • safe
    • strict
  • Explicit late-intervention handling when response headers were already sent
  • Dedicated phase4 JSON logging via:
    • modsecurity_phase4_log
  • Response MIME selection and inspection limits controlled by libModSecurity through:
    • SecResponseBodyAccess
    • SecResponseBodyMimeType and SecResponseBodyMimeTypesClear
    • SecResponseBodyLimit and SecResponseBodyLimitAction
  • Additional parser, logging, HTTP/2, proxy, and regression test coverage
  • Production-oriented example configurations and English documentation

Additional fixes included in this PR:

  • Removed the redundant minimal mode and the connector MIME-file directive/extra MIME gate from earlier revisions of this PR
  • Kept modsecurity_phase4_body_limit as a deprecated, ignored compatibility setting, with positive-value validation; inspection limits belong to the engine
  • Fixed phase4 log inheritance across http, server, and location, with a server-to-location regression test
  • Added bounded 32 KiB file reads with per-chunk intervention collection; cumulative counters saturate without imposing a SIZE_MAX delivery limit
  • Corrected response completion, main/subrequest ownership, response-header failures, redirect cleanup, and internal redirect recovery
  • Added the missing configuration NULL check in event logging, corrected NGINX status formatting, and replaced locale-dependent content-type whitespace checks with fixed ASCII checks
  • Improved JSON escaping, redacted engine intervention messages, and detected event-buffer allocation/log-write failures
  • Moved phase4-unit-tests from test.yml to test_new.yml and disabled credential persistence for all ten checkouts in the new workflow
  • Made response failures terminal across repeated data/NULL callbacks, accepted valid empty EOS buffers, and retained main/subrequest ownership
  • Validated native return contracts, ignored nondisruptive results, released owned intervention outputs on every collected outcome, and propagated early logging failures
  • Reused request transactions for identical inherited policy, retained every changed-policy owner's audit attempt, and allowed only the captured native-block error_page target to render without reblocking
  • Streamed request temp-file descriptors through the same native append contract as memory, preserving ProcessPartial zero and queued Reject through P2 completion
  • Corrected failed context construction/cleanup and NUL-terminated custom transaction IDs before calling the native SDK
  • Integrated the current upstream README/security policy while retaining the Phase 4 directives and previous workflow fixes
  • Applied fzipi's review corrections: late Safe survives optional event-log failure; replacement bodies emit writer-safe controls once; response-limit Reject is recognized at a valid append boundary without English log-text matching
  • Parsed complete, unambiguous native metadata records for rule IDs and redacted summaries; preserved JSON controls and valid UTF-8 while escaping invalid bytes
  • Suppressed synthetic Transfer-Encoding inspection headers on HTTP/2/HTTP/3 and checked response status before native integer conversion
  • Bundled the versioned NGINX HTTP/2 close-stream correction, with an explicit pre-build helper, repeat-application checks and Linux/Windows CI integration

Why these changes

phase:4 operates on response bodies. At this stage, nginx may already have sent response headers or even part of the response body.

Because of this technical limitation, actions such as:

  • deny
  • status
  • redirect

cannot always be converted into a clean client-visible HTTP response once headers are already sent. Bytes already delivered to the client cannot be recalled.

This PR introduces explicit handling for those late-intervention scenarios:

  • off
    → default; disables the additional safe/strict connector policy while leaving native ModSecurity response inspection and interventions active
  • safe
    → opt-in; attempts log_only logging and continues after headers are sent, diagnosing optional event-log failures
  • strict
    → opt-in; records connection_abort and terminates the response after headers are sent

Before headers are committed, interventions can use the normal status/redirect path. Genuine engine, buffer, body-processing allocation, file-read and native audit errors remain failures, including in safe. Cumulative accounting saturates without cutting large responses. After commitment, optional JSON event-log failures are diagnosed without aborting Safe delivery; the existing pre-commit error path remains. Native response-limit Reject is distinct from a rule intervention: it produces 403 before headers are sent, or fails the committed transport, in all modes. ProcessPartial still allows full forwarding of the original response. Append-time unknown disruptive results cannot become Safe log-only events. Neither safe nor strict can guarantee a new client-visible status after headers are sent.

The goal is to:

  • avoid false guarantees around late blocking behavior,
  • make the additional late-intervention policy explicitly opt-in,
  • improve operational visibility,
  • keep MIME selection and inspection limits in the engine,
  • clearly document the limits of production and test/demo behavior.

On HTTP/2, a late Strict decision resets the affected stream while unrelated streams and the physical connection remain usable. The bundled host-core patch schedules queued reset delivery even while another stream remains open. Apply it before rebuilding NGINX; dynamic-module installations require the patched host binary as well. Replacing only the connector module cannot correct an existing host.

One compatibility point remains open: off does not reproduce every pre-PR intervention control-flow decision. Native failures remain terminal, and late native intervention handling fails the committed transport without trying to emit a second response header. This PR does not claim exact legacy callback/return-value equivalence.


Logging and observability

The new phase4 logging implementation provides optional structured JSON-lines logging including fields such as:

  • intervention mode
  • requested vs actual action
  • header-sent state
  • response status and WAF status
  • content type
  • rule ID
  • reason
  • a redacted intervention summary

JSON controls are escaped without losing their values; valid UTF-8 is preserved and invalid bytes are explicitly escaped. Rule IDs and presence summaries come only from complete, unambiguous engine metadata, so reflected delimiter-like content cannot override them. Raw engine intervention messages are redacted. Tests also verify that response-body payload data is not leaked into dedicated phase4 events. These events are not a complete inventory of engine errors or a replacement for nginx/ModSecurity logging.

The intervention field is still retained alongside rule_id; the review suggestion to rename or remove it has not been adopted. Late completed-P4 Safe/Strict decisions do not invoke native status replacement or early audit logging. Native limit errors use an nginx error diagnostic rather than a misleading rule-policy event.


Testing

This PR adds and extends coverage for:

  • parser validation and mode behavior
  • regression handling and server-to-location log inheritance
  • proxy paths and HTTP/2 fixtures
  • late-intervention scenarios
  • engine-controlled MIME selection and Reject/ProcessPartial response limits
  • large responses, subrequests, redirects, and processing/error paths
  • memory/file-backed buffers, overflow, file-read failures, sticky filter failure, empty EOS, native return contracts, and transaction ownership in standalone C helper tests
  • JSON logging, UTF-8/control-byte handling, redaction, metadata ambiguity and reflected-ID regressions
  • inherited-policy transaction reuse, source/changed/disabled-target audit ownership, custom block pages and file/memory ProcessPartial parity
  • writer-safe replacement controls and HTTP/2/HTTP/3 synthetic-header guards
  • prompt HTTP/2 Strict resets after an asynchronous auth gate, passive client observation, sibling completion and connection reuse; one Linux job first verifies the stalled unpatched baseline before applying the fix
  • actual patch-helper execution across LF/CRLF source and patch combinations, idempotency, source-drift rejection and paths with spaces

The standalone helper tests run with GCC and Clang as phase4-unit-tests in .github/workflows/test_new.yml, rather than the legacy workflow. All ten checkouts there use persist-credentials: false. The new native matrix covers nginx 1.29.x/1.30.x/1.31.x on Ubuntu 24.04/26.04 and Windows 2022. The duplicate legacy Windows job is removed; the legacy Linux jobs remain.

Validation for head 7a5a336:

  • All 143 standalone tests pass locally with GCC and Clang, including eight actual patch-helper cases. Both helper CI jobs also pass all 143 tests at this head.
  • All 14 native Linux jobs pass: both legacy GCC/Clang jobs and all twelve cells in the new Linux matrix. The normal native suite reports 23 Perl suites / 385 assertions.
  • The controlled HTTP/2 baseline and rebuild job passes all 23 unpatched checks, applies the host patch, rebuilds NGINX, and passes the normal 385-assertion suite. This proves both the original queued-reset behavior and prompt patched delivery without new client input.
  • The three native Windows build/test cells for NGINX 1.29.x/1.30.x/1.31.x are included in the same matrix; their current results are shown by the linked checks. The zlib download uses the official release asset for the unchanged pinned version.
  • SonarQube Cloud passes with zero new issues. CodeRabbit is paused/skipped, so this is not presented as a new CodeRabbit review.

Optional unsafe proxy/HTTP/2 assertions require their test environment; this does not claim that every optional protocol assertion ran in those checks.


Documentation

Comprehensive documentation for the implemented behavior was added and updated, including:

  • operational limitations of phase:4
  • behavior differences between off, safe, and strict
  • configuration examples for each mode and engine-controlled MIME selection
  • logging semantics, inheritance, and redaction
  • known technical boundaries and the remaining off compatibility difference
  • a detailed Phase 4 entry in CHANGES and the corrected README count of nine directives
  • provenance and application of the NGINX host-core patch, including the rebuild requirement for dynamic-module installations

The English documentation is the maintained source. The German translation is removed.

Summary by CodeRabbit

  • New Features

    • Added configurable Phase 4 response handling with off, safe, and strict modes, plus optional JSON event logging.
    • Added configuration examples for each mode and for selecting response content types through ModSecurity rules.
  • Behavior Changes

    • Phase 4 defaults to off; this disables the additional safe/strict connector policy while leaving native response inspection active. Full pre-PR intervention control-flow compatibility remains open. The minimal mode and content-types-file directive are no longer supported.
    • Interventions before headers are sent can apply their status. Afterward, safe continues rule-only delivery even if optional event logging fails (with a diagnostic), while strict terminates the response. Native response-limit Reject remains a failure in every mode.
    • ModSecurity rules control inspection limits and MIME selection. The deprecated body-limit setting remains accepted with positive-value validation but no longer limits inspection.
  • Documentation

    • Updated English guidance on configuration, logging, engine policy, transaction/audit ownership and response-body handling; retained the upstream security policy, removed the German translation and expanded CHANGES.

…rvention-handling-in-modsecurity-nginx

Phase 4: configurable handling, content-type scoping and JSON logging
…igs-and-document-phase-4-functionality

docs: add bilingual Phase-4 handling docs and example configurations
…n-modsecurity-proxy

Guard allocations, tighten phase4 content-type parsing, and update tests
…nability-issues-in-modsecurity-nginx

Reduce nesting and split declarations to address SonarQube maintainability issues
…in-src/ngx_http_modsecurity_body_filter

Fix SonarQube issues in body filter (split declarations, reduce nesting)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds configurable Phase 4 response-body intervention handling for the ModSecurity nginx connector, including safer late-intervention behavior, scoped content-type handling, structured logging, and related tests/docs.

Changes:

  • Adds minimal, safe, and strict Phase 4 modes with late-intervention downgrade/abort behavior.
  • Adds Phase 4 JSON logging and content-type scoping configuration.
  • Expands regression, proxy, HTTP/2, parser, documentation, and example coverage.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
src/ngx_http_modsecurity_module.c Adds new Phase 4 directives, config parsing, defaults, and intervention metadata capture.
src/ngx_http_modsecurity_common.h Extends request/location config structs for Phase 4 state.
src/ngx_http_modsecurity_body_filter.c Implements Phase 4 scope checks, late-intervention handling, and JSON logging.
tests/modsecurity.t Enables strict mode for existing Phase 4 baseline tests.
tests/modsecurity-h2.t Enables strict mode for HTTP/2 Phase 4 tests.
tests/modsecurity-proxy.t Enables strict mode and adds in-scope proxy backend responses for Phase 4 triggers.
tests/modsecurity-proxy-h2.t Enables strict mode for HTTP/2 proxy Phase 4 tests.
tests/modsecurity-phase4-regression.t Adds large-response minimal-mode regression/logging coverage.
tests/modsecurity-phase4-modes.t Adds mode behavior and JSON-line logging tests.
tests/modsecurity-phase4-invalid-config.t Adds intended invalid content-type config coverage.
tests/modsecurity-phase4-content-types.t Adds content-type scoping behavior tests.
docs/phase4-handling.en.md Documents English Phase 4 behavior, modes, logging, and limits.
docs/phase4-handling.de.md Adds German Phase 4 behavior documentation.
docs/examples/phase4-minimal.conf Adds minimal-mode production example.
docs/examples/phase4-safe.conf Adds safe-mode production example.
docs/examples/phase4-strict.conf Adds strict-mode production example.
docs/examples/phase4-content-types.conf Adds example scoped content-type list.
Comments suppressed due to low confidence (1)

src/ngx_http_modsecurity_module.c:659

  • Read failures are also returned as a generic configuration error without the file path or errno. Please log the read error before returning so operators can distinguish truncated/unreadable content-type files from parser validation failures.
    n = ngx_read_file(&file, buf, ngx_file_size(&fi), 0);
    ngx_close_file(file.fd);
    if (n < 0) return NGX_CONF_ERROR;

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

}
location /phase4 {
modsecurity on;
modsecurity_phase4_mode strict;
Comment thread tests/modsecurity-phase4-modes.t Resolved
Comment thread src/ngx_http_modsecurity_body_filter.c Resolved
Comment thread src/ngx_http_modsecurity_module.c Resolved
Comment thread src/ngx_http_modsecurity_body_filter.c Resolved
Comment thread src/ngx_http_modsecurity_common.h Resolved
Comment thread src/ngx_http_modsecurity_module.c Resolved
Comment thread src/ngx_http_modsecurity_body_filter.c Resolved
Comment thread src/ngx_http_modsecurity_module.c Resolved
@airween

airween commented May 14, 2026

Copy link
Copy Markdown
Member

Excellent PR, thanks @Easton97-Jens!

Please take a look at Copilot's suggestions.

I think it would be great to add some minimal description about the new directives to main README.

Other than that, I would like to test this PR locally, and play with it for a wile - I need some time for that.

Thanks again.

@Easton97-Jens

Copy link
Copy Markdown
Contributor Author

Hi @airween,

I've pushed updates following your review. The current head is 468ebd2.

The following changes are implemented:

  • off is the default, and the additional safe/strict policy is opt-in. The redundant minimal mode is removed.
  • The connector MIME-file directive and additional MIME gate are removed. Response MIME selection and inspection limits remain owned by libModSecurity.
  • Phase 4 log inheritance is fixed across http, server and location, with a server-to-location regression test.
  • The logging path checks mcf for NULL. Logged content-type normalization uses fixed ASCII whitespace checks, and the WAF status uses the correct NGINX integer type for formatting.
  • CHANGES now has a detailed Phase 4 entry, the README directive count is corrected to nine, the German translation is removed, and the duplicated MIME assertion is cleaned up.
  • The GCC/Clang helper tests now run in test_new.yml. At the latest code commit (85604d5, before the documentation-only commits), all 52 helper tests pass with each compiler, and native Linux GCC/Clang runs pass 21 suites / 352 assertions.

Two points still need to be settled:

  • off does not yet reproduce every pre-PR intervention control-flow decision: negative mid-chain and positive final-body results are handled differently. I am not claiming full legacy compatibility yet.
  • The JSON intervention field is still retained as a redacted summary alongside rule_id; I have not renamed or removed it.

The current behavior is described in the updated English documentation and diff; the PR description still awaits replacement.

Could you take another look at the updated diff, especially the remaining off compatibility and your preferred treatment of the intervention field? Thanks for the review.

@Easton97-Jens

Copy link
Copy Markdown
Contributor Author

Hi @airween,

I've updated the PR following your review and integrated the relevant NGINX corrections from ModSecurity-conector. The current head is 776de3a, including the latest upstream merge and resolved workflow conflicts.

Implemented changes include:

  • off is the default; the additional safe/strict policy is opt-in. minimal and the separate connector MIME-file gate are removed. The engine owns MIME selection and inspection limits.
  • Log inheritance is fixed, with a server-to-location regression. The missing configuration check, ASCII whitespace normalization, correct NGINX status formatting, directive count, duplicated assertion, expanded CHANGES, and removal of the German translation are addressed.
  • Native return values and nondisruptive outputs are validated. Native response-limit Reject remains a failure in all modes, including Safe; it cannot become a rule-policy log-only event. ProcessPartial remains supported.
  • Empty EOS and repeated filter/error callbacks are handled explicitly. File-backed responses collect interventions after each chunk, retaining the existing request ownership and outgoing chain rules.
  • Request decisions survive error_page routing, internal target locations use their own transaction, native audit logging is attempted once, and partial context creation/custom transaction IDs are handled safely.
  • phase4-unit-tests runs in test_new.yml, with credential persistence disabled on all ten checkouts there. GCC and Clang each pass all 85 helper tests, all twelve new-workflow Linux matrix jobs and both legacy Linux jobs pass, and SonarCloud reports zero new issues. Windows 1.29 and 1.30 also pass; Windows 1.31 remains in progress in the current workflow.

Two review points remain explicit: off does not claim exact pre-PR callback/return-value compatibility, and the redacted JSON intervention summary is still retained alongside rule_id.

Could you take another look at the current diff and the updated English documentation, particularly those remaining design points? Thank you for reviewing it.

Comment thread src/ngx_http_modsecurity_access.c Resolved
Comment thread src/ngx_http_modsecurity_access.c Resolved
Comment thread src/ngx_http_modsecurity_access.c Resolved
Comment thread src/ngx_http_modsecurity_body_filter.c Resolved
Comment thread src/ngx_http_modsecurity_body_filter.c Resolved
Comment thread src/ngx_http_modsecurity_body_filter.c Resolved
Comment thread src/ngx_http_modsecurity_body_filter.c Resolved
Comment thread src/ngx_http_modsecurity_body_filter.c Resolved
Comment thread src/ngx_http_modsecurity_module.c Resolved
Comment thread src/ngx_http_modsecurity_body_filter.c Resolved
@Easton97-Jens
Easton97-Jens requested a review from fzipi November 15, 2026 00:09
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants