[dmsolr]

What this PR does

The shenyu-plugin-ai-pii protocol profiles only extracted the plain text
member of a content part, so text nested in structured parts was never scanned
and REDACT / BLOCK rules silently let PII through. This PR extends the
chat / responses extractors to cover the missed cases.

• document content parts: extract source.data when source.type == "text"
  (a base64 document source is binary and stays out of scope).
• file content parts: add PiiTextSlot.ofBase64TextData, which decodes a
  textual base64 file_data (text/* or application/json) for scanning and
  re-encodes it after redaction, keeping the data: URL header. Binary media
  types (image / audio) and non textual file_data yield no slot.
• Tool call arguments: extract tool_calls[].function.arguments, the legacy
  function_call.arguments (chat) and the arguments of a responses
  function_call item as text slots.
• Extraction is shared between the two profiles via a new PiiContentParts
  helper; the existing overlapping-range merge in PiiScanner is reused, so the
  same text reached via multiple paths yields a single merged redaction.

This changes the previously documented design ("a redaction never touches ...
function arguments"), so the PiiProtocolProfile javadoc, the plugin README and
the affected tests are updated accordingly. The broader "non chat/responses
shapes" item (#4 in the issue, a design topic) is intentionally out of scope.

Related Issue

Fixes #8112

Type of change

• Bug fix
• New feature / enhancement
• Refactor (no functional change)
• Documentation
• Chore / build

How was this tested

• Unit tests added / updated (PiiProtocolProfilesTest, PiiResponseDecoratorTest)
• ./mvnw test -pl shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-pii passed (82 tests)
• ./mvnw checkstyle:check passed (0 violations) and apache-rat:check clean
• New regression tests cover document text, textual/binary file_data,
  chat tool_calls and responses function_call arguments

Checklist

• Apache 2.0 license header added to new source files
• README updated for the changed extraction scope
• No breaking changes to public API / config / DB schema

[Aias00]

This closes the gap I reported in #8112, and the approach is right: rather than widening the profiles ad hoc, extraction moved into PiiContentParts and both profiles share it, with the existing overlapping-range merge in PiiScanner doing the dedup. Confirmed correct:

• collectDocumentSlots only takes source.data when source.type == "text", so a base64 document source is genuinely out of scope rather than accidentally decoded.
• isTextualMediaType restricts file_data to text/* and application/json, which excludes image and audio parts.
• dataUrlPrefix requires the ;base64 suffix before the comma, so a plain data:text/plain,hello is not treated as base64.
• PiiScanner.scan already skips a null or empty text() (line 70), so a slot that cannot be read does not NPE downstream.
• README, PiiProtocolProfile javadoc and the renamed testToolCallArgumentsAreRedactedInResponse all track the behaviour change, instead of leaving the old "a redaction never touches function arguments" sentence behind.

One item I would like fixed before merge, and four smaller ones.

1. Base64.getDecoder() silently drops a large class of real payloads (should fix)

decodeDataUrl uses Base64.getDecoder(), which is the strict RFC 4648 decoder. It rejects:

• base64 with line breaks — the MIME-style wrapping that many clients and libraries emit, especially for anything a user pasted or that was chunked at 76 characters
• the URL-safe alphabet (- and _) instead of + and /

Either one throws IllegalArgumentException, which you catch and turn into null, and PiiScanner then treats the slot as empty and continues. So a file_data of data:text/plain;base64,<wrapped base64 containing a phone number> is passed through untouched — even under a fail-closed rule, because nothing about the outcome is treated as a failure. That is the same shape of defect this PR exists to fix: PII leaves unredacted and nothing is logged.

Suggested fix: try Base64.getMimeDecoder() first (it tolerates line breaks and skips other non-alphabet characters), fall back to Base64.getUrlDecoder(), and only then give up. Re-encoding with getEncoder() in replace() is fine for the output side.

2. text() re-decodes on every call, with no size bound

text() calls decodeDataUrl(parent.get(member)) each time it is invoked. PiiScanner calls it at least once per slot, and redact reads the text again. A multi-megabyte file_data is therefore base64-decoded repeatedly per request, allocating a fresh String each time.

The per-direction deadline does not help here: it is enforced by the deadline-aware CharSequence that the detectors walk, but decoding happens before any CharSequence exists. A large enough payload spends its time and memory in decodeDataUrl with nothing bounding it.

Two things worth doing: cache the decoded text per slot for the lifetime of one scan, and cap the payload length you are willing to decode, treating an over-limit payload as an explicit fail-closed rejection (or at minimum a warning) rather than a silent skip.

3. replace() normalises the base64 encoding

Re-encoding with Base64.getEncoder() emits padded, unwrapped base64. If the original was MIME-wrapped or URL-safe, the member changes encoding style after a redaction. It stays a valid data URL, so this is cosmetic rather than a correctness bug, but it does mean the "everything outside the redacted range is preserved" property does not hold for this member. Worth a sentence in the javadoc so it is a documented tradeoff rather than a surprise.

4. OpenAiResponsesProfile reads arguments from any item

PiiTextSlot arguments = PiiTextSlot.ofMember(object, "arguments");

This is unconditional in collectItemSlots, while document and file parts are both gated on their type. Any responses item that happens to carry a string arguments member gets scanned as message text. Today only function_call has one, so the practical risk is low, but gating on "function_call".equals(typeOf(object)) would make the rule explicit and keep the three extractions consistent with each other.

5. Minor

addIfPresent is called for the plain text member before the type dispatch, so a document part that also carries a text member yields two slots. Harmless given the overlap merge, just noting it is intentional rather than an oversight.

The test coverage is good — document text, textual and binary file_data, chat tool_calls, responses function_call, and the textless-message regression are all pinned. The testStrictToolCallOnlyResponseRemainsUnchanged → testToolCallArgumentsAreRedactedInResponse rewrite is the right call: keeping the old name while asserting the opposite would have been confusing.

Item 1 is the only one I would hold the merge on; the rest can be follow-ups if you prefer.

[dmsolr]

Thanks for the thorough review @Aias00! All five items are addressed in 3eca7cf:

1. Robust base64 decoding (blocker) — Decoding moved into a new Base64TextData helper: payloads are decoded with Base64.getMimeDecoder() (tolerates line breaks and skips other non-alphabet characters), and when the payload contains -/_ the URL-safe decoder is used instead (whitespace stripped first, since the URL-safe decoder is strict about separators). Only when both fail is the payload treated as undecodable. Tests cover a MIME-wrapped payload with CRLF breaks and a URL-safe payload.

2. Decode cost & silent skips — Two changes:

• The decoded text is cached per slot (Base64TextDataSlot), so a payload is decoded at most once per scan.
• A textual file_data that cannot be decoded is now rejected by shape validation (PiiProtocolProfiles.validFilePart), so select() returns null and the payload follows the existing unsupported-mode path: fail-closed rejection, or pass-through with a warning under fail-open — no more silent skip.

On the length cap: I deliberately did not add a separate decode cap. The whole request body is already bounded by the rule's maxBodySize (enforced against the actual UTF-8 byte length before parsing), so any base64 payload this code sees is at most maxBodySize bytes. A second hardcoded cap would either be redundant or conflict with the operator-configured bound. The caching above removes the repeated-decode cost, which was the practical concern.

3. Encoding normalisation on replace — Documented in the ofBase64TextData javadoc: re-encoding uses the standard padded alphabet, so a MIME-wrapped or URL-safe original is normalised; this is called out as a documented tradeoff, not corruption.

4. Arguments gating — Responses extraction now gates arguments on type == "function_call" via PiiContentParts.collectFunctionCallArguments; a stray arguments member on another item type is no longer mistaken for message text (test added).

5. text + document double slot — Comment added in PiiContentParts.collectPartSlots clarifying this is intentional: the overlapping-range merge in PiiScanner collapses the two slots into a single redaction.

Verification: ./mvnw test -pl shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-pii — 86 tests, 0 failures; checkstyle and RAT clean.

[Aias00]

Found a JSON-escaping gap in the new tool-argument scanning path. A targeted PiiScanner regression test, testEscapedEmailInToolArgumentsIsRedacted, confirms that a valid JSON argument value decoding to alice@example.com passes the EMAIL scanner unchanged when @ is encoded as a Unicode escape.

[Aias00]

[P2] Scan decoded JSON values inside function arguments

The arguments member is itself serialized JSON. Treating it as an ordinary text slot scans the JSON representation rather than its decoded values, so valid escaping hides PII: arguments containing {"email":"alice\u0040example.com"} decode to alice@example.com when the tool parses them, but the EMAIL detector sees no @ and forwards the value unchanged. The Chat tool_calls and legacy function_call paths above have the same issue.

Please expose decoded string values from the nested argument JSON as slots and serialize the modified argument object back, with explicit handling for malformed arguments. I reproduced this through OpenAiResponsesProfile.responseSlots() and PiiScanner with EMAIL enabled; testEscapedEmailInToolArgumentsIsRedacted fails because outcome.isChanged() is false.