Skip to content

[iast] Validate meta_struct IAST payloads in the vulnerability schema test (APPSEC-71866) - #8592

Merged
avara1986 merged 2 commits into
alberto.vara/enable-python-untrusted-deserialization-iastfrom
alberto.vara/iast-schema-validate-meta-struct
Nov 19, 2026
Merged

avara1986 merged 2 commits into
alberto.vara/enable-python-untrusted-deserialization-iastfrom
alberto.vara/iast-schema-validate-meta-struct

Conversation

@avara1986

@avara1986 avara1986 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Motivation

APPSEC-71866: TestIastVulnerabilitySchema only validates meta["_dd.iast.json"]. Since dd-trace-py 3.11.0, Python sends IAST data only in meta_struct["iast"], so the test has not validated a Python payload since then. That is why it did not catch APPSEC-71849: Python reported untrusted deserialization (a sink added in 3.15.0) as UNTRUSTED_SERIALIZATION, a type that is not in the schema.

.NET has the same gap: from 3.7.0, it sends IAST data in meta_struct["iast"] when the agent supports it, which is the case in system-tests.

Stacked on #8585, which adds the Python untrusted deserialization endpoints that produce those findings.

Where each tracer sends the IAST data

The Security events in meta struct RFC moves the IAST data from the _dd.iast.json tag to meta_struct["iast"] when the agent reports span_meta_structs in /info, with _dd.iast.json as the fallback. A tracer sends one or the other, never both. Sources: the RFC status table, the tracers' code, and the declarations of tests/appsec/test_metastruct.py.

Tracer Where IAST data goes Fallback when the agent lacks meta_struct Schema test before this PR
Python ≥ 3.11 meta_struct["iast"] only none skipped
Python 3.8 to 3.10 _dd.iast.json n/a validated
.NET ≥ 3.7.0 meta_struct["iast"] _dd.iast.json skipped
Java _dd.iast.json (its meta_struct tests are missing_feature, APPSEC-4766) n/a validated
Node.js _dd.iast.json (missing_feature) n/a validated
PHP _dd.iast.json (missing_feature) n/a validated
Go, Ruby, C++ no IAST n/a nothing to check

In Python, the choice is _use_metastruct_for_iast = True, hardcoded in ddtrace/internal/settings/asm.py. It is not read from an environment variable and does not depend on the agent; only unit tests override it. In the local runs below, the agent forwards the Python report to the backend as the iast struct attribute: no payload contains _dd.iast.json, neither from the tracer nor from the agent.

Changes

  • TestIastVulnerabilitySchema validates meta_struct["iast"] too. A failure now shows the field, the span and the error messages.
  • manifests/python.yml: the test is declared bug (APPSEC-71849) from 3.15.0 (the first release with the sink) to before 4.18.0rc1, except on aiohttp and tornado. These two weblogs do not report the finding: tornado has no untrusted deserialization endpoint, and aiohttp has no IAST before 4.18. v3.8.0.dev becomes the equivalent missing_feature below 3.8.0-dev.
  • .NET: this PR validates the .NET IAST payloads for the first time. I did not run the .NET weblogs locally; this PR's CI shows whether they match the schema. A mismatch gets a .NET declaration and its own ticket.

Local results (Python)

DEFAULT scenario, all of tests/appsec/iast. I also validated every captured payload against the schema outside the test.

Weblog ddtrace 4.18.0rc1 (main with DataDog/dd-trace-py#21918) ddtrace 4.17.1
flask-poc 99 payloads, all valid. The test passes 3 of 99 invalid (UNTRUSTED_SERIALIZATION). The test xfails with bug (APPSEC-71849)
django-poc 90 payloads, all valid. The test passes 3 of 90 invalid. The test failed before the manifest change
fastapi 87 payloads, all valid. The test passes 3 of 87 invalid. The test failed before the manifest change
aiohttp 87 payloads, all valid. The test passes No IAST payloads. The test passes
tornado 6 payloads, all valid. The test passes 6 payloads, all valid. The test passes

The only invalid Python payloads are the untrusted deserialization findings of the versions before the fix. For django-poc and fastapi on 4.17.1, the manifest evaluates to bug (APPSEC-71849), the same as for flask-poc. The re-run that confirmed the xfail used flask-poc.

Follow-up

The 4.17, 4.16 and 3.19 backports of DataDog/dd-trace-py#21896 and DataDog/dd-trace-py#21918 have not shipped yet. When they ship, the bug range must stop at the first fixed patch version of each line, like the declarations of #8585.

🤖 Generated with Claude Code

…ma test (APPSEC-71866)

TestIastVulnerabilitySchema only read meta["_dd.iast.json"]. Python sends
the IAST data in meta_struct["iast"], so the test never validated a Python
payload and did not catch UNTRUSTED_SERIALIZATION (APPSEC-71849).

Validate meta_struct["iast"] too, and declare the test as a bug for the
Python versions that report UNTRUSTED_SERIALIZATION (3.15.0 up to
4.18.0rc1), except aiohttp and tornado, which do not report it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

manifests/python.yml                                                    @DataDog/system-tests-reviewers
tests/appsec/iast/test_vulnerability_schema.py                          @DataDog/asm-libraries

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Pipelines  Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c285411 | Docs | View more details | Give us feedback!

…st' into alberto.vara/iast-schema-validate-meta-struct
@avara1986
avara1986 marked this pull request as ready for review November 19, 2026 13:53
@avara1986
avara1986 requested review from a team as code owners November 19, 2026 13:53
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T13:55:50.302606Z c285411 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@avara1986 avara1986 changed the title [python] Validate meta_struct IAST payloads in the vulnerability schema test (APPSEC-71866) [iast] Validate meta_struct IAST payloads in the vulnerability schema test (APPSEC-71866) Nov 19, 2026
@avara1986
avara1986 merged commit d79d9ae into alberto.vara/enable-python-untrusted-deserialization-iast Nov 19, 2026
926 of 928 checks passed
@avara1986
avara1986 deleted the alberto.vara/iast-schema-validate-meta-struct branch November 19, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants