Skip to content

AI Guard: validate MCP server IDs across host vectors, shared IDs and reconnects - #8599

Open
avara1986 wants to merge 4 commits into
mainfrom
alberto.vara/APPSEC-71867-mcp-server-id-tests
Open

avara1986 wants to merge 4 commits into
mainfrom
alberto.vara/APPSEC-71867-mcp-server-id-tests

Conversation

@avara1986

@avara1986 avara1986 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Motivation

Jira: APPSEC-71867

An AI Guard MCP server ID is mcp_ followed by the first 16 hex characters of the SHA-256 of a single identity key: the URL origin, then the connector ID, then the stdio command, then the configured name. Every tracer must compute the same ID for the same server. The existing MCP tests already cover the url, connector and stdio keys, URL sanitization, ID resolution and the kill switch. These tests add the rest of the contract.

Changes

  • test_ai_guard_openai_mcp_server_id_host_vectors (OpenAI hosted MCP): each server_url host is canonicalized with the WHATWG URL rules before the ID is computed. The shared vectors are a numeric IPv4, an IPv4 with port, an IPv6, a percent-encoded host and an underscore host, and each one gets its exact expected ID. An IDN host has no url and falls back to the name key.
  • test_ai_guard_openai_mcp_servers_sharing_an_id_merge (OpenAI hosted MCP): two servers on the same origin with different paths share an ID. They are reported as one mcp_servers entry with the first server's fields and the tools of both.
  • test_ai_guard_anthropic_mcp_reconnect_same_server_id (Anthropic stdio MCP): two requests, each starting a new stdio server process and MCP session, report the same server entry and ID.
  • test_ai_guard_anthropic_mcp_command_path (Anthropic stdio MCP): the weblog /mcp/create takes an optional command_path flag that starts the echo server from the absolute path of the interpreter. Only the program name (python) may reach AI Guard, so the server ID stays the same. It is declared missing_feature for Python until chore(aiguard): merge MCP servers sharing an ID and report stdio commands by program name dd-trace-py#21929 ships.
  • The expected IDs are written out in the tests as shared vectors, not derived from a tracer.
  • OpenAI cassettes: generated by utils/scripts/gen_aiguard_mcp_cassettes.py. The new scenarios are appended last, so no existing cassette changes.
  • Anthropic cassettes: the new tests reuse copies of the recorded test_ai_guard_anthropic_mcp_tool_use cassettes, renamed to their own test names. Cassette names hash only the request body, so the copies match unchanged. Identifying response headers and IDs were scrubbed from the copies.

No manifest changes. The new tests live in files that already have entries:

  • Python: enabled from v4.18.0-rc1. The WHATWG host canonicalization is not in 4.17.x.
  • Node.js: missing_feature for OpenAI and incomplete_test_app for Anthropic (the weblog has no /mcp/create endpoint).
  • Java: missing_feature.

When Node.js gets MCP support it will need OpenAI cassettes of its own, because the AI Guard meta it sends is part of the hashed request body.

Validation, run locally against the dd-trace-py main wheel (python@4.18.0-rc1):

  • ./run.sh INTEGRATION_FRAMEWORKS -L python --weblog openai-py@2.0.0 tests/integration_frameworks/llm/openai/test_openai_ai_guard_mcp.py: 8 passed
  • ./run.sh INTEGRATION_FRAMEWORKS -L python --weblog anthropic-py@0.84.0 tests/integration_frameworks/llm/anthropic/test_anthropic_ai_guard_mcp.py: 5 passed
  • test_ai_guard_anthropic_mcp_command_path: XFAIL on the dd-trace-py main wheel, for the right reason (the response evaluation misses its cassette because the full path is sent), and passes with the chore(aiguard): merge MCP servers sharing an ID and report stdio commands by program name dd-trace-py#21929 branch loaded locally.
  • ./format.sh passes, including the AI Guard MCP cassette drift check.

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Tests, manifest, weblog are modified -> you'll need a review from system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.
    • Framework is modified, or non obvious usage of it -> get a review from system-tests-core (slack)

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

🤖 Generated with Claude Code

avara1986 and others added 2 commits November 19, 2026 14:25
… reconnects

Add system tests for the MCP server ID contract:

- OpenAI hosted MCP: the RFC host canonicalization vectors (IPv4 number
  forms, IPv6, percent-encoded and underscore hosts) get the RFC IDs, and an
  IDN host falls back to the name key.
- OpenAI hosted MCP: two servers on one origin share an ID and are reported
  as one merged mcp_servers entry with the tools of both.
- Anthropic stdio MCP: two requests, each with a new server process and MCP
  session, report the same server entry and ID.

The OpenAI cassettes are generated by gen_aiguard_mcp_cassettes.py; the
Anthropic test reuses copies of the recorded tool_use cassettes.

APPSEC-71867

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the Anthropic organization, workspace and request IDs, CF-RAY, trace
and timing headers, datadog-entity-id and the CSP header carrying a client
token, and replace the real evaluation and message IDs with synthetic ones.
Only response data and headers change: the hashed request bodies are kept.

APPSEC-71867

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/integration_frameworks/utils/vcr-cassettes/aiguard/test_ai_guard_anthropic_mcp_command_path_aiguard_evaluate_post_67a603bb.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/aiguard/test_ai_guard_anthropic_mcp_command_path_aiguard_evaluate_post_f5a80811.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/aiguard/test_ai_guard_anthropic_mcp_reconnect_same_server_id_aiguard_evaluate_post_67a603bb.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/aiguard/test_ai_guard_anthropic_mcp_reconnect_same_server_id_aiguard_evaluate_post_f5a80811.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/aiguard/test_ai_guard_openai_mcp_server_id_host_vectors_aiguard_evaluate_post_5398b618.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/aiguard/test_ai_guard_openai_mcp_server_id_host_vectors_aiguard_evaluate_post_d3bbc0e6.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/aiguard/test_ai_guard_openai_mcp_servers_sharing_an_id_merge_aiguard_evaluate_post_80404fae.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/aiguard/test_ai_guard_openai_mcp_servers_sharing_an_id_merge_aiguard_evaluate_post_ee7211f7.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/anthropic/test_ai_guard_anthropic_mcp_command_path_anthropic_v1_messages_post_6ea6f9b0.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/anthropic/test_ai_guard_anthropic_mcp_reconnect_same_server_id_anthropic_v1_messages_post_6ea6f9b0.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/openai/test_ai_guard_openai_mcp_server_id_host_vectors_openai_responses_post_ebb3e99a.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/utils/vcr-cassettes/openai/test_ai_guard_openai_mcp_servers_sharing_an_id_merge_openai_responses_post_5d147bbb.json  @DataDog/asm-libraries @DataDog/k9-ai-guard
manifests/python.yml                                                    @DataDog/system-tests-reviewers
tests/integration_frameworks/llm/anthropic/test_anthropic_ai_guard_mcp.py  @DataDog/asm-libraries @DataDog/k9-ai-guard
tests/integration_frameworks/llm/openai/test_openai_ai_guard_mcp.py     @DataDog/asm-libraries @DataDog/k9-ai-guard
utils/build/docker/python/anthropic_app/anthropic.py                    @DataDog/system-tests-reviewers
utils/scripts/gen_aiguard_mcp_cassettes.py                              @DataDog/system-tests-core

The anthropic-py weblog /mcp/create can start the echo server from the
absolute path of the interpreter (command_path). The new test checks that
only the program name reaches AI Guard and the server ID is unchanged. It
reuses the scrubbed cassettes, and is declared missing_feature for Python
until DataDog/dd-trace-py#21929 ships.

APPSEC-71867

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Pipelines  Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 47fe661 | Docs | View more details | Give us feedback!

@avara1986
avara1986 marked this pull request as ready for review November 19, 2026 21:22
@avara1986
avara1986 requested review from a team as code owners November 19, 2026 21:22
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T21:25:37.117495Z 47fe661 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 47fe6610f3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +86 to +87
# Start the echo server with the absolute path of the interpreter instead of "python".
command_path: bool = False

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Document the new MCP request parameter

INFRA-010 in .agents/rules/code-review/infrastructure-and-conventions.md requires weblog interface changes to update the corresponding specification. The Anthropic /mcp/create endpoint now accepts command_path, changing how the interpreter is launched, but this commit makes no update under docs/understand/weblogs/. Document the parameter, its default, and its behavior so other weblog implementations can reproduce this test interface.

AGENTS.md reference: AGENTS.md:L49-L49

Useful? React with 👍 / 👎.

Comment thread manifests/python.yml
"*": v4.18.0-rc1
anthropic-py@0.75.0: "irrelevant (anthropic<0.84.0 has no MCP tool helpers)"
? tests/integration_frameworks/llm/anthropic/test_anthropic_ai_guard_mcp.py::Test_AnthropicAiGuardMcp::test_ai_guard_anthropic_mcp_command_path
: missing_feature (stdio commands are reported with their full path until DataDog/dd-trace-py#21929)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Quote the manifest declaration containing a hash character

META-004 in .agents/rules/code-review/activation-and-metadata.md explicitly requires quoting manifest values containing #, even when YAML accepts them unquoted. This new declaration contains DataDog/dd-trace-py#21929; quote the whole value to follow that convention and avoid ambiguity with YAML comment syntax.

AGENTS.md reference: AGENTS.md:L48-L48

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant