Skip to content

bazel/tap: Add tap2pcap, to convert Envoy socket tap traces to PCAP - #5622

Merged
phlax merged 1 commit into
envoyproxy:mainfrom
phlax:bazel-tap2pcap
Nov 18, 2026
Merged

phlax merged 1 commit into
envoyproxy:mainfrom
phlax:bazel-tap2pcap

Conversation

@phlax

@phlax phlax commented Nov 18, 2026

Copy link
Copy Markdown
Member

A pure Python replacement for the tap2pcap tool of the Envoy API (@envoy_api//tools:tap2pcap, documented in Envoy's traffic tapping docs).

The existing tool shells out to od and to Wireshark's text2pcap, and its test compares tshark output to a golden file. Neither tool is in the Envoy CI image, so @envoy_api//tools:tap2pcap_test only passes while its result is in the remote cache: anything that changes its action key (eg envoyproxy/envoy#49376, a rules_python bump) runs it for real and fails with FileNotFoundError: 'text2pcap'.

What this adds

  • //tap:tap2pcap: a library that writes the PCAP itself. A classic PCAP file, with one TCP segment (dummy Ethernet and IPv4/IPv6 headers, real checksums, per-direction sequence numbers) for each read or write of a socket buffered trace. A payload larger than an IP packet can carry is split into several segments.
  • The tap2pcap macro (//tap:defs.bzl), which makes the binary from the caller's trace wrapper message. envoy_api depends on this module, so this module cannot depend on the Envoy API: the proto comes from the caller.
tap2pcap(
    name = "tap2pcap",
    message = "envoy.data.tap.v3.wrapper_pb2.TraceWrapper",
    proto = "//envoy/data/tap/v3:pkg_py_proto",
)

Differences from the Envoy tool

  • No od, text2pcap or tshark: Python standard library only (and the caller's protobuf).
  • IPv6 traces are written with our own headers, so the "malformed TCP packets" known issue of the text2pcap path does not apply.
  • Timestamps are written from the trace's epoch time directly, without the local timezone adjustment.
  • Body.as_string is read as well as as_bytes.
  • A trace that is not a socket buffered trace, or has no IP addresses, is an error rather than an empty or broken PCAP.

Testing

  • //tap/tests:tap2pcap_test: the PCAP bytes, decoded back with every length and checksum verified (IPv4, IPv6, sequence numbers, segmentation, errors).
  • //tap/tests:tap2pcap_binary_test: end to end on a binary made by the macro for a test proto with the fields of Envoy's, for text and binary traces.
  • By hand: converted Envoy's tap2pcap_h2_ipv4.pb_text fixture with this library and ran tshark (4.4.19) on it with the arguments of the Envoy test. It dissects as the same six HTTP/2 frames, with the same frame lengths and times, as Envoy's golden file; the only difference is the precision tshark prints the time column with.

The unit tests were run with plain Python; the Bazel targets have only been run by this PR's CI.

Envoy would then replace its tool, test and fixtures with one tap2pcap(...) target.

A pure Python replacement for the `tap2pcap` tool of the Envoy API
(`@envoy_api//tools:tap2pcap`), which shells out to `od` and to
Wireshark's `text2pcap`, and is tested by comparing `tshark` output to a
golden file. Neither tool is in the Envoy CI image, so that test only
passes while its result is in the remote cache.

`//tap:tap2pcap` writes the PCAP itself: a classic PCAP file with one
TCP segment (dummy Ethernet and IPv4/IPv6 headers, real checksums,
per-direction sequence numbers) for each read or write of a socket
buffered trace. Payloads over what an IP packet can carry are split.

The `tap2pcap` macro (`//tap:defs.bzl`) makes the binary from the
caller's trace wrapper message, so this module does not depend on the
Envoy API (which depends on this module):

    tap2pcap(
        name = "tap2pcap",
        message = "envoy.data.tap.v3.wrapper_pb2.TraceWrapper",
        proto = "//envoy/data/tap/v3:pkg_py_proto",
    )

Tested on the PCAP bytes (decoded back, with every length and checksum
verified), and end to end on a binary made for a test proto with the
fields of Envoy's, for text and binary traces.

Signed-off-by: Ryan Northey <ryan@synca.io>
@phlax
phlax requested a balanced review from Copilot November 18, 2026 12:09
phlax added a commit to envoyproxy/envoy that referenced this pull request Nov 18, 2026
The tap2pcap tool shelled out to `od` and to Wireshark's `text2pcap`, and
its test compared `tshark` output to a golden file. The CI image has
neither tool, so the test only passed while its result was in the
remote cache, and failed whenever its action key changed (eg a
`rules_python` bump).

`@envoy_toolshed//tap:defs.bzl` (envoyproxy/toolshed#5622) writes the
PCAP in Python and is tested there, so the tool is now made from that
with the v3 trace wrapper, and the test, its fixtures and its line in
`do_ci.sh api` are removed. `bazel run @envoy_api//tools:tap2pcap` is
unchanged for users, and no longer needs Wireshark.

Signed-off-by: Ryan Northey <ryan@synca.io>
phlax added a commit to envoyproxy/envoy that referenced this pull request Nov 18, 2026
Not for merge: `git_override`s of `envoy_toolshed` and `envoy_toolshed_jq`
in every workspace, and a genrule that runs the tap2pcap tool on a real
trace, until the toolshed change is released and in the registry.

Signed-off-by: Ryan Northey <ryan@synca.io>

Copilot AI commented Nov 18, 2026

Copy link
Copy Markdown

One or more custom setup steps configured for this repository failed during this Copilot code review run:

Install deps

Setup steps run before each review. If the review above is missing context, or no review was posted at all, the failing step above may be the cause. See the workflow run for failure details, fix your setup steps configuration, and re-request a review.

Note

You can configure setup steps for Copilot code review separately from Copilot cloud agent with a copilot-code-review.yml file. Read the docs for details.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the pure-Python tap-to-PCAP converter and its Bazel integration.

No open issues 🎉

What changed in this PR

Adds a pure-Python converter for Envoy socket tap traces, removing the need for external packet-conversion tools. A Bazel macro accepts the caller’s protobuf target, avoiding a dependency from toolshed back to the Envoy API.

Changes:

  • Generates PCAP files with IPv4/IPv6 headers, checksums, per-direction sequence numbers, and payload segmentation.
  • Provides a reusable library and macro-generated command-line tool supporting binary and text traces.
  • Adds packet-validation and end-to-end tests with a standalone test proto and fixtures.
File Description
bazel/​tap/​tests/​trace.proto Defines an Envoy-shaped test trace schema.
bazel/​tap/​tests/​tap2pcap_test.py Tests packet generation, segmentation, and validation errors.
bazel/​tap/​tests/​tap2pcap_binary_test.py Tests CLI conversion of text and binary traces.
bazel/​tap/​tests/​socket.pb_text Supplies a socket trace with reads, writes, and timestamps.
bazel/​tap/​tests/​pcap.py Decodes generated packets and verifies lengths and checksums.
bazel/​tap/​tests/​http.pb_text Supplies an unsupported trace fixture for error testing.
bazel/​tap/​tests/​BUILD Wires test libraries, protobuf generation, and test targets.
bazel/​tap/​tap2pcap.py Implements trace parsing, packet synthesis, and PCAP output.
bazel/​tap/​defs.bzl Generates a CLI using the caller’s protobuf message.
bazel/​tap/​BUILD Exposes the converter library and Bazel macro.

Review effort: Balanced

@phlax phlax changed the title [WIP] bazel/tap: Add tap2pcap, to convert Envoy socket tap traces to PCAP bazel/tap: Add tap2pcap, to convert Envoy socket tap traces to PCAP Nov 18, 2026
@phlax
phlax marked this pull request as ready for review November 18, 2026 12:19
@phlax
phlax merged commit 5f5e48b into envoyproxy:main Nov 18, 2026
56 checks passed
phlax added a commit to envoyproxy/envoy that referenced this pull request Nov 18, 2026
The `tap2pcap` tool (`@envoy_api//tools:tap2pcap`, from the traffic
tapping docs) shelled out to `od` and to Wireshark's `text2pcap`, and
its test compared `tshark` output to a golden file.

The CI image has neither tool, so `@envoy_api//tools:tap2pcap_test` only
passed while its result was in the remote cache: the API job reports
"Executed 0 out of 5 tests". Anything that changes its action key runs
it for real, and it fails with `FileNotFoundError: 'text2pcap'`, as on
#49376 (a `rules_python` bump).

`@envoy_toolshed//tap:defs.bzl` (envoyproxy/toolshed#5622, in
`envoy_toolshed` 0.5.24.envoy from #49389) writes the PCAP in Python and
is tested there. This makes the tool from that, with the v3 trace
wrapper (`envoy.data.tap.v3`, where the old tool read the frozen
`v2alpha` one), and removes `tap2pcap.py`, its test, the two fixtures,
and the test's line in `do_ci.sh api`.

For users `bazel run @envoy_api//tools:tap2pcap <tap .pb/.pb_text> <pcap
path>` is unchanged, and no longer needs Wireshark installed. Other
differences, from the toolshed implementation: IPv6 traces are written
with correct headers (a known issue of the `text2pcap` path),
`as_string` bodies are read as well as `as_bytes`, and a trace that is
not a socket buffered trace is an error.

Risk Level: Low (a developer tool and its test)
Testing: These changes ran in CI on #49376 against a pre-release of the
toolshed module, with a temporary target that converted the old
`tap2pcap_h2_ipv4.pb_text` fixture with the new tool; the API job
passed. The toolshed output for that fixture was also checked with
`tshark` by hand: the same six HTTP/2 frames, lengths and times as the
old golden file. Nothing in Envoy runs the tool after this change.
Docs Changes: N/A (the documented command is unchanged)
Release Notes: N/A

Signed-off-by: Ryan Northey <ryan@synca.io>
update-envoy Bot added a commit to envoyproxy/data-plane-api that referenced this pull request Nov 18, 2026
The `tap2pcap` tool (`@envoy_api//tools:tap2pcap`, from the traffic
tapping docs) shelled out to `od` and to Wireshark's `text2pcap`, and
its test compared `tshark` output to a golden file.

The CI image has neither tool, so `@envoy_api//tools:tap2pcap_test` only
passed while its result was in the remote cache: the API job reports
"Executed 0 out of 5 tests". Anything that changes its action key runs
it for real, and it fails with `FileNotFoundError: 'text2pcap'`, as on
#49376 (a `rules_python` bump).

`@envoy_toolshed//tap:defs.bzl` (envoyproxy/toolshed#5622, in
`envoy_toolshed` 0.5.24.envoy from #49389) writes the PCAP in Python and
is tested there. This makes the tool from that, with the v3 trace
wrapper (`envoy.data.tap.v3`, where the old tool read the frozen
`v2alpha` one), and removes `tap2pcap.py`, its test, the two fixtures,
and the test's line in `do_ci.sh api`.

For users `bazel run @envoy_api//tools:tap2pcap <tap .pb/.pb_text> <pcap
path>` is unchanged, and no longer needs Wireshark installed. Other
differences, from the toolshed implementation: IPv6 traces are written
with correct headers (a known issue of the `text2pcap` path),
`as_string` bodies are read as well as `as_bytes`, and a trace that is
not a socket buffered trace is an error.

Risk Level: Low (a developer tool and its test)
Testing: These changes ran in CI on #49376 against a pre-release of the
toolshed module, with a temporary target that converted the old
`tap2pcap_h2_ipv4.pb_text` fixture with the new tool; the API job
passed. The toolshed output for that fixture was also checked with
`tshark` by hand: the same six HTTP/2 frames, lengths and times as the
old golden file. Nothing in Envoy runs the tool after this change.
Docs Changes: N/A (the documented command is unchanged)
Release Notes: N/A

Signed-off-by: Ryan Northey <ryan@synca.io>

Mirrored from https://github.com/envoyproxy/envoy @ c11136586782858deaa4ee7b0f270d83350fbe4f
phlax added a commit to phlax/envoy that referenced this pull request Nov 18, 2026
The `tap2pcap` tool (`@envoy_api//tools:tap2pcap`, from the traffic
tapping docs) shelled out to `od` and to Wireshark's `text2pcap`, and
its test compared `tshark` output to a golden file.

The CI image has neither tool, so `@envoy_api//tools:tap2pcap_test` only
passed while its result was in the remote cache: the API job reports
"Executed 0 out of 5 tests". Anything that changes its action key runs
it for real, and it fails with `FileNotFoundError: 'text2pcap'`, as on
envoyproxy#49376 (a `rules_python` bump).

`@envoy_toolshed//tap:defs.bzl` (envoyproxy/toolshed#5622, in
`envoy_toolshed` 0.5.24.envoy from envoyproxy#49389) writes the PCAP in Python and
is tested there. This makes the tool from that, with the v3 trace
wrapper (`envoy.data.tap.v3`, where the old tool read the frozen
`v2alpha` one), and removes `tap2pcap.py`, its test, the two fixtures,
and the test's line in `do_ci.sh api`.

For users `bazel run @envoy_api//tools:tap2pcap <tap .pb/.pb_text> <pcap
path>` is unchanged, and no longer needs Wireshark installed. Other
differences, from the toolshed implementation: IPv6 traces are written
with correct headers (a known issue of the `text2pcap` path),
`as_string` bodies are read as well as `as_bytes`, and a trace that is
not a socket buffered trace is an error.

Risk Level: Low (a developer tool and its test)
Testing: These changes ran in CI on envoyproxy#49376 against a pre-release of the
toolshed module, with a temporary target that converted the old
`tap2pcap_h2_ipv4.pb_text` fixture with the new tool; the API job
passed. The toolshed output for that fixture was also checked with
`tshark` by hand: the same six HTTP/2 frames, lengths and times as the
old golden file. Nothing in Envoy runs the tool after this change.
Docs Changes: N/A (the documented command is unchanged)
Release Notes: N/A

Signed-off-by: Ryan Northey <ryan@synca.io>
phlax added a commit that referenced this pull request Nov 18, 2026
…#5622) (#5627)

A pure Python replacement for the `tap2pcap` tool of the Envoy API
(`@envoy_api//tools:tap2pcap`), which shells out to `od` and to
Wireshark's `text2pcap`, and is tested by comparing `tshark` output to a
golden file. Neither tool is in the Envoy CI image, so that test only
passes while its result is in the remote cache.

`//tap:tap2pcap` writes the PCAP itself: a classic PCAP file with one
TCP segment (dummy Ethernet and IPv4/IPv6 headers, real checksums,
per-direction sequence numbers) for each read or write of a socket
buffered trace. Payloads over what an IP packet can carry are split.

The `tap2pcap` macro (`//tap:defs.bzl`) makes the binary from the
caller's trace wrapper message, so this module does not depend on the
Envoy API (which depends on this module):

    tap2pcap(
        name = "tap2pcap",
        message = "envoy.data.tap.v3.wrapper_pb2.TraceWrapper",
        proto = "//envoy/data/tap/v3:pkg_py_proto",
    )

Tested on the PCAP bytes (decoded back, with every length and checksum
verified), and end to end on a binary made for a test proto with the
fields of Envoy's, for text and binary traces.

Signed-off-by: Ryan Northey <ryan@synca.io>
Backport note (workspace): picked as is, for the Envoy release branches that
build with WORKSPACE (v1.37-v1.39), which have the same `tap2pcap_test` and
the same CI image as `main` had.

(cherry picked from commit 5f5e48b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants