Repository navigation
bazel/tap: Add tap2pcap, to convert Envoy socket tap traces to PCAP - #5622
Conversation
A pure Python replacement for the `tap2pcap` tool of the Envoy API
(`@envoy_api//tools:tap2pcap`), which shells out to `od` and to
Wireshark's `text2pcap`, and is tested by comparing `tshark` output to a
golden file. Neither tool is in the Envoy CI image, so that test only
passes while its result is in the remote cache.
`//tap:tap2pcap` writes the PCAP itself: a classic PCAP file with one
TCP segment (dummy Ethernet and IPv4/IPv6 headers, real checksums,
per-direction sequence numbers) for each read or write of a socket
buffered trace. Payloads over what an IP packet can carry are split.
The `tap2pcap` macro (`//tap:defs.bzl`) makes the binary from the
caller's trace wrapper message, so this module does not depend on the
Envoy API (which depends on this module):
tap2pcap(
name = "tap2pcap",
message = "envoy.data.tap.v3.wrapper_pb2.TraceWrapper",
proto = "//envoy/data/tap/v3:pkg_py_proto",
)
Tested on the PCAP bytes (decoded back, with every length and checksum
verified), and end to end on a binary made for a test proto with the
fields of Envoy's, for text and binary traces.
Signed-off-by: Ryan Northey <ryan@synca.io>
The tap2pcap tool shelled out to `od` and to Wireshark's `text2pcap`, and its test compared `tshark` output to a golden file. The CI image has neither tool, so the test only passed while its result was in the remote cache, and failed whenever its action key changed (eg a `rules_python` bump). `@envoy_toolshed//tap:defs.bzl` (envoyproxy/toolshed#5622) writes the PCAP in Python and is tested there, so the tool is now made from that with the v3 trace wrapper, and the test, its fixtures and its line in `do_ci.sh api` are removed. `bazel run @envoy_api//tools:tap2pcap` is unchanged for users, and no longer needs Wireshark. Signed-off-by: Ryan Northey <ryan@synca.io>
Not for merge: `git_override`s of `envoy_toolshed` and `envoy_toolshed_jq` in every workspace, and a genrule that runs the tap2pcap tool on a real trace, until the toolshed change is released and in the registry. Signed-off-by: Ryan Northey <ryan@synca.io>
|
One or more custom setup steps configured for this repository failed during this Copilot code review run: Setup steps run before each review. If the review above is missing context, or no review was posted at all, the failing step above may be the cause. See the workflow run for failure details, fix your setup steps configuration, and re-request a review. Note You can configure setup steps for Copilot code review separately from Copilot cloud agent with a |
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the pure-Python tap-to-PCAP converter and its Bazel integration.
No open issues 🎉
What changed in this PR
Adds a pure-Python converter for Envoy socket tap traces, removing the need for external packet-conversion tools. A Bazel macro accepts the caller’s protobuf target, avoiding a dependency from toolshed back to the Envoy API.
Changes:
- Generates PCAP files with IPv4/IPv6 headers, checksums, per-direction sequence numbers, and payload segmentation.
- Provides a reusable library and macro-generated command-line tool supporting binary and text traces.
- Adds packet-validation and end-to-end tests with a standalone test proto and fixtures.
| File | Description |
|---|---|
bazel/tap/tests/trace.proto |
Defines an Envoy-shaped test trace schema. |
bazel/tap/tests/tap2pcap_test.py |
Tests packet generation, segmentation, and validation errors. |
bazel/tap/tests/tap2pcap_binary_test.py |
Tests CLI conversion of text and binary traces. |
bazel/tap/tests/socket.pb_text |
Supplies a socket trace with reads, writes, and timestamps. |
bazel/tap/tests/pcap.py |
Decodes generated packets and verifies lengths and checksums. |
bazel/tap/tests/http.pb_text |
Supplies an unsupported trace fixture for error testing. |
bazel/tap/tests/BUILD |
Wires test libraries, protobuf generation, and test targets. |
bazel/tap/tap2pcap.py |
Implements trace parsing, packet synthesis, and PCAP output. |
bazel/tap/defs.bzl |
Generates a CLI using the caller’s protobuf message. |
bazel/tap/BUILD |
Exposes the converter library and Bazel macro. |
Review effort: Balanced
tap2pcap, to convert Envoy socket tap traces to PCAPtap2pcap, to convert Envoy socket tap traces to PCAP
The `tap2pcap` tool (`@envoy_api//tools:tap2pcap`, from the traffic tapping docs) shelled out to `od` and to Wireshark's `text2pcap`, and its test compared `tshark` output to a golden file. The CI image has neither tool, so `@envoy_api//tools:tap2pcap_test` only passed while its result was in the remote cache: the API job reports "Executed 0 out of 5 tests". Anything that changes its action key runs it for real, and it fails with `FileNotFoundError: 'text2pcap'`, as on #49376 (a `rules_python` bump). `@envoy_toolshed//tap:defs.bzl` (envoyproxy/toolshed#5622, in `envoy_toolshed` 0.5.24.envoy from #49389) writes the PCAP in Python and is tested there. This makes the tool from that, with the v3 trace wrapper (`envoy.data.tap.v3`, where the old tool read the frozen `v2alpha` one), and removes `tap2pcap.py`, its test, the two fixtures, and the test's line in `do_ci.sh api`. For users `bazel run @envoy_api//tools:tap2pcap <tap .pb/.pb_text> <pcap path>` is unchanged, and no longer needs Wireshark installed. Other differences, from the toolshed implementation: IPv6 traces are written with correct headers (a known issue of the `text2pcap` path), `as_string` bodies are read as well as `as_bytes`, and a trace that is not a socket buffered trace is an error. Risk Level: Low (a developer tool and its test) Testing: These changes ran in CI on #49376 against a pre-release of the toolshed module, with a temporary target that converted the old `tap2pcap_h2_ipv4.pb_text` fixture with the new tool; the API job passed. The toolshed output for that fixture was also checked with `tshark` by hand: the same six HTTP/2 frames, lengths and times as the old golden file. Nothing in Envoy runs the tool after this change. Docs Changes: N/A (the documented command is unchanged) Release Notes: N/A Signed-off-by: Ryan Northey <ryan@synca.io>
The `tap2pcap` tool (`@envoy_api//tools:tap2pcap`, from the traffic tapping docs) shelled out to `od` and to Wireshark's `text2pcap`, and its test compared `tshark` output to a golden file. The CI image has neither tool, so `@envoy_api//tools:tap2pcap_test` only passed while its result was in the remote cache: the API job reports "Executed 0 out of 5 tests". Anything that changes its action key runs it for real, and it fails with `FileNotFoundError: 'text2pcap'`, as on #49376 (a `rules_python` bump). `@envoy_toolshed//tap:defs.bzl` (envoyproxy/toolshed#5622, in `envoy_toolshed` 0.5.24.envoy from #49389) writes the PCAP in Python and is tested there. This makes the tool from that, with the v3 trace wrapper (`envoy.data.tap.v3`, where the old tool read the frozen `v2alpha` one), and removes `tap2pcap.py`, its test, the two fixtures, and the test's line in `do_ci.sh api`. For users `bazel run @envoy_api//tools:tap2pcap <tap .pb/.pb_text> <pcap path>` is unchanged, and no longer needs Wireshark installed. Other differences, from the toolshed implementation: IPv6 traces are written with correct headers (a known issue of the `text2pcap` path), `as_string` bodies are read as well as `as_bytes`, and a trace that is not a socket buffered trace is an error. Risk Level: Low (a developer tool and its test) Testing: These changes ran in CI on #49376 against a pre-release of the toolshed module, with a temporary target that converted the old `tap2pcap_h2_ipv4.pb_text` fixture with the new tool; the API job passed. The toolshed output for that fixture was also checked with `tshark` by hand: the same six HTTP/2 frames, lengths and times as the old golden file. Nothing in Envoy runs the tool after this change. Docs Changes: N/A (the documented command is unchanged) Release Notes: N/A Signed-off-by: Ryan Northey <ryan@synca.io> Mirrored from https://github.com/envoyproxy/envoy @ c11136586782858deaa4ee7b0f270d83350fbe4f
The `tap2pcap` tool (`@envoy_api//tools:tap2pcap`, from the traffic tapping docs) shelled out to `od` and to Wireshark's `text2pcap`, and its test compared `tshark` output to a golden file. The CI image has neither tool, so `@envoy_api//tools:tap2pcap_test` only passed while its result was in the remote cache: the API job reports "Executed 0 out of 5 tests". Anything that changes its action key runs it for real, and it fails with `FileNotFoundError: 'text2pcap'`, as on envoyproxy#49376 (a `rules_python` bump). `@envoy_toolshed//tap:defs.bzl` (envoyproxy/toolshed#5622, in `envoy_toolshed` 0.5.24.envoy from envoyproxy#49389) writes the PCAP in Python and is tested there. This makes the tool from that, with the v3 trace wrapper (`envoy.data.tap.v3`, where the old tool read the frozen `v2alpha` one), and removes `tap2pcap.py`, its test, the two fixtures, and the test's line in `do_ci.sh api`. For users `bazel run @envoy_api//tools:tap2pcap <tap .pb/.pb_text> <pcap path>` is unchanged, and no longer needs Wireshark installed. Other differences, from the toolshed implementation: IPv6 traces are written with correct headers (a known issue of the `text2pcap` path), `as_string` bodies are read as well as `as_bytes`, and a trace that is not a socket buffered trace is an error. Risk Level: Low (a developer tool and its test) Testing: These changes ran in CI on envoyproxy#49376 against a pre-release of the toolshed module, with a temporary target that converted the old `tap2pcap_h2_ipv4.pb_text` fixture with the new tool; the API job passed. The toolshed output for that fixture was also checked with `tshark` by hand: the same six HTTP/2 frames, lengths and times as the old golden file. Nothing in Envoy runs the tool after this change. Docs Changes: N/A (the documented command is unchanged) Release Notes: N/A Signed-off-by: Ryan Northey <ryan@synca.io>
…#5622) (#5627) A pure Python replacement for the `tap2pcap` tool of the Envoy API (`@envoy_api//tools:tap2pcap`), which shells out to `od` and to Wireshark's `text2pcap`, and is tested by comparing `tshark` output to a golden file. Neither tool is in the Envoy CI image, so that test only passes while its result is in the remote cache. `//tap:tap2pcap` writes the PCAP itself: a classic PCAP file with one TCP segment (dummy Ethernet and IPv4/IPv6 headers, real checksums, per-direction sequence numbers) for each read or write of a socket buffered trace. Payloads over what an IP packet can carry are split. The `tap2pcap` macro (`//tap:defs.bzl`) makes the binary from the caller's trace wrapper message, so this module does not depend on the Envoy API (which depends on this module): tap2pcap( name = "tap2pcap", message = "envoy.data.tap.v3.wrapper_pb2.TraceWrapper", proto = "//envoy/data/tap/v3:pkg_py_proto", ) Tested on the PCAP bytes (decoded back, with every length and checksum verified), and end to end on a binary made for a test proto with the fields of Envoy's, for text and binary traces. Signed-off-by: Ryan Northey <ryan@synca.io> Backport note (workspace): picked as is, for the Envoy release branches that build with WORKSPACE (v1.37-v1.39), which have the same `tap2pcap_test` and the same CI image as `main` had. (cherry picked from commit 5f5e48b)
A pure Python replacement for the
tap2pcaptool of the Envoy API (@envoy_api//tools:tap2pcap, documented in Envoy's traffic tapping docs).The existing tool shells out to
odand to Wireshark'stext2pcap, and its test comparestsharkoutput to a golden file. Neither tool is in the Envoy CI image, so@envoy_api//tools:tap2pcap_testonly passes while its result is in the remote cache: anything that changes its action key (eg envoyproxy/envoy#49376, arules_pythonbump) runs it for real and fails withFileNotFoundError: 'text2pcap'.What this adds
//tap:tap2pcap: a library that writes the PCAP itself. A classic PCAP file, with one TCP segment (dummy Ethernet and IPv4/IPv6 headers, real checksums, per-direction sequence numbers) for each read or write of a socket buffered trace. A payload larger than an IP packet can carry is split into several segments.tap2pcapmacro (//tap:defs.bzl), which makes the binary from the caller's trace wrapper message.envoy_apidepends on this module, so this module cannot depend on the Envoy API: the proto comes from the caller.Differences from the Envoy tool
od,text2pcaportshark: Python standard library only (and the caller's protobuf).text2pcappath does not apply.Body.as_stringis read as well asas_bytes.Testing
//tap/tests:tap2pcap_test: the PCAP bytes, decoded back with every length and checksum verified (IPv4, IPv6, sequence numbers, segmentation, errors).//tap/tests:tap2pcap_binary_test: end to end on a binary made by the macro for a test proto with the fields of Envoy's, for text and binary traces.tap2pcap_h2_ipv4.pb_textfixture with this library and rantshark(4.4.19) on it with the arguments of the Envoy test. It dissects as the same six HTTP/2 frames, with the same frame lengths and times, as Envoy's golden file; the only difference is the precisiontsharkprints the time column with.The unit tests were run with plain Python; the Bazel targets have only been run by this PR's CI.
Envoy would then replace its tool, test and fixtures with one
tap2pcap(...)target.