Repository navigation
cpworker: output counters have no definition; byte and packet counts differ per output and do not add up #308
Description
Problem
The output counters (fwd_*, direction_drop_*, ratelimit_drop_*, error_drop_*) have no written definition, and each output counts them differently. A byte counter can mean either "captured bytes this output handled" or "bytes this output put on the wire", depending on the output and on the counter. The values are not comparable across outputs, and the per-output totals do not add up to what was captured.
These counters are shown by cpctl stats and uploaded to CPM (fwdPackets, fwdBytes, ... in cpdaemon/pkg/cpm/models.go).
Current behaviour (0.9.x a57d2f90)
| Output | Counter | Counts |
|---|---|---|
| all capturers | cap_bytes |
caplen (libpcap.c:47, pcap_file.c:36) |
| gre / vxlan | fwd_bytes, ratelimit_drop_bytes, error_drop_bytes |
encapsulation header + frame (output_gre.c:64,130, output_vxlan.c:178,207) |
| gre / vxlan | direction_drop_bytes |
frame only, no encapsulation header (output_gre.c:55, output_vxlan.c:198) |
vxlan with split |
fwd_packets |
1 per fragment (output_vxlan.c:174,179, called once per fragment); fwd_packets can exceed cap_packets |
| gre / vxlan partial send | fwd_bytes + error_drop_bytes |
one packet's bytes split across both counters, packet counted in fwd_packets (output_gre.c:124-126, output_vxlan.c:172-174) |
| zmq | fwd_bytes, error_drop_bytes |
whole batch: batch header + record headers + heartbeat records (output_zmq.c:119,130) |
| zmq | fwd_packets |
includes heartbeat records (#299) |
| zmq | direction_drop_bytes, ratelimit_drop_bytes |
caplen after slice + 4-byte MPLS header (output_zmq.c:194,199,208) |
| zmq runt | error_drop_bytes |
caplen (output_zmq.c:183) |
| file / rotating_file / null | fwd_bytes and drop bytes |
frame (caplen, after slice) |
Proposed definition
Each packet the capturer hands to an output ends in exactly one bucket: fwd, direction_drop, ratelimit_drop or error_drop. Packets are counted as captured packets, bytes as the captured packet's original caplen. For every output:
fwd_packets + direction_drop_packets + ratelimit_drop_packets + error_drop_packets = packets handed to the output
fwd_bytes + direction_drop_bytes + ratelimit_drop_bytes + error_drop_bytes = sum of their caplen
Consequences:
- Encapsulation overhead (GRE/VXLAN header, ZMQ batch and record headers) is not counted.
- A split packet counts once, as
fwdonly if every fragment was sent, otherwise aserror_drop. - A partial send counts the whole packet as
error_drop. slicedoes not change the counted bytes.- Heartbeats are not captured packets and are counted only in
heartbeat_packets.
If egress bandwidth is needed, add separate tx_packets / tx_bytes counters (wire packets and bytes including encapsulation, fragments and heartbeats) instead of overloading fwd_*.
Open question
How does CPM use fwdBytes? The definition above lowers fwdBytes for gre, vxlan and zmq by the encapsulation overhead. If CPM charts it as egress bandwidth to the collector, fwd_bytes should keep the wire meaning instead, and the conservation counters should get new names. This needs to be settled before any code change.
Scope
- Write the counter definitions into the docs (stats RPC /
docs/REFACTOR-CPCTL.md§6.2, and the per-output wire-format docs where they mention counters). - Apply the definition to every output: gre, vxlan (including
split), zmq, file, rotating_file, null. - Unit tests per output that check the conservation equations above, including split, partial send, rate limit and direction drop.
- cpworker: zmq heartbeats are counted in fwd_packets/fwd_bytes (an idle output reports forwarded packets) #299 (heartbeats in
fwd_*) is fixed separately and is consistent with this definition.
Activity
timmy21 commentedon Oct 9, 2026
Decided definition
This replaces the byte rule in "Proposed definition" above: bytes are counted after
slice, not as the originalcaplen. CPM only displaysfwdBytes/fwdPacketsand gives them no meaning of its own, andfwd_packets <= cap_packetsis not a requirement. The goal is one definition that every output follows.Counted length
For each packet the capturer hands to an output, the counted length
Lis the packet as the output carries it, without any encapsulation:L=caplenafterslice.slicekeeps the Ethernet header and the whole VLAN tag stack, soLis the record's frame bytes (the MPLS label is not included).L=caplen.capture_timetrailer, ZMQ batch/record headers and MPLS label, outer IP/UDP headers, repeated headers in split fragments.Buckets
Each input packet ends in exactly one bucket, once its outcome is known, counting 1 packet and
Lbytes:direction_dropratelimit_droperror_dropfwdheartbeat_packets.Rate limit
rate_limit_mbpsconsumesLper packet, once per input packet (also when split). While the limit is active, thefwd_bytesrate therefore stays at or belowrate_limit_mbps. For gre, vxlan and zmq the limit becomes slightly looser than today, because the 4-8 byte headers are no longer charged.Scope notes
output_stats_tshared by every output of every task (task.c:221,385). The equations above hold for the sum over outputs: with two outputs per task, the right-hand side is about2 × cap_packets. Per-output conservation is checked in unit tests.cap_bytesstays the unslicedcaplen, so withsliceset,fwd_bytesis lower thancap_bytes.fwd_*no longer includes heartbeats, butfwd_bytes/error_drop_bytesstill include batch and record headers.