Skip to content

[test] Coverage/mutation run: tests that pass without verifying behaviour (dispatcher, wire bytes, checksums, parse_packet, cpm) #243

Closed

Description

@vaderyang

Environment

  • CP version: v0.9.4 (f925e5f6); measured with gcov (C unit + integration suites) and go test -coverprofile; mutation testing with a source-level C mutator and gremlins (Go)

Description

A coverage + mutation run over the current test suites found areas where tests pass but do not actually verify the behaviour. Tracking them here so they can be picked up individually:

  1. The control-socket dispatcher is never executed. unix-manager.c::unix_command_execute has 0% coverage. tests/unit/unix_control.c calls the ping/info handlers directly, and the reload tests use SIGHUP, so handshake parsing, unknown commands and malformed JSON are untested over the real socket.
  2. Wire bytes are not verified. Every GRE/VXLAN/ZMQ integration case (testdata/cases/*/verify.json) uses packet_count_only/packet_count_range. No test checks the VXLAN header/VNI, the GRE protocol field, or the ZMQ batch/record/packet-header layout in docs/ZMQ-WIRE-FORMAT.md §3–§6 (only the heartbeat record is byte-checked).
  3. Checksum tests are round-trips. The TCP/UDP (IPv4 + IPv6) checksum tests compute a checksum and then verify it with the same code, so a wrong pseudo-header sum still passes: 26 surviving mutants in calculate_pseudo_header_sum/cksum_accumulate. Known-answer vectors are needed (only the IPv4 header has the RFC 1071 example).
  4. parse_packet boundaries. 34 surviving mutants, e.g. caplen < offset + ip_hdr_len → <= is undetected (no frame with caplen exactly equal to the header length), and there are no IPv6 extension-header cases.
  5. Untested features with no test at all: req_pattern_judge_pkt_direction on real packets (only the expression grammar is tested), rotating_file output, ZMQ batch flush rules (§8), corrupt/truncated pcap input to pcap_file, numeric config range checks (see [cpworker] Out-of-range numeric config silently clamped by cJSON (snaplen/buffer_size_mb/timeout_ms/ring_size) #234/[cpworker] zmq.hwm=0 means an unbounded queue and is not validated #235), and live libpcap capture fidelity (reload tests drive libpcap on lo but only assert log lines).
  6. Go: gremlins test efficacy is 53% for cpdaemon/pkg/cpm (63 surviving mutants, mostly syncer.go, worker_mgr.go, synclog.go) and 74% for cpgolib/cpworker (handshake recv-error and invalid-JSON branches, arguments omitted when empty).

Expected

Each area has at least one test derived from its spec (RFC / docs/*.md) that fails when the behaviour is wrong.


中文原文

测试环境

  • CP 版本:v0.9.4(f925e5f6);用 gcov(C 单测 + 集成测试)和 go test -coverprofile 测量;mutation 测试使用源码级 C 变异器与 gremlins(Go)

问题描述

对现有测试做了一次覆盖率 + mutation 测量,发现一些"测试通过但并未真正验证行为"的地方,在此汇总跟踪,可逐项认领:

  1. 控制 socket 分发器从未被执行:unix-manager.c::unix_command_execute 覆盖率 0%。单测直接调用 ping/info handler,reload 测试走 SIGHUP;经真实 socket 的握手解析、未知命令、畸形 JSON 均无测试。
  2. 线上字节未被验证:GRE/VXLAN/ZMQ 集成用例全部只用 packet_count_only/packet_count_range;没有测试检查 VXLAN 头/VNI、GRE 协议字段,或 docs/ZMQ-WIRE-FORMAT.md §3–§6 的 batch/record/包头布局(只有心跳记录做了字节校验)。
  3. 校验和测试是 round-trip:TCP/UDP(IPv4 + IPv6)校验和测试用同一段代码计算再验证,伪首部求和写错也能通过(calculate_pseudo_header_sum/cksum_accumulate 共 26 个变异体存活)。需要已知答案向量(目前只有 IPv4 头用了 RFC 1071 示例)。
  4. parse_packet 边界:34 个变异体存活,例如 caplen < offset + ip_hdr_len 改成 <= 不被发现(没有 caplen 恰好等于头长的帧),也没有 IPv6 扩展头用例。
  5. 完全没有测试的功能:真实报文上的 req_pattern_judge_pkt_direction(只测了表达式语法)、rotating_file 输出、ZMQ batch flush 规则(§8)、pcap_file 读入损坏/截断的 pcap、数值配置范围校验(见 [cpworker] Out-of-range numeric config silently clamped by cJSON (snaplen/buffer_size_mb/timeout_ms/ring_size) #234/[cpworker] zmq.hwm=0 means an unbounded queue and is not validated #235)、libpcap 实时抓包保真度(reload 测试在 lo 上跑 libpcap 但只断言日志)。
  6. Go:gremlins 测试有效率 cpdaemon/pkg/cpm 53%(63 个存活,主要在 syncer.go、worker_mgr.go、synclog.go),cpgolib/cpworker 74%(握手接收错误与非法 JSON 分支、参数为空时省略 arguments)。

期望

每一项至少有一个从其规范(RFC / docs/*.md)推导出的测试,并且在行为出错时会失败。

Activity

  1. timmy21 commented on Oct 13, 2026

    @timmy21
    Contributor

    Addressed in #314 (1e2b82e), #315 (c4e1a8e) and #316 (0ace1c5):

    1. Dispatcher: unknown command, malformed JSON and bad handshake are tested over the real socket (test(cpworker): test checksums, parse_packet bounds, zmq flush and control-socket errors by spec #314).
    2. Wire bytes: GRE, VXLAN and ZMQ output bytes are checked against RFC 2784/2890 and docs/*-WIRE-FORMAT.md (test(cpworker): check gre, vxlan and zmq wire bytes against the wire-format docs #315).
    3. Checksums: TCP/UDP IPv4/IPv6 known-answer vectors (test(cpworker): test checksums, parse_packet bounds, zmq flush and control-socket errors by spec #314).
    4. parse_packet: exact-length and one-short frames per layer (test(cpworker): test checksums, parse_packet bounds, zmq flush and control-socket errors by spec #314); IPv6 extension headers were added in fix(cpworker): harden packet_split header parsing and UDP checksum #282.
    5. ZMQ batch flush rules (test(cpworker): test checksums, parse_packet bounds, zmq flush and control-socket errors by spec #314); req_pattern on real frames (fix(cpworker): walk QinQ tag stacks in custom req_pattern direction detection #287), corrupt pcap input (fix(cpworker): report pcap_file read errors instead of end of file #278) and numeric ranges (fix(cpworker): validate numeric config ranges instead of silently truncating #279) were covered earlier. Not done: live libpcap capture fidelity, which needs a real interface or veth setup.
    6. Go: gremlins efficacy 63.4% -> 83.3% for cpdaemon/pkg/cpm and 76.0% -> 85.7% for cpgolib/cpworker (test(cpdaemon,cpgolib): check cpm config mapping, buffer sizing, sync state and handshake #316); the remaining mutants are log-only error branches, live process metrics and equivalent mutants.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions