Repository navigation
Review learning: rule changes from 2026-10-20 to 2026-11-18 - #8601
dd-apm-ecosystems[bot] wants to merge 7 commits into
Conversation
|
|
| - Severity: P1 | ||
| - Flag: Pick the marker that matches the cause. When one PR excludes tests for different causes, give each exclusion its own reason, and make each reason state the real cause. By default, markers other than `irrelevant` and `flaky` become a non-strict xfail, and `conftest.py` also records a fixture error as `xfailed`. When a marked test fails before its assertions, the reason must name that failure. | ||
| - Detect: A marker contradicts its cause, or several exclusions with different causes share one vague reason. A marked test fails during setup because of invalid configuration or a missing required secret that the marker does not describe. | ||
| - Detect: A marker contradicts its cause, or several exclusions with different causes share one vague reason. A file-level declaration covers tests that pass or that fail for different causes. A `flaky` declaration hides a failure that the test itself causes, such as a wrong assertion or a timing assumption. A marked test fails in setup from invalid configuration or a missing secret, and the marker does not say so. |
There was a problem hiding this comment.
I'm not sure to get this point. The outcome can be flaky because the test logic is crap Or because the underlying feature is underterministic. In both use cases, we declare it as flaky. Same for bug
There was a problem hiding this comment.
Edited META-006 in 28a309c. I removed the Detect check about a flaky marker hiding a failure in the test logic, because flaky and bug apply whether the test or the feature causes the failure.
The review learning workflow answers once a day. It does not read this reply.
| ### META-010: Activate a test only for the versions and weblogs that have its behavior. | ||
| - Severity: P0 | ||
| - Flag: A manifest version range must match the releases that implement the tested behavior. If a later release changes the behavior, close the range of the old variant. If a version is validated on one weblog only, do not apply it to every weblog. | ||
| - Detect: A manifest activates a test from a version that lacks the behavior. A legacy variant has no upper bound, although newer releases use a new behavior. A change lowers a minimum version for all weblogs after validation on one weblog. |
There was a problem hiding this comment.
This one seems useless : if you activate a test for a version/weblog does not have the behavior, the CI will fail, no ?
There was a problem hiding this comment.
Discarded META-010 in 28a309c: the rule is back as it is on main. The reviewer says the rule is useless because CI already fails when a test runs for a version or weblog that lacks the behavior, so the rule is removed from this pull request.
The review learning workflow answers once a day. It does not read this reply.
| - ❌ Poll for traces in `test_*` after the containers have stopped, or assert expected trace contents from `setup_*`. | ||
| - Ref: [End-to-end test flow](../../../docs/understand/test-flow.md#end-to-end-scenario), [Flushing default behavior](../../../docs/edit/flushing.md#default-behavior) | ||
|
|
||
| ### LIFE-005: Do not tie exact counts to flush timing. |
There was a problem hiding this comment.
This one is slippery : many test relies on time condition in a way or another. I'm afraid this rules will cause some harm.
There was a problem hiding this comment.
Discarded LIFE-005 in 60feb35: the rule is back as it is on main. The reviewer thinks this rule can harm many tests that depend on timing, so this change to the rule is dropped from this pull request.
The review learning workflow answers once a day. It does not read this reply.
| - Severity: P1 | ||
| - Flag: When checking one protocol flag, mask that bit and accept other bits the protocol allows, unless the whole field is under test. When checking one member of a list field, such as a `tracestate` vendor or an `x-datadog-tags` entry, find that member. Do not require a count or an order that the protocol does not define. When checking for redaction leaks, use a distinctive marker or the relevant parsed field so unrelated payload values cannot count as leaked input. | ||
| - Detect: A sampled-bit assertion requires exact flags `01`. A test requires `len(tracestate) == 1` or compares `x-datadog-tags` as a raw string. A redaction check searches an entire payload for a short marker such as `1234`. | ||
| - Detect: A sampled-bit assertion requires exact flags `01`. A test requires `len(tracestate) == 1` or compares `x-datadog-tags` as a raw string. A redaction check searches an entire payload for a short marker such as `1234`. A test compares serialized JSON as a string instead of parsed values. A test requires one stats request, one bucket, or an exact metric count, but the tracer flushes on a timer. |
There was a problem hiding this comment.
Edited TEST-001 in a158884. I rewrote the rule in shorter, plainer sentences so its intent is clear, and kept its meaning, severity, and examples.
The review learning workflow answers once a day. It does not read this reply.
Note
Monthly review learning. 9 changes to the code review rules, learned from the 364 PRs merged from 2026-10-20 to 2026-11-18.
Review them in Files changed, and answer on each rule:
ok,discard, a suggestion, or a comment. Once a day, a workflow applies the answers and replies.Changes in this PR
META-006META-010INFRA-011INFRA-002INFRA-013LIFE-005TEST-001TEST-008TEST-0099 more changes with evidence from one PR only. They stay out of the diff. Comment
add <ID>to apply one.META-0077.0.0-preis the same defect as a minimum dev version: builds without the change also match.INFRA-005INFRA-012INFRA-014LIFE-003LIFE-002TEST-002TEST-007TEST-010META-007· sharpen Detect · activation-and-metadataGating on
7.0.0-preis the same defect as a minimum dev version: builds without the change also match. The current Detect names only-devand RCs.Current:
Proposed:
Evidence:
correction: "i had thought the proper way of doing this was to first disable the affected test"INFRA-005· sharpen Detect · infrastructure-and-conventionsThe log upload was tied to the weblog build step, and PARAMETRIC skips that step, so no logs were saved. The current Detect only covers a missing
iforalways().Current:
Proposed:
Evidence:
correction: "with parametric, you won't have any logs saved"INFRA-012· new rule: Check downstream consumers of the shared CI pipeline. · infrastructure-and-conventionsA change to the shared GitLab pipeline for a system-tests-only job broke dd-trace-php and had to be reverted. No rule in any lens covers downstream consumers of the shared CI.
Proposed:
Proposed severity: P1. The breakage shows up in other repositories rather than hiding a system-tests failure, so it is informative.
Evidence:
miss(culprit #8472)INFRA-014· new rule: Keep test_the_test for system-tests internals. · infrastructure-and-conventionsA maintainer asked the author to remove unit tests of FFE test classes from test_the_test, because that folder is for system-tests internals. No rule states this convention.
Proposed:
Proposed severity: P1. This is a repository convention that does not by itself hide failures.
Evidence:
correction: "We usually use test_the_test for system-tests internals"LIFE-003· sharpen Detect · lifecycle-and-isolationLIFE-003 covers matching the data to this action. Its Detect misses stale telemetry from the old process after a restart. The root cause is the test design, not the helper that holds the fix.
Current:
Proposed:
Evidence:
miss(culprit #7196)LIFE-002· sharpen Detect · lifecycle-and-isolationLIFE-002 says to use the scenario flush timeout, but its Detect does not catch a timeout too short for the telemetry flush. In that case the weblog stops before the data arrives.
Current:
Proposed:
Evidence:
correction: "Setlibrary_interface_timeout=15here."TEST-002· sharpen Detect · test-effectivenessAn allowed subject can serve as the control, but if the test sums its counts with the suppressed subject's counts, the suppression check proves nothing. The Detect line does not name this aggregation.
Current:
Proposed:
Evidence:
correction: "Please validate the shared flag separately for the opted-out and allowed subjects."TEST-007· sharpen Detect · test-effectivenessThe mock compared the whole query to an exact dict, so it rejected valid requests that the real service accepts. TEST-007 asks for the real contract, but its Detect only covers response shape.
Current:
Proposed:
Evidence:
absence(culprit #8143)TEST-010· new rule: Limit a relaxed assertion to the case that needs it. · test-effectivenessHere, a fix for V1 payloads weakened the checks for other encodings and dropped a run-wide guarantee. No rule names this loss of coverage.
Proposed:
Proposed severity: P0. The relaxed check skips coverage for the cases that never needed the relaxation.
Evidence:
correction: "Shouldn't we still be validating the git tags are only on the first span for them?"correction: "it also drops the run-wide guarantee for every other tracer that runsTest_V1PayloadByDefaultin DEFAULT"Full evidence of the 9 changes in this PR
META-006· sharpen Detect · activation-and-metadataThe current Detect misses two cases. In one, a file-level incomplete_test_app declaration hides passing tests and gives inaccurate reasons (8352-2). In the other, flaky manifest entries hide a test-side defect: an assertion that contradicts the RFC (8371-1) or a single-stats-bucket timing assumption (8519-2). In both PRs, the fix corrected the test and removed the flaky entry.
Current:
Proposed:
Evidence:
correction: "Would you consider declaring these per class with the real cause, or letting the ones that pass run?"absence(culprit #8348)miss(culprit #5543)META-010· new rule: Activate a test only for the versions and weblogs that have its behavior. · activation-and-metadataThree PRs set version ranges that were too wide. One activated a legacy-key variant for every later Go version. One enabled a test for Go versions that lack the drop telemetry. One lowered a Java Lambda minimum for every weblog after testing only one. No existing rule covers version-range scope.
Proposed:
Proposed severity: P0. A range that is too wide makes the test fail for the wrong reason on releases or weblogs that never had the behavior.
Evidence:
correction: "This activates the legacydd.serialIdclass for every Go version from v2.10.1."correction: "Gate this class atv2.11.0-devfor Go."correction: "this PR moves them to v1.67.0 for every weblog, ALB included"INFRA-011· sharpen Detect · infrastructure-and-conventionsProvision and build steps went on after failures (a failed agent restart, a missing Docker engine, a fragile install gate). The real error then showed up later in some other step. The current Detect lists only shell stderr and pipeline patterns, so it misses these cases.
Current:
Proposed:
Evidence:
correction: "Restart-Service -Name datadogagent -Force fails with Cannot stop datadogagent service, but the script continues."miss(culprit #3420)correction: "Checking for themcp.pyfile feels a bit fragile"INFRA-002· sharpen Flag · infrastructure-and-conventionsA shared mocked-backend option was added, but a parallel lifecycle and a hardcoded API key were left in place, so they drifted apart. In another PR a backend response was overridden inside one test. Both repeat logic that a shared utility already owns.
Current:
Proposed:
Evidence:
absence(culprit #8143)correction: "can you add it directly in the mocked backend, rather dynamically in a test"INFRA-013· new rule: Require scenario inputs only when the scenario uses them. · infrastructure-and-conventionsTwo PRs made scenarios demand inputs they did not need: credentials in mocked mode, and an opt-in env var for an environment the scenario can detect. No existing rule covers this.
Proposed:
Proposed severity: P1. Extra required inputs make scenarios harder to run but do not hide test failures.
Evidence:
correction: "we only need this env variable when we use the real backend"correction: "My concern is I'd like to avoid to let users think about it"LIFE-005· new rule: Do not tie exact counts to flush timing. · lifecycle-and-isolationFive candidates show assertions whose result depends on flush timers, time buckets, rate limiters, or other test classes. LIFE-002 covers sleeps and TEST-001 covers protocol list fields, but no rule names this defect.
Proposed:
Proposed severity: P0. Timing-dependent counts make a test fail for the wrong reason or miss a late extra item.
Evidence:
correction: "dropped >= 1will flake: each request already sleeps ~800ms"correction: "I am leaving it approved but I think this is a valid point."miss(culprit #5543)correction: "so it depends on every metric point being flushed before teardown"correction: "a third evaluation landing in a later flush wouldn't be caught"TEST-001· sharpen Detect · test-effectivenessTEST-001 already forbids counts and formats that the protocol does not define. Its Detect misses timer-flushed request or bucket counts, scenario-wide metric counts, and JSON compared as a raw string. Valid tracers fail these checks or flake.
Current:
Proposed:
Evidence:
correction: "I am leaving it approved but I think this is a valid point."miss(culprit #5543)correction: "so it depends on every metric point being flushed before teardown"correction: "Once nodejs implements this"TEST-008· new rule: Check the type when the contract fixes a boolean type. · test-effectivenessTwo PRs had equality checks that accepted a number in place of a boolean. No rule names strict type checks. TEST-004 covers only the case where the contract accepts these values as the same.
Proposed:
Proposed severity: P0. A wrong value type passes the assertion, so the test passes for the wrong reason.
Evidence:
correction: "Please compare bool-ness in both directions here."correction: "I think it could be a float with value1.0or0.0and not pass the test here, but worth verifying."TEST-009· new rule: Let the weblog report a failure of the action under test. · test-effectivenessIn two PRs, the weblog apps swallowed failures, so status checks could not detect broken packaging or a broken install. INFRA-011 covers only shell scripts.
Proposed:
Proposed severity: P0. A swallowed failure makes a status assertion pass while the behavior under test never ran.
Evidence:
correction: "checks only prove that the proxy and Handler routes exist"correction: "Checking for themcp.pyfile feels a bit fragile"7 defects that a rule already describes
META-009· activation-and-metadata: META-009 already flags a new scenario that has no step in run-end-to-end.yml. #8080INFRA-010· infrastructure-and-conventions: New weblog endpoints were not documented in end-to-end_weblog.md. #7425INFRA-006· infrastructure-and-conventions: A new base image was not picked up by the mirroring job. #8121INFRA-007· infrastructure-and-conventions: A bare curl download with no retry is the remaining-registry-command-without-retry case that INFRA-007 already names. #8202INFRA-008· infrastructure-and-conventions: Transitive dependencies of the weblog image were left unpinned. #8441INFRA-011· infrastructure-and-conventions: The hiddenfinderror and the silent fallback to the latest release are already the example in INFRA-011. #8572TEST-002· test-effectiveness: These are absence or timing-only checks with no positive control, and one adds a skip.yml entry. #8144 · #8398How to answer, and run details
How to answer each change
ok,LGTMor a thumbs up: keep the change.discardordiscard: <reason>: put the rule back as it is onmain. Outside a review thread, name the rule:discard LIFE-005.add <ID>: apply a folded change, for example one with evidence from one PR only.Not when:. The next runs read the rules.This PR only edits
.agents/rules/code-review/. It never merges by itself: a person fromsystem-tests-coreapproves it. The next monthly run publishes nothing while it is open.Not applied
META-011Human-only review and outside context: never rule material.
context: CI base images: GitLab CI has no Docker Hub credentials outside the mirroring jobs, so unmirrored base images could turn main red.context: Manifests: tests were disabled temporarily to unblock a dd-trace-dotnet merge, and the easy-wins script will re-enable them later.context: Manifests: the Java Akka HTTP and Play OTel semantics tests were enabled, but they fail on dev builds until dd-trace-java PR 12901 is merged.Coverage: test-effectiveness 10 · lifecycle-and-isolation 4 · activation-and-metadata 5 · infrastructure-and-conventions 12. 22 candidates went to another lens. 0 candidates had a quote that does not match.
Run: the workflow run in Datadog.