Skip to content

Review learning: rule changes from 2026-10-20 to 2026-11-18 - #8601

Draft
dd-apm-ecosystems[bot] wants to merge 7 commits into
mainfrom
review-learning/run-2026-10-20-aab1fd6a
Draft

dd-apm-ecosystems[bot] wants to merge 7 commits into
mainfrom
review-learning/run-2026-10-20-aab1fd6a

Conversation

@dd-apm-ecosystems

Copy link
Copy Markdown

Note

Monthly review learning. 9 changes to the code review rules, learned from the 364 PRs merged from 2026-10-20 to 2026-11-18.
Review them in Files changed, and answer on each rule: ok, discard, a suggestion, or a comment. Once a day, a workflow applies the answers and replies.

Merged PRs With human review Fixes linked to their cause Verified lessons Proposals Changes in this PR
364 283 16 52 31 9

Changes in this PR

Rule Lens Change Why Evidence
META-006 activation-and-metadata sharpen Detect The current Detect misses two cases. #8352 · #8371 · #8519
META-010 activation-and-metadata new rule: Activate a test only for the versions and weblogs that have its behavior. Three PRs set version ranges that were too wide. #7844 · #8093 · #8457
INFRA-011 infrastructure-and-conventions sharpen Detect Provision and build steps went on after failures (a failed agent restart, a missing Docker engine, a fragile install gate). #8071 · #8136 · #8441
INFRA-002 infrastructure-and-conventions sharpen Flag A shared mocked-backend option was added, but a parallel lifecycle and a hardcoded API key were left in place, so they drifted apart. #8147 · #8266
INFRA-013 infrastructure-and-conventions new rule: Require scenario inputs only when the scenario uses them. Two PRs made scenarios demand inputs they did not need: credentials in mocked mode, and an opt-in env var for an environment the scenario can detect. #8071 · #8064
LIFE-005 lifecycle-and-isolation new rule: Do not tie exact counts to flush timing. Five candidates show assertions whose result depends on flush timers, time buckets, rate limiters, or other test classes. #7425 · #8519 · #8352 · #8441
TEST-001 test-effectiveness sharpen Detect TEST-001 already forbids counts and formats that the protocol does not define. #8519 · #8352 · #8441
TEST-008 test-effectiveness new rule: Check the type when the contract fixes a boolean type. Two PRs had equality checks that accepted a number in place of a boolean. #8080 · #8130
TEST-009 test-effectiveness new rule: Let the weblog report a failure of the action under test. In two PRs, the weblog apps swallowed failures, so status checks could not detect broken packaging or a broken install. #8182 · #8441
9 more changes with evidence from one PR only. They stay out of the diff. Comment add <ID> to apply one.
Rule Lens Change Why Evidence
META-007 activation-and-metadata sharpen Detect Gating on 7.0.0-pre is the same defect as a minimum dev version: builds without the change also match. #8306
INFRA-005 infrastructure-and-conventions sharpen Detect The log upload was tied to the weblog build step, and PARAMETRIC skips that step, so no logs were saved. #8313
INFRA-012 infrastructure-and-conventions new rule: Check downstream consumers of the shared CI pipeline. A change to the shared GitLab pipeline for a system-tests-only job broke dd-trace-php and had to be reverted. #8548
INFRA-014 infrastructure-and-conventions new rule: Keep test_the_test for system-tests internals. A maintainer asked the author to remove unit tests of FFE test classes from test_the_test, because that folder is for system-tests internals. #8153
LIFE-003 lifecycle-and-isolation sharpen Detect LIFE-003 covers matching the data to this action. #7464
LIFE-002 lifecycle-and-isolation sharpen Detect LIFE-002 says to use the scenario flush timeout, but its Detect does not catch a timeout too short for the telemetry flush. #8093
TEST-002 test-effectiveness sharpen Detect An allowed subject can serve as the control, but if the test sums its counts with the suppressed subject's counts, the suppression check proves nothing. #8080
TEST-007 test-effectiveness sharpen Detect The mock compared the whole query to an exact dict, so it rejected valid requests that the real service accepts. #8399
TEST-010 test-effectiveness new rule: Limit a relaxed assertion to the case that needs it. Here, a fix for V1 payloads weakened the checks for other encodings and dropped a run-wide guarantee. #7843

META-007 · sharpen Detect · activation-and-metadata

Gating on 7.0.0-pre is the same defect as a minimum dev version: builds without the change also match. The current Detect names only -dev and RCs.

Current:

A manifest declares a minimum dev version, and an RC of that version predates the change.

Proposed:

A manifest declares a minimum dev or pre-release version, such as -dev or -pre. An RC or master build of that version predates the change.

Evidence:

  • #8306 correction: "i had thought the proper way of doing this was to first disable the affected test"

INFRA-005 · sharpen Detect · infrastructure-and-conventions

The log upload was tied to the weblog build step, and PARAMETRIC skips that step, so no logs were saved. The current Detect only covers a missing if or always().

Current:

A required upload, tag, or cleanup has no if, or it uses if: always(). In one set -e script it follows a command that can fail, and no trap runs it.

Proposed:

A required upload, tag, or cleanup has no if, or it uses if: always(). In one set -e script it follows a command that can fail, and no trap runs it. Its if depends on the outcome of a step that some scenarios skip, such as the weblog build for PARAMETRIC.

Evidence:

  • #8313 correction: "with parametric, you won't have any logs saved"

INFRA-012 · new rule: Check downstream consumers of the shared CI pipeline. · infrastructure-and-conventions

A change to the shared GitLab pipeline for a system-tests-only job broke dd-trace-php and had to be reverted. No rule in any lens covers downstream consumers of the shared CI.

Proposed:

INFRA-012: Check downstream consumers of the shared CI pipeline.

  • Severity: P1
  • Flag: Other repositories, such as dd-trace-php, consume the shared GitLab pipeline. If a change edits that pipeline, it must keep those consumers working. Keep a job that only system-tests needs out of the shared include path.
  • Detect: The diff changes the include condition in .gitlab-ci.yml or edits utils/ci/gitlab/build_ci_image.py to support a system-tests-only job.

Proposed severity: P1. The breakage shows up in other repositories rather than hiding a system-tests failure, so it is informative.

Evidence:

INFRA-014 · new rule: Keep test_the_test for system-tests internals. · infrastructure-and-conventions

A maintainer asked the author to remove unit tests of FFE test classes from test_the_test, because that folder is for system-tests internals. No rule states this convention.

Proposed:

INFRA-014: Keep test_the_test for system-tests internals.

  • Severity: P1
  • Flag: tests/test_the_test/ holds tests of system-tests internals, such as the framework, scripts, and mocked backends. Do not add unit tests there for product test classes that CI already runs.
  • Detect: A new file under tests/test_the_test/ tests product test classes, such as FFE tests. It does not test framework code.

Proposed severity: P1. This is a repository convention that does not by itself hide failures.

Evidence:

  • #8153 correction: "We usually use test_the_test for system-tests internals"

LIFE-003 · sharpen Detect · lifecycle-and-isolation

LIFE-003 covers matching the data to this action. Its Detect misses stale telemetry from the old process after a restart. The root cause is the test design, not the helper that holds the fix.

Current:

The wait accepts a trace or event whose request id is from an earlier request. It accepts a Remote Config acknowledgement for an older config version. Watching starts after the request. The test sends traffic as soon as Remote Config says "received".

Proposed:

The wait accepts a trace or event whose request id is from an earlier request. It accepts a Remote Config acknowledgement for an older config version. Watching starts after the request. The test sends traffic as soon as Remote Config says "received". After an app restart, the test reads telemetry without requiring the runtime ID of the new process.

Evidence:

LIFE-002 · sharpen Detect · lifecycle-and-isolation

LIFE-002 says to use the scenario flush timeout, but its Detect does not catch a timeout too short for the telemetry flush. In that case the weblog stops before the data arrives.

Current:

A setup or test uses a fixed sleep, polls without a deadline, or increases an arbitrary delay instead of waiting for an observable condition.

Proposed:

A setup or test uses a fixed sleep, polls without a deadline, or increases an arbitrary delay instead of waiting for an observable condition. A scenario sets library_interface_timeout or agent_interface_timeout shorter than the flush interval of the data that its tests check.

Evidence:

  • #8093 correction: "Set library_interface_timeout=15 here."

TEST-002 · sharpen Detect · test-effectiveness

An allowed subject can serve as the control, but if the test sums its counts with the suppressed subject's counts, the suppression check proves nothing. The Detect line does not name this aggregation.

Current:

A test expects no spans, logs, metrics, or events, and no assertion shows that data when the feature is on. Its control bypasses the proxy or receiver used for the absence check. A new entry in utils/scripts/activate_easy_wins/skip.yml is a review signal for this case.

Proposed:

A test expects no spans, logs, metrics, or events, and no assertion shows that data when the feature is on. Its control bypasses the proxy or receiver used for the absence check. The test sums counts across the suppressed subject and the control subject, so data from the wrong subject passes. A new entry in utils/scripts/activate_easy_wins/skip.yml is a review signal for this case.

Evidence:

  • #8080 correction: "Please validate the shared flag separately for the opted-out and allowed subjects."

TEST-007 · sharpen Detect · test-effectiveness

The mock compared the whole query to an exact dict, so it rejected valid requests that the real service accepts. TEST-007 asks for the real contract, but its Detect only covers response shape.

Current:

The self-test imports every expected value from the fake. The fake returns a bare body where the real service uses an envelope. The fake sends config for a product the client did not ask for.

Proposed:

The self-test imports every expected value from the fake. The fake returns a bare body where the real service uses an envelope. The fake sends config for a product the client did not ask for. The fake compares the whole request query, so a valid extra parameter gets an error.

Evidence:

TEST-010 · new rule: Limit a relaxed assertion to the case that needs it. · test-effectiveness

Here, a fix for V1 payloads weakened the checks for other encodings and dropped a run-wide guarantee. No rule names this loss of coverage.

Proposed:

TEST-010: Limit a relaxed assertion to the case that needs it.

  • Severity: P0
  • Flag: Keep the original check for every encoding that does not need a looser check. Do not narrow a run-wide guarantee to one request.
  • Detect: A change relaxes an assertion for all cases to fix one case. A check over all captured requests becomes a check over one request.
  • ✅ Accept git tags on several spans for V1 payloads only, and keep the first-span check for v0.4 and v0.5.
  • ❌ Check only the GET / request, so a v0.4 payload sent later passes.

Proposed severity: P0. The relaxed check skips coverage for the cases that never needed the relaxation.

Evidence:

  • #7843 correction: "Shouldn't we still be validating the git tags are only on the first span for them?"
  • #7843 correction: "it also drops the run-wide guarantee for every other tracer that runs Test_V1PayloadByDefault in DEFAULT"
Full evidence of the 9 changes in this PR

META-006 · sharpen Detect · activation-and-metadata

The current Detect misses two cases. In one, a file-level incomplete_test_app declaration hides passing tests and gives inaccurate reasons (8352-2). In the other, flaky manifest entries hide a test-side defect: an assertion that contradicts the RFC (8371-1) or a single-stats-bucket timing assumption (8519-2). In both PRs, the fix corrected the test and removed the flaky entry.

Current:

A marker contradicts its cause, or several exclusions with different causes share one vague reason. A marked test fails during setup because of invalid configuration or a missing required secret that the marker does not describe.

Proposed:

A marker contradicts its cause, or several exclusions with different causes share one vague reason. A file-level declaration covers tests that pass or that fail for different causes. A flaky declaration hides a failure that the test itself causes, such as a wrong assertion or a timing assumption. A marked test fails in setup from invalid configuration or a missing secret, and the marker does not say so.

Evidence:

  • #8352 correction: "Would you consider declaring these per class with the real cause, or letting the ones that pass run?"
  • #8371 absence (culprit #8348)
  • #8519 miss (culprit #5543)

META-010 · new rule: Activate a test only for the versions and weblogs that have its behavior. · activation-and-metadata

Three PRs set version ranges that were too wide. One activated a legacy-key variant for every later Go version. One enabled a test for Go versions that lack the drop telemetry. One lowered a Java Lambda minimum for every weblog after testing only one. No existing rule covers version-range scope.

Proposed:

META-010: Activate a test only for the versions and weblogs that have its behavior.

  • Severity: P0
  • Flag: A manifest version range must match the releases that implement the tested behavior. If a later release changes the behavior, close the range of the old variant. If a version is validated on one weblog only, do not apply it to every weblog.
  • Detect: A manifest activates a test from a version that lacks the behavior. A legacy variant has no upper bound, although newer releases use a new behavior. A change lowers a minimum version for all weblogs after validation on one weblog.

Proposed severity: P0. A range that is too wide makes the test fail for the wrong reason on releases or weblogs that never had the behavior.

Evidence:

  • #7844 correction: "This activates the legacy dd.serialId class for every Go version from v2.10.1."
  • #8093 correction: "Gate this class at v2.11.0-dev for Go."
  • #8457 correction: "this PR moves them to v1.67.0 for every weblog, ALB included"

INFRA-011 · sharpen Detect · infrastructure-and-conventions

Provision and build steps went on after failures (a failed agent restart, a missing Docker engine, a fragile install gate). The real error then showed up later in some other step. The current Detect lists only shell stderr and pipeline patterns, so it misses these cases.

Current:

The script discards stderr (2>/dev/null, &>/dev/null) or ignores a failure (|| true) on a command whose output is counted, parsed, or tested, and an error then looks like an empty result. A pipeline drops the failure of its first command because pipefail is off, as in cmd 2>/dev/null | wc -l or cmd | jq. A failed check goes silently to a fallback branch. A command runs under if ! with its stderr discarded, so the failure message loses the cause.

Proposed:

A command whose output is counted, parsed, or tested discards stderr or uses || true, so an error looks empty. A pipeline without pipefail drops a failure, as in cmd 2>/dev/null | wc -l or cmd | jq. A failed check goes silently to a fallback branch, or runs under if ! with its stderr discarded. A provision step continues after a failed command, such as Restart-Service, so the agent runs with wrong settings. A step exits 0 without checking its key result, such as docker info after a Docker install. A build gates an install on a file path inside a package, or the app catches ImportError broadly.

Evidence:

  • #8071 correction: "Restart-Service -Name datadogagent -Force fails with Cannot stop datadogagent service, but the script continues."
  • #8136 miss (culprit #3420)
  • #8441 correction: "Checking for the mcp.py file feels a bit fragile"

INFRA-002 · sharpen Flag · infrastructure-and-conventions

A shared mocked-backend option was added, but a parallel lifecycle and a hardcoded API key were left in place, so they drifted apart. In another PR a backend response was overridden inside one test. Both repeat logic that a shared utility already owns.

Current:

Do not redefine constants, enums, or helpers that already exist in utils/. In particular, check utils/dd_constants.py and utils/_features.py for constants and enums, and use from utils import logger.

Proposed:

Do not redefine constants, enums, or helpers that already exist in utils/. In particular, check utils/dd_constants.py and utils/_features.py for constants and enums, and use from utils import logger. If a PR adds a shared scenario option, subclasses and callers must use it, not a parallel lifecycle or a hardcoded constant. Implement a backend response in utils/mocked_backend/, not as an override inside one test.

Evidence:

  • #8147 absence (culprit #8143)
  • #8266 correction: "can you add it directly in the mocked backend, rather dynamically in a test"

INFRA-013 · new rule: Require scenario inputs only when the scenario uses them. · infrastructure-and-conventions

Two PRs made scenarios demand inputs they did not need: credentials in mocked mode, and an opt-in env var for an environment the scenario can detect. No existing rule covers this.

Proposed:

INFRA-013: Require scenario inputs only when the scenario uses them.

  • Severity: P1
  • Flag: A scenario must require a credential or an env var only in the mode that uses it. If the scenario can detect its environment, it must not ask the user for an opt-in env var.
  • Detect: The environment check of a scenario requires real-backend credentials while the scenario uses the mocked backend. A scenario adds an opt-in env var for a condition it can detect, such as the /vm-host path.

Proposed severity: P1. Extra required inputs make scenarios harder to run but do not hide test failures.

Evidence:

  • #8071 correction: "we only need this env variable when we use the real backend"
  • #8064 correction: "My concern is I'd like to avoid to let users think about it"

LIFE-005 · new rule: Do not tie exact counts to flush timing. · lifecycle-and-isolation

Five candidates show assertions whose result depends on flush timers, time buckets, rate limiters, or other test classes. LIFE-002 covers sleeps and TEST-001 covers protocol list fields, but no rule names this defect.

Proposed:

LIFE-005: Do not tie exact counts to flush timing.

  • Severity: P0
  • Flag: The SDK flushes data on its own timer and buckets it by wall-clock time. If the count of requests, buckets, or points depends on that timing, do not assert it exactly. Merge the data across requests and buckets, and assert the aggregated values. Read only the data of this test, not a scenario-wide total. If the test asserts an exact count, keep waiting after it appears, so a late extra item fails. Do not assert that a rate limiter drops data when the request pace decides the result.
  • Detect: A test asserts exactly one stats request or one bucket, or reads only bucket [0]. It asserts an exact scenario-wide telemetry count. A wait returns at the expected count before an == N assertion. It asserts dropped >= 1 against a time-based limiter.
  • Not when: The SDK contract fixes the count independently of flush timing.

Proposed severity: P0. Timing-dependent counts make a test fail for the wrong reason or miss a late extra item.

Evidence:

  • #7425 correction: "dropped >= 1 will flake: each request already sleeps ~800ms"
  • #8519 correction: "I am leaving it approved but I think this is a valid point."
  • #8519 miss (culprit #5543)
  • #8352 correction: "so it depends on every metric point being flushed before teardown"
  • #8441 correction: "a third evaluation landing in a later flush wouldn't be caught"

TEST-001 · sharpen Detect · test-effectiveness

TEST-001 already forbids counts and formats that the protocol does not define. Its Detect misses timer-flushed request or bucket counts, scenario-wide metric counts, and JSON compared as a raw string. Valid tracers fail these checks or flake.

Current:

A sampled-bit assertion requires exact flags 01. A test requires len(tracestate) == 1 or compares x-datadog-tags as a raw string. A redaction check searches an entire payload for a short marker such as 1234.

Proposed:

A sampled-bit assertion requires exact flags 01. A test requires len(tracestate) == 1 or compares x-datadog-tags as a raw string. A redaction check searches an entire payload for a short marker such as 1234. A test compares serialized JSON as a string instead of parsed values. A test requires one stats request, one bucket, or an exact metric count, but the tracer flushes on a timer.

Evidence:

  • #8519 correction: "I am leaving it approved but I think this is a valid point."
  • #8519 miss (culprit #5543)
  • #8352 correction: "so it depends on every metric point being flushed before teardown"
  • #8441 correction: "Once nodejs implements this"

TEST-008 · new rule: Check the type when the contract fixes a boolean type. · test-effectiveness

Two PRs had equality checks that accepted a number in place of a boolean. No rule names strict type checks. TEST-004 covers only the case where the contract accepts these values as the same.

Proposed:

TEST-008: Check the type when the contract fixes a boolean type.

  • Severity: P0
  • Flag: Python treats 1, 1.0, and True as equal. If the contract fixes a boolean or numeric type, check the type of both values.
  • Detect: An assertion compares a typed attribute with == alone. An isinstance check covers only one side of the comparison.
  • Not when: The contract accepts True, 1, and "true" as the same value. Then use is_same_boolean() as TEST-004 says.
  • ✅ assert isinstance(actual, bool) == isinstance(expected, bool) and actual == expected.
  • ❌ assert actual == expected when expected is True and the SDK sends 1.0.

Proposed severity: P0. A wrong value type passes the assertion, so the test passes for the wrong reason.

Evidence:

  • #8080 correction: "Please compare bool-ness in both directions here."
  • #8130 correction: "I think it could be a float with value 1.0 or 0.0 and not pass the test here, but worth verifying."

TEST-009 · new rule: Let the weblog report a failure of the action under test. · test-effectiveness

In two PRs, the weblog apps swallowed failures, so status checks could not detect broken packaging or a broken install. INFRA-011 covers only shell scripts.

Proposed:

TEST-009: Let the weblog report a failure of the action under test.

  • Severity: P0
  • Flag: A weblog endpoint must not catch every exception and return success. If an import or action fails, the endpoint must fail clearly. Then a status check proves that the action ran.
  • Detect: An endpoint catches all exceptions and returns 200 or OK. The app catches ImportError broadly and returns another status. A test uses that status as proof.
  • ✅ Let the exception reach the framework, so the test sees a server error.
  • ❌ Catch every exception in the reachability endpoint and always return OK.

Proposed severity: P0. A swallowed failure makes a status assertion pass while the behavior under test never ran.

Evidence:

  • #8182 correction: "checks only prove that the proxy and Handler routes exist"
  • #8441 correction: "Checking for the mcp.py file feels a bit fragile"
7 defects that a rule already describes
  • META-009 · activation-and-metadata: META-009 already flags a new scenario that has no step in run-end-to-end.yml. #8080
  • INFRA-010 · infrastructure-and-conventions: New weblog endpoints were not documented in end-to-end_weblog.md. #7425
  • INFRA-006 · infrastructure-and-conventions: A new base image was not picked up by the mirroring job. #8121
  • INFRA-007 · infrastructure-and-conventions: A bare curl download with no retry is the remaining-registry-command-without-retry case that INFRA-007 already names. #8202
  • INFRA-008 · infrastructure-and-conventions: Transitive dependencies of the weblog image were left unpinned. #8441
  • INFRA-011 · infrastructure-and-conventions: The hidden find error and the silent fallback to the latest release are already the example in INFRA-011. #8572
  • TEST-002 · test-effectiveness: These are absence or timing-only checks with no positive control, and one adds a skip.yml entry. #8144 · #8398
How to answer, and run details

How to answer each change

  • Nothing, ok, LGTM or a thumbs up: keep the change.
  • discard or discard: <reason>: put the rule back as it is on main. Outside a review thread, name the rule: discard LIFE-005.
  • A suggestion block: replace those lines.
  • add <ID>: apply a folded change, for example one with evidence from one PR only.
  • Any other comment on a rule: an agent edits that rule as you ask.
  • A change goes into the diff when its evidence comes from 2 or more distinct PRs. Each change must also pass the evidence thresholds and lint.
  • If the reason of a discard holds for the future, write it in the rule, for example as Not when:. The next runs read the rules.

This PR only edits .agents/rules/code-review/. It never merges by itself: a person from system-tests-core approves it. The next monthly run publishes nothing while it is open.

Not applied

Rule Action Status Reason
META-011 new_rule not_enough_evidence needs a fix, an applied maintainer correction, or two PRs

Human-only review and outside context: never rule material.

  • #8121 context: CI base images: GitLab CI has no Docker Hub credentials outside the mirroring jobs, so unmirrored base images could turn main red.
  • #8344 context: Manifests: tests were disabled temporarily to unblock a dd-trace-dotnet merge, and the easy-wins script will re-enable them later.
  • #8324 context: Manifests: the Java Akka HTTP and Play OTel semantics tests were enabled, but they fail on dev builds until dd-trace-java PR 12901 is merged.

Coverage: test-effectiveness 10 · lifecycle-and-isolation 4 · activation-and-metadata 5 · infrastructure-and-conventions 12. 22 candidates went to another lens. 0 candidates had a quote that does not match.

Run: the workflow run in Datadog.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

.agents/rules/code-review/activation-and-metadata.md                    @DataDog/system-tests-core
.agents/rules/code-review/infrastructure-and-conventions.md             @DataDog/system-tests-core
.agents/rules/code-review/test-effectiveness.md                         @DataDog/system-tests-core

- Severity: P1
- Flag: Pick the marker that matches the cause. When one PR excludes tests for different causes, give each exclusion its own reason, and make each reason state the real cause. By default, markers other than `irrelevant` and `flaky` become a non-strict xfail, and `conftest.py` also records a fixture error as `xfailed`. When a marked test fails before its assertions, the reason must name that failure.
- Detect: A marker contradicts its cause, or several exclusions with different causes share one vague reason. A marked test fails during setup because of invalid configuration or a missing required secret that the marker does not describe.
- Detect: A marker contradicts its cause, or several exclusions with different causes share one vague reason. A file-level declaration covers tests that pass or that fail for different causes. A `flaky` declaration hides a failure that the test itself causes, such as a wrong assertion or a timing assumption. A marked test fails in setup from invalid configuration or a missing secret, and the marker does not say so.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure to get this point. The outcome can be flaky because the test logic is crap Or because the underlying feature is underterministic. In both use cases, we declare it as flaky. Same for bug

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edited META-006 in 28a309c. I removed the Detect check about a flaky marker hiding a failure in the test logic, because flaky and bug apply whether the test or the feature causes the failure.

The review learning workflow answers once a day. It does not read this reply.

### META-010: Activate a test only for the versions and weblogs that have its behavior.
- Severity: P0
- Flag: A manifest version range must match the releases that implement the tested behavior. If a later release changes the behavior, close the range of the old variant. If a version is validated on one weblog only, do not apply it to every weblog.
- Detect: A manifest activates a test from a version that lacks the behavior. A legacy variant has no upper bound, although newer releases use a new behavior. A change lowers a minimum version for all weblogs after validation on one weblog.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one seems useless : if you activate a test for a version/weblog does not have the behavior, the CI will fail, no ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discarded META-010 in 28a309c: the rule is back as it is on main. The reviewer says the rule is useless because CI already fails when a test runs for a version or weblog that lacks the behavior, so the rule is removed from this pull request.

The review learning workflow answers once a day. It does not read this reply.

- ❌ Poll for traces in `test_*` after the containers have stopped, or assert expected trace contents from `setup_*`.
- Ref: [End-to-end test flow](../../../docs/understand/test-flow.md#end-to-end-scenario), [Flushing default behavior](../../../docs/edit/flushing.md#default-behavior)

### LIFE-005: Do not tie exact counts to flush timing.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is slippery : many test relies on time condition in a way or another. I'm afraid this rules will cause some harm.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discarded LIFE-005 in 60feb35: the rule is back as it is on main. The reviewer thinks this rule can harm many tests that depend on timing, so this change to the rule is dropped from this pull request.

The review learning workflow answers once a day. It does not read this reply.

- Severity: P1
- Flag: When checking one protocol flag, mask that bit and accept other bits the protocol allows, unless the whole field is under test. When checking one member of a list field, such as a `tracestate` vendor or an `x-datadog-tags` entry, find that member. Do not require a count or an order that the protocol does not define. When checking for redaction leaks, use a distinctive marker or the relevant parsed field so unrelated payload values cannot count as leaked input.
- Detect: A sampled-bit assertion requires exact flags `01`. A test requires `len(tracestate) == 1` or compares `x-datadog-tags` as a raw string. A redaction check searches an entire payload for a short marker such as `1234`.
- Detect: A sampled-bit assertion requires exact flags `01`. A test requires `len(tracestate) == 1` or compares `x-datadog-tags` as a raw string. A redaction check searches an entire payload for a short marker such as `1234`. A test compares serialized JSON as a string instead of parsed values. A test requires one stats request, one bucket, or an exact metric count, but the tracer flushes on a timer.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edited TEST-001 in a158884. I rewrote the rule in shorter, plainer sentences so its intent is clear, and kept its meaning, severity, and examples.

The review learning workflow answers once a day. It does not read this reply.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant