Skip to content

Fix alarm windows retaining initially unresolved layers - #14246

Merged
wu-sheng merged 1 commit into
masterfrom
fix/alarm-window-layers
Nov 17, 2026
Merged

wu-sheng merged 1 commit into
masterfrom
fix/alarm-window-layers

Conversation

@wu-sheng

Copy link
Copy Markdown
Member

Fix alarm windows retaining initially unresolved layers

  • Add a unit test to verify that the fix works.
  • Explain briefly why the bug exists and how to fix it.

NotifyHandler resolves layers through MetadataQueryService for incoming metric notifications. Metrics reach alarm rules during persistence preparation, so a new service's first notification can arrive before its traffic metadata is queryable or visible in the metadata cache.

RunningRule reuses windows keyed by AlarmEntity, whose equality excludes layers. When the first lookup returned no layers, the window retained that entity even after later lookups succeeded. Subsequent firing messages therefore kept an empty layer list, persisted as UNDEFINED, and did not match queries filtering for the service's actual layer.

The window now fills missing layers from a later notification with resolved layers. The replacement entity is volatile because the alarm check runs on another thread. This affects subsequently built firing messages; it does not backfill existing records or refresh non-empty layer lists. RunningRuleTest#testAlarmTakesTheLayersResolvedAfterTheFirstMetric feeds a first metric without layers and later ones with GENERAL; it fails without the fix (expected: <[GENERAL]> but was: <[]>).

Found by the GraalVM distro's alarm e2e using upstream's test/e2e-v2/cases/alarm/alarm-cases.yaml: the GENERAL layer-filtered query returned no alarms in 3/3 local runs, while the unfiltered, entity-filtered and rule-name-filtered queries passed. In an instrumented run, the first observed ServiceRespTimeMetrics addition to RunningRule was at 18:14:11. A listServices poll saw no service at 18:14:03 and saw the provider with GENERAL at 18:14:24; 12 alarms followed, none matching the GENERAL filter. These observations are consistent with an initial metadata lookup miss being retained by the window. Upstream JVM alarm e2e passes at ae9ea84636 on BanyanDB, Elasticsearch, MySQL and PostgreSQL. A manually started native stack using the same Compose file also produced alarms that all matched GENERAL. With this change applied to the native OAP, the distro's alarm e2e passes 21/21, the layer-filtered case included.

NotifyHandler resolves layers through MetadataQueryService for incoming metric notifications. Metrics reach alarm
rules during persistence preparation, so a new service's first notification can arrive before its traffic metadata is
queryable or visible in the metadata cache.

RunningRule reuses windows keyed by AlarmEntity, whose equality excludes layers. When the first lookup returned no
layers, the window retained that entity even after later lookups succeeded. Subsequent firing messages therefore kept
an empty layer list, persisted as UNDEFINED, and did not match queries filtering for the service's actual layer.

The window now fills missing layers from a later notification with resolved layers. The replacement entity is
volatile because the alarm check runs on another thread. This affects subsequently built firing messages; it does not
backfill existing records or refresh non-empty layer lists.
@wu-sheng wu-sheng added this to the 11.2.0 milestone Nov 17, 2026
@wu-sheng wu-sheng added bug Something isn't working and you are sure it's a bug! backend OAP backend related. labels Nov 17, 2026
@wu-sheng
wu-sheng merged commit ab9dcee into master Nov 17, 2026
248 checks passed
@wu-sheng
wu-sheng deleted the fix/alarm-window-layers branch November 17, 2026 12:36
wu-sheng added a commit to apache/skywalking-graalvm-distro that referenced this pull request Nov 17, 2026
…er) and build with GraalVM 25.5 (#46)

Upstream moved 22 commits past the v11.1.0 tag (d34eff29ba..ab9dceee9d, 11.2.0-SNAPSHOT):

- SWIP-17 (apache/skywalking#14208) moves agent-segment analysis into seven sw-native TAL files and adds
  otlp-ai-evaluation for OTLP spans. Including otlp-general, all nine files are pre-compiled into 14 expressions;
  the configured file lists match upstream, and with default modules the two AI-evaluation files are skipped,
  leaving two OTLP rules and ten native rules active. Generated TAL classes call functions through function-object
  fields upstream sets from a TALFunctionRegistry after instantiating the class; the precompiler records each
  field's registry key and the same-FQCN TalDSL sets the same objects from the factory's registry at boot.
  TALConfigs honors requiredModules and loadInOrder; the precompiler gains the ai-evaluation dependency.
- The query-server module (#14238) is wired; GraphQL, the AI agent conversation API and /status/config/ttl mount on
  it. The GraphQL query protocol adds TraceScopeCondition.otlpSpanId (#14213).
- LAL fallback rules (#14244), the alarm module's grpcTimeout, the e2e pins and cases follow upstream.
- apache/skywalking#14246 fixes the alarm race this sync's native e2e exposed: an alarm window kept the layers of its
  entity's first metric, so a first metadata lookup miss left every alarm of the window without a layer.

The native image is built with GraalVM 25.5: CI selects Oracle GraalVM 25.5.5.0.3 through setup-graalvm
version '25.5'; local auto-detection prefers sdkman Oracle 25.5, then CE 25.5, with fallback and override paths;
the Linux cross-build uses native-image-community:25i5.

The precompiler now fails the build where upstream fails the boot: a MAL, Zabbix, LAL or hierarchy rule that does
not load or compile, a LAL inputType/outputType that does not resolve, a missing hierarchy definition or its
auto-matching-rules section.

NativeTALComparisonTest runs every native rule over upstream's native-trace checker fixtures through
SwNativeRuleRunner, pre-compiled vs fresh compile.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend OAP backend related. bug Something isn't working and you are sure it's a bug!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants