Repository navigation
Fix alarm windows retaining initially unresolved layers - #14246
Merged
Merged
Conversation
NotifyHandler resolves layers through MetadataQueryService for incoming metric notifications. Metrics reach alarm rules during persistence preparation, so a new service's first notification can arrive before its traffic metadata is queryable or visible in the metadata cache. RunningRule reuses windows keyed by AlarmEntity, whose equality excludes layers. When the first lookup returned no layers, the window retained that entity even after later lookups succeeded. Subsequent firing messages therefore kept an empty layer list, persisted as UNDEFINED, and did not match queries filtering for the service's actual layer. The window now fills missing layers from a later notification with resolved layers. The replacement entity is volatile because the alarm check runs on another thread. This affects subsequently built firing messages; it does not backfill existing records or refresh non-empty layer lists.
wankai123
approved these changes
Nov 17, 2026
4 of 5 tasks
wu-sheng
added a commit
to apache/skywalking-graalvm-distro
that referenced
this pull request
Nov 17, 2026
…er) and build with GraalVM 25.5 (#46) Upstream moved 22 commits past the v11.1.0 tag (d34eff29ba..ab9dceee9d, 11.2.0-SNAPSHOT): - SWIP-17 (apache/skywalking#14208) moves agent-segment analysis into seven sw-native TAL files and adds otlp-ai-evaluation for OTLP spans. Including otlp-general, all nine files are pre-compiled into 14 expressions; the configured file lists match upstream, and with default modules the two AI-evaluation files are skipped, leaving two OTLP rules and ten native rules active. Generated TAL classes call functions through function-object fields upstream sets from a TALFunctionRegistry after instantiating the class; the precompiler records each field's registry key and the same-FQCN TalDSL sets the same objects from the factory's registry at boot. TALConfigs honors requiredModules and loadInOrder; the precompiler gains the ai-evaluation dependency. - The query-server module (#14238) is wired; GraphQL, the AI agent conversation API and /status/config/ttl mount on it. The GraphQL query protocol adds TraceScopeCondition.otlpSpanId (#14213). - LAL fallback rules (#14244), the alarm module's grpcTimeout, the e2e pins and cases follow upstream. - apache/skywalking#14246 fixes the alarm race this sync's native e2e exposed: an alarm window kept the layers of its entity's first metric, so a first metadata lookup miss left every alarm of the window without a layer. The native image is built with GraalVM 25.5: CI selects Oracle GraalVM 25.5.5.0.3 through setup-graalvm version '25.5'; local auto-detection prefers sdkman Oracle 25.5, then CE 25.5, with fallback and override paths; the Linux cross-build uses native-image-community:25i5. The precompiler now fails the build where upstream fails the boot: a MAL, Zabbix, LAL or hierarchy rule that does not load or compile, a LAL inputType/outputType that does not resolve, a missing hierarchy definition or its auto-matching-rules section. NativeTALComparisonTest runs every native rule over upstream's native-trace checker fixtures through SwNativeRuleRunner, pre-compiled vs fresh compile.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix alarm windows retaining initially unresolved layers
NotifyHandlerresolves layers throughMetadataQueryServicefor incoming metric notifications. Metrics reach alarm rules during persistence preparation, so a new service's first notification can arrive before its traffic metadata is queryable or visible in the metadata cache.RunningRulereuses windows keyed byAlarmEntity, whose equality excludes layers. When the first lookup returned no layers, the window retained that entity even after later lookups succeeded. Subsequent firing messages therefore kept an empty layer list, persisted asUNDEFINED, and did not match queries filtering for the service's actual layer.The window now fills missing layers from a later notification with resolved layers. The replacement entity is volatile because the alarm check runs on another thread. This affects subsequently built firing messages; it does not backfill existing records or refresh non-empty layer lists.
RunningRuleTest#testAlarmTakesTheLayersResolvedAfterTheFirstMetricfeeds a first metric without layers and later ones withGENERAL; it fails without the fix (expected: <[GENERAL]> but was: <[]>).Found by the GraalVM distro's alarm e2e using upstream's
test/e2e-v2/cases/alarm/alarm-cases.yaml: theGENERALlayer-filtered query returned no alarms in 3/3 local runs, while the unfiltered, entity-filtered and rule-name-filtered queries passed. In an instrumented run, the first observedServiceRespTimeMetricsaddition toRunningRulewas at 18:14:11. AlistServicespoll saw no service at 18:14:03 and saw the provider withGENERALat 18:14:24; 12 alarms followed, none matching theGENERALfilter. These observations are consistent with an initial metadata lookup miss being retained by the window. Upstream JVM alarm e2e passes atae9ea84636on BanyanDB, Elasticsearch, MySQL and PostgreSQL. A manually started native stack using the same Compose file also produced alarms that all matchedGENERAL. With this change applied to the native OAP, the distro's alarm e2e passes 21/21, the layer-filtered case included.CHANGESlog.