Skip to content

Add cluster remote client so11y metrics - #14245

Open
songzhendong wants to merge 2 commits into
apache:masterfrom
songzhendong:feat/cluster-remote-so11y
Open

songzhendong wants to merge 2 commits into
apache:masterfrom
songzhendong:feat/cluster-remote-so11y

Conversation

@songzhendong

@songzhendong songzhendong commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add the existing cluster remote metrics (remote in/out counts, errors, in latency, and cluster size) to the OAP self-observability MAL rules and dashboards, add remote client out latency, in-flight streams, stream-slot wait time and send-queue usage metrics, and count asynchronous gRPC send failures in remote_out_error_count, which were only logged before.

Companion Horizon UI: apache/skywalking-horizon-ui#222

Test plan

Fork CI

@songzhendong

songzhendong commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

UI screenshots (local verification)

Grafana cluster

Grafana cluster Remote panels

Grafana instance

Grafana instance Remote panels

Horizon OAP so11y

Horizon cluster remote widgets

@songzhendong
songzhendong force-pushed the feat/cluster-remote-so11y branch from 808c1e0 to 9736715 Compare November 18, 2026 01:15

@wu-sheng wu-sheng left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Getting the cluster metrics into the so11y MAL rules and dashboards is a useful addition. Three points:

1. Please drop the reason label from remote_out_error_count.

Every failure behind those reasons is already logged with its exception, and the exception carries the gRPC status: onError logs each failed stream at ERROR (RESOURCE_EXHAUSTED at WARN, once per 120s), and the consume catch, the interrupted slot wait and the self client all log at ERROR. Deployments keep OAP's error logs, so the metric only needs to show that pushes to a peer are failing and how often; the logs say why.

Keeping remote_out_error_count{dest, self} as it is today also avoids changing the label set of an existing metric, and avoids pre-creating 8 series per client. Some values also don't fit this link: auth_failed can never fire, because RemoteServiceHandler is registered without an authentication interceptor, and rate_limited / rejected don't describe what RESOURCE_EXHAUSTED / CANCELLED mean.

This touches RemoteClientMetrics (the reason constants, reasonOf, the pre-created counters), RemoteClientMetricsTest, oap.yaml, oap.data.yaml, dashboards-so11y.md and the two Grafana JSON files. The Horizon UI PR (apache/skywalking-horizon-ui#222) already sums by dest only, so it is unaffected.

2. Please keep counting errors in onError.

On master, onError logs async failures but never increments remote_out_error_count, so deadline-exceeded and unavailable peers, the most common real failures, never reach the metric. Counting them there (without a reason) is a real fix.

3. Changelog and PR description wording.

remote_in_count, remote_in_error_count, remote_in_target_not_found_count, remote_in_latency, remote_out_count, remote_out_error_count and cluster_size have been exported on the Prometheus endpoint since #2133, and the Grafana dashboards already chart the remote in/out counts. What this PR adds is getting them into the so11y MAL rules and dashboard, plus the new client metrics. Suggested changelog line:

* Add the existing cluster remote metrics (remote in/out counts, errors, in latency, and cluster size) to the OAP self-observability MAL rules and dashboards, add remote client out latency, in-flight streams, stream-slot wait time and send-queue usage metrics, and count asynchronous gRPC send failures in `remote_out_error_count`, which were only logged before.

@songzhendong

songzhendong commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Addressed all three points:

  1. Dropped the reason label from remote_out_error_count. It is back to {dest, self} only (metrics, MAL, Grafana, docs, and tests). Failure details stay in the existing ERROR/WARN logs.
  2. Kept counting asynchronous failures in onError via recordError() (no reason).
  3. Updated the changelog and PR description to the suggested wording.

Fork CI is green after the change:
songzhendong#36

Pushed to this PR branch as 159b2cb.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants