Repository navigation
Add cluster remote client so11y metrics - #14245
songzhendong wants to merge 2 commits into
Conversation
808c1e0 to
9736715
Compare
wu-sheng
left a comment
There was a problem hiding this comment.
Thanks for this. Getting the cluster metrics into the so11y MAL rules and dashboards is a useful addition. Three points:
1. Please drop the reason label from remote_out_error_count.
Every failure behind those reasons is already logged with its exception, and the exception carries the gRPC status: onError logs each failed stream at ERROR (RESOURCE_EXHAUSTED at WARN, once per 120s), and the consume catch, the interrupted slot wait and the self client all log at ERROR. Deployments keep OAP's error logs, so the metric only needs to show that pushes to a peer are failing and how often; the logs say why.
Keeping remote_out_error_count{dest, self} as it is today also avoids changing the label set of an existing metric, and avoids pre-creating 8 series per client. Some values also don't fit this link: auth_failed can never fire, because RemoteServiceHandler is registered without an authentication interceptor, and rate_limited / rejected don't describe what RESOURCE_EXHAUSTED / CANCELLED mean.
This touches RemoteClientMetrics (the reason constants, reasonOf, the pre-created counters), RemoteClientMetricsTest, oap.yaml, oap.data.yaml, dashboards-so11y.md and the two Grafana JSON files. The Horizon UI PR (apache/skywalking-horizon-ui#222) already sums by dest only, so it is unaffected.
2. Please keep counting errors in onError.
On master, onError logs async failures but never increments remote_out_error_count, so deadline-exceeded and unavailable peers, the most common real failures, never reach the metric. Counting them there (without a reason) is a real fix.
3. Changelog and PR description wording.
remote_in_count, remote_in_error_count, remote_in_target_not_found_count, remote_in_latency, remote_out_count, remote_out_error_count and cluster_size have been exported on the Prometheus endpoint since #2133, and the Grafana dashboards already chart the remote in/out counts. What this PR adds is getting them into the so11y MAL rules and dashboard, plus the new client metrics. Suggested changelog line:
* Add the existing cluster remote metrics (remote in/out counts, errors, in latency, and cluster size) to the OAP self-observability MAL rules and dashboards, add remote client out latency, in-flight streams, stream-slot wait time and send-queue usage metrics, and count asynchronous gRPC send failures in `remote_out_error_count`, which were only logged before.
|
Thanks for the review. Addressed all three points:
Fork CI is green after the change: Pushed to this PR branch as 159b2cb. |



Summary
Add the existing cluster remote metrics (remote in/out counts, errors, in latency, and cluster size) to the OAP self-observability MAL rules and dashboards, add remote client out latency, in-flight streams, stream-slot wait time and send-queue usage metrics, and count asynchronous gRPC send failures in
remote_out_error_count, which were only logged before.Companion Horizon UI: apache/skywalking-horizon-ui#222
Test plan
GRPCRemoteClient/RemoteClientMetrics/SelfRemoteClientFork CI
reason, keeponErrorcounting): WIP: drop remote_out_error reason labels (review) songzhendong/skywalking#36