Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/en/changes/changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@
* Fix the async-profiler task OAP sends to an agent whose service has two tasks created within five minutes of each other: it could send the older one.
* Fix alarm windows retaining empty layers after an initial metadata lookup miss. Later metrics can supply the missing layers for subsequent alarms, allowing layer-filtered `queryAlarms` queries to match the persisted layer.
* Rename the etcd dynamic configuration password environment variable from `SW_CONFIG_ETCD_password` to `SW_CONFIG_ETCD_PASSWORD`, matching every other environment variable. The lower-case name no longer has any effect, so a deployment that sets it must switch to the new name.
* Add the existing cluster remote metrics (remote in/out counts, errors, in latency, and cluster size) to the OAP self-observability MAL rules and dashboards, add remote client out latency, in-flight streams, stream-slot wait time and send-queue usage metrics, and count asynchronous gRPC send failures in `remote_out_error_count`, which were only logged before.

#### UI

Expand Down
11 changes: 11 additions & 0 deletions docs/en/setup/backend/dashboards-so11y.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,17 @@ Self observability monitoring provides monitoring of the status and resources of
| Count per minute | meter_oap_instance_alarm_http_delivery_count | Alarm hook deliveries by `hook_type` and `result` (Per Minute), including `gRPC` | oap self observability |
| Count per minute | meter_oap_instance_alarm_http_delivery_status_count | HTTP alarm hook responses by `hook_type` and `status_class` (Per Minute) | oap self observability |
| ms | meter_oap_instance_alarm_http_delivery_latency_percentile | Alarm hook delivery latency by `hook_type` (ms), including `gRPC` | oap self observability |
| Count per minute | meter_oap_instance_remote_out_count | Cluster remote client push count by `dest` and `self` (Per Minute); peers: send attempts, `self=Y`: successful local fan-out | oap self observability |
| Count per minute | meter_oap_instance_remote_out_error_count | Cluster remote client push errors by `dest` and `self` (Per Minute); one sample per failed stream | oap self observability |
| ms | meter_oap_instance_remote_out_latency_percentile | Cluster remote client push latency by `dest` and `self` (ms, one sample per successful stream) | oap self observability |
| Count | meter_oap_instance_remote_out_stream_inflight | In-flight remote gRPC streams by `dest` and `self` (peers only) | oap self observability |
| Milliseconds per minute | meter_oap_instance_remote_out_stream_block_milliseconds | Time waiting for a free remote stream slot by `dest` and `self` (ms Per Minute, peers only) | oap self observability |
| Count | meter_oap_instance_remote_out_queue_used_per_ten_thousand | Remote send queue used percentage ×100 by `dest` and `self` (peers only; divide by 100 for 0–100%) | oap self observability |
| Count per minute | meter_oap_instance_remote_in_count | Cluster remote server receive count (Per Minute) | oap self observability |
| Count per minute | meter_oap_instance_remote_in_error_count | Cluster remote server receive errors (Per Minute) | oap self observability |
| Count per minute | meter_oap_instance_remote_in_target_not_found_count | Cluster remote server target-worker-not-found count (Per Minute) | oap self observability |
| ms | meter_oap_instance_remote_in_latency_percentile | Cluster remote server receive latency (ms, includes discarded messages) | oap self observability |
| Count | meter_oap_instance_cluster_size | OAP cluster size seen by the current node | oap self observability |

## Customizations
You can customize your own metrics/expression/dashboard panel.
Expand Down
Loading