Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .claude/skills/test/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,19 @@ CI runs this on JDK 11, 17, 21, 25 (ubuntu only).

### `slow` — slow integration tests

The slow matrix has two entries, each building the whole project once:

```bash
# ElasticSearch / OpenSearch: the Elasticsearch client against Elasticsearch 7, 8 and 9, and OpenSearch 1 and 2
./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true \
-Dit.test=org.apache.skywalking.library.elasticsearch.ElasticSearchIT \
-Dfailsafe.failIfNoSpecifiedTests=false

# Elastic IT: the storage-elasticsearch-plugin ITs tagged @Tag("elastic-it") (Zipkin query, span-attached events)
./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -Dgroups=elastic-it
```

CI runs on JDK 11 (ubuntu only). Currently only ElasticSearch/OpenSearch IT is in the slow matrix.
CI runs on JDK 11 (ubuntu only).

### Module name — single module tests

Expand Down Expand Up @@ -64,7 +70,7 @@ CI runs on JDK 11 (ubuntu only). Currently only ElasticSearch/OpenSearch IT is i

## Slow test tagging

Tests tagged with `@Tag("slow")` are excluded from the normal integration-test job and run separately in the slow-integration-test job. Use this tag for tests that spin up external services (Elasticsearch, etc.) and take significant time.
Tests tagged with `@Tag("slow")` are excluded from the normal integration-test job and run separately in the slow-integration-test job. Use this tag for tests that spin up external services (Elasticsearch, etc.) and take significant time. A slow IT must also be selected by a slow-matrix entry (by class, or by a second tag such as `@Tag("elastic-it")`), or it never runs in CI.

## CI retry behavior

Expand Down
27 changes: 19 additions & 8 deletions .github/workflows/skywalking.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,13 @@ jobs:
- name: Unit test
run: ./mvnw clean test -q -B -D"checkstyle.skip" || ./mvnw clean test -q -B -D"checkstyle.skip"

# Every `*IT` not tagged `@Tag("slow")`, on each supported JDK. Each IT starts its own server in Docker
# (Testcontainers):
# - BanyanDB storage: storage-banyandb-plugin's `BanyanDB*IT`, OAP's BanyanDB DAOs on a BanyanDB container.
# - JDBC storage: storage-jdbc-hikaricp-plugin's `JDBC*PostgreSQLIT` and `JDBC*MariaDBIT`, OAP's JDBC DAOs on
# PostgreSQL and on MariaDB, and `JDBCHistoryDeleteDAOIT`.
# - Cluster coordinators and dynamic configuration: the ZooKeeper, etcd, Consul, Nacos and Apollo plugins' ITs.
# - library-client's Elasticsearch bulk processor IT.
integration-test:
if: |
( always() && ! cancelled() ) &&
Expand Down Expand Up @@ -294,7 +301,13 @@ jobs:
./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -DexcludedGroups=slow || \
./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -DexcludedGroups=slow

# Slow tests
# Slow tests: the `@Tag("slow")` ITs, all on Elasticsearch and OpenSearch containers, on JDK 11. Each matrix entry
# builds the whole project once and runs its own ITs:
# - `ElasticSearchIT`, by class: library-elasticsearch-client against Elasticsearch 7, 8 and 9, and OpenSearch 1
# and 2.
# - Elastic IT, by the `@Tag("elastic-it")` the storage-elasticsearch-plugin's ITs also carry: the Zipkin query DAO
# (`ZipkinQueryEsDAO*IT`) and the span-attached event queries (`SpanAttachedEventQuery*IT`), each on
# Elasticsearch 8, 9 and 9 sharded, and OpenSearch 2, 3 and 3 sharded.
slow-integration-test:
if: |
( always() && ! cancelled() ) &&
Expand All @@ -307,11 +320,9 @@ jobs:
matrix:
test:
- name: ElasticSearch / OpenSearch
class: org.apache.skywalking.library.elasticsearch.ElasticSearchIT
- name: Zipkin query on ElasticSearch / OpenSearch
class: ZipkinQueryEsDAO*IT
- name: Span attached events on ElasticSearch / OpenSearch
class: SpanAttachedEventQuery*IT
select: -Dit.test=org.apache.skywalking.library.elasticsearch.ElasticSearchIT -Dfailsafe.failIfNoSpecifiedTests=false
- name: Elastic IT
select: -Dgroups=elastic-it
steps:
- uses: actions/checkout@v7
with:
Expand All @@ -329,8 +340,8 @@ jobs:
distribution: temurin
- name: ${{ matrix.test.name }}
run: |
./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -Dit.test=${{ matrix.test.class }} -Dfailsafe.failIfNoSpecifiedTests=false || \
./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -Dit.test=${{ matrix.test.class }} -Dfailsafe.failIfNoSpecifiedTests=false
./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true ${{ matrix.test.select }} || \
./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true ${{ matrix.test.select }}

e2e-test:
if: |
Expand Down
20 changes: 20 additions & 0 deletions docs/en/changes/changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,26 @@
* LAL: save the logs of the OpenTelemetry services that set `service.layer=OTLP_GENERAL` through the new `otlp-general` rule in `lal/default.yaml`; no rule matched that layer, so these logs were dropped. A LAL rule declaring an unregistered or missing layer (names are case-sensitive) is now skipped with a warning at boot and rejected when applied as a runtime rule, instead of loading silently under `UNDEFINED` and never running.
* LAL: a log without a layer that no `layer: auto` rule claims runs the new `fallback: true` auto rules, instead of the rules declared for `GENERAL`, and is dropped when none claims it either, so where such a log is filed is decided by a rule, not hard-coded. The new `default-fallback` rule in `lal/default.yaml` files it under `GENERAL`, as before; edit its extractor to file it under another layer. A fallback rule must set the layer, as any auto rule. A custom `layer: GENERAL` rule written for the logs without a layer must declare `layer: auto` and `fallback: true` and set `layer "GENERAL"` in its extractor; it then runs next to `default-fallback`, as it ran next to `default` before.
* LAL: handle the mini-program error logs with the `miniprogram-errors-wechat` and `miniprogram-errors-alipay` rules in `lal/miniprogram.yaml`, declared for the `WECHAT_MINI_PROGRAM` and `ALIPAY_MINI_PROGRAM` layers, replacing the `layer: auto` rule `miniprogram-errors`. The mini-program SDK v0.5.0 sets `service.layer`, which routes its error logs to these rules; it is now the minimum supported SDK version, since an older SDK's error logs carry no layer and are filed under `GENERAL` without the `error_count` metric.
* Fix bugs of the JDBC storage, `mysql` (MySQL and MariaDB) and `postgresql`. Many showed only when a query's time range crossed midnight, as this storage keeps records and metrics in one table per day:
- The trace list came out of order when its time range crossed midnight, and its pages skipped or repeated traces.
- A Zipkin trace search returned more traces than its limit, and a trace with spans on both days came back as two partial traces, in a search and in a lookup of several trace IDs (`traceMany`).
- A heatmap had extra, zeroed columns.
- A Top N list ranked an entity by the average of its daily averages and could miss the top entities, and it failed on values above 2147483647, such as memory in bytes.
- The slow statement list was not ranked across the days.
- The async-profiler and pprof task lists could leave out the newest tasks.
- A trace profiling task scheduled to start on a later day than it was created could fail to reach the agent, and a new task could be created over it.
- Trace profiling of a Go service read its thread snapshots as Java, and the profile analysis could read thread snapshots outside the selected time range.
- Querying the targets of a continuous profiling policy saved without targets failed with an SQL syntax error, and an eBPF profiling analysis with no schedule logged one.
- The continuous profiling instance list missed instances first seen on an earlier day.
- Rover could receive eBPF profiling tasks for processes that had stopped reporting.
- Tag autocomplete returned more keys or values than its limit.
- Querying browser error logs without a time range failed.
- Filtering events by layer failed on PostgreSQL, and an event at the first millisecond of a time range was left out of it.
- Disabling a UI template always failed with "Can't find the template", and changing a template that does not exist reported success.
- A metric whose row was gone from its table while OAP still cached it was not written again: its update found no row and OAP kept the cached copy. It is now dropped from the cache, so the next flush inserts the row again.
- The table check at boot logged wrong table and column states.
* Fix creating an async-profiler task for a service while another one runs on it: the check that refuses it never found the running task on the JDBC and Elasticsearch storage.
* Fix the async-profiler task OAP sends to an agent whose service has two tasks created within five minutes of each other: it could send the older one.

#### UI

Expand Down
14 changes: 14 additions & 0 deletions oap-server-bom/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@
<commons-beanutils.version>1.11.0</commons-beanutils.version>
<flatbuffers-java.version>1.12.0</flatbuffers-java.version>
<postgresql.version>42.7.14</postgresql.version>
<mariadb-java-client.version>3.5.1</mariadb-java-client.version>
<jetcd.version>0.6.1</jetcd.version>
<testcontainers.version>1.17.6</testcontainers.version>
<armeria.version>1.34.2</armeria.version>
Expand Down Expand Up @@ -557,6 +558,13 @@
<artifactId>postgresql</artifactId>
<version>${postgresql.version}</version>
</dependency>
<!-- LGPL: for the MariaDB storage ITs only, never distributed. -->
<dependency>
<groupId>org.mariadb.jdbc</groupId>
<artifactId>mariadb-java-client</artifactId>
<version>${mariadb-java-client.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.linecorp.armeria</groupId>
Expand Down Expand Up @@ -623,6 +631,12 @@
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>mariadb</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,19 @@ public AsyncProfilerTask getAsyncProfilerTask(String serviceId) {
return serviceId2taskCache.getIfPresent(serviceId);
}

/**
* Caches the newest task of the service. A refresh lists every task created within five minutes of now, newest
* first, so a task replaces the cached one only when it was created no earlier.
*/
public void saveTask(String serviceId, AsyncProfilerTask task) {
if (task == null) {
return ;
}

serviceId2taskCache.put(serviceId, task);
final AsyncProfilerTask cached = serviceId2taskCache.getIfPresent(serviceId);
if (cached == null || cached.getCreateTime() <= task.getCreateTime()) {
serviceId2taskCache.put(serviceId, task);
}
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,8 @@ private String checkArgumentError(String serviceId,
private String checkTaskProfiling(String serviceId,
long createTime) throws IOException {
// Each service can only enable one task at a time
long endTimeBucket = TimeBucket.getMinuteTimeBucket(createTime);
// Task rows carry the second bucket of their creation (createTask), so bound them by a second bucket too.
final long endTimeBucket = TimeBucket.getRecordTimeBucket(createTime);
final List<AsyncProfilerTask> alreadyHaveTaskList = getAsyncProfileTaskDAO().getTaskList(
serviceId, null, endTimeBucket, 1
);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.skywalking.oap.server.core.cache;

import org.apache.skywalking.oap.server.core.CoreModuleConfig;
import org.apache.skywalking.oap.server.core.query.type.AsyncProfilerTask;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.mock;

class AsyncProfilerTaskCacheTest {
private static final String SERVICE_ID = "c2VydmljZQ==.1";

/**
* A refresh lists the tasks created within five minutes of now, newest first as the storage DAOs return them: the
* task created a moment ago, then one that just ended. The service's agent must be sent the newer one, whichever
* order the list has.
*/
@Test
void keepsTheNewestTaskOfAServiceWhateverTheListOrder() {
final long now = System.currentTimeMillis();
final AsyncProfilerTask ended = task("ended", now - 180_000L);
final AsyncProfilerTask created = task("created", now - 30_000L);

final AsyncProfilerTaskCache newestFirst = new AsyncProfilerTaskCache(mock(ModuleManager.class), new CoreModuleConfig());
newestFirst.saveTask(SERVICE_ID, created);
newestFirst.saveTask(SERVICE_ID, ended);
assertEquals("created", newestFirst.getAsyncProfilerTask(SERVICE_ID).getId());

final AsyncProfilerTaskCache oldestFirst = new AsyncProfilerTaskCache(mock(ModuleManager.class), new CoreModuleConfig());
oldestFirst.saveTask(SERVICE_ID, ended);
oldestFirst.saveTask(SERVICE_ID, created);
assertEquals("created", oldestFirst.getAsyncProfilerTask(SERVICE_ID).getId());
}

private static AsyncProfilerTask task(final String id, final long createTime) {
return AsyncProfilerTask.builder().id(id).serviceId(SERVICE_ID).createTime(createTime).duration(60).build();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.skywalking.oap.server.core.profiling.asyncprofiler;

import java.util.Collections;
import java.util.List;
import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
import org.apache.skywalking.oap.server.core.query.type.AsyncProfilerEventType;
import org.apache.skywalking.oap.server.core.query.type.AsyncProfilerTask;
import org.apache.skywalking.oap.server.core.query.type.AsyncProfilerTaskCreationResult;
import org.apache.skywalking.oap.server.core.query.type.AsyncProfilerTaskCreationType;
import org.apache.skywalking.oap.server.core.storage.StorageModule;
import org.apache.skywalking.oap.server.core.storage.profiling.asyncprofiler.IAsyncProfilerTaskQueryDAO;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.library.module.ModuleProviderHolder;
import org.apache.skywalking.oap.server.library.module.ModuleServiceHolder;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

class AsyncProfilerMutationServiceTest {
private static final String SERVICE_ID = "c2VydmljZQ==.1";

/**
* A task created ten seconds ago runs for a minute. Its row carries the second bucket of its creation, as
* {@code createTask} stamps it, and the DAO keeps the rows whose time bucket is not past the bound, as the JDBC
* and Elasticsearch DAOs filter.
*/
@Test
void refusesASecondTaskForTheServiceWhileTheFirstRuns() throws Exception {
final long createTime = System.currentTimeMillis() - 10_000L;
final AsyncProfilerTask running = AsyncProfilerTask.builder()
.id(createTime + "_" + SERVICE_ID)
.serviceId(SERVICE_ID)
.createTime(createTime)
.duration(60)
.build();
final long storedTimeBucket = TimeBucket.getRecordTimeBucket(createTime);
final IAsyncProfilerTaskQueryDAO dao = mock(IAsyncProfilerTaskQueryDAO.class);
when(dao.getTaskList(eq(SERVICE_ID), any(), anyLong(), any())).thenAnswer(invocation -> {
final long endTimeBucket = invocation.getArgument(2);
return storedTimeBucket <= endTimeBucket
? Collections.singletonList(running) : Collections.<AsyncProfilerTask>emptyList();
});
final ModuleManager moduleManager = mock(ModuleManager.class);
final ModuleProviderHolder providerHolder = mock(ModuleProviderHolder.class);
final ModuleServiceHolder serviceHolder = mock(ModuleServiceHolder.class);
when(moduleManager.find(StorageModule.NAME)).thenReturn(providerHolder);
when(providerHolder.provider()).thenReturn(serviceHolder);
when(serviceHolder.getService(IAsyncProfilerTaskQueryDAO.class)).thenReturn(dao);

final AsyncProfilerTaskCreationResult result = new AsyncProfilerMutationService(moduleManager).createTask(
SERVICE_ID, List.of("instance-1"), 60, List.of(AsyncProfilerEventType.CPU), "");

assertEquals(AsyncProfilerTaskCreationType.ALREADY_PROFILING_ERROR, result.getCode());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import org.testcontainers.junit.jupiter.Testcontainers;

@Tag("slow")
@Tag("elastic-it")
@Testcontainers
public class SpanAttachedEventQueryElasticsearch8IT extends SpanAttachedEventQueryITBase {
@Container
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import org.testcontainers.junit.jupiter.Testcontainers;

@Tag("slow")
@Tag("elastic-it")
@Testcontainers
public class SpanAttachedEventQueryElasticsearch9IT extends SpanAttachedEventQueryITBase {
@Container
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import org.testcontainers.junit.jupiter.Testcontainers;

@Tag("slow")
@Tag("elastic-it")
@Testcontainers
public class SpanAttachedEventQueryElasticsearch9ShardedIT extends SpanAttachedEventQueryITBase {
@Container
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import org.testcontainers.junit.jupiter.Testcontainers;

@Tag("slow")
@Tag("elastic-it")
@Testcontainers
public class SpanAttachedEventQueryOpenSearch2IT extends SpanAttachedEventQueryITBase {
@Container
Expand Down
Loading
Loading