Skip to content

Test every JDBC and Elasticsearch storage DAO and the Elasticsearch client on each request format, and fix what the tests found - #14252

Open
wu-sheng wants to merge 69 commits into
masterfrom
test/jdbc-es-dao-coverage
Open

wu-sheng wants to merge 69 commits into
masterfrom
test/jdbc-es-dao-coverage

Conversation

@wu-sheng

Copy link
Copy Markdown
Member

Test every JDBC and Elasticsearch storage DAO and the Elasticsearch client on each request format, and fix what the tests found

  • Add a unit test to verify that the fix works.
  • Explain briefly why the bug exists and how to fix it.

The BanyanDB storage has an IT for every DAO method OAP can reach (#14216). This PR brings the JDBC and Elasticsearch storages to the same coverage. It also tests the Elasticsearch client on a server of each request format it picks by version, and runs the Elasticsearch e2e cases on Elasticsearch 9. Each fix below has a test that failed before the fix and passes after it.

Storage DAO ITs. Each IT installs the schema the way OAP does at boot, writes through OAP's own write path, and reads through the DAO the storage provider registers. It compares whole results: near misses for every filter, rows on and just outside the time bounds, paging, order and limits. Rows sit on both sides of midnight, so a read crosses two day tables or two daily indices. Every DAO method of both storages is now called by an IT, and no IT is disabled.

  • JDBC: PostgreSQL 13 and MariaDB 11.4, with one JDBC<Area>ITBase and a subclass per database.
  • Elasticsearch: one IT per server, with each of its 22 areas as a nested class, so each server starts once and each area writes under its own index namespace.
    • Servers: Elasticsearch 8.18.8 and 9.3.0, OpenSearch 1.3.20, 2.8.0 and 3.0.0, and Elasticsearch 9.3.0 and OpenSearch 3.0.0 with logic sharding and custom routing.
    • Writes refresh their indices.
    • A new installer, with nothing cached, must find every model an IT installs, as on OAP's next boot.
    • The Zipkin and span-attached-event ITs moved into this layout.

Elasticsearch client IT. It has one nested class per server version. Each sends every request OAP's storage sends, in the shape OAP sends it:

  • index templates with their alias and order;
  • indices and appended mappings, read back for the installer's comparison;
  • refreshed single-document writes, with ids that need escaping in a path;
  • bulk inserts, updates and routed inserts;
  • every query and aggregation shape the DAOs build;
  • routed searches;
  • searches over day indices of which some are missing;
  • scrolls, with their context cleared.

The versions cover each request format the client picks:

Client request format Versions
Elasticsearch 6 6.8.23 (added)
Elasticsearch 7.0-7.7, legacy index templates 7.7.1 (was 7.4.2)
Elasticsearch 7.8-7.x, composable index templates 7.8.0, 7.17.29 (was 7.15.0)
Elasticsearch 8 and later 8.19.25, 9.3.0 (8.9.0 removed)
OpenSearch 1.3.20, 2.8.0, 3.0.0 (3.0.0 added; 2.4.0 removed)

Each server starts once for all its tests, so the job takes about 3 minutes instead of 20.

e2e.

  • Every case that stores in Elasticsearch now takes ES_VERSION and runs on 8.18.8 and 9.3.0, adding 25 jobs. Before, only the storage case ran on 9.3.0. Outside CI the cases default to 8.18.8.
  • The OTLP native trace case keeps 7.15.0 as well.
  • The Elasticsearch monitoring case does not store in Elasticsearch and stays on 8.18.8.
  • The mini-program log expectations now quote content with printf "%q". A log holding a ' broke the expected YAML, and when it was the oldest log the verifier failed the case, about 1 run in 4.

Fixes.

Elasticsearch client:

  • indexTemplateOrder reached the server only on Elasticsearch 6. A legacy template now sends its order, a composable template its priority, and both are read back as 64-bit values. A template stored with another order is updated at boot. A legacy template keeps a negative order; a composable priority cannot be negative, so it takes 0.
  • Document ids went into request paths unescaped. An id holding /, ? or # could not be written; examples are a runtime rule named like aws-gateway/gateway-service, or the continuous-profiling policy of a service whose Base64 id holds /. The server decoded a percent escape, storing a%20b as a b, which a read by id never found; such a document, written by an earlier version, is not found after the upgrade. Horizon's template names never hold these characters. A + still reaches the server as +, because the HTTP client decodes %2B, and Elasticsearch before 7.4 reads it as a space.
  • A delete reports a missing document instead of throwing. A management delete is now one request: a refused delete reaches the caller as an IOException instead of passing for a deletion, and the existence check before it, which hid failures and raced other deletes, is gone.
  • A legacy template was read back without its name.

Elasticsearch DAOs:

  • Topology kept one component of a relation whose component changed within the range. Its component sub-aggregations also stopped at 10 values.
  • An instance's hierarchy stopped at 10 relations.
  • The JFR and pprof data reads stopped at the default 10 hits. They now read every upload, one per page, so large uploads never exceed the client's 10 MiB response limit. A pprof read with no instance listed now reads every instance.
  • getProcess never found a process in the default (merged) mode.
  • Tag autocomplete had four faults: it returned fewer values than its limit when a value was seen on several days; it returned other days' tags when dayStep > 1; it read every page before cutting; and it failed for a limit of 0.
  • An eBPF task query with no target types returned every task of the service.
  • The event query's endpoint filter used a field that the event mapping does not have.
  • Alarm recovery times were lost past the 10th alarm of a page.
  • The TTL deletion's skip cache was read by model name and written by table name, so it never skipped.

Both storages:

  • End instant. A query's end is inclusive: Duration#getEndTimestamp() is the start of the next step, and a row exactly on it is read, as on BanyanDB. Most reads compared second buckets against the end step and chose day tables and indices by the end step. So they left out a row on the end instant, and a range ending in a day's last minute never read the next day, where that end lies. These reads now filter their millisecond column through the end instant and read that day: logs, browser error logs, alarms, GenAI evaluations, AI agent rounds, events, the trace list, sampled records, and the JDBC Zipkin and OTLP searches. Tag autocomplete is stored per day, so honoring the end instant would add a whole day; it reads the days of the requested steps, as BanyanDB does.
  • listInstances and findEndpoint returned rows first seen in the minute after the range.
  • listProcesses of an instance compared hour or day buckets with minute ping buckets.
  • The process topology kept only one client-side component of a call (ProcessTopologyBuilder).

JDBC:

  • The Zipkin search and the OTLP latest-first query ranked a trace per day table and merged the results, so a trace with spans on both sides of midnight could take another trace's place. Both now rank in one query: a UNION ALL of each day table's per-trace minimum (or maximum, for the longest-first order), grouped again by trace and only then sorted and cut. It is exact for both orders. A range within one day table is ranked in that table's query alone, and the longest-first order limits each table's ranking too: a trace among the longest of the range is among the longest of the table holding its longest span.
  • isExpiredCache counted a metric on the last dropped day as alive.
  • A network alias updated in the asked minute was left out.
  • eBPF schedules did not come back newest first.

Profiling task rows:

  • Elasticsearch now writes a row into the daily index of its time bucket's day, whatever the bucket's precision. Async-profiler, pprof and continuous-profiling tasks carry a second-precision bucket on a minute model, so they went into an index per hour that the TTL never deleted, and could not be written at all with dayStep > 1. The TTL now deletes those hour-named indices with their day, so an upgrade needs no manual cleanup. Async-profiler and pprof task rows keep the second buckets earlier versions wrote, so rows from before and after an upgrade are read the same way; an eBPF task that a continuous-profiling policy triggers now carries the minute it is created in, as its model's downsampling says.
  • The eBPF keep-alive task copy carried time bucket 0, an index dated 0 that the next TTL run deleted. It now carries the minute it is written.
  • BanyanDB's task reads missed a task in a range's last time bucket.
  • The trace profiling conflict check read the latest task by whole minutes, so on JDBC and Elasticsearch, and now on BanyanDB, it refused a new task when another starts later in the minute the new one ends. It now reads the latest two tasks and compares both ends; a service's tasks never overlap and last a minute or more, so at most one starts in a minute.

Accepted Elasticsearch behavior, documented at the aggregations.

  • Top-N (sortMetrics) is a terms aggregation ordered by avg. The Zipkin and OTLP latest-first searches order by min descending.
  • Elastic documents that sorting terms by a sub-aggregation "generally produces incorrect ordering" across shards. Each shard returns its own top shard_size, which is size * 1.5 + 10 by default.
  • It is exact when one shard holds the data. Across several indices or shards, an entity can be missed or ranked by part of its data.
  • Reading every entity, or making extra requests, was judged not worth the cost.
  • The ES ITs assert what the servers return for these cases. The JDBC ITs assert the exact answer for the same fixtures.

Validation (local, JDK 11, on the final commit):

  • Elasticsearch DAO ITs: 2,100 run, 0 failed, 0 skipped.

  • JDBC DAO ITs: 583 run, 0 failed, 0 skipped.

  • BanyanDB ITs: 189 run, 0 failed.

  • Elasticsearch client IT: 108 of 108 on the nine servers. Wrapper IT: 28 of 28 on Elasticsearch 6.3.2 and 7.8.0.

  • Unit tests: core 426, eBPF receiver 13, JDBC 123, Elasticsearch 69, BanyanDB 73, Elasticsearch client 12, all passing. Checkstyle passes on every changed module.

  • The e2e changes run in CI.

  • Update the CHANGES log.

Each area's IT is a nested class of the IT of the server it runs on, and writes under a namespace of its own. Writes
refresh their indices instead of waiting for the next refresh.
…e2e expectations

A single-quoted scalar cannot hold a raw quote, and one of the simulator's js errors reads "(reading 'id')". The
verifier renders the expectation once per log; the render of that log failed to parse, and when it was the oldest log
(the warmup error, last in the newest-first list) the verifier expected no element and failed the case. Go's %q
always renders a valid YAML double-quoted scalar.
…torages

Add ITs for every ITopologyQueryDAO and IHierarchyQueryDAO method on
PostgreSQL, MariaDB and the six Elasticsearch / OpenSearch servers. The
relations are written on both sides of midnight, on and just outside the
range bounds, on both detect points, and rolled up per hour in the same
tables and indices; each filter has a near miss.

Fix TopologyQueryEsDAO returning only the most frequent component of a
service or process relation in the range. A relation stores its components
per minute, so a range may hold several; each is now a call of its own, as
the JDBC and BanyanDB DAOs already return them.
JDBC (PostgreSQL, MariaDB): the trace-profiling, async-profiler and pprof
task lookups by id, their task logs, the JFR and pprof data reads, more task
list near misses, and thread-snapshot batches and window bounds.

Elasticsearch and OpenSearch, default and sharded modes: new ProfilingTasks
and ProfileThreadSnapshots ITs on all six servers, covering task lists, ids,
logs, data and thread snapshots.

Fix: the JFR and pprof data reads on Elasticsearch searched without a size
and returned only the first 10 uploads of a task. They now scroll over every
hit in pages of profileDataQueryBatchSize.
… Elasticsearch

JDBC (PostgreSQL, MariaDB): the native trace search, by-id and profiling reads,
the OTLP trace search, by-id read and catalogs, the Zipkin catalogs, by-id
reads and every search criterion, and the span attached events with the trace
and Zipkin query services on top. Elasticsearch and OpenSearch: the native
trace and the OTLP DAOs, as nested ITs of the six server ITs.

Two JDBC tests are disabled: the Zipkin and OTLP searches cut each day table
at the limit by a trace's earliest match in that table, so a trace that
crosses midnight is ranked by its later span and pushes a later trace out.
… the JDBC and Elasticsearch storages

JDBCMetadataQueryITBase now covers every IMetadataQueryDAO method, network
address alias reloads and service labels on PostgreSQL and MariaDB, and the new
MetadataQueryITBase covers the same plus tag autocompletion on every
Elasticsearch and OpenSearch server IT, in both storage modes.

Fixes found by the tests:
- JDBC and ES listInstances and findEndpoint read rows created in the minute
  after the window's end: the exclusive end timestamp was compared inclusively.
- JDBC and ES listProcesses by instance compared minute buckets with the
  duration's own step, so an hour or day duration returned nothing.
- JDBC loadLastUpdate skipped aliases updated in the given minute.
- ES getProcess never found a process in the default (merged) storage mode.
- ES tag value autocompletion counted a value seen on several days more than
  once against the limit, returning fewer values than exist.
…asticsearch storages

JDBCMetricsQueryITBase now also reads metric values at the minute and hour steps, the values of an int
column, labeled values (every label and a subset), the labeled values of every entity and the entities
seen in a range, and ranks services, endpoints of a service and an int metric either way and by an
attribute; the sampled records keep statements on and just outside the range bounds, of another database
and cache commands of the same entity. MetricsQueryITBase covers the same on the six Elasticsearch and
OpenSearch servers. Both build the metrics from the shipped core.oal through the OAL engine (oal-rt, test
scope) and MAL meters from their function classes, and read across midnight.

On Elasticsearch, a top-N over more entities than a shard returns misses the range's top entity and
returns a partial mean; that test is disabled.
…AOs on JDBC and Elasticsearch

ITs for the eBPF profiling task, schedule and data DAOs, continuous-profiling
policies, runtime rules, UI templates (Elasticsearch), Elasticsearch history
delete, and the batch DAO's insert with the none-stream and management DAOs the
storage DAO hands out, on PostgreSQL, MariaDB and the six Elasticsearch and
OpenSearch servers.

Fixes found by them:
- JDBC listed a task's eBPF profiling schedules in day-table order; they are
  latest start first, as on Elasticsearch and BanyanDB.
- Elasticsearch read every task of a service for no target type (a
  continuous-profiling policy saved without targets); it reads none.
- Elasticsearch threw on deleting a runtime rule that is not there; the delete
  returns silently, as the DAO promises.
…ns on JDBC and Elasticsearch

New storage ITs read through the production DAOs what OAP's own write path
stored, on PostgreSQL and MariaDB and on the six Elasticsearch/OpenSearch
servers: logs (JDBC and Elasticsearch), browser error logs and events
(Elasticsearch), both alarm queries, GenAI evaluation records, and the four AI
agent conversation reads. Each filter has a near miss, rows lie on and just
outside the time bounds, and reads cross midnight, page, and compare whole rows.

Elasticsearch fixes, each shown by a failing IT:
- Events filtered by endpoint matched nothing: the read sent a match_phrase on
  an endpoint_match field the event index does not have. It is now a term on
  the endpoint, as JDBC and BanyanDB compare it.
- The event time range left out the events on the duration's first
  millisecond and on its end instant (gt/lt); it is now gte/lte.
- The recovery times of an alarm page were read with the default search size,
  so a page of more than 10 recovered alarms lost the rest.

One test per area is disabled: logs, browser error logs, alarms, GenAI
evaluations and AI agent rounds filter by the second time bucket up to the end
step's last second, so a row on Duration#getEndTimestamp() is left out on both
storages.
# Conflicts:
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
# Conflicts:
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/ElasticSearchQueryITSupport.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
# Conflicts:
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
#	oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/pom.xml
# Conflicts:
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/ElasticSearchQueryITSupport.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
# Conflicts:
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/ElasticSearchQueryITSupport.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
# Conflicts:
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java
#	oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
…uest format it picks by version

The client IT runs each server version once, as a nested class, and sends every request OAP's storage sends in the
shape it sends it: index templates with their alias and order, indices, appended mappings and their comparison on the
next boot, refreshed single-document writes, bulk inserts, updates and routed inserts, every query and aggregation
shape, routed searches, day indices of which some are missing, and scrolls with their context cleared. The versions
cover each request format: Elasticsearch 6.8.23, 7.7.1 (legacy templates), 7.8.0 and 7.17.29 (composable templates),
8.19.25 and 9.3.0, and OpenSearch 1.3.20, 2.8.0 and 3.0.0.

Fixes it found:
- indexTemplateOrder reached only Elasticsearch 6: a legacy template now sends its order, a composable one its
  priority, and both are read back.
- The installer wrote copy_to as a string, which the server returns as a list, so its comparison never found a stored
  template current and rewrote it on every boot.
- Document ids were put into request paths unescaped: an id with a slash, such as a runtime rule name or a Base64
  service id that keys a continuous-profiling policy, could not be written.
- A legacy index template was read back without its name.
… IT model is found installed

An index template returns copy_to as written; only an index lists it, and the installer compares indices by column
name. Every model with a match column is compared through its template, so the installer's string form was never
compared against the listed one. The client IT now asserts each read-back as the server returns it, and the storage
ITs check, for every model they install, that the installer finds it installed, as it must on the next boot.
…9.3.0

Only the storage case ran on Elasticsearch 9. Each case storing in Elasticsearch now takes ES_VERSION and runs on
8.18.8 and 9.3.0: alarms and baseline alarms, browser, cluster with ZooKeeper, events, LogQL, async-profiler, pprof
and trace profiling, Zipkin, the sharding variants, and the cases that already took ES_VERSION (storage sharding,
inspect API, runtime-rule MAL storage, TTL, logs, Fluent Bit logs, AI agent conversations). The OTLP native trace
case keeps Elasticsearch 7.15.0 and runs on 8.18.8 and 9.3.0 too. The Elasticsearch monitoring case, which does not
store in Elasticsearch, stays on 8.18.8.
…range

Each day table answered its own top `limit` traces, ranked by the trace's
earliest match in that table, and the answers were merged and cut once. A
trace that crosses midnight ranked by its later span in the later table,
took a place there, and kept that rank when its earlier span was not in the
earlier table's top `limit`: a later trace was dropped, or the crossing
trace was returned out of order.

Both searches now select their traces with the threshold algorithm: each
table is read in rank order a page of `limit` traces at a time, each trace
read is looked up by id in the other tables, and the reading stops once
`limit` traces rank at least as high as any trace not read yet. Without a
crossing trace near the top this is one page and one lookup per table.
…rof data of every instance on Elasticsearch

The task rows of async-profiler and pprof carried the second bucket of their
creation while their none-stream model is minute-downsampled. Elasticsearch
names a row's index from the bucket by the model's downsampling, so each task
went into an index named after its hour, which the TTL never deleted. The
mutation services now write minute buckets, and every reader bounds the rows
by minute buckets: the creation checks, the agent task caches, and the task
lists of a duration.

BanyanDB bounds a task read by timestamps, and ended it at the start of the
end bucket; a minute end bucket would have missed the tasks created in the
current minute. The trace-profiling, async-profiler and pprof task reads now
end at the last millisecond of the end bucket, as JDBC and Elasticsearch keep
the rows of the end bucket.

The Elasticsearch pprof data read returned nothing for an empty instance
list; it now reads every instance of the task, as JDBC, BanyanDB and the JFR
read do.
…nds through the duration's end instant

A query's end bound is inclusive: a row at Duration#getEndTimestamp() is read.
These reads filtered time_bucket up to the end step's last second, so a row
on the end instant was left out, and they picked the day tables and daily
indices by the end step's day, so a duration ending in a day's last step
missed the next midnight's table or index.

On JDBC and Elasticsearch, logs, browser error logs, both alarm queries, GenAI
evaluation records and AI agent conversation rounds now filter each record's
millisecond column (timestamp, start_time, evaluation_time) from the start
through the end instant, and read the tables or indices of both instants'
days. The alarm recovery lookup covers the time bucket of an alarm starting
on the end instant. Reads without a duration are unchanged.

The ITs that were disabled for this now run, each also on a duration ending
at midnight; JDBC browser error logs get the same bound tests.
…ticsearch

HierarchyQueryEsDAO read an instance's hierarchy relations with a plain
search that set no size, so Elasticsearch returned at most its default of
10. It now scrolls them as it does the service relations, in pages of
scrollingBatchSize up to metadataQueryMaxSize.

TopologyQueryEsDAO read a relation's components with a terms aggregation of
the default size, 10, so a relation naming more component values in a range
lost some. The aggregation now keeps up to 1000 per relation, more than the
component library defines.

The ITs gain a pod below instances in twelve layers, and a service and a
process relation that name a different component in each of twelve minutes,
on both storages.
The topology DAOs return one call per component a relation named in the
range, so a process relation detected at the client side may come back
several times with the same id. ProcessTopologyBuilder added the source
component only when it first created the call, so the process topology
kept one of them; it now adds each, as it already does for the target
components and as the service topology does on both sides. The instance
and endpoint topologies carry no components.

The JDBC and Elasticsearch topology ITs now also build the service and
process topologies through TopologyQueryService and compare their calls,
with the detect points and both sides' components, and the process nodes
read from the process traffic.
…ery server runs every Elasticsearch IT

- `indexTemplateOrder` may be negative, as a legacy template's order can be, but a composable template's priority
  cannot: Elasticsearch refused the template and OAP could not install its schema. A composable template now takes
  priority 0 for a negative order, and the installer takes that as the configured order. The client IT writes a
  negative order on every version.
- A unit test fails when one of the seven Elasticsearch server ITs misses an area another runs. Each server IT keeps
  its own nested classes, so a report still names the server a test ran on.
…one HTTP client in the ES ITs

The threshold ranking read each day table a page at a time and looked up the
traces it read in the other tables, so a range whose newest spans belong to
traces that also matched the day before paged through many queries. The
search is now one query: each day table ranks its traces over their matching
spans, with its own tag or annotation joins, and the union of those ranks is
grouped by trace once more and cut at the limit. It is exact for both orders
and returns only the traces kept.

The Elasticsearch IT support sends its refreshes, node stats and shard probes
through one HTTP client instead of creating one per request.
… the day its end instant opens

A tag is stored under the first minute of the day it is seen, so a read takes
whole days. Reading through the end instant, Duration#getEndTimestamp(),
admitted the whole next day: a one-day duration, or a minute duration ending
at 23:59, also returned every tag first seen on the following day, which can
displace the requested day's tags at the limit. On JDBC and Elasticsearch the
day tables or indices and the day filter now end with the day of the
duration's last step, as on BanyanDB. The Elasticsearch day filter for indices
rolled by more than one day, the early end of the value scroll and the empty
read for a limit of 0 stay.

The tag ITs on both storages now assert that a window ending in the last
minute before midnight, and a window of one day, read none of the next day's
tags.
…ch bulk processor be closed

JDBCMetricsDAO.isExpiredCache runs for every metric the workers receive and
built three calendars a call. It now reads the metric's day off the digits of
its time bucket, and keeps the first day table the TTL keeps, counted as
JDBCHistoryDeleteDAO counts it, for as long as that day stays the same.

BulkProcessor ran its periodical flush on a scheduler nothing could stop. It
and BatchProcessEsDAO get a close that stops it and sends what is queued. OAP's
modules have no stop phase, so OAP keeps its processor for the life of the
process; the Elasticsearch ITs close theirs before shutting their client down.
…elete their Elasticsearch indices by day

Writing these task rows with minute buckets broke reads of rows written
before an upgrade (second buckets fall outside minute bounds, so a running
task stopped blocking a new one) and coarsened task lists of a second-step
duration. The rows keep their second buckets, and every reader bounds them
by second buckets again.

The Elasticsearch defect was the write index: IndexNames read a row's day by
its model's downsampling, so a second bucket on a minute model named an hourly
index, or failed with a day step above one. The day is now read by the
bucket's own precision. HistoryDeleteEsDAO reads the hour-named indices
earlier versions created, <table>-yyyyMMddHH, as the day of their first eight
digits, so the TTL deletes them with that day; no other index name has ten
digits after its last dash.
… and keep a legacy template's negative order

- BulkProcessor.close() waits for a periodical flush that has already taken the queued requests; the client IT
  closes the processors it creates.
- A duration's end instant and BanyanDB's end of a time bucket are where the next step starts on the wall clock,
  so an hour repeated when the clocks are set back is read whole, and a day is not taken as 24 hours.
- The JDBC metrics cache takes its day starts from calendar dates, so a day whose midnight is skipped expires at
  its own start.
- The Elasticsearch installer compares a legacy template's order as configured; only a composable template's
  priority takes 0 for a negative order.
- A JDBC trace search within one day table ranks and limits in that table's query alone.
- The changes say that the server decoded a percent escape into a stored document id.
…ex without its midnight, and check profiling task overlap exactly

- DurationUtils resolves a step's start and end instants with the JDK's zone rules, by which time buckets are
  written, not with the bundled Joda-Time's older zone data. A step repeated when the clocks are set back starts at
  its first occurrence and ends after its second. BanyanDB's task reads bound their time buckets the same way.
- IndexNames walks and groups days as calendar dates, so a day whose midnight is skipped has an index.
- The trace profiling task check reads the latest two tasks and compares both ends, so a task starting in the new
  task's last minute after the new task ends is not a conflict.
- The JDBC longest-first trace ranking limits each day table's ranking too.
- BulkProcessor.close() waits without being interrupted and sends what is still queued itself.
…lk processor without losing a request

Every conversion of a timestamp into a duration's steps (PromQL, LogQL, TraceQL, Zipkin, MQE's trend range,
the trace profiling segment list) now uses the JDK's zone rules that DurationUtils reads the steps back by.
A step ends after the last instant on its wall clock, so an hour the clocks set back by half an hour ends after
its second half hour, and a step starting in a skipped hour starts where the skip ends.

JDBC lists day tables by calendar date, so a read from a day whose midnight is skipped reaches the next day, and
the TTL's table list keeps the day before a 23-hour day. Elasticsearch looks up alarm recoveries in the buckets of
the alarms it read. The trace profiling conflict check reads every task that can overlap the new one, bounded by
the least and greatest minute bucket of those instants. The bulk processor refuses a request added once closing
has begun and waits for the server's version without being interrupted.
…lists by their buckets, and close the batch DAO safely

DurationUtils no longer uses Joda-Time's zones: a step's format is checked as a local time, and the points of a
series and their timestamps in MQE and PromQL follow the JDK's rules, as their ranges already do. The storage ITs
build their midnights and query durations by the same rules.

BanyanDB keeps no time bucket for task rows and bounds a task list by instants; it now leaves out the tasks of other
buckets that the instants hold when the clocks are set back, and near such a time reads more rows so they take no
place of the tasks asked for. The async-profiler and pprof checks read the latest tasks without bounding them by the
new task's bucket, which left out a running task created before the clocks were set back.

The Elasticsearch batch DAO marks itself closed under the lock that creates its bulk processor, so a processor is
never started once closing has begun.
…t from OAP

OAP never closes a bulk processor: its periodical flush runs as long as OAP. BulkProcessor and BatchProcessEsDAO
lose the close methods only the ITs called, and stay as they were but for the scheduler, now kept in a field. The ITs
close their processors through BulkProcessorITExt and BatchProcessEsDAOITExt, which stop the periodical flush by
reflection and send what is still queued.
…rom where a skipped hour ends, and read BanyanDB task lists by both occurrences of a repeated hour

The JDBC day walk names each date through TimeBucket, as the tables are named, so a default locale with another
calendar, such as Thai, does not skip or fail on 29 February. A metric series over a range starting or ending in the
hour skipped when the clocks are set forward steps from where the skip ends, as the range's instants do, instead of
failing on the range limit. BanyanDB task lists no longer leave out the tasks of other minutes that a repeated hour
holds: a range in that hour covers both occurrences, as it does on every read.
…orage ITs does not need

OAP runs in UTC or a fixed UTC offset. DurationUtils, the query plugins, the BanyanDB bucket bounds, the JDBC table
walk and metrics cache day, the Elasticsearch index days and alarm recovery read, the profiling task checks and the IT
fixtures go back to what they were before daylight saving, zone-data and calendar cases were handled. The fixes the ITs
found stay.
@wu-sheng wu-sheng added this to the 11.2.0 milestone Nov 18, 2026
@wu-sheng wu-sheng added bug Something isn't working and you are sure it's a bug! test Test requirements about performance, feature or before release. labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working and you are sure it's a bug! test Test requirements about performance, feature or before release.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant