Repository navigation
Conversation
Each area's IT is a nested class of the IT of the server it runs on, and writes under a namespace of its own. Writes refresh their indices instead of waiting for the next refresh.
…e2e expectations A single-quoted scalar cannot hold a raw quote, and one of the simulator's js errors reads "(reading 'id')". The verifier renders the expectation once per log; the render of that log failed to parse, and when it was the oldest log (the warmup error, last in the newest-first list) the verifier expected no element and failed the case. Go's %q always renders a valid YAML double-quoted scalar.
…torages Add ITs for every ITopologyQueryDAO and IHierarchyQueryDAO method on PostgreSQL, MariaDB and the six Elasticsearch / OpenSearch servers. The relations are written on both sides of midnight, on and just outside the range bounds, on both detect points, and rolled up per hour in the same tables and indices; each filter has a near miss. Fix TopologyQueryEsDAO returning only the most frequent component of a service or process relation in the range. A relation stores its components per minute, so a range may hold several; each is now a call of its own, as the JDBC and BanyanDB DAOs already return them.
JDBC (PostgreSQL, MariaDB): the trace-profiling, async-profiler and pprof task lookups by id, their task logs, the JFR and pprof data reads, more task list near misses, and thread-snapshot batches and window bounds. Elasticsearch and OpenSearch, default and sharded modes: new ProfilingTasks and ProfileThreadSnapshots ITs on all six servers, covering task lists, ids, logs, data and thread snapshots. Fix: the JFR and pprof data reads on Elasticsearch searched without a size and returned only the first 10 uploads of a task. They now scroll over every hit in pages of profileDataQueryBatchSize.
… Elasticsearch JDBC (PostgreSQL, MariaDB): the native trace search, by-id and profiling reads, the OTLP trace search, by-id read and catalogs, the Zipkin catalogs, by-id reads and every search criterion, and the span attached events with the trace and Zipkin query services on top. Elasticsearch and OpenSearch: the native trace and the OTLP DAOs, as nested ITs of the six server ITs. Two JDBC tests are disabled: the Zipkin and OTLP searches cut each day table at the limit by a trace's earliest match in that table, so a trace that crosses midnight is ranked by its later span and pushes a later trace out.
… the JDBC and Elasticsearch storages JDBCMetadataQueryITBase now covers every IMetadataQueryDAO method, network address alias reloads and service labels on PostgreSQL and MariaDB, and the new MetadataQueryITBase covers the same plus tag autocompletion on every Elasticsearch and OpenSearch server IT, in both storage modes. Fixes found by the tests: - JDBC and ES listInstances and findEndpoint read rows created in the minute after the window's end: the exclusive end timestamp was compared inclusively. - JDBC and ES listProcesses by instance compared minute buckets with the duration's own step, so an hour or day duration returned nothing. - JDBC loadLastUpdate skipped aliases updated in the given minute. - ES getProcess never found a process in the default (merged) storage mode. - ES tag value autocompletion counted a value seen on several days more than once against the limit, returning fewer values than exist.
…asticsearch storages JDBCMetricsQueryITBase now also reads metric values at the minute and hour steps, the values of an int column, labeled values (every label and a subset), the labeled values of every entity and the entities seen in a range, and ranks services, endpoints of a service and an int metric either way and by an attribute; the sampled records keep statements on and just outside the range bounds, of another database and cache commands of the same entity. MetricsQueryITBase covers the same on the six Elasticsearch and OpenSearch servers. Both build the metrics from the shipped core.oal through the OAL engine (oal-rt, test scope) and MAL meters from their function classes, and read across midnight. On Elasticsearch, a top-N over more entities than a shard returns misses the range's top entity and returns a partial mean; that test is disabled.
…AOs on JDBC and Elasticsearch ITs for the eBPF profiling task, schedule and data DAOs, continuous-profiling policies, runtime rules, UI templates (Elasticsearch), Elasticsearch history delete, and the batch DAO's insert with the none-stream and management DAOs the storage DAO hands out, on PostgreSQL, MariaDB and the six Elasticsearch and OpenSearch servers. Fixes found by them: - JDBC listed a task's eBPF profiling schedules in day-table order; they are latest start first, as on Elasticsearch and BanyanDB. - Elasticsearch read every task of a service for no target type (a continuous-profiling policy saved without targets); it reads none. - Elasticsearch threw on deleting a runtime rule that is not there; the delete returns silently, as the DAO promises.
…ns on JDBC and Elasticsearch New storage ITs read through the production DAOs what OAP's own write path stored, on PostgreSQL and MariaDB and on the six Elasticsearch/OpenSearch servers: logs (JDBC and Elasticsearch), browser error logs and events (Elasticsearch), both alarm queries, GenAI evaluation records, and the four AI agent conversation reads. Each filter has a near miss, rows lie on and just outside the time bounds, and reads cross midnight, page, and compare whole rows. Elasticsearch fixes, each shown by a failing IT: - Events filtered by endpoint matched nothing: the read sent a match_phrase on an endpoint_match field the event index does not have. It is now a term on the endpoint, as JDBC and BanyanDB compare it. - The event time range left out the events on the duration's first millisecond and on its end instant (gt/lt); it is now gte/lte. - The recovery times of an alarm page were read with the default search size, so a page of more than 10 recovered alarms lost the rest. One test per area is disabled: logs, browser error logs, alarms, GenAI evaluations and AI agent rounds filter by the second time bucket up to the end step's last second, so a row on Duration#getEndTimestamp() is left out on both storages.
# Conflicts: # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
# Conflicts: # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/ElasticSearchQueryITSupport.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
# Conflicts: # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java # oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/pom.xml
# Conflicts: # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/ElasticSearchQueryITSupport.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
# Conflicts: # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/ElasticSearchQueryITSupport.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
# Conflicts: # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch8IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/Elasticsearch9ShardedIT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch2IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3IT.java # oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/OpenSearch3ShardedIT.java
…uest format it picks by version The client IT runs each server version once, as a nested class, and sends every request OAP's storage sends in the shape it sends it: index templates with their alias and order, indices, appended mappings and their comparison on the next boot, refreshed single-document writes, bulk inserts, updates and routed inserts, every query and aggregation shape, routed searches, day indices of which some are missing, and scrolls with their context cleared. The versions cover each request format: Elasticsearch 6.8.23, 7.7.1 (legacy templates), 7.8.0 and 7.17.29 (composable templates), 8.19.25 and 9.3.0, and OpenSearch 1.3.20, 2.8.0 and 3.0.0. Fixes it found: - indexTemplateOrder reached only Elasticsearch 6: a legacy template now sends its order, a composable one its priority, and both are read back. - The installer wrote copy_to as a string, which the server returns as a list, so its comparison never found a stored template current and rewrote it on every boot. - Document ids were put into request paths unescaped: an id with a slash, such as a runtime rule name or a Base64 service id that keys a continuous-profiling policy, could not be written. - A legacy index template was read back without its name.
… IT model is found installed An index template returns copy_to as written; only an index lists it, and the installer compares indices by column name. Every model with a match column is compared through its template, so the installer's string form was never compared against the listed one. The client IT now asserts each read-back as the server returns it, and the storage ITs check, for every model they install, that the installer finds it installed, as it must on the next boot.
…9.3.0 Only the storage case ran on Elasticsearch 9. Each case storing in Elasticsearch now takes ES_VERSION and runs on 8.18.8 and 9.3.0: alarms and baseline alarms, browser, cluster with ZooKeeper, events, LogQL, async-profiler, pprof and trace profiling, Zipkin, the sharding variants, and the cases that already took ES_VERSION (storage sharding, inspect API, runtime-rule MAL storage, TTL, logs, Fluent Bit logs, AI agent conversations). The OTLP native trace case keeps Elasticsearch 7.15.0 and runs on 8.18.8 and 9.3.0 too. The Elasticsearch monitoring case, which does not store in Elasticsearch, stays on 8.18.8.
… the fixes they found
…range Each day table answered its own top `limit` traces, ranked by the trace's earliest match in that table, and the answers were merged and cut once. A trace that crosses midnight ranked by its later span in the later table, took a place there, and kept that rank when its earlier span was not in the earlier table's top `limit`: a later trace was dropped, or the crossing trace was returned out of order. Both searches now select their traces with the threshold algorithm: each table is read in rank order a page of `limit` traces at a time, each trace read is looked up by id in the other tables, and the reading stops once `limit` traces rank at least as high as any trace not read yet. Without a crossing trace near the top this is one page and one lookup per table.
…rof data of every instance on Elasticsearch The task rows of async-profiler and pprof carried the second bucket of their creation while their none-stream model is minute-downsampled. Elasticsearch names a row's index from the bucket by the model's downsampling, so each task went into an index named after its hour, which the TTL never deleted. The mutation services now write minute buckets, and every reader bounds the rows by minute buckets: the creation checks, the agent task caches, and the task lists of a duration. BanyanDB bounds a task read by timestamps, and ended it at the start of the end bucket; a minute end bucket would have missed the tasks created in the current minute. The trace-profiling, async-profiler and pprof task reads now end at the last millisecond of the end bucket, as JDBC and Elasticsearch keep the rows of the end bucket. The Elasticsearch pprof data read returned nothing for an empty instance list; it now reads every instance of the task, as JDBC, BanyanDB and the JFR read do.
…nds through the duration's end instant A query's end bound is inclusive: a row at Duration#getEndTimestamp() is read. These reads filtered time_bucket up to the end step's last second, so a row on the end instant was left out, and they picked the day tables and daily indices by the end step's day, so a duration ending in a day's last step missed the next midnight's table or index. On JDBC and Elasticsearch, logs, browser error logs, both alarm queries, GenAI evaluation records and AI agent conversation rounds now filter each record's millisecond column (timestamp, start_time, evaluation_time) from the start through the end instant, and read the tables or indices of both instants' days. The alarm recovery lookup covers the time bucket of an alarm starting on the end instant. Reads without a duration are unchanged. The ITs that were disabled for this now run, each also on a duration ending at midnight; JDBC browser error logs get the same bound tests.
…ticsearch HierarchyQueryEsDAO read an instance's hierarchy relations with a plain search that set no size, so Elasticsearch returned at most its default of 10. It now scrolls them as it does the service relations, in pages of scrollingBatchSize up to metadataQueryMaxSize. TopologyQueryEsDAO read a relation's components with a terms aggregation of the default size, 10, so a relation naming more component values in a range lost some. The aggregation now keeps up to 1000 per relation, more than the component library defines. The ITs gain a pod below instances in twelve layers, and a service and a process relation that name a different component in each of twelve minutes, on both storages.
The topology DAOs return one call per component a relation named in the range, so a process relation detected at the client side may come back several times with the same id. ProcessTopologyBuilder added the source component only when it first created the call, so the process topology kept one of them; it now adds each, as it already does for the target components and as the service topology does on both sides. The instance and endpoint topologies carry no components. The JDBC and Elasticsearch topology ITs now also build the service and process topologies through TopologyQueryService and compare their calls, with the detect points and both sides' components, and the process nodes read from the process traffic.
…ery server runs every Elasticsearch IT - `indexTemplateOrder` may be negative, as a legacy template's order can be, but a composable template's priority cannot: Elasticsearch refused the template and OAP could not install its schema. A composable template now takes priority 0 for a negative order, and the installer takes that as the configured order. The client IT writes a negative order on every version. - A unit test fails when one of the seven Elasticsearch server ITs misses an area another runs. Each server IT keeps its own nested classes, so a report still names the server a test ran on.
…one HTTP client in the ES ITs The threshold ranking read each day table a page at a time and looked up the traces it read in the other tables, so a range whose newest spans belong to traces that also matched the day before paged through many queries. The search is now one query: each day table ranks its traces over their matching spans, with its own tag or annotation joins, and the union of those ranks is grouped by trace once more and cut at the limit. It is exact for both orders and returns only the traces kept. The Elasticsearch IT support sends its refreshes, node stats and shard probes through one HTTP client instead of creating one per request.
… the day its end instant opens A tag is stored under the first minute of the day it is seen, so a read takes whole days. Reading through the end instant, Duration#getEndTimestamp(), admitted the whole next day: a one-day duration, or a minute duration ending at 23:59, also returned every tag first seen on the following day, which can displace the requested day's tags at the limit. On JDBC and Elasticsearch the day tables or indices and the day filter now end with the day of the duration's last step, as on BanyanDB. The Elasticsearch day filter for indices rolled by more than one day, the early end of the value scroll and the empty read for a limit of 0 stay. The tag ITs on both storages now assert that a window ending in the last minute before midnight, and a window of one day, read none of the next day's tags.
…ch bulk processor be closed JDBCMetricsDAO.isExpiredCache runs for every metric the workers receive and built three calendars a call. It now reads the metric's day off the digits of its time bucket, and keeps the first day table the TTL keeps, counted as JDBCHistoryDeleteDAO counts it, for as long as that day stays the same. BulkProcessor ran its periodical flush on a scheduler nothing could stop. It and BatchProcessEsDAO get a close that stops it and sends what is queued. OAP's modules have no stop phase, so OAP keeps its processor for the life of the process; the Elasticsearch ITs close theirs before shutting their client down.
…elete their Elasticsearch indices by day Writing these task rows with minute buckets broke reads of rows written before an upgrade (second buckets fall outside minute bounds, so a running task stopped blocking a new one) and coarsened task lists of a second-step duration. The rows keep their second buckets, and every reader bounds them by second buckets again. The Elasticsearch defect was the write index: IndexNames read a row's day by its model's downsampling, so a second bucket on a minute model named an hourly index, or failed with a day step above one. The day is now read by the bucket's own precision. HistoryDeleteEsDAO reads the hour-named indices earlier versions created, <table>-yyyyMMddHH, as the day of their first eight digits, so the TTL deletes them with that day; no other index name has ten digits after its last dash.
… and keep a legacy template's negative order - BulkProcessor.close() waits for a periodical flush that has already taken the queued requests; the client IT closes the processors it creates. - A duration's end instant and BanyanDB's end of a time bucket are where the next step starts on the wall clock, so an hour repeated when the clocks are set back is read whole, and a day is not taken as 24 hours. - The JDBC metrics cache takes its day starts from calendar dates, so a day whose midnight is skipped expires at its own start. - The Elasticsearch installer compares a legacy template's order as configured; only a composable template's priority takes 0 for a negative order. - A JDBC trace search within one day table ranks and limits in that table's query alone. - The changes say that the server decoded a percent escape into a stored document id.
…ex without its midnight, and check profiling task overlap exactly - DurationUtils resolves a step's start and end instants with the JDK's zone rules, by which time buckets are written, not with the bundled Joda-Time's older zone data. A step repeated when the clocks are set back starts at its first occurrence and ends after its second. BanyanDB's task reads bound their time buckets the same way. - IndexNames walks and groups days as calendar dates, so a day whose midnight is skipped has an index. - The trace profiling task check reads the latest two tasks and compares both ends, so a task starting in the new task's last minute after the new task ends is not a conflict. - The JDBC longest-first trace ranking limits each day table's ranking too. - BulkProcessor.close() waits without being interrupted and sends what is still queued itself.
…lk processor without losing a request Every conversion of a timestamp into a duration's steps (PromQL, LogQL, TraceQL, Zipkin, MQE's trend range, the trace profiling segment list) now uses the JDK's zone rules that DurationUtils reads the steps back by. A step ends after the last instant on its wall clock, so an hour the clocks set back by half an hour ends after its second half hour, and a step starting in a skipped hour starts where the skip ends. JDBC lists day tables by calendar date, so a read from a day whose midnight is skipped reaches the next day, and the TTL's table list keeps the day before a 23-hour day. Elasticsearch looks up alarm recoveries in the buckets of the alarms it read. The trace profiling conflict check reads every task that can overlap the new one, bounded by the least and greatest minute bucket of those instants. The bulk processor refuses a request added once closing has begun and waits for the server's version without being interrupted.
…lists by their buckets, and close the batch DAO safely DurationUtils no longer uses Joda-Time's zones: a step's format is checked as a local time, and the points of a series and their timestamps in MQE and PromQL follow the JDK's rules, as their ranges already do. The storage ITs build their midnights and query durations by the same rules. BanyanDB keeps no time bucket for task rows and bounds a task list by instants; it now leaves out the tasks of other buckets that the instants hold when the clocks are set back, and near such a time reads more rows so they take no place of the tasks asked for. The async-profiler and pprof checks read the latest tasks without bounding them by the new task's bucket, which left out a running task created before the clocks were set back. The Elasticsearch batch DAO marks itself closed under the lock that creates its bulk processor, so a processor is never started once closing has begun.
…t from OAP OAP never closes a bulk processor: its periodical flush runs as long as OAP. BulkProcessor and BatchProcessEsDAO lose the close methods only the ITs called, and stay as they were but for the scheduler, now kept in a field. The ITs close their processors through BulkProcessorITExt and BatchProcessEsDAOITExt, which stop the periodical flush by reflection and send what is still queued.
…rom where a skipped hour ends, and read BanyanDB task lists by both occurrences of a repeated hour The JDBC day walk names each date through TimeBucket, as the tables are named, so a default locale with another calendar, such as Thai, does not skip or fail on 29 February. A metric series over a range starting or ending in the hour skipped when the clocks are set forward steps from where the skip ends, as the range's instants do, instead of failing on the range limit. BanyanDB task lists no longer leave out the tasks of other minutes that a repeated hour holds: a range in that hour covers both occurrences, as it does on every read.
…orage ITs does not need OAP runs in UTC or a fixed UTC offset. DurationUtils, the query plugins, the BanyanDB bucket bounds, the JDBC table walk and metrics cache day, the Elasticsearch index days and alarm recovery read, the profiling task checks and the IT fixtures go back to what they were before daylight saving, zone-data and calendar cases were handled. The fixes the ITs found stay.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test every JDBC and Elasticsearch storage DAO and the Elasticsearch client on each request format, and fix what the tests found
The BanyanDB storage has an IT for every DAO method OAP can reach (#14216). This PR brings the JDBC and Elasticsearch storages to the same coverage. It also tests the Elasticsearch client on a server of each request format it picks by version, and runs the Elasticsearch e2e cases on Elasticsearch 9. Each fix below has a test that failed before the fix and passes after it.
Storage DAO ITs. Each IT installs the schema the way OAP does at boot, writes through OAP's own write path, and reads through the DAO the storage provider registers. It compares whole results: near misses for every filter, rows on and just outside the time bounds, paging, order and limits. Rows sit on both sides of midnight, so a read crosses two day tables or two daily indices. Every DAO method of both storages is now called by an IT, and no IT is disabled.
JDBC<Area>ITBaseand a subclass per database.Elasticsearch client IT. It has one nested class per server version. Each sends every request OAP's storage sends, in the shape OAP sends it:
The versions cover each request format the client picks:
Each server starts once for all its tests, so the job takes about 3 minutes instead of 20.
e2e.
ES_VERSIONand runs on 8.18.8 and 9.3.0, adding 25 jobs. Before, only the storage case ran on 9.3.0. Outside CI the cases default to 8.18.8.contentwithprintf "%q". A log holding a'broke the expected YAML, and when it was the oldest log the verifier failed the case, about 1 run in 4.Fixes.
Elasticsearch client:
indexTemplateOrderreached the server only on Elasticsearch 6. A legacy template now sends itsorder, a composable template itspriority, and both are read back as 64-bit values. A template stored with another order is updated at boot. A legacy template keeps a negative order; a composable priority cannot be negative, so it takes 0./,?or#could not be written; examples are a runtime rule named likeaws-gateway/gateway-service, or the continuous-profiling policy of a service whose Base64 id holds/. The server decoded a percent escape, storinga%20basa b, which a read by id never found; such a document, written by an earlier version, is not found after the upgrade. Horizon's template names never hold these characters. A+still reaches the server as+, because the HTTP client decodes%2B, and Elasticsearch before 7.4 reads it as a space.IOExceptioninstead of passing for a deletion, and the existence check before it, which hid failures and raced other deletes, is gone.Elasticsearch DAOs:
getProcessnever found a process in the default (merged) mode.dayStep> 1; it read every page before cutting; and it failed for a limit of 0.Both storages:
Duration#getEndTimestamp()is the start of the next step, and a row exactly on it is read, as on BanyanDB. Most reads compared second buckets against the end step and chose day tables and indices by the end step. So they left out a row on the end instant, and a range ending in a day's last minute never read the next day, where that end lies. These reads now filter their millisecond column through the end instant and read that day: logs, browser error logs, alarms, GenAI evaluations, AI agent rounds, events, the trace list, sampled records, and the JDBC Zipkin and OTLP searches. Tag autocomplete is stored per day, so honoring the end instant would add a whole day; it reads the days of the requested steps, as BanyanDB does.listInstancesandfindEndpointreturned rows first seen in the minute after the range.listProcessesof an instance compared hour or day buckets with minute ping buckets.ProcessTopologyBuilder).JDBC:
UNION ALLof each day table's per-trace minimum (or maximum, for the longest-first order), grouped again by trace and only then sorted and cut. It is exact for both orders. A range within one day table is ranked in that table's query alone, and the longest-first order limits each table's ranking too: a trace among the longest of the range is among the longest of the table holding its longest span.isExpiredCachecounted a metric on the last dropped day as alive.Profiling task rows:
dayStep> 1. The TTL now deletes those hour-named indices with their day, so an upgrade needs no manual cleanup. Async-profiler and pprof task rows keep the second buckets earlier versions wrote, so rows from before and after an upgrade are read the same way; an eBPF task that a continuous-profiling policy triggers now carries the minute it is created in, as its model's downsampling says.Accepted Elasticsearch behavior, documented at the aggregations.
sortMetrics) is a terms aggregation ordered byavg. The Zipkin and OTLP latest-first searches order bymindescending.shard_size, which issize * 1.5 + 10by default.Validation (local, JDK 11, on the final commit):
Elasticsearch DAO ITs: 2,100 run, 0 failed, 0 skipped.
JDBC DAO ITs: 583 run, 0 failed, 0 skipped.
BanyanDB ITs: 189 run, 0 failed.
Elasticsearch client IT: 108 of 108 on the nine servers. Wrapper IT: 28 of 28 on Elasticsearch 6.3.2 and 7.8.0.
Unit tests: core 426, eBPF receiver 13, JDBC 123, Elasticsearch 69, BanyanDB 73, Elasticsearch client 12, all passing. Checkstyle passes on every changed module.
The e2e changes run in CI.
Update the
CHANGESlog.