Skip to content

Fix JDBC storage query bugs and two async-profiler task bugs, and test the JDBC DAOs on PostgreSQL and MariaDB - #14243

Merged
wu-sheng merged 1 commit into
masterfrom
fix/jdbc-query-dao-bugs
Nov 17, 2026
Merged

wu-sheng merged 1 commit into
masterfrom
fix/jdbc-query-dao-bugs

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Fix bugs of the JDBC storage, and two async-profiler task bugs

  • Add a unit test to verify that the fix works. Every fix below has a test that fails on master's code and passes here: the new PostgreSQL and MariaDB ITs for the JDBC fixes, and unit tests for the Top N ranking and the two async-profiler fixes.
  • Explain briefly why the bug exists and how to fix it.

#14239 was fixed by #14242, which merged first. This PR gives JDBCTraceQueryDAO.queryByTraceIdWithInstanceId the same guard: an empty list would also build in (). Its only caller, ProfileTaskQueryService.getProfileTaskSegments, returns before calling it when there is no segment, so this guard is defensive. The new ITs read both lookups with an empty list on PostgreSQL and MariaDB.

Day tables. The JDBC storage keeps records and metrics in one table per day. Many queries ran their SQL on each day table and concatenated the answers, so ranking, limits and grouping applied to each day instead of to the queried range:

  • Trace list (JDBCTraceQueryDAO.queryBasicTraces): each table applied the page's offset and limit, and the pages were concatenated unsorted. Each table now returns its first from + limit traces, and the page is cut after one sort by start time or duration, as the query asks.
  • Zipkin search (JDBCZipkinQueryDAO): each table returned up to limit trace IDs, so the search could return more. Each table still selects up to limit trace IDs; these are now merged, ranked newest first by the earliest matching span each table returned for the trace, and cut once. Spans were grouped into traces per table, so a trace with spans on both days came back as two partial traces, in a search and in traceMany; spans are now grouped across tables.
  • Heatmap (JDBCMetricsQueryDAO.readHeatMap): each table filled the points it did not hold with zeroed columns, so a range across midnight got extra columns. The columns are now built in point order once every table is read.
  • Top N (JDBCAggregationQueryDAO.sortMetrics): each table averaged and cut its own top N, and the results were averaged again, so an entity was ranked by the average of its daily averages, and one that topped the range but no single day was missed. The values were also read with getInt, which fails on values above 2147483647. With one table, the database still ranks and cuts. Across tables, each table now returns every entity's sum and sample count; entities are ranked by the exact mean, comparing cross-products of sums and counts, and the value returned is the truncated mean.
  • Slow statements (JDBCRecordsQueryDAO): each table returned its own top N, concatenated unranked. They are now ranked together and cut once.
  • Tag autocomplete (JDBCTagAutoCompleteQueryDAO): the union of the tables could exceed the limit, so the limit now applies to it.
  • Async-profiler and pprof task lists (JDBCAsyncProfilerTaskQueryDAO, JDBCPprofTaskQueryDAO): with a time range, tables are read oldest day first, so the limit kept the older days' tasks. The tasks are now sorted newest first before the limit.
  • Trace profiling tasks (JDBCProfileTaskQueryDAO.getTaskList): a task is stored in the table of the day it was created, but its time bucket is its scheduled start. The agent's task refresh queries with both time bounds, and the tables were chosen by those bounds, so a task scheduled for a later day than it was created could not reach the agent. The check of a new task against the latest-starting one queries without a start bound, so it read every table, but each table was sorted and cut on its own, and the check could inspect the wrong task and let a new task be created over a running one. Every table within the TTL is now read, and the tasks are sorted by start time, latest first, before the limit.
  • Trace profiling snapshots (JDBCProfileThreadSnapshotQueryDAO): the language column was never read, so a Go service's thread snapshots were analyzed as Java. A day table holding none of a segment's snapshots answered the min/max sequence query with SQL NULL, read as sequence 0, so the analysis could read snapshots outside the selected time range. NULL now counts as no answer.
  • Instances (JDBCMetadataQueryDAO.getInstances): it returned the first day table's answer only, so the continuous profiling instance list missed instances first seen on an earlier day. It now reads every table.

Other JDBC bugs:

  • Processes of an agent (JDBCMetadataQueryDAO.listProcesses(agentId, …)): the ping time range was dropped, so Rover could receive eBPF profiling tasks for processes that had stopped reporting.
  • Empty lists in SQL: JDBCEBPFProfilingTaskDAO.queryTasksByTargets (a continuous profiling policy saved without targets) and JDBCEBPFProfilingDataDAO.queryData (an analysis with no schedule) built in (). They now return nothing.
  • Browser error logs without a time range (JDBCBrowserLogQueryDAO) failed with a NullPointerException. They are now read from every table within the TTL.
  • Events (JDBCEventQueryDAO): the layer was bound as a string, which PostgreSQL rejects against the integer column. The start of the time range was exclusive, leaving out an event at its first millisecond.
  • UI templates: JDBCSQLExecutor.getByID always prefixed the ID with the model name, which the insert does only for records and function metrics; a management row such as a UI template is stored under its raw ID. So disabling a template always failed with "Can't find the template", and changeTemplate, which didn't check, reported success for a template that does not exist. The lookup now builds the ID as the insert does, and changeTemplate reports a missing template. JDBCRuntimeRuleManagementDAO's comment about working around that lookup is updated.
  • Cached metrics whose row is gone (BatchSQLExecutor): an insert and an update both reach it as a SQLExecutor, which implements both request types, so every batch was taken as inserts and an update that found no row never called onUpdateFailure. OAP kept the metric cached and kept updating its missing row. The row count now decides: an UPDATE of zero rows calls onUpdateFailure, which drops the metric from the cache, so the next flush of that metric checks storage and inserts the row again.
  • The table check at boot (JDBCTableInstaller.isExists) logged the table as missing and the column state inverted.

Async profiler (shared OAP core):

  • The check that refuses a second task on a service never fired on JDBC and Elasticsearch. AsyncProfilerMutationService.checkTaskProfiling bounded the time bucket by the minute bucket of now (12 digits), while a task row carries the second bucket of its creation (14 digits), so no task was ever at or below the bound. It now uses the second bucket.
  • An agent could be sent the older of two tasks. A refresh of AsyncProfilerTaskCache saves each task its query returns, those created from five minutes before now to five minutes after, and each save replaced the service's cached task, so a newest-first list left the older task cached. A task now replaces the cached task only if it was created no earlier.

Tests

  • New JDBC storage ITs on PostgreSQL 13 and MariaDB 11.4 (Testcontainers): JDBC<Area>PostgreSQLIT and JDBC<Area>MariaDBIT for eight areas (TraceQuery, ZipkinQuery, LogQuery, MetricsQuery, MetadataQuery, ProfilingTaskQuery, ProfilingDataQuery, Management), 40 cases on each database. They install the tables with the plugin's installers as JDBCStorageProvider does (column renames included), write through OAP's DAOs and read through the production query DAOs, with time-series data on yesterday's and today's tables. They carry no slow tag, so they run in the default Integration test job; the whole set takes about 45 seconds locally.
  • MariaDB Connector/J (LGPL) and Testcontainers' mariadb module are test-scope dependencies of the JDBC plugin only, never distributed. The MySQL driver stays out.
  • Unit tests: two new JDBCAggregationQueryDAOTest cases (fractional sums across days, ranking by the exact mean), and the new AsyncProfilerMutationServiceTest and AsyncProfilerTaskCacheTest. No existing test body or assertion changed.

CI

  • The Zipkin and span-attached event ITs of the Elasticsearch storage now also carry @Tag("elastic-it"), and the slow-integration-test job runs them as one "Elastic IT" entry (-Dgroups=elastic-it) instead of two, saving one whole-project build. ElasticSearchIT keeps its own entry. Comments in skywalking.yaml list what each IT job runs, and the /test skill matches the new matrix.

Validation

  • On master's code, the final tests fail as expected: 73 of the 80 new IT cases (62 failures, 11 errors), and the 2 new Top N unit tests. The 7 that pass are controls (segments by ID on both days, and the boot check of a model without a table, on each database), the empty segment-ID list that Handle empty JDBC trace profile segment lists #14242 fixed, on each database, and the layer filter on MariaDB, which converts the string itself. On master, AsyncProfilerMutationServiceTest fails with expected: <ALREADY_PROFILING_ERROR> but was: <SUCCESS> and AsyncProfilerTaskCacheTest with expected: <created> but was: <ended>.

  • At f074ed4bab, rebased onto master e564f75793 after Handle empty JDBC trace profile segment lists #14242: the JDBC plugin's 118 unit tests and 81 ITs (the 80 new and JDBCHistoryDeleteDAOIT) passed; the whole project's unit tests, 4,201 on JDK 17, passed (0 failures, 0 errors, 5 skipped); checkstyle passed, and the JDK 11 clean flatten:flatten install javadoc:javadoc -Pall build succeeded. Before that rebase, which changed no Elasticsearch code, -Dgroups=elastic-it ran exactly the 12 tagged classes, 174 tests, 0 failures, in 9:22 min.

  • After rebasing onto master 381fcbd488 (Decide the layer of a log without one by LAL fallback rules, and save OTLP_GENERAL logs #14244) as 1224096b83: checkstyle passes, the JDK 11 clean flatten:flatten install javadoc:javadoc -Pall build succeeds, and the whole project's unit tests, 4,284 on JDK 17, pass (0 failures, 0 errors, 5 skipped), the JDBC plugin's 118 among them.

  • If this pull request closes/resolves/fixes an existing issue, replace the issue number. Not applicable: [Bug] JDBC trace profiling query fails with SQL syntax error when the segment list is empty #14239 was already addressed by Handle empty JDBC trace profile segment lists #14242.

  • Update the CHANGES log.

@wu-sheng
wu-sheng marked this pull request as ready for review November 16, 2026 23:44
@wu-sheng
wu-sheng requested a balanced review from Copilot November 16, 2026 23:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@wu-sheng
wu-sheng requested a review from wankai123 November 16, 2026 23:52
@wu-sheng wu-sheng added this to the 11.2.0 milestone Nov 16, 2026
@wu-sheng wu-sheng added bug Something isn't working and you are sure it's a bug! backend OAP backend related. test Test requirements about performance, feature or before release. and removed bug Something isn't working and you are sure it's a bug! labels Nov 16, 2026
wankai123
wankai123 previously approved these changes Nov 16, 2026
@wu-sheng
wu-sheng force-pushed the fix/jdbc-query-dao-bugs branch from 7bfe20a to f074ed4 Compare November 17, 2026 03:25
@wu-sheng wu-sheng changed the title Fix #14239 and other JDBC storage query bugs, and two async-profiler task bugs, and test the JDBC DAOs on PostgreSQL and MariaDB Fix JDBC storage query bugs and two async-profiler task bugs, and test the JDBC DAOs on PostgreSQL and MariaDB Nov 17, 2026
…t the JDBC DAOs on PostgreSQL and MariaDB

JDBC storage fixes, each proven by a new IT or unit test that fails on the
previous code:
- Empty lists: the continuous profiling targets and the eBPF profiling data
  built `in ()`, an SQL syntax error on MySQL, MariaDB and PostgreSQL; an
  empty list now reads nothing. The segment lookup by trace and instance
  gets the guard #14242 added to the lookup by segment ID.
- Queries across day tables: the trace list pages, the Zipkin search limit
  and traces split at midnight, heatmap columns, Top N (ranked by the exact
  mean of the whole range, values above the int range), slow statements,
  tag autocomplete limits, the async-profiler, pprof and trace profiling
  task lists, and the instances of every day table.
- Trace profiling: Go thread snapshots read as Java, and a day table without
  a segment's snapshots read as sequence 0.
- The process list of an agent ignored the ping time range; browser error
  logs failed without a time range; events filtered by layer failed on
  PostgreSQL, and an event at a time range's first millisecond was left out.
- UI templates: the lookup by ID missed every template, so disabling one
  failed, and changing a missing one reported success.
- A zero-row UPDATE did not drop the metric from OAP's cache, so later
  flushes kept updating its missing row. The failure callback now drops it,
  and a later flush inserts the row again.
- The boot table check logged wrong table and column states.

Async-profiler fixes in shared OAP core:
- On JDBC and Elasticsearch, the check refusing a second task on a service
  compared a minute bucket with the tasks' second buckets and never found
  the running task. It now uses a second bucket.
- A cache refresh could keep the older of two recent tasks for the agent.

Tests and CI:
- New JDBC storage ITs on PostgreSQL 13 and MariaDB 11.4 (Testcontainers),
  40 cases on each, run in the default Integration test job. MariaDB
  Connector/J (LGPL) is a test-scope dependency only.
- The Zipkin and span-attached event ITs of the Elasticsearch storage also
  carry `@Tag("elastic-it")` and run as one slow-IT matrix entry.
@wu-sheng
wu-sheng force-pushed the fix/jdbc-query-dao-bugs branch from f074ed4 to 1224096 Compare November 17, 2026 05:35
@wu-sheng
wu-sheng requested a review from wankai123 November 17, 2026 05:36
@wu-sheng
wu-sheng merged commit ae9ea84 into master Nov 17, 2026
491 of 493 checks passed
@wu-sheng
wu-sheng deleted the fix/jdbc-query-dao-bugs branch November 17, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend OAP backend related. test Test requirements about performance, feature or before release.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants