Repository navigation
Fix JDBC storage query bugs and two async-profiler task bugs, and test the JDBC DAOs on PostgreSQL and MariaDB - #14243
Merged
Merged
Conversation
wu-sheng
marked this pull request as ready for review
November 16, 2026 23:44
wankai123
previously approved these changes
Nov 16, 2026
4 tasks done
wu-sheng
force-pushed
the
fix/jdbc-query-dao-bugs
branch
from
November 17, 2026 03:25
7bfe20a to
f074ed4
Compare
…t the JDBC DAOs on PostgreSQL and MariaDB JDBC storage fixes, each proven by a new IT or unit test that fails on the previous code: - Empty lists: the continuous profiling targets and the eBPF profiling data built `in ()`, an SQL syntax error on MySQL, MariaDB and PostgreSQL; an empty list now reads nothing. The segment lookup by trace and instance gets the guard #14242 added to the lookup by segment ID. - Queries across day tables: the trace list pages, the Zipkin search limit and traces split at midnight, heatmap columns, Top N (ranked by the exact mean of the whole range, values above the int range), slow statements, tag autocomplete limits, the async-profiler, pprof and trace profiling task lists, and the instances of every day table. - Trace profiling: Go thread snapshots read as Java, and a day table without a segment's snapshots read as sequence 0. - The process list of an agent ignored the ping time range; browser error logs failed without a time range; events filtered by layer failed on PostgreSQL, and an event at a time range's first millisecond was left out. - UI templates: the lookup by ID missed every template, so disabling one failed, and changing a missing one reported success. - A zero-row UPDATE did not drop the metric from OAP's cache, so later flushes kept updating its missing row. The failure callback now drops it, and a later flush inserts the row again. - The boot table check logged wrong table and column states. Async-profiler fixes in shared OAP core: - On JDBC and Elasticsearch, the check refusing a second task on a service compared a minute bucket with the tasks' second buckets and never found the running task. It now uses a second bucket. - A cache refresh could keep the older of two recent tasks for the agent. Tests and CI: - New JDBC storage ITs on PostgreSQL 13 and MariaDB 11.4 (Testcontainers), 40 cases on each, run in the default Integration test job. MariaDB Connector/J (LGPL) is a test-scope dependency only. - The Zipkin and span-attached event ITs of the Elasticsearch storage also carry `@Tag("elastic-it")` and run as one slow-IT matrix entry.
wu-sheng
force-pushed
the
fix/jdbc-query-dao-bugs
branch
from
November 17, 2026 05:35
f074ed4 to
1224096
Compare
wankai123
approved these changes
Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix bugs of the JDBC storage, and two async-profiler task bugs
#14239 was fixed by #14242, which merged first. This PR gives
JDBCTraceQueryDAO.queryByTraceIdWithInstanceIdthe same guard: an empty list would also buildin (). Its only caller,ProfileTaskQueryService.getProfileTaskSegments, returns before calling it when there is no segment, so this guard is defensive. The new ITs read both lookups with an empty list on PostgreSQL and MariaDB.Day tables. The JDBC storage keeps records and metrics in one table per day. Many queries ran their SQL on each day table and concatenated the answers, so ranking, limits and grouping applied to each day instead of to the queried range:
JDBCTraceQueryDAO.queryBasicTraces): each table applied the page's offset and limit, and the pages were concatenated unsorted. Each table now returns its firstfrom + limittraces, and the page is cut after one sort by start time or duration, as the query asks.JDBCZipkinQueryDAO): each table returned up tolimittrace IDs, so the search could return more. Each table still selects up tolimittrace IDs; these are now merged, ranked newest first by the earliest matching span each table returned for the trace, and cut once. Spans were grouped into traces per table, so a trace with spans on both days came back as two partial traces, in a search and intraceMany; spans are now grouped across tables.JDBCMetricsQueryDAO.readHeatMap): each table filled the points it did not hold with zeroed columns, so a range across midnight got extra columns. The columns are now built in point order once every table is read.JDBCAggregationQueryDAO.sortMetrics): each table averaged and cut its own top N, and the results were averaged again, so an entity was ranked by the average of its daily averages, and one that topped the range but no single day was missed. The values were also read withgetInt, which fails on values above 2147483647. With one table, the database still ranks and cuts. Across tables, each table now returns every entity's sum and sample count; entities are ranked by the exact mean, comparing cross-products of sums and counts, and the value returned is the truncated mean.JDBCRecordsQueryDAO): each table returned its own top N, concatenated unranked. They are now ranked together and cut once.JDBCTagAutoCompleteQueryDAO): the union of the tables could exceed the limit, so the limit now applies to it.JDBCAsyncProfilerTaskQueryDAO,JDBCPprofTaskQueryDAO): with a time range, tables are read oldest day first, so the limit kept the older days' tasks. The tasks are now sorted newest first before the limit.JDBCProfileTaskQueryDAO.getTaskList): a task is stored in the table of the day it was created, but its time bucket is its scheduled start. The agent's task refresh queries with both time bounds, and the tables were chosen by those bounds, so a task scheduled for a later day than it was created could not reach the agent. The check of a new task against the latest-starting one queries without a start bound, so it read every table, but each table was sorted and cut on its own, and the check could inspect the wrong task and let a new task be created over a running one. Every table within the TTL is now read, and the tasks are sorted by start time, latest first, before the limit.JDBCProfileThreadSnapshotQueryDAO): the language column was never read, so a Go service's thread snapshots were analyzed as Java. A day table holding none of a segment's snapshots answered the min/max sequence query with SQL NULL, read as sequence 0, so the analysis could read snapshots outside the selected time range. NULL now counts as no answer.JDBCMetadataQueryDAO.getInstances): it returned the first day table's answer only, so the continuous profiling instance list missed instances first seen on an earlier day. It now reads every table.Other JDBC bugs:
JDBCMetadataQueryDAO.listProcesses(agentId, …)): the ping time range was dropped, so Rover could receive eBPF profiling tasks for processes that had stopped reporting.JDBCEBPFProfilingTaskDAO.queryTasksByTargets(a continuous profiling policy saved without targets) andJDBCEBPFProfilingDataDAO.queryData(an analysis with no schedule) builtin (). They now return nothing.JDBCBrowserLogQueryDAO) failed with a NullPointerException. They are now read from every table within the TTL.JDBCEventQueryDAO): the layer was bound as a string, which PostgreSQL rejects against the integer column. The start of the time range was exclusive, leaving out an event at its first millisecond.JDBCSQLExecutor.getByIDalways prefixed the ID with the model name, which the insert does only for records and function metrics; a management row such as a UI template is stored under its raw ID. So disabling a template always failed with "Can't find the template", andchangeTemplate, which didn't check, reported success for a template that does not exist. The lookup now builds the ID as the insert does, andchangeTemplatereports a missing template.JDBCRuntimeRuleManagementDAO's comment about working around that lookup is updated.BatchSQLExecutor): an insert and an update both reach it as aSQLExecutor, which implements both request types, so every batch was taken as inserts and an update that found no row never calledonUpdateFailure. OAP kept the metric cached and kept updating its missing row. The row count now decides: an UPDATE of zero rows callsonUpdateFailure, which drops the metric from the cache, so the next flush of that metric checks storage and inserts the row again.JDBCTableInstaller.isExists) logged the table as missing and the column state inverted.Async profiler (shared OAP core):
AsyncProfilerMutationService.checkTaskProfilingbounded the time bucket by the minute bucket of now (12 digits), while a task row carries the second bucket of its creation (14 digits), so no task was ever at or below the bound. It now uses the second bucket.AsyncProfilerTaskCachesaves each task its query returns, those created from five minutes before now to five minutes after, and each save replaced the service's cached task, so a newest-first list left the older task cached. A task now replaces the cached task only if it was created no earlier.Tests
JDBC<Area>PostgreSQLITandJDBC<Area>MariaDBITfor eight areas (TraceQuery,ZipkinQuery,LogQuery,MetricsQuery,MetadataQuery,ProfilingTaskQuery,ProfilingDataQuery,Management), 40 cases on each database. They install the tables with the plugin's installers asJDBCStorageProviderdoes (column renames included), write through OAP's DAOs and read through the production query DAOs, with time-series data on yesterday's and today's tables. They carry noslowtag, so they run in the default Integration test job; the whole set takes about 45 seconds locally.mariadbmodule are test-scope dependencies of the JDBC plugin only, never distributed. The MySQL driver stays out.JDBCAggregationQueryDAOTestcases (fractional sums across days, ranking by the exact mean), and the newAsyncProfilerMutationServiceTestandAsyncProfilerTaskCacheTest. No existing test body or assertion changed.CI
@Tag("elastic-it"), and the slow-integration-test job runs them as one "Elastic IT" entry (-Dgroups=elastic-it) instead of two, saving one whole-project build.ElasticSearchITkeeps its own entry. Comments inskywalking.yamllist what each IT job runs, and the/testskill matches the new matrix.Validation
On master's code, the final tests fail as expected: 73 of the 80 new IT cases (62 failures, 11 errors), and the 2 new Top N unit tests. The 7 that pass are controls (segments by ID on both days, and the boot check of a model without a table, on each database), the empty segment-ID list that Handle empty JDBC trace profile segment lists #14242 fixed, on each database, and the layer filter on MariaDB, which converts the string itself. On master,
AsyncProfilerMutationServiceTestfails withexpected: <ALREADY_PROFILING_ERROR> but was: <SUCCESS>andAsyncProfilerTaskCacheTestwithexpected: <created> but was: <ended>.At
f074ed4bab, rebased onto mastere564f75793after Handle empty JDBC trace profile segment lists #14242: the JDBC plugin's 118 unit tests and 81 ITs (the 80 new andJDBCHistoryDeleteDAOIT) passed; the whole project's unit tests, 4,201 on JDK 17, passed (0 failures, 0 errors, 5 skipped); checkstyle passed, and the JDK 11clean flatten:flatten install javadoc:javadoc -Pallbuild succeeded. Before that rebase, which changed no Elasticsearch code,-Dgroups=elastic-itran exactly the 12 tagged classes, 174 tests, 0 failures, in 9:22 min.After rebasing onto master
381fcbd488(Decide the layer of a log without one by LAL fallback rules, and save OTLP_GENERAL logs #14244) as1224096b83: checkstyle passes, the JDK 11clean flatten:flatten install javadoc:javadoc -Pallbuild succeeds, and the whole project's unit tests, 4,284 on JDK 17, pass (0 failures, 0 errors, 5 skipped), the JDBC plugin's 118 among them.If this pull request closes/resolves/fixes an existing issue, replace the issue number. Not applicable: [Bug] JDBC trace profiling query fails with SQL syntax error when the segment list is empty #14239 was already addressed by Handle empty JDBC trace profile segment lists #14242.
Update the
CHANGESlog.