Skip to content

Cut per-round overhead of the executor and explorer - #82

Closed
nils-loose wants to merge 7 commits into
svcomp27/integrationfrom
perf/round-overhead
Closed

nils-loose wants to merge 7 commits into
svcomp27/integrationfrom
perf/round-overhead

Conversation

@nils-loose

Copy link
Copy Markdown
Contributor

Cuts the per-round cost of the SV-COMP loop (one executor JVM per round, then branch selection in the explorer). The verdict logic is unchanged. JVM reuse (#56) is out of scope.

Changes

Executor

  • C1 only (-XX:TieredStopAtLevel=1, set in SVCompDriver.build_command). A round runs the target once, mostly for less than a second, and C2 compiles do not pay off in that time. A MinePump round drops from 1.58 to 0.95 executor CPU-s. Long executions are no slower (Hanoi with n=13/14: 13.6 -> 12.1 and 44 -> 42 CPU-s).
  • No solver-context close at the end of an SV-COMP execution. The JVM exits right after the execution. Z3SolverContext.close (delContext) took 61% of a long-trace run (Hanoi n=14: 44 -> 12.5 CPU-s together with the next change).
  • FormulaDumper: one Z3 solver for all trace formulas. java-smt's dumpFormula creates and initialises a new solver for every formula. The dumper uses one solver with push/assert/print/pop, then runs java-smt's own sanitize, so the output is identical. It falls back to dumpFormula if the reflective lookup fails. I checked the trace JSON of about 50 tasks from all families against the base jar: identical. The only differences were stack-overflow depths, which also vary from run to run on the base.
  • DTOBuilder:
    • variables are extracted once per branch, with a precompiled input-name pattern;
    • compact JSON (still indented in PRINT mode);
    • constraint/inst/concreteOnly are left out when at their defaults (the explorer's pydantic defaults are the same).
  • Null log manager for java-smt. BasicLogManager registers a JMX bean, which loads about 300 classes per run, and its log is never read.

Explorer

  • Branch selection walks the Database's tree under the lock instead of a deep copy. Before, every round deep-copied the tree three times: once in select_branch and twice in determine_next_step, which only needs the count of symbolic variables. The driver keeps a shallow copy of the target node, so check_divergence still sees the side that was open when the branch was solved.
  • is_symbolic_branch is memoised per (gid, trace_id). A non-symbolic open branch used to be re-checked and logged in every round.
  • The per-node [DFS] line is only written when an SA graph is in use.
  • _extract_variables is iterative and visits each subterm once (cherry-pick of 58c15ae from fix/explorer-robustness, same patch).

Measurements

Config: --sv-comp, --mem15g, nosa, 2 cores per run. Base is svcomp27/integration.

Speed (base and branch run side by side):

  • 40 baseline timeouts at 60 s CPU, rounds per CPU-s:
    • median 1.56x, geomean 1.46x, p75 2.01x, p90 2.15x, max 6.93x;
    • on the 23 tasks with more than 5 rounds: median 1.85x, range 1.0-2.19x;
    • by family: MinePump 2.08x, TCAS 1.88x, jdart 1.85x, float_unboundedloop 1.52x;
    • unchanged (1.0x) where a run is stuck in one long solver query or execution (float-nonlinear, securibench).
  • 40 quick correct runs: CPU per round 2.00 -> 1.20 s (median), total CPU 186 -> 109 s.
  • All 1047 baseline-correct runs: total CPU 7687 -> 5040 s.

Verdicts

  • Regression set, all 1047 baseline-correct runs: 0 changed, 0 new wrong verdicts.
  • Timeout set, 229 of the 276 baseline timeouts (all except 47 of the 67 float-nonlinear-calculation runs, which are solver-bound):
    • 3 flips, +5 points, 0 lost, 0 new wrong verdicts;
    • all 3 re-run once on both sides with the same result.
task prop base branch
argv-tasks/ProjectionGridRoundTripper_exceptionless no-runtime-exception timeout SAFE (276 s)
float_unboundedloop/Adjustable-Inner-Count-Halving valid-assert timeout VIOLATION (216 s)
jdart-regression/addition01 no-runtime-exception timeout SAFE (166 s)

Other changes, all 0 points:

  • 3 jayhorn McCarthy91 runs: timeout -> UNKNOWN. Round 103 is reached sooner, and its input (I_0=-2147483647) overflows the stack inside the executor. The base jar does the same with that input.
  • 2 SatHanoi01 runs: timeout -> stall. Round 4 now gets its trace to the explorer without spending the rest of the budget in delContext. Tree.add then hits RecursionError and the Database lock leaks. fix/explorer-robustness fixes that.

Tests

  • New explorer unit tests in tests/test_round_overhead.py:
    • selection order and filtering;
    • no tree copy;
    • the memoised symbolic check;
    • the target snapshot;
    • variable extraction on shared and deep terms.
  • Existing explorer tests pass.
  • Executor test and agentTest pass.

Known limits

  • Conflicts are likely with fix/explorer-robustness (Database with lock rewrite, DFS, ConstraintManager) and with branches that touch DTOBuilder/check_divergence. Each overlap is a few lines.
  • The JVM floor is about 0.93 CPU-s per round: class loading and agent setup.
    • AppCDS cannot be used here: dumping needs the diagnostic AllowArchivingWithJavaAgent, and the benchmark class directories block it.
    • SerialGC/ParallelGC, -XX:-UsePerfData and CICompilerCount gave less than 2%. -Xverify:none was not adopted, since it would hide bad instrumented bytecode.
  • Not addressed:
    • shadow-array-eager-copy-init: a constant-array initialisation changes what out-of-bounds selects mean and removes a precision-loss signal, so it needs its own soundness argument;
    • trace-encoding-quadratic-simplify: simplify is about 16% of a long-trace encode now.

🤖 Generated with Claude Code

nils-loose and others added 7 commits October 18, 2026 22:43
The walk was recursive and visited shared subterms once per path to them,
which is exponential in the depth of a DAG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The JVM runs one execution and exits right after it, and Z3 frees the
terms of the trace one by one when the context is closed: minutes after a
long trace, while the explorer waits for the JVM to exit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
java-smt dumps a formula by creating a solver, asserting the formula and
printing the solver; initialising the solver costs more than printing a
small formula. FormulaDumper keeps one solver and asserts each formula in
its own scope, then sanitises the output as java-smt does, so the dump is
the same.

Also extract the variables of a branch formula once (and not after
precision loss is known), send the JSON without indentation (still
indented in PRINT mode) and leave default fields out of branch elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BasicLogManager registers a JMX bean, which loads about 300 JMX classes
in every executor run; java-smt's log is not read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A round runs the target once, for well under a second in most tasks. C2
compiles cost CPU that the short-lived JVM does not get back: with C1
only, an executor run of a MinePump round takes 0.95 instead of 1.58 CPU
seconds, and long executions are no slower.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every round deep-copied the whole tree three times (branch selection and
two counts of the symbolic variables), and the solutions with it, which
grows with the tree. Branch selection now walks the Database's tree under
its lock, and the driver keeps a shallow copy of the target branch, whose
open side check_divergence needs after the next trace has filled it in.

A branch whose side was not taken stays a candidate and was checked for
symbolic variables again every round, with two log lines each time; the
answer only depends on the node, so it is kept. The per-node DFS log line
is only written when there is an SA graph to report on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nils-loose added a commit that referenced this pull request Oct 19, 2026
_extract_variables: one copy (#82 cherry-picks #72's patch).
StrategyService.py, Database.py: #82 walks the Database's tree under the
lock; visit_tree passes the solved and #74's retired branches, in #72's
'with lock' style.
SVCompDriver.py: #82's shallow snapshot of the target node for
check_divergence, with #74's solution record and #80's guess flag; the
ingest-failure (#72) and stack-overflow (#78) checks read the Database
without copying the tree.
DTOBuilder.java: #82's FormulaDumper with #71's SymbolRegistry check.
DFS.py: #82 logs the per-node line only with an SA graph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nils-loose added a commit that referenced this pull request Oct 19, 2026
Combining #74 and #82: check_divergence gets a shallow copy of the
target, so #74's check whether another run reached the given-up side
looked at a node that never changes, and every given-up branch blocked
SAFE. The copy keeps a reference to the tree's node, which give_up_branch
records. The driver test fakes gain #82's Database reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nils-loose added a commit that referenced this pull request Oct 19, 2026
SVCompDriver.py:
- one C1-only option (#82 and #88 both added it), in #88's
  EXECUTOR_JVM_OPTIONS, with #88's budget-derived heap;
- determine_next_step drops a stack-overflow run's violations (#78)
  before #88's out-of-memory and resource-limit handling, and reads the
  Database without copying the tree (#82);
- a run stopped by the budget is not checked for divergence; #72's
  no-trace check applies to the others;
- every query, first or retry (#74), runs under the budget's interrupt;
- #88's cut-short check joins #72's incomplete-search check, and both
  apply to every SAFE;
- the stats keep the pre-check (#73), divergence retries (#74) and
  resources (#88).
DFS.py: the per-node line is at #88's TRACE level, with the CHOICE label.
TestcaseStats.py, test_solver_rlimit.py: both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nils-loose added a commit that referenced this pull request Oct 19, 2026
Combining #72, #74, #78, #82 and #88: the driver tests create #88's
budget and verdict lock, and #88's tests get the stack-overflow (#78) and
copy-free (#82) Database reads and the choice flag (#76). A run reports
one verdict (#88), so the incomplete-search subtests reset the flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nils-loose added a commit that referenced this pull request Oct 19, 2026
Tree.py: #84's loop-state positions are registered at the start of #72's
iterative Tree.add; the recursive add_recursive is not brought back.
Node/Branch/TraceParser/DataTransferObjects.py, BranchDTO/DTOBuilder.java:
both the choice flag (#69/#76) and the loop states (#84).
SymbolicTraceHandler.java: addChoiceBranch, checkAndSetLibraryBranch and
the loop-state collection side by side.
StrategyService.py: the walk on the live tree (#82) reports #84's count
of pruned subtrees.
SVCompDriver.py: #84's --no-loop-state-matching next to the other state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nils-loose added a commit that referenced this pull request Oct 19, 2026
…tates to every branch

Combining #72, #82 and #84:
- #84 pruned a subtree below a repeated loop-head state by returning from
  the recursive dfs. In #72's iterative walk that return ended the whole
  walk, so the open branches of every later subtree were never offered,
  and an exhausted search could report SAFE. It now skips that subtree
  only (regression test added).
- The pruned-subtree count is reset for each walk, as #82 walks the
  persistent tree instead of a copy.
- The choice branches (#69/#76) and library-call branches (#75/#90) take
  the pending loop-head states too, so a state always goes with the
  branch right after it, which its position assumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nils-loose nils-loose mentioned this pull request Oct 20, 2026
@nils-loose

Copy link
Copy Markdown
Contributor Author

Included in dev through #92 (merge commit 534c182), together with the fixes for interactions between PRs. This PR stays as the record of the change and its measured flips.

@nils-loose nils-loose closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant