Separate harness implementation from the harness framework by rafaelfranca · Pull Request #446 · ruby/ruby-bench

rafaelfranca · 2025-11-21T22:44:05Z

I had to spent some time understanding how the harness worked in this project. I realized that there were multiple directories for them and didn't know why.

I later realized that some were different harness implementation, and the harness folder was both the implementation and the framework.

I made this change the separate those two concepts, and remove the need to have different directories for implementation.

The framework lives in lib/harness and the implementations live in harness, in separated files (ractor.rb, perf.rb, etc.)

k0kubun · 2025-11-21T23:06:01Z

https://github.com/ruby/ruby-bench/actions/runs/19585546525/job/56093508019?pr=446#step:4:873
Running benchmark "railsbench" (46/67)

/home/runner/work/ruby-bench/ruby-bench/lib/harness/extra.rb:62: warning: already initialized constant MIN_BENCH_TIME
/home/runner/work/ruby-bench/ruby-bench/harness/default.rb:10: warning: previous definition of MIN_BENCH_TIME was here

Could you fix this warning? (I'm sorry if it wasn't new)

/home/runner/work/ruby-bench/ruby-bench/harness/stackprof.rb:72:in 'Object#run_benchmark': uninitialized constant StackProf (NameError)

StackProf.run(out: out, **opts) do
^^^^^^^^^

Do you know why the stackprof harness is loaded on the benchmark-default CI?

eregon · 2025-12-02T14:32:48Z

Given there is a once harness, maybe we should just remove run_once.sh and run_once.rb?
When running a single iteration of a benchmark (the meaning of once in this context) I don't think the harness matters much, with the one exception maybe of Ractors (if that's useful then maybe we can keep the trivial run_once.sh script).

My concern is I think we should not introduce "yet another way to run benchmarks". To be fair run_once.rb is pretty small but it seems unnecessary to me, I think HARNESS=once ruby some/benchmark.rb is enough.

eregon

LGTM but I would just remove run_once.rb, it seems unnecessary and we already have 2 ways to run benchmarks (run_benchmarks.rb and directly with ruby ... benchmark.rb)

eregon · 2025-12-02T14:53:03Z

I can take a look at the test failures on truffleruby and push a commit to fix them if you'd like, let me know if so.

XrXr

LGTM since the ruby benchmark.rb interface is preserved. Having to say -r... instead of -I... is fine.

eightbitraptor · 2026-10-13T11:35:41Z

Ok. This PR has been approved and unmerged fora while now. I've taken it on and rebased it on top of the latest main and fixed a few things:

I removed run_once.rb. I concur with @eregon that it's not necessary since we can use either ruby with --harness and --iterations or via run_benchmarks.rb --once
Some issues with harness loading have been ironed out. We now require HARNESS to be loaded by an absolute path because otherwise the stackprof and vernier harness ended up attempting to load them gem by mistake.
--with-pre-init is loaded before the harness now - previously the Ractor harness froze main when it loaded so loading a pre-init file after it raised FrozenError
I moved the stackprof and the vernier harness output back to the top level data dir rather than lib/data

Few other documentation fixes and nits. Because of these changes I'm going to re-request review before merging. Thanks folks.

Use file based harness instead of directory based harness

… selection So we don't need to deal with the `-r` ruby argument directly for selecting harnesses.

This allow `ruby` script direct calls to still select the harness without using any wrapper script.

lib/harness/extra.rb moved from harness/ to lib/harness/ but still built its output path from ../data, so the stackprof and vernier harnesses wrote to lib/data, which does not exist. harness_name also matched any */harness/*.rb feature, so lib/harness/loader.rb won and vernier named its profile loader-*.json. In this commit we build the output path from ../../data and take the harness name only from files directly inside the top level harness directory.

The loader passed HARNESS to require as a bare feature name. When a gem has the same name, such as stackprof or vernier, require loaded the gem and run_benchmark was never defined. In this commit we require harness/<name>.rb by its absolute path and keep the error that lists the available harnesses.

The default harness was selected by passing no -r at all, which left the choice to lib/harness/loader.rb. The loader reads HARNESS first, so an exported HARNESS replaced --harness=default and default_harness: default in benchmarks.yml, while the result was still labelled default. In this commit we set HARNESS to the resolved harness in the environment of the benchmark process. The harness that BenchmarkSuite picks is now the harness that the benchmark loads, whatever the parent environment holds.

The harness was passed as -r ahead of the --with-pre-init library, so Ruby loaded a non-default harness first. The Ractor harness calls Ractor.make_shareable(self) at load time, which froze main before the pre-init file ran, and a pre-init that set a top-level instance variable raised FrozenError. The default harness still loaded after pre-init, from the benchmark, so the order also depended on the harness. In this commit we pass the pre-init library before the harness. The pre-init file now runs before every harness, as it did when the benchmark itself loaded the harness.

Before the harnesses moved to harness/<name>.rb, --harness=harness selected the default harness. The prefix stripping turned harness-<name> into <name> but left harness unchanged, so every benchmark tried to load harness/harness.rb and failed with LoadError. The help text still named harness as the default. In this commit we map harness to default and change the help text to say default.

There are already two ways to run a benchmark: run_benchmarks.rb, and ruby on the benchmark file, where HARNESS or -r selects the harness and WARMUP_ITRS, MIN_BENCH_ITRS and MIN_BENCH_TIME set the iterations. run_once.rb added a third way that wrapped the second one in an exec. In this commit we remove run_once.rb and its tests. The CI job that ran it now runs railsbench once directly with ruby, and the README and lib/harness/loader.rb no longer refer to the script. The README example that runs a benchmark directly now sets WARMUP_ITRS=0 MIN_BENCH_ITRS=1 MIN_BENCH_TIME=0, the same values that run_benchmarks.rb --once sets.

burn_in.rb ran ruby -Iharness -r<name>. Now that harness/ holds stackprof.rb and vernier.rb, -Iharness let those files shadow the stackprof and vernier gems. railsbench calls Bundler.require, so it loaded harness/stackprof.rb instead of the gem and failed with uninitialized constant StackProf. In this commit we pass the harness to -r by absolute path and drop -Iharness, the same way lib/benchmark_suite.rb does. The harness directory never enters $LOAD_PATH, so gem requires resolve to gems.

The README said chained harnesses run in sequence and that once,default runs the benchmark once and then with default iterations. harness/chain.rb nests them instead: the first harness is the outermost, so once,default ran the default iterations inside a single once call and then once overwrote the results with a fake 1ms sample. The ractor example also pointed at benchmarks-ractor/, which does not exist. In this commit we describe the nesting, state that the outermost harness writes the results that are kept, and replace both examples with vernier,default and vernier,ractor on benchmarks/fib.rb.

The usage examples still passed -I harness-stackprof, a directory that no longer exists, and the second example set MIN_BENCH_ITRS=10 twice. HARNESS=stackprof already selects the harness, so the commands only worked because Ruby ignores a missing -I path.

The usage examples still passed -I harness-vernier, a directory that no longer exists, and the header named the variable NO_VIERWER although the harness reads NO_VIEWER. HARNESS=vernier already selects the harness.

rafaelfranca requested a review from a team November 21, 2025 22:44

eregon reviewed Nov 22, 2025

View reviewed changes

Comment thread README.md Outdated

rafaelfranca force-pushed the rmf-better-harness branch 3 times, most recently from eccdfb4 to d3153ab Compare November 24, 2025 20:46

eregon reviewed Nov 24, 2025

View reviewed changes

Comment thread README.md Outdated

XrXr requested changes Nov 25, 2025

View reviewed changes

Comment thread README.md Outdated

rafaelfranca force-pushed the rmf-better-harness branch 3 times, most recently from c82ec54 to 82e0b03 Compare November 27, 2025 19:50

rafaelfranca requested a review from XrXr November 27, 2025 19:51

k0kubun approved these changes Dec 1, 2025

View reviewed changes

rafaelfranca force-pushed the rmf-better-harness branch from 82e0b03 to 01698a8 Compare December 1, 2025 17:38

eregon approved these changes Dec 2, 2025

View reviewed changes

Comment thread README.md Outdated

Comment thread harness/stackprof.rb Outdated

eregon reviewed Dec 2, 2025

View reviewed changes

Comment thread README.md Outdated

XrXr approved these changes Dec 8, 2025

View reviewed changes

eightbitraptor force-pushed the rmf-better-harness branch 2 times, most recently from 4a80027 to 62c21c2 Compare October 13, 2026 11:35

eightbitraptor requested review from XrXr, eregon, k0kubun and tekknolagi October 13, 2026 11:35

rafaelfranca added 4 commits October 13, 2026 12:37

Movel all hardness to a single directory

762cb0e

Use file based harness instead of directory based harness

Keep the harness implementation separated from the harness framework

c0b0bc7

Improve the run_once script to better handle Ruby options and harness…

e0139ba

… selection So we don't need to deal with the `-r` ruby argument directly for selecting harnesses.

Allow using HARNESS environment variable to select harness

f072f07

This allow `ruby` script direct calls to still select the harness without using any wrapper script.

rafaelfranca and others added 13 commits October 13, 2026 12:37

Improve chain harness documentation and error handling

ecd6717

Build harness arguments once during initialization

45e110e

Close the perf code block in the README

68c769c

Drop the removed harness directory from the vernier usage

47775f9

The usage examples still passed -I harness-vernier, a directory that no longer exists, and the header named the variable NO_VIERWER although the harness reads NO_VIEWER. HARNESS=vernier already selects the harness.

eightbitraptor force-pushed the rmf-better-harness branch from 62c21c2 to 47775f9 Compare October 13, 2026 11:37

eightbitraptor self-assigned this Oct 13, 2026

Navigation Menu

Repository navigation

Separate harness implementation from the harness framework - #446

Separate harness implementation from the harness framework#446
rafaelfranca wants to merge 17 commits into
mainfrom
rmf-better-harness

rafaelfranca commented Nov 21, 2025 •

edited

Loading

Uh oh!

k0kubun commented Nov 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eregon commented Dec 2, 2025 •

edited

Loading

Uh oh!

eregon left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eregon commented Dec 2, 2025

Uh oh!

XrXr left a comment •

edited

Loading

Uh oh!

eightbitraptor commented Oct 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

rafaelfranca commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k0kubun commented Nov 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eregon commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eregon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eregon commented Dec 2, 2025

Uh oh!

XrXr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eightbitraptor commented Oct 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rafaelfranca commented Nov 21, 2025 •

edited

Loading

eregon commented Dec 2, 2025 •

edited

Loading

XrXr left a comment •

edited

Loading