diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 0960f70c..0c6e32d2 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -152,8 +152,12 @@ jobs: with: ruby-version: head - - name: Test run_once.sh - run: ./run_once.sh --yjit-stats benchmarks/railsbench/benchmark.rb + - name: Run a benchmark once directly with Ruby + run: ruby --yjit-stats benchmarks/railsbench/benchmark.rb + env: + WARMUP_ITRS: '0' + MIN_BENCH_ITRS: '1' + MIN_BENCH_TIME: '0' all-tests: runs-on: ubuntu-latest diff --git a/README.md b/README.md index fb758955..f418c0a3 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Small set of benchmarks and scripts for the Ruby programming language. The benchmarks are found in the `benchmarks` directory. Individual Ruby files in `benchmarks` are microbenchmarks. Subdirectories under `benchmarks` are larger macrobenchmarks. Each benchmark relies on a harness found in -[./harness/harness.rb](harness/harness.rb). The harness controls the number of times a benchmark is +[./harness/default.rb](harness/default.rb). The harness controls the number of times a benchmark is run, and writes timing values into an output file. The `run_benchmarks.rb` script (optional) traverses the `benchmarks` directory and @@ -97,8 +97,8 @@ This is the easiest way to run a single benchmark. It requires no setup at all and assumes nothing about the Ruby you are benchmarking. It's also convenient for profiling, debugging, etc, especially since all benchmarked code runs in that process. -``` -ruby benchmarks/some_benchmark.rb +```bash +ruby benchmarks/fib.rb ``` ### Benchmark organization @@ -138,19 +138,19 @@ There are two Ractor-related categories: * **`--category ractor`** - Runs both regular benchmarks marked with `ractor: true` in `benchmarks.yml` AND all benchmarks from the `benchmarks-ractor` - directory. The `harness-ractor` harness is used for both types of benchmark. + directory. The `ractor` harness is used for both types of benchmark. * **`--category ractor-only`** - Runs ONLY benchmarks from the `benchmarks-ractor` directory, ignoring regular benchmarks even if they are marked with `ractor: true`. This category also automatically uses the - `harness-ractor` harness. + `ractor` harness. ### Directory Structure The `benchmarks-ractor/` directory sits at the same level as the main `benchmarks` directory, and contains Ractor-specific benchmark implementations that are designed to test Ractor functionality. They are not -intended to be used with any harness except `harness-ractor`. +intended to be used with any harness except `ractor`. ### Usage Examples @@ -162,7 +162,7 @@ intended to be used with any harness except `harness-ractor`. ./run_benchmarks.rb --category ractor-only ``` -Note: The `harness-ractor` harness is automatically selected when using these +Note: The `ractor` harness is automatically selected when using these categories, so there's no need to specify `--harness` manually. ### Ractor counts @@ -188,12 +188,12 @@ because no single process ran all counts. The text summary, the CSV output, and `misc/zjit_diff.rb` show one row for each count, with the RSS and JIT stats of that count's process. -When you run a benchmark directly with `-Iharness-ractor`, the harness runs +When you run a benchmark directly with `-r./harness/ractor`, the harness runs all counts in one process, one count after another. ### Ractor Scenario Benchmarks -By default, `harness-ractor` spawns the worker Ractors and runs the benchmark +By default, the `ractor` harness spawns the worker Ractors and runs the benchmark block inside each of them. A benchmark that calls `run_benchmark(n, scenario: true)` uses scenario mode instead. The harness calls the block one time per trial in the main Ractor, with the Ractor count as @@ -232,7 +232,7 @@ with multiple ractors, for GC work that reclaims ractor-local memory: ```bash # Every count in one process -ruby -Iharness-ractor benchmarks/ractor-dead-set/benchmark.rb +ruby -r./harness/ractor benchmarks/ractor-dead-set/benchmark.rb # One process per count, comparing two Rubies, with Ractor GC metrics RUBY_BENCH_RACTORS=1,2,4,6,8 ./run_benchmarks.rb --category ractor --ractor-gc \ @@ -321,26 +321,62 @@ This file will then be passed to the underlying Ruby interpreter with ## Harnesses -You can find several test harnesses in this repository: +You can find several test harnesses in the `harness/` directory: + +* `default` - the normal default harness, with duration controlled by warmup iterations and time/count limits +* `bips` - a harness that measures iterations/second until stable +* `continuous` - a harness that adjusts the batch sizes of iterations to run in stable iteration size batches +* `once` - a simplified harness that simply runs once +* `perf` - a simplified harness that runs for exactly the hinted number of iterations +* `ractor` - a harness that runs the benchmark inside Ractors to measure scaling +* `gc` - the default harness plus GC timing, count and heap statistics +* `stackprof` - a harness to profile the benchmark with stackprof +* `stats` - count method calls and loop iterations +* `vernier` - a harness to profile the benchmark with vernier +* `warmup` - a harness which runs as long as needed to find warmed up (peak) performance +* `chain` - a harness to chain multiple harnesses together +* `mplr` - a harness for multiple iterations with time limits -* harness - the normal default harness, with duration controlled by warmup iterations and time/count limits -* harness-bips - a harness that measures iterations/second until stable -* harness-continuous - a harness that adjusts the batch sizes of iterations to run in stable iteration size batches -* harness-once - a simplified harness that simply runs once -* harness-perf - a simplified harness that runs for exactly the hinted number of iterations -* harness-stackprof - a harness to profile the benchmark with stackprof -* harness-stats - count method calls and loop iterations -* harness-vernier - a harness to profile the benchmark with vernier -* harness-warmup - a harness which runs as long as needed to find warmed up (peak) performance +### Selecting a harness -To use it, run a benchmark script directly, specifying a harness directory with `-I`: +**Using the `HARNESS` environment variable** + +```bash +# Run with specific harness +HARNESS=perf ruby benchmarks/fib.rb +HARNESS=stackprof ruby benchmarks/railsbench/benchmark.rb +HARNESS=vernier ruby benchmarks/optcarrot/benchmark.rb +# Combine with Ruby options +HARNESS=perf ruby --yjit benchmarks/fib.rb +HARNESS=once ruby --yjit-stats benchmarks/railsbench/benchmark.rb ``` -ruby -Iharness benchmarks/railsbench/benchmark.rb + +When using `run_benchmarks.rb`, you can specify a harness with the `--harness` option: + +```bash +./run_benchmarks.rb --harness=once +./run_benchmarks.rb --harness=perf ``` There is also a robust but complex CI harness in [the yjit-metrics repo](https://github.com/Shopify/yjit-metrics). +### Chain harness + +The `chain` harness allows you to combine multiple harnesses together. This is useful when you want to run a benchmark through an analysis tool, such as a profiler, while another harness drives the iterations. + +Use the `HARNESS_CHAIN` environment variable to specify which harnesses to chain (comma-separated, at least 2 required): + +```bash +# Profile the iterations of the default harness with vernier +HARNESS=chain HARNESS_CHAIN="vernier,default" ruby benchmarks/fib.rb + +# Profile a benchmark running under the ractor harness with vernier +HARNESS=chain HARNESS_CHAIN="vernier,ractor" ruby benchmarks/fib.rb +``` + +The harnesses are nested, not run one after another. The first harness is the outermost one: its benchmark block calls the `run_benchmark` of the next harness, and the last harness runs the benchmark itself. Every harness writes its results to the same file, and the outermost harness writes last, so its results are the ones kept. + ### Iterations and duration With the default harness, the number of iterations and duration @@ -360,12 +396,11 @@ You can also use `--warmup`, `--bench`, or `--once` to set these environment var ./run_benchmarks.rb railsbench --once ``` -There is also a handy script for running benchmarks just once using -`WARMUP_ITRS=0 MIN_BENCH_ITRS=1 MIN_BENCH_TIME=0`, for example -with the `--yjit-stats` command-line option: +To run a single benchmark once directly with Ruby, set the same variables as `--once`: -``` -./run_once.sh --yjit-stats benchmarks/railsbench/benchmark.rb +```bash +# Run once with YJIT stats +WARMUP_ITRS=0 MIN_BENCH_ITRS=1 MIN_BENCH_TIME=0 ruby --yjit-stats benchmarks/railsbench/benchmark.rb ``` ### Using perf @@ -373,12 +408,12 @@ with the `--yjit-stats` command-line option: There is also a harness to use Linux perf. By default, it only runs a fixed number of iterations. If `PERF` environment variable is present, it starts the perf subcommand after warmup. -```sh +```bash # Use `perf record` for both warmup and benchmark -perf record ruby --yjit-perf=map -Iharness-perf benchmarks/railsbench/benchmark.rb +HARNESS=perf perf record ruby --yjit-perf=map benchmarks/railsbench/benchmark.rb # Use `perf record` only for benchmark -PERF=record ruby --yjit-perf=map -Iharness-perf benchmarks/railsbench/benchmark.rb +HARNESS=perf PERF=record ruby --yjit-perf=map benchmarks/railsbench/benchmark.rb ``` This is the only harness that uses `run_benchmark`'s argument, `num_itrs_hint`. @@ -390,14 +425,14 @@ Linux that shows its profiles in the Firefox Profiler. Run the benchmark script directly under `samply record`: ```sh -samply record ruby -Iharness benchmarks/railsbench/benchmark.rb +samply record ruby benchmarks/railsbench/benchmark.rb # Name JIT-compiled methods. Both JITs write /tmp/perf-.map, which samply reads. -samply record ruby --yjit --yjit-perf -Iharness benchmarks/railsbench/benchmark.rb -samply record ruby --zjit --zjit-perf -Iharness benchmarks/railsbench/benchmark.rb +samply record ruby --yjit --yjit-perf benchmarks/railsbench/benchmark.rb +samply record ruby --zjit --zjit-perf benchmarks/railsbench/benchmark.rb # Save the profile without opening it, and open it later -samply record --save-only -o railsbench.json.gz ruby -Iharness benchmarks/railsbench/benchmark.rb +samply record --save-only -o railsbench.json.gz ruby benchmarks/railsbench/benchmark.rb samply load railsbench.json.gz ``` @@ -412,7 +447,7 @@ benchmark, also set `RUBY_BENCH_PROFILING=1` to keep the harness's retention GCs out of the profile. The worker Ractors show up as separate thread tracks. ```sh -RUBY_BENCH_RACTORS=4 RUBY_BENCH_PROFILING=1 samply record ruby -Iharness-ractor benchmarks/ractor-dead-set/benchmark.rb +RUBY_BENCH_RACTORS=4 RUBY_BENCH_PROFILING=1 samply record ruby -r./harness/ractor benchmarks/ractor-dead-set/benchmark.rb ``` samply also records child processes, so `samply record ruby run_benchmarks.rb ...` @@ -465,7 +500,7 @@ fail before warmup. ./run_benchmarks.rb --category ractor --chruby=base::ruby-base --ractor-gc # One benchmark, run directly. --ractor-gc sets RUBY_BENCH_RACTOR_GC=1. -RUBY_BENCH_RACTOR_GC=1 ruby -Iharness-ractor benchmarks/ractor-dead-set/benchmark.rb +RUBY_BENCH_RACTOR_GC=1 ruby -r./harness/ractor benchmarks/ractor-dead-set/benchmark.rb ``` `run_benchmarks.rb` only runs Ractor harness benchmarks with `--category ractor` diff --git a/benchmarks.yml b/benchmarks.yml index d11b7570..aa848b9e 100644 --- a/benchmarks.yml +++ b/benchmarks.yml @@ -81,7 +81,7 @@ gcbench: desc: Ellis-Kovac-Boehm GCBench builds binary trees of various depths to exercise GC marking, sweeping, and write barriers. category: gc single_file: true - default_harness: harness-gc + default_harness: gc erubi: desc: erubi compiles a simple Erb template into a method with erubi, then times evaluating that method. ractor: true @@ -105,7 +105,7 @@ knucleotide-ractor: desc: k-nucleotide from the Computer Language Benchmarks Game - counts nucleotide frequencies using hash tables in parallel using Ractors ractor: true ractor_only: true - default_harness: harness + default_harness: default lee: desc: lee is a circuit-board layout solver, deployed in a plausibly reality-like way matmul: @@ -145,7 +145,7 @@ string_malloc_pressure: desc: Allocates and retains many empty Strings with large reserved capacity to exercise GC under malloc-backed String buffer pressure. category: gc single_file: true - default_harness: harness-gc + default_harness: gc tinygql: desc: TinyGQL gem parsing a large file in pure Ruby @@ -285,37 +285,37 @@ gvl_release_acquire: desc: microbenchmark designed to test how fast the gvl can be acquired and released between ractors. ractor: true ractor_only: true - default_harness: harness-ractor + default_harness: ractor json_parse_float: desc: test the performance of parsing multiple lists of json floats with ractors. ractor: true ractor_only: true - default_harness: harness-ractor + default_harness: ractor json_parse_string: desc: test the performance of parsing multiple lists of strings with ractors. ractor: true ractor_only: true - default_harness: harness-ractor + default_harness: ractor ractor-dead-set: desc: multiple ractors each build a large live set and terminate, then memory retention after their death is measured. ractor: true ractor_only: true ractor_scenario: true - default_harness: harness-ractor + default_harness: ractor ractor-idle-garbage: desc: multiple ractors each build and drop a large garbage set, then idle uncollected while retention is measured. ractor: true ractor_only: true ractor_scenario: true - default_harness: harness-ractor + default_harness: ractor ractor-msg-backlog: desc: unshareable message payloads flood the queues of gated consumer ractors, duplicating data per consumer and driving peak RSS and retention. ractor: true ractor_only: true ractor_scenario: true - default_harness: harness-ractor + default_harness: ractor symbol-name-ractor: desc: repeatedly calls Symbol#name on a static symbol under the ractor harness to stress ID-to-string lookup. ractor: true ractor_only: true - default_harness: harness-ractor + default_harness: ractor diff --git a/benchmarks/30k_ifelse.rb b/benchmarks/30k_ifelse.rb index ff9f32c3..a525fda4 100644 --- a/benchmarks/30k_ifelse.rb +++ b/benchmarks/30k_ifelse.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' if ENV["RUBY_BENCH_RACTOR_HARNESS"] eval_recv = Object.new eval_meth = :instance_eval diff --git a/benchmarks/30k_methods.rb b/benchmarks/30k_methods.rb index f3d75f61..4d7bdccd 100644 --- a/benchmarks/30k_methods.rb +++ b/benchmarks/30k_methods.rb @@ -121011,7 +121011,7 @@ def fun_l29_n999() end end -require_relative '../harness/loader' +require_relative '../lib/harness/loader' INTERNAL_ITRS = Integer(ENV.fetch("INTERNAL_ITRS", 200)) diff --git a/benchmarks/30k_variables.rb b/benchmarks/30k_variables.rb index 44e77de0..342171d2 100644 --- a/benchmarks/30k_variables.rb +++ b/benchmarks/30k_variables.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true # Generated by misc/gen_30k_variables.rb: 300 methods x 100 local variables = 30000 local variables -require_relative '../harness/loader' +require_relative '../lib/harness/loader' def fun_l0_n0(x) v0 = x diff --git a/benchmarks/activerecord/benchmark.rb b/benchmarks/activerecord/benchmark.rb index e820d5ae..4f336daa 100644 --- a/benchmarks/activerecord/benchmark.rb +++ b/benchmarks/activerecord/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/addressable/equality.rb b/benchmarks/addressable/equality.rb index ca48dffe..95141423 100644 --- a/benchmarks/addressable/equality.rb +++ b/benchmarks/addressable/equality.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/addressable/getters.rb b/benchmarks/addressable/getters.rb index 301439f2..43d0d03a 100644 --- a/benchmarks/addressable/getters.rb +++ b/benchmarks/addressable/getters.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/addressable/join.rb b/benchmarks/addressable/join.rb index a4df7b39..4d600795 100644 --- a/benchmarks/addressable/join.rb +++ b/benchmarks/addressable/join.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/addressable/merge.rb b/benchmarks/addressable/merge.rb index 39c07f2d..20f33283 100644 --- a/benchmarks/addressable/merge.rb +++ b/benchmarks/addressable/merge.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/addressable/new.rb b/benchmarks/addressable/new.rb index 5fd9731e..ae5d44ae 100644 --- a/benchmarks/addressable/new.rb +++ b/benchmarks/addressable/new.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/addressable/normalize.rb b/benchmarks/addressable/normalize.rb index 121a8ad0..014c81fa 100644 --- a/benchmarks/addressable/normalize.rb +++ b/benchmarks/addressable/normalize.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/addressable/parse.rb b/benchmarks/addressable/parse.rb index 02d29a83..d95f1094 100644 --- a/benchmarks/addressable/parse.rb +++ b/benchmarks/addressable/parse.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/addressable/setters.rb b/benchmarks/addressable/setters.rb index 41ee88b3..3431bba3 100644 --- a/benchmarks/addressable/setters.rb +++ b/benchmarks/addressable/setters.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/addressable/to-s.rb b/benchmarks/addressable/to-s.rb index 355d931f..9f337ff7 100644 --- a/benchmarks/addressable/to-s.rb +++ b/benchmarks/addressable/to-s.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/attr_accessor.rb b/benchmarks/attr_accessor.rb index d5e397dc..1ab4ae7f 100644 --- a/benchmarks/attr_accessor.rb +++ b/benchmarks/attr_accessor.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class TheClass attr_accessor :levar diff --git a/benchmarks/binarytrees/benchmark.rb b/benchmarks/binarytrees/benchmark.rb index 4e54528d..ca460c07 100644 --- a/benchmarks/binarytrees/benchmark.rb +++ b/benchmarks/binarytrees/benchmark.rb @@ -22,7 +22,7 @@ def bottom_up_tree(depth) MAX_DEPTH = MIN_DEPTH + 2 if MIN_DEPTH + 2 > MAX_DEPTH STRETCH_DEPTH = MAX_DEPTH + 1 -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' run_benchmark(60) do max_depth = MAX_DEPTH diff --git a/benchmarks/block_methods.rb b/benchmarks/block_methods.rb index 3db42080..8fc910c7 100644 --- a/benchmarks/block_methods.rb +++ b/benchmarks/block_methods.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' def array_each(array) array.each do |x| diff --git a/benchmarks/blurhash/benchmark.rb b/benchmarks/blurhash/benchmark.rb index 1f081f4b..327fc225 100644 --- a/benchmarks/blurhash/benchmark.rb +++ b/benchmarks/blurhash/benchmark.rb @@ -172,7 +172,7 @@ def blurHashForPixels(xComponents, yComponents, width, height, rgb, bytesPerRow) # :startdoc: end -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" FILE = File.join(__dir__, "test.bin") diff --git a/benchmarks/cfunc_itself.rb b/benchmarks/cfunc_itself.rb index 2c39137e..8560da66 100644 --- a/benchmarks/cfunc_itself.rb +++ b/benchmarks/cfunc_itself.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' run_benchmark(500) do # 500K calls diff --git a/benchmarks/chunky-png/benchmark.rb b/benchmarks/chunky-png/benchmark.rb index ef511893..29054d29 100644 --- a/benchmarks/chunky-png/benchmark.rb +++ b/benchmarks/chunky-png/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/erubi-rails/benchmark.rb b/benchmarks/erubi-rails/benchmark.rb index 38c22634..1817a0f8 100644 --- a/benchmarks/erubi-rails/benchmark.rb +++ b/benchmarks/erubi-rails/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/erubi/benchmark.rb b/benchmarks/erubi/benchmark.rb index 0a75f1bb..f62a2f3d 100644 --- a/benchmarks/erubi/benchmark.rb +++ b/benchmarks/erubi/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/etanni/benchmark.rb b/benchmarks/etanni/benchmark.rb index 77e86386..48c41f1a 100644 --- a/benchmarks/etanni/benchmark.rb +++ b/benchmarks/etanni/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir __dir__ # This is an Etanni translation of the Erb template in the Erubi diff --git a/benchmarks/fannkuchredux/benchmark.rb b/benchmarks/fannkuchredux/benchmark.rb index e73dc0ee..2e44c7d1 100644 --- a/benchmarks/fannkuchredux/benchmark.rb +++ b/benchmarks/fannkuchredux/benchmark.rb @@ -56,7 +56,7 @@ def fannkuch(n) #n = (ARGV[0] || 1).to_i N = 9 # Benchmarks Game uses n = 12, but it's too slow -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' run_benchmark(10) do sum, flips = fannkuch(N) diff --git a/benchmarks/fib.rb b/benchmarks/fib.rb index 4e5b3428..02be31b8 100644 --- a/benchmarks/fib.rb +++ b/benchmarks/fib.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' def fib(n) if n < 2 diff --git a/benchmarks/fluentd/benchmark.rb b/benchmarks/fluentd/benchmark.rb index c503a653..4e3f9a6d 100644 --- a/benchmarks/fluentd/benchmark.rb +++ b/benchmarks/fluentd/benchmark.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir(__dir__) use_gemfile diff --git a/benchmarks/gcbench.rb b/benchmarks/gcbench.rb index be6399fc..582551a3 100644 --- a/benchmarks/gcbench.rb +++ b/benchmarks/gcbench.rb @@ -11,7 +11,7 @@ # Tree construction uses both top-down (populate — creates old-to-young pointers, # exercises write barriers) and bottom-up (make_tree — young-to-young only). -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class GCBench class Node diff --git a/benchmarks/getivar-module.rb b/benchmarks/getivar-module.rb index e99930a2..ee6e1d06 100644 --- a/benchmarks/getivar-module.rb +++ b/benchmarks/getivar-module.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class TheClass @v0 = 1 diff --git a/benchmarks/getivar.rb b/benchmarks/getivar.rb index 80acfae1..d77cc33d 100644 --- a/benchmarks/getivar.rb +++ b/benchmarks/getivar.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class TheClass def initialize diff --git a/benchmarks/grape/benchmark.rb b/benchmarks/grape/benchmark.rb index 17b513ae..c139b41b 100644 --- a/benchmarks/grape/benchmark.rb +++ b/benchmarks/grape/benchmark.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir(__dir__) use_gemfile diff --git a/benchmarks/graphql-native/benchmark.rb b/benchmarks/graphql-native/benchmark.rb index d4fd7077..ff5858c9 100644 --- a/benchmarks/graphql-native/benchmark.rb +++ b/benchmarks/graphql-native/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/graphql/benchmark.rb b/benchmarks/graphql/benchmark.rb index 85b5da1f..26c1feb6 100644 --- a/benchmarks/graphql/benchmark.rb +++ b/benchmarks/graphql/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/gvl_release_acquire/benchmark.rb b/benchmarks/gvl_release_acquire/benchmark.rb index e735bfe2..b1b22863 100644 --- a/benchmarks/gvl_release_acquire/benchmark.rb +++ b/benchmarks/gvl_release_acquire/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" # Windows has no /dev/zero, so read the same zeros from a regular file kept in data/. ZERO_PATH = if File.exist?("/dev/zero") diff --git a/benchmarks/hexapdf/benchmark.rb b/benchmarks/hexapdf/benchmark.rb index bcd2cf90..147eb50f 100644 --- a/benchmarks/hexapdf/benchmark.rb +++ b/benchmarks/hexapdf/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/json_parse_float/benchmark.rb b/benchmarks/json_parse_float/benchmark.rb index ef81bc49..2aef033f 100644 --- a/benchmarks/json_parse_float/benchmark.rb +++ b/benchmarks/json_parse_float/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir(__dir__) use_gemfile diff --git a/benchmarks/json_parse_string/benchmark.rb b/benchmarks/json_parse_string/benchmark.rb index a579f805..71417cd3 100644 --- a/benchmarks/json_parse_string/benchmark.rb +++ b/benchmarks/json_parse_string/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir(__dir__) use_gemfile diff --git a/benchmarks/keyword_args.rb b/benchmarks/keyword_args.rb index a9b035f3..386ef894 100644 --- a/benchmarks/keyword_args.rb +++ b/benchmarks/keyword_args.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' def add(left:, right:) left + right diff --git a/benchmarks/knucleotide-ractor/benchmark.rb b/benchmarks/knucleotide-ractor/benchmark.rb index 41d265d6..779ae3c6 100644 --- a/benchmarks/knucleotide-ractor/benchmark.rb +++ b/benchmarks/knucleotide-ractor/benchmark.rb @@ -6,7 +6,7 @@ Warning[:experimental] = false -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' def frequency(seq, length) frequencies = Hash.new(0) diff --git a/benchmarks/knucleotide/benchmark.rb b/benchmarks/knucleotide/benchmark.rb index 94bbb028..4ea79b37 100644 --- a/benchmarks/knucleotide/benchmark.rb +++ b/benchmarks/knucleotide/benchmark.rb @@ -4,7 +4,7 @@ # k-nucleotide benchmark - Fastest implementation # Based on Ruby #1 with byteslice optimization -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' def frequency(seq, length) frequencies = Hash.new(0) diff --git a/benchmarks/lee/benchmark.rb b/benchmarks/lee/benchmark.rb index 5d0475fc..5e6d3c84 100644 --- a/benchmarks/lee/benchmark.rb +++ b/benchmarks/lee/benchmark.rb @@ -84,7 +84,7 @@ def lay(depth, solution) end end -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/liquid-c/benchmark.rb b/benchmarks/liquid-c/benchmark.rb index e074e33c..4a649dff 100644 --- a/benchmarks/liquid-c/benchmark.rb +++ b/benchmarks/liquid-c/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/liquid-compile/benchmark.rb b/benchmarks/liquid-compile/benchmark.rb index 0e0d0f1c..956e0fb8 100644 --- a/benchmarks/liquid-compile/benchmark.rb +++ b/benchmarks/liquid-compile/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/liquid-il/benchmark.rb b/benchmarks/liquid-il/benchmark.rb index 144894ca..99dac298 100644 --- a/benchmarks/liquid-il/benchmark.rb +++ b/benchmarks/liquid-il/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/liquid-render/benchmark.rb b/benchmarks/liquid-render/benchmark.rb index 661015a8..b851474c 100644 --- a/benchmarks/liquid-render/benchmark.rb +++ b/benchmarks/liquid-render/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/lobsters/benchmark.rb b/benchmarks/lobsters/benchmark.rb index 3a28fe81..63c36100 100644 --- a/benchmarks/lobsters/benchmark.rb +++ b/benchmarks/lobsters/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' ENV['RAILS_ENV'] ||= 'production' ENV['DISABLE_DATABASE_ENVIRONMENT_CHECK'] = '1' # Benchmarks don't really have 'production', so trash it at will. diff --git a/benchmarks/loops-times.rb b/benchmarks/loops-times.rb index 653fe989..22c196bd 100644 --- a/benchmarks/loops-times.rb +++ b/benchmarks/loops-times.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' # Fix these values for determinism U = 5 diff --git a/benchmarks/mail/benchmark.rb b/benchmarks/mail/benchmark.rb index 0d625650..9bef261d 100644 --- a/benchmarks/mail/benchmark.rb +++ b/benchmarks/mail/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/matmul.rb b/benchmarks/matmul.rb index c4087dbb..80187ade 100644 --- a/benchmarks/matmul.rb +++ b/benchmarks/matmul.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' def matgen(n) tmp = 1.0 / n / n diff --git a/benchmarks/nbody/benchmark.rb b/benchmarks/nbody/benchmark.rb index 737e298b..86a11a3c 100644 --- a/benchmarks/nbody/benchmark.rb +++ b/benchmarks/nbody/benchmark.rb @@ -5,7 +5,7 @@ # modified by Jesse Millikan # modified by Yusuke Endoh -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' SOLAR_MASS = 4 * Math::PI**2 DAYS_PER_YEAR = 365.24 diff --git a/benchmarks/nqueens.rb b/benchmarks/nqueens.rb index 9a1b7a8c..0912a64a 100644 --- a/benchmarks/nqueens.rb +++ b/benchmarks/nqueens.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' def nq_solve(n) a = Array.new(n, -1) diff --git a/benchmarks/object-new-initialize.rb b/benchmarks/object-new-initialize.rb index bb90dbf1..b51881a7 100644 --- a/benchmarks/object-new-initialize.rb +++ b/benchmarks/object-new-initialize.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class C def initialize(a, b, c, d) diff --git a/benchmarks/object-new-no-escape.rb b/benchmarks/object-new-no-escape.rb index 5c38e03d..069784df 100644 --- a/benchmarks/object-new-no-escape.rb +++ b/benchmarks/object-new-no-escape.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class Point attr_reader :x, :y diff --git a/benchmarks/object-new.rb b/benchmarks/object-new.rb index b079c393..ed3b37b8 100644 --- a/benchmarks/object-new.rb +++ b/benchmarks/object-new.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' run_benchmark(100) do i = 0 diff --git a/benchmarks/optcarrot/benchmark.rb b/benchmarks/optcarrot/benchmark.rb index 3c48515f..29e6e58f 100755 --- a/benchmarks/optcarrot/benchmark.rb +++ b/benchmarks/optcarrot/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' require_relative "lib/optcarrot" if ENV["RUBY_BENCH_RACTOR_HARNESS"] diff --git a/benchmarks/protoboeuf-encode/benchmark.rb b/benchmarks/protoboeuf-encode/benchmark.rb index a7afcdb7..16551acc 100644 --- a/benchmarks/protoboeuf-encode/benchmark.rb +++ b/benchmarks/protoboeuf-encode/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' # Protoboeuf decoder require_relative 'benchmark_pb' diff --git a/benchmarks/protoboeuf/benchmark.rb b/benchmarks/protoboeuf/benchmark.rb index 6e3078f4..9f477b79 100644 --- a/benchmarks/protoboeuf/benchmark.rb +++ b/benchmarks/protoboeuf/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' # Protoboeuf decoder require_relative 'benchmark_pb' diff --git a/benchmarks/psych-load/benchmark.rb b/benchmarks/psych-load/benchmark.rb index 795135f3..08b99c64 100644 --- a/benchmarks/psych-load/benchmark.rb +++ b/benchmarks/psych-load/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' Dir.chdir __dir__ use_gemfile diff --git a/benchmarks/rack/benchmark.rb b/benchmarks/rack/benchmark.rb index a3fde656..57c25052 100644 --- a/benchmarks/rack/benchmark.rb +++ b/benchmarks/rack/benchmark.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir(__dir__) use_gemfile diff --git a/benchmarks/ractor-dead-set/benchmark.rb b/benchmarks/ractor-dead-set/benchmark.rb index 4f48e50e..9a693b97 100644 --- a/benchmarks/ractor-dead-set/benchmark.rb +++ b/benchmarks/ractor-dead-set/benchmark.rb @@ -4,7 +4,7 @@ Warning[:experimental] = false -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" DEAD_SET_ITEMS = Integer(ENV.fetch("RACTOR_DEAD_SET_ITEMS", 100_000)) diff --git a/benchmarks/ractor-idle-garbage/benchmark.rb b/benchmarks/ractor-idle-garbage/benchmark.rb index 7c69366f..87b66d01 100644 --- a/benchmarks/ractor-idle-garbage/benchmark.rb +++ b/benchmarks/ractor-idle-garbage/benchmark.rb @@ -6,7 +6,7 @@ Warning[:experimental] = false -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" IDLE_GARBAGE_ITEMS = Integer(ENV.fetch("RACTOR_IDLE_GARBAGE_ITEMS", 100_000)) diff --git a/benchmarks/ractor-msg-backlog/benchmark.rb b/benchmarks/ractor-msg-backlog/benchmark.rb index 1648bbdc..fdca9adb 100644 --- a/benchmarks/ractor-msg-backlog/benchmark.rb +++ b/benchmarks/ractor-msg-backlog/benchmark.rb @@ -6,7 +6,7 @@ Warning[:experimental] = false -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" BACKLOG_MESSAGES = Integer(ENV.fetch("RACTOR_BACKLOG_MESSAGES", 20_000)) BACKLOG_GATE_SLEEP = Float(ENV.fetch("RACTOR_BACKLOG_GATE_SLEEP", 1.0)) diff --git a/benchmarks/railsbench/benchmark.rb b/benchmarks/railsbench/benchmark.rb index f89f91a3..965c70e3 100644 --- a/benchmarks/railsbench/benchmark.rb +++ b/benchmarks/railsbench/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' ENV['RAILS_ENV'] ||= 'production' diff --git a/benchmarks/respond_to.rb b/benchmarks/respond_to.rb index 9a419060..55f49f70 100644 --- a/benchmarks/respond_to.rb +++ b/benchmarks/respond_to.rb @@ -1,7 +1,7 @@ # Microbenchmark to test the performance of respond_to? # This is one of the top most called methods in rack/railsbench -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class A def foo diff --git a/benchmarks/rubocop/benchmark.rb b/benchmarks/rubocop/benchmark.rb index 60b49d05..12f415a0 100644 --- a/benchmarks/rubocop/benchmark.rb +++ b/benchmarks/rubocop/benchmark.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir(__dir__) use_gemfile diff --git a/benchmarks/ruby-json/benchmark.rb b/benchmarks/ruby-json/benchmark.rb index 3c2d3e4e..f8c085b8 100644 --- a/benchmarks/ruby-json/benchmark.rb +++ b/benchmarks/ruby-json/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" require "json" require "strscan" diff --git a/benchmarks/ruby-lsp/benchmark.rb b/benchmarks/ruby-lsp/benchmark.rb index 232a0e8d..0bbc304b 100644 --- a/benchmarks/ruby-lsp/benchmark.rb +++ b/benchmarks/ruby-lsp/benchmark.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir(__dir__) use_gemfile diff --git a/benchmarks/ruby-xor.rb b/benchmarks/ruby-xor.rb index 5352c76a..2a9f9d1b 100644 --- a/benchmarks/ruby-xor.rb +++ b/benchmarks/ruby-xor.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' # XOR strings in place # Expected to behave like Xorcist.xor! from the xorcist gem diff --git a/benchmarks/rubyboy/benchmark.rb b/benchmarks/rubyboy/benchmark.rb index 5a086fc2..18afcbd9 100644 --- a/benchmarks/rubyboy/benchmark.rb +++ b/benchmarks/rubyboy/benchmark.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir(__dir__) use_gemfile diff --git a/benchmarks/rubykon/benchmark.rb b/benchmarks/rubykon/benchmark.rb index e8f0e527..d312daaf 100644 --- a/benchmarks/rubykon/benchmark.rb +++ b/benchmarks/rubykon/benchmark.rb @@ -2,7 +2,7 @@ # https://github.com/benchmark-driver/sky2-bench/blob/master/benchmark/rubykon-benchmark.rb, # part of benchmark-driver's default benchmarking suite, by Takashi Kokubun. -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' require_relative 'lib/rubykon' diff --git a/benchmarks/send_bmethod.rb b/benchmarks/send_bmethod.rb index 649df891..07774ea0 100644 --- a/benchmarks/send_bmethod.rb +++ b/benchmarks/send_bmethod.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' # Call with and without args define_method(:zero) { :b } diff --git a/benchmarks/send_cfunc_block.rb b/benchmarks/send_cfunc_block.rb index 2c7dc34f..1238b041 100644 --- a/benchmarks/send_cfunc_block.rb +++ b/benchmarks/send_cfunc_block.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' ARR = [] diff --git a/benchmarks/send_rubyfunc_block.rb b/benchmarks/send_rubyfunc_block.rb index ae3f3734..95db2e01 100644 --- a/benchmarks/send_rubyfunc_block.rb +++ b/benchmarks/send_rubyfunc_block.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class C CALL_BLOCK = false diff --git a/benchmarks/send_rubyfunc_inline.rb b/benchmarks/send_rubyfunc_inline.rb index 50f0efd8..230ba169 100644 --- a/benchmarks/send_rubyfunc_inline.rb +++ b/benchmarks/send_rubyfunc_inline.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' INNER_ITERATIONS = 10_000_000 EXPECTED_RESULT = 2 * INNER_ITERATIONS * INNER_ITERATIONS + 4 * INNER_ITERATIONS diff --git a/benchmarks/sequel/benchmark.rb b/benchmarks/sequel/benchmark.rb index 81b07c67..7c2491b4 100644 --- a/benchmarks/sequel/benchmark.rb +++ b/benchmarks/sequel/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" require "securerandom" # Provides `Random::Formatter` in Ruby 2.6+ Dir.chdir __dir__ diff --git a/benchmarks/setivar.rb b/benchmarks/setivar.rb index e64dca4d..9e9ebf3a 100644 --- a/benchmarks/setivar.rb +++ b/benchmarks/setivar.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class TheClass def initialize diff --git a/benchmarks/setivar_object.rb b/benchmarks/setivar_object.rb index 6f34cba6..0c316833 100644 --- a/benchmarks/setivar_object.rb +++ b/benchmarks/setivar_object.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class TheClass def initialize diff --git a/benchmarks/setivar_young.rb b/benchmarks/setivar_young.rb index 2f673756..d463d811 100644 --- a/benchmarks/setivar_young.rb +++ b/benchmarks/setivar_young.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class TheClass def initialize diff --git a/benchmarks/shipit/benchmark.rb b/benchmarks/shipit/benchmark.rb index d7b2ce7d..afe83a71 100644 --- a/benchmarks/shipit/benchmark.rb +++ b/benchmarks/shipit/benchmark.rb @@ -1,4 +1,4 @@ -require_relative '../../harness/loader' +require_relative '../../lib/harness/loader' ENV['RAILS_ENV'] ||= 'production' ENV['DISABLE_DATABASE_ENVIRONMENT_CHECK'] = '1' # Benchmarks don't really have 'production', so trash it at will. diff --git a/benchmarks/splay.rb b/benchmarks/splay.rb index e38e0f84..f9903551 100644 --- a/benchmarks/splay.rb +++ b/benchmarks/splay.rb @@ -29,7 +29,7 @@ # Ported to Ruby from the V8/WebKit JavaScript benchmark suite. # https://browserbench.org/JetStream2.1/Octane/splay.js -require_relative '../harness/loader' +require_relative '../lib/harness/loader' class SplayTree class Node diff --git a/benchmarks/str_concat.rb b/benchmarks/str_concat.rb index ee514cc4..9bbeaf2b 100644 --- a/benchmarks/str_concat.rb +++ b/benchmarks/str_concat.rb @@ -1,5 +1,5 @@ # frozen_string_literal: true -require_relative '../harness/loader' +require_relative '../lib/harness/loader' NUM_ITERS = 10 * 1024 TEST_STR = 'sssssséé'.freeze diff --git a/benchmarks/string_malloc_pressure.rb b/benchmarks/string_malloc_pressure.rb index ebda6860..39026d22 100644 --- a/benchmarks/string_malloc_pressure.rb +++ b/benchmarks/string_malloc_pressure.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' def build_string_malloc_pressure live = [] diff --git a/benchmarks/struct-new-no-escape.rb b/benchmarks/struct-new-no-escape.rb index 1a45269f..18751bde 100644 --- a/benchmarks/struct-new-no-escape.rb +++ b/benchmarks/struct-new-no-escape.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' Point = Struct.new(:x, :y) diff --git a/benchmarks/structaref.rb b/benchmarks/structaref.rb index 95bb3575..b319092e 100644 --- a/benchmarks/structaref.rb +++ b/benchmarks/structaref.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' TheClass = Struct.new(:v0, :v1, :v2, :levar) diff --git a/benchmarks/structaset.rb b/benchmarks/structaset.rb index 5d73035e..bcff9b0e 100644 --- a/benchmarks/structaset.rb +++ b/benchmarks/structaset.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' TheClass = Struct.new(:v0, :v1, :v2, :levar) diff --git a/benchmarks/sudoku.rb b/benchmarks/sudoku.rb index 62308b36..e8a298e0 100644 --- a/benchmarks/sudoku.rb +++ b/benchmarks/sudoku.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' def sd_genmat mr = Array.new(324) { [] } diff --git a/benchmarks/symbol-name-ractor/benchmark.rb b/benchmarks/symbol-name-ractor/benchmark.rb index 3d60356f..367e71e2 100644 --- a/benchmarks/symbol-name-ractor/benchmark.rb +++ b/benchmarks/symbol-name-ractor/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" SYMBOL = :ruby_bench_id2str_pressure_static_symbol EXPECTED_NAME = SYMBOL.name diff --git a/benchmarks/throw.rb b/benchmarks/throw.rb index 3d3a7b2b..e3aa8b9c 100644 --- a/benchmarks/throw.rb +++ b/benchmarks/throw.rb @@ -1,4 +1,4 @@ -require_relative '../harness/loader' +require_relative '../lib/harness/loader' def foo yield diff --git a/benchmarks/tinygql/benchmark.rb b/benchmarks/tinygql/benchmark.rb index 7195a9ed..8457772d 100644 --- a/benchmarks/tinygql/benchmark.rb +++ b/benchmarks/tinygql/benchmark.rb @@ -1,4 +1,4 @@ -require_relative "../../harness/loader" +require_relative "../../lib/harness/loader" Dir.chdir __dir__ use_gemfile diff --git a/burn_in.rb b/burn_in.rb index 91b1ada1..1d4a63b2 100755 --- a/burn_in.rb +++ b/burn_in.rb @@ -62,7 +62,7 @@ def free_file_path(parent_dir, name_prefix) def run_benchmark(bench_name, no_yjit, logs_path, run_time, ruby_version, metadata) bench_dir = 'benchmarks' entry = metadata[bench_name] || {} - harness = entry.fetch('default_harness', 'harness') + harness = entry.fetch('default_harness', 'default') script_path = File.join(bench_dir, bench_name, 'benchmark.rb') if not File.exist?(script_path) @@ -104,7 +104,7 @@ def run_benchmark(bench_name, no_yjit, logs_path, run_time, ruby_version, metada cmd = [ 'ruby', *test_options, - "-I#{harness}", + "-r#{File.expand_path("harness/#{harness}", __dir__)}", script_path, ].compact cmd_str = cmd.shelljoin @@ -225,7 +225,7 @@ def test_loop(bench_names, no_yjit, logs_path, run_time, ruby_version, metadata) metadata.each do |name, entry| category = entry.fetch('category', 'other') is_ractor_only = entry['ractor_only'] || - (entry['ractor'] && entry['default_harness'] == 'harness-ractor') + (entry['ractor'] && entry['default_harness'] == 'ractor') if args.categories.include?(category) && !is_ractor_only bench_names << name end diff --git a/harness-bips/harness.rb b/harness/bips.rb similarity index 79% rename from harness-bips/harness.rb rename to harness/bips.rb index d1319a19..dddb546e 100644 --- a/harness-bips/harness.rb +++ b/harness/bips.rb @@ -1,5 +1,5 @@ require 'benchmark/ips' -require_relative "../harness/harness-common" +require_relative "../lib/harness" puts RUBY_DESCRIPTION diff --git a/harness-chain/harness.rb b/harness/chain.rb similarity index 82% rename from harness-chain/harness.rb rename to harness/chain.rb index 48e6b0f9..20e1fa66 100644 --- a/harness-chain/harness.rb +++ b/harness/chain.rb @@ -1,4 +1,4 @@ -require_relative '../harness/harness-common' +require_relative '../lib/harness' # Ex: HARNESS_CHAIN="vernier,ractor" # Wraps the ractor harness in ther vernier harness @@ -10,7 +10,7 @@ end if CHAIN.include?("vernier") && CHAIN.last != "vernier" - require_relative "../harness/harness-extra" + require_relative "../lib/harness/extra" def run_enough_to_profile(n, **kwargs, &block) block.call end @@ -30,12 +30,13 @@ def self.method_added(name) def run_benchmark(n, **kwargs, &block) CHAIN.each do |h| begin - path = "../harness-#{h}/harness" + path = "#{h}" $current_harness = h require_relative path rescue LoadError => e - if e.path == path - $stderr.puts "Can't find harness harness-#{h}/harness.rb. Exiting." + expected_path = File.expand_path(h, __dir__) + if e.path == expected_path + $stderr.puts "Can't find harness #{h}.rb in harness/. Exiting." exit 1 end raise diff --git a/harness-continuous/harness.rb b/harness/continuous.rb similarity index 94% rename from harness-continuous/harness.rb rename to harness/continuous.rb index 6606b182..05a9162f 100644 --- a/harness-continuous/harness.rb +++ b/harness/continuous.rb @@ -1,4 +1,4 @@ -require_relative "../harness/harness-common" +require_relative "../lib/harness" puts RUBY_DESCRIPTION diff --git a/harness/harness.rb b/harness/default.rb similarity index 98% rename from harness/harness.rb rename to harness/default.rb index f6bd6099..1d4aae93 100644 --- a/harness/harness.rb +++ b/harness/default.rb @@ -1,4 +1,4 @@ -require_relative "./harness-common" +require_relative "../lib/harness" # Warmup iterations WARMUP_ITRS = Integer(ENV.fetch('WARMUP_ITRS', 15)) diff --git a/harness-gc/harness.rb b/harness/gc.rb similarity index 99% rename from harness-gc/harness.rb rename to harness/gc.rb index ab627e1f..4c93fc69 100644 --- a/harness-gc/harness.rb +++ b/harness/gc.rb @@ -1,4 +1,4 @@ -require_relative "../harness/harness-common" +require_relative "../lib/harness" require_relative "../lib/gc_stats" WARMUP_ITRS = Integer(ENV.fetch('WARMUP_ITRS', 15)) diff --git a/harness/loader.rb b/harness/loader.rb deleted file mode 100644 index 5feb027e..00000000 --- a/harness/loader.rb +++ /dev/null @@ -1,17 +0,0 @@ -# Use harness/harness.rb by default. You can change it with -I option. -# i.e. ruby -Iharness benchmarks/railsbench/benchmark.rb -if RUBY_ENGINE == "spinel" - require_relative "../harness-spinel/harness" -else - retries = 0 - begin - require "harness" - rescue LoadError => e - if retries == 0 && e.path == "harness" - retries += 1 - $LOAD_PATH << File.expand_path(__dir__) - retry - end - raise - end -end diff --git a/harness-mplr/harness.rb b/harness/mplr.rb similarity index 93% rename from harness-mplr/harness.rb rename to harness/mplr.rb index 73234934..f91bfd94 100644 --- a/harness-mplr/harness.rb +++ b/harness/mplr.rb @@ -1,5 +1,5 @@ require 'benchmark' -require_relative "../harness/harness-common" +require_relative "../lib/harness" # Minimum number of benchmarking iterations MAX_BENCH_ITRS = Integer(ENV.fetch('MAX_BENCH_ITRS', 1000)) diff --git a/harness-once/harness.rb b/harness/once.rb similarity index 88% rename from harness-once/harness.rb rename to harness/once.rb index c03d9db5..29b3340a 100644 --- a/harness-once/harness.rb +++ b/harness/once.rb @@ -5,7 +5,7 @@ # Intended only for checking whether the benchmark can set itself up properly # and can run to completion. -require_relative '../harness/harness-common' +require_relative '../lib/harness' def run_benchmark(_hint, **) yield diff --git a/harness-perf/harness.rb b/harness/perf.rb similarity index 94% rename from harness-perf/harness.rb rename to harness/perf.rb index 4991cb54..e13978ea 100644 --- a/harness-perf/harness.rb +++ b/harness/perf.rb @@ -3,14 +3,14 @@ # This is a relatively minimal harness meant for use with Linux perf(1). # Example usage: # -# $ PERF='record -e cycles' ruby -Iharness-perf benchmarks/fib.rb +# $ PERF='record -e cycles' HARNESS=perf ruby benchmarks/fib.rb # # When recording with perf(1), make sure the benchmark runs long enough; you # can tweak the MIN_BENCH_ITRS environment variable to lengthen the run. A race # condition is possible where the benchmark finishes before the perf(1) # subprocess has a chance to attach, in which case perf outputs no profile. -require_relative "../harness/harness-common" +require_relative "../lib/harness" # Run $WARMUP_ITRS or 10 iterations of a given block. Then run $MIN_BENCH_ITRS # or `num_itrs_int` iterations of the block, attaching a perf command to the diff --git a/harness-ractor/harness.rb b/harness/ractor.rb similarity index 99% rename from harness-ractor/harness.rb rename to harness/ractor.rb index ee5bdfa0..a74bc6a4 100644 --- a/harness-ractor/harness.rb +++ b/harness/ractor.rb @@ -1,5 +1,5 @@ # frozen_string_literal: true -require_relative '../harness/harness-common' +require_relative '../lib/harness' require 'etc' Warning[:experimental] = false diff --git a/harness-spinel/harness.rb b/harness/spinel.rb similarity index 95% rename from harness-spinel/harness.rb rename to harness/spinel.rb index 969524fe..3abb72c5 100644 --- a/harness-spinel/harness.rb +++ b/harness/spinel.rb @@ -1,5 +1,5 @@ # A harness for Ruby implementations that compile the whole program ahead of -# time, such as Spinel (https://github.com/matz/spinel). harness/harness.rb +# time, such as Spinel (https://github.com/matz/spinel). harness/default.rb # relies on CRuby-only libraries and reflection (RubyVM, RbConfig, Fiddle, # Bundler, $LOAD_PATH) that a whole-program compiler refuses to compile, so # this file reimplements the same protocol in a portable subset of Ruby: @@ -7,9 +7,9 @@ # * WARMUP_ITRS, MIN_BENCH_ITRS and MIN_BENCH_TIME select the iteration count. # * RESULT_JSON_PATH receives the JSON that run_benchmarks.rb reads. # -# harness/loader.rb selects this harness automatically when RUBY_ENGINE is +# lib/harness/loader.rb selects this harness automatically when RUBY_ENGINE is # "spinel", so `spinel -E benchmarks/fib.rb` works without any -I option. It -# also runs on CRuby (`ruby -Iharness-spinel benchmarks/fib.rb`), which is how +# also runs on CRuby (`ruby -r./harness/spinel benchmarks/fib.rb`), which is how # its output is kept compatible with the default harness. require "json" diff --git a/harness-stackprof/harness.rb b/harness/stackprof.rb similarity index 86% rename from harness-stackprof/harness.rb rename to harness/stackprof.rb index 6a63d1c5..998c3c26 100644 --- a/harness-stackprof/harness.rb +++ b/harness/stackprof.rb @@ -3,11 +3,11 @@ # Profile the benchmark (ignoring initialization code) with stackprof. # Customize stackprof options with an env var of STACKPROF_OPTS='key:value,...'. # Usage: -# STACKPROF_OPTS='mode:object' MIN_BENCH_TIME=0 MIN_BENCH_ITRS=1 ruby -v -I harness-stackprof benchmarks/.../benchmark.rb -# STACKPROF_OPTS='mode:cpu,interval:10' MIN_BENCH_TIME=1 MIN_BENCH_ITRS=10 ruby -v -I harness-stackprof benchmarks/.../benchmark.rb +# STACKPROF_OPTS='mode:object' HARNESS=stackprof MIN_BENCH_TIME=0 MIN_BENCH_ITRS=1 ruby -v benchmarks/.../benchmark.rb +# STACKPROF_OPTS='mode:cpu,interval:10' HARNESS=stackprof MIN_BENCH_TIME=1 MIN_BENCH_ITRS=10 ruby -v benchmarks/.../benchmark.rb -require_relative "../harness/harness-common" -require_relative "../harness/harness-extra" +require_relative "../lib/harness" +require_relative "../lib/harness/extra" ensure_global_gem("stackprof") diff --git a/harness-stats/harness.rb b/harness/stats.rb similarity index 98% rename from harness-stats/harness.rb rename to harness/stats.rb index a0b9a2a3..2971c667 100644 --- a/harness-stats/harness.rb +++ b/harness/stats.rb @@ -1,4 +1,4 @@ -require_relative '../harness/harness' +require_relative 'default' # Using Module#prepend to enable TracePoint right before #run_benchmark # while also reusing the original implementation. diff --git a/harness-vernier/harness.rb b/harness/vernier.rb similarity index 62% rename from harness-vernier/harness.rb rename to harness/vernier.rb index fe32ff1c..fe7e81d1 100644 --- a/harness-vernier/harness.rb +++ b/harness/vernier.rb @@ -1,13 +1,13 @@ # frozen_string_literal: true # Profile the benchmark (ignoring initialization code) using vernier and display the profile. -# Set NO_VIERWER=1 to disable automatically opening the profile in a browser. +# Set NO_VIEWER=1 to disable automatically opening the profile in a browser. # Usage: -# MIN_BENCH_TIME=1 MIN_BENCH_ITRS=1 ruby -v -I harness-vernier benchmarks/... -# NO_VIEWER=1 MIN_BENCH_TIME=1 MIN_BENCH_ITRS=1 ruby -v -I harness-vernier benchmarks/... +# HARNESS=vernier MIN_BENCH_TIME=1 MIN_BENCH_ITRS=1 ruby -v benchmarks/... +# NO_VIEWER=1 HARNESS=vernier MIN_BENCH_TIME=1 MIN_BENCH_ITRS=1 ruby -v benchmarks/... -require_relative "../harness/harness-common" -require_relative "../harness/harness-extra" +require_relative "../lib/harness" +require_relative "../lib/harness/extra" ensure_global_gem("vernier") ensure_global_gem_exe("profile-viewer") diff --git a/harness-warmup/harness.rb b/harness/warmup.rb similarity index 98% rename from harness-warmup/harness.rb rename to harness/warmup.rb index 5f1050cc..84321234 100644 --- a/harness-warmup/harness.rb +++ b/harness/warmup.rb @@ -1,5 +1,5 @@ require 'benchmark' -require_relative '../harness/harness-common' +require_relative '../lib/harness' require_relative '../misc/stats' MIN_ITERS = Integer(ENV['MIN_ITERS'] || 10) diff --git a/lib/argument_parser.rb b/lib/argument_parser.rb index 556dc648..18b45dc9 100644 --- a/lib/argument_parser.rb +++ b/lib/argument_parser.rb @@ -77,7 +77,7 @@ def parse(argv) opts.on("--category=headline,other,micro,ractor", "when given, only benchmarks with specified categories will run") do |v| args.categories += v.split(",") if args.categories == ["ractor"] - args.harness = "harness-ractor" + args.harness = "ractor" end end @@ -101,8 +101,9 @@ def parse(argv) args.skip_zjit = true end - opts.on("--harness=HARNESS_DIR", "which harness to use") do |v| - v = "harness-#{v}" unless v.start_with?('harness') + opts.on("--harness=HARNESS_NAME", "which harness to use (default: default, options: once, bips, perf, ractor, gc, stackprof, vernier, warmup, stats, continuous, chain, mplr)") do |v| + # Accept the old directory names (harness, harness-) for backward compatibility + v = v == 'harness' ? 'default' : v.sub(/^harness-/, '') args.harness = v args.harness_explicit = true end @@ -246,7 +247,7 @@ def default_args executables: {}, out_path: File.expand_path("../data", __dir__), out_override: nil, - harness: "harness", + harness: "default", harness_explicit: false, yjit_opts: "", zjit_opts: "", diff --git a/lib/benchmark_filter.rb b/lib/benchmark_filter.rb index a9ed5586..18f2b0b8 100644 --- a/lib/benchmark_filter.rb +++ b/lib/benchmark_filter.rb @@ -30,7 +30,7 @@ def matches_category?(name) def ractor_harness_benchmark?(name) benchmark_metadata = @metadata[name] || {} benchmark_metadata['ractor_only'] || - (benchmark_metadata['ractor'] && benchmark_metadata['default_harness'] == 'harness-ractor') + (benchmark_metadata['ractor'] && benchmark_metadata['default_harness'] == 'ractor') end def matches_name_filter?(name) diff --git a/lib/benchmark_suite.rb b/lib/benchmark_suite.rb index 48c474b9..1a247748 100644 --- a/lib/benchmark_suite.rb +++ b/lib/benchmark_suite.rb @@ -18,7 +18,8 @@ class BenchmarkSuite BENCHMARKS_DIR = "benchmarks" RACTOR_CATEGORY = ["ractor"].freeze RACTOR_ONLY_CATEGORY = ["ractor-only"].freeze - RACTOR_HARNESS = "harness-ractor" + RACTOR_HARNESS = "ractor" + HARNESS_DIR = File.expand_path("../harness", __dir__) PIN_CPU_ENV = "RUBY_BENCH_PIN_CPU" attr_reader :categories, :name_filters, :excludes, :out_path, :harness, :harness_explicit, :pre_init, :no_pinning, :force_pinning, :bench_dir @@ -207,13 +208,13 @@ def run_single_benchmark(script_path, result_json_path, ruby, cmd_prefix, env, b # Set up the benchmarking command cmd = (cmd_prefix + [ *ruby, - "-I", benchmark_harness, *pre_init, + *harness_args_for(benchmark_harness), script_path, ].compact).map(&:to_s) # Do the benchmarking - result = BenchmarkRunner.check_call(cmd, env: env, raise_error: false, quiet: quiet) + result = BenchmarkRunner.check_call(cmd, env: env.merge("HARNESS" => benchmark_harness), raise_error: false, quiet: quiet) result[:command] = cmd.shelljoin result ensure @@ -329,6 +330,17 @@ def setarch_prefix prefix end + # If harness is 'default', use default (no -r needed) + # Otherwise use -r to load the specific harness file with full path + def harness_args_for(benchmark_harness) + @harness_args ||= {} + @harness_args[benchmark_harness] ||= if benchmark_harness == "default" + [] + else + ["-r", File.join(HARNESS_DIR, benchmark_harness)] + end + end + # Resolve the pre_init file path into a form that can be required def expand_pre_init(path) path = Pathname.new(path) diff --git a/harness/harness-common.rb b/lib/harness.rb similarity index 100% rename from harness/harness-common.rb rename to lib/harness.rb diff --git a/harness/harness-extra.rb b/lib/harness/extra.rb similarity index 88% rename from harness/harness-extra.rb rename to lib/harness/extra.rb index 18a3654f..c3e526bb 100644 --- a/harness/harness-extra.rb +++ b/lib/harness/extra.rb @@ -32,12 +32,12 @@ def benchmark_name # Get name of harness (stackprof, vernier, etc) from the file path of the loaded harness. def harness_name + harness_dir = File.expand_path("../../harness", __dir__) $LOADED_FEATURES.reverse_each do |feat| - if m = feat.match(%r{/harness-([^/]+)/harness\.rb$}) - return m[1] - end + return File.basename(feat, ".rb") if File.dirname(feat) == harness_dir end - raise "Unable to determine harness name" + # Default to 'harness' if we can't determine the name + 'harness' end # Share a single timestamp for everything from this execution. @@ -46,7 +46,7 @@ def harness_name # Create a consistent file path in the data directory # so that the data can be further analyzed. def output_file_path(prefix: harness_name, suffix: benchmark_name, ruby_info: ruby_version_info, timestamp: TIMESTAMP, ext: "bin") - File.expand_path("../data/#{prefix}-#{timestamp}-#{ruby_info}-#{suffix}.#{ext}", __dir__) + File.expand_path("../../data/#{prefix}-#{timestamp}-#{ruby_info}-#{suffix}.#{ext}", __dir__) end # Can we get the benchmark config name from somewhere? diff --git a/lib/harness/loader.rb b/lib/harness/loader.rb new file mode 100644 index 00000000..e4a5800e --- /dev/null +++ b/lib/harness/loader.rb @@ -0,0 +1,26 @@ +# Use harness/default.rb by default. You can change it with: +# 1. HARNESS environment variable (preferred, keeps harness close to code) +# 2. -r option when calling Ruby directly +# +# Examples: +# HARNESS=perf ruby benchmarks/railsbench/benchmark.rb # uses harness/perf.rb +# HARNESS=once ruby benchmarks/railsbench/benchmark.rb # uses harness/once.rb +# ruby -r./harness/ractor benchmarks/railsbench/benchmark.rb # uses harness/ractor.rb + +if RUBY_ENGINE == "spinel" + require_relative "../../harness/spinel" +else + # Only load the default harness if no other harness has defined run_benchmark + unless defined?(run_benchmark) + harness_name = ENV['HARNESS'] || 'default' + harness_dir = File.expand_path("../../harness", __dir__) + harness_path = File.join(harness_dir, "#{harness_name}.rb") + + unless File.file?(harness_path) + available = Dir.glob("#{harness_dir}/*.rb").map { |f| File.basename(f, '.rb') }.sort + raise LoadError, "Harness '#{harness_name}' not found. Available harnesses: #{available.join(', ')}" + end + + require harness_path + end +end diff --git a/misc/gen_30k_variables.rb b/misc/gen_30k_variables.rb index a55db4ad..df24c92e 100644 --- a/misc/gen_30k_variables.rb +++ b/misc/gen_30k_variables.rb @@ -22,7 +22,7 @@ out = +"" out << "# frozen_string_literal: true\n" out << "# Generated by misc/gen_30k_variables.rb: #{num_methods} methods x #{vars_per_method} local variables = #{num_methods * vars_per_method} local variables\n" -out << "require_relative '../harness/loader'\n\n" +out << "require_relative '../lib/harness/loader'\n\n" num_methods.times do |m| out << "def fun_l0_n#{m}(x)\n" diff --git a/run_once.sh b/run_once.sh deleted file mode 100755 index d2035e8d..00000000 --- a/run_once.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/usr/bin/env bash -# Script to run a single benchmark once -# You can pass --yjit-stats and other ruby arguments to this script. -# Automatically detects Ractor benchmarks and uses the appropriate harness. -# Examples: -# ./run_once.sh --yjit-stats benchmarks/railsbench/benchmark.rb -# ./run_once.sh benchmarks-ractor/optcarrot/benchmark.rb - -# Detect if any argument contains benchmarks-ractor/ to determine harness -HARNESS="./harness" -for arg in "$@"; do - if [[ "$arg" == *"benchmarks-ractor/"* ]]; then - HARNESS="./harness-ractor" - break - fi -done - -WARMUP_ITRS=0 MIN_BENCH_ITRS=1 MIN_BENCH_TIME=0 ruby -I"$HARNESS" "$@" \ No newline at end of file diff --git a/test/argument_parser_test.rb b/test/argument_parser_test.rb index 739e10a3..5e732078 100644 --- a/test/argument_parser_test.rb +++ b/test/argument_parser_test.rb @@ -44,7 +44,7 @@ def setup_mock_ruby(path) assert_equal({ 'ruby' => [mock_ruby] }, args.executables) assert_equal File.expand_path("./data"), args.out_path assert_nil args.out_override - assert_equal "harness", args.harness + assert_equal "default", args.harness assert_equal "", args.yjit_opts assert_equal [], args.categories assert_equal [], args.name_filters @@ -296,12 +296,12 @@ def setup_mock_ruby(path) assert_equal ['headline', 'micro'], args.categories end - it 'sets harness to harness-ractor when category is ractor' do + it 'sets harness to ractor when category is ractor' do parser = ArgumentParser.new args = parser.parse(['--category=ractor']) assert_equal ['ractor'], args.categories - assert_equal 'harness-ractor', args.harness + assert_equal 'ractor', args.harness end it 'allows multiple category flags' do @@ -354,18 +354,25 @@ def setup_mock_ruby(path) end describe '--harness option' do - it 'sets harness directory' do + it 'sets harness name' do parser = ArgumentParser.new args = parser.parse(['--harness=once']) - assert_equal 'harness-once', args.harness + assert_equal 'once', args.harness end - it 'accepts harness- prefix' do + it 'strips harness- prefix for backward compatibility' do parser = ArgumentParser.new args = parser.parse(['--harness=harness-stats']) - assert_equal 'harness-stats', args.harness + assert_equal 'stats', args.harness + end + + it 'maps the old default harness name to default' do + parser = ArgumentParser.new + args = parser.parse(['--harness=harness']) + + assert_equal 'default', args.harness end end diff --git a/test/benchmark_filter_test.rb b/test/benchmark_filter_test.rb index 1abdfada..b7302bb3 100644 --- a/test/benchmark_filter_test.rb +++ b/test/benchmark_filter_test.rb @@ -48,16 +48,16 @@ end it 'excludes ractor harness benchmarks from default runs' do - metadata = @metadata.merge('ractor_harness_bench' => { 'category' => 'other', 'ractor' => true, 'default_harness' => 'harness-ractor' }) + metadata = @metadata.merge('ractor_harness_bench' => { 'category' => 'other', 'ractor' => true, 'default_harness' => 'ractor' }) filter = BenchmarkFilter.new(categories: [], name_filters: [], excludes: [], metadata: metadata) assert_equal true, filter.match?('fib') - assert_equal true, filter.match?('ractor_bench') # ractor: true without harness-ractor runs in default - assert_equal false, filter.match?('ractor_harness_bench') # ractor: true with harness-ractor excluded + assert_equal true, filter.match?('ractor_bench') # ractor: true without ractor harness runs in default + assert_equal false, filter.match?('ractor_harness_bench') # ractor: true with ractor harness excluded end it 'includes ractor harness benchmarks in ractor category' do - metadata = @metadata.merge('ractor_harness_bench' => { 'category' => 'other', 'ractor' => true, 'default_harness' => 'harness-ractor' }) + metadata = @metadata.merge('ractor_harness_bench' => { 'category' => 'other', 'ractor' => true, 'default_harness' => 'ractor' }) filter = BenchmarkFilter.new(categories: ['ractor'], name_filters: [], excludes: [], metadata: metadata) assert_equal true, filter.match?('ractor_harness_bench') diff --git a/test/benchmark_runner_cli_test.rb b/test/benchmark_runner_cli_test.rb index dadff083..ef039e3c 100644 --- a/test/benchmark_runner_cli_test.rb +++ b/test/benchmark_runner_cli_test.rb @@ -44,7 +44,7 @@ def create_args(overrides = {}) executables: executables, out_path: nil, out_override: nil, - harness: 'harness', + harness: 'default', yjit_opts: '', categories: [], name_filters: [], @@ -107,13 +107,13 @@ def create_args(overrides = {}) } } bench_harnesses = { - 'fib' => 'harness', - 'symbol-name-ractor' => 'harness-ractor' + 'fib' => 'default', + 'symbol-name-ractor' => 'ractor' } sections = cli.send(:build_output_sections, ['ruby'], bench_data, bench_harnesses, {}) - assert_equal ['harness-ractor', 'harness'], sections.map { |section| section[:title] } + assert_equal ['ractor', 'default'], sections.map { |section| section[:title] } assert_equal ['bench', 'ractors', 'ruby (ms)'], sections[0][:table][0] assert_equal ['symbol-name-ractor', '0', '1000.0 ± 0.0%'], sections[0][:table][1] assert_equal ['', '2', '2000.0 ± 0.0%'], sections[0][:table][2] @@ -153,11 +153,11 @@ def create_args(overrides = {}) } } } - bench_harnesses = { 'fib' => 'harness', 'object-new' => 'harness-ractor' } + bench_harnesses = { 'fib' => 'default', 'object-new' => 'ractor' } sections = cli.send(:build_output_sections, ['ruby'], bench_data, bench_harnesses, {}) - ractor = sections.find { |section| section[:title] == 'harness-ractor' } - normal = sections.find { |section| section[:title] == 'harness' } + ractor = sections.find { |section| section[:title] == 'ractor' } + normal = sections.find { |section| section[:title] == 'default' } assert_equal 'ractor-local-workload', ractor[:gc_scope] refute normal.key?(:gc_scope) @@ -195,7 +195,7 @@ def create_args(overrides = {}) } } - sections = cli.send(:build_output_sections, ['ruby'], bench_data, { 'object-new' => 'harness-ractor' }, {}) + sections = cli.send(:build_output_sections, ['ruby'], bench_data, { 'object-new' => 'ractor' }, {}) assert sections.one?, 'single-harness fixture must produce exactly one section' section = sections.first @@ -230,7 +230,7 @@ def create_args(overrides = {}) } } - sections = cli.send(:build_output_sections, ['ruby'], bench_data, { 'ractor-dead-set' => 'harness-ractor' }, {}) + sections = cli.send(:build_output_sections, ['ruby'], bench_data, { 'ractor-dead-set' => 'ractor' }, {}) output = BenchmarkRunner.build_output_text({ 'ruby' => 'ruby 4.1.0dev' }, nil, nil, {}, sections: sections) assert_match(/Scenario benchmarks sample only the worker Ractors whose bodies the scenario wraps in measure_worker_gc/, output) @@ -259,7 +259,7 @@ def create_args(overrides = {}) } } - sections = cli.send(:build_output_sections, ['ruby'], bench_data, { 'ractor-dead-set' => 'harness-ractor' }, {}) + sections = cli.send(:build_output_sections, ['ruby'], bench_data, { 'ractor-dead-set' => 'ractor' }, {}) memory = sections.first[:memory_tables] assert_equal 1, memory.size @@ -283,7 +283,7 @@ def create_args(overrides = {}) 'ruby' => { 'fib' => { 'warmup' => [0.1], 'bench' => [0.1], 'rss' => 10 * 1024 * 1024 } } } - sections = cli.send(:build_output_sections, ['ruby'], bench_data, { 'fib' => 'harness' }, {}) + sections = cli.send(:build_output_sections, ['ruby'], bench_data, { 'fib' => 'default' }, {}) assert_nil sections.first[:memory_tables] output = BenchmarkRunner.build_output_text({ 'ruby' => 'ruby 4.1.0dev' }, nil, nil, {}, sections: sections) @@ -314,7 +314,7 @@ def create_args(overrides = {}) data end bench_data = { 'ruby' => { 'object-new' => blob.call(%w[0 2], nil), 'ractor-dead-set' => blob.call(%w[1 2], 'scenario') } } - harnesses = { 'object-new' => 'harness-ractor', 'ractor-dead-set' => 'harness-ractor' } + harnesses = { 'object-new' => 'ractor', 'ractor-dead-set' => 'ractor' } sections = cli.send(:build_output_sections, ['ruby'], bench_data, harnesses, {}) diff --git a/test/benchmark_runner_test.rb b/test/benchmark_runner_test.rb index 77dcb4e7..e7c66a94 100644 --- a/test/benchmark_runner_test.rb +++ b/test/benchmark_runner_test.rb @@ -474,7 +474,7 @@ def git(dir, *args) ruby_descriptions = { 'ruby' => 'ruby 4.1.0dev' } sections = [ { - title: 'harness-ractor', + title: 'ractor', table: [['bench', 'ractors', 'ruby (ms)'], ['object-new', '1', '200.0']], format: ['%s', '%s', '%.1f'], failures: {}, @@ -493,7 +493,7 @@ def git(dir, *args) ruby_descriptions, sections.first[:table], sections.first[:format], {}, sections: sections ) - assert_includes result, "GC summary (worker sum, harness-ractor):\n" + assert_includes result, "GC summary (worker sum, ractor):\n" assert_includes result, "Hidden columns (zero or N/A in every row): global/iter, compacts*\n" refute_includes result, 'GC metric notes:', 'a hidden compaction column needs no note' assert_includes result, '- Tables marked worker sum add Ractor-local counters' @@ -580,7 +580,7 @@ def git(dir, *args) ruby_descriptions = { 'ruby' => 'ruby 4.1.0dev' } sections = [ { - title: 'harness-ractor', + title: 'ractor', table: [['bench', 'ractors', 'ruby (ms)'], ['object-new', '0', '200.0']], format: ['%s', '%s', '%.1f'], failures: {}, @@ -660,14 +660,14 @@ def git(dir, *args) ruby_descriptions = { 'ruby' => 'ruby 3.3.0' } sections = [ { - title: 'harness', + title: 'default', table: [['bench', 'ruby (ms)'], ['fib', '100.0']], format: ['%s', '%.1f'], failures: {}, include_gc: false, }, { - title: 'harness-ractor', + title: 'ractor', table: [['bench', 'ractors', 'ruby (ms)'], ['symbol-name-ractor', '0', '200.0']], format: ['%s', '%s', '%.1f'], failures: {}, @@ -679,8 +679,8 @@ def git(dir, *args) ruby_descriptions, sections.first[:table], sections.first[:format], {}, sections: sections ) - assert_includes result, "harness:\n" - assert_includes result, "harness-ractor:\n" + assert_includes result, "default:\n" + assert_includes result, "ractor:\n" assert_includes result, 'fib' assert_includes result, 'symbol-name-ractor' end @@ -689,7 +689,7 @@ def git(dir, *args) ruby_descriptions = { 'ruby' => 'ruby 3.3.0', 'exp' => 'ruby 3.3.0 exp' } sections = [ { - title: 'harness-gc', + title: 'gc', table: [['bench', 'ruby (ms)', 'exp (ms)'], ['gcbench', '100.0', '90.0']], format: ['%s', '%.1f', '%.1f'], failures: {}, @@ -702,7 +702,7 @@ def git(dir, *args) ruby_descriptions, sections.first[:table], sections.first[:format], {}, sections: sections ) - assert_includes result, "GC time ratios (harness-gc):\n" + assert_includes result, "GC time ratios (gc):\n" assert_includes result, '- GC time ratios are ruby/comparison;' end diff --git a/test/benchmark_suite_test.rb b/test/benchmark_suite_test.rb index 27ae3c46..1f7366aa 100644 --- a/test/benchmark_suite_test.rb +++ b/test/benchmark_suite_test.rb @@ -51,13 +51,13 @@ categories: ['micro'], name_filters: [], out_path: @out_path, - harness: 'harness' + harness: 'default' ) assert_equal ['micro'], suite.categories assert_equal [], suite.name_filters assert_equal @out_path, suite.out_path - assert_equal 'harness', suite.harness + assert_equal 'default', suite.harness assert_nil suite.pre_init assert_equal false, suite.no_pinning end @@ -67,7 +67,7 @@ categories: [], name_filters: [], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -79,11 +79,11 @@ categories: ['micro'], name_filters: [], out_path: @out_path, - harness: 'harness' + harness: 'default' ) assert_equal 'benchmarks', suite.bench_dir - assert_equal 'harness', suite.harness + assert_equal 'default', suite.harness assert_equal ['micro'], suite.categories end @@ -92,11 +92,11 @@ categories: ['ractor'], name_filters: [], out_path: @out_path, - harness: 'harness' + harness: 'default' ) assert_equal 'benchmarks', suite.bench_dir - assert_equal 'harness', suite.harness + assert_equal 'default', suite.harness assert_equal ['ractor'], suite.categories end @@ -114,7 +114,7 @@ categories: [], name_filters: [], out_path: @out_path, - harness: 'harness-ractor' + harness: 'ractor' ) assert_equal false, suite_auto.harness_explicit end @@ -124,7 +124,7 @@ before do @metadata_with_harness = { 'simple' => { 'category' => 'micro' }, - 'custom_harness_bench' => { 'category' => 'other', 'default_harness' => 'harness' } + 'custom_harness_bench' => { 'category' => 'other', 'default_harness' => 'default' } } File.write('benchmarks.yml', YAML.dump(@metadata_with_harness)) end @@ -134,12 +134,12 @@ categories: [], name_filters: [], out_path: @out_path, - harness: 'harness-ractor', + harness: 'ractor', harness_explicit: false ) - assert_equal 'harness', suite.send(:benchmark_harness_for, 'custom_harness_bench') - assert_equal 'harness-ractor', suite.send(:benchmark_harness_for, 'simple') + assert_equal 'default', suite.send(:benchmark_harness_for, 'custom_harness_bench') + assert_equal 'ractor', suite.send(:benchmark_harness_for, 'simple') end it 'ignores default_harness when harness is explicit' do @@ -162,7 +162,7 @@ categories: ['ractor'], name_filters: [], out_path: @out_path, - harness: 'harness-ractor' + harness: 'ractor' ) assert_equal true, suite.send(:ractor_category_run?) @@ -173,7 +173,7 @@ categories: ['micro'], name_filters: [], out_path: @out_path, - harness: 'harness' + harness: 'default' ) assert_equal false, suite.send(:ractor_category_run?) @@ -186,7 +186,7 @@ def base_cmd_for(ruby_description) categories: ['micro'], name_filters: [], out_path: @out_path, - harness: 'harness' + harness: 'default' ) suite.stub :linux?, true do suite.stub :setarch_prefix, [] do @@ -212,7 +212,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -234,7 +234,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -257,7 +257,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -278,7 +278,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['failing'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -309,7 +309,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['subdir'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -323,6 +323,7 @@ def base_cmd_for(ruby_description) end it 'handles ractor category with ractor benchmarks' do + skip 'target Ruby has no Ractor' unless defined?(Ractor) # Create a ractor benchmark File.write('benchmarks/ractor_test.rb', <<~RUBY) require 'json' @@ -344,7 +345,7 @@ def base_cmd_for(ruby_description) categories: ['ractor'], name_filters: [], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -357,9 +358,12 @@ def base_cmd_for(ruby_description) assert_includes bench_data, 'ractor_test' refute_includes bench_data, 'simple' assert_empty bench_failures + + end it 'runs each Ractor count in its own process and merges the results' do + skip 'target Ruby has no Ractor' unless defined?(Ractor) File.write('benchmarks/per_count.rb', <<~RUBY) require 'json' count = ENV.fetch('RUBY_BENCH_RACTORS') @@ -377,7 +381,7 @@ def base_cmd_for(ruby_description) custom_json_path = File.join(@out_path, 'nested', 'custom_result.json') ENV['RESULT_JSON_PATH'] = custom_json_path ENV['RUBY_BENCH_RACTORS'] = '0,2,1' - suite = BenchmarkSuite.new(categories: ['ractor'], name_filters: [], out_path: @out_path, harness: 'harness', no_pinning: true) + suite = BenchmarkSuite.new(categories: ['ractor'], name_filters: [], out_path: @out_path, harness: 'default', no_pinning: true) bench_data, bench_failures = nil capture_io do @@ -399,6 +403,7 @@ def base_cmd_for(ruby_description) end it 'reports a Ractor benchmark as failed when one count fails' do + skip 'target Ruby has no Ractor' unless defined?(Ractor) File.write('benchmarks/fails_at_two.rb', <<~RUBY) require 'json' count = ENV.fetch('RUBY_BENCH_RACTORS') @@ -408,7 +413,7 @@ def base_cmd_for(ruby_description) File.write('benchmarks.yml', YAML.dump('fails_at_two' => { 'category' => 'other', 'ractor' => true })) ENV['RUBY_BENCH_RACTORS'] = '0,2,4' - suite = BenchmarkSuite.new(categories: ['ractor'], name_filters: [], out_path: @out_path, harness: 'harness', no_pinning: true) + suite = BenchmarkSuite.new(categories: ['ractor'], name_filters: [], out_path: @out_path, harness: 'default', no_pinning: true) bench_data, bench_failures = nil capture_io do @@ -423,6 +428,7 @@ def base_cmd_for(ruby_description) end it 'skips count 0 for a scenario benchmark unless it is the only count' do + skip 'target Ruby has no Ractor' unless defined?(Ractor) File.write('benchmarks/scenario.rb', <<~RUBY) require 'json' count = ENV.fetch('RUBY_BENCH_RACTORS') @@ -430,7 +436,7 @@ def base_cmd_for(ruby_description) File.write(ENV['RESULT_JSON_PATH'], JSON.generate('warmup' => [], 'bench' => [count.to_f], 'rss' => 1)) RUBY File.write('benchmarks.yml', YAML.dump('scenario' => { 'category' => 'other', 'ractor' => true, 'ractor_scenario' => true })) - suite = BenchmarkSuite.new(categories: ['ractor'], name_filters: [], out_path: @out_path, harness: 'harness', no_pinning: true) + suite = BenchmarkSuite.new(categories: ['ractor'], name_filters: [], out_path: @out_path, harness: 'default', no_pinning: true) ENV['RUBY_BENCH_RACTORS'] = '0,1,2' bench_data, bench_failures = nil @@ -461,7 +467,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', pre_init: pre_init_file, no_pinning: true ) @@ -483,7 +489,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', pre_init: pre_init_file, no_pinning: true ) @@ -503,7 +509,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', pre_init: pre_init_file, no_pinning: true ) @@ -513,6 +519,29 @@ def base_cmd_for(ruby_description) assert_equal 'setup', suite.pre_init[3].to_s end + it 'runs pre_init before the Ractor harness freezes main' do + skip 'target Ruby has no Ractor' unless defined?(Ractor) + pre_init_file = File.join(@temp_dir, 'pre_init.rb') + File.write(pre_init_file, "@pre_init_ran = true\n") + File.write('benchmarks/ractor_pre_init.rb', <<~RUBY) + require 'json' + File.write(ENV['RESULT_JSON_PATH'], JSON.generate('warmup' => [], 'bench' => [1.0], 'pre_init_ran' => @pre_init_ran)) + RUBY + File.write('benchmarks.yml', YAML.dump('ractor_pre_init' => { 'category' => 'other', 'ractor' => true, 'default_harness' => 'ractor' })) + suite = BenchmarkSuite.new(categories: ['ractor'], name_filters: [], out_path: @out_path, harness: 'default', pre_init: pre_init_file, no_pinning: true) + + ENV['RUBY_BENCH_RACTORS'] = '1' + bench_data, bench_failures = nil + capture_io do + bench_data, bench_failures = suite.run(ruby: [RbConfig.ruby], ruby_description: 'ruby 3.2.0') + end + + assert_empty bench_failures + assert_equal true, bench_data['ractor_pre_init']['pre_init_ran'] + ensure + ENV.delete('RUBY_BENCH_RACTORS') + end + it 'exits when pre_init file does not exist' do output = capture_io do assert_raises(SystemExit) do @@ -520,7 +549,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', pre_init: '/nonexistent/file.rb', no_pinning: true ) @@ -536,7 +565,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', pre_init: @temp_dir, no_pinning: true ) @@ -550,7 +579,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -569,7 +598,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -590,7 +619,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -624,7 +653,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['bench_a'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -644,7 +673,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -659,7 +688,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -673,7 +702,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -688,7 +717,7 @@ def base_cmd_for(ruby_description) assert_includes result[:data], 'warmup' assert_includes result[:data], 'bench' assert_includes result[:data], 'rss' - assert_equal 'harness', result[:harness] + assert_equal 'default', result[:harness] assert_nil result[:failure] end @@ -699,7 +728,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['failing'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -712,7 +741,7 @@ def base_cmd_for(ruby_description) assert_equal 'failing', result[:name] assert_nil result[:data] assert_equal 1, result[:failure] - assert_equal 'harness', result[:harness] + assert_equal 'default', result[:harness] end it 'produces same data as run for the same benchmark' do @@ -720,7 +749,7 @@ def base_cmd_for(ruby_description) categories: [], name_filters: ['simple'], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) @@ -738,6 +767,30 @@ def base_cmd_for(ruby_description) assert_equal run_data['simple'].keys.sort, single_result[:data].keys.sort end + it 'loads the selected harness when HARNESS is set in the environment' do + loader_path = File.expand_path('../lib/harness/loader', __dir__) + harness_dir = File.expand_path('../harness', __dir__) + File.write('benchmarks/harness_probe.rb', <<~RUBY) + require #{loader_path.inspect} + require 'json' + harnesses = $LOADED_FEATURES.filter_map { |f| File.basename(f, '.rb') if File.dirname(f) == #{harness_dir.inspect} } + File.write(ENV['RESULT_JSON_PATH'], JSON.generate('warmup' => [], 'bench' => [0.001], 'harnesses' => harnesses)) + RUBY + suite = BenchmarkSuite.new(categories: [], name_filters: ['harness_probe'], out_path: @out_path, harness: 'default', harness_explicit: true, no_pinning: true) + + result = nil + ENV['HARNESS'] = 'once' + begin + capture_io do + result = suite.run_benchmark(suite.benchmarks.first, ruby: [RbConfig.ruby], ruby_description: 'ruby 3.2.0') + end + ensure + ENV.delete('HARNESS') + end + + assert_equal ['default'], result[:data]['harnesses'] + end + describe 'on Windows' do before do File.write('benchmarks/pin_cpu.rb', <<~RUBY) @@ -747,7 +800,7 @@ def base_cmd_for(ruby_description) end def pin_cpu_seen_by_benchmark(ruby_description: 'ruby 3.2.0', **options) - suite = BenchmarkSuite.new(categories: [], name_filters: ['pin_cpu'], out_path: @out_path, harness: 'harness', **options) + suite = BenchmarkSuite.new(categories: [], name_filters: ['pin_cpu'], out_path: @out_path, harness: 'default', **options) result = nil suite.stub(:windows?, true) do suite.stub(:linux?, false) do @@ -800,7 +853,7 @@ def pin_cpu_seen_by_benchmark(ruby_description: 'ruby 3.2.0', **options) categories: ['micro'], name_filters: [], out_path: @out_path, - harness: 'harness', + harness: 'default', no_pinning: true ) diff --git a/test/benchmarks_test.rb b/test/benchmarks_test.rb index a5ca16d3..c249f737 100644 --- a/test/benchmarks_test.rb +++ b/test/benchmarks_test.rb @@ -31,7 +31,7 @@ benchmarks_yml.each do |name, metadata| category = metadata.fetch('category') do # Ractor scaling benchmarks are intentionally kept in their own section. - metadata['default_harness'] == 'harness-ractor' ? 'ractor-scaling' : 'other' + metadata['default_harness'] == 'ractor' ? 'ractor-scaling' : 'other' end benchmark_names_by_category[category] << name end diff --git a/test/ractor_gc_harness_test.rb b/test/ractor_gc_harness_test.rb index 8c76c51d..f4df718c 100644 --- a/test/ractor_gc_harness_test.rb +++ b/test/ractor_gc_harness_test.rb @@ -190,7 +190,7 @@ def run_workload(body, env: {}) Dir.mktmpdir do |dir| result_path = File.join(dir, 'results.json') script = File.join(dir, 'workload.rb') - File.write(script, "Warning[:experimental] = false\nrequire #{File.join(ROOT, 'harness', 'loader').inspect}\n" + body) + File.write(script, "Warning[:experimental] = false\nrequire #{File.join(ROOT, 'lib', 'harness', 'loader').inspect}\n" + body) env = CLEAN_ENV.merge( 'RUBY_BENCH_RACTOR_GC' => '1', 'RUBY_BENCH_RACTORS' => '0,1,2', @@ -200,7 +200,7 @@ def run_workload(body, env: {}) 'MIN_BENCH_TIME' => '0', 'RESULT_JSON_PATH' => result_path ).merge(env) - stdout, stderr, status = Open3.capture3(env, @ruby, "-I#{File.join(ROOT, 'harness-ractor')}", script, chdir: ROOT) + stdout, stderr, status = Open3.capture3(env, @ruby, "-r#{File.join(ROOT, 'harness', 'ractor')}", script, chdir: ROOT) yield stdout, stderr, status, result_path end end @@ -465,7 +465,7 @@ def run_workload(body, env: {}) 'RESULT_JSON_PATH' => result_path ) script = File.join(ROOT, 'benchmarks', bench, 'benchmark.rb') - stdout, stderr, status = Open3.capture3(env, @ruby, "-I#{File.join(ROOT, 'harness-ractor')}", script, chdir: ROOT) + stdout, stderr, status = Open3.capture3(env, @ruby, "-r#{File.join(ROOT, 'harness', 'ractor')}", script, chdir: ROOT) assert status.success?, "#{bench} failed:\n#{stdout}\n#{stderr}" data = JSON.parse(File.read(result_path)) diff --git a/test/ractor_harness_test.rb b/test/ractor_harness_test.rb index cab55a44..2c4568c0 100644 --- a/test/ractor_harness_test.rb +++ b/test/ractor_harness_test.rb @@ -13,7 +13,7 @@ Dir.mkdir(calls_dir) script = File.join(dir, 'workload.rb') File.write(script, <<~RUBY) - require #{File.join(root, 'harness', 'loader').inspect} + require #{File.join(root, 'lib', 'harness', 'loader').inspect} run_benchmark(1) do |count| File.open(File.join(#{calls_dir.inspect}, Ractor.current.object_id.to_s), 'a') { |f| f.puts(count) } end @@ -28,7 +28,7 @@ 'RESULT_JSON_PATH' => result_path } - stdout, stderr, status = Open3.capture3(env, RbConfig.ruby, "-I#{File.join(root, 'harness-ractor')}", script) + stdout, stderr, status = Open3.capture3(env, RbConfig.ruby, "-r#{File.join(root, 'harness', 'ractor')}", script) assert status.success?, "workload failed:\n#{stdout}\n#{stderr}" data = JSON.parse(File.read(result_path))