Skip to content

Refactor evaluation to use Inspect, bump package versions, rename package to steerability - #25

Open
emiehling wants to merge 32 commits into
generative-computing:mainfrom
emiehling:eval-refactor
Open

Refactor evaluation to use Inspect, bump package versions, rename package to steerability#25
emiehling wants to merge 32 commits into
generative-computing:mainfrom
emiehling:eval-refactor

Conversation

@emiehling

Copy link
Copy Markdown
Collaborator
  • Replace the Benchmark, metrics, and use-case evaluation modules with an Inspect AI evaluation package (steerability/evaluation/)
  • Add as_inspect_model and ProviderOptions, a generation-only Inspect model provider over a steered pipeline, with a collator that batches concurrent requests (evaluation/batching.py)
  • Add runtime_kwargs_solver for per-sample runtime kwargs, sample_scorer_from_inspect for using Inspect scorers as per-row rewards, and InspectSuite for defining task sets
  • Add the SteeringEval runner (configurations x trials x suites), which resumes from the .eval logs and exposes results, runs_frame, samples_frame, and summarize_runs; rework viz_utils into evaluation/plotting.py over the summarized runs frame
  • Move identity.py into core and add core/sweeps.py (expand_configurations, preflight, PipelineFactory) and core/scoring.py (SampleScorer); the sweep layer has no Inspect dependency
  • Port to transformers v5 and bump the ML dependencies (trl 1.0, peft 0.20, datasets 4.7, accelerate 1.4, vllm 0.26 with trl[vllm], mergekit from a pinned commit); require Python 3.12
  • Replace the plots extra with inspect and viz, declare merging as a uv conflict with inspect, all, dev, docs, and vllm, and commit uv.lock
  • Rename the package, distribution, identifiers, and documentation from aisteer360 to steerability
  • Add the .spipe serialization format (steerability/spipe/), which stores the recipe and, once steered, the frozen resolution in a content-addressed artifact store with a lock section; add SteeringPipeline.to_spipe() and SPipe.load, save, pipeline, verify, and thaw
  • Add the freeze methods on controls (steer_fits, export_state, frozen_form, fit_identity) and the load_checkpoint and load_lora structural controls, which are the frozen forms of trained structural controls
  • Add core/internals/model_layout.py (resolve_model_layout, text_config, register_layout_detector, per-layer head geometry) and support composite multimodal wrappers (Gemma 3), unmerged PEFT/LoRA wrappers, and hybrid attention stacks (Qwen3.5 / Qwen3-Next) in state controls and activation capture
  • Add the system_prompt and user_prefix input controls
  • Add PASTA head profiling as a steer-time fit (HeadProfile); fix scale_position and the decode-phase attention mask under transformers v5
  • Add RAD prefix-reward training (rad/utils/reward_training.py) and sequence-classification heads for the Granite families
  • Extend SASA to fit from contrastive pairs and rework the ITI estimator
  • Left-pack batched prompts on the HF session and before state-hook assembly in _execute_generation
  • Add seeding scoped per item or per dispatch (seed_scope) and counts of the tokens generated across rollouts (Output.generated_tokens, DecodingDriver.max_rollouts_per_query)
  • Fix the phased, routed, and budget-forcing decoding paths across the backend interface; make SearchDriver reward_params row-scoped and fix finished-beam detection in the search Frontier
  • Declare a scope (row or call) on every RUNTIME_KWARGS_SCHEMA entry, validated at steer() and read by the Inspect provider
  • Add backends/vllm/environment.py, which turns the FlashInfer sampler off by default at offline engine boot, sets VLLM_HOOK_WORKER=unified for engines using the vLLM-Hook plugin, and provides serve_environment for launched servers; install vllm-hook-plugins from the steerability-interface branch
  • Extend the TRL wrappers with prompt_format on the DPO/APO preference path (with a boundary warning), validation of training_args against the installed TRL config, DPO multi-loss, and resume_from_checkpoint passed to trainer.train()
  • Support reasoning models by splitting reasoning on token ids when the delimiters are special tokens (utils/thinking.py); add numeric answer extraction (utils/answers.py)
  • Fix canonical_value to recurse into the elements of NumPy object arrays
  • Rewrite the evaluation tutorial (evaluate_steering_pipelines.md) and remove the benchmark, metric, and use-case tutorials; add the spipe concept page and reference pages for the evaluation modules, backends, spipe, utils, and the new controls
  • Add the bibtex-warning hook (docs/_hooks/), make mkdocs build --strict pass, add a test over the docs nav, and rewrite AGENTS.md and README
  • Move the generics and library-wrapper notebooks under algorithms/ and port the benchmark notebooks to Inspect as studies/ with task modules
  • Add the system_prompt, honest-persona prompting, routing-vs-prompting, vLLM serve, and working-with-spipes notebooks; remove the TruthfulQA composite study
  • Refresh the SASA (style self-steering), RAD (prefix-reward detoxification on Granite 4.1), best-of-n, and budget-forcing notebooks
  • Rerun notebooks
  • Add/modify tests for the Inspect provider, batching, solvers, scorers, suite, runner, and frames; spipe codec, format, store, identity, and freeze; sweeps and the runtime-kwargs schema; Gemma 3, hybrid-attention, and LoRA-composition steering; system_prompt, user_prefix, and load_checkpoint / load_lora; the DPO wrapper, TRL resume, and training-args validation; ITI, SASA, and RAD reward training; the search frontier, reward params, phased token boundaries, max rollouts, generated tokens, and padding alignment; the vLLM engine environment; the docs nav
  • Bump version to 0.5.0

Fix the PASTA decode-phase attention mask under transformers v5, tidy the
execution contracts, and update the notebooks for the new stack.
Remove Benchmark and the metrics package in favor of an Inspect AI provider,
suites, and scorers. Add dispatch-scoped seeding and evaluation progress
readouts, rewrite the evaluation tutorial and docs, and rework the notebooks
against the new stack.
Add .spipe as a portable serialization format for SteeringPipeline. Support
composite and PEFT model families in state controls and activation capture,
resolve hybrid attention stacks (Qwen3.5 / Qwen3-Next) in the model layout,
tidy core/internals docstrings, and rework the best-of-n and budget-forcing
notebooks on harder tasks.
Rename the package directory, distribution, references, identifiers, and
documentation from aisteer360 to steerability. Assert the spipe lock records
the toolkit under its package name, normalize formatting after the rename,
and make mkdocs build --strict pass.
Left-pack batched prompts before batched generation on the HF session. Add
prompt_format to the DPO/APO preference path with a boundary warning,
validate TRL training_args against the installed config, add DPO multi-loss
support, and move the mkdocs bibtex-warning hook into docs/_hooks.
Interpret runtime kwargs by declared scope in the Inspect provider and port
the instruction-following study. Make SearchDriver reward_params row-scoped
for per-sample rewards, fix finished-beam detection in the search Frontier,
add PASTA head profiling as a steer-time fit with HeadProfile in the PASTA
notebook, correct scale_position, and apply readability and style edits to
the docs prose.
Add the system_prompt and user_prefix input controls with docs and tests, and
rework the ITI estimator. Split reasoning by token ids when delimiters are
special tokens and left-pack batched prompts before state-hook assembly in
_execute_generation. Move the generics and library-wrapper notebooks under
algorithms/, add the honest-persona prompting recipe and the routed decoding
recipe and routing study, batch the instruction-following study at 32 in
bfloat16, and install mergekit from a pinned commit.
Default the FlashInfer sampler off at offline vLLM boot and add
serve_environment for launched vLLM servers. Keep the vllm extra inside trl's
supported window, thread resume_from_checkpoint to trainer.train() for the
TRL wrappers, silence sklearn lbfgs warnings in ITI head-probe fits, and
refresh the TRL and honest-persona notebooks.
Refresh SASA as style self-steering on an instruct model and RAD as
prefix-reward detoxification on Granite 4.1. Fix the phased, routed, and
budget-forcing decoding paths across the backend seam with tests, adjust the
Inspect provider and suite, normalize trailing whitespace across the tree,
refresh the ITI and TRL notebook outputs, and update the uv lock.
canonical_value() returned ndarray.tolist() without recursing, so an
object-dtype array holding Path, set, or NumPy scalar values escaped the
canonical rules and config_digest() raised TypeError in json.dumps().
This aborted expand_configurations(), batch-key derivation, and .spipe
config ids for any control parameter or gen_kwargs value that arrived
as an object array, for example from a pandas or datasets row. Arrays
now convert through tolist() and recurse, so their elements follow the
same rules as list elements. Numeric arrays produce the same form as
before, so existing config ids are unchanged.

Same pattern as the to_jsonable() fix in generative-computing#24 (issue generative-computing#23); the checkpoint
converter that fix targets is removed by this branch.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
Signed-off-by: Erik Miehling <emiehling@gmail.com>

# Conflicts:
#	aisteer360/evaluation/utils/data_utils.py
#	tests/core/test_benchmark.py
#	tests/core/test_evaluation_utils.py
@emiehling
emiehling requested a review from ingelise September 4, 2026 02:17
Artifact records and sidecars are first-write-wins by content id. A tensor
that appeared in an earlier control's recipe args was recorded as opaque, and
a later control exporting the same content as fitted state kept that opaque
record, so its manifest entry carried no artifact class or fit digest and the
staleness check skipped it. The freeze walk now runs over every enabled
control before any recipe encoding.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
…ifact

`_decode_artifact` wrapped a `SteeringVector` in `VerifiedPrecomputed` based
on the store sidecar's `artifact_class`, while the schema-validated manifest
record was consulted only for staleness. A shared external store can hold a
sidecar written by another bundle for the same content id, so the sidecar
could say `opaque` where the manifest says `direction`. `DecodeContext` now
carries the manifest's artifact records, collected at `SPipe` construction,
and the manifest record decides the wrap when one exists; the sidecar keeps
supplying the encoding, type, and reconstruction metadata.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
Saving onto the backing directory rewrote the manifest and returned regardless
of `artifacts=`, so a thin directory loaded with `artifact_store=` and saved
back with `artifacts="fat"` still held no artifacts. The in-place branch now
copies every referenced artifact into the directory for a fat save (external
sources resolve through the store, and present ids are skipped) and refuses a
thin save onto a directory that embeds artifacts.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
`verify()` took the thin-bundle branch only when no referenced artifact was
present, so a bundle with one embedded artifact and one missing reference ran
integrity verification on the missing id and reported FAILED, while `load()`
counted availability per artifact and deferred. `verify()` now reports the
missing ids in one warning and verifies the present ones, matching `load()`.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
The non-string-key mapping branch of `encode` emitted `$map` pairs in
insertion order, and `json.dumps(sort_keys=True)` sorts object keys only, so
two insertion orders of one mapping produced different fit digests and a false
staleness error. Entries are now sorted by key type and repr; `$map` decodes to
a dict, so the order is not semantic. The shipped `pasta_profiled.spipe` still
verifies, since its fit identity contains no non-string-keyed mapping.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
`split_thinking_ids` with `opened_at_start=False` returned the whole
continuation as the answer when the open subsequence was absent and never
looked for the close subsequence, while the text-mode split treats the open
tag as optional. A chat template that opens the channel in the generation
prompt combined with special-token tags (which route "auto" to token mode)
therefore graded the chain of thought as the answer. Token mode now starts
the reasoning at position 0 when only the close subsequence is present; the
`opened_at_start` flag matters only for a continuation carrying neither tag.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
The `RuntimeError` was gated on the list of tasks whose logs report a
non-success status, so `success=False` with an empty or all-success log list
returned partial results and the runner recorded the cell. The suite now
raises on any failed `eval_set` call, naming the failed tasks when the logs
identify them and `unknown` otherwise.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
`_dispatch` paired records with outputs by `zip`, so a `pipeline.generate`
call returning fewer outputs than prompts left trailing records with neither
an output nor an error. `_run_batch` now raises `RuntimeError` naming both
counts, which routes a multi-row dispatch through the serial poison-isolation
pass so every record ends with an `Output` or a specific error.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
`evaluate_probe` pools each class in descending rendered text length, so the
per-class score tensors follow that order rather than the input order.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
The helper returns None only when the config cannot be loaded; a loaded
config lacking `hidden_size` or `num_hidden_layers` raises `AttributeError`,
consistent with never defaulting a structural fact.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
`SteeringEval(seed=...)` derives one seed per (configuration, trial) and
records it in every frame, but the provider attaches a seed to sampling
dispatches only, so with no `temperature` in `generate_defaults` or any
suite's `generate_overrides` the seeds were never applied and the only signal
was a debug log per request. The runner now warns once before any model work,
recommending `generate_defaults={"temperature": 0}` or an explicit sampling
temperature.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
The shared-vocabulary path fed `[B*K, T+1]` rows to the classifier and relied
on its pooling, which selects the last id not equal to the configured pad id.
With pad set to eos (as `train_prefix_reward_model` does), an eos candidate
pooled onto the last prefix token and received the prefix's reward, while the
cached value scored the candidate itself. For a reward model with a `score`
head the backbone now runs once and the head is applied at the last position,
which is the candidate for every row; a model without `score` keeps the full
forward.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
`prefix_reward_loss` cast the mask to the prediction dtype, so under bf16
autocast the prefix positions, lengths, and the l(l+1)/2 normalizer were
accumulated in bf16, which is integer-exact only to 256. The predictions,
labels, and mask are now cast to float32 before the reduction.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
Only `cache_position` is passed to the candidate forward, and decoder models
derive `position_ids` from it when absent, so a prefix with leading pads read
the candidate at a rotary position shifted by the pad count. The pipeline
never pads a single prompt, so this was reachable only through caller-supplied
`input_ids=` with a padded mask; `last_hidden_states` now raises `ValueError`
for an attention mask containing zeros.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
The regression test for the reward head read at the candidate position used
the hub-free Granite classifier. Under transformers 5.16.1 the Granite head
registration is a no-op (the auto mapping skips registrations for native
config classes), so the test moves onto the hub-free Llama classifier, which
uses the same generic sequence-classification head, and sits with the other
cached/stateless equivalence tests.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
inspect-ai 0.3.263 adds `stream_idle_timeout`, a streaming stall timeout that
Inspect applies with its other retry and timeout knobs and that has no effect
on calls that do not stream. The provider does not stream, so the field joins
`timeout` and `attempt_timeout` in `UPSTREAM_FIELDS`, which keeps the
full-field classification check complete. This is outside the PR generative-computing#25 review
list; it follows from installing the toml-resolved inspect-ai.

Signed-off-by: Erik Miehling <emiehling@gmail.com>
Collapse the former `inspect` and `viz` extras into a single `eval` extra, and
fold `cpo` and `guided` into the core dependencies (`xgrammar`) or leave them to
be installed separately (`econml`). Move contributor tooling out of the extras
and into `[dependency-groups]`, with a `notebooks` group that `dev` includes.

Add the IFEval checker to the `notebooks` group as the contents of the
`inspect-evals[ifeval]` extra, which cannot be named directly since it resolves
inspect-evals down to 0.3.x.

Narrow the `[tool.uv]` conflicts to the extras that can actually co-install, and
trim the core dependency list to what the package imports.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant