Refactor evaluation to use Inspect, bump package versions, rename package to steerability - #25
Open
emiehling wants to merge 32 commits into
Open
Refactor evaluation to use Inspect, bump package versions, rename package to steerability#25emiehling wants to merge 32 commits into
emiehling wants to merge 32 commits into
Conversation
Fix the PASTA decode-phase attention mask under transformers v5, tidy the execution contracts, and update the notebooks for the new stack.
Remove Benchmark and the metrics package in favor of an Inspect AI provider, suites, and scorers. Add dispatch-scoped seeding and evaluation progress readouts, rewrite the evaluation tutorial and docs, and rework the notebooks against the new stack.
Add .spipe as a portable serialization format for SteeringPipeline. Support composite and PEFT model families in state controls and activation capture, resolve hybrid attention stacks (Qwen3.5 / Qwen3-Next) in the model layout, tidy core/internals docstrings, and rework the best-of-n and budget-forcing notebooks on harder tasks.
Rename the package directory, distribution, references, identifiers, and documentation from aisteer360 to steerability. Assert the spipe lock records the toolkit under its package name, normalize formatting after the rename, and make mkdocs build --strict pass.
Left-pack batched prompts before batched generation on the HF session. Add prompt_format to the DPO/APO preference path with a boundary warning, validate TRL training_args against the installed config, add DPO multi-loss support, and move the mkdocs bibtex-warning hook into docs/_hooks.
Interpret runtime kwargs by declared scope in the Inspect provider and port the instruction-following study. Make SearchDriver reward_params row-scoped for per-sample rewards, fix finished-beam detection in the search Frontier, add PASTA head profiling as a steer-time fit with HeadProfile in the PASTA notebook, correct scale_position, and apply readability and style edits to the docs prose.
Add the system_prompt and user_prefix input controls with docs and tests, and rework the ITI estimator. Split reasoning by token ids when delimiters are special tokens and left-pack batched prompts before state-hook assembly in _execute_generation. Move the generics and library-wrapper notebooks under algorithms/, add the honest-persona prompting recipe and the routed decoding recipe and routing study, batch the instruction-following study at 32 in bfloat16, and install mergekit from a pinned commit.
Default the FlashInfer sampler off at offline vLLM boot and add serve_environment for launched vLLM servers. Keep the vllm extra inside trl's supported window, thread resume_from_checkpoint to trainer.train() for the TRL wrappers, silence sklearn lbfgs warnings in ITI head-probe fits, and refresh the TRL and honest-persona notebooks.
Refresh SASA as style self-steering on an instruct model and RAD as prefix-reward detoxification on Granite 4.1. Fix the phased, routed, and budget-forcing decoding paths across the backend seam with tests, adjust the Inspect provider and suite, normalize trailing whitespace across the tree, refresh the ITI and TRL notebook outputs, and update the uv lock.
canonical_value() returned ndarray.tolist() without recursing, so an object-dtype array holding Path, set, or NumPy scalar values escaped the canonical rules and config_digest() raised TypeError in json.dumps(). This aborted expand_configurations(), batch-key derivation, and .spipe config ids for any control parameter or gen_kwargs value that arrived as an object array, for example from a pandas or datasets row. Arrays now convert through tolist() and recurse, so their elements follow the same rules as list elements. Numeric arrays produce the same form as before, so existing config ids are unchanged. Same pattern as the to_jsonable() fix in generative-computing#24 (issue generative-computing#23); the checkpoint converter that fix targets is removed by this branch. Signed-off-by: Erik Miehling <emiehling@gmail.com>
Signed-off-by: Erik Miehling <emiehling@gmail.com> # Conflicts: # aisteer360/evaluation/utils/data_utils.py # tests/core/test_benchmark.py # tests/core/test_evaluation_utils.py
Artifact records and sidecars are first-write-wins by content id. A tensor that appeared in an earlier control's recipe args was recorded as opaque, and a later control exporting the same content as fitted state kept that opaque record, so its manifest entry carried no artifact class or fit digest and the staleness check skipped it. The freeze walk now runs over every enabled control before any recipe encoding. Signed-off-by: Erik Miehling <emiehling@gmail.com>
…ifact `_decode_artifact` wrapped a `SteeringVector` in `VerifiedPrecomputed` based on the store sidecar's `artifact_class`, while the schema-validated manifest record was consulted only for staleness. A shared external store can hold a sidecar written by another bundle for the same content id, so the sidecar could say `opaque` where the manifest says `direction`. `DecodeContext` now carries the manifest's artifact records, collected at `SPipe` construction, and the manifest record decides the wrap when one exists; the sidecar keeps supplying the encoding, type, and reconstruction metadata. Signed-off-by: Erik Miehling <emiehling@gmail.com>
Saving onto the backing directory rewrote the manifest and returned regardless of `artifacts=`, so a thin directory loaded with `artifact_store=` and saved back with `artifacts="fat"` still held no artifacts. The in-place branch now copies every referenced artifact into the directory for a fat save (external sources resolve through the store, and present ids are skipped) and refuses a thin save onto a directory that embeds artifacts. Signed-off-by: Erik Miehling <emiehling@gmail.com>
`verify()` took the thin-bundle branch only when no referenced artifact was present, so a bundle with one embedded artifact and one missing reference ran integrity verification on the missing id and reported FAILED, while `load()` counted availability per artifact and deferred. `verify()` now reports the missing ids in one warning and verifies the present ones, matching `load()`. Signed-off-by: Erik Miehling <emiehling@gmail.com>
The non-string-key mapping branch of `encode` emitted `$map` pairs in insertion order, and `json.dumps(sort_keys=True)` sorts object keys only, so two insertion orders of one mapping produced different fit digests and a false staleness error. Entries are now sorted by key type and repr; `$map` decodes to a dict, so the order is not semantic. The shipped `pasta_profiled.spipe` still verifies, since its fit identity contains no non-string-keyed mapping. Signed-off-by: Erik Miehling <emiehling@gmail.com>
`split_thinking_ids` with `opened_at_start=False` returned the whole continuation as the answer when the open subsequence was absent and never looked for the close subsequence, while the text-mode split treats the open tag as optional. A chat template that opens the channel in the generation prompt combined with special-token tags (which route "auto" to token mode) therefore graded the chain of thought as the answer. Token mode now starts the reasoning at position 0 when only the close subsequence is present; the `opened_at_start` flag matters only for a continuation carrying neither tag. Signed-off-by: Erik Miehling <emiehling@gmail.com>
The `RuntimeError` was gated on the list of tasks whose logs report a non-success status, so `success=False` with an empty or all-success log list returned partial results and the runner recorded the cell. The suite now raises on any failed `eval_set` call, naming the failed tasks when the logs identify them and `unknown` otherwise. Signed-off-by: Erik Miehling <emiehling@gmail.com>
`_dispatch` paired records with outputs by `zip`, so a `pipeline.generate` call returning fewer outputs than prompts left trailing records with neither an output nor an error. `_run_batch` now raises `RuntimeError` naming both counts, which routes a multi-row dispatch through the serial poison-isolation pass so every record ends with an `Output` or a specific error. Signed-off-by: Erik Miehling <emiehling@gmail.com>
`evaluate_probe` pools each class in descending rendered text length, so the per-class score tensors follow that order rather than the input order. Signed-off-by: Erik Miehling <emiehling@gmail.com>
The helper returns None only when the config cannot be loaded; a loaded config lacking `hidden_size` or `num_hidden_layers` raises `AttributeError`, consistent with never defaulting a structural fact. Signed-off-by: Erik Miehling <emiehling@gmail.com>
`SteeringEval(seed=...)` derives one seed per (configuration, trial) and
records it in every frame, but the provider attaches a seed to sampling
dispatches only, so with no `temperature` in `generate_defaults` or any
suite's `generate_overrides` the seeds were never applied and the only signal
was a debug log per request. The runner now warns once before any model work,
recommending `generate_defaults={"temperature": 0}` or an explicit sampling
temperature.
Signed-off-by: Erik Miehling <emiehling@gmail.com>
The shared-vocabulary path fed `[B*K, T+1]` rows to the classifier and relied on its pooling, which selects the last id not equal to the configured pad id. With pad set to eos (as `train_prefix_reward_model` does), an eos candidate pooled onto the last prefix token and received the prefix's reward, while the cached value scored the candidate itself. For a reward model with a `score` head the backbone now runs once and the head is applied at the last position, which is the candidate for every row; a model without `score` keeps the full forward. Signed-off-by: Erik Miehling <emiehling@gmail.com>
`prefix_reward_loss` cast the mask to the prediction dtype, so under bf16 autocast the prefix positions, lengths, and the l(l+1)/2 normalizer were accumulated in bf16, which is integer-exact only to 256. The predictions, labels, and mask are now cast to float32 before the reduction. Signed-off-by: Erik Miehling <emiehling@gmail.com>
Only `cache_position` is passed to the candidate forward, and decoder models derive `position_ids` from it when absent, so a prefix with leading pads read the candidate at a rotary position shifted by the pad count. The pipeline never pads a single prompt, so this was reachable only through caller-supplied `input_ids=` with a padded mask; `last_hidden_states` now raises `ValueError` for an attention mask containing zeros. Signed-off-by: Erik Miehling <emiehling@gmail.com>
The regression test for the reward head read at the candidate position used the hub-free Granite classifier. Under transformers 5.16.1 the Granite head registration is a no-op (the auto mapping skips registrations for native config classes), so the test moves onto the hub-free Llama classifier, which uses the same generic sequence-classification head, and sits with the other cached/stateless equivalence tests. Signed-off-by: Erik Miehling <emiehling@gmail.com>
inspect-ai 0.3.263 adds `stream_idle_timeout`, a streaming stall timeout that Inspect applies with its other retry and timeout knobs and that has no effect on calls that do not stream. The provider does not stream, so the field joins `timeout` and `attempt_timeout` in `UPSTREAM_FIELDS`, which keeps the full-field classification check complete. This is outside the PR generative-computing#25 review list; it follows from installing the toml-resolved inspect-ai. Signed-off-by: Erik Miehling <emiehling@gmail.com>
Collapse the former `inspect` and `viz` extras into a single `eval` extra, and fold `cpo` and `guided` into the core dependencies (`xgrammar`) or leave them to be installed separately (`econml`). Move contributor tooling out of the extras and into `[dependency-groups]`, with a `notebooks` group that `dev` includes. Add the IFEval checker to the `notebooks` group as the contents of the `inspect-evals[ifeval]` extra, which cannot be named directly since it resolves inspect-evals down to 0.3.x. Narrow the `[tool.uv]` conflicts to the extras that can actually co-install, and trim the core dependency list to what the package imports.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
as_inspect_modelandProviderOptions, a generation-only Inspect model provider over a steered pipeline, with a collator that batches concurrent requests (evaluation/batching.py)runtime_kwargs_solverfor per-sample runtime kwargs,sample_scorer_from_inspectfor using Inspect scorers as per-row rewards, andInspectSuitefor defining task setsSteeringEvalrunner (configurations x trials x suites), which resumes from the.evallogs and exposesresults,runs_frame,samples_frame, andsummarize_runs; reworkviz_utilsintoevaluation/plotting.pyover the summarized runs frameidentity.pyinto core and addcore/sweeps.py(expand_configurations,preflight,PipelineFactory) andcore/scoring.py(SampleScorer); the sweep layer has no Inspect dependencytrl[vllm], mergekit from a pinned commit); require Python 3.12plotsextra withinspectandviz, declaremergingas a uv conflict withinspect,all,dev,docs, andvllm, and commituv.lock.spipeserialization format (steerability/spipe/), which stores the recipe and, once steered, the frozen resolution in a content-addressed artifact store with a lock section; addSteeringPipeline.to_spipe()andSPipe.load,save,pipeline,verify, andthawsteer_fits,export_state,frozen_form,fit_identity) and theload_checkpointandload_lorastructural controls, which are the frozen forms of trained structural controlscore/internals/model_layout.py(resolve_model_layout,text_config,register_layout_detector, per-layer head geometry) and support composite multimodal wrappers (Gemma 3), unmerged PEFT/LoRA wrappers, and hybrid attention stacks (Qwen3.5 / Qwen3-Next) in state controls and activation capturesystem_promptanduser_prefixinput controlsHeadProfile); fixscale_positionand the decode-phase attention mask under transformers v5_execute_generationseed_scope) and counts of the tokens generated across rollouts (Output.generated_tokens,DecodingDriver.max_rollouts_per_query)SearchDriverreward_paramsrow-scoped and fix finished-beam detection in the searchFrontierscope(roworcall) on everyRUNTIME_KWARGS_SCHEMAentry, validated atsteer()and read by the Inspect providerVLLM_HOOK_WORKER=unifiedfor engines using the vLLM-Hook plugin, and providesserve_environmentfor launched servers; installvllm-hook-pluginsfrom the steerability-interface branchprompt_formaton the DPO/APO preference path (with a boundary warning), validation oftraining_argsagainst the installed TRL config, DPO multi-loss, andresume_from_checkpointpassed totrainer.train()canonical_valueto recurse into the elements of NumPy object arraysmkdocs build --strictpass, add a test over the docs nav, and rewrite AGENTS.md and README