Read config values from the environment: _env, EnvField, and _select/_env in pipeline cfg (v0.1.20)

[Volv-G]

(AI-assisted)

What

Config files can now read values from the environment, and Python pipeline configs get the same _select / _env resolution as --config files. Mechanism only: no existing field is wired to an environment variable.

1. _env value directive in any --config file or pipeline config.yaml:

   token: {_env: TANGLE_PROD_TOKEN}
   limit: {_env: RUN_LIMIT, default: 10}
   header:
     - {_env: GATEWAY_HEADER}

2. EnvField tier for ArgsContainer field specs: token=EnvField("TANGLE_PROD_TOKEN", (token, None)) wraps any existing 1–6-tuple spec unchanged.

3. Pipeline cfg (@pipeline(config=...) / default config.yaml, root and children, including a propagate_config broadcast) now resolves a root _select and _env. Before this change a _select there was silently treated as an ordinary nested value.

How

One shared resolver

resolve_config_document(parsed, source, *, mapping_only=False) in args_container.py is the single document-level resolution. It validates the selector tree and every _env directive without reading the environment (dormant branches and helper sections included), then resolves the root _select chain. It returns the selected document plus a per-entry _env resolver, so each caller keeps its own shape rules:

• ArgsContainer._load_config_file: the object / list / _defaults + configs shapes. _env is looked up only in the entries and _defaults it actually returns.
• cfg.read_cfg_document (used by load_cfg, _load_cfg_and_raw and _read_raw_cfg): mapping_only=True, so every branch must be a mapping and _defaults / configs are ordinary keys.

_select messages and behavior for --config files are byte-identical. The only internal change is a keyword-only mapping_only flag threaded through the selector validators.

_env rules

• Exact node shape: {_env: NAME} or {_env: NAME, default: <scalar>}. Any other sibling key is rejected, underscore-prefixed ones included. NAME reuses the _select.env name rule through a single _is_env_name.
• The directive is recognized in value positions only (nested maps and lists, _defaults, configs entries). A document or config-entry mapping is never itself a directive, so an _env: helper or anchor key at the top level still works.
• Eager structure, lazy lookup: every directive in every _select branch is shape-checked in every environment, but only directives in the selected document are read.
• A missing variable with no default fails closed with the variable name, the key path (for example configs[1].token) and the file. An empty string counts as set.
• The resolved value is always a string. default is stringified the same way, so a field has one type whether or not the variable is set. Numbers and booleans use their JSON spelling (10, 1.5, true), so JSON-typed fields round-trip the authored value; dates use ISO format. null is rejected, because it has no honest string form; quote '' for an empty default. Maps and lists are rejected too.
• There is no ${VAR} interpolation. YAML alias sharing is resolved once, with linear cost. An _env inside a recursive alias is rejected. Files with no directive skip the resolver entirely.

Precedence

CLI > config > env > default, per ArgsContainer field. A config value read through _env counts as config, so it beats an EnvField tier. Fields not wrapped in EnvField never read the environment, and no names are mapped automatically. The env string goes through the spec's usual converter (JSON fields, repeatables, enums, typed converters). As before, a CLI value equal to the option default is indistinguishable from an omitted one.

Pipeline cfg ordering

Resolve _select / _env first, then layer --override, .override_config and the broadcast on top.

• An .override_config key must exist in the child's selected branch.
• The broadcast payload (raw_cfg) is the resolved dict and never holds _select / _env nodes.
• Each child resolves its own config.

template_file is fail-closed

template_file: is rejected in every candidate branch, dormant ones included, in every environment, not just the selected one. Non-mapping branches are likewise rejected in every environment.

cfg coercion choice (please weigh in)

_select branch values keep their native YAML types. _env values stay strings and are not YAML-coerced the way raw --override strings are. An environment value is opaque: 007 stays "007" and no stays "no". This matches ArgsContainer. Authors convert explicitly (int(cfg.limit)) or put typed values in _select branches.

Compile identity

PipelineCompileKey never included config content. Within one compile the environment is fixed, so nothing can collide there. Across compiles, however, different selections of the same file produced the same sidecar name (<child>-<hash8>.yaml) for different content.

Now, only for a config that uses _select or _env, a SHA-256 of the resolved config is folded into the key through a reserved "\x00config" fingerprint envelope entry. This applies to the root and to every child that loads its config.

• Different values → different key and sidecar name.
• Identical resolved values → identical key.
• Configs without directives → byte-identical keys and sidecar names, pinned by a test against the legacy key.

The digest is never printed; only hash8 of the whole key reaches a file name.

No value echo

Diagnostics name variables and key paths only. The EnvField conversion errors are raised from None.

• Changed error message: the enum converter no longer echoes the rejected value, for any source including the CLI (Invalid value for X. Valid values: [...]). Without this, an enum value that arrived through _env would be echoed.
• Tests assert on the rendered traceback, not just the message.

Provisional surface: ArgsContainer.origin()

args.origin(name) returns cli, config, env:NAME or default, backed by _origins and excluded from to_dict(). Upstream had no value-source reporting to extend, so this was added to make env:NAME provenance observable. It is currently unused and should be treated as provisional: it may be removed or reshaped before anything depends on it.

Failure modes

• _env becomes a reserved key in value positions. An existing config value that is a mapping containing an _env key would now be read as a directive.
• For files that use _env, value nesting is capped at 256 levels.
• Children that load their config now have it read by the parent loop, to compute the identity. A malformed child config therefore fails slightly earlier, with the same message.
• _env values end up wherever the config value goes, including compiled YAML if a pipeline passes them as task constants. The directive keeps secrets out of checked-in files, not out of outputs.
• Pre-existing, unchanged: the shared name regex uses $, so a name ending in a newline passes validation (affects _select too). Worth a follow-up with fullmatch.

Review focus

• resolve_config_document as the single path, and mapping_only keeping the --config behavior identical.
• The dormant-branch walker (_validate_env_directives): linear, cycle-safe, never reads the environment.
• _config_identity and the envelope in _compile_key_for, especially that directive-free configs keep byte-identical keys.
• The no-echo guarantees.

Tophatting

uv run --frozen pytest tests/test_args_container_env.py tests/test_pipeline_cfg_select_env.py

The two files add 112 tests.

• --config side: _env set / unset / empty / default, stringified defaults, null rejection, nested maps and lists, all document shapes, JSON configs, selected vs dormant _select branches, invalid names, stray siblings, no value echo, the full CLI/config/env/default precedence matrix, JSON and repeatable conversion, and unchanged behavior without directives.
• Pipeline side: real compiles for root and child _select, fail-closed without echo, default fallback, _env in cfg, --override and .override_config precedence over the selected branch, broadcast of resolved values, identity differing across selections and env values, identical identity for identical values, and unchanged sidecar names for directive-free configs.
• Mutation-checked: 26 deliberate breaks of the key logic are all caught.

Checklist

• Full suite green locally on Python 3.12 and 3.13: 1701 passed.
• pyright: 0 new errors (the remaining strict-mode errors are pre-existing on master). ruff clean. git diff --check clean.
• uv build for both packages. The wheel metadata reports Version: 0.1.20, and test_packaging passes.
• Version bumped to 0.1.20 in pyproject.toml, packages/tangle-cli/src/tangle_cli/__init__.py, tests/test_packaging.py, and the uv.lock editable self-entry. The lock diff is the two-line version change only; uv lock was not run.
• README documents _env, the _select interplay, precedence, EnvField / origin(), and the pipeline cfg section.