Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ limit: {_env: RUN_LIMIT, default: 10}
- The directive is exactly `{_env: NAME}` or `{_env: NAME, default: <scalar>}`. Any other key beside `_env` — including underscore-prefixed keys and aliases such as `fallback` — is rejected, and `NAME` follows the same rule as `_select.env` (`[A-Za-z_][A-Za-z0-9_]*`).
- It may appear anywhere a value may appear: under a config key, in `_defaults`, in `configs` entries, and inside nested maps and lists. A document or config-entry mapping is never itself a directive, so an `_env:` helper/anchor key at the top level of a config object is unaffected.
- A missing variable without `default` fails closed with the variable name, the key path (for example `configs[1].token`), and the config file. An empty string counts as set and is used as-is.
- The resolved value is always a string, and a `default` is stringified the same way: numbers and booleans use their JSON spelling (`10`, `1.5`, `true`), dates use ISO format, and `null`/maps/lists are rejected (quote `''` for an empty default). A field therefore has one type whether or not the variable is set, and the command's usual conversion (JSON fields, repeatable options, enums, typed converters) applies downstream in both cases.
- The resolved value is always a string, and a `default` is stringified the same way: numbers and booleans use their JSON spelling (`10`, `1.5`, `true`), dates use ISO format, and `null`/maps/lists are rejected (quote `''` for an empty default). A field therefore has one type whether or not the variable is set, and the command's field typing (below) applies in both cases.
- Diagnostics name the variable, never its value, and no directive value is logged. There is no `${VAR}` string interpolation.

With `_select`, selection happens first, and `_env` applies to the selected document:
Expand All @@ -315,6 +315,17 @@ Every `_env` directive — in every case and `default` branch, and in helper sec

Precedence per field is **CLI > config > environment > default**: an explicit CLI value wins, then the config key (including a value read through `_env`), then an environment variable the command has opted that field into (see [`EnvField`](#shared-cli-helpers-and-logging)), then the default. As before, a CLI value equal to the option default is indistinguishable from an omitted one.

#### Scalar field typing

Command flags and numeric options are typed strictly, so an environment or config string can never reach them as the wrong type. This matters most for flags: `force: {_env: DELETE_FORCE, default: false}` yields the string `"false"`, which Python treats as truthy, so without typing it would skip the delete confirmation.

- A field is typed when its default is a `bool`, `int`, or `float`, or when its spec names a strict converter (`strict_bool`, `strict_int`, `strict_float` from `tangle_cli.args_container`). The latter is how a `bool | None` / `int | None` option with a `None` default is typed; the built-in commands use it for their flags, e.g. `dry_run`, `trusted_hydration_cli`, `allow_downgrade`, and `limit`.
- **bool** accepts a YAML/JSON boolean, `0`/`1`, or exactly one of `true`/`false`, `yes`/`no`, `1`/`0` (case-insensitive, no surrounding whitespace). Anything else is rejected, including `""`, `on`/`off`, lists, and maps.
- **int** accepts an integer or a string of ASCII digits with an optional sign. `1.0`, `1e3`, `0x1f`, `1_000`, and booleans are rejected. **float** accepts a number or a decimal string with an optional exponent; `nan`/`inf` are rejected.
- The same rule applies to a config **string literal** (`force: "false"` → `False`), a value read through `_env`, and an `EnvField` variable. Before 0.1.21 a quoted `"false"` also reached a flag as a truthy string. CLI values are already typed by the parser.
- A rejection names the field and its source — config key, variable, or `_env` at a config key — never the value.
- Only scalar fields are typed. String fields, JSON fields, repeatable options, enums, and values nested inside maps or lists are unchanged: an `_env` inside a nested structure stays a string. Pipeline `cfg` files keep their own typing rules (native YAML values; `_env` strings are not coerced).

## API schema cache and dynamic commands

Refresh the local schema cache for a live backend with:
Expand Down Expand Up @@ -1021,7 +1032,7 @@ Use these for generic downstream behavior such as alternate storage, extra annot

`cli_options.py` centralizes shared Cyclopts annotations such as `BaseUrlOption`, `TokenOption`, `AuthHeaderOption`, `HeaderOption`, `ConfigOption`, and `LogTypeOption`. `cli_helpers.py` centralizes config loading, JSON printing, credential-isolation helpers, and the native-safe `LazyTangleApiClient` proxy. `logger.py` provides `ConsoleLogger`, `NullLogger`, `CaptureLogger`, `logger_for_log_type(...)`, and `run_with_logging(...)`.

`ArgsContainer.load(...)` field specs are tuples (see `ArgsContainer._resolve`). To give one field an environment tier, wrap its unchanged spec: `token=EnvField("TANGLE_PROD_TOKEN", (token, None))`. The field then resolves CLI > config > `os.environ["TANGLE_PROD_TOKEN"]` > default; the raw string (empty counts as set) goes through the spec's usual converter, and a conversion error names the variable without echoing its value. Nothing is mapped automatically: fields that are not wrapped never read the environment. `args.origin(name)` reports where each field came from — `cli`, `config`, `env:NAME`, or `default` — without the value.
`ArgsContainer.load(...)` field specs are tuples (see `ArgsContainer._resolve`). To give one field an environment tier, wrap its unchanged spec: `token=EnvField("TANGLE_PROD_TOKEN", (token, None))`. The field then resolves CLI > config > `os.environ["TANGLE_PROD_TOKEN"]` > default; the raw string (empty counts as set) goes through the spec's converter or the [scalar typing](#scalar-field-typing), and a conversion error names the variable without echoing its value. Nothing is mapped automatically: fields that are not wrapped never read the environment. `args.origin(name)` reports where each field came from — `cli`, `config`, `env:NAME`, or `default` — without the value.

Use these helpers for new SDK commands so top-level imports remain native-free, `--config` behavior stays consistent, credentials from config do not accidentally mix with ambient environment auth, and progress logs stay off structured stdout.

Expand Down
2 changes: 1 addition & 1 deletion packages/tangle-cli/src/tangle_cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@
try:
__version__ = metadata_version("tangle-cli")
except PackageNotFoundError:
__version__ = "0.1.20"
__version__ = "0.1.21"

__all__ = ["TangleDynamicDiscoveryClient", "__version__"]
189 changes: 168 additions & 21 deletions packages/tangle-cli/src/tangle_cli/args_container.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@
Precedence per field is CLI > config > environment (only for fields wrapped
in :class:`EnvField`) > default. Config values may themselves be read from
the environment with the ``{_env: NAME}`` value directive.

Scalar fields are typed strictly: a field whose default is a bool/int/float
(or whose spec names :func:`strict_bool` / :func:`strict_int` /
:func:`strict_float`) parses a config or environment string into that type
and rejects anything else, so ``"false"`` can never reach a flag as a truthy
string.
"""

from __future__ import annotations
Expand Down Expand Up @@ -274,6 +280,72 @@ def __post_init__(self) -> None:
raise ValueError("EnvField.spec must be an ArgsContainer field-spec tuple")


class _ScalarValueError(ConfigFileError):
"""A strict scalar converter's value-free rejection reason."""


#: The only accepted boolean spellings, matched case-insensitively and untrimmed.
_BOOL_STRINGS = {"true": True, "false": False, "yes": True, "no": False, "1": True, "0": False}
_INT_PATTERN = re.compile(r"[+-]?[0-9]+")
_FLOAT_PATTERN = re.compile(r"[+-]?(?:[0-9]+(?:\.[0-9]*)?|\.[0-9]+)(?:[eE][+-]?[0-9]+)?")


def strict_bool(value: Any) -> bool:
"""Field converter: a bool, the integer 0/1, or one of ``true``/``false``,
``yes``/``no``, ``1``/``0`` (case-insensitive). Anything else -- the empty
string included -- is rejected without echoing the value."""

if isinstance(value, bool):
return value
if isinstance(value, int) and value in (0, 1):
return bool(value)
if isinstance(value, str) and value.isascii() and value.lower() in _BOOL_STRINGS:
return _BOOL_STRINGS[value.lower()]
raise _ScalarValueError("expected a boolean: true/false, yes/no, or 1/0 (case-insensitive)")


def strict_int(value: Any) -> int:
"""Field converter: an int, or a string of ASCII digits with an optional sign."""

if isinstance(value, int) and not isinstance(value, bool):
return value
if isinstance(value, str) and _INT_PATTERN.fullmatch(value):
return int(value)
raise _ScalarValueError("expected an integer")


def strict_float(value: Any) -> float:
"""Field converter: an int/float, or a decimal string (optional exponent).

``nan``/``inf`` spellings are rejected.
"""

if isinstance(value, (int, float)) and not isinstance(value, bool):
return value
if isinstance(value, str) and _FLOAT_PATTERN.fullmatch(value):
return float(value)
raise _ScalarValueError("expected a number")


_STRICT_SCALARS: dict[type, Callable[[Any], Any]] = {
bool: strict_bool,
int: strict_int,
float: strict_float,
}


def _direct_env_sources(entry: dict[str, Any]) -> dict[str, str]:
"""Map each top-level key whose value is an ``_env`` directive to its variable."""

sources: dict[str, str] = {}
for key, value in entry.items():
if isinstance(value, dict) and ENV_KEY in value:
name = cast(dict[Any, Any], value)[ENV_KEY]
if _is_env_name(name):
sources[key] = name
return sources


class ArgsContainer:
"""Container for resolved CLI arguments with config-file defaults."""

Expand Down Expand Up @@ -665,6 +737,21 @@ def _load_config_file(
) -> list[dict[str, Any]]:
"""Load a YAML/JSON config file as a list of config dictionaries.

See :meth:`_load_config_entries`, which this wraps without provenance.
"""

return [entry for entry, _ in ArgsContainer._load_config_entries(config_path, logger)]

@staticmethod
def _load_config_entries(
config_path: str | Path | None,
logger: Logger | None = None,
) -> list[tuple[dict[str, Any], dict[str, str]]]:
"""Load a YAML/JSON config file as ``(config, env_sources)`` pairs.

``env_sources`` maps each top-level key read through an ``_env``
directive to its variable name, for diagnostics only.

Supported shapes are a single object, a list of objects, or an object
with ``_defaults`` and ``configs`` where defaults are applied to each
config entry. Other top-level keys are ignored, which lets YAML files
Expand All @@ -681,7 +768,7 @@ def _load_config_file(

log = logger or get_default_logger()
if config_path is None:
return [{}]
return [({}, {})]

path = Path(config_path)
if not path.exists():
Expand All @@ -692,15 +779,19 @@ def _load_config_file(
if path.suffix in (".yaml", ".yml"):
parsed = yaml.safe_load(f)
if parsed is None:
return [{}]
return [({}, {})]
else:
parsed = json.load(f)
except (OSError, json.JSONDecodeError, yaml.YAMLError) as exc:
raise ConfigFileError(f"Error loading config file: {exc}") from exc

resolution = resolve_config_document(parsed, path)
parsed = resolution.document
resolve_entry = resolution.resolve_entry

def resolve_entry(
entry: dict[str, Any], entry_path: _ConfigPath
) -> tuple[dict[str, Any], dict[str, str]]:
return resolution.resolve_entry(entry, entry_path), _direct_env_sources(entry)

if isinstance(parsed, dict):
parsed_dict = cast(dict[str, Any], parsed)
Expand All @@ -721,15 +812,26 @@ def _load_config_file(
"configs entry "
f"{index} must be an object, got {type(item).__name__}"
)
defaults = resolve_entry(cast(dict[str, Any], defaults), ("_defaults",))
configs_list = [
defaults, defaults_sources = resolve_entry(
cast(dict[str, Any], defaults), ("_defaults",)
)
entries = [
resolve_entry(item, ("configs", index))
for index, item in enumerate(cast(list[dict[str, Any]], configs_list))
]
merged = apply_defaults(configs_list, defaults)
merged = apply_defaults([entry for entry, _ in entries], defaults)
assert isinstance(merged, list)
log.info(f"Loaded config: {path} ({len(merged)} configs with defaults)")
return merged
return [
(
merged_entry,
{
**{k: v for k, v in defaults_sources.items() if k not in entry},
**sources,
},
)
for merged_entry, (entry, sources) in zip(merged, entries, strict=True)
]
log.info(f"Loaded config: {path} (1 config)")
return [resolve_entry(parsed_dict, ())]

Expand Down Expand Up @@ -796,7 +898,12 @@ def convert(value: Any) -> Any:
return convert

@staticmethod
def _resolve(config: dict[str, Any], **kwargs: Any) -> ArgsContainer:
def _resolve(
config: dict[str, Any],
env_sources: dict[str, str] | None = None,
/,
**kwargs: Any,
) -> ArgsContainer:
"""Resolve CLI args against a single config dict.

Field specs can be:
Expand All @@ -811,6 +918,13 @@ def _resolve(config: dict[str, Any], **kwargs: Any) -> ArgsContainer:
Precedence: an explicit CLI value (one differing from the spec
default), then the config key, then the ``EnvField`` variable, then
the CLI/default value.

Without a converter, a bool/int/float default types the field: config
and environment values go through :func:`strict_bool` /
:func:`strict_int` / :func:`strict_float` (CLI values are already
typed by the parser). A rejection names the field and its source --
config key or variable -- never the value. *env_sources* maps config
keys read through ``_env`` to their variables, for those messages.
"""

resolved: dict[str, Any] = {}
Expand Down Expand Up @@ -853,8 +967,12 @@ def _resolve(config: dict[str, Any], **kwargs: Any) -> ArgsContainer:
if required:
required_fields.append(param_name)

inferred_scalar = False
if converter is None and isinstance(default_value, Enum):
converter = ArgsContainer._make_enum_converter(param_name, type(default_value))
elif converter is None and type(default_value) in _STRICT_SCALARS:
converter = _STRICT_SCALARS[type(default_value)]
inferred_scalar = True

if cli_value is not None and cli_value != default_value:
value, origin = cli_value, "cli"
Expand All @@ -865,18 +983,26 @@ def _resolve(config: dict[str, Any], **kwargs: Any) -> ArgsContainer:
else:
value, origin = cli_value, "default"

if converter and value is not None:
if env_name is not None and origin == f"env:{env_name}":
try:
value = converter(value)
except (ConfigFileError, ValueError, TypeError):
# A converter message may quote the raw value.
raise ConfigFileError(
f"Invalid value for {param_name} from environment "
f"variable {env_name}"
) from None
else:
# An inferred type only parses config/env values; CLI values are typed.
if converter and value is not None and not (inferred_scalar and origin in ("cli", "default")):
from_env_tier = env_name is not None and origin == f"env:{env_name}"
try:
value = converter(value)
except _ScalarValueError as exc:
source = ArgsContainer._describe_source(
origin, config_key, env_name, env_sources or {}
)
raise ConfigFileError(
f"Invalid value for {param_name} from {source}: {exc}"
) from None
except (ConfigFileError, ValueError, TypeError):
if not from_env_tier:
raise
# A converter message may quote the raw value.
raise ConfigFileError(
f"Invalid value for {param_name} from environment "
f"variable {env_name}"
) from None
resolved[param_name] = value
origins[param_name] = origin

Expand All @@ -893,6 +1019,24 @@ def _resolve(config: dict[str, Any], **kwargs: Any) -> ArgsContainer:

return ArgsContainer(resolved, config, origins)

@staticmethod
def _describe_source(
origin: str, config_key: Any, env_name: str | None, env_sources: dict[str, str]
) -> str:
"""Name where a rejected value came from, without the value."""

if origin == "config":
key = _render_config_key(config_key)
if config_key in env_sources:
return (
f"environment variable {env_sources[config_key]} "
f"({ENV_KEY} at config key {key})"
)
return f"config key {key}"
if env_name is not None and origin == f"env:{env_name}":
return f"environment variable {env_name}"
return "the CLI argument" if origin == "cli" else "the default"

@staticmethod
def load(
config_path: str | Path | None,
Expand All @@ -901,8 +1045,8 @@ def load(
) -> list[ArgsContainer]:
"""Load a config file and resolve CLI args against each config entry."""

configs = ArgsContainer._load_config_file(config_path, logger=logger)
return [ArgsContainer._resolve(config, **kwargs) for config in configs]
entries = ArgsContainer._load_config_entries(config_path, logger=logger)
return [ArgsContainer._resolve(config, sources, **kwargs) for config, sources in entries]


@dataclass(frozen=True)
Expand Down Expand Up @@ -969,4 +1113,7 @@ def resolve_config_document(
"EnvField",
"ResolvedConfigDocument",
"resolve_config_document",
"strict_bool",
"strict_float",
"strict_int",
]
7 changes: 4 additions & 3 deletions packages/tangle-cli/src/tangle_cli/components_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

from cyclopts import App, Parameter

from .args_container import strict_bool
from .cli_helpers import load_args_or_exit, optional_path
from .cli_options import ConfigOption, LogTypeOption
from .logger import logger_for_log_type
Expand Down Expand Up @@ -115,8 +116,8 @@ def _components_generate_from_python_impl(
function_name=("function", function_name, None, False),
image=(image, None),
dependencies_from=(dependencies_from, None, optional_path),
strip_code=(strip_code, None),
use_legacy_naming=(use_legacy_naming, None),
strip_code=(strip_code, None, strict_bool),
use_legacy_naming=(use_legacy_naming, None, strict_bool),
mode=(mode, None),
resolve_root=(resolve_root, None, optional_path),
log_type=(log_type, "console"),
Expand Down Expand Up @@ -246,7 +247,7 @@ def components_bump_version(
config,
yaml_file=("yaml_file", yaml_file, None, False, True, optional_path),
set_version=(set_version, None),
update_timestamp=(update_timestamp, None),
update_timestamp=(update_timestamp, None, strict_bool),
log_type=(log_type, "console"),
)
result: dict[str, Any] = {}
Expand Down
Loading
Loading