Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -654,6 +654,31 @@ Semantics:

The rules live in one place, `tangle_cli.schema_validation`: `check_annotations(mapping, policy=..., error_cls=...)` applied under a named `AnnotationPolicy`. `CALLER_ANNOTATION_POLICY` is the strict input policy described above; `DOCUMENT_ANNOTATION_POLICY` is the lenient policy every pipeline document is validated against (scalar-or-null values, no key rules), matching the schema and hand-authored YAML. Only the caller-supplied input surface is strict: existing documents are accepted exactly as before, and the document check still runs on the merged result, so annotations reaching the output by any route are validated.

##### Root pipeline labels

`@pipeline(labels={...})` writes the compiled pipeline's root `metadata.labels` block — a block separate from `annotations`:

```python
@pipeline(
"Search signals",
labels={"team": "discovery", "domain": "search-signals", "stage": "analysis"},
)
def search_signals() -> Out[str]:
...
```

Semantics:

- **`str -> str`, and stricter than annotations.** Both the dehydrated and the pipeline schema type `metadata.labels` as `additionalProperties: {"type": "string"}`, whereas `metadata.annotations` also admits numbers, booleans and null. A non-mapping argument, a non-string key or value, an empty key, or a template delimiter (`{{`, `{%`, `{#`) raises `InvalidPipelineLabelsError` (a `CompileError`). Diagnostics name the key and the type and never echo a value.
- **The `system/` prefix is allowed here.** That prefix is reserved for Tangle's own *annotations*; nothing reserves a label prefix, so rejecting one would refuse a document both schemas accept.
- **Written before `annotations`** in the `metadata` block, whichever order the keywords were passed in.
- **Author key order is preserved** inside the block; it is not sorted.
- **Omitted, `{}` or `None` is a no-op**, byte for byte — a pipeline that does not use labels compiles exactly as before.
- **Root only.** A `subpipeline` child keeps exactly what its own `@pipeline` declared, so child sidecar names, bytes and component digests are unaffected. A child that wants labels declares its own.
- **Descriptive only.** Nothing in the CLI, the hydrator or the generated API client reads labels; they appear only in the schemas and in `MetadataSpec`. They are not part of compile identity or the overrides fingerprint, and cannot influence placement, routing, scheduling or run identity.

There is no `pipeline_labels` compile keyword mirroring `pipeline_annotations`. Annotations got one because the varying part of that block (environment, owner) comes from a caller's per-environment config; no such need exists for labels today, and the rules are already shared, so adding one later is a small change.

A distribution that reads these annotations from its own config file should call `check_annotations(mapping, policy=CALLER_ANNOTATION_POLICY, error_cls=...)` at config-parse time — passing its own error type and adding the config path and key to the message — so one user mistake produces one diagnostic instead of two competing ones. The compiler's own call is then the backstop for anything arriving by another route.

##### Declaring graph inputs and outputs from the body
Expand Down
2 changes: 1 addition & 1 deletion packages/tangle-cli/src/tangle_cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@
try:
__version__ = metadata_version("tangle-cli")
except PackageNotFoundError:
__version__ = "0.1.26"
__version__ = "0.1.27"

__all__ = ["TangleDynamicDiscoveryClient", "__version__"]
12 changes: 9 additions & 3 deletions packages/tangle-cli/src/tangle_cli/python_pipeline/emit.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,15 @@ def emit_pipeline(g: GraphBuilder) -> tuple[dict[str, Any], set[str]]:
if g.description:
out["description"] = g.description

if g.annotations:
# metadata.annotations preserves user-specified order.
out["metadata"] = {"annotations": dict(g.annotations)}
if g.labels or g.annotations:
# Both blocks preserve user-specified key order. ``labels`` is
# written first, matching the corpus majority.
metadata: dict[str, Any] = {}
if g.labels:
metadata["labels"] = dict(g.labels)
if g.annotations:
metadata["annotations"] = dict(g.annotations)
out["metadata"] = metadata

if g.inputs:
out["inputs"] = list(g.inputs)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,15 @@ class InvalidGraphIoError(CompileError):
"""


class InvalidPipelineLabelsError(CompileError):
"""Raised on a malformed ``@pipeline(labels=...)`` mapping.

Separate from :class:`InvalidPipelineAnnotationsError` so a caller can
tell which metadata block it got wrong. Messages name the key and the
type, never the value.
"""


class InvalidPipelineAnnotationsError(CompileError):
"""Raised on a malformed caller-supplied ``pipeline_annotations`` mapping.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ class GraphBuilder:
name: str
description: str | None = None
annotations: dict[str, str] = field(default_factory=dict)
labels: dict[str, str] = field(default_factory=dict)
inputs: list[dict[str, Any]] = field(default_factory=list)
outputs: list[dict[str, Any]] = field(default_factory=list)
# MULTI-output map (Decision D): ``{output_name: EdgeRef}`` in field
Expand Down
18 changes: 17 additions & 1 deletion packages/tangle-cli/src/tangle_cli/python_pipeline/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,10 @@
validate_flow_direction,
)

from tangle_cli.schema_validation import PIPELINE_LABELS_POLICY, check_annotations

from . import emit
from .errors import InvalidEditorLayoutError
from .errors import InvalidEditorLayoutError, InvalidPipelineLabelsError
from .graph import GraphBuilder


Expand All @@ -39,6 +41,9 @@ class PipelineFn:
description: str | None = None
config_path: str | None = None # path relative to caller_dir
annotations: dict[str, Any] = field(default_factory=dict)
# ``metadata.labels`` — a separate block from annotations, and typed
# ``str -> str`` by both schemas.
labels: dict[str, str] = field(default_factory=dict)
task_annotations: dict[str, Any] = field(default_factory=dict)
caller_dir: Path | None = None
# Convention for the single Out[T] slot's name. Defaults to the PoC
Expand Down Expand Up @@ -135,6 +140,7 @@ def pipeline(
description: str | None = None,
config: str | None = None,
annotations: dict[str, Any] | None = None,
labels: dict[str, str] | None = None,
task_annotations: dict[str, Any] | None = None,
flow_direction: str | None = None,
output_name: str = "wait_for_output",
Expand All @@ -156,6 +162,10 @@ def pipeline(
time so ``--override key=value`` pairs can merge in.
annotations: ``metadata.annotations`` block (e.g. ``version``,
``author``).
labels: ``metadata.labels`` block (e.g. ``team``, ``domain``,
``stage``). A separate block from ``annotations``, restricted
by both schemas to string values. Descriptive only, and root
only — a ``subpipeline`` child declares its own.
flow_direction: Editor rendering direction, written as the
``editor.flow-direction`` root annotation
(``"left-to-right"`` / ``"top-to-bottom"``). Sugar over
Expand All @@ -180,6 +190,11 @@ def decorator(fn: Callable[..., Any]) -> PipelineFn:
caller_dir = None

merged_annotations = dict(annotations or {})
checked_labels = check_annotations(
labels,
policy=PIPELINE_LABELS_POLICY,
error_cls=InvalidPipelineLabelsError,
)
if flow_direction is not None:
# Assigned after the mapping: the typed keyword wins, and an
# existing key keeps its position in key order.
Expand All @@ -193,6 +208,7 @@ def decorator(fn: Callable[..., Any]) -> PipelineFn:
description=description,
config_path=config,
annotations=merged_annotations,
labels=checked_labels,
task_annotations=dict(task_annotations or {}),
caller_dir=caller_dir,
output_name=output_name,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,7 @@ def trace_pipeline(
name=pipeline_fn.name,
description=pipeline_fn.description,
annotations=dict(pipeline_fn.annotations),
labels=dict(pipeline_fn.labels),
)
# AST pre-pass: needed by CallableRef.__call__ to derive task IDs
# from LHS variable names. Stashed on the builder so the contextvar
Expand Down
16 changes: 16 additions & 0 deletions packages/tangle-cli/src/tangle_cli/schema_validation.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,22 @@ class AnnotationPolicy:
reject_non_mapping=True,
)

#: Applied to a ``@pipeline(labels=...)`` mapping. Strict ``str -> str``
#: because both schemas type ``metadata.labels`` as
#: ``additionalProperties: {"type": "string"}`` — narrower than
#: ``metadata.annotations``, which also admits numbers, booleans and null.
#: ``reject_reserved_key_prefix`` is deliberately OFF: ``system/`` is
#: documented as reserved for Tangle's own ANNOTATIONS, and nothing reserves
#: a label prefix, so enabling it would refuse a document the schema accepts.
PIPELINE_LABELS_POLICY = AnnotationPolicy(
label="labels",
require_string_values=True,
require_string_keys=True,
require_non_empty_keys=True,
reject_template_delimiters=True,
reject_non_mapping=True,
)


def check_annotations(
annotations: Any,
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "tangle-cli"
version = "0.1.26"
version = "0.1.27"
description = "CLI for Tangle, the open-source ML pipeline orchestration platform"
readme = "README.md"
authors = [
Expand Down
2 changes: 1 addition & 1 deletion tests/test_packaging.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ def test_tangle_cli_wheel_supports_expert_no_deps_import_path_without_tangle_api
requires_dist = [line for line in metadata.splitlines() if line.startswith("Requires-Dist: ")]
assert not any(name.startswith("tangle_api/") for name in names)
assert "tangle_cli/openapi/openapi.json" not in names
assert "Version: 0.1.26" in metadata
assert "Version: 0.1.27" in metadata
assert "Requires-Dist: tangle-api==0.1.1" in requires_dist
assert not any("extra == 'native'" in line for line in requires_dist)
assert "Provides-Extra: native" in metadata
Expand Down
Loading
Loading