Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ TDD required. Red → Green → Refactor. No production code without a failing t

Python 3.12, uv, hatchling src layout, pytest, ruff.

**co-core is pinned exactly (`==0.38.0`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"<co-core version>+<LOCAL_GENERATION>"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core).
**co-core is pinned exactly (`==0.38.0`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"<co-core version>+<LOCAL_GENERATION>"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models or `canonical_text` semantics. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core).

**Cannobserv wheelhouse.** `co-core`, `co-core-aio`, `co-core-sync` resolve from `./.wheelhouse` (git-ignored) via `[tool.uv] find-links`. Populate it before any `uv` command:

Expand Down
2 changes: 1 addition & 1 deletion docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,7 @@ co-core is pinned `==`, and Processor owns the pin (spec D5, §5; amended 2026-1
1. Rebuild the wheelhouse at the new tag, then update the pin in `pyproject.toml`, `EXPECTED` in `tests/test_pin.py` and the default tag in `scripts/build_wheelhouse.sh` (the pin test holds all three), and `uv lock`.
2. A release that adds an essence or an extraction rule gets its cases in `tests/fixtures/parity/cases.json` first, so their goldens come from co-core. Then regenerate the goldens from co-core directly: `uv run python scripts/gen_parity_goldens.py tests/fixtures/parity`. It prints every digest that moved, and every case that started or stopped being refused. Paste that list into the bump note: each line is a deliberate re-baseline. Never regenerate to make a parity test pass outside a bump. A catch-up across releases reads each CHANGELOG's Downstream lines for the extract, bus and blob-store surfaces, and diffs those packages' code between the tags (#29 did 0.19.7 → 0.32.0).
3. The real corpus (`tests/fixtures/parity/real/`) is never regenerated. It passes unchanged, or the bump note says which recorded fingerprints moved.
4. **Notice to Watcher** before deploy: the new `processor_version`, and the moved digests. A change to the contract needs coordination with Watcher first, not just notice. The contract is the `content.process` / `content.derived` models, `canonical_text` semantics and `resolve_dispatch_essence`.
4. **Notice to Watcher** before deploy: the new `processor_version`, and the moved digests. A change to the contract needs coordination with Watcher first, not just notice. The contract is the `content.process` / `content.derived` models and `canonical_text` semantics. A change to essence resolution is notice, not coordination (spec §5, #57).
5. Merge, then `scripts/deploy.sh`.

## Tailscale DNS
Expand Down
34 changes: 34 additions & 0 deletions docs/plans/2026-10-21-57-dispatch-essence-notice.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
title: The co-core bump rule — a dispatch-essence change is notice (#57)
date: 2026-10-21
status: per #57's hand-off from epic #24 (2026-10-21)
---

# A dispatch-essence change is notice, not coordination

## Problem

The bump rule lists `resolve_dispatch_essence` in the contract, so a change to it needs coordination with Watcher. Processor never calls it: the issuer resolves the essence on its own pin and puts it on the command, and Processor dispatches what arrives (spec §3, cannobserv#486 D1). `extractor_for_essence` and `extraction_overrides_for_essence`, which Processor does call, don't call it either (co-core 0.38.0). #29 coordinated (watcher#366) where notice sufficed; #37 used notice, against the rule as written.

## Approach

- Drop `resolve_dispatch_essence` from the contract list at the four rule sites: `AGENTS.md:21` (the contract list only), `docs/DEPLOYMENT.md` "Bumping co-core" step 4, spec D5, spec §5's bump list.
- §5 says why, once: resolution is the issuer's, on its pin; Processor never re-resolves; so a change is notice, with the either-order reasoning and the moved digests; coordination again only if Processor starts resolving essences itself, which the guard test catches.
- Spec amendment entry, 2026-10-21 (processor#57): #29's coordination on watcher#366 was more than the rule needed; #37 used notice. The 2026-10-12 / 2026-10-20 entries and §5's per-bump notes stay as written.
- Untouched: notice on every `processor_version` move; `EXTRACTOR_BY_ESSENCE`.

## Test

`tests/test_pin.py` (the bump rule's test home): walk every `src/processor/**/*.py` AST, as `test_gen_parity_goldens.py` does, and fail on an `import`/`from … import` of, or a `Name` / `Attribute` use of, `resolve_dispatch_essence` or `extension_media_type`. Code, not text: a docstring mentioning them passes. The message points at §5. A parametrised self-test holds the walker to catching each form and passing a docstring.

Red: the guard on a scratch copy of `src/processor/` with `from co_core.pure.extract import resolve_dispatch_essence` planted, output in the PR.

Known limit: a string-keyed lookup (`getattr(m, "resolve_dispatch_essence")`) isn't caught; that is deliberate evasion, not drift.

## Ship

PR (`Refs #57` until the Watcher notice is posted or declined) → FF merge → `git pull --ff-only`. No deploy: nothing under `src/`, `deploy/`, `scripts/deploy.sh`, `pyproject.toml`, `uv.lock` changes. Then the notice on CannObserv/watcher#368, drafted in the PR, posted only with the operator's approval.

## Open questions

None.
26 changes: 23 additions & 3 deletions docs/specs/2026-09-29-processor-service-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,14 @@ on `co-processor`.
- The corpus covers the JSON essences, `jsonpath` narrowing, and the new
refusals (Section 7).

**Amended 2026-10-21 (the bump rule, processor#57):**
- Essence resolution leaves the contract (D5, Section 5). Processor never
calls `resolve_dispatch_essence` or `extension_media_type`; the issuer
resolves on its own pin. A change to them is notice, with the either-order
reasoning and the moved digests, not coordination. A test holds the premise.
- So #29's coordination on watcher#366 was more than the rule needed, and #37's
notice on watcher#368 was what it needed. The entries above stand as written.

---

## Why a new service
Expand Down Expand Up @@ -131,7 +139,7 @@ state and UI it should not have).
| D2 | **Headless and stateless in v1**: no UI, no database. Idempotency is the write-if-absent store; correlation and dedupe are the issuer's (by `command_id`). |
| D3 | **Own identities**: broker ACL user, GCS service account, bucket, tailnet tag. Observo's broker#62 `observo` user is stripped. |
| D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so Watcher's read grant is a plain bucket-level `objectViewer`. |
| D5 | **co-core pinned exactly; Processor owns the pin** (0.38.0 today, processor#37). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged.* |
| D5 | **co-core pinned exactly; Processor owns the pin** (0.38.0 today, processor#37). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models or `canonical_text` semantics. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged. Amended 2026-10-21 (processor#57): essence resolution left the contract, so a change to it is notice (Section 5).* |
| D6 | **Extraction runs in a killable child process** with a timeout and an address-space limit, never a thread. |
| D7 | Broker `NOPERM` and `OOM` are **transient**; a reclaim **re-runs** the extraction (no dedupe key). |
| D8 | **v1 scope is #629's requirements only.** |
Expand Down Expand Up @@ -505,8 +513,20 @@ reclaim re-running (not re-publishing) the extraction.
Option A re-baselines silently on a processor change alone and audits it as
`check.rebaselined`;
- **is coordinated with Watcher** only when it changes the contract: the
`content.process` / `content.derived` models, `canonical_text` semantics,
or `resolve_dispatch_essence`.
`content.process` / `content.derived` models or `canonical_text`
semantics.
- **Essence resolution is not in the contract** (amended 2026-10-21,
processor#57). The issuer resolves the dispatch essence
(`resolve_dispatch_essence`, with `extension_media_type`) on its own co-core
pin and puts it on the command. Processor dispatches the essence that arrives
and never re-resolves it (Section 3). So either adoption order is safe when a
bump changes resolution: an old issuer keeps sending the essence it sent,
which reaches the same extractor as before; a new issuer's essence reaches the
old extractor on an old Processor, and the new one on a new Processor. Such a
bump is **notice**: it carries that reasoning and the moved digests, which
already show any dispatch change for an essence Watcher sends today. It
becomes coordination again only if Processor starts resolving essences
itself, and `tests/test_pin.py` fails first if it does.
- Until watcher#350 (2026-10-10) both sides extracted, so the pin was in
lockstep with Watcher's, and both repos moved together or neither did.

Expand Down
55 changes: 55 additions & 0 deletions tests/test_pin.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,13 @@
it must be a deliberate act: edit ``EXPECTED`` here, in the same commit as the pin
and ``scripts/build_wheelhouse.sh``'s default tag, with the goldens regenerated
co-core-direct and every moved digest in the bump note.

Whether a bump is notice or coordination turns on the contract (§5). Essence
resolution is outside it only because Processor never resolves an essence itself, and
a test holds that premise (processor#57).
"""

import ast
import re
import tomllib
from importlib.metadata import version
Expand All @@ -18,6 +23,25 @@
EXPECTED = "0.38.0"
ROOT = Path(__file__).resolve().parent.parent
PINNED = ("co-core", "co-core-aio", "co-core-sync")
SRC = ROOT / "src" / "processor"
RESOLVERS = frozenset({"resolve_dispatch_essence", "extension_media_type"})


def _resolver_uses(source: str) -> list[str]:
# Code, not text: an import, a name or an attribute. A docstring may quote them.
found = []
for node in ast.walk(ast.parse(source)):
if isinstance(node, ast.alias):
name = node.name.rsplit(".", 1)[-1]
elif isinstance(node, ast.Name):
name = node.id
elif isinstance(node, ast.Attribute):
name = node.attr
else:
continue
if name in RESOLVERS:
found.append(f"line {node.lineno}: {name}")
return found


def _lock_versions() -> dict[str, str]:
Expand Down Expand Up @@ -53,3 +77,34 @@ def test_the_wheelhouse_builder_defaults_to_the_pin() -> None:
def test_processor_version_is_co_core_plus_generation() -> None:
assert LOCAL_GENERATION == 1 # Watcher's LOCAL_EXTRACTION_GENERATION at the cutover
assert PROCESSOR_VERSION == f"{EXPECTED}+1"


@pytest.mark.parametrize(
("source", "caught"),
[
("from co_core.pure.extract import resolve_dispatch_essence", True),
("from co_core.pure.extract.media_type import extension_media_type as ext", True),
("import co_core.pure.extract.media_type as m\nm.resolve_dispatch_essence(t, u)", True),
("essence = resolve_dispatch_essence", True),
('"""Watcher calls ``resolve_dispatch_essence``; Processor never does."""', False),
("# resolve_dispatch_essence is the issuer's", False),
("from co_core.pure.extract.dispatch import extractor_for_essence", False),
],
)
def test_the_guard_catches_code_not_text(source: str, caught: bool) -> None:
assert bool(_resolver_uses(source)) is caught


def test_processor_never_resolves_a_dispatch_essence() -> None:
# The premise of §5's rule: the issuer resolves on its own pin, Processor dispatches
# what arrives. If this fails, a change to essence resolution is a contract change
# again, and the coordination rule comes back with it.
files = sorted(SRC.rglob("*.py"))
assert SRC / "processors" / "extract.py" in files
uses = {str(f.relative_to(ROOT)): u for f in files if (u := _resolver_uses(f.read_text()))}
assert not uses, (
f"Processor resolves a dispatch essence itself: {uses}. A change to essence "
"resolution is then a contract change, coordinated with Watcher, not notice: "
"restore it to the contract (spec §5, D5; AGENTS.md; DEPLOYMENT.md step 4; "
"processor#57)."
)
Loading