#37 co-core 0.38.0: JsonExtractor, the jsonpath corpus

AGENTS.md

@@ -8,7 +8,7 @@ The cohort's **headless discrete-transform service**. Consumes `content.process`
 
 Design of record: [docs/specs/2026-09-29-processor-service-design.md](docs/specs/2026-09-29-processor-service-design.md) (the spec). Tracking issue: #1. **Read the spec before changing behaviour**; it is self-contained and names every co-core symbol used.
 
-**Owns:** transforms (co-core's HTML/PDF/CSV/XML extractors now; org adapters later), `gs://co-gcs-processor`, draining `content.process.dlq`. **Does not own:** spec authoring or interpretation (issuers), fetching (Replicator), change detection/diffing (Watcher), the registry (Archiver).
+**Owns:** transforms (co-core's HTML/PDF/CSV/XML/JSON extractors now; org adapters later), `gs://co-gcs-processor`, draining `content.process.dlq`. **Does not own:** spec authoring or interpretation (issuers), fetching (Replicator), change detection/diffing (Watcher), the registry (Archiver).
 
 ## Development Methodology
 
@@ -18,7 +18,7 @@ TDD required. Red → Green → Refactor. No production code without a failing t
 
 Python 3.12, uv, hatchling src layout, pytest, ruff.
 
-**co-core is pinned exactly (`==0.37.0`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"<co-core version>+<LOCAL_GENERATION>"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core).
+**co-core is pinned exactly (`==0.38.0`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"<co-core version>+<LOCAL_GENERATION>"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core).
 
 **Cannobserv wheelhouse.** `co-core`, `co-core-aio`, `co-core-sync` resolve from `./.wheelhouse` (git-ignored) via `[tool.uv] find-links`. Populate it before any `uv` command:
 

docs/plans/2026-10-20-37-co-core-0.38.0.md

@@ -0,0 +1,66 @@
+---
+title: co-core 0.37.0 → 0.38.0 — JsonExtractor, xpath on HTML, ignore_selectors (#37)
+date: 2026-10-20
+status: per #37's hand-off from epic #24 (2026-10-20) and the v0.38.0 release comment
+---
+
+# co-core 0.38.0
+
+## Problem
+
+`application/json` falls back to `HtmlExtractor`, so a `jsonpath` spec was refused (since 0.32.0) and a `full_page` one fingerprinted the raw bytes. cannobserv v0.38.0 (`33676e6`) carries:
+- **cannobserv#354**: `JsonExtractor` for `application/json`, `application/ld+json`, `application/vnd.api+json` (no `+json` suffix rule); the translator maps `jsonpath` → `{"jsonpath": selector}`; `resolve_dispatch_essence` maps a `.json` path to `application/json`.
+- **cannobserv#557**: `xpath` on HTML is honoured.
+- **cannobserv#570**: `ignore_selectors` (HTML only); an uncompilable CSS selector raises `UnsupportedExtractionError`.
+
+## The catch-up reading (0.37.0 → 0.38.0, 0.37.1 between)
+
+Checked in code (`git diff v0.37.0 v0.38.0 -- packages/`), not only the CHANGELOG:
+- **Contract models** (`changes.py`): `ContentProcessCommand`, `ProcessingCompleteEvent`/`Emit`, `ProcessingFailedEvent`/`Emit`, `ContentProcessCommandEmit`, `ProcessingFailureReason`: identical with docstrings stripped (AST compare). The rest is the capture pair (#571) and `InfoItemPrimaryChangedEvent`'s docstring.
+- **Bus**: `envelope.py` registers the capture pair and moves `info_item_primary_changed`'s key to `…:occurred_at` (#578); `streams.py` adds `CONTENT_CAPTURE`. Processor reads neither. `dead_letter`, `exceptions`, `co_core_aio`, `co_core_sync`: version strings only.
+- **`canonical_text`, blob store, hashing**: unchanged. **`spec_fingerprint`**: serialization moved into the shared `_canonical_json`, same `json.dumps` arguments, so `spec1:` is unchanged (the goldens' unmoved `spec_fingerprint`s confirm it).
+- **Extract**: `json_.py`, `_xpath.py`, `_canonical_json.py` new; `html.py` (xpath, ignore_selectors, the selector refusal); the dispatch map; the translator; `refuse_unhonoured`'s two new keys; `media_type`'s `.json` tiebreak.
+- **Dependencies**: the `extract` extra adds `python-jsonpath>=2.2,<3` and declares `soupsieve>=2.0,<3` (already installed via bs4). Sibling floors move to `>=0.38,<0.39`.
+- **CHANGELOG** 0.37.1 (bus-only) and 0.38.0: no security entries.
+
+`processors/extract.py` needs no code change. `resolve_dispatch_essence` is the contract, but Processor never calls it: it dispatches whatever essence arrives, so either adoption order is safe, and the bump is notice to Watcher, not coordination (D5).
+
+## What moves on the existing corpus
+
+0.38.0's generator on `main`'s corpus, unchanged (scratch copy): exactly the three cases the hand-off names.
+- `refused_jsonpath` → extracts (`d6be94c4…`, 64 bytes).
+- `refused_xpath_on_html` → extracts (`c331fdd4…`, 90 bytes).
+- `unknown_essence_is_html`: `52492823…` → `d1da4585…` (now `JsonExtractor`).
+
+## Approach
+
+**Pin.** `pyproject.toml` (three packages), `EXPECTED` in `tests/test_pin.py`, `scripts/build_wheelhouse.sh`'s default tag, `uv lock`. `LOCAL_GENERATION` stays 1: `processor_version` = `0.38.0+1`.
+
+**Corpus** (`cases.json`, before regenerating):
+- Renamed, since they no longer refuse: `refused_jsonpath` → `json_jsonpath`, `refused_xpath_on_html` → `html_xpath`.
+- `unknown_essence_is_html` re-pointed to `application/x-unknown`, same bytes: the HTML fallback again (expect the v1 digest `52492823…` back).
+- New: `json_full_page` (`application/json`, the `$` document chunk); `json_jsonpath_titles` (`$.items[*].title`, the narrowing case); `json_vnd_api` (`application/vnd.api+json`); `html_ignore_selectors` (`full_page`, ignoring `#side`); refusals `refused_empty_jsonpath` (translator), `refused_jsonpath_on_html` and `refused_jsonpath_on_unlisted_plus_json` (`application/problem+json` → HTML), `refused_css_on_json` (`JsonExtractor`).
+
+**Tests** (red first):
+- `test_pin.py`: `EXPECTED = "0.38.0"`.
+- `test_extract.py`: the coverage test names each layer's refusals afresh and the JSON essences; the two renamed cases are not refused; `json_jsonpath_titles`' text is exactly `Agenda item one\nItem two` (Ask 2); `html_ignore_selectors` drops the sidebar; the fallback case's essence is not in `EXTRACTOR_BY_ESSENCE` (so the next essence co-core maps can't silently retire it again); JSON parse failures (`NaN`, `Infinity`, `1e400`, deep nesting, an unpaired surrogate) raise `ValueError`, as a PDF parse failure does.
+- `test_child.py`: those JSON bodies come back `raised` with `ValueError:` under containment, the PDF path, which the handler publishes as `extraction_error`, terminal. The whole-corpus test runs the JSON cases under containment (python-jsonpath's imports).
+- `test_gen_parity_goldens.py`: re-point `test_a_case_that_starts_or_stops_being_refused_is_listed` at `refused_css_on_xml` (trap 1).
+
+**Goldens.** Regenerate co-core-direct; list every line printed. Real corpus: 9/9 `text/html`, expect unchanged.
+
+**Docs.** Spec: amendment entry; D5's "today"; §2 dependency row; §4 refusal row; §5 pin bullet and a 0.37.0 → 0.38.0 sub-bullet; §7 pin-test line and the refusal-coverage prose; the contract quick reference; Open Question 2's range; the Owns line (JSON). AGENTS.md: pin, Owns line. DEPLOYMENT.md: the catch-up example if it names a version.
+
+**Off-repo (operator approval for each post).** Watcher notice on CannObserv/watcher#368 before the deploy: `0.38.0+1`, moved digests, the either-order reasoning. After live checks: Archiver notice on CannObserv/archiver#355 / #356 / #378.
+
+## Steps
+
+1. Red: pin, corpus, tests above.
+2. Green: pin, `uv lock`, regenerate goldens; record the printed list.
+3. Docs.
+4. Gate: `ruff check`, `ruff format --check`, full `pytest` (containment `required`), `uv lock --locked`; PR `Refs #37`.
+5. After FF merge: `git pull --ff-only`; Watcher notice (approved); `scripts/deploy.sh`; live: `starting` shows `0.38.0+1`, each source's first `0.38.0+1` digest equals its last `0.37.0+1` digest (any move: stop, report), `NRestarts=0`; Archiver notice (approved).
+
+## Open questions
+
+None blocking.