diff --git a/AGENTS.md b/AGENTS.md index 69cd76e..ae8fe2e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ The cohort's **headless discrete-transform service**. Consumes `content.process` Design of record: [docs/specs/2026-09-29-processor-service-design.md](docs/specs/2026-09-29-processor-service-design.md) (the spec). Tracking issue: #1. **Read the spec before changing behaviour**; it is self-contained and names every co-core symbol used. -**Owns:** transforms (co-core's HTML/PDF/CSV/XML extractors now; org adapters later), `gs://co-gcs-processor`, draining `content.process.dlq`. **Does not own:** spec authoring or interpretation (issuers), fetching (Replicator), change detection/diffing (Watcher), the registry (Archiver). +**Owns:** transforms (co-core's HTML/PDF/CSV/XML/JSON extractors now; org adapters later), `gs://co-gcs-processor`, draining `content.process.dlq`. **Does not own:** spec authoring or interpretation (issuers), fetching (Replicator), change detection/diffing (Watcher), the registry (Archiver). ## Development Methodology @@ -18,7 +18,7 @@ TDD required. Red → Green → Refactor. No production code without a failing t Python 3.12, uv, hatchling src layout, pytest, ruff. -**co-core is pinned exactly (`==0.37.0`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"+"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core). +**co-core is pinned exactly (`==0.38.0`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"+"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core). **Cannobserv wheelhouse.** `co-core`, `co-core-aio`, `co-core-sync` resolve from `./.wheelhouse` (git-ignored) via `[tool.uv] find-links`. Populate it before any `uv` command: diff --git a/docs/plans/2026-10-20-37-co-core-0.38.0.md b/docs/plans/2026-10-20-37-co-core-0.38.0.md new file mode 100644 index 0000000..f4acecc --- /dev/null +++ b/docs/plans/2026-10-20-37-co-core-0.38.0.md @@ -0,0 +1,66 @@ +--- +title: co-core 0.37.0 → 0.38.0 — JsonExtractor, xpath on HTML, ignore_selectors (#37) +date: 2026-10-20 +status: per #37's hand-off from epic #24 (2026-10-20) and the v0.38.0 release comment +--- + +# co-core 0.38.0 + +## Problem + +`application/json` falls back to `HtmlExtractor`, so a `jsonpath` spec was refused (since 0.32.0) and a `full_page` one fingerprinted the raw bytes. cannobserv v0.38.0 (`33676e6`) carries: +- **cannobserv#354**: `JsonExtractor` for `application/json`, `application/ld+json`, `application/vnd.api+json` (no `+json` suffix rule); the translator maps `jsonpath` → `{"jsonpath": selector}`; `resolve_dispatch_essence` maps a `.json` path to `application/json`. +- **cannobserv#557**: `xpath` on HTML is honoured. +- **cannobserv#570**: `ignore_selectors` (HTML only); an uncompilable CSS selector raises `UnsupportedExtractionError`. + +## The catch-up reading (0.37.0 → 0.38.0, 0.37.1 between) + +Checked in code (`git diff v0.37.0 v0.38.0 -- packages/`), not only the CHANGELOG: +- **Contract models** (`changes.py`): `ContentProcessCommand`, `ProcessingCompleteEvent`/`Emit`, `ProcessingFailedEvent`/`Emit`, `ContentProcessCommandEmit`, `ProcessingFailureReason`: identical with docstrings stripped (AST compare). The rest is the capture pair (#571) and `InfoItemPrimaryChangedEvent`'s docstring. +- **Bus**: `envelope.py` registers the capture pair and moves `info_item_primary_changed`'s key to `…:occurred_at` (#578); `streams.py` adds `CONTENT_CAPTURE`. Processor reads neither. `dead_letter`, `exceptions`, `co_core_aio`, `co_core_sync`: version strings only. +- **`canonical_text`, blob store, hashing**: unchanged. **`spec_fingerprint`**: serialization moved into the shared `_canonical_json`, same `json.dumps` arguments, so `spec1:` is unchanged (the goldens' unmoved `spec_fingerprint`s confirm it). +- **Extract**: `json_.py`, `_xpath.py`, `_canonical_json.py` new; `html.py` (xpath, ignore_selectors, the selector refusal); the dispatch map; the translator; `refuse_unhonoured`'s two new keys; `media_type`'s `.json` tiebreak. +- **Dependencies**: the `extract` extra adds `python-jsonpath>=2.2,<3` and declares `soupsieve>=2.0,<3` (already installed via bs4). Sibling floors move to `>=0.38,<0.39`. +- **CHANGELOG** 0.37.1 (bus-only) and 0.38.0: no security entries. + +`processors/extract.py` needs no code change. `resolve_dispatch_essence` is the contract, but Processor never calls it: it dispatches whatever essence arrives, so either adoption order is safe, and the bump is notice to Watcher, not coordination (D5). + +## What moves on the existing corpus + +0.38.0's generator on `main`'s corpus, unchanged (scratch copy): exactly the three cases the hand-off names. +- `refused_jsonpath` → extracts (`d6be94c4…`, 64 bytes). +- `refused_xpath_on_html` → extracts (`c331fdd4…`, 90 bytes). +- `unknown_essence_is_html`: `52492823…` → `d1da4585…` (now `JsonExtractor`). + +## Approach + +**Pin.** `pyproject.toml` (three packages), `EXPECTED` in `tests/test_pin.py`, `scripts/build_wheelhouse.sh`'s default tag, `uv lock`. `LOCAL_GENERATION` stays 1: `processor_version` = `0.38.0+1`. + +**Corpus** (`cases.json`, before regenerating): +- Renamed, since they no longer refuse: `refused_jsonpath` → `json_jsonpath`, `refused_xpath_on_html` → `html_xpath`. +- `unknown_essence_is_html` re-pointed to `application/x-unknown`, same bytes: the HTML fallback again (expect the v1 digest `52492823…` back). +- New: `json_full_page` (`application/json`, the `$` document chunk); `json_jsonpath_titles` (`$.items[*].title`, the narrowing case); `json_vnd_api` (`application/vnd.api+json`); `html_ignore_selectors` (`full_page`, ignoring `#side`); refusals `refused_empty_jsonpath` (translator), `refused_jsonpath_on_html` and `refused_jsonpath_on_unlisted_plus_json` (`application/problem+json` → HTML), `refused_css_on_json` (`JsonExtractor`). + +**Tests** (red first): +- `test_pin.py`: `EXPECTED = "0.38.0"`. +- `test_extract.py`: the coverage test names each layer's refusals afresh and the JSON essences; the two renamed cases are not refused; `json_jsonpath_titles`' text is exactly `Agenda item one\nItem two` (Ask 2); `html_ignore_selectors` drops the sidebar; the fallback case's essence is not in `EXTRACTOR_BY_ESSENCE` (so the next essence co-core maps can't silently retire it again); JSON parse failures (`NaN`, `Infinity`, `1e400`, deep nesting, an unpaired surrogate) raise `ValueError`, as a PDF parse failure does. +- `test_child.py`: those JSON bodies come back `raised` with `ValueError:` under containment, the PDF path, which the handler publishes as `extraction_error`, terminal. The whole-corpus test runs the JSON cases under containment (python-jsonpath's imports). +- `test_gen_parity_goldens.py`: re-point `test_a_case_that_starts_or_stops_being_refused_is_listed` at `refused_css_on_xml` (trap 1). + +**Goldens.** Regenerate co-core-direct; list every line printed. Real corpus: 9/9 `text/html`, expect unchanged. + +**Docs.** Spec: amendment entry; D5's "today"; §2 dependency row; §4 refusal row; §5 pin bullet and a 0.37.0 → 0.38.0 sub-bullet; §7 pin-test line and the refusal-coverage prose; the contract quick reference; Open Question 2's range; the Owns line (JSON). AGENTS.md: pin, Owns line. DEPLOYMENT.md: the catch-up example if it names a version. + +**Off-repo (operator approval for each post).** Watcher notice on CannObserv/watcher#368 before the deploy: `0.38.0+1`, moved digests, the either-order reasoning. After live checks: Archiver notice on CannObserv/archiver#355 / #356 / #378. + +## Steps + +1. Red: pin, corpus, tests above. +2. Green: pin, `uv lock`, regenerate goldens; record the printed list. +3. Docs. +4. Gate: `ruff check`, `ruff format --check`, full `pytest` (containment `required`), `uv lock --locked`; PR `Refs #37`. +5. After FF merge: `git pull --ff-only`; Watcher notice (approved); `scripts/deploy.sh`; live: `starting` shows `0.38.0+1`, each source's first `0.38.0+1` digest equals its last `0.37.0+1` digest (any move: stop, report), `NRestarts=0`; Archiver notice (approved). + +## Open questions + +None blocking. diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index ff46361..7d22309 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -75,6 +75,20 @@ on `co-processor`. Watcher decodes the token, since `reason` is a `str` on the consumer class (Section 5). +**Amended 2026-10-20 (co-core 0.38.0, processor#37):** +- The pin moves from 0.37.0 to **0.38.0** (Sections 2 and 5). It carries + cannobserv#354: `application/json`, `application/ld+json` and + `application/vnd.api+json` dispatch to co-core's `JsonExtractor`, which + honours `jsonpath`. It also carries `xpath` on HTML (cannobserv#557) and + `ignore_selectors` on HTML (cannobserv#570). +- A JSON body co-core can't parse is a `ValueError`, a PDF parse failure's path: + `extraction_error`, terminal (Section 4). +- `resolve_dispatch_essence` maps a `.json` path to `application/json`. That is + the contract, but Processor never calls it, so either adoption order is safe: + notice to Watcher on watcher#368, not coordination (Section 5). +- The corpus covers the JSON essences, `jsonpath` narrowing, and the new + refusals (Section 7). + --- ## Why a new service @@ -92,7 +106,8 @@ first named Observo. Reviewing it showed a mismatch of shape, not of capability. consumers. - **Discrete transforms are a class, not a one-off.** Watcher's text extraction is first; Observo's own event work needs agendas and meeting materials - transformed next; JSON extraction (cannobserv#354), OCR and similar follow. + transformed next; JSON extraction (cannobserv#354, taken in processor#37), OCR and + similar follow. - **Transformation separates cleanly from interpretation.** Taking agendas as the example: a generic transform (any agenda → chronological item list) and an org adapter (TVW's agenda format → precise items) are both @@ -116,7 +131,7 @@ state and UI it should not have). | D2 | **Headless and stateless in v1**: no UI, no database. Idempotency is the write-if-absent store; correlation and dedupe are the issuer's (by `command_id`). | | D3 | **Own identities**: broker ACL user, GCS service account, bucket, tailnet tag. Observo's broker#62 `observo` user is stripped. | | D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so Watcher's read grant is a plain bucket-level `objectViewer`. | -| D5 | **co-core pinned exactly; Processor owns the pin** (0.37.0 today, processor#21). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged.* | +| D5 | **co-core pinned exactly; Processor owns the pin** (0.38.0 today, processor#37). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models, `canonical_text` semantics, or `resolve_dispatch_essence`. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged.* | | D6 | **Extraction runs in a killable child process** with a timeout and an address-space limit, never a thread. | | D7 | Broker `NOPERM` and `OOM` are **transient**; a reclaim **re-runs** the extraction (no dedupe key). | | D8 | **v1 scope is #629's requirements only.** | @@ -133,7 +148,7 @@ an ops concern (logs, the broker probe, lag); a cohort-wide plane is later work. function of (input bytes, spec, `processor_version`). Consumes `content.process`, publishes `content.derived`. -**Owns:** the transforms (co-core's generic HTML/PDF/CSV/XML extractors now; org +**Owns:** the transforms (co-core's generic HTML/PDF/CSV/XML/JSON extractors now; org adapters later, selected by `command.processor` + spec); its bucket; draining `content.process.dlq` (the writer of a queue is its drainer). @@ -154,7 +169,7 @@ processor-version pin on the command; no job API — the bus is the interface. | Thing | Value | |---|---| | Repo | `CannObserv/processor` — Python 3.12, uv, hatchling src layout | -| Dependencies | `co-core[extract]`, `co-core-aio[bus]`, `co-core-sync[gcs]`, all `==0.37.0` (Section 5), from the private CannObserv index | +| Dependencies | `co-core[extract]`, `co-core-aio[bus]`, `co-core-sync[gcs]`, all `==0.38.0` (Section 5), from the private CannObserv index | | CI | wheelhouse pulled through WIF as `co-pypi-reader` (org var `GCP_WIF_PROVIDER`), as in the other cohort repos | | VM | exe.dev `co-processor`, 8 GB, default `exeuntu` image, tag `processor` | | Service | systemd unit `processor`: `MemoryMax` below VM RAM, `Restart=on-failure`, `OOMPolicy=continue` (Section 4). Runs as its own user `processor` (no sudo, not in `docker`, no home), from a read-only release at `/srv/processor/live` that `scripts/deploy.sh` builds from a commit on `origin/main` (amended 2026-10-06, #2: the cohort's release standard, broker#22 / status#9) | @@ -320,7 +335,7 @@ Observo adopted in #395/#407) so the later plane can ingest them unchanged. | Malformed digest, or `input_uri` the input store does not recognize | `invalid_input`, terminal (issuer does not re-fetch) | yes | | Input bytes gone, or the download fails its checksum (`DataCorruption`, amended 2026-09-30) | `input_unreadable`, terminal for the command (issuer re-fetches, capped) | yes | | Bytes hash ≠ `input_digest` | `input_digest_mismatch`, terminal | yes | -| Extractor raises, including `MemoryError` under `RLIMIT_AS`, and a spec co-core refuses (`UnsupportedExtractionError` since 0.32.0, cannobserv#523: `regex`, `jsonpath`, an empty or invalid `xpath`, an unknown algorithm, `xpath` on HTML / PDF / CSV, `css` on XML / PDF / CSV), amended 2026-10-12 | `extraction_error`, terminal | yes | +| Extractor raises, including `MemoryError` under `RLIMIT_AS`, a JSON body co-core can't parse (`ValueError`, as for PDF and XML), and a spec co-core refuses (`UnsupportedExtractionError` since 0.32.0, cannobserv#523; on 0.38.0: `regex`, an empty `xpath` or `jsonpath`, an invalid `xpath`, `jsonpath` or CSS selector, an unknown algorithm, `xpath` on PDF / CSV / JSON, `css` on XML / PDF / CSV / JSON, `jsonpath` on anything but JSON, `ignore_selectors` on anything but HTML), amended 2026-10-12 and 2026-10-20 | `extraction_error`, terminal | yes | | Child timeout or crash | — | no; the 3rd attempt publishes `extraction_error`, terminal, and acks | | A non-transient exception escaping the handler (a bug, or an ack or dead-letter refused), amended 2026-09-30, 2026-10-02, 2026-10-05 and 2026-10-17 | at the cap, `dead_lettered`, terminal, unless this entry's fact already went out (a fact published before a refused ack stands) | no; counted like a strike, and the 3rd attempt publishes, then dead-letters the entry with the exception as its reason. A dead-letter that does not land is retried alone, without running the command again (#28) | | GCS 5xx / 429 / timeout / auth; broker `NOPERM`; broker `OOM command not allowed`; broker `MISCONF` / `BUSY` / `MASTERDOWN` / `TRYAGAIN` / `CLUSTERDOWN` / `NOREPLICAS` (amended 2026-09-30) | — | no; the entry is reclaimed | @@ -433,13 +448,29 @@ reclaim re-running (not re-publishing) the extraction. co-core. Since watcher#350 nothing on Watcher's side has to match. Bump it by hand only when Processor's own logic (config merging, dispatch) changes output in a way co-core's version cannot see. -- **co-core `==0.37.0`** since processor#21 (2026-10-17). Before it, - `==0.32.0` since processor#29 (2026-10-12), the catch-up jump from +- **co-core `==0.38.0`** since processor#37 (2026-10-20). Before it, + `==0.37.0` since processor#21 (2026-10-17), and `==0.32.0` since processor#29 + (2026-10-12), the catch-up jump from 0.19.7, the version Watcher's `uv.lock` resolved at the cutover (Watcher's `pyproject.toml` allowed `>=0.19.6,<0.20`), so both sides reported `"0.19.7+1"` from the first shadow command. Since D5's 2026-10-10 amendment, Watcher's lock no longer constrains this pin. Processor needs `GcsBlobStore` (0.19.3+) and the `content.process` types (0.19.4). + - Across 0.37.0 → 0.38.0 (0.37.1 between), the `content.process` / + `content.derived` models (docstrings aside), `canonical_text`, the stores + and `co_core_aio.bus` are unchanged in code. `spec_fingerprint` moves its + serialization into a shared `_canonical_json` with the same `json.dumps` + arguments, so `spec1:` is unchanged. The extract package gains + `JsonExtractor` (cannobserv#354, with `python-jsonpath`), `xpath` on HTML + (cannobserv#557) and `ignore_selectors` (cannobserv#570). `envelope.py` and + `streams.py` gain the `content.capture` pair and move + `info_item_primary_changed`'s key, none of which Processor reads. + `resolve_dispatch_essence` maps `.json` to `application/json`: Processor + never re-resolves an essence, so an old Watcher's ambiguous essence still + reaches HTML, and a new Watcher's `application/json` reaches HTML before the + bump and `JsonExtractor` after it. Notice, on watcher#368. Three goldens + moved on the existing corpus, each a case 0.38.0 newly honours or dispatches + (processor#37). - Across 0.32.0 → 0.37.0, the extract package, `canonical_text`, `spec_fingerprint`, `resolve_dispatch_essence`, the stores and `co_core_aio.bus` are unchanged in code. `ProcessingFailureReason` gains @@ -540,9 +571,17 @@ TDD, red first. (`empty`), a generic document, and an `xpath`. A spec co-core refuses is recorded by the generator as `{"refused": "UnsupportedExtractionError"}`; the core must raise it, and through the child it comes back `raised`. One - refusal at each layer: `extraction_config_from_spec` (`regex`, `jsonpath`, - an empty `xpath`, an unknown algorithm) and each extractor (`xpath` on HTML - and CSV; `css` on XML, PDF and CSV; an invalid `xpath` on XML). + refusal at each layer: `extraction_config_from_spec` (`regex`, an empty + `xpath` or `jsonpath`, an unknown algorithm) and each extractor (`xpath` on + CSV; `css` on XML, PDF, CSV and JSON; `jsonpath` on HTML and on an unlisted + `+json` essence; an invalid `xpath` on XML). + - **JSON (amended 2026-10-20, processor#37).** Each mapped JSON essence but + `ld+json` (the same extractor), `full_page` (one `$` chunk of canonical + JSON) and `jsonpath`, one of whose cases asserts the text narrows to the + selected nodes. `xpath` on HTML and `ignore_selectors` on HTML extract. The + HTML-fallback case uses `application/x-unknown`, and a test holds it outside + `EXTRACTOR_BY_ESSENCE`. JSON parse failures come back `raised` through the + child, as a PDF parse failure does. - **The golden source (amended 2026-10-10, processor#47).** The v1 goldens were Watcher's: `_extract_and_fingerprint` at Watcher `a5d6f34`, on co-core 0.19.7. watcher#350 deleted that code. `scripts/gen_parity_goldens.py` now @@ -569,7 +608,7 @@ TDD, red first. - Parity with Watcher's local path should be proven before shadow, not discovered in it. - **Pin test.** The lock's co-core equals the expected exact version and - `processor_version(LOCAL_GENERATION) == "0.37.0+1"` — a bump is a deliberate, + `processor_version(LOCAL_GENERATION) == "0.38.0+1"` — a bump is a deliberate, test-failing act. - **Shells — one test per Section 4 row.** Input/output stores: a local co-core store on a temp dir behind the same interface, plus fakes raising GCS 5xx / @@ -641,7 +680,7 @@ writing-plans for v1 there. Implementation happens on `co-processor`. --- -## Contract quick reference (co-core 0.19.4–0.37.0, unchanged across them but the refusal, added in 0.32.0, and `dead_lettered`, in 0.37.0) +## Contract quick reference (co-core 0.19.4–0.38.0, unchanged across them but the refusal, added in 0.32.0, and `dead_lettered`, in 0.37.0) | Need | Name | Module | |---|---|---| @@ -682,7 +721,7 @@ echoed, reporting only. `ChangeRevision.processor_version` is never rewritten. `spec_fingerprint` gets the same treatment. 2. ~~**co-core-aio:** does `AsyncBusConsumer` expose a delivery count or a - max-deliveries hook?~~ **Answered, no (0.19.4 through 0.37.0).** It offers + max-deliveries hook?~~ **Answered, no (0.19.4 through 0.38.0).** It offers primitives only: `ensure_group`, `read`, `ack`, `claim_stale` / `claim_stale_page`, and `dead_letter`. It has no delivery count, no hook and no run loop. The in-memory counter stays, and Processor owns its loop. diff --git a/pyproject.toml b/pyproject.toml index 5bf7883..151d4ae 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,9 +8,9 @@ dependencies = [ # Exact, and Processor owns the pin (spec D5, §5). processor_version is # "+", so a bump moves every fact Watcher # compares; tests/test_pin.py makes one a deliberate, test-failing act. - "co-core[extract]==0.37.0", - "co-core-aio[bus]==0.37.0", - "co-core-sync[gcs]==0.37.0", + "co-core[extract]==0.38.0", + "co-core-aio[bus]==0.38.0", + "co-core-sync[gcs]==0.38.0", "pydantic>=2.12,<3", "pydantic-settings>=2.6,<3", # Imported directly: the consumer classifies redis-py's ResponseError / diff --git a/scripts/build_wheelhouse.sh b/scripts/build_wheelhouse.sh index 7e62088..72631ea 100755 --- a/scripts/build_wheelhouse.sh +++ b/scripts/build_wheelhouse.sh @@ -8,7 +8,7 @@ set -euo pipefail root="$(cd "$(dirname "$0")/.." && pwd)" -tag="${1:-v0.37.0}" +tag="${1:-v0.38.0}" : "${GH_TOKEN_CANNOBSERV:?set GH_TOKEN_CANNOBSERV (read access to CannObserv/cannobserv)}" work="$(mktemp -d)" diff --git a/tests/fixtures/parity/cases.json b/tests/fixtures/parity/cases.json index a360147..67560a6 100644 --- a/tests/fixtures/parity/cases.json +++ b/tests/fixtures/parity/cases.json @@ -18,7 +18,7 @@ {"id": "xlsx", "input": "licenses.xlsx", "media_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", "source_spec": {"extraction": {"algorithm": "full_page"}}}, - {"id": "unknown_essence_is_html", "input": "items.json", "media_type": "application/json", + {"id": "unknown_essence_is_html", "input": "items.json", "media_type": "application/x-unknown", "source_spec": {"extraction": {"algorithm": "full_page"}}}, {"id": "none_essence_is_html", "input": "agenda.html", "media_type": null, "source_spec": {"extraction": {"algorithm": "full_page"}}}, @@ -39,7 +39,7 @@ "source_spec": {"extraction": {"algorithm": "full_page"}}}, {"id": "xml_xpath", "input": "register.xml", "media_type": "application/xml", "source_spec": {"extraction": {"algorithm": "xpath", "selector": "//entry/title"}}}, - {"id": "refused_xpath_on_html", "input": "agenda.html", "media_type": "text/html", + {"id": "html_xpath", "input": "agenda.html", "media_type": "text/html", "source_spec": {"extraction": {"algorithm": "xpath", "selector": "//p"}}}, {"id": "refused_css_on_xml", "input": "tailscale_security_bulletins.xml", "media_type": "text/xml", "source_spec": {"extraction": {"algorithm": "css", "selector": "item"}}}, @@ -52,12 +52,29 @@ {"id": "refused_regex", "input": "tailscale_security_bulletins.xml", "media_type": "application/rss+xml", "source_spec": {"extraction": {"algorithm": "regex", "selector": "TS-\\d+"}}}, - {"id": "refused_jsonpath", "input": "items.json", "media_type": "application/json", + {"id": "json_jsonpath", "input": "items.json", "media_type": "application/json", "source_spec": {"extraction": {"algorithm": "jsonpath", "selector": "$.items"}}}, {"id": "refused_empty_xpath", "input": "register.xml", "media_type": "application/xml", "source_spec": {"extraction": {"algorithm": "xpath", "selector": ""}}}, {"id": "refused_unknown_algorithm", "input": "agenda.html", "media_type": "text/html", "source_spec": {"extraction": {"algorithm": "semantic"}}}, {"id": "refused_invalid_xpath", "input": "register.xml", "media_type": "application/xml", - "source_spec": {"extraction": {"algorithm": "xpath", "selector": "//["}}} + "source_spec": {"extraction": {"algorithm": "xpath", "selector": "//["}}}, + {"id": "json_full_page", "input": "items.json", "media_type": "application/json", + "source_spec": {"extraction": {"algorithm": "full_page"}}}, + {"id": "json_jsonpath_titles", "input": "items.json", "media_type": "application/json", + "source_spec": {"extraction": {"algorithm": "jsonpath", "selector": "$.items[*].title"}}}, + {"id": "json_vnd_api", "input": "items.json", "media_type": "application/vnd.api+json", + "source_spec": {"extraction": {"algorithm": "full_page"}}}, + {"id": "html_ignore_selectors", "input": "agenda.html", "media_type": "text/html", + "source_spec": {"extraction": {"algorithm": "full_page", "ignore_selectors": ["#side"]}}}, + {"id": "refused_empty_jsonpath", "input": "items.json", "media_type": "application/json", + "source_spec": {"extraction": {"algorithm": "jsonpath", "selector": ""}}}, + {"id": "refused_jsonpath_on_html", "input": "agenda.html", "media_type": "text/html", + "source_spec": {"extraction": {"algorithm": "jsonpath", "selector": "$.items"}}}, + {"id": "refused_jsonpath_on_unlisted_plus_json", "input": "items.json", + "media_type": "application/problem+json", + "source_spec": {"extraction": {"algorithm": "jsonpath", "selector": "$.items"}}}, + {"id": "refused_css_on_json", "input": "items.json", "media_type": "application/json", + "source_spec": {"extraction": {"algorithm": "css", "selector": "#main"}}} ] diff --git a/tests/fixtures/parity/goldens.json b/tests/fixtures/parity/goldens.json index ae3ff80..d4c364d 100644 --- a/tests/fixtures/parity/goldens.json +++ b/tests/fixtures/parity/goldens.json @@ -1,5 +1,5 @@ { - "co_core": "0.37.0", + "co_core": "0.38.0", "generated_by": "scripts/gen_parity_goldens.py (co-core's pure extract API, direct; #47)", "goldens": { "csv": { @@ -50,12 +50,48 @@ "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", "spec_schema_version": 1 }, + "html_ignore_selectors": { + "content_fingerprint": "sha256:54fb39f5ee423006094f811bc0a36e0345187dfbbdb9935bd0425d954d6a77b6", + "content_size_bytes": 142, + "spec_fingerprint": "spec1:sha256:7d23a355be19080eba7e55104ec097fd2573a49f12db3fa5f38f42193421a5c3", + "spec_schema_version": 1 + }, "html_latin1": { "content_fingerprint": "sha256:449e97f5b3d7bb19f30da0d29126d21417d3b7fe722ffb7556ec3d23fceb72ee", "content_size_bytes": 35, "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", "spec_schema_version": 1 }, + "html_xpath": { + "content_fingerprint": "sha256:c331fdd4416460510d486e08aa51245cf42b53482beee1e70f2f369dff81962f", + "content_size_bytes": 90, + "spec_fingerprint": "spec1:sha256:88f08919e886596ef061e38187464eb95d52a0c468f3cfc3a3580a5494daa022", + "spec_schema_version": 1 + }, + "json_full_page": { + "content_fingerprint": "sha256:d1da45850448ef0dc317a67de8685e7a6e505db6fe94a1cd553786a888f74eac", + "content_size_bytes": 74, + "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", + "spec_schema_version": 1 + }, + "json_jsonpath": { + "content_fingerprint": "sha256:d6be94c49f4d0c9c4084abadf66e4974c89e591c729aa792438ee51d47dee587", + "content_size_bytes": 64, + "spec_fingerprint": "spec1:sha256:e3a8d6e4efe2e815eef50bbedb7d13591a6b0594f90e374eaa6e84920ab75141", + "spec_schema_version": 1 + }, + "json_jsonpath_titles": { + "content_fingerprint": "sha256:7ac461583fec44c91e6e12ab3b76fd84c9c962c7fe4ab7fd8fce852ff70caa0c", + "content_size_bytes": 24, + "spec_fingerprint": "spec1:sha256:d94d5bbb60fb07feaf139a9201523c7be96079adb14afe6ca53c0d43b8524103", + "spec_schema_version": 1 + }, + "json_vnd_api": { + "content_fingerprint": "sha256:d1da45850448ef0dc317a67de8685e7a6e505db6fe94a1cd553786a888f74eac", + "content_size_bytes": 74, + "spec_fingerprint": "spec1:sha256:71ec7a46fe3cd2c105ad4029e31f1fdbe3fdffc17ab976167e793143957c9a60", + "spec_schema_version": 1 + }, "none_essence_is_html": { "content_fingerprint": "sha256:8c815816bbcc0e3ab0fb9ec8b9e77f7aa638bb2ada24cf373112ded580acaebd", "content_size_bytes": 181, @@ -77,19 +113,28 @@ "refused_css_on_csv": { "refused": "UnsupportedExtractionError" }, + "refused_css_on_json": { + "refused": "UnsupportedExtractionError" + }, "refused_css_on_pdf": { "refused": "UnsupportedExtractionError" }, "refused_css_on_xml": { "refused": "UnsupportedExtractionError" }, + "refused_empty_jsonpath": { + "refused": "UnsupportedExtractionError" + }, "refused_empty_xpath": { "refused": "UnsupportedExtractionError" }, "refused_invalid_xpath": { "refused": "UnsupportedExtractionError" }, - "refused_jsonpath": { + "refused_jsonpath_on_html": { + "refused": "UnsupportedExtractionError" + }, + "refused_jsonpath_on_unlisted_plus_json": { "refused": "UnsupportedExtractionError" }, "refused_regex": { @@ -101,9 +146,6 @@ "refused_xpath_on_csv": { "refused": "UnsupportedExtractionError" }, - "refused_xpath_on_html": { - "refused": "UnsupportedExtractionError" - }, "spec_fingerprint_raises": { "content_fingerprint": "sha256:8c815816bbcc0e3ab0fb9ec8b9e77f7aa638bb2ada24cf373112ded580acaebd", "content_size_bytes": 181, diff --git a/tests/processors/test_extract.py b/tests/processors/test_extract.py index dd314d6..fbcb894 100644 --- a/tests/processors/test_extract.py +++ b/tests/processors/test_extract.py @@ -17,6 +17,7 @@ from pathlib import Path import pytest +from co_core.pure.extract.dispatch import EXTRACTOR_BY_ESSENCE from co_core.pure.extract.errors import UnsupportedExtractionError from processor.processors import TRANSFORMS @@ -92,28 +93,54 @@ def test_parity_with_refusals(case: str) -> None: _run(spec) -def test_the_corpus_covers_the_refusals_and_the_xml_essences() -> None: - # The #523 surface: every XML essence dispatched, and a spec refused at each layer. +def test_the_corpus_covers_the_refusals_and_the_dispatched_essences() -> None: + # The #523 and cannobserv#354 surface: every XML and JSON essence dispatched, and a + # spec refused at each layer. essences = {c["media_type"] for c in EXTRACTED} assert {"text/xml", "application/xml", "application/rss+xml", "application/atom+xml"} <= ( essences ) + assert {"application/json", "application/vnd.api+json"} <= essences # extraction_config_from_spec refuses these whatever the essence... assert { "refused_regex", - "refused_jsonpath", "refused_empty_xpath", + "refused_empty_jsonpath", "refused_unknown_algorithm", } <= REFUSED - # ...and the dispatched extractor these. xpath on HTML moves at cannobserv#557. + # ...and the dispatched extractor these. An unlisted +json essence goes to HTML. assert { - "refused_xpath_on_html", "refused_xpath_on_csv", "refused_css_on_xml", "refused_css_on_pdf", "refused_css_on_csv", + "refused_css_on_json", "refused_invalid_xpath", + "refused_jsonpath_on_html", + "refused_jsonpath_on_unlisted_plus_json", } <= REFUSED + # Refused until 0.38.0: jsonpath (cannobserv#354) and xpath on HTML (cannobserv#557). + assert {"json_jsonpath", "json_jsonpath_titles", "html_xpath"}.isdisjoint(REFUSED) + + +def test_the_fallback_case_uses_an_essence_co_core_does_not_map() -> None: + # application/json was this case's "unknown" essence until cannobserv#354 mapped it. + (case,) = [c for c in CASES if c["id"] == "unknown_essence_is_html"] + assert case["media_type"] not in EXTRACTOR_BY_ESSENCE + + +def test_jsonpath_narrows_to_the_selected_nodes() -> None: + # #37 Ask 2: one chunk per selected node, not the whole document. + (case,) = [c for c in CASES if c["id"] == "json_jsonpath_titles"] + assert _run(case).text == b"Agenda item one\nItem two" + + +def test_ignore_selectors_removes_the_matched_elements() -> None: + # cannobserv#570: the sidebar goes, the rest of the page stays. + (case,) = [c for c in CASES if c["id"] == "html_ignore_selectors"] + text = _run(case).text + assert b"Sidebar text" not in text + assert b"Work session on cannabis retail licensing" in text def test_the_feed_golden_is_co_cores_own_pin() -> None: @@ -137,6 +164,34 @@ def test_extractor_errors_propagate() -> None: extract(b"this is not a pdf", "application/pdf", {"extraction": {}}) +# Bodies JsonExtractor can't extract: a ValueError, the PDF and XML parse failures' shape. +UNEXTRACTABLE_JSON = { + "nan": b'{"a": NaN}', + "infinity": b'{"a": Infinity}', + "float_overflow": b'{"a": 1e400}', + "too_deep": b"[" * 100_000 + b"]" * 100_000, + "unpaired_surrogate": b'["\\ud800"]', + "not_json": b"not json", +} + + +@pytest.mark.parametrize("body", UNEXTRACTABLE_JSON.values(), ids=UNEXTRACTABLE_JSON) +def test_a_json_parse_failure_is_a_value_error_not_a_refusal(body: bytes) -> None: + with pytest.raises(ValueError) as raised: + extract(body, "application/json", {"extraction": {"algorithm": "full_page"}}) + assert not isinstance(raised.value, UnsupportedExtractionError) + + +def test_a_descendant_query_past_the_depth_limit_is_a_value_error() -> None: + # python-jsonpath stops ``..`` at 100 levels; co-core turns its RecursionError into + # a ValueError, so it is an extraction_error, never a crash. + body = b'{"a":' * 150 + b"1" + b"}" * 150 + spec = {"extraction": {"algorithm": "jsonpath", "selector": "$..a"}} + with pytest.raises(ValueError) as raised: + extract(body, "application/json", spec) + assert not isinstance(raised.value, UnsupportedExtractionError) + + def _extract_or_refuse(raw: bytes, case: dict) -> None: try: extract(raw, case["media_type"], case["source_spec"]) diff --git a/tests/test_child.py b/tests/test_child.py index 612637d..5acc5db 100644 --- a/tests/test_child.py +++ b/tests/test_child.py @@ -194,6 +194,31 @@ async def test_the_whole_parity_corpus_is_identical_through_the_child() -> None: assert result.value == extract(*args), job_id +@pytest.mark.parametrize( + ("essence", "body"), + [ + ("application/pdf", b"this is not a pdf"), + ("application/json", b'{"a": NaN}'), + ("application/json", b'{"a": 1e400}'), + ("application/json", b"[" * 100_000 + b"]" * 100_000), + ("application/json", b'["\\ud800"]'), + ], + ids=["pdf", "json_nan", "json_float_overflow", "json_too_deep", "json_unpaired_surrogate"], +) +async def test_a_parse_failure_comes_back_raised(essence: str, body: bytes) -> None: + # A JSON body JsonExtractor can't extract takes a PDF parse failure's path: raised, + # a ValueError, which the handler publishes as extraction_error, terminal (#37). + result = await run_in_child( + transform_target(extract), + (body, essence, {"extraction": {"algorithm": "full_page"}}), + timeout_s=60, + rlimit_as_bytes=3 * GiB, + containment=CONTAINMENT, + ) + assert result.kind == "raised", result.detail + assert result.detail.startswith("ValueError: "), result.detail + + async def test_a_compromised_child_cannot_run_code_in_the_parent(tmp_path: Path) -> None: marker = tmp_path / "pwned" result = await _run("evil", str(marker)) diff --git a/tests/test_gen_parity_goldens.py b/tests/test_gen_parity_goldens.py index 89a16f9..4ac6dd4 100644 --- a/tests/test_gen_parity_goldens.py +++ b/tests/test_gen_parity_goldens.py @@ -127,17 +127,17 @@ def test_a_spec_co_core_refuses_is_recorded_as_refused(corpus: Path) -> None: def test_a_case_that_starts_or_stops_being_refused_is_listed(corpus: Path) -> None: - # The bump note's view of e.g. cannobserv#557, when HTML starts honouring xpath. + # The bump note's view of e.g. cannobserv#557, when HTML started honouring xpath (#37). doc = json.loads((corpus / "goldens.json").read_text()) - was_refused = doc["goldens"]["refused_xpath_on_html"] + was_refused = doc["goldens"]["refused_css_on_xml"] was_extracted = doc["goldens"]["xml_xpath"] - doc["goldens"]["refused_xpath_on_html"] = was_extracted + doc["goldens"]["refused_css_on_xml"] = was_extracted doc["goldens"]["xml_xpath"] = was_refused (corpus / "goldens.json").write_text(json.dumps(doc)) moved = gen.main(corpus) - assert "refused_xpath_on_html refused: None -> UnsupportedExtractionError" in moved + assert "refused_css_on_xml refused: None -> UnsupportedExtractionError" in moved assert "xml_xpath refused: UnsupportedExtractionError -> None" in moved assert ( f"xml_xpath content_fingerprint: None -> {was_extracted['content_fingerprint']}" diff --git a/tests/test_pin.py b/tests/test_pin.py index 49fad86..6e6c794 100644 --- a/tests/test_pin.py +++ b/tests/test_pin.py @@ -15,7 +15,7 @@ from processor.processors.extract import LOCAL_GENERATION, PROCESSOR_VERSION -EXPECTED = "0.37.0" +EXPECTED = "0.38.0" ROOT = Path(__file__).resolve().parent.parent PINNED = ("co-core", "co-core-aio", "co-core-sync") diff --git a/uv.lock b/uv.lock index 78ba189..4743697 100644 --- a/uv.lock +++ b/uv.lock @@ -268,14 +268,14 @@ wheels = [ [[package]] name = "co-core" -version = "0.37.0" +version = "0.38.0" source = { registry = ".wheelhouse" } dependencies = [ { name = "dateparser" }, { name = "pydantic" }, ] wheels = [ - { path = "co_core-0.37.0-py3-none-any.whl" }, + { path = "co_core-0.38.0-py3-none-any.whl" }, ] [package.optional-dependencies] @@ -285,18 +285,20 @@ extract = [ { name = "lxml" }, { name = "openpyxl" }, { name = "pypdf" }, + { name = "python-jsonpath" }, + { name = "soupsieve" }, ] [[package]] name = "co-core-aio" -version = "0.37.0" +version = "0.38.0" source = { registry = ".wheelhouse" } dependencies = [ { name = "co-core" }, { name = "httpx" }, ] wheels = [ - { path = "co_core_aio-0.37.0-py3-none-any.whl" }, + { path = "co_core_aio-0.38.0-py3-none-any.whl" }, ] [package.optional-dependencies] @@ -306,7 +308,7 @@ bus = [ [[package]] name = "co-core-sync" -version = "0.37.0" +version = "0.38.0" source = { registry = ".wheelhouse" } dependencies = [ { name = "co-core" }, @@ -314,7 +316,7 @@ dependencies = [ { name = "urllib3" }, ] wheels = [ - { path = "co_core_sync-0.37.0-py3-none-any.whl" }, + { path = "co_core_sync-0.38.0-py3-none-any.whl" }, ] [package.optional-dependencies] @@ -867,9 +869,9 @@ dev = [ [package.metadata] requires-dist = [ - { name = "co-core", extras = ["extract"], specifier = "==0.37.0" }, - { name = "co-core-aio", extras = ["bus"], specifier = "==0.37.0" }, - { name = "co-core-sync", extras = ["gcs"], specifier = "==0.37.0" }, + { name = "co-core", extras = ["extract"], specifier = "==0.38.0" }, + { name = "co-core-aio", extras = ["bus"], specifier = "==0.38.0" }, + { name = "co-core-sync", extras = ["gcs"], specifier = "==0.38.0" }, { name = "google-api-core", specifier = ">=2.27,<3" }, { name = "google-auth", specifier = ">=2,<3" }, { name = "google-cloud-storage", specifier = ">=3,<4" }, @@ -1133,6 +1135,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0d/17/c5c6b53ddc18f297992099b3d9ec16c855c0ccc83263a21fe4d1c625ec6c/python_dotenv-1.2.3-py3-none-any.whl", hash = "sha256:904552145e8bfed22162c09dab1c2b9b54fefa7b23ba780f4f26ca0316b0f0d9", size = 22780, upload-time = "2026-08-16T16:54:52.473Z" }, ] +[[package]] +name = "python-jsonpath" +version = "2.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/10/6c/9fc15bfc9d311ba7d828e58daa7864cc5e92aff165fe5e58d9d24928627f/python_jsonpath-2.2.2.tar.gz", hash = "sha256:bb1a66e82b8c586fcdd53a1df46df92e1cd5e4164aa57f315b03afa76dd14947", size = 51619, upload-time = "2026-10-17T08:41:38.363Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/97/42/9f23be33ed490b89a4acf2e87dcc319f52f8a97239b23fb6409c90199e07/python_jsonpath-2.2.2-py3-none-any.whl", hash = "sha256:b088251c7b63ee8c8ac14e54cf2eefbfe0cf5f59f0d26609a96bacb444ec40ac", size = 65738, upload-time = "2026-10-17T08:41:36.832Z" }, +] + [[package]] name = "pytz" version = "2026.4"