diff --git a/AGENTS.md b/AGENTS.md index a549ba8..b43b715 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## What this is -A review-driven document ingestion pipeline (product name "Bharat Vistaar Docs Pipeline"): Temporal workflows + FastAPI + Postgres + MinIO + Qdrant. Documents move through explicit stages (OCR → review → translate → review → chunk → review → ingest DEV → Super Admin promote to PROD), with every major output persisted as an inspectable artifact rather than hidden. +Knowledge Provider receives files, pushed JSON, and content fetched from approved URLs. FastAPI exposes intake and management, Temporal runs processing, and PostgreSQL stores documents and revisions. Each network selects a revision independently. Qdrant holds the search projection. Full narrative docs (read these before deep architectural changes, don't re-derive): - `README.md` — what the system does, API surface, supported inputs, auth model @@ -53,44 +53,36 @@ Interactive API docs once running: `http://localhost:8001/docs`. ### Everything lives in `pipeline/`, not split into microservices `pipeline/api/` layout: -- `api.py` — the FastAPI app and most routes. `__init__.py` only re-exports `app`. -- `search.py` — `POST /search` as an `APIRouter`. -- `documents.py` — collection-level `/documents` routes (`POST /documents`, `POST /documents/batch`, `GET /documents`, `/summary`, `/cohorts`) as an `APIRouter` with prefix `/documents`; reads shared state (`temporal_client`, `limiter`, helpers) from `api.py` at call time. Per-document `/documents/{workflow_id}` routes stay in `api.py`. -- `upload.py` — `POST /upload` (multipart file + optional `display_name`) as an `APIRouter`; same call-time access to `api.py` state (`minio_client`, `MINIO_BUCKET`, `temporal_client`). Remaining resources move out the same way; see `TODO(api-split)` in `api.py`. +- `api.py` composes the app and exports runtime dependencies. `__init__.py` re-exports `app`. +- `application.py` constructs FastAPI and its middleware. `runtime.py` owns service startup and shutdown. +- `routers/` holds feature handlers: documents and reviews, upload and crawl, search and catalog, provider and Registry management, authentication, and operations. +- `document_helpers.py`, `file_helpers.py`, `audit_helpers.py`, and `index_helpers.py` hold shared handler helpers. +- `request_logging.py` and `rate_limiting.py` configure request diagnostics and limits. Rules: -- Add a new endpoint to its resource's router file if one exists, else near its siblings in `api.py`. +- Add new endpoints to their feature router under `routers/`. Register new routers in `api.py`. - Update `tests/test_api_routes.py` whenever a route is added, removed or renamed. -- Patch module globals (e.g. `temporal_client`) on `pipeline.api.api`, not the package. +- Patch runtime dependencies (for example, `temporal_client`, `object_store`, and startup factories) on `pipeline.api.api`, not the package. Routers and startup read those overrides at call time. +- Shared helper exports remain available on `pipeline.api.api` for existing callers and tests. - Walk routes with `tests/route_inventory.py`, not `app.routes`: FastAPI hides included routers there. -Core modules (all under `pipeline/`): -- `api/` — FastAPI app and routers (see layout above) -- `workflows.py` — Temporal workflow definitions (`DocumentPipelineWorkflow` is the main one; also `ReingestionWorkflow`, `PromoteToProdWorkflow`, `TranslationOnlyWorkflow`, `OcrOnlyWorkflow`, `ChunkingOnlyWorkflow`) -- `activities.py` — Temporal activities invoked by the workflows (OCR, translate, chunk, ingest, state updates) -- `worker.py` — Temporal worker process; registers workflows/activities on task queue `ocr-pipeline` -- `db.py` — Postgres access layer (psycopg3 sync connection pool); Postgres is the **canonical** metadata/review-state store (documents, workflows, pages, chunks, artifacts, audit log, settings), not just a cache. Documents and jobs come back as flat dicts keyed by `workflow_id` (the document's main Temporal workflow id); every function also accepts the document UUID. Schema: `migrations/` (applied at api/worker startup by `migrate.py`), described in `docs/POSTGRES_SCHEMA.md` -- `document_repository.py` — `DocumentRepository`, a domain layer over `db.py` for document reads; `db.py` stays one-function-per-query with no domain knowledge -- `network_constants.py` — fixed values sent on the network (catalog/resource ids, topics, languages, Beckn version). The file v2 edits when the AI layer starts deriving them -- `catalog_builder.py` — pure builder mapping a document's knowledge kind (`advisory`/`scheme`) to the single `OnDemand` Beckn catalog announced for that kind; takes the resolved provider and prefixes catalog/resource ids for a Group; returns `None` for any other kind. No env, no I/O -- `network_provider.py` — pure provider resolution for Publish to Network: one provider per instance — id (overrides, then `{instance}.{network_id}` template) and name — plus each Knowledge Kind's catalog name. `network_group.py` builds its `ProviderNaming` from env and the overrides file; `catalog_builder.py` takes the resolved provider -- `document_validity.py` — pure module owning Document Validity: the period a document's chunks answer searches in, the upload-day start with no end by default, and the clock injection point. Shared by `api/api.py` (stamping the default at upload), `scheme_catalog.py` (validating a reviewer's edit), `activities.py` (stamping chunk payloads) and `vector_store/qdrant_store.py` (filtering at search time). `db.py` stores the dates it is handed and defaults nothing -- `discovery_publish_service.py` — `DiscoveryPublishService`: owns the Publish to Network env vars and the HTTP call. Deliberately Temporal-free -- `models.py` — Pydantic models, including `DocumentStage` enum and `PIPELINE_STAGES` (the stepper-UI stage list) -- `config.py` — `Config` dataclass reading env vars, with defaults -- `observability.py` — bridges root-logger `INFO`+ lines to OTLP when `OTEL_EXPORTER_OTLP_ENDPOINT` is set (ClickHouse/HyperDX); no-op otherwise. See ADR 0007 -- `instances.py` — tenant instance code ↔ human name mapping; `bv` = Bharat Vistaar platform-wide, others are state codes matching Keycloak group paths (`/states/MH/...` → `mh`). A tenant code is an `instances` row (name = code); a document's `instance` is derived through `documents.catalog_id → catalogs.instance_id`, there is no `documents.instance` column -- `scheme_catalog.py` — scheme catalog behind the pull-based `/catalog/v1/*` API, served from the computed `catalog_status` view (one row per document; `catalogs` rows are `(instance, type)`, `documents` carry `code`/`name`/`aliases`) -- `ocr/`, `translation/`, `chunking/`, `vector_store/` — pluggable provider packages, each with a `base.py` protocol/ABC and a `service.py` (or equivalent) that dispatches to the configured provider -- `auth/` — Keycloak JWT validation, group-based tenancy/permissions, email-OTP flow, admin user management +Core packages under `pipeline/` + +- `temporal/workflows/` owns workflow definitions and retry policies. `temporal/activities/` owns the I/O activities. Run `python -m pipeline.worker` to start the worker on `ocr-pipeline`. Preserve registered workflow names, activity names, and patch markers so saved histories can replay. +- `db/` owns PostgreSQL queries. Its public interface remains `pipeline.db`. Add queries to the module that owns the affected record. Documents and jobs return application-shaped records keyed by workflow ID or document UUID. See [the schema guide](docs/POSTGRES_SCHEMA.md). +- `documents/` owns content models, format admission, dates, text cleanup and the document repository. `intake/` receives pushed content, fetches URLs and runs scheduled pulls. `metadata/` owns schema packs, language variants and metadata normalization. +- `network/` owns connection configuration, admission and release policy. Its `adapters/` package owns protocol dispatch for Beckn and other implementations. `network/catalog/` owns catalog projection and its deployment profile. +- `search/` owns passage projection, projection identity and query expansion. `vector_store/` owns Qdrant transport and embedding providers. +- `telemetry/logs.py` exports structured logs to OTLP when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. +- `ocr/`, `translation/`, `chunking/`, `object_storage/` and `vector_store/` contain implementation interfaces and factories. `auth/` validates Keycloak tokens, resolves instance membership, and administers identities. ### Provider pattern -`ocr/`, `translation/`, `chunking/`, `vector_store/` each define a `base.py` protocol and one-or-more concrete implementations selected by env var (e.g. `OCR_PROVIDER`, `TRANSLATION_PROVIDER`, `CHUNKING_PROVIDER`) via a `service.py` factory. Adding a new provider means implementing the base protocol and registering it in that package's service dispatcher — not touching `activities.py` call sites. +`ocr/`, `translation/`, `chunking/`, `vector_store/` each define a `base.py` protocol and one-or-more concrete implementations selected by env var (e.g. `OCR_PROVIDER`, `TRANSLATION_PROVIDER`, `CHUNKING_PROVIDER`) via a `service.py` factory. Adding a new provider means implementing the base protocol and registering it in that package's service dispatcher — without changing Temporal activity call sites. ### Temporal workflow model -`DocumentPipelineWorkflow` pauses at review stages via Temporal **signals** and resumes on operator action (the approve-* API endpoints send the signal). Stage order (`DocumentStage` in `models.py` / `PIPELINE_STAGES`): +`DocumentPipelineWorkflow` pauses at review stages via Temporal **signals** and resumes on operator action (the approve-* API endpoints send the signal). Stage order (`DocumentStage` in `documents/models.py` / `PIPELINE_STAGES`): ``` registered → ocr_processing → ocr_review → translation_processing → translation_review @@ -98,11 +90,11 @@ registered → ocr_processing → ocr_review → translation_processing → tran → approval_for_prod → ingesting_prod → completed (or → failed at any point) ``` -`approval_for_prod`/`ingesting_prod` are **PROD-only stages** (`PROD_ONLY_STAGES` in `models.py`); with `DISABLE_PROD_SETTING=true` a document goes straight from `ingesting` to `completed`. That flag is read only at workflow *start* and passed in as an argument — never read inside a running workflow, since that would break Temporal replay determinism if flipped mid-flight (see the docstring in `instances.py`). +`approval_for_prod`/`ingesting_prod` are **PROD-only stages** (`PROD_ONLY_STAGES` in `documents/models.py`); with `DISABLE_PROD_SETTING=true` a document goes straight from `ingesting` to `completed`. That flag is read only at workflow *start* and passed in as an argument — never read inside a running workflow, since that would break Temporal replay determinism if flipped mid-flight (see the docstring in `network/release_policy.py`). -### DEV vs PROD dual indexing +### Revisions and networks -`ingest_document_from_db` (activity) writes the **DEV** index via `pipeline/vector_store` (`VectorStore` protocol in `base.py`, `QdrantVectorStore` the only implementation). A separate Super-Admin-only approval gate (`approve-prod` / `request-prod-ready` endpoints, `PromoteToProdWorkflow`) triggers `promote_document_to_prod_qdrant`, which promotes into the **PROD** index — a *different* Qdrant collection/deployment (`PROD_VECTOR_DB_*` env vars), reached by constructing a `QdrantVectorStore` directly rather than through the `VectorStore` protocol. DEV and PROD are both Qdrant; they're just two separate indexes, not two backends. +PostgreSQL freezes document content in a revision. Release staging holds destination details. Each document and network has its own release pointer. A managed publication activates that pointer after indexing and a matching network receipt. A legacy approval path remains for deployments that use the older publishing settings. ### Storage responsibilities (don't blur these) @@ -111,9 +103,9 @@ registered → ocr_processing → ocr_review → translation_processing → tran - **Temporal** — orchestration/retries/review-gate signaling only; not a content store. - **Qdrant** — downstream search projections of approved chunk state; never the place to edit content. -### Auth (optional, off by default) +### Authentication -`AUTH_DISABLED=true` by default — no login, a synthetic local admin is used. When enabled (`AUTH_DISABLED=false`), the API validates Keycloak RS256 JWTs against realm JWKS. Preferred model is Keycloak **groups** (`/states/{STATE}/contributor`, `/global/super-admin`); realm roles + `instances` claims are a legacy fallback. `pipeline/auth/` holds `jwt.py` (validation), `groups.py`/`tenancy.py` (state scoping), `permissions.py` (role→permission mapping), `email_otp.py` (OTP login flow), `keycloak_admin.py` (admin API calls). See `docs/architecture-pipeline-rbac.md` for the full role table and login flow before changing anything here. +Keycloak authentication is required by default. `AUTH_DISABLED=true` is limited to the local runtime profile. Keycloak owns people, service accounts, and provider membership. The console uses the configured Keycloak browser flow with PKCE. Use [Keycloak passwordless setup](docs/KEYCLOAK_PASSWORDLESS.md) for console sign-in. ### UI (`ui/`) @@ -121,4 +113,4 @@ React 18 + Vite + Tailwind v4 + Radix UI operator console (not a plain upload fo ### Scripts (`scripts/`) -Operational tools, not one-off hacks — e.g. `list_failed_workflows.py`, `terminate_stuck_workflows.py`, `keycloak_bootstrap_docs_pipeline.py`, `drop_legacy_marqo_columns.py`, `backfill_provider_id.py` (sets `provider_id` on existing DEV/PROD chunks; dry run unless `--apply`), `migrate_sqlite_to_postgres.py`. Check here before writing a new maintenance script — an equivalent likely exists. +Operational tools, not one-off hacks — e.g. `list_failed_workflows.py`, `terminate_stuck_workflows.py`, `keycloak_bootstrap_docs_pipeline.py`, `drop_legacy_marqo_columns.py`, `backfill_provider_id.py` (sets `provider_id` on existing DEV/PROD chunks; dry run unless `--apply`). The legacy migration kit is in the workspace beside the backup. Check here before writing a new maintenance script — an equivalent likely exists. diff --git a/docs/ADR/0006-instances-and-catalogs-model.md b/docs/ADR/0006-instances-and-catalogs-model.md index 91f4190..fca9db7 100644 --- a/docs/ADR/0006-instances-and-catalogs-model.md +++ b/docs/ADR/0006-instances-and-catalogs-model.md @@ -1,5 +1,7 @@ # Providers are instances, catalogs are (instance, type), documents are identified by (catalog, code) +Migrations 0007 and 0008 supersede the document ownership and classification portions of this decision. Documents store their provider and kind directly. The legacy catalog API derives catalog information from those fields. + Context: the first Postgres schema modelled a tenant as a free-text `documents.instance`, a `catalog_entries` table keyed by `(type, code)` with one entry per scheme, and a free-form `document_kind` that had to match the entry's type through a composite foreign key. Review feedback was that the unit published on the network is a provider's catalog, that a provider is a Keycloak group, that each scheme is itself a document, and that database enums and demo flags do not belong in the schema. Decision: @@ -12,7 +14,7 @@ Decision: - Period, uploader and vector index state moved into `documents.metadata`; counts are derived from `pages` and `chunks`. - Database enums are replaced by plain `TEXT`, validated by `pipeline/models.py` (`DocumentStage`, `WorkflowType`, `WorkflowStatus`, `AuditEntityType`). - `audit_logs.instance` is replaced by `audit_logs.instance_id` (snapshot, `ON DELETE SET NULL`). -- No Postgres held the first schema, so `migrations/0001` was rewritten in place instead of adding an ALTER migration. `scripts/migrate_sqlite_to_postgres.py` produces v2 and skips `kind=document` and `is_demo` documents, reporting the counts. +- No Postgres held the first schema, so `migrations/0001` was rewritten in place instead of adding an ALTER migration. The SQLite importer then produced v2 with per-table outcome counts. The current importer lives in the operator kit beside the backup. Alternatives considered: - **Keep `documents.instance`.** Rejected: the catalog already carries the provider, and two sources of tenant truth drift. diff --git a/docs/POSTGRES_SCHEMA.md b/docs/POSTGRES_SCHEMA.md index f9e260e..1c29c2e 100644 --- a/docs/POSTGRES_SCHEMA.md +++ b/docs/POSTGRES_SCHEMA.md @@ -1,18 +1,18 @@ -# Postgres Schema — knowledge-provider +# Knowledge Provider PostgreSQL schema -Postgres schema behind `pipeline/db.py`. A document is content and metadata. A workflow is one run of processing against a document. A document belongs to a catalog (a provider instance plus a resource type) and is identified there by its `code`. Source file and vector index state are stored as JSONB on the document. Translations are stored as JSONB on each page. +PostgreSQL stores provider ownership, documents, processing runs, and network releases. A provider owns each document. The combination of provider, kind, and code identifies it within that provider. Each network independently selects a frozen document revision. The legacy catalog API derives catalog membership from the provider and kind. Documents store source and working index metadata as JSONB, and pages store translations as JSONB. -The schema is created by `migrations/0001_initial_schema.sql` and applied by the api and worker at startup (`pipeline/migrate.py`). `pipeline/db.py` is the only code that reads or writes it. The previous SQLite store is described in `docs/SQLITE_SCHEMA.md` (legacy) and is moved over once by `scripts/migrate_sqlite_to_postgres.py`. +At startup, the API and worker call `pipeline/migrate.py` to create the active schema or apply later numbered migrations. Use the migration kit beside the legacy backup to import SQLite data or apply archived pre-release upgrades. ## Conventions -- Every table has a primary key `id`, a UUID v4 (`DEFAULT gen_random_uuid()`, built in from Postgres 13). +- Entity tables generally use UUID primary keys. Join and checkpoint tables can use keys formed from their owning records. - A foreign key is named `_id` (for example `document_id`). A role prefix is added when the role matters (`uploaded_by_user_id`, `started_by_user_id`). - Text identifiers other systems rely on (`documents.code`, `temporal_workflow_id`) are ordinary columns, not keys. - `created_at` and `updated_at` are the last two columns of a table. `updated_at` is maintained by one shared trigger. Append-only or short-lived tables skip `updated_at`, as noted per table. - Timestamps are `TIMESTAMPTZ` (UTC), booleans are `BOOLEAN`, JSON is `JSONB`, lists of strings are `TEXT[]`. - JSONB columns have a `CHECK` that the value is an object (or array). Value rules inside them live in Pydantic models, not in the database. -- There are no database enums. Every value set (stages, workflow types and statuses, audit entity types) is plain `TEXT`, validated in code by `pipeline/models.py`; open sets (`artifact_type`, `action_type`) are `TEXT` too. +- There are no database enums. `pipeline/documents/models.py` validates the plain `TEXT` value sets for stages, workflow types and statuses, and audit entity types. Open sets such as `artifact_type` and `action_type` also use `TEXT`. - Counts have `CHECK >= 0`. Slugs and identifiers have pattern `CHECK`s. ### Shared trigger @@ -30,7 +30,7 @@ END $$ LANGUAGE plpgsql; ### Value sets (plain `TEXT`, validated in code) -| Column(s) | Python type (`pipeline/models.py`) | Values | +| Column(s) | Python type (`pipeline/documents/models.py`) | Values | |---|---|---| | `workflows.current_stage`, `workflow_artifacts.stage` | `DocumentStage` | `registered`, `ocr_processing`, `ocr_review`, `translation_processing`, `translation_review`, `chunking`, `chunk_review`, `ready_for_ingestion`, `ingesting`, `publishing_to_network`, `approval_for_prod`, `ingesting_prod`, `completed`, `failed` | | `workflows.workflow_type` | `WorkflowType` | `pipeline`, `ocr_only`, `ocr_retry`, `translation_retry`, `chunking_retry`, `reingest`, `promote_prod` | @@ -41,36 +41,33 @@ The database does not check these values, so adding one needs no migration. Raw ## Tables -| # | Table | Purpose | +The grouped baseline creates 25 application tables plus its migration ledger. Keycloak and Temporal use their own databases on the same PostgreSQL server. Startup applies [the shared extension and trigger](../migrations/bootstrap/00_shared.sql) followed by the domain files in the order shown below. The migration kit owns `legacy_import_evidence` and creates it only when it imports old data. `webhook_records` and `email_otps` are absent from a clean install. + +| Bootstrap file | Tables | Purpose | |---|---|---| -| 1 | `instances` | Provider tenant nodes (`/states/MH`, `/global`), mirrored from Keycloak | -| 2 | `users` | People who upload, review and act, mirrored from Keycloak | -| 3 | `catalogs` | One row per provider instance and resource type (scheme, advisory, video) | -| 4 | `documents` | One row per document. Content, metadata, source file, index state | -| 5 | `workflows` | One row per processing run of a document. Stage, status, config | -| 6 | `workflow_artifacts` | Files each workflow produces, stored in MinIO | -| 7 | `pages` | OCR layer and translations of each page | -| 8 | `chunks` | Chunked text and review state | -| 9 | `audit_logs` | Append-only action history | -| 10 | `search_settings` | Single-row search configuration | -| 11 | `email_otps` | Email OTP login codes | -| 12 | `webhook_records` | Records pushed to `POST /webhook` | -| — | `schema_migrations` | Applied migration files (created by the runner, not by a migration) | -| — | `catalog_status` (view) | One row per document with its catalog status. `/catalog/v1/*` serves the scheme type | -| — | `document_current_workflow` (view) | Latest workflow of each document | - -Relationships: +| [Instances and access](../migrations/bootstrap/10_instances_access.sql) | `instances`, `users`, `email_sign_in_limits` | Ownership, actor attribution and email send cooldown. Keycloak owns credentials and group membership. | +| [Documents](../migrations/bootstrap/20_documents_processing.sql) | `documents`, `pages`, `chunks`, `document_revisions` | Editable content and frozen snapshots. | +| [Workflows](../migrations/bootstrap/25_workflows.sql) | `workflows`, `workflow_artifacts`, `document_current_workflow` | Processing history, object references and the revision-to-workflow foreign key. | +| [Intake and source evidence](../migrations/bootstrap/30_intake_source_evidence.sql) | `provider_source_settings`, `webhook_sources`, `webhook_mapping_versions`, `webhook_deliveries`, `source_intake_receipts`, `scheduled_source_pulls` | Provider settings, pinned mappings, delivery and document evidence, and restartable cursors. | +| [Search, audit and catalog](../migrations/bootstrap/40_search_audit_catalog.sql) | `audit_logs`, `search_settings`, `catalogs` | Action history, shared search configuration and catalog identifiers used by document reads and catalog routes. | +| [Network and Registry](../migrations/bootstrap/50_network_registry.sql) | `network_release_staging`, `network_releases`, `publication_deliveries`, `schema_pack_versions`, `registry_operator_subjects`, `registry_operator_proposals`, `registry_operator_decisions` | Release state, delivery receipts, schema bundles and proposal decisions. | +| Migration ledger | `schema_migrations` | Baseline and later applied migrations. | + +The views `catalog_status`, `document_current_workflow`, and `legacy_document_catalog_types` store no rows of their own. + +Document ownership and processing relationships ``` instances 1 ── * users (users.instance_id, nullable) -instances 1 ── * catalogs 1 ── * documents 1 ── * workflows 1 ── * workflow_artifacts +instances 1 ── * documents 1 ── * workflows 1 ── * workflow_artifacts │ ├── * pages └── * chunks instances 1 ── * audit_logs (audit_logs.instance_id, snapshot, SET NULL) +instances 1 ── * catalogs (legacy compatibility) ``` -Dependency order for creation: `instances`, `users`, `catalogs`, `documents`, `workflows`, `workflow_artifacts`, `pages`, `chunks`, `audit_logs`, `search_settings`, `email_otps`, `webhook_records`, views. +Release staging, source receipts, and scheduled cursors survive worker restarts. A source receipt holds replay evidence. The worker commits `last_page_policy_conflicts` with the scheduled page cursor and receipts. --- @@ -148,7 +145,7 @@ Unique: `(instance_id, type)`. Pydantic (`CatalogMetadata`) validates the keys; the database only checks for an object. Notes -- `type` stores the schema name, and code asks `pipeline/network_constants.py` what it means (`catalog_type_for_kind`, `kind_for_catalog_type`) instead of comparing `openagrinet:*` strings. The mapping is a placeholder: `scheme` is `openagrinet:AgricultureResource`, `advisory` is `openagrinet:KnowledgeAdvisory`, `video` is `openagrinet:VideoResource` (see ADR 0003 and ADR 0006). +- `type` stores the schema name. `pipeline/network/network_constants.py` defines `catalog_type_for_kind` and `kind_for_catalog_type`. The mappings are `scheme` to `openagrinet:AgricultureResource`, `advisory` to `openagrinet:KnowledgeAdvisory`, and `video` to `openagrinet:VideoResource`. See ADR 0003 and ADR 0006. - Every instance gets one catalog per supported type when the instance row is created (`upsert_instance` calls `ensure_instance_catalogs` in the same transaction). Startup (`init_db`) runs `sync_catalogs_for_all_instances`, so a type added to `KIND_TO_CATALOG_TYPE` reaches existing instances. Upload keeps `get_or_create_catalog` as a fallback for a catalog that is missing. Names are ` catalog`, for example `MH scheme catalog`. - Computed catalog values (status, chunk count, network visibility, `promoted_at`) come from the `catalog_status` view, not stored here. @@ -156,15 +153,16 @@ Notes ## 4. `documents` -One row per document: one resource (a scheme, an advisory) in one catalog, identified there by `code`. +One row per document, classified by its own kind and owned by a provider. -Unique: `(catalog_id, code)`. +Unique: `(provider_id, kind, code)`. | Column | Type | Constraints / Default | Purpose | |---|---|---|---| | id | UUID | PRIMARY KEY, DEFAULT gen_random_uuid() | Document ID. | -| catalog_id | UUID | NOT NULL, FK → `catalogs(id)` ON DELETE RESTRICT | Catalog (provider instance + type) the document belongs to. | -| code | TEXT | NOT NULL, CHECK `^[a-z0-9]+(-[a-z0-9]+)*$` | Short code (`pm-kmy`) that catalog consumers see. Unique within the catalog. | +| provider_id | UUID | NOT NULL, FK → `instances(id)` ON DELETE RESTRICT | Provider that owns the document. | +| kind | TEXT | NOT NULL, nonblank | Document classification. Legacy catalog IDs are derived for compatibility. | +| code | TEXT | NOT NULL, CHECK `^[a-z0-9]+(-[a-z0-9]+)*$` | Short code. Unique within its provider and kind. | | name | TEXT | NOT NULL | Display name. | | aliases | TEXT[] | NOT NULL DEFAULT '{}', CHECK no empty string | Ordered alternate names used to match queries. The first one is the prompt hint. | | source | JSONB | NOT NULL, CHECK object, CHECK `source->>'fingerprint' ~ '^[0-9a-f]{32}$'` | The uploaded file. See shape below. | @@ -173,7 +171,7 @@ Unique: `(catalog_id, code)`. | created_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | Row created. | | updated_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | Last change (trigger). | -Indexes: `(created_at DESC)`, `(catalog_id, (source->>'fingerprint'))`, plus the unique `(catalog_id, code)`. +Indexes include `(created_at DESC)`, `(provider_id, (source->>'fingerprint'))`, and the unique key `(provider_id, kind, code)`. ### `source` @@ -231,12 +229,11 @@ Column default: `'{"index_status": {}, "catalog_visible": true, "network_visible ### Notes - A document's stage is the `current_stage` of its latest workflow. List and filter queries join the `document_current_workflow` view. -- The API's `instance` is derived: `documents.catalog_id → catalogs.instance_id → instances.name`. There is no `documents.instance` column, and tenant filters join `instances`. -- The API's `document_kind` is derived from `catalogs.type` through the map in `network_constants.py`. `scheme_code`, `scheme_name` and `scheme_aliases` are `code`, `name` and `aliases`. +- The API's `instance` comes from `documents.provider_id → instances.name`. Migration 0007 backfills this reference. Migration 0008 backfills the kind and removes the catalog foreign key. +- The document's kind is stored in `documents.kind`. The legacy catalog view maps known kinds to their catalog types. `scheme_code`, `scheme_name` and `scheme_aliases` are `code`, `name` and `aliases`. - `chunk_count` and `page_count` are counted from `chunks` and `pages`; they are not stored. -- A new document needs a `document_kind` and a `code`. Upload gets the catalog from the tenant and the kind (seeded with the instance; created when missing). A document's kind is fixed after upload, so there is no kind change. `DuplicateDocumentCode` is raised when the code is taken in that catalog. +- A new document needs a `document_kind` and a `code`. A document's kind is fixed after upload. `DuplicateDocumentCode` is raised when that provider already uses the code for that kind. - Re-uploading an existing code updates the same document: a new workflow starts, `source` is replaced, pages and chunks are cleared and `reindex_required` is set (`reset_document_for_reupload`). Earlier workflows and artifacts stay as history. -- Changing a document's kind moves it to the same instance's catalog of the new type (`update_document_fields(document_kind=...)`). - SQL does not check types inside the JSONB columns. The insert path writes the default metadata keys, and Pydantic validates `source`, `metadata` and each `index_status` entry. - Each document keeps its own `source`. There is no unique key on the fingerprint. - Identifier lookup (`find_document_by_doc_identifier`) matches `documents.id`, `workflows.temporal_workflow_id`, `source->>'fingerprint'`, `source->>'original_filename'`, an index `doc_id` and the legacy `md5(source->>'fingerprint')`. @@ -483,28 +480,11 @@ Notes --- -## 11. `email_otps` - -Server-side email-OTP login codes. Short-lived. - -Unique: `email`; index on `(expires_at)`. +## 11. Retired email challenges -| Column | Type | Constraints / Default | Purpose | -|---|---|---|---| -| id | UUID | PRIMARY KEY, DEFAULT gen_random_uuid() | Row ID. | -| email | TEXT | NOT NULL UNIQUE, CHECK `email = lower(email)` | Address the code was requested for. One live code per email. | -| code_hash | TEXT | NOT NULL | Hash of the one-time code. The plain code is never stored. Random for unknown accounts. | -| expires_at | TIMESTAMPTZ | NOT NULL | When the code stops being valid. | -| attempts | INTEGER | NOT NULL DEFAULT 0, CHECK ≥ 0 | Failed verifications. Reset to 0 on resend. | -| last_sent_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | When the code was last sent. Used for the resend cooldown. | +Migration `0015` retires the legacy `email_otps` table. It waits for outstanding challenges to expire by refusing the migration while any unexpired row remains. The SQLite importer counts old challenges without copying their hashes. Keycloak authenticates console users. Migration `0018` adds `email_sign_in_limits` for the one-minute send cooldown. It stores an HMAC of the normalized address and the next allowed send time. The API removes expired rows as it handles new requests. Keycloak holds the one-time code and attempt counter. -Notes -- No `created_at` or `updated_at`: rows are overwritten on each send. -- `email` is deliberately not an FK to `users`. A row is written for every request, even for unknown addresses (anti-enumeration, `email_otp.py:276-310`). -- Periodic cleanup (`DELETE FROM email_otps WHERE expires_at < now() - interval '1 day'`) is not scheduled yet. The endpoint is unauthenticated, so unverified rows stay until it is. -- `verify_otp` increments attempts atomically: `UPDATE ... SET attempts = attempts + 1 WHERE email = $1 RETURNING attempts`, or `SELECT ... FOR UPDATE`. -- Code constants (length 6, TTL 300 s, resend cooldown 30 s, max 5 attempts) stay in code. -- Optional: make the table `UNLOGGED`. Losing it in a crash only forces users to request a new code. +The same migration moves the first raw JSON item observed on legacy crawl replay into four optional columns on `source_intake_receipts`. It preserves the original payload, receipt identity, digest, and observation time. Stop older API and worker processes before upgrading their schema. --- @@ -515,8 +495,8 @@ Computed from `documents`, `catalogs`, `instances`, `workflows` and `document_cu | Column | Type | How it is computed | |---|---|---| | document_id | UUID | `documents.id`. | -| catalog_id | UUID | `documents.catalog_id`. | -| instance_id | UUID | `catalogs.instance_id`. | +| catalog_id | UUID | Matching legacy catalog, if present. | +| instance_id | UUID | `documents.provider_id`. | | instance_name | TEXT | `instances.name`. | | type | TEXT | `catalogs.type`. | | code | TEXT | `documents.code`. | @@ -532,44 +512,41 @@ Computed from `documents`, `catalogs`, `instances`, `workflows` and `document_cu --- -## 12. `webhook_records` +## Retired shared-key advisory intake -Records pushed to `POST /webhook` (`x-api-key` = `WEBHOOK_API_KEY`). Carried over from the previous store unchanged; not part of the instances and catalogs model. +The migration kit archives old webhook rows in `legacy_import_evidence`. Its archived PostgreSQL upgrade `0022` retains the full original row, keyed by source and external ID, and removes `webhook_records`. The SQLite importer also preserves raw legacy rows in the archive. Current webhook processing uses `webhook_sources`, `webhook_mapping_versions`, `webhook_deliveries`, and `source_intake_receipts` for delivery and document history. -Unique: `(source, external_id)`; index on `(source, status)`. - -| Column | Type | Constraints / Default | Purpose | -|---|---|---|---| -| id | UUID | PRIMARY KEY, DEFAULT gen_random_uuid() | Record ID. | -| source | TEXT | NOT NULL DEFAULT 'webhook' | Sender. | -| external_id | TEXT | NOT NULL | `contentID`, else `id`, else `slug`, else a hash of the payload. | -| title | TEXT | | From the payload. | -| slug | TEXT | | From the payload. | -| status | TEXT | | From the payload. | -| is_published | BOOLEAN | | From `isPublished`. | -| category | TEXT | | From `domainName`. | -| payload | JSONB | NOT NULL, CHECK object | The full record. | -| payload_hash | TEXT | | SHA-256 of the canonical payload; an upsert with the same hash writes nothing. | -| indexed_hash | TEXT | | `payload_hash` last written to the DEV collection (migration 0002). | -| promoted_hash | TEXT | | `payload_hash` last written to the PROD collection (migration 0002). | -| created_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | The payload's `createdAt` when it has one. | -| updated_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | The payload's `updatedAt` when it has one. | - -The payload's own timestamps are stored as given, so these two columns are not maintained by the shared trigger. +New deliveries keep their exact request bytes in object storage. The nullable `webhook_deliveries.raw_storage_uri` points to that object. Older rows retain the parsed JSON, which cannot recover the original formatting or key order. --- +## Revisions and network releases + +`document_revisions` freezes the content and its checksum. `network_release_staging` associates a revision with a network, collection, provider identity, and the release generation observed when indexing began. The release worker compares that generation before changing the active pointer in `network_releases`. Search checks the active pointer and excludes staged, superseded, and withdrawn revisions. + +The worker records a delivery intent under the network and transaction ID in `publication_deliveries`. It stores the publisher's receipt before writing the exchange artifact or activating the revision. A retry reuses a recorded receipt and the exchange artifact ID. If the worker loses the response before saving it, a retry may send another request with the same transaction ID. + +## Schema pack versions + +`schema_pack_versions` stores each approved bundle under its network ID, capability, and SHA-256 digest. Installing a new digest keeps earlier versions available. The upload and structured intake paths validate supplied pack attributes against the selected digest. The revision snapshot retains the selection and attributes for later validation. + +Prepare a local JSON or YAML manifest with `capability`, `version`, `entrypoint`, `component`, and `files`. The `files` object maps each schema's HTTPS identifier to a path under the manifest directory, including every referenced dependency. The validator resolves references from those files without fetching remote schemas. + +Run `uv run python -m scripts.install_schema_pack manifest.json` to print the checked digest. Add `--install-for-network NETWORK_ID` to register the bundle in PostgreSQL. With `--verify-registry`, the command also checks the capability, version, and entrypoint against the configured network's active Registry entry. Supply the resulting `network_id`, `capability`, `digest`, and `attributes` in the `schema_pack` object inside `knowledge_metadata`. + +A Beckn catalog profile can also pin its outbound schema. Add a `schema_pack` object containing `capability` and `digest` to the kind's projection. For managed publication, the worker checks the catalog's `@type` and validates its resource attributes against that pack on the destination network before sending the catalog. + ## `schema_migrations` -Created by the runner, not by a migration file: `version` (the `NNNN` prefix, primary key), `name` (the file name) and `applied_at`. The runner takes a session advisory lock, so concurrent api and worker starts apply each file once, and runs each file in its own transaction. +The runner creates the ledger with `version` (the `NNNN` prefix and primary key), `name`, and `applied_at`. It installs all six bootstrap files and records one `0022` baseline marker in the same transaction. A pre-release database already at `0022` keeps its existing ledger. The runner takes a session advisory lock across startup and commits each later migration in its own transaction. --- ## Implementation notes -- **Identifiers.** `db.py` functions take a `workflow_id` argument that may be the document UUID, a Temporal workflow id, the fingerprint, the filename, an index `doc_id` or the legacy `md5(fingerprint)` doc id (`find_document_by_doc_identifier`). Documents, jobs and artifacts come back as flat dicts; `id` is the UUID and `workflow_id` is the Temporal workflow id of the document's main run (the earliest `pipeline` or `ocr_only` workflow). The API rewrites `/documents/{id}/...` to that workflow id before routing, because handlers signal and query Temporal with it. +- **Identifiers.** `pipeline.db` accepts the document UUID, Temporal workflow ID, fingerprint, filename, index `doc_id` or legacy `md5(fingerprint)` doc ID through `find_document_by_doc_identifier`. Documents, jobs and artifacts come back as flat dicts. `id` is the UUID and `workflow_id` is the Temporal workflow ID of the document's main run, the earliest `pipeline` or `ocr_only` workflow. The API rewrites `/documents/{id}/...` to the workflow ID before routing because handlers use it to signal and query Temporal. - **Jobs.** The API's "jobs" and "runs" are `workflows` rows; job ids and artifact ids are the UUIDs of `workflows` and `workflow_artifacts`. `latest_job_id`, `original_artifact_id` and `normalized_artifact_id` are derived by join, not stored. -- **Instance.** The API returns an `instance` string on documents, runs, queue rows and audit entries. It is the lowercase `instances.name` reached through `catalogs`; filters compare `lower(instances.name)`. +- **Instance.** The API returns an `instance` string on documents, runs, queue rows and audit entries. Document queries join `documents.provider_id` to `instances.id` and compare `lower(instances.name)`. - **Catalog version.** `/catalog/v1/version` counts `audit_logs` rows with `entity_type = 'catalog'` and `action_type = 'catalog_version_bump'`, so it only grows. `updated_at` is the newest such row. - **Document stage.** The stage a document shows is `document_current_workflow.current_stage`; `update_document_stage` writes to that workflow. - **Page translations.** `save_pages` keeps stored translations unless the page carries `translations` (replaces them) or flat `translated_*` fields (merged into `translation_target_language`). A page whose OCR text changed loses its translations. diff --git a/docs/SQLITE_SCHEMA.md b/docs/SQLITE_SCHEMA.md index c3fa539..e75637c 100644 --- a/docs/SQLITE_SCHEMA.md +++ b/docs/SQLITE_SCHEMA.md @@ -1,6 +1,6 @@ # SQLite Schema — knowledge-provider (legacy) -> Legacy. The store is now Postgres, described in `docs/POSTGRES_SCHEMA.md`. This file documents the shape of the old SQLite file that `scripts/migrate_sqlite_to_postgres.py` reads. +> Legacy. The store is now Postgres, described in `docs/POSTGRES_SCHEMA.md`. This file documents the shape of the old SQLite file that the operator migration kit reads. ## Overview diff --git a/docs/SYSTEM_DESIGN.md b/docs/SYSTEM_DESIGN.md index 7aa88e4..a203634 100644 --- a/docs/SYSTEM_DESIGN.md +++ b/docs/SYSTEM_DESIGN.md @@ -255,7 +255,8 @@ Supporting workflows: `OcrOnly`, `TranslationOnly`, `ChunkingOnly`, ```text documents (id UUID PK) - ├── catalog_id # → catalogs.instance_id → instances.name + ├── provider_id # → instances.id (tenant ownership) + ├── kind # Document classification ├── source (JSONB) # fingerprint, filename, storage_uri, page_count ├── index_status (JSONB) # dev / prod vector index state ├── metadata (JSONB) # chunk_count, reindex_required, is_demo, ... diff --git a/docs/architecture-pipeline-rbac.md b/docs/architecture-pipeline-rbac.md index 8721c8e..a74bf73 100644 --- a/docs/architecture-pipeline-rbac.md +++ b/docs/architecture-pipeline-rbac.md @@ -220,7 +220,7 @@ stateDiagram-v2 | **Approval for Prod** (queue) | Document waits | Still **DEV-only** in search until promote | | **Approve for Prod** | **Super Admin only** (`admin` / `RequireAdmin`) | `promote_document_to_prod_qdrant` → **Qdrant PROD** → `completed` | -Signals (workflow): `approve_ocr`, `approve_translation`, `approve_chunks`, `approve_ingestion`, `approve_prod` — see `pipeline/workflows.py`. +Signals (workflow): `approve_ocr`, `approve_translation`, `approve_chunks`, `approve_ingestion`, `approve_prod` — see `pipeline/temporal/workflows/`. API (examples): @@ -400,10 +400,10 @@ flowchart TB | Concern | Location | |---|---| -| Stages enum | `pipeline/models.py` → `DocumentStage` | -| Workflow + signals | `pipeline/workflows.py` | -| DEV ingest / PROD promote activities | `pipeline/activities.py` | -| Approve prod API | `pipeline/api/api.py` → `POST .../approve-prod` | +| Stages enum | `pipeline/documents/models.py` → `DocumentStage` | +| Workflow + signals | `pipeline/temporal/workflows/` | +| DEV ingest / PROD promote activities | `pipeline/temporal/activities/` | +| Approve prod API | `pipeline/api/routers/document_review.py` → `POST .../approve-prod` | | Permissions | `pipeline/auth/permissions.py` | | Groups / super_admin | `pipeline/auth/groups.py` | | Tenancy | `pipeline/auth/tenancy.py` | diff --git a/migrations/0001_initial_schema.sql b/migrations/0001_initial_schema.sql deleted file mode 100644 index b20e900..0000000 --- a/migrations/0001_initial_schema.sql +++ /dev/null @@ -1,262 +0,0 @@ -CREATE EXTENSION IF NOT EXISTS pg_trgm; - -CREATE FUNCTION set_updated_at() RETURNS trigger AS $$ -BEGIN - NEW.updated_at := now(); - RETURN NEW; -END $$ LANGUAGE plpgsql; - --- 1. instances: one row per tenant node (provider), mirrored from Keycloak -CREATE TABLE instances ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - keycloak_group_id TEXT UNIQUE, - name TEXT NOT NULL, - path TEXT NOT NULL UNIQUE, - created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT now() -); -CREATE TRIGGER trg_instances_updated_at BEFORE UPDATE ON instances - FOR EACH ROW EXECUTE FUNCTION set_updated_at(); - --- 2. users -CREATE TABLE users ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - keycloak_user_id TEXT NOT NULL UNIQUE, - username TEXT, - email TEXT, - roles TEXT[] NOT NULL DEFAULT '{}', - instance_id UUID REFERENCES instances(id) ON DELETE SET NULL, - first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), - last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now() -); -CREATE UNIQUE INDEX users_email_lower_key ON users (lower(email)) WHERE email IS NOT NULL; - -INSERT INTO users (keycloak_user_id, username, roles) -VALUES ('local-admin', 'local-admin', '{superadmin}'); - --- 3. catalogs: one row per provider instance and resource type -CREATE TABLE catalogs ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - instance_id UUID NOT NULL REFERENCES instances(id) ON DELETE RESTRICT, - type TEXT NOT NULL CHECK (type ~ '^[a-z0-9]+:[A-Za-z0-9]+$'), - name TEXT NOT NULL, - metadata JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(metadata) = 'object'), - created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), - UNIQUE (instance_id, type) -); -CREATE TRIGGER trg_catalogs_updated_at BEFORE UPDATE ON catalogs - FOR EACH ROW EXECUTE FUNCTION set_updated_at(); - --- 4. documents -CREATE TABLE documents ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - catalog_id UUID NOT NULL REFERENCES catalogs(id) ON DELETE RESTRICT, - code TEXT NOT NULL CHECK (code ~ '^[a-z0-9]+(-[a-z0-9]+)*$'), - name TEXT NOT NULL, - aliases TEXT[] NOT NULL DEFAULT '{}' CHECK (array_position(aliases, '') IS NULL), - source JSONB NOT NULL - CHECK (jsonb_typeof(source) = 'object') - CHECK (source->>'fingerprint' ~ '^[0-9a-f]{32}$'), - is_disabled BOOLEAN NOT NULL DEFAULT FALSE, - metadata JSONB NOT NULL - DEFAULT '{"index_status": {}, "catalog_visible": true, "network_visible": true, "reindex_required": false}' - CHECK (jsonb_typeof(metadata) = 'object'), - created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), - UNIQUE (catalog_id, code) -); -CREATE INDEX documents_created_at_idx ON documents (created_at DESC); -CREATE INDEX documents_fingerprint_idx ON documents (catalog_id, (source->>'fingerprint')); -CREATE TRIGGER trg_documents_updated_at BEFORE UPDATE ON documents - FOR EACH ROW EXECUTE FUNCTION set_updated_at(); - --- 5. workflows -CREATE TABLE workflows ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE, - workflow_type TEXT NOT NULL, - temporal_workflow_id TEXT NOT NULL, - status TEXT NOT NULL DEFAULT 'running', - current_stage TEXT NOT NULL DEFAULT 'registered', - stage_timestamps JSONB NOT NULL DEFAULT '{}' - CHECK (jsonb_typeof(stage_timestamps) = 'object' - AND (stage_timestamps - ARRAY['ocr', 'translation', 'chunks']) = '{}'::jsonb), - stop_after_ocr BOOLEAN NOT NULL DEFAULT FALSE, - error_message TEXT, - config JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(config) = 'object'), - started_by_user_id UUID REFERENCES users(id) ON DELETE SET NULL, - started_at TIMESTAMPTZ NOT NULL DEFAULT now(), - completed_at TIMESTAMPTZ, - updated_at TIMESTAMPTZ NOT NULL DEFAULT now() -); -CREATE INDEX workflows_document_started_idx ON workflows (document_id, started_at DESC); -CREATE INDEX workflows_status_started_idx ON workflows (status, started_at DESC); -CREATE INDEX workflows_temporal_id_idx ON workflows (temporal_workflow_id); -CREATE TRIGGER trg_workflows_updated_at BEFORE UPDATE ON workflows - FOR EACH ROW EXECUTE FUNCTION set_updated_at(); - -CREATE VIEW document_current_workflow AS -SELECT DISTINCT ON (document_id) * -FROM workflows -ORDER BY document_id, started_at DESC, id DESC; - --- 6. workflow_artifacts -CREATE TABLE workflow_artifacts ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - workflow_id UUID NOT NULL REFERENCES workflows(id) ON DELETE CASCADE, - artifact_type TEXT NOT NULL CHECK (btrim(artifact_type) <> ''), - stage TEXT, - storage_uri TEXT NOT NULL, - mime_type TEXT, - filename TEXT, - size_bytes BIGINT CHECK (size_bytes >= 0), - metadata JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(metadata) = 'object'), - created_at TIMESTAMPTZ NOT NULL DEFAULT now() -); -CREATE INDEX workflow_artifacts_workflow_idx ON workflow_artifacts (workflow_id, created_at DESC); -CREATE INDEX workflow_artifacts_type_idx ON workflow_artifacts (workflow_id, artifact_type, created_at DESC); - --- 7. pages -CREATE TABLE pages ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE, - page_number INTEGER NOT NULL CHECK (page_number >= 1), - original_markdown TEXT, - edited_markdown TEXT, - translations JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(translations) = 'object'), - metadata JSONB NOT NULL DEFAULT '{"is_reviewed": false}' CHECK (jsonb_typeof(metadata) = 'object'), - created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), - UNIQUE (document_id, page_number) -); -CREATE TRIGGER trg_pages_updated_at BEFORE UPDATE ON pages - FOR EACH ROW EXECUTE FUNCTION set_updated_at(); - --- 8. chunks -CREATE TABLE chunks ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE, - chunk_number INTEGER NOT NULL CHECK (chunk_number >= 1), - original_text TEXT NOT NULL, - edited_text TEXT, - token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0), - page_start INTEGER NOT NULL DEFAULT 1 CHECK (page_start >= 1), - page_end INTEGER NOT NULL DEFAULT 1, - source_spans JSONB NOT NULL DEFAULT '[]' CHECK (jsonb_typeof(source_spans) = 'array'), - section_title TEXT, - content_type TEXT, - is_reference BOOLEAN NOT NULL DEFAULT FALSE, - is_reviewed BOOLEAN NOT NULL DEFAULT FALSE, - is_excluded BOOLEAN NOT NULL DEFAULT FALSE, - reviewer_notes TEXT, - created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), - UNIQUE (document_id, chunk_number), - CHECK (page_end >= page_start) -); -CREATE INDEX chunks_text_trgm_idx ON chunks - USING gin (lower(coalesce(edited_text, original_text)) gin_trgm_ops); -CREATE TRIGGER trg_chunks_updated_at BEFORE UPDATE ON chunks - FOR EACH ROW EXECUTE FUNCTION set_updated_at(); - --- 9. audit_logs -CREATE TABLE audit_logs ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - document_id UUID REFERENCES documents(id) ON DELETE SET NULL, - workflow_id TEXT, - instance_id UUID REFERENCES instances(id) ON DELETE SET NULL, - action_type TEXT NOT NULL CHECK (btrim(action_type) <> ''), - entity_type TEXT, - entity_number INTEGER, - field_name TEXT, - old_value TEXT, - new_value TEXT, - metadata JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(metadata) = 'object'), - actor_user_id UUID REFERENCES users(id) ON DELETE SET NULL, - actor JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(actor) = 'object'), - created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - CHECK (entity_type IN ('setting', 'catalog') OR workflow_id IS NOT NULL) -); -CREATE INDEX audit_logs_document_idx ON audit_logs (document_id, created_at DESC); -CREATE INDEX audit_logs_instance_idx ON audit_logs (instance_id, created_at DESC); -CREATE INDEX audit_logs_created_idx ON audit_logs (created_at DESC); -CREATE INDEX audit_logs_action_idx ON audit_logs (action_type, created_at DESC); -CREATE INDEX audit_logs_settings_idx ON audit_logs (created_at DESC) WHERE entity_type = 'setting'; - --- 10. search_settings -CREATE TABLE search_settings ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - singleton BOOLEAN NOT NULL DEFAULT TRUE UNIQUE CHECK (singleton), - retrieval JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(retrieval) = 'object'), - candidates JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(candidates) = 'object'), - results JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(results) = 'object'), - query JSONB NOT NULL DEFAULT '{}' CHECK (jsonb_typeof(query) = 'object'), - created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT now() -); -CREATE TRIGGER trg_search_settings_updated_at BEFORE UPDATE ON search_settings - FOR EACH ROW EXECUTE FUNCTION set_updated_at(); -INSERT INTO search_settings DEFAULT VALUES; - --- 11. email_otps -CREATE TABLE email_otps ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - email TEXT NOT NULL UNIQUE CHECK (email = lower(email)), - code_hash TEXT NOT NULL, - expires_at TIMESTAMPTZ NOT NULL, - attempts INTEGER NOT NULL DEFAULT 0 CHECK (attempts >= 0), - last_sent_at TIMESTAMPTZ NOT NULL DEFAULT now() -); -CREATE INDEX email_otps_expires_idx ON email_otps (expires_at); - --- webhook_records -CREATE TABLE webhook_records ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - source TEXT NOT NULL DEFAULT 'webhook', - external_id TEXT NOT NULL, - title TEXT, - slug TEXT, - status TEXT, - is_published BOOLEAN, - category TEXT, - payload JSONB NOT NULL CHECK (jsonb_typeof(payload) = 'object'), - payload_hash TEXT, - created_at TIMESTAMPTZ NOT NULL DEFAULT now(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), - UNIQUE (source, external_id) -); -CREATE INDEX webhook_records_source_status_idx ON webhook_records (source, status); - --- catalog_status view: one row per document, computed from catalogs, documents and workflows -CREATE VIEW catalog_status AS -SELECT - d.id AS document_id, - c.id AS catalog_id, - c.instance_id, - g.name AS instance_name, - c.type, - d.code, - d.name, - d.aliases, - CASE - WHEN x.is_live AND COALESCE((d.metadata->>'reindex_required')::boolean, FALSE) THEN 'pending_reindex' - WHEN x.is_live THEN 'live' - WHEN cw.current_stage = 'approval_for_prod' AND NOT d.is_disabled THEN 'pending_prod' - ELSE 'disabled' - END AS status, - (SELECT w.temporal_workflow_id FROM workflows w - WHERE w.document_id = d.id AND w.workflow_type = 'pipeline' - ORDER BY w.started_at ASC, w.id ASC LIMIT 1) AS workflow_id, - (SELECT count(*) FROM chunks ch WHERE ch.document_id = d.id)::INTEGER AS chunk_count, - COALESCE((d.metadata->>'network_visible')::boolean, TRUE) AS network_visible, - CASE WHEN x.is_live - THEN (d.metadata->'index_status'->'prod'->>'last_indexed_at')::timestamptz END AS promoted_at -FROM documents d -JOIN catalogs c ON c.id = d.catalog_id -JOIN instances g ON g.id = c.instance_id -LEFT JOIN document_current_workflow cw ON cw.document_id = d.id -CROSS JOIN LATERAL ( - SELECT (cw.current_stage = 'completed' AND NOT d.is_disabled - AND COALESCE((d.metadata->>'catalog_visible')::boolean, TRUE)) AS is_live -) x; diff --git a/migrations/0002_webhook_advisory_index_state.sql b/migrations/0002_webhook_advisory_index_state.sql deleted file mode 100644 index 328fc93..0000000 --- a/migrations/0002_webhook_advisory_index_state.sql +++ /dev/null @@ -1,3 +0,0 @@ --- Advisory indexing state: the payload_hash last written to the dev / PROD Qdrant collection. -ALTER TABLE webhook_records ADD COLUMN indexed_hash TEXT; -ALTER TABLE webhook_records ADD COLUMN promoted_hash TEXT; diff --git a/migrations/bootstrap/00_shared.sql b/migrations/bootstrap/00_shared.sql new file mode 100644 index 0000000..866b2fc --- /dev/null +++ b/migrations/bootstrap/00_shared.sql @@ -0,0 +1,13 @@ +-- Shared extension and update timestamp trigger. + +CREATE EXTENSION IF NOT EXISTS pg_trgm WITH SCHEMA public; + +COMMENT ON EXTENSION pg_trgm IS 'text similarity measurement and index searching based on trigrams'; + +CREATE FUNCTION public.set_updated_at() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + NEW.updated_at := now(); + RETURN NEW; +END $$; diff --git a/migrations/bootstrap/10_instances_access.sql b/migrations/bootstrap/10_instances_access.sql new file mode 100644 index 0000000..771d6e7 --- /dev/null +++ b/migrations/bootstrap/10_instances_access.sql @@ -0,0 +1,56 @@ +-- Instances, actor attribution, and sign-in throttling. + +CREATE TABLE public.instances ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + keycloak_group_id text, + name text NOT NULL, + path text NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + +CREATE TABLE public.email_sign_in_limits ( + address_digest bytea NOT NULL, + available_at timestamp with time zone NOT NULL +); + +CREATE TABLE public.users ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + keycloak_user_id text NOT NULL, + username text, + email text, + roles text[] DEFAULT '{}'::text[] NOT NULL, + instance_id uuid, + first_seen_at timestamp with time zone DEFAULT now() NOT NULL, + last_seen_at timestamp with time zone DEFAULT now() NOT NULL +); + +ALTER TABLE ONLY public.email_sign_in_limits + ADD CONSTRAINT email_sign_in_limits_pkey PRIMARY KEY (address_digest); + +ALTER TABLE ONLY public.instances + ADD CONSTRAINT instances_keycloak_group_id_key UNIQUE (keycloak_group_id); + +ALTER TABLE ONLY public.instances + ADD CONSTRAINT instances_path_key UNIQUE (path); + +ALTER TABLE ONLY public.instances + ADD CONSTRAINT instances_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.users + ADD CONSTRAINT users_keycloak_user_id_key UNIQUE (keycloak_user_id); + +ALTER TABLE ONLY public.users + ADD CONSTRAINT users_pkey PRIMARY KEY (id); + +CREATE INDEX email_sign_in_limits_available_at_idx ON public.email_sign_in_limits USING btree (available_at); + +CREATE UNIQUE INDEX users_email_lower_key ON public.users USING btree (lower(email)) WHERE (email IS NOT NULL); + +CREATE TRIGGER trg_instances_updated_at BEFORE UPDATE ON public.instances FOR EACH ROW EXECUTE FUNCTION public.set_updated_at(); + +ALTER TABLE ONLY public.users + ADD CONSTRAINT users_instance_id_fkey FOREIGN KEY (instance_id) REFERENCES public.instances(id) ON DELETE SET NULL; + +INSERT INTO public.users (keycloak_user_id, username, roles) +VALUES ('local-admin', 'local-admin', '{superadmin}'); diff --git a/migrations/bootstrap/20_documents_processing.sql b/migrations/bootstrap/20_documents_processing.sql new file mode 100644 index 0000000..5a76767 --- /dev/null +++ b/migrations/bootstrap/20_documents_processing.sql @@ -0,0 +1,129 @@ +-- Documents, pages, chunks, and frozen revisions. + +CREATE TABLE public.chunks ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + document_id uuid NOT NULL, + chunk_number integer NOT NULL, + original_text text NOT NULL, + edited_text text, + token_count integer DEFAULT 0 NOT NULL, + page_start integer DEFAULT 1 NOT NULL, + page_end integer DEFAULT 1 NOT NULL, + source_spans jsonb DEFAULT '[]'::jsonb NOT NULL, + section_title text, + content_type text, + is_reference boolean DEFAULT false NOT NULL, + is_reviewed boolean DEFAULT false NOT NULL, + is_excluded boolean DEFAULT false NOT NULL, + reviewer_notes text, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + tags text[] DEFAULT '{}'::text[] NOT NULL, + CONSTRAINT chunks_check CHECK ((page_end >= page_start)), + CONSTRAINT chunks_chunk_number_check CHECK ((chunk_number >= 1)), + CONSTRAINT chunks_page_start_check CHECK ((page_start >= 1)), + CONSTRAINT chunks_source_spans_check CHECK ((jsonb_typeof(source_spans) = 'array'::text)), + CONSTRAINT chunks_tags_max_count CHECK ((cardinality(tags) <= 16)), + CONSTRAINT chunks_token_count_check CHECK ((token_count >= 0)) +); + +CREATE TABLE public.documents ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + code text NOT NULL, + name text NOT NULL, + aliases text[] DEFAULT '{}'::text[] NOT NULL, + source jsonb NOT NULL, + is_disabled boolean DEFAULT false NOT NULL, + metadata jsonb DEFAULT '{"index_status": {}, "catalog_visible": true, "network_visible": true, "reindex_required": false}'::jsonb NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + provider_id uuid NOT NULL, + kind text NOT NULL, + CONSTRAINT documents_aliases_check CHECK ((array_position(aliases, ''::text) IS NULL)), + CONSTRAINT documents_code_check CHECK ((code ~ '^[a-z0-9]+(-[a-z0-9]+)*$'::text)), + CONSTRAINT documents_kind_nonblank CHECK ((btrim(kind) <> ''::text)), + CONSTRAINT documents_metadata_check CHECK ((jsonb_typeof(metadata) = 'object'::text)), + CONSTRAINT documents_source_check CHECK ((jsonb_typeof(source) = 'object'::text)), + CONSTRAINT documents_source_check1 CHECK (((source ->> 'fingerprint'::text) ~ '^[0-9a-f]{32}$'::text)) +); + +CREATE TABLE public.document_revisions ( + id uuid NOT NULL, + document_id uuid NOT NULL, + workflow_id uuid NOT NULL, + content_sha256 text NOT NULL, + snapshot jsonb NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT document_revisions_content_sha256_check CHECK ((content_sha256 ~ '^[0-9a-f]{64}$'::text)), + CONSTRAINT document_revisions_snapshot_check CHECK ((jsonb_typeof(snapshot) = 'object'::text)) +); + +CREATE TABLE public.pages ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + document_id uuid NOT NULL, + page_number integer NOT NULL, + original_markdown text, + edited_markdown text, + translations jsonb DEFAULT '{}'::jsonb NOT NULL, + metadata jsonb DEFAULT '{"is_reviewed": false}'::jsonb NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT pages_metadata_check CHECK ((jsonb_typeof(metadata) = 'object'::text)), + CONSTRAINT pages_page_number_check CHECK ((page_number >= 1)), + CONSTRAINT pages_translations_check CHECK ((jsonb_typeof(translations) = 'object'::text)) +); + +ALTER TABLE ONLY public.chunks + ADD CONSTRAINT chunks_document_id_chunk_number_key UNIQUE (document_id, chunk_number); + +ALTER TABLE ONLY public.chunks + ADD CONSTRAINT chunks_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.document_revisions + ADD CONSTRAINT document_revisions_document_id_id_key UNIQUE (document_id, id); + +ALTER TABLE ONLY public.document_revisions + ADD CONSTRAINT document_revisions_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.document_revisions + ADD CONSTRAINT document_revisions_workflow_id_key UNIQUE (workflow_id); + +ALTER TABLE ONLY public.documents + ADD CONSTRAINT documents_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.documents + ADD CONSTRAINT documents_provider_kind_code_key UNIQUE (provider_id, kind, code); + +ALTER TABLE ONLY public.pages + ADD CONSTRAINT pages_document_id_page_number_key UNIQUE (document_id, page_number); + +ALTER TABLE ONLY public.pages + ADD CONSTRAINT pages_pkey PRIMARY KEY (id); + +CREATE INDEX chunks_text_trgm_idx ON public.chunks USING gin (lower(COALESCE(edited_text, original_text)) public.gin_trgm_ops); + +CREATE INDEX document_revisions_document_idx ON public.document_revisions USING btree (document_id, created_at DESC); + +CREATE INDEX documents_created_at_idx ON public.documents USING btree (created_at DESC); + +CREATE INDEX documents_provider_fingerprint_idx ON public.documents USING btree (provider_id, ((source ->> 'fingerprint'::text))); + +CREATE INDEX documents_provider_id_idx ON public.documents USING btree (provider_id); + +CREATE TRIGGER trg_chunks_updated_at BEFORE UPDATE ON public.chunks FOR EACH ROW EXECUTE FUNCTION public.set_updated_at(); + +CREATE TRIGGER trg_documents_updated_at BEFORE UPDATE ON public.documents FOR EACH ROW EXECUTE FUNCTION public.set_updated_at(); + +CREATE TRIGGER trg_pages_updated_at BEFORE UPDATE ON public.pages FOR EACH ROW EXECUTE FUNCTION public.set_updated_at(); + +ALTER TABLE ONLY public.chunks + ADD CONSTRAINT chunks_document_id_fkey FOREIGN KEY (document_id) REFERENCES public.documents(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.document_revisions + ADD CONSTRAINT document_revisions_document_id_fkey FOREIGN KEY (document_id) REFERENCES public.documents(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.documents + ADD CONSTRAINT documents_provider_id_fkey FOREIGN KEY (provider_id) REFERENCES public.instances(id) ON DELETE RESTRICT; + +ALTER TABLE ONLY public.pages + ADD CONSTRAINT pages_document_id_fkey FOREIGN KEY (document_id) REFERENCES public.documents(id) ON DELETE CASCADE; diff --git a/migrations/bootstrap/25_workflows.sql b/migrations/bootstrap/25_workflows.sql new file mode 100644 index 0000000..c10e8d0 --- /dev/null +++ b/migrations/bootstrap/25_workflows.sql @@ -0,0 +1,84 @@ +-- Workflow history, artifacts, and the revision-to-workflow foreign key. + +CREATE TABLE public.workflows ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + document_id uuid NOT NULL, + workflow_type text NOT NULL, + temporal_workflow_id text NOT NULL, + status text DEFAULT 'running'::text NOT NULL, + current_stage text DEFAULT 'registered'::text NOT NULL, + stage_timestamps jsonb DEFAULT '{}'::jsonb NOT NULL, + stop_after_ocr boolean DEFAULT false NOT NULL, + error_message text, + config jsonb DEFAULT '{}'::jsonb NOT NULL, + started_by_user_id uuid, + started_at timestamp with time zone DEFAULT now() NOT NULL, + completed_at timestamp with time zone, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT workflows_config_check CHECK ((jsonb_typeof(config) = 'object'::text)), + CONSTRAINT workflows_stage_timestamps_check CHECK (((jsonb_typeof(stage_timestamps) = 'object'::text) AND ((stage_timestamps - ARRAY['ocr'::text, 'translation'::text, 'chunks'::text]) = '{}'::jsonb))) +); + +CREATE VIEW public.document_current_workflow AS + SELECT DISTINCT ON (workflows.document_id) workflows.id, + workflows.document_id, + workflows.workflow_type, + workflows.temporal_workflow_id, + workflows.status, + workflows.current_stage, + workflows.stage_timestamps, + workflows.stop_after_ocr, + workflows.error_message, + workflows.config, + workflows.started_by_user_id, + workflows.started_at, + workflows.completed_at, + workflows.updated_at + FROM public.workflows + ORDER BY workflows.document_id, workflows.started_at DESC, workflows.id DESC; + +CREATE TABLE public.workflow_artifacts ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + workflow_id uuid NOT NULL, + artifact_type text NOT NULL, + stage text, + storage_uri text NOT NULL, + mime_type text, + filename text, + size_bytes bigint, + metadata jsonb DEFAULT '{}'::jsonb NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT workflow_artifacts_artifact_type_check CHECK ((btrim(artifact_type) <> ''::text)), + CONSTRAINT workflow_artifacts_metadata_check CHECK ((jsonb_typeof(metadata) = 'object'::text)), + CONSTRAINT workflow_artifacts_size_bytes_check CHECK ((size_bytes >= 0)) +); + +ALTER TABLE ONLY public.workflow_artifacts + ADD CONSTRAINT workflow_artifacts_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.workflows + ADD CONSTRAINT workflows_pkey PRIMARY KEY (id); + +CREATE INDEX workflow_artifacts_type_idx ON public.workflow_artifacts USING btree (workflow_id, artifact_type, created_at DESC); + +CREATE INDEX workflow_artifacts_workflow_idx ON public.workflow_artifacts USING btree (workflow_id, created_at DESC); + +CREATE INDEX workflows_document_started_idx ON public.workflows USING btree (document_id, started_at DESC); + +CREATE INDEX workflows_status_started_idx ON public.workflows USING btree (status, started_at DESC); + +CREATE INDEX workflows_temporal_id_idx ON public.workflows USING btree (temporal_workflow_id); + +CREATE TRIGGER trg_workflows_updated_at BEFORE UPDATE ON public.workflows FOR EACH ROW EXECUTE FUNCTION public.set_updated_at(); + +ALTER TABLE ONLY public.document_revisions + ADD CONSTRAINT document_revisions_workflow_id_fkey FOREIGN KEY (workflow_id) REFERENCES public.workflows(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.workflow_artifacts + ADD CONSTRAINT workflow_artifacts_workflow_id_fkey FOREIGN KEY (workflow_id) REFERENCES public.workflows(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.workflows + ADD CONSTRAINT workflows_document_id_fkey FOREIGN KEY (document_id) REFERENCES public.documents(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.workflows + ADD CONSTRAINT workflows_started_by_user_id_fkey FOREIGN KEY (started_by_user_id) REFERENCES public.users(id) ON DELETE SET NULL; diff --git a/migrations/bootstrap/30_intake_source_evidence.sql b/migrations/bootstrap/30_intake_source_evidence.sql new file mode 100644 index 0000000..86465a9 --- /dev/null +++ b/migrations/bootstrap/30_intake_source_evidence.sql @@ -0,0 +1,153 @@ +-- Source policies, mappings, deliveries, receipts, and scheduled pulls. + +CREATE TABLE public.provider_source_settings ( + provider_id uuid NOT NULL, + settings jsonb NOT NULL, + version integer NOT NULL, + updated_by text NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT provider_source_settings_settings_check CHECK ((jsonb_typeof(settings) = 'object'::text)), + CONSTRAINT provider_source_settings_version_check CHECK ((version > 0)) +); + +CREATE TABLE public.scheduled_source_pulls ( + provider text NOT NULL, + source_key text NOT NULL, + profile_sha256 text NOT NULL, + cursor text, + visited_cursors text[] DEFAULT '{}'::text[] NOT NULL, + next_run_at timestamp with time zone DEFAULT now() NOT NULL, + lease_id uuid, + lease_expires_at timestamp with time zone, + failure_count integer DEFAULT 0 NOT NULL, + last_fetch_at timestamp with time zone, + last_payload_sha256 text, + last_error text, + last_page_policy_conflicts integer DEFAULT 0 NOT NULL, + CONSTRAINT scheduled_source_pulls_last_page_policy_conflicts_check CHECK ((last_page_policy_conflicts >= 0)), + CONSTRAINT scheduled_source_pulls_profile_sha256_check CHECK ((profile_sha256 ~ '^[0-9a-f]{64}$'::text)) +); + +CREATE TABLE public.source_intake_receipts ( + id uuid NOT NULL, + instance_id uuid NOT NULL, + source_kind text NOT NULL, + source_id text NOT NULL, + idempotency_key text NOT NULL, + external_id text NOT NULL, + raw_payload jsonb NOT NULL, + raw_sha256 text NOT NULL, + normalized_document jsonb NOT NULL, + mapping_version_id uuid, + document_id uuid NOT NULL, + job_id uuid NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + replay_evidence_key text, + replay_evidence_payload jsonb, + replay_evidence_sha256 text, + replay_evidence_observed_at timestamp with time zone, + CONSTRAINT source_intake_receipts_normalized_document_check CHECK ((jsonb_typeof(normalized_document) = 'object'::text)), + CONSTRAINT source_intake_receipts_raw_sha256_check CHECK ((raw_sha256 ~ '^[0-9a-f]{64}$'::text)), + CONSTRAINT source_intake_receipts_replay_evidence_check CHECK (((num_nonnulls(replay_evidence_key, replay_evidence_payload, replay_evidence_sha256, replay_evidence_observed_at) = 0) OR ((source_kind = 'crawl'::text) AND (num_nonnulls(replay_evidence_key, replay_evidence_payload, replay_evidence_sha256, replay_evidence_observed_at) = 4) AND (replay_evidence_key ~ '^[0-9a-f]{64}$'::text) AND (replay_evidence_sha256 ~ '^[0-9a-f]{64}$'::text)))), + CONSTRAINT source_intake_receipts_source_kind_check CHECK ((source_kind = ANY (ARRAY['webhook'::text, 'crawl'::text]))) +); + +CREATE TABLE public.webhook_deliveries ( + source_id uuid NOT NULL, + idempotency_key text NOT NULL, + body_sha256 text NOT NULL, + raw_payload jsonb NOT NULL, + raw_storage_uri text, + mapping_version_id uuid NOT NULL, + receipt_ids uuid[] NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT webhook_deliveries_body_sha256_check CHECK ((body_sha256 ~ '^[0-9a-f]{64}$'::text)) +); + +CREATE TABLE public.webhook_mapping_versions ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + source_id uuid NOT NULL, + version integer NOT NULL, + mapping jsonb NOT NULL, + mapping_sha256 text NOT NULL, + created_by text NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT webhook_mapping_versions_mapping_check CHECK ((jsonb_typeof(mapping) = 'object'::text)), + CONSTRAINT webhook_mapping_versions_mapping_sha256_check CHECK ((mapping_sha256 ~ '^[0-9a-f]{64}$'::text)), + CONSTRAINT webhook_mapping_versions_version_check CHECK ((version > 0)) +); + +CREATE TABLE public.webhook_sources ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + instance_id uuid NOT NULL, + source_key text NOT NULL, + service_client text NOT NULL, + service_subject text NOT NULL, + enabled boolean DEFAULT true NOT NULL, + auto_release boolean DEFAULT false NOT NULL, + mapping_version integer NOT NULL, + release_network_id text, + CONSTRAINT webhook_sources_mapping_version_check CHECK ((mapping_version > 0)), + CONSTRAINT webhook_sources_release_network_check CHECK (((release_network_id IS NULL) OR (auto_release AND (release_network_id ~ '^[A-Za-z0-9][A-Za-z0-9._:-]{0,252}$'::text)))) +); + +ALTER TABLE ONLY public.provider_source_settings + ADD CONSTRAINT provider_source_settings_pkey PRIMARY KEY (provider_id); + +ALTER TABLE ONLY public.scheduled_source_pulls + ADD CONSTRAINT scheduled_source_pulls_pkey PRIMARY KEY (provider, source_key, profile_sha256); + +ALTER TABLE ONLY public.source_intake_receipts + ADD CONSTRAINT source_intake_receipts_instance_id_source_kind_source_id_id_key UNIQUE (instance_id, source_kind, source_id, idempotency_key); + +ALTER TABLE ONLY public.source_intake_receipts + ADD CONSTRAINT source_intake_receipts_job_id_key UNIQUE (job_id); + +ALTER TABLE ONLY public.source_intake_receipts + ADD CONSTRAINT source_intake_receipts_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.webhook_deliveries + ADD CONSTRAINT webhook_deliveries_pkey PRIMARY KEY (source_id, idempotency_key); + +ALTER TABLE ONLY public.webhook_mapping_versions + ADD CONSTRAINT webhook_mapping_versions_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.webhook_mapping_versions + ADD CONSTRAINT webhook_mapping_versions_source_id_version_key UNIQUE (source_id, version); + +ALTER TABLE ONLY public.webhook_sources + ADD CONSTRAINT webhook_sources_instance_id_source_key_key UNIQUE (instance_id, source_key); + +ALTER TABLE ONLY public.webhook_sources + ADD CONSTRAINT webhook_sources_pkey PRIMARY KEY (id); + +CREATE INDEX source_intake_document_history ON public.source_intake_receipts USING btree (instance_id, source_kind, source_id, external_id, created_at DESC, id DESC); + +CREATE UNIQUE INDEX source_intake_receipts_replay_evidence_key ON public.source_intake_receipts USING btree (instance_id, source_id, replay_evidence_key) WHERE (replay_evidence_key IS NOT NULL); + +ALTER TABLE ONLY public.provider_source_settings + ADD CONSTRAINT provider_source_settings_provider_id_fkey FOREIGN KEY (provider_id) REFERENCES public.instances(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.source_intake_receipts + ADD CONSTRAINT source_intake_receipts_document_id_fkey FOREIGN KEY (document_id) REFERENCES public.documents(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.source_intake_receipts + ADD CONSTRAINT source_intake_receipts_instance_id_fkey FOREIGN KEY (instance_id) REFERENCES public.instances(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.source_intake_receipts + ADD CONSTRAINT source_intake_receipts_job_id_fkey FOREIGN KEY (job_id) REFERENCES public.workflows(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.source_intake_receipts + ADD CONSTRAINT source_intake_receipts_mapping_version_id_fkey FOREIGN KEY (mapping_version_id) REFERENCES public.webhook_mapping_versions(id); + +ALTER TABLE ONLY public.webhook_deliveries + ADD CONSTRAINT webhook_deliveries_mapping_version_id_fkey FOREIGN KEY (mapping_version_id) REFERENCES public.webhook_mapping_versions(id); + +ALTER TABLE ONLY public.webhook_deliveries + ADD CONSTRAINT webhook_deliveries_source_id_fkey FOREIGN KEY (source_id) REFERENCES public.webhook_sources(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.webhook_mapping_versions + ADD CONSTRAINT webhook_mapping_versions_source_id_fkey FOREIGN KEY (source_id) REFERENCES public.webhook_sources(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.webhook_sources + ADD CONSTRAINT webhook_sources_instance_id_fkey FOREIGN KEY (instance_id) REFERENCES public.instances(id) ON DELETE CASCADE; diff --git a/migrations/bootstrap/40_search_audit_catalog.sql b/migrations/bootstrap/40_search_audit_catalog.sql new file mode 100644 index 0000000..20df036 --- /dev/null +++ b/migrations/bootstrap/40_search_audit_catalog.sql @@ -0,0 +1,133 @@ +-- Search settings, audit history, and catalog interfaces. + +CREATE TABLE public.audit_logs ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + document_id uuid, + workflow_id text, + instance_id uuid, + action_type text NOT NULL, + entity_type text, + entity_number integer, + field_name text, + old_value text, + new_value text, + metadata jsonb DEFAULT '{}'::jsonb NOT NULL, + actor_user_id uuid, + actor jsonb DEFAULT '{}'::jsonb NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT audit_logs_action_type_check CHECK ((btrim(action_type) <> ''::text)), + CONSTRAINT audit_logs_actor_check CHECK ((jsonb_typeof(actor) = 'object'::text)), + CONSTRAINT audit_logs_check CHECK (((entity_type = ANY (ARRAY['setting'::text, 'catalog'::text])) OR (workflow_id IS NOT NULL))), + CONSTRAINT audit_logs_metadata_check CHECK ((jsonb_typeof(metadata) = 'object'::text)) +); + +CREATE TABLE public.catalogs ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + instance_id uuid NOT NULL, + type text NOT NULL, + name text NOT NULL, + metadata jsonb DEFAULT '{}'::jsonb NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT catalogs_metadata_check CHECK ((jsonb_typeof(metadata) = 'object'::text)), + CONSTRAINT catalogs_type_check CHECK ((type ~ '^[a-z0-9]+:[A-Za-z0-9]+$'::text)) +); + +CREATE VIEW public.legacy_document_catalog_types AS + SELECT mapping.kind, + mapping.catalog_type + FROM ( VALUES ('scheme'::text,'openagrinet:AgricultureResource'::text), ('advisory'::text,'openagrinet:KnowledgeAdvisory'::text), ('video'::text,'openagrinet:VideoResource'::text)) mapping(kind, catalog_type); + +CREATE VIEW public.catalog_status AS + SELECT d.id AS document_id, + c.id AS catalog_id, + d.provider_id AS instance_id, + g.name AS instance_name, + COALESCE(m.catalog_type, d.kind) AS type, + d.code, + d.name, + d.aliases, + CASE + WHEN (x.is_live AND COALESCE(((d.metadata ->> 'reindex_required'::text))::boolean, false)) THEN 'pending_reindex'::text + WHEN x.is_live THEN 'live'::text + WHEN ((cw.current_stage = 'approval_for_prod'::text) AND (NOT d.is_disabled)) THEN 'pending_prod'::text + ELSE 'disabled'::text + END AS status, + ( SELECT w.temporal_workflow_id + FROM public.workflows w + WHERE ((w.document_id = d.id) AND (w.workflow_type = 'pipeline'::text)) + ORDER BY w.started_at, w.id + LIMIT 1) AS workflow_id, + (( SELECT count(*) AS count + FROM public.chunks ch + WHERE (ch.document_id = d.id)))::integer AS chunk_count, + COALESCE(((d.metadata ->> 'network_visible'::text))::boolean, true) AS network_visible, + CASE + WHEN x.is_live THEN ((((d.metadata -> 'index_status'::text) -> 'prod'::text) ->> 'last_indexed_at'::text))::timestamp with time zone + ELSE NULL::timestamp with time zone + END AS promoted_at + FROM (((((public.documents d + LEFT JOIN public.legacy_document_catalog_types m ON ((m.kind = d.kind))) + LEFT JOIN public.catalogs c ON (((c.instance_id = d.provider_id) AND (c.type = COALESCE(m.catalog_type, d.kind))))) + JOIN public.instances g ON ((g.id = d.provider_id))) + LEFT JOIN public.document_current_workflow cw ON ((cw.document_id = d.id))) + CROSS JOIN LATERAL ( SELECT ((cw.current_stage = 'completed'::text) AND (NOT d.is_disabled) AND COALESCE(((d.metadata ->> 'catalog_visible'::text))::boolean, true)) AS is_live) x); + +CREATE TABLE public.search_settings ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + singleton boolean DEFAULT true NOT NULL, + retrieval jsonb DEFAULT '{}'::jsonb NOT NULL, + candidates jsonb DEFAULT '{}'::jsonb NOT NULL, + results jsonb DEFAULT '{}'::jsonb NOT NULL, + query jsonb DEFAULT '{}'::jsonb NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT search_settings_candidates_check CHECK ((jsonb_typeof(candidates) = 'object'::text)), + CONSTRAINT search_settings_query_check CHECK ((jsonb_typeof(query) = 'object'::text)), + CONSTRAINT search_settings_results_check CHECK ((jsonb_typeof(results) = 'object'::text)), + CONSTRAINT search_settings_retrieval_check CHECK ((jsonb_typeof(retrieval) = 'object'::text)), + CONSTRAINT search_settings_singleton_check CHECK (singleton) +); + +ALTER TABLE ONLY public.audit_logs + ADD CONSTRAINT audit_logs_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.catalogs + ADD CONSTRAINT catalogs_instance_id_type_key UNIQUE (instance_id, type); + +ALTER TABLE ONLY public.catalogs + ADD CONSTRAINT catalogs_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.search_settings + ADD CONSTRAINT search_settings_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.search_settings + ADD CONSTRAINT search_settings_singleton_key UNIQUE (singleton); + +CREATE INDEX audit_logs_action_idx ON public.audit_logs USING btree (action_type, created_at DESC); + +CREATE INDEX audit_logs_created_idx ON public.audit_logs USING btree (created_at DESC); + +CREATE INDEX audit_logs_document_idx ON public.audit_logs USING btree (document_id, created_at DESC); + +CREATE INDEX audit_logs_instance_idx ON public.audit_logs USING btree (instance_id, created_at DESC); + +CREATE INDEX audit_logs_settings_idx ON public.audit_logs USING btree (created_at DESC) WHERE (entity_type = 'setting'::text); + +CREATE TRIGGER trg_catalogs_updated_at BEFORE UPDATE ON public.catalogs FOR EACH ROW EXECUTE FUNCTION public.set_updated_at(); + +CREATE TRIGGER trg_search_settings_updated_at BEFORE UPDATE ON public.search_settings FOR EACH ROW EXECUTE FUNCTION public.set_updated_at(); + +ALTER TABLE ONLY public.audit_logs + ADD CONSTRAINT audit_logs_actor_user_id_fkey FOREIGN KEY (actor_user_id) REFERENCES public.users(id) ON DELETE SET NULL; + +ALTER TABLE ONLY public.audit_logs + ADD CONSTRAINT audit_logs_document_id_fkey FOREIGN KEY (document_id) REFERENCES public.documents(id) ON DELETE SET NULL; + +ALTER TABLE ONLY public.audit_logs + ADD CONSTRAINT audit_logs_instance_id_fkey FOREIGN KEY (instance_id) REFERENCES public.instances(id) ON DELETE SET NULL; + +ALTER TABLE ONLY public.catalogs + ADD CONSTRAINT catalogs_instance_id_fkey FOREIGN KEY (instance_id) REFERENCES public.instances(id) ON DELETE RESTRICT; + +INSERT INTO public.search_settings DEFAULT VALUES; diff --git a/migrations/bootstrap/50_network_registry.sql b/migrations/bootstrap/50_network_registry.sql new file mode 100644 index 0000000..7f70f74 --- /dev/null +++ b/migrations/bootstrap/50_network_registry.sql @@ -0,0 +1,166 @@ +-- Network releases, publication delivery, and Registry state. + +CREATE FUNCTION public.reject_registry_operator_history_mutation() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + RAISE EXCEPTION 'Registry operator history is append-only' USING ERRCODE = '55000'; +END +$$; + +CREATE TABLE public.network_release_staging ( + revision_id uuid NOT NULL, + network_id text NOT NULL, + collection text, + provider_id text, + expected_generation uuid, + generation_captured boolean DEFAULT true NOT NULL, + CONSTRAINT network_release_staging_network_id_check CHECK ((btrim(network_id) <> ''::text)) +); + +CREATE TABLE public.network_releases ( + document_id uuid NOT NULL, + network_id text NOT NULL, + revision_id uuid NOT NULL, + transaction_id uuid NOT NULL, + released_at timestamp with time zone DEFAULT now() NOT NULL, + withdrawn_at timestamp with time zone, + generation uuid DEFAULT gen_random_uuid() NOT NULL, + CONSTRAINT network_releases_network_id_check CHECK ((btrim(network_id) <> ''::text)) +); + +CREATE TABLE public.publication_deliveries ( + network_id text NOT NULL, + transaction_id uuid NOT NULL, + document_id uuid NOT NULL, + request_identity jsonb NOT NULL, + receipt jsonb, + created_at timestamp with time zone DEFAULT now() NOT NULL, + received_at timestamp with time zone, + CONSTRAINT publication_deliveries_network_id_check CHECK ((btrim(network_id) <> ''::text)), + CONSTRAINT publication_deliveries_receipt_check CHECK ((jsonb_typeof(receipt) = 'object'::text)), + CONSTRAINT publication_deliveries_request_identity_check CHECK ((jsonb_typeof(request_identity) = 'object'::text)) +); + +CREATE TABLE public.registry_operator_decisions ( + proposal_id uuid NOT NULL, + outcome text NOT NULL, + decided_by text NOT NULL, + reason text NOT NULL, + decided_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT registry_operator_decisions_decided_by_check CHECK (((char_length(btrim(decided_by)) >= 1) AND (char_length(btrim(decided_by)) <= 200))), + CONSTRAINT registry_operator_decisions_outcome_check CHECK ((outcome = ANY (ARRAY['approved'::text, 'rejected'::text]))), + CONSTRAINT registry_operator_decisions_reason_check CHECK (((char_length(btrim(reason)) >= 1) AND (char_length(btrim(reason)) <= 1000))) +); + +CREATE TABLE public.registry_operator_proposals ( + id uuid DEFAULT gen_random_uuid() NOT NULL, + network_id text NOT NULL, + provider_id uuid NOT NULL, + request_id uuid NOT NULL, + capability text NOT NULL, + version bigint NOT NULL, + schema_pack_digest text, + change jsonb NOT NULL, + observed_record_id text, + observed_revision text, + submitted_by text NOT NULL, + submission_reason text NOT NULL, + submitted_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT registry_operator_proposals_change_check CHECK (((jsonb_typeof(change) = 'object'::text) AND (octet_length((change)::text) <= 16384))), + CONSTRAINT registry_operator_proposals_check CHECK (((observed_record_id IS NULL) = (observed_revision IS NULL))), + CONSTRAINT registry_operator_proposals_check1 CHECK (((capability = ''::text) = (schema_pack_digest IS NULL))), + CONSTRAINT registry_operator_proposals_observed_record_id_check CHECK (((char_length(btrim(observed_record_id)) >= 1) AND (char_length(btrim(observed_record_id)) <= 253))), + CONSTRAINT registry_operator_proposals_observed_revision_check CHECK (((char_length(btrim(observed_revision)) >= 1) AND (char_length(btrim(observed_revision)) <= 253))), + CONSTRAINT registry_operator_proposals_submission_reason_check CHECK (((char_length(btrim(submission_reason)) >= 1) AND (char_length(btrim(submission_reason)) <= 1000))), + CONSTRAINT registry_operator_proposals_submitted_by_check CHECK (((char_length(btrim(submitted_by)) >= 1) AND (char_length(btrim(submitted_by)) <= 200))), + CONSTRAINT registry_operator_proposals_version_check CHECK ((version > 0)) +); + +CREATE TABLE public.registry_operator_subjects ( + network_id text NOT NULL, + provider_id uuid NOT NULL, + participant_id text NOT NULL, + CONSTRAINT registry_operator_subjects_network_id_check CHECK (((char_length(btrim(network_id)) >= 1) AND (char_length(btrim(network_id)) <= 253))), + CONSTRAINT registry_operator_subjects_participant_id_check CHECK (((char_length(btrim(participant_id)) >= 3) AND (char_length(btrim(participant_id)) <= 253))) +); + +CREATE TABLE public.schema_pack_versions ( + network_id text NOT NULL, + capability text NOT NULL, + digest text NOT NULL, + version text NOT NULL, + bundle jsonb NOT NULL, + installed_at timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT schema_pack_versions_bundle_check CHECK ((jsonb_typeof(bundle) = 'object'::text)), + CONSTRAINT schema_pack_versions_capability_check CHECK ((btrim(capability) <> ''::text)), + CONSTRAINT schema_pack_versions_digest_check CHECK ((digest ~ '^[0-9a-f]{64}$'::text)), + CONSTRAINT schema_pack_versions_network_id_check CHECK ((btrim(network_id) <> ''::text)), + CONSTRAINT schema_pack_versions_version_check CHECK ((btrim(version) <> ''::text)) +); + +ALTER TABLE ONLY public.network_release_staging + ADD CONSTRAINT network_release_staging_pkey PRIMARY KEY (revision_id, network_id); + +ALTER TABLE ONLY public.network_releases + ADD CONSTRAINT network_releases_pkey PRIMARY KEY (document_id, network_id); + +ALTER TABLE ONLY public.publication_deliveries + ADD CONSTRAINT publication_deliveries_pkey PRIMARY KEY (network_id, transaction_id); + +ALTER TABLE ONLY public.registry_operator_decisions + ADD CONSTRAINT registry_operator_decisions_pkey PRIMARY KEY (proposal_id); + +ALTER TABLE ONLY public.registry_operator_proposals + ADD CONSTRAINT registry_operator_proposals_network_id_provider_id_capabili_key UNIQUE (network_id, provider_id, capability, version); + +ALTER TABLE ONLY public.registry_operator_proposals + ADD CONSTRAINT registry_operator_proposals_network_id_provider_id_request__key UNIQUE (network_id, provider_id, request_id); + +ALTER TABLE ONLY public.registry_operator_proposals + ADD CONSTRAINT registry_operator_proposals_pkey PRIMARY KEY (id); + +ALTER TABLE ONLY public.registry_operator_subjects + ADD CONSTRAINT registry_operator_subjects_network_id_participant_id_key UNIQUE (network_id, participant_id); + +ALTER TABLE ONLY public.registry_operator_subjects + ADD CONSTRAINT registry_operator_subjects_pkey PRIMARY KEY (network_id, provider_id); + +ALTER TABLE ONLY public.schema_pack_versions + ADD CONSTRAINT schema_pack_versions_pkey PRIMARY KEY (network_id, capability, digest); + +CREATE INDEX network_release_staging_scope_idx ON public.network_release_staging USING btree (network_id, collection); + +CREATE INDEX network_releases_revision_idx ON public.network_releases USING btree (revision_id, network_id); + +CREATE INDEX publication_deliveries_document_idx ON public.publication_deliveries USING btree (document_id); + +CREATE TRIGGER registry_operator_decisions_append_only BEFORE DELETE OR UPDATE ON public.registry_operator_decisions FOR EACH ROW EXECUTE FUNCTION public.reject_registry_operator_history_mutation(); + +CREATE TRIGGER registry_operator_proposals_append_only BEFORE DELETE OR UPDATE ON public.registry_operator_proposals FOR EACH ROW EXECUTE FUNCTION public.reject_registry_operator_history_mutation(); + +CREATE TRIGGER registry_operator_subjects_append_only BEFORE DELETE OR UPDATE ON public.registry_operator_subjects FOR EACH ROW EXECUTE FUNCTION public.reject_registry_operator_history_mutation(); + +ALTER TABLE ONLY public.network_release_staging + ADD CONSTRAINT network_release_staging_revision_id_fkey FOREIGN KEY (revision_id) REFERENCES public.document_revisions(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.network_releases + ADD CONSTRAINT network_releases_document_id_fkey FOREIGN KEY (document_id) REFERENCES public.documents(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.network_releases + ADD CONSTRAINT network_releases_document_id_revision_id_fkey FOREIGN KEY (document_id, revision_id) REFERENCES public.document_revisions(document_id, id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.publication_deliveries + ADD CONSTRAINT publication_deliveries_document_id_fkey FOREIGN KEY (document_id) REFERENCES public.documents(id) ON DELETE CASCADE; + +ALTER TABLE ONLY public.registry_operator_decisions + ADD CONSTRAINT registry_operator_decisions_proposal_id_fkey FOREIGN KEY (proposal_id) REFERENCES public.registry_operator_proposals(id); + +ALTER TABLE ONLY public.registry_operator_proposals + ADD CONSTRAINT registry_operator_proposals_network_id_capability_schema_p_fkey FOREIGN KEY (network_id, capability, schema_pack_digest) REFERENCES public.schema_pack_versions(network_id, capability, digest); + +ALTER TABLE ONLY public.registry_operator_proposals + ADD CONSTRAINT registry_operator_proposals_network_id_provider_id_fkey FOREIGN KEY (network_id, provider_id) REFERENCES public.registry_operator_subjects(network_id, provider_id); + +ALTER TABLE ONLY public.registry_operator_subjects + ADD CONSTRAINT registry_operator_subjects_provider_id_fkey FOREIGN KEY (provider_id) REFERENCES public.instances(id) ON DELETE RESTRICT; diff --git a/pipeline/__init__.py b/pipeline/__init__.py index cd462cf..adf2e90 100644 --- a/pipeline/__init__.py +++ b/pipeline/__init__.py @@ -1,10 +1 @@ -""" -Temporal-based document ingestion pipeline. - -Components: -- workflows.py: Temporal workflow definitions -- activities.py: Retryable work units -- worker.py: Temporal worker process -- api.py: FastAPI REST interface -- models.py: Data models -""" +"""Knowledge Provider document ingestion, publication, and search.""" diff --git a/pipeline/activities.py b/pipeline/activities.py deleted file mode 100644 index ec82210..0000000 --- a/pipeline/activities.py +++ /dev/null @@ -1,1757 +0,0 @@ -""" -Temporal activities for the OCR pipeline. -Each activity is a retryable unit of work. -""" - -import asyncio -import csv -import hashlib -import json -import mimetypes -import os -import re -import shutil -import subprocess -import tempfile -from datetime import datetime -from pathlib import Path -from threading import Lock -from uuid import uuid4 - -import tiktoken -from minio import Minio -from temporalio import activity -from temporalio.exceptions import ApplicationError - -from . import scheme_catalog -from .documents import document_validity -from .auth.tenancy import group_for_instance -from .catalog_builder import normalize_document_kind -from .chunking import chunk_pages, load_chunking_config -from .discovery_publish_service import DiscoveryPublishService, PublishConfigError -from .document_repository import DocumentRepository -from .instances import instance_display_name, publish_to_network_enabled -from .models import PUBLISH_SKIPPED_DISABLED, PUBLISH_SKIPPED_UNMAPPED_KIND -from .network_constants import RESULT_REJECTED -from .network_group import load_provider_naming, named_group, provider_for -from .network_provider import InvalidProviderIdError -from .ocr import ocr_pdf as run_ocr_pdf -from .ocr import ocr_pdf_in_segments as run_ocr_pdf_in_segments -from .translation import load_translation_config, translate_pages - -SUPPORTED_INPUT_EXTENSIONS = { - ".pdf", ".doc", ".docx", ".ppt", ".pptx", ".xls", ".xlsx", ".csv", ".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff" -} -IMAGE_INPUT_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff"} -OFFICE_INPUT_EXTENSIONS = {".doc", ".docx", ".ppt", ".pptx", ".xls", ".xlsx"} -DELIMITED_INPUT_EXTENSIONS = {".csv"} -NATIVE_SPREADSHEET_EXTENSIONS = {".xlsx"} - -_doc_metadata_cache: dict[str, dict] = {} -_doc_descriptions_cache: dict[str, str] = {} -_metadata_loaded = False -_metadata_lock = Lock() - - -def get_minio_client(): - """Get MinIO client from environment. Credentials are required.""" - access_key = os.environ.get("MINIO_ACCESS_KEY") - secret_key = os.environ.get("MINIO_SECRET_KEY") - - if not access_key or not secret_key: - raise RuntimeError("MINIO_ACCESS_KEY and MINIO_SECRET_KEY environment variables are required") - - return Minio( - os.environ.get("MINIO_ENDPOINT", "localhost:9000"), - access_key=access_key, - secret_key=secret_key, - secure=False, - ) - - -def download_from_minio(minio_path: str) -> str: - """Download file from MinIO and return local temp path.""" - path = minio_path.replace("minio://", "") - parts = path.split("/", 1) - bucket = parts[0] - object_name = parts[1] if len(parts) > 1 else "" - - client = get_minio_client() - - suffix = Path(object_name).suffix - temp_file = tempfile.NamedTemporaryFile(delete=False, suffix=suffix) - temp_path = temp_file.name - temp_file.close() - - client.fget_object(bucket, object_name, temp_path) - return temp_path - - -def _minio_object_name(workflow_id: str, artifact_type: str, filename: str) -> str: - safe_name = re.sub(r"[^A-Za-z0-9._-]+", "_", filename or "artifact") - return f"documents/{workflow_id}/{artifact_type}/{safe_name}" - - -def _upload_file_to_minio(local_path: str, workflow_id: str, artifact_type: str, filename: str) -> tuple[str, int, str]: - client = get_minio_client() - bucket = os.environ.get("MINIO_BUCKET", "documents") - object_name = _minio_object_name(workflow_id, artifact_type, filename) - mime_type = mimetypes.guess_type(filename)[0] or "application/octet-stream" - client.fput_object(bucket, object_name, local_path, content_type=mime_type) - size_bytes = os.path.getsize(local_path) - return (f"minio://{bucket}/{object_name}", size_bytes, mime_type) - - -def _write_json_temp(data: object, suffix: str = ".json") -> str: - with tempfile.NamedTemporaryFile("w", delete=False, suffix=suffix, encoding="utf-8") as f: - json.dump(data, f, ensure_ascii=False, indent=2) - return f.name - - -def _source_type_from_path(filepath: str) -> tuple[str, str]: - ext = Path(filepath).suffix.lower() - if ext in {".csv", ".xlsx"}: - return ("spreadsheet", "spreadsheet") - if ext in IMAGE_INPUT_EXTENSIONS | OFFICE_INPUT_EXTENSIONS | {".pdf"}: - return ("document", "pdf") - return ("unknown", "pdf") - - -def _normalized_filename(original_name: str, canonical_input_type: str) -> str: - stem = Path(original_name).stem or "document" - if canonical_input_type == "spreadsheet": - ext = Path(original_name).suffix.lower() - return f"{stem}{ext if ext in {'.csv', '.xlsx'} else '.csv'}" - return f"{stem}.pdf" - - -def _resolve_local_path(filepath: str) -> tuple[str, bool]: - """ - Resolve path into local file path. - Returns (path, should_cleanup). - """ - if filepath.startswith("minio://") or filepath.startswith("minio:/"): - minio_path = filepath - if filepath.startswith("minio:/") and not filepath.startswith("minio://"): - minio_path = filepath.replace("minio:/", "minio://", 1) - local_path = download_from_minio(minio_path) - activity.logger.info(f"Downloaded from MinIO to {local_path}") - return local_path, True - return filepath, False - - -def _normalize_filename(name: str) -> str: - return (name or "").strip().lower() - - -def _load_metadata_once() -> None: - global _metadata_loaded - if _metadata_loaded: - return - with _metadata_lock: - if _metadata_loaded: - return - - metadata_csv_path = os.getenv( - "DOCUMENT_METADATA_CSV_PATH", "/app/workspace/document_manifest.csv" - ) - descriptions_jsonl_path = os.getenv( - "DOCUMENT_DESCRIPTIONS_JSONL_PATH", - "/app/workspace/document_descriptions.jsonl", - ) - - if os.path.exists(metadata_csv_path): - try: - with open(metadata_csv_path, "r", encoding="utf-8-sig", newline="") as f: - reader = csv.DictReader(f) - for row in reader: - file_name = (row.get("File Name") or "").strip() - if not file_name or file_name.startswith("http://") or file_name.startswith("https://"): - continue - key = _normalize_filename(file_name) - _doc_metadata_cache[key] = { - "title_en": (row.get("Title (English)") or "").strip(), - "title_gu": (row.get("Title (Gujarati)") or "").strip(), - "doc_language": (row.get("Language (Gujarati / English)") or "").strip(), - "category_tags": (row.get("Category Tags (") or "").strip(), - "doc_short_description": (row.get("Description") or "").strip(), - "quality_score": (row.get("Quality(1-5)") or "").strip(), - "priority_rank": (row.get("Priority(1-5)") or "").strip(), - "ingestion_status": (row.get("Status ingested in the system") or "").strip(), - } - except Exception as e: - activity.logger.warning(f"Failed loading document metadata CSV: {e}") - - if os.path.exists(descriptions_jsonl_path): - try: - with open(descriptions_jsonl_path, "r", encoding="utf-8") as f: - for line in f: - line = line.strip() - if not line: - continue - obj = json.loads(line) - key = _normalize_filename(obj.get("file", "")) - if key and obj.get("description"): - _doc_descriptions_cache[key] = str(obj["description"]).strip() - except Exception as e: - activity.logger.warning(f"Failed loading document descriptions JSONL: {e}") - - _metadata_loaded = True - - -def _get_doc_metadata(filename: str) -> dict: - _load_metadata_once() - return _doc_metadata_cache.get(_normalize_filename(filename), {}) - - -def _get_doc_description(filename: str) -> str: - _load_metadata_once() - return _doc_descriptions_cache.get(_normalize_filename(filename), "") - - -def _convert_image_to_pdf(input_path: str, output_path: str) -> None: - from PIL import Image, ImageOps - - with Image.open(input_path) as img: - rgb_img = ImageOps.exif_transpose(img).convert("RGB") - rgb_img.save(output_path, "PDF", resolution=300.0) - - -def _convert_office_to_pdf(input_path: str, output_dir: str) -> str: - soffice_bin = shutil.which("soffice") - if not soffice_bin: - raise RuntimeError( - "LibreOffice (soffice) is not installed. Install libreoffice to convert office files to PDF." - ) - - cmd = [ - soffice_bin, - "--headless", - "--nologo", - "--nodefault", - "--nolockcheck", - "--nofirststartwizard", - "--convert-to", - "pdf", - "--outdir", - output_dir, - input_path, - ] - result = subprocess.run(cmd, capture_output=True, text=True) - if result.returncode != 0: - raise RuntimeError( - f"Failed to convert office document to PDF: {result.stderr.strip() or result.stdout.strip()}" - ) - - output_pdf = Path(output_dir) / f"{Path(input_path).stem}.pdf" - if not output_pdf.exists(): - raise RuntimeError("Office-to-PDF conversion finished but output PDF was not found") - return str(output_pdf) - - -def _ensure_pdf_input(local_path: str) -> tuple[str, bool]: - """ - Ensure the given file path points to a PDF. - Returns (pdf_path, should_cleanup). - """ - ext = Path(local_path).suffix.lower() - if ext not in SUPPORTED_INPUT_EXTENSIONS: - raise ValueError( - f"Unsupported file extension '{ext}'. Supported extensions: {sorted(SUPPORTED_INPUT_EXTENSIONS)}" - ) - - if ext == ".pdf": - return local_path, False - - work_dir = tempfile.mkdtemp(prefix="doc_convert_") - output_pdf = Path(work_dir) / f"{Path(local_path).stem}.pdf" - - if ext in IMAGE_INPUT_EXTENSIONS: - _convert_image_to_pdf(local_path, str(output_pdf)) - elif ext in OFFICE_INPUT_EXTENSIONS: - converted = _convert_office_to_pdf(local_path, work_dir) - if converted != str(output_pdf): - shutil.move(converted, output_pdf) - else: - raise ValueError(f"Unsupported conversion path for extension: {ext}") - - activity.logger.info(f"Converted {local_path} -> {output_pdf}") - return str(output_pdf), True - - -def _csv_to_pages(input_path: str, rows_per_page: int = 80) -> list[dict]: - import csv - - def _open_reader(): - last_err = None - for enc in ("utf-8-sig", "utf-8", "latin-1"): - try: - f = open(input_path, "r", encoding=enc, newline="") - sample = f.read(4096) - f.seek(0) - try: - dialect = csv.Sniffer().sniff(sample, delimiters=",;\t|") - except Exception: - dialect = csv.excel - return f, csv.reader(f, dialect=dialect) - except Exception as e: - last_err = e - raise RuntimeError(f"Could not read CSV file: {last_err}") - - f, reader = _open_reader() - try: - rows = [[(c or "").strip() for c in r] for r in reader if any((c or "").strip() for c in r)] - finally: - f.close() - - if not rows: - return [{ - "page_number": 1, - "original_markdown": f"# {Path(input_path).name}\n\n(Empty CSV file)", - "edited_markdown": None, - "is_reviewed": False, - "reviewer_notes": None, - }] - - header = rows[0] - body = rows[1:] if len(rows) > 1 else [] - - pages: list[dict] = [] - page_num = 1 - for i in range(0, len(body), rows_per_page): - batch = body[i:i + rows_per_page] - lines = [f"# Spreadsheet Data: {Path(input_path).name}", "", f"Columns: {', '.join(header)}", ""] - for row_idx, row in enumerate(batch, start=i + 1): - pairs = [] - for col_i, val in enumerate(row): - col = header[col_i] if col_i < len(header) and header[col_i] else f"col_{col_i+1}" - pairs.append(f"{col}: {val}") - if pairs: - lines.append(f"- Row {row_idx}: " + " | ".join(pairs)) - pages.append({ - "page_number": page_num, - "original_markdown": "\n".join(lines), - "edited_markdown": None, - "is_reviewed": False, - "reviewer_notes": None, - }) - page_num += 1 - return pages - - -def _xlsx_to_pages(input_path: str, rows_per_page: int = 80) -> list[dict]: - from datetime import date, datetime - - from openpyxl import load_workbook - - def _cell_to_str(value) -> str: - if value is None: - return "" - if isinstance(value, (datetime, date)): - return value.isoformat() - return str(value).strip() - - pages: list[dict] = [] - page_num = 1 - wb = load_workbook(filename=input_path, data_only=True, read_only=True) - - for sheet in wb.worksheets: - header: list[str] | None = None - batch: list[tuple[int, list[str]]] = [] - seen_rows = 0 - - def emit_page(rows_batch: list[tuple[int, list[str]]]) -> None: - nonlocal page_num - if not rows_batch: - return - assert header is not None - lines = [ - f"# Spreadsheet Data: {Path(input_path).name}", - "", - f"Sheet: {sheet.title}", - f"Columns: {', '.join(header)}", - "", - ] - for excel_row_num, row in rows_batch: - pairs = [] - for col_i, val in enumerate(row): - col = header[col_i] if col_i < len(header) and header[col_i] else f"col_{col_i+1}" - pairs.append(f"{col}: {val}") - if pairs: - lines.append(f"- Row {excel_row_num}: " + " | ".join(pairs)) - pages.append({ - "page_number": page_num, - "original_markdown": "\n".join(lines), - "edited_markdown": None, - "is_reviewed": False, - "reviewer_notes": None, - }) - page_num += 1 - - for excel_row_num, row in enumerate(sheet.iter_rows(values_only=True), start=1): - values = [_cell_to_str(v) for v in row] - if not any(values): - continue - seen_rows += 1 - - if header is None: - header = values - continue - - batch.append((excel_row_num, values)) - if len(batch) >= rows_per_page: - emit_page(batch) - batch = [] - - if header is not None: - emit_page(batch) - elif seen_rows == 0: - pages.append({ - "page_number": page_num, - "original_markdown": f"# Spreadsheet Data: {Path(input_path).name}\n\nSheet: {sheet.title}\n\n(Empty sheet)", - "edited_markdown": None, - "is_reviewed": False, - "reviewer_notes": None, - }) - page_num += 1 - - wb.close() - - if not pages: - return [{ - "page_number": 1, - "original_markdown": f"# Spreadsheet Data: {Path(input_path).name}\n\n(Empty workbook)", - "edited_markdown": None, - "is_reviewed": False, - "reviewer_notes": None, - }] - return pages - - -# ============================================================================= -# Text Processing Utilities -# ============================================================================= - -def count_tokens(text: str, model: str = "cl100k_base") -> int: - if not text: - return 0 - encoder = tiktoken.get_encoding(model) - return len(encoder.encode(str(text), disallowed_special=())) - - -def clean_html_tags(text: str) -> str: - if not isinstance(text, str): - return text - text = text.replace("
", "\n").replace("
", "\n").replace("
", "\n") - text = re.sub(r"<[^>]+>", "", text) - return text.strip() - - -def clean_latex_notation(text: str) -> str: - if not isinstance(text, str): - return text - text = re.sub(r"\[\^[0-9]+\]", "", text) - text = re.sub(r"\$\s*\{\s*\}\s*\^\{[0-9]+\}\s*\$", "", text) - text = re.sub(r"\$\s*\^\{[0-9]+\}\s*\$", "", text) - text = re.sub(r"\$\s*\$", "", text) - text = re.sub(r"\\[a-zA-Z]+\{[^}]*\}", "", text) - text = re.sub(r"\\[a-zA-Z]+", "", text) - text = re.sub(r"[\\{}]", "", text) - return text - - -def format_table_content(text: str) -> str: - if not isinstance(text, str): - return text - lines = text.split("\n") - cleaned_lines = [] - for line in lines: - if re.match(r"^[\s\|]*$", line): - continue - if re.match(r"^[\s\|\-\:]*$", line): - continue - line = re.sub(r"\|\s*\|", "|", line) - line = re.sub(r"^\|\s*", "", line) - line = re.sub(r"\s*\|$", "", line) - line = re.sub(r"\s+", " ", line).strip() - if line: - cleaned_lines.append(line) - return "\n".join(cleaned_lines) - - -def clean_text(text: str) -> str: - if not isinstance(text, str): - return text - text = clean_html_tags(text) - text = clean_latex_notation(text) - text = format_table_content(text) - text = re.sub(r"\n{3,}", "\n\n", text) - text = re.sub(r"[ \t]+", " ", text) - lines = [line.strip() for line in text.split("\n")] - text = "\n".join(lines) - text = re.sub(r"\n\s*\n\s*\n", "\n\n", text) - text = re.sub(r"\n ", "\n", text) - text = re.sub(r" \n", "\n", text) - return text.strip() - - -def _infer_section(text: str, section_title: str | None = None) -> str: - """Best-effort section heading for chunk provenance.""" - if section_title and str(section_title).strip(): - return str(section_title).strip() - if not text: - return "" - for line in text.splitlines(): - stripped = line.strip() - if stripped.startswith("## "): - return stripped[3:].strip() - if stripped.startswith("# "): - return stripped[2:].strip() - return "" - - -def is_reference_section(text: str) -> bool: - """ - Detect if text is primarily a reference/bibliography section. - Returns True if the text appears to be citations/references. - """ - if not text or len(text) < 50: - return False - - ref_headers = [ - r"^\s*#{1,3}\s*(?:references|bibliography|citations|works cited|literature cited)\s*$", - r"^\s*\*{1,2}(?:references|bibliography)\*{1,2}\s*$", - ] - for pattern in ref_headers: - if re.search(pattern, text, re.IGNORECASE | re.MULTILINE): - return True - - lines = text.split("\n") - total_lines = len([line for line in lines if line.strip()]) - if total_lines == 0: - return False - - citation_patterns = [ - r"^\s*\d{1,3}[\.\)]\s+[A-Z][a-z]+[\s,].*(?:\d{4}|\(\d{4}\))", - r"doi[:\s]*10\.\d{4,}", - r"(?:J\.|Journal|Int\.|Proceedings|Trans\.).*\d{4}", - r"\(\d{4}\)\s*$", - r"\bet\s+al\b", - r"(?:Vol\.?\s*\d+|\d+\s*\(\d+\)\s*:)", - r"(?:pp?\.?\s*\d+[-–]\d+|:\s*\d+[-–]\d+)", - ] - - citation_line_count = 0 - for line in lines: - if not line.strip(): - continue - for pattern in citation_patterns: - if re.search(pattern, line, re.IGNORECASE): - citation_line_count += 1 - break - - return (citation_line_count / total_lines) > 0.4 - - -def _ocr_pdf(local_pdf_path: str) -> list[dict]: - return run_ocr_pdf(local_pdf_path, clean_text) - - -def _ocr_pdf_in_segments( - local_pdf_path: str, - segment_pages: int, - on_segment_complete=None, - completed_page_numbers: set[int] | None = None, -) -> list[dict]: - return run_ocr_pdf_in_segments( - local_pdf_path, - segment_pages, - clean_text, - on_segment_complete=on_segment_complete, - completed_page_numbers=completed_page_numbers, - ) - - -def clean_text_for_ingestion(text: str) -> str: - """Clean translation preambles from text before ingestion.""" - if not text: - return text - - result = text - result = re.sub( - r"^Here is the translated text from \*\*[^*]+\*\* to English[^:]*:?\s*\n*-{0,3}\s*\n*", - "", - result, - flags=re.IGNORECASE, - ) - result = re.sub( - r"^Here is the translated text from [^:]+?:\s*\n*-{0,3}\s*\n*", - "", - result, - flags=re.IGNORECASE, - ) - result = re.sub( - r"^Here is the translated text[^:]*:?\s*\n*-{0,3}\s*\n*", - "", - result, - flags=re.IGNORECASE, - ) - result = re.sub( - r"^Here is the (?:English )?translation[^:]*:?\s*\n*", - "", - result, - flags=re.IGNORECASE | re.MULTILINE, - ) - result = re.sub( - r"^Here is the translated text with[^:]+:?\s*\n*-{0,3}\s*\n*", - "", - result, - flags=re.IGNORECASE, - ) - - prefixes = [ - r"^(?:the\s+)?english\s+translation:?\s*\n*", - r"^(?:the\s+)?translation:?\s*\n*", - r"^translated\s+(?:text|content):?\s*\n*", - r"^##?\s*(?:english\s+)?translation\s*\n+", - r"^---+\s*\n+", - r"^\*\*Translation:?\*\*\s*\n*", - ] - for pattern in prefixes: - result = re.sub(pattern, "", result, flags=re.IGNORECASE | re.MULTILINE) - - result = re.sub(r"\n*-{3,}\s*$", "", result) - return result.strip() - - -def _normalize_instance(value: str | None) -> str: - """Ingest-side instance normalizer (mirrors auth.tenancy without importing FastAPI).""" - text = (value or "").strip().lower() - if text: - return text - return (os.environ.get("DEFAULT_INSTANCE") or "default").strip().lower() or "default" - - -def _prepare_records( - document_id: str, - filename: str, - chunks: list[dict], - workflow_id: str | None = None, - name_gu: str | None = None, - name_en: str | None = None, - description: str | None = None, - include_e5_prefix_field: bool = True, - instance: str | None = None, - kind: str = "document", - code: str | None = None, - name: str | None = None, - aliases: list[str] | None = None, - valid_from: str | None = None, - valid_to: str | None = None, - provider_id: str | None = None, - display_name: str | None = None, -) -> list[dict]: - metadata = _get_doc_metadata(filename) - resolved_instance = _normalize_instance(instance) - llm_doc_description = _get_doc_description(filename) - - doc_hash = hashlib.md5(document_id.encode()).hexdigest() - external_slug = workflow_id or filename - default_name = filename.replace(".pdf", "").replace(".PDF", "") - name_gu = name_gu or metadata.get("title_gu") or default_name - name_en = name_en or display_name or metadata.get("title_en") or default_name - category_tags = metadata.get("category_tags", "") - quality_score = metadata.get("quality_score", "") - priority_rank = metadata.get("priority_rank", "") - ingestion_status = metadata.get("ingestion_status", "") - doc_language = metadata.get("doc_language", "") - short_description = metadata.get("doc_short_description", "") - effective_description = description or llm_doc_description or short_description - - kind = (kind or "document").strip().lower() - is_scheme = kind == "scheme" and bool((code or "").strip()) - resolved_name = name or name_en or default_name - # Merge curator-supplied, bootstrap and derived aliases so a scheme is never - # indexed with an empty alias list (it would then only match its exact name). - resolved_aliases = ( - scheme_catalog.resolve_scheme_aliases(code, resolved_name, aliases) - if is_scheme - else [] - ) - instance_name = instance_display_name(resolved_instance) - # One period for the whole document, resolved before the loop: every chunk - # of a document expires together, and re-deriving it per chunk would let a - # midnight boundary split one document across two periods. - validity = document_validity.period_from_row(valid_from, valid_to) - - records = [] - for chunk in chunks: - if chunk.get("is_excluded", False): - continue - - raw_text = chunk.get("edited_text") or chunk.get("original_text", "") - chunk_num = chunk.get("chunk_number", 0) - text = clean_text_for_ingestion(raw_text) - is_ref = is_reference_section(text) - - section = _infer_section(text, chunk.get("section_title")) - point_id = hashlib.md5(f"{doc_hash}_{chunk_num}_{text[:50]}".encode()).hexdigest() - record = { - "_id": point_id, - "doc_id": document_id, - "workflow_id": workflow_id or "", - "instance": resolved_instance, # vector payload key, kept for consumers - "instance_name": instance_name, - "provider_id": provider_id, - "type": kind, - "source": "docs-pipeline", - "filename": external_slug, - "name_gu": name_gu, - "name_en": name_en, - "title_en": metadata.get("title_en", ""), - "title_gu": metadata.get("title_gu", ""), - "doc_language": doc_language, - "category_tags": category_tags, - "doc_short_description": short_description, - "doc_llm_description": llm_doc_description, - "ingestion_status": ingestion_status, - "description": effective_description, - "text": text, - "chunk_num": chunk_num, - "chunk_id": point_id, - "chunk_index": chunk_num, - "section": section, - "token_count": chunk.get("token_count", 0), - "page_start": chunk.get("page_start", 1), - "page_end": chunk.get("page_end", 1), - "is_reference": is_ref, - "quality_score": float(quality_score) if str(quality_score).strip().replace(".", "", 1).isdigit() else 0.0, - "priority_rank": float(priority_rank) if str(priority_rank).strip().replace(".", "", 1).isdigit() else 0.0, - } - if validity is not None: - record["start_date"] = validity.start_date - # Omitted, not null, when the document never expires: the search - # filter's "end_date missing" branch is what keeps it answerable - # forever, and _record_payload drops None anyway. - if validity.end_date is not None: - record["end_date"] = validity.end_date - if is_scheme: - record["scheme_code"] = (code or "").strip().lower() - record["scheme_name"] = resolved_name - record["scheme_aliases"] = resolved_aliases - if include_e5_prefix_field: - record["text_for_embedding"] = f"passage: {text}" if text else "passage:" - records.append(record) - - return records - - -def _scheme_fields_from_doc(doc: dict | None) -> dict: - """Extract kind/code/name/aliases kwargs for _prepare_records from a documents row.""" - doc = doc or {} - return { - "kind": (doc.get("kind") or "document"), - "code": doc.get("code"), - "name": doc.get("name"), - "aliases": [str(a) for a in (doc.get("aliases") or [])], - } - - -def _provider_id_from_doc(doc: dict | None) -> str | None: - doc = doc or {} - group = named_group(group_for_instance(doc.get("instance"))) - try: - provider = provider_for(load_provider_naming(), doc.get("kind"), group) - except InvalidProviderIdError as exc: - raise ApplicationError(str(exc), non_retryable=True) from exc - return provider["id"] if provider else None - - -def _validity_fields_from_doc(doc: dict | None) -> dict: - """Extract the validity kwargs for `_prepare_records` from a documents row. - - A row with no stored period is stamped with the default anchored on its - upload day - start there, no end - so every chunk written from here on - carries a start. Old points already in the index keep none until their - document is reingested, which is what keeps them searchable meanwhile. - """ - doc = doc or {} - period = document_validity.period_from_row( - doc.get("valid_from"), doc.get("valid_to") - ) - if period is None: - period = document_validity.period_from_upload_timestamp(doc.get("created_at")) - return {"valid_from": period.start_date, "valid_to": period.end_date} - - -def prepare_ingestion_records( - document_id: str, - filename: str, - chunks: list[dict], - workflow_id: str | None = None, - name_gu: str | None = None, - name_en: str | None = None, - description: str | None = None, - instance: str | None = None, -) -> list[dict]: - """Public helper used by tests and scripts when preparing vector-index payloads.""" - return _prepare_records( - document_id, - filename, - chunks, - workflow_id=workflow_id, - name_gu=name_gu, - name_en=name_en, - description=description, - instance=instance, - ) - - -def _passage_schema_field_names() -> set[str]: - """Field names for the canonical passage schema (E5 text_for_embedding + full metadata).""" - settings = _passage_schema_definition(use_tensor_prefix_field=True) - return {f.get("name") for f in settings.get("allFields", []) if isinstance(f, dict) and f.get("name")} - - -def _passage_schema_definition(use_tensor_prefix_field: bool = True) -> dict: - tensor_field = "text_for_embedding" if use_tensor_prefix_field else "text" - all_fields = [ - {"name": "doc_id", "type": "text", "features": ["filter"]}, - {"name": "workflow_id", "type": "text", "features": ["filter"]}, - {"name": "instance", "type": "text", "features": ["filter"]}, - {"name": "instance_name", "type": "text", "features": ["filter"]}, - {"name": "provider_id", "type": "text", "features": ["filter"]}, - {"name": "type", "type": "text", "features": ["filter"]}, - {"name": "source", "type": "text", "features": ["filter"]}, - {"name": "filename", "type": "text", "features": ["filter"]}, - {"name": "name_gu", "type": "text", "features": ["filter"]}, - {"name": "name_en", "type": "text", "features": ["filter"]}, - {"name": "title_en", "type": "text", "features": ["filter"]}, - {"name": "title_gu", "type": "text", "features": ["filter"]}, - {"name": "doc_language", "type": "text", "features": ["filter"]}, - {"name": "category_tags", "type": "text", "features": ["filter"]}, - {"name": "doc_short_description", "type": "text", "features": ["filter"]}, - {"name": "doc_llm_description", "type": "text", "features": ["filter"]}, - {"name": "ingestion_status", "type": "text", "features": ["filter"]}, - {"name": "description", "type": "text", "features": ["lexical_search"]}, - {"name": "chunk_num", "type": "int", "features": ["filter"]}, - {"name": "section", "type": "text", "features": ["filter"]}, - {"name": "token_count", "type": "int", "features": ["filter"]}, - {"name": "page_start", "type": "int", "features": ["filter"]}, - {"name": "page_end", "type": "int", "features": ["filter"]}, - {"name": "is_reference", "type": "bool", "features": ["filter"]}, - {"name": "quality_score", "type": "float", "features": ["filter"]}, - {"name": "priority_rank", "type": "float", "features": ["filter"]}, - {"name": "text", "type": "text", "features": ["lexical_search"]}, - {"name": "priority", "type": "float", "features": ["score_modifier", "filter"]}, - # Document validity - filtered on at search time, so the day a chunk - # starts and stops answering travels with the chunk itself. - {"name": "start_date", "type": "date", "features": ["filter"]}, - {"name": "end_date", "type": "date", "features": ["filter"]}, - ] - if use_tensor_prefix_field: - all_fields.append({"name": "text_for_embedding", "type": "text"}) - - return { - "type": "structured", - "vectorNumericType": "float", - "model": "hf/multilingual-e5-large", - "normalizeEmbeddings": False, - "textPreprocessing": {"splitLength": 3, "splitOverlap": 1, "splitMethod": "sentence"}, - "allFields": all_fields, - "tensorFields": [tensor_field], - } - - -async def _detect_and_translate_impl( - pages: list[dict], - target_language: str = "en", - source_language: str | None = None, -) -> list[dict]: - del source_language - config = load_translation_config(target_language=target_language) - return await translate_pages( - pages, - target_language=target_language, - config=config, - log=activity.logger.info, - ) - - -# ============================================================================= -# Activities -# ============================================================================= - -@activity.defn -async def run_ocr(filepath: str) -> list[dict]: - """Run OCR on a supported file and return page dicts.""" - activity.logger.info(f"Running OCR on {filepath}") - - local_path, cleanup_local = _resolve_local_path(filepath) - ext = Path(local_path).suffix.lower() - pdf_path = local_path - cleanup_pdf_dir = False - - try: - if ext in DELIMITED_INPUT_EXTENSIONS: - pages = _csv_to_pages(local_path) - activity.logger.info(f"CSV parsed into {len(pages)} pages") - return pages - if ext in NATIVE_SPREADSHEET_EXTENSIONS: - pages = _xlsx_to_pages(local_path) - activity.logger.info(f"XLSX parsed into {len(pages)} pages") - return pages - pdf_path, cleanup_pdf_dir = _ensure_pdf_input(local_path) - return _ocr_pdf(pdf_path) - finally: - if cleanup_pdf_dir: - try: - shutil.rmtree(Path(pdf_path).parent, ignore_errors=True) - except Exception: - pass - if cleanup_local and os.path.exists(local_path): - os.remove(local_path) - - -@activity.defn -async def run_ocr_and_store(workflow_id: str, filepath: str) -> dict: - """Run OCR and persist pages to Postgres to avoid large Temporal payloads.""" - from . import db - - local_path, cleanup_local = _resolve_local_path(filepath) - ext = Path(local_path).suffix.lower() - source_type, canonical_input_type = _source_type_from_path(local_path) - original_filename = Path(local_path).name - normalized_path = local_path - cleanup_normalized = False - segment_pages = max( - 1, - int(os.environ.get("OCR_SEGMENT_PAGES", "20")), - ) - - try: - if ext in DELIMITED_INPUT_EXTENSIONS: - pages = _csv_to_pages(local_path) - elif ext in NATIVE_SPREADSHEET_EXTENSIONS: - pages = _xlsx_to_pages(local_path) - else: - normalized_path, cleanup_normalized = _ensure_pdf_input(local_path) - saved_page_numbers = set(db.get_saved_page_numbers(workflow_id)) - loop = asyncio.get_running_loop() - - def persist_segment(segment_pages_result: list[dict], total_pages: int) -> None: - db.save_pages(workflow_id, segment_pages_result) - current_saved = len(saved_page_numbers.union({p["page_number"] for p in segment_pages_result})) - saved_page_numbers.update(p["page_number"] for p in segment_pages_result) - db.update_document_fields(workflow_id, page_count=current_saved) - # Heartbeat must run on the activity event loop (persist_segment runs in to_thread). - loop.call_soon_threadsafe( - activity.heartbeat, - {"workflow_id": workflow_id, "pages_saved": current_saved, "total_pages": total_pages}, - ) - activity.logger.info( - "Persisted OCR segment for %s: %s/%s pages saved", - workflow_id, - current_saved, - total_pages, - ) - - pages = await asyncio.to_thread( - _ocr_pdf_in_segments, - normalized_path, - segment_pages=segment_pages, - on_segment_complete=persist_segment, - completed_page_numbers=saved_page_numbers, - ) - - pages = db.get_pages(workflow_id) - - db.save_pages(workflow_id, pages) - - latest_job = db.get_latest_document_job(workflow_id) - job_id = latest_job["id"] if latest_job else None - - normalized_filename = _normalized_filename(original_filename, canonical_input_type) - normalized_uri, normalized_size, normalized_mime = _upload_file_to_minio( - normalized_path, - workflow_id, - "normalized_spreadsheet" if canonical_input_type == "spreadsheet" else "normalized_pdf", - normalized_filename, - ) - normalized_artifact_id = db.add_document_artifact( - workflow_id=workflow_id, - job_id=job_id, - artifact_type="normalized_spreadsheet" if canonical_input_type == "spreadsheet" else "normalized_pdf", - stage="ocr_processing", - storage_uri=normalized_uri, - mime_type=normalized_mime, - filename=normalized_filename, - size_bytes=normalized_size, - metadata={"source_filepath": filepath, "canonical_input_type": canonical_input_type}, - ) - - if filepath.startswith("minio://"): - original_uri = filepath - original_size = None - original_mime = mimetypes.guess_type(original_filename)[0] or "application/octet-stream" - else: - original_uri, original_size, original_mime = _upload_file_to_minio( - local_path, - workflow_id, - "original_upload", - original_filename, - ) - original_artifact_id = db.add_document_artifact( - workflow_id=workflow_id, - job_id=job_id, - artifact_type="original_upload", - stage="registered", - storage_uri=original_uri, - mime_type=original_mime, - filename=original_filename, - size_bytes=original_size, - metadata={"source_filepath": filepath}, - ) - - pages_json_path = _write_json_temp(pages) - try: - pages_uri, pages_size, pages_mime = _upload_file_to_minio( - pages_json_path, workflow_id, "ocr_pages_json", "pages.json" - ) - finally: - if os.path.exists(pages_json_path): - os.remove(pages_json_path) - - db.add_document_artifact( - workflow_id=workflow_id, - job_id=job_id, - artifact_type="ocr_pages_json", - stage="ocr_review", - storage_uri=pages_uri, - mime_type=pages_mime, - filename="pages.json", - size_bytes=pages_size, - metadata={"page_count": len(pages)}, - ) - - db.update_document_fields( - workflow_id, - page_count=len(pages), - source_type=source_type, - canonical_input_type=canonical_input_type, - original_artifact_id=original_artifact_id, - normalized_artifact_id=normalized_artifact_id, - ) - return {"page_count": len(pages), "normalized_artifact_id": normalized_artifact_id} - finally: - if cleanup_normalized and os.path.exists(normalized_path): - shutil.rmtree(Path(normalized_path).parent, ignore_errors=True) - if cleanup_local and os.path.exists(local_path): - os.remove(local_path) - - -@activity.defn -async def create_chunks( - pages: list[dict], - chunk_size: int = 450, - chunk_overlap: int = 128, - min_tokens: int = 100, -) -> list[dict]: - """Create chunks from pages with page range tracking.""" - activity.logger.info(f"Creating chunks from {len(pages)} pages") - config = load_chunking_config(chunk_size=chunk_size, chunk_overlap=chunk_overlap, min_tokens=min_tokens) - result = await chunk_pages(pages, config) - chunks = [] - for idx, chunk in enumerate(result.chunks, 1): - chunks.append( - { - "chunk_number": idx, - "original_text": chunk.text, - "edited_text": None, - "token_count": chunk.token_count, - "page_start": chunk.page_start, - "page_end": chunk.page_end, - "source_page_numbers_json": json.dumps(chunk.source_page_numbers), - "source_spans_json": json.dumps(chunk.source_spans), - "section_title": chunk.section_title, - "content_type": chunk.content_type, - "is_reference": chunk.is_reference, - "chunking_provider": result.provider, - "chunking_model": result.model, - "chunking_config_json": result.config.to_json(), - "chunking_run_id": "", - "chunk_version": 1, - "is_reviewed": False, - "is_excluded": False, - "reviewer_notes": None, - } - ) - activity.logger.info(f"Created {len(chunks)} chunks with page tracking") - return chunks - - -@activity.defn -async def create_chunks_from_db( - workflow_id: str, - chunk_size: int = 450, - chunk_overlap: int = 128, - min_tokens: int = 100, -) -> dict: - """Create chunks from persisted pages and persist chunks in Postgres.""" - from . import db - - pages = db.get_pages(workflow_id) - activity.logger.info(f"Creating chunks from DB pages for {workflow_id}: {len(pages)} pages") - config = load_chunking_config(chunk_size=chunk_size, chunk_overlap=chunk_overlap, min_tokens=min_tokens) - latest_job = db.get_latest_document_job(workflow_id) - base_job_config = {} - if latest_job and latest_job.get("config_json"): - try: - base_job_config = json.loads(latest_job["config_json"]) or {} - except Exception: - base_job_config = {} - - async def _persist_chunking_progress(event: dict) -> None: - if not latest_job: - return - pages_total = int(event.get("pages_total") or len(pages) or 0) - pages_processed = int(event.get("pages_processed") or 0) - chunks_emitted = int(event.get("chunks_emitted") or 0) - raw_percent = float(event.get("percent") or 0.0) - percent = max(0.0, min(100.0, raw_percent)) - progress = { - "status": "running" if percent < 100.0 else "completed", - "provider": event.get("provider") or config.provider, - "pages_processed": pages_processed, - "pages_total": pages_total, - "chunks_emitted": chunks_emitted, - "percent": round(percent, 2), - "updated_at": datetime.utcnow().isoformat(), - } - next_config = {**base_job_config, "chunking_progress": progress} - db.update_document_job(latest_job["id"], config_json=next_config) - - await _persist_chunking_progress( - { - "provider": config.provider, - "pages_processed": 0, - "pages_total": len(pages), - "chunks_emitted": 0, - "percent": 0.0, - } - ) - - result = await chunk_pages(pages, config, progress_callback=_persist_chunking_progress) - chunking_run_id = f"chunk-{workflow_id}-{datetime.utcnow().strftime('%Y%m%d%H%M%S')}" - chunks = [] - for idx, chunk in enumerate(result.chunks, 1): - chunks.append( - { - "chunk_number": idx, - "original_text": chunk.text, - "edited_text": None, - "token_count": chunk.token_count, - "page_start": chunk.page_start, - "page_end": chunk.page_end, - "source_page_numbers_json": json.dumps(chunk.source_page_numbers), - "source_spans_json": json.dumps(chunk.source_spans), - "section_title": chunk.section_title, - "content_type": chunk.content_type, - "is_reference": chunk.is_reference, - "chunking_provider": result.provider, - "chunking_model": result.model, - "chunking_config_json": result.config.to_json(), - "chunking_run_id": chunking_run_id, - "is_reviewed": False, - "is_excluded": False, - "reviewer_notes": None, - } - ) - db.save_chunks(workflow_id, chunks) - chunks_json_path = _write_json_temp(chunks) - try: - chunks_uri, chunks_size, chunks_mime = _upload_file_to_minio( - chunks_json_path, workflow_id, "chunk_json_export", "chunks.json" - ) - finally: - if os.path.exists(chunks_json_path): - os.remove(chunks_json_path) - db.add_document_artifact( - workflow_id=workflow_id, - job_id=latest_job["id"] if latest_job else None, - artifact_type="chunk_json_export", - stage="chunk_review", - storage_uri=chunks_uri, - mime_type=chunks_mime, - filename="chunks.json", - size_bytes=chunks_size, - metadata={ - "chunk_count": len(chunks), - "chunking_provider": result.provider, - "chunking_model": result.model, - "chunking_run_id": chunking_run_id, - "chunking_config": json.loads(result.config.to_json()), - "warnings": result.warnings, - "stats": result.stats, - }, - ) - # Reconcile the document row immediately after chunk persistence so Postgres - # remains truthful even if the workflow fails before its final state update. - db.reconcile_materialized_state(workflow_id) - await _persist_chunking_progress( - { - "provider": result.provider, - "pages_processed": len(pages), - "pages_total": len(pages), - "chunks_emitted": len(chunks), - "percent": 100.0, - } - ) - return {"chunk_count": len(chunks)} - - -@activity.defn -async def prepare_for_ingestion( - document_id: str, - filename: str, - chunks: list[dict], - workflow_id: str | None = None, - name_gu: str = None, - name_en: str = None, - description: str = None, -) -> list[dict]: - """Prepare chunks for vector-index ingestion.""" - from . import db - - activity.logger.info(f"Preparing {len(chunks)} chunks for ingestion") - doc = db.get_document(workflow_id) if workflow_id else None - records = _prepare_records( - document_id, - filename, - chunks, - workflow_id=workflow_id, - name_gu=name_gu, - name_en=name_en, - description=description, - instance=(doc or {}).get("instance"), - **_validity_fields_from_doc(doc), - provider_id=_provider_id_from_doc(doc), - display_name=(doc or {}).get("name"), - ) - activity.logger.info(f"Prepared {len(records)} records") - return records - - -@activity.defn -async def ingest_to_vector_db( - records: list[dict], - index_name: str = "documents-index", - batch_size: int = 10, -) -> dict: - """Ingest records into the Qdrant vector index.""" - from .vector_store import get_default_index_name, get_vector_store - - if not index_name or index_name == "documents-index": - index_name = get_default_index_name() or index_name - - activity.logger.info( - "Ingesting %s records to Qdrant collection %s", - len(records), - index_name, - ) - store = get_vector_store() - # store.upsert() runs CPU-bound embedding synchronously (model.encode()) — - # off the event loop so it doesn't stall Temporal's query/heartbeat handling. - result = await asyncio.to_thread( - store.upsert, index_name, records, batch_size=max(batch_size, 8) - ) - activity.logger.info("Qdrant ingestion complete: %s", result.get("index_stats")) - return result - - -@activity.defn -async def promote_document_to_prod_qdrant( - workflow_id: str, - document_id: str, - filename: str, - batch_size: int = 10, -) -> dict: - """Re-upsert document chunks into the PROD Qdrant collection (from Postgres).""" - from . import db - from .vector_store.qdrant_store import QdrantVectorStore, get_qdrant_client - - prod_url = (os.environ.get("PROD_VECTOR_DB_URL") or "").strip() - prod_key = (os.environ.get("PROD_VECTOR_DB_API_KEY") or "").strip() or None - prod_collection = ( - os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME") - or os.environ.get("VECTOR_DB_COLLECTION_NAME") - or "documents-index" - ).strip() - if not prod_url: - raise RuntimeError("PROD_VECTOR_DB_URL is required to promote documents to prod") - - timeout = float( - os.environ.get("PROD_VECTOR_DB_TIMEOUT_SECONDS") - or os.environ.get("VECTOR_DB_TIMEOUT_SECONDS") - or "30" - ) - # Temporarily honor prod timeout for this client build via env used by get_qdrant_client - prev_timeout = os.environ.get("VECTOR_DB_TIMEOUT_SECONDS") - os.environ["VECTOR_DB_TIMEOUT_SECONDS"] = str(timeout) - try: - client = get_qdrant_client(url=prod_url, api_key=prod_key) - finally: - if prev_timeout is None: - os.environ.pop("VECTOR_DB_TIMEOUT_SECONDS", None) - else: - os.environ["VECTOR_DB_TIMEOUT_SECONDS"] = prev_timeout - - chunks = db.get_chunks(workflow_id, include_excluded=True) - doc = db.get_document(workflow_id) or {} - scheme_kwargs = _scheme_fields_from_doc(doc) - kind = (scheme_kwargs.get("kind") or "document").strip().lower() - is_scheme = kind == "scheme" and bool((scheme_kwargs.get("code") or "").strip()) - - if is_scheme: - prod_url = ( - (os.environ.get("PROD_SCHEME_VECTOR_DB_URL") or "").strip() - or prod_url - ) - prod_key = ( - (os.environ.get("PROD_SCHEME_VECTOR_DB_API_KEY") or "").strip() - or prod_key - ) - prod_collection = ( - os.environ.get("PROD_SCHEME_VECTOR_DB_COLLECTION_NAME") or "schemes-index" - ).strip() - docs_default = ( - os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME") - or os.environ.get("VECTOR_DB_COLLECTION_NAME") - or "documents-index" - ).strip() - if prod_collection == docs_default: - raise RuntimeError( - "Scheme promote refused: PROD_SCHEME_VECTOR_DB_COLLECTION_NAME must not " - f"equal documents collection ({docs_default}). Set schemes-index." - ) - # Rebuild client if scheme URL differs - scheme_url = (os.environ.get("PROD_SCHEME_VECTOR_DB_URL") or "").strip() - if scheme_url and scheme_url != (os.environ.get("PROD_VECTOR_DB_URL") or "").strip(): - client = get_qdrant_client(url=scheme_url, api_key=prod_key) - - records = _prepare_records( - document_id, - filename, - chunks, - workflow_id=workflow_id, - instance=doc.get("instance"), - **scheme_kwargs, - **_validity_fields_from_doc(doc), - provider_id=_provider_id_from_doc(doc), - display_name=doc.get("name"), - ) - activity.logger.info( - "Promoting %s records to PROD Qdrant collection %s (kind=%s)", - len(records), - prod_collection, - kind, - ) - store = QdrantVectorStore(client=client) - result = await asyncio.to_thread( - store.upsert, prod_collection, records, batch_size=max(batch_size, 8) - ) - activity.logger.info("PROD promotion complete: %s", result.get("index_stats")) - await _drop_superseded_points("prod", workflow_id, doc, document_id, prod_collection, store=store) - db.log_audit( - workflow_id=workflow_id, - document_id=document_id, - action_type="promote_to_prod", - entity_type="document", - metadata={ - "collection": prod_collection, - "records_ingested": result.get("records_ingested", len(records)), - "prod_qdrant_url": prod_url, - "kind": kind, - "code": scheme_kwargs.get("code"), - }, - ) - catalog_version = None - if is_scheme: - try: - from .scheme_catalog import on_scheme_document_promoted - - catalog_version = on_scheme_document_promoted(workflow_id) - except Exception as exc: - activity.logger.warning("Catalog update after promote failed: %s", exc) - return { - **result, - "collection": prod_collection, - "target": "prod", - "document_kind": kind, - "catalog_version": catalog_version, - } - - -async def _drop_superseded_points( - env: str, - workflow_id: str, - doc: dict, - current_doc_id: str, - collection: str, - store=None, -) -> None: - """Delete points of earlier file versions (see ``metadata.superseded_doc_ids``). - - Runs after the new content is indexed, so search never has a gap. A failed - delete keeps the key for the next run and does not fail the ingest. - """ - from . import db - - stale = [ - doc_id - for doc_id in (doc.get("superseded_doc_ids") or {}).get(env, []) - if doc_id and doc_id != current_doc_id - ] - if not stale: - return - if store is None: - from .vector_store import get_vector_store - - store = get_vector_store() - deleted = [] - for doc_id in stale: - # Two documents holding the same file share one index key. While another - # live document is indexed under it the points stay, and the key stays on - # the list so a later ingest, disable or purge can still remove it. - if await asyncio.to_thread(db.doc_id_used_by_other_document, doc_id, doc.get("id"), env): - activity.logger.info( - "Keeping %s points for %s: another document is indexed under the same key", env, doc_id - ) - continue - result = await asyncio.to_thread(store.delete_by_doc_id, collection, doc_id) - if result.get("error"): - activity.logger.warning( - "Could not delete superseded %s points for %s: %s", env, doc_id, result["error"] - ) - else: - deleted.append(doc_id) - if deleted: - db.clear_superseded_doc_ids(workflow_id, env, deleted) - - -@activity.defn -async def ingest_document_from_db( - workflow_id: str, - document_id: str, - filename: str, - index_name: str = "documents-index", - batch_size: int = 10, -) -> dict: - """Prepare and ingest chunks directly from Postgres by workflow_id.""" - from . import db - - chunks = db.get_chunks(workflow_id, include_excluded=True) - doc = db.get_document(workflow_id) or {} - records = _prepare_records( - document_id, - filename, - chunks, - workflow_id=workflow_id, - instance=doc.get("instance"), - **_scheme_fields_from_doc(doc), - **_validity_fields_from_doc(doc), - provider_id=_provider_id_from_doc(doc), - display_name=doc.get("name"), - ) - payload_path = _write_json_temp(records) - try: - payload_uri, payload_size, payload_mime = _upload_file_to_minio( - payload_path, workflow_id, "vector_payload_export", "vector_payload.json" - ) - finally: - if os.path.exists(payload_path): - os.remove(payload_path) - latest_job = db.get_latest_document_job(workflow_id) - db.add_document_artifact( - workflow_id=workflow_id, - job_id=latest_job["id"] if latest_job else None, - artifact_type="vector_payload_export", - stage="ingesting", - storage_uri=payload_uri, - mime_type=payload_mime, - filename="vector_payload.json", - size_bytes=payload_size, - metadata={"record_count": len(records), "index_name": index_name}, - ) - result = await ingest_to_vector_db(records, index_name=index_name, batch_size=batch_size) - db.upsert_document_index_status( - workflow_id=workflow_id, - index_name=index_name, - doc_id=document_id, - chunk_count_indexed=result.get("records_ingested", 0), - last_indexed_at=datetime.utcnow().isoformat(), - last_verified_at=datetime.utcnow().isoformat(), - schema_version="passage-v1", - status="indexed", - details=result.get("index_stats"), - ) - await _drop_superseded_points("dev", workflow_id, doc, document_id, index_name) - return result - - -@activity.defn -async def publish_catalog_to_network(workflow_id: str, transaction_id: str) -> dict: - """POST a catalog/publish envelope to the Discovery Service and record the exchange. - - The catalog depends on the document's knowledge kind, read here rather than - passed in as an activity argument - that keeps every workflow call site, and - any workflow already in flight, untouched. - """ - from . import db - - repository = DocumentRepository() - document_kind = normalize_document_kind(repository.get_document_kind(workflow_id)) - - if publish_to_network_enabled(): - group = repository.get_document_group(workflow_id) - try: - service = DiscoveryPublishService() - except PublishConfigError as exc: - raise ApplicationError(str(exc), non_retryable=True) from exc - try: - result = await asyncio.to_thread( - service.publish, transaction_id, document_kind, workflow_id, group - ) - except InvalidProviderIdError as exc: - raise ApplicationError(str(exc), non_retryable=True) from exc - skip = (PUBLISH_SKIPPED_UNMAPPED_KIND, f"No network catalog is mapped to document kind '{document_kind}'.") if result["skipped"] else None - else: - result = {"skipped": True, "result_status": None} - skip = (PUBLISH_SKIPPED_DISABLED, "Publish to Network is disabled (ENABLE_PUBLISH_TO_NETWORK is not true).") - - if skip: - artifact_type = "network_publish_skipped" - filename = "network_publish_skipped.json" - exchange = { - "skipped": True, - "document_kind": document_kind, - "reason_code": skip[0], - "reason": skip[1], - } - else: - artifact_type = "network_publish_payload" - filename = "network_publish.json" - exchange = { - "request": result["envelope"], - "response_status": result["status_code"], - "response_body": result["response_body"], - "result_status": result["result_status"], - "errors": result["errors"], - } - - # Recorded even when skipped: add_document_artifact requires a storage_uri, - # and an empty one 404s the artifact-content endpoint if an operator clicks it. - exchange_path = _write_json_temp(exchange) - try: - exchange_uri, exchange_size, exchange_mime = _upload_file_to_minio( - exchange_path, workflow_id, artifact_type, filename - ) - finally: - if os.path.exists(exchange_path): - os.remove(exchange_path) - - latest_job = db.get_latest_document_job(workflow_id) - db.add_document_artifact( - workflow_id=workflow_id, - job_id=latest_job["id"] if latest_job else None, - artifact_type=artifact_type, - stage="publishing_to_network", - storage_uri=exchange_uri, - mime_type=exchange_mime, - filename=filename, - size_bytes=exchange_size, - metadata=exchange, - ) - - if result["result_status"] == RESULT_REJECTED: - # Recorded before the raise: a raise records no artifact, and the - # errors[] is the only explanation an operator gets. Non-retryable - # because a rejection is deterministic. - raise ApplicationError( - f"Discovery Service rejected catalog for {document_kind}: " - f"{result['errors']}", - non_retryable=True, - ) - - return { - "status": "skipped" if skip else "published", - "reason_code": skip[0] if skip else None, - "document_kind": document_kind, - "transaction_id": transaction_id, - "message_id": None if skip else result["envelope"]["context"]["messageId"], - "result_status": result["result_status"], - } - - -@activity.defn -async def update_document_state( - workflow_id: str, - stage: str, - page_count: int = 0, - chunk_count: int = 0, - error_message: str = None, -) -> dict: - """Update document state in Postgres.""" - from . import db - - activity.logger.info(f"Updating state for {workflow_id}: stage={stage}") - db.update_document_stage( - workflow_id=workflow_id, - stage=stage, - page_count=page_count, - chunk_count=chunk_count, - error_message=error_message, - ) - latest_job = db.get_latest_document_job(workflow_id) - if latest_job: - job_updates = {"current_stage": stage} - if stage in { - "ocr_review", - "translation_review", - "chunk_review", - "ready_for_ingestion", - "approval_for_prod", - }: - job_updates["status"] = "waiting_review" - elif stage == "completed": - job_updates["status"] = "completed" - job_updates["completed_at"] = datetime.utcnow().isoformat() - elif stage == "failed": - job_updates["status"] = "failed" - job_updates["completed_at"] = datetime.utcnow().isoformat() - job_updates["error_message"] = error_message - else: - job_updates["status"] = "running" - db.update_document_job(latest_job["id"], **job_updates) - return {"updated": True, "stage": stage} - - -@activity.defn -async def persist_document_content(workflow_id: str, pages: list[dict], chunks: list[dict]) -> dict: - """Persist pages and chunks to Postgres.""" - from . import db - - activity.logger.info(f"Persisting content for {workflow_id}: {len(pages)} pages, {len(chunks)} chunks") - db.save_pages(workflow_id, pages) - db.save_chunks(workflow_id, chunks) - return {"persisted": True, "pages": len(pages), "chunks": len(chunks)} - - -@activity.defn -async def auto_tag_chunks_from_db(workflow_id: str, filename: str = "") -> dict: - """No-op retained so existing workflow histories that scheduled this - activity before the auto-tag-v1 patch continue to replay deterministically. - """ - return {"tagged_chunks": 0, "skipped": True} - - -@activity.defn -async def detect_and_translate_pages( - pages: list[dict], - target_language: str = "en", - source_language: str = None, -) -> list[dict]: - """Detect language and translate non-English pages.""" - return await _detect_and_translate_impl(pages, target_language=target_language, source_language=source_language) - - -@activity.defn -async def detect_and_translate_pages_from_db( - workflow_id: str, - target_language: str = "en", - source_language: str = None, -) -> dict: - """Detect and translate pages loaded from Postgres; persist updated pages back to Postgres.""" - from . import db - - pages = db.get_pages(workflow_id) - translated = await _detect_and_translate_impl(pages, target_language=target_language, source_language=source_language) - db.save_pages(workflow_id, translated) - translated_count = sum(1 for p in translated if p.get("translated_markdown")) - latest_job = db.get_latest_document_job(workflow_id) - translation_config = load_translation_config(target_language=target_language) - translated_json_path = _write_json_temp(translated) - try: - translated_uri, translated_size, translated_mime = _upload_file_to_minio( - translated_json_path, workflow_id, "translation_pages_json", "translated_pages.json" - ) - finally: - if os.path.exists(translated_json_path): - os.remove(translated_json_path) - db.add_document_artifact( - workflow_id=workflow_id, - job_id=latest_job["id"] if latest_job else None, - artifact_type="translation_pages_json", - stage="translation_review", - storage_uri=translated_uri, - mime_type=translated_mime, - filename="translated_pages.json", - size_bytes=translated_size, - metadata={ - "page_count": len(translated), - "translated_count": translated_count, - "translation_provider": translation_config.provider, - "translation_model": translation_config.model, - "translation_target_language": target_language, - "translation_run_id": str(uuid4()), - }, - ) - return {"page_count": len(translated), "translated_count": translated_count} - - -async def _run_advisory_activity(record_id: str, promote: bool) -> dict: - """Index (dev) or promote (PROD) one webhook advisory record; shared by both activities.""" - from . import advisory_index - from .vector_store import get_default_index_name, get_vector_store - from .vector_store.embeddings import count_tokens - - also_delete_from = None - if promote: - target = advisory_index.prod_target() - if target is None: - raise RuntimeError("PROD_VECTOR_DB_URL is required to promote advisory to prod") - store, collection = target - mark = "promoted" - else: - store, collection = get_vector_store(), get_default_index_name() - mark = "indexed" - try: - # An unpublished record must also disappear from PROD (deletion only). - also_delete_from = advisory_index.prod_target() - except Exception as exc: - activity.logger.warning("PROD target unavailable for advisory cleanup: %s", exc) - - # Embedding is CPU-bound and synchronous: keep it off the event loop. - result = await asyncio.to_thread( - advisory_index.index_advisory_record, - record_id, - store, - collection, - count_tokens, - mark, - also_delete_from, - ) - advisory_index.log_advisory_audit(record_id, mark, collection, result) - return result - - -@activity.defn -async def index_webhook_advisory(record_id: str) -> dict: - """Write one webhook advisory record's chunks to the dev collection (or delete if unpublished).""" - return await _run_advisory_activity(record_id, promote=False) - - -@activity.defn -async def promote_webhook_advisory(record_id: str) -> dict: - """Copy one webhook advisory record's chunks to the PROD collection (or delete if unpublished).""" - return await _run_advisory_activity(record_id, promote=True) diff --git a/pipeline/advisory_chunks.py b/pipeline/advisory_chunks.py deleted file mode 100644 index cf496e2..0000000 --- a/pipeline/advisory_chunks.py +++ /dev/null @@ -1,153 +0,0 @@ -"""Turn a published advisory ``webhook_records`` row into Qdrant chunk records. - -One advisory entry (``popList`` item) becomes one chunk when its whole text fits the -embedding model's token limit, otherwise up to three chunks that each carry the advice -(overview + advice, symptoms + advice, conditions + advice). The stored ``text`` is -always the complete chunk text; the embedding model reads only its first tokens. -""" - -from __future__ import annotations - -import hashlib -from typing import Callable - -MAX_EMBEDDING_TOKENS = 512 - -# (section, label) for the question-side fields, in display order. -_FIELD_SECTIONS = ( - ("overview", "Overview"), - ("symptoms", "Symptoms"), - ("conditions", "Conditions"), -) - - -def clean_value(value: object) -> str: - """Return a trimmed string with surrounding quote characters removed ('' for None).""" - if value is None: - return "" - return str(value).strip().strip('"').strip() - - -def _clean_list(value: object) -> str: - """Join the non-empty cleaned items of a list (or a single value) with spaces.""" - if value is None: - return "" - items = value if isinstance(value, (list, tuple)) else [value] - return " ".join(part for part in (clean_value(item) for item in items) if part) - - -def _is_published(row: dict) -> bool: - return row.get("is_published") in (1, True) and clean_value(row.get("status")) == "Published" - - -def _header(title: str, crop: str, problem_type: str, season: str) -> list[str]: - lines = [title] if title else [] - parts = [] - if crop: - parts.append(f"Crop: {crop}") - if problem_type: - parts.append(f"Problem type: {problem_type}") - if season: - parts.append(f"Season: {season}") - if parts: - lines.append(" | ".join(parts)) - return lines - - -def _chunk_id(doc_id: str, pop_id: object, section: str) -> str: - return hashlib.md5(f"{doc_id}:{pop_id}:{section}".encode("utf-8")).hexdigest() - - -def build_chunks( - row: dict, - count_tokens: Callable[[str], int], - max_tokens: int = MAX_EMBEDDING_TOKENS, -) -> list[dict]: - """Build Qdrant records for one ``webhook_records`` row (empty list if unpublished).""" - if not _is_published(row): - return [] - payload = row.get("payload") or {} - pop_list = payload.get("popList") - if not isinstance(pop_list, list): - return [] - - doc_id = str(row["id"]) - title = clean_value(row.get("title")) or clean_value(payload.get("title")) - crop = clean_value(payload.get("commodityName")) - base = { - "doc_id": doc_id, - "workflow_id": "", - "type": "advisory", - "source": "webhook", - "title_en": title, - "slug": clean_value(row.get("slug")), - "external_id": clean_value(row.get("external_id")), - "crop": crop, - "domain": clean_value(payload.get("domainName")), - "subdomain": clean_value(payload.get("subdomainName")), - "institute": clean_value(payload.get("instituteName")), - "creator": clean_value(payload.get("creatorName")), - "updated_at": clean_value(row.get("updated_at")), - } - - records: list[dict] = [] - - def add(pop: dict, index: int, section: str, text: str, meta: dict) -> None: - pop_id = pop.get("popID") - key = pop_id if pop_id is not None else f"idx{index}" - chunk_id = _chunk_id(doc_id, key, section) - records.append( - { - **base, - **meta, - "_id": chunk_id, - "chunk_id": chunk_id, - "pop_id": pop_id, - "section": section, - "chunk_num": len(records), - "text": text, - "text_for_embedding": text, - } - ) - - for index, pop in enumerate(pop_list): - if not isinstance(pop, dict) or clean_value(pop.get("status")) != "Published": - continue - problem_type = clean_value(pop.get("problemType")) - season = clean_value(pop.get("season")) - meta = { - "problem_type": problem_type, - "problem": clean_value(pop.get("problemDetails")), - "season": season, - "start_period": clean_value(pop.get("startPeriod")), - "end_period": clean_value(pop.get("endPeriod")), - } - header = _header(title, crop, problem_type, season) - fields = { - "overview": clean_value(pop.get("description")), - "symptoms": _clean_list(pop.get("applicableSymptoms")), - "conditions": _clean_list(pop.get("applicableCondition")), - } - advisory = clean_value(pop.get("advisory")) - labelled = [(label, fields[section]) for section, label in _FIELD_SECTIONS if fields[section]] - if advisory: - labelled.append(("Advisory", advisory)) - if not labelled: - continue - - full_text = "\n".join(header + [f"{label}: {body}" for label, body in labelled]) - if count_tokens(full_text) <= max_tokens: - add(pop, index, "full", full_text, meta) - continue - - others = [(section, label) for section, label in _FIELD_SECTIONS if fields[section]] - if not others: - add(pop, index, "advisory", full_text, meta) - continue - for section, label in others: - lines = header + [f"{label}: {fields[section]}"] - if advisory: - lines.append(f"Advisory: {advisory}") - add(pop, index, section, "\n".join(lines), meta) - - return records diff --git a/pipeline/advisory_index.py b/pipeline/advisory_index.py deleted file mode 100644 index c91f2f3..0000000 --- a/pipeline/advisory_index.py +++ /dev/null @@ -1,84 +0,0 @@ -"""Index (or remove) one advisory ``webhook_records`` row in a vector store collection.""" - -from __future__ import annotations - -import logging -import os -from typing import Callable, Literal, Optional - -from . import db -from .advisory_chunks import build_chunks - -logger = logging.getLogger(__name__) - - -def _delete_points(store, collection: str, doc_id: str) -> None: - """Delete a record's points; a missing collection is fine, any other failure is not.""" - result = store.delete_by_doc_id(collection, doc_id) or {} - error = str(result.get("error") or "") - if error and "doesn't exist" not in error and "not found" not in error.lower(): - raise RuntimeError(f"Could not delete points for {doc_id} from {collection}: {error}") - - -def index_advisory_record( - record_id: str, - store, - collection: str, - count_tokens: Callable[[str], int], - mark: Literal["indexed", "promoted"] = "indexed", - also_delete_from: Optional[tuple[object, str]] = None, -) -> dict: - """Replace a record's points in ``collection`` with its current chunks. - - Order: read row, delete the record's old points, write the new chunks, then store the - row's payload_hash (``indexed_hash`` or ``promoted_hash``). An unpublished record (no - chunks) only gets its points deleted; ``also_delete_from=(store, collection)`` removes - them from a second collection too (deletion only, never a content push). - """ - row = db.get_webhook_record(record_id) - if row is None: - return {"record_id": record_id, "status": "missing", "chunks": 0} - - doc_id = str(row["id"]) - chunks = build_chunks(row, count_tokens) - _delete_points(store, collection, doc_id) - if chunks: - store.upsert(collection, chunks, batch_size=8) - elif also_delete_from is not None: - other_store, other_collection = also_delete_from - _delete_points(other_store, other_collection, doc_id) - - marker = db.mark_webhook_record_promoted if mark == "promoted" else db.mark_webhook_record_indexed - marker(record_id, row["payload_hash"]) - logger.info("advisory %s record=%s collection=%s chunks=%s", mark, record_id, collection, len(chunks)) - return {"record_id": record_id, "status": "indexed" if chunks else "deleted", "chunks": len(chunks)} - - -def prod_target() -> Optional[tuple[object, str]]: - """The PROD (store, collection) pair from the PROD_VECTOR_DB_* env, or None when not configured.""" - url = (os.environ.get("PROD_VECTOR_DB_URL") or "").strip() - if not url: - return None - from .vector_store.qdrant_store import QdrantVectorStore, get_qdrant_client - - api_key = (os.environ.get("PROD_VECTOR_DB_API_KEY") or "").strip() or None - collection = ( - os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME") - or os.environ.get("VECTOR_DB_COLLECTION_NAME") - or "documents-index" - ).strip() - return QdrantVectorStore(client=get_qdrant_client(url=url, api_key=api_key)), collection - - -def log_advisory_audit(record_id: str, mark: str, collection: str, result: dict) -> None: - """Write one audit_logs row for an indexed / deleted / promoted advisory record.""" - action = "advisory_promoted" if mark == "promoted" else ( - "advisory_deleted" if result.get("status") == "deleted" else "advisory_indexed" - ) - db.log_audit( - workflow_id=f"advisory-{record_id}", - document_id=record_id, - action_type=action, - entity_type="webhook_record", - metadata={"collection": collection, "chunks": result.get("chunks", 0), "status": result.get("status")}, - ) diff --git a/pipeline/advisory_start.py b/pipeline/advisory_start.py deleted file mode 100644 index cc3ef07..0000000 --- a/pipeline/advisory_start.py +++ /dev/null @@ -1,58 +0,0 @@ -"""Start advisory index / promote workflows: one Temporal workflow per webhook record.""" - -from __future__ import annotations - -import logging -import os - -from temporalio.exceptions import WorkflowAlreadyStartedError - -from .workflows import AdvisoryIndexWorkflow, AdvisoryPromoteWorkflow - -logger = logging.getLogger(__name__) - -TASK_QUEUE = "ocr-pipeline" - - -async def _start_each(client, workflow_run, id_prefix: str, record_ids: list[str]) -> int: - started = 0 - for record_id in record_ids: - try: - await client.start_workflow( - workflow_run, - args=[record_id], - id=f"{id_prefix}{record_id}", - task_queue=TASK_QUEUE, - ) - started += 1 - except WorkflowAlreadyStartedError: - logger.info("advisory workflow %s%s already running; skipped", id_prefix, record_id) - except Exception: - logger.exception("could not start advisory workflow %s%s", id_prefix, record_id) - return started - - -async def start_advisory_index(client, record_ids: list[str]) -> int: - """Start ``AdvisoryIndexWorkflow`` (id ``advisory-``) for each id; returns how many started.""" - return await _start_each(client, AdvisoryIndexWorkflow.run, "advisory-", record_ids) - - -async def start_advisory_promote(client, record_ids: list[str]) -> int: - """Start ``AdvisoryPromoteWorkflow`` (id ``advisory-promote-``) for each id.""" - return await _start_each(client, AdvisoryPromoteWorkflow.run, "advisory-promote-", record_ids) - - -def _backfill_enabled() -> bool: - return os.environ.get("ADVISORY_BACKFILL_ON_START", "false").strip().lower() in {"1", "true", "yes", "on"} - - -async def backfill_on_start(client) -> int: - """Worker startup hook: when ADVISORY_BACKFILL_ON_START is on (default off), queue every unindexed row.""" - if not _backfill_enabled(): - return 0 - from . import db - - record_ids = db.get_unindexed_webhook_record_ids() - started = await start_advisory_index(client, record_ids) - logger.info("advisory startup backfill: %s unindexed rows, %s workflows started", len(record_ids), started) - return started diff --git a/pipeline/api/__init__.py b/pipeline/api/__init__.py index be3c5dc..c95bef7 100644 --- a/pipeline/api/__init__.py +++ b/pipeline/api/__init__.py @@ -1,4 +1,4 @@ -"""HTTP API package. The FastAPI app lives in ``api.py``; per-resource routers sit beside it.""" +"""HTTP application entry point; feature handlers live in ``routers/``.""" from .api import app diff --git a/pipeline/api/api.py b/pipeline/api/api.py index 4ec262e..86f38d2 100644 --- a/pipeline/api/api.py +++ b/pipeline/api/api.py @@ -1,20 +1,17 @@ """ FastAPI REST API for the Temporal-based OCR pipeline. -This API provides HTTP endpoints that interact with Temporal workflows. +Composition and dependency overrides live here; feature routes and shared helpers +are grouped in the adjacent modules. """ -import asyncio -import hashlib -import json + +# Imports are also the supported runtime dependency override boundary. +# ruff: noqa: F401 import logging import os -import re -import time -from contextlib import asynccontextmanager -from datetime import datetime from pathlib import Path -from typing import Any, Dict, List, Optional +from typing import Optional # Load repo-root .env so KEYCLOAK_ADMIN_* etc. work without manual `source .env` try: @@ -27,92 +24,34 @@ except Exception: pass -from urllib.parse import quote - -from fastapi import Depends, FastAPI, Header, HTTPException, Query, Request -from fastapi import Path as PathParam -from fastapi.middleware.cors import CORSMiddleware -from fastapi.responses import StreamingResponse -from minio import Minio -from pydantic import BaseModel, Field -from slowapi import Limiter, _rate_limit_exceeded_handler -from slowapi.errors import RateLimitExceeded -from slowapi.util import get_remote_address -from temporalio.client import Client, WorkflowFailureError +from fastapi import FastAPI, HTTPException, Request +from temporalio.client import Client -from .. import db, scheme_catalog -from ..documents import display_name as display_name_rules -from ..advisory_start import start_advisory_index, start_advisory_promote +from pipeline import db +from pipeline.network.catalog import scheme_catalog from ..auth.config import load_auth_config, validate_auth_config -from ..auth.deps import ( - CurrentUser, - RequireAdmin, - RequireApproveIngestion, - RequireManageUsers, - RequirePipeline, - RequireReview, - RequireSearch, -) -from ..auth.email_otp import ( - exchange_authorization_code, - normalize_email, - refresh_tokens, - send_otp, - verify_otp, -) -from ..auth.keycloak_admin import ( - list_access_options, - list_realm_users, - provision_user, - set_user_access, -) from ..auth.models import AuthUser from ..auth.permissions import Permission from ..auth.tenancy import ( - PORTAL_INSTANCE, allowed_instances, assert_document_instance_access, normalize_instance, resolve_create_instance, - unrestricted, - user_can_access_instance, -) -from ..instances import prod_stage_disabled -from ..models import ( - PIPELINE_STAGES, - PROD_ONLY_STAGES, - PUBLISH_SKIPPED_UNMAPPED_KIND, - AuditLogResponse, - BulkWorkflowActionRequest, - BulkWorkflowActionResponse, - BulkWorkflowActionResult, - ChunkUpdate, - DocumentDetail, - DocumentGraph, - DocumentStage, - DocumentSummary, - NetworkPublishOutcome, - OperationQueueEntry, - OperationQueueResponse, - PageUpdate, - ReindexStateRequest, - SchemeMetadataUpdate, - SearchSettings, - SearchSettingsUpdate, - SettingsAuditResponse, ) -from ..network_constants import RESULT_REJECTED, catalog_type_for_kind -from ..network_group import validate_network_provider_config -from ..telemetry.logs import configure_logs, reset_logs -from ..workflows import ( - ChunkingOnlyWorkflow, - DocumentPipelineWorkflow, - OcrOnlyWorkflow, - PromoteToProdWorkflow, - ReingestionWorkflow, - TranslationOnlyWorkflow, +from pipeline.documents.document_formats import allowed_file_extensions, installed_formats +from pipeline.documents.document_kinds import validate_document_kind +from pipeline.network.release_policy import prod_stage_disabled +from pipeline.documents.models import DocumentStage, DocumentSummary +from pipeline.network.network_publisher import validate_network_adapters +from ..object_storage import ( + ObjectStore, + is_object_uri, + load_object_store_config, + open_object_store, + parse_object_uri, ) -from . import request_logging +from pipeline.telemetry.logs import configure_logs, reset_logs +from .rate_limiting import limiter LOG_LEVEL = getattr(logging, os.environ.get("LOG_LEVEL", "INFO").strip().upper(), logging.INFO) # uvicorn only configures its own "uvicorn"/"uvicorn.access" loggers, so plain @@ -127,3654 +66,98 @@ logger = logging.getLogger(__name__) -TASK_QUEUE = "ocr-pipeline" +TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "ocr-pipeline") # Global clients temporal_client: Optional[Client] = None -minio_client: Optional[Minio] = None -MINIO_BUCKET = "documents" - - -@asynccontextmanager -async def lifespan(app: FastAPI): - """Initialize Temporal and MinIO clients on startup.""" - configure_logs() - try: - await _start_services() - except Exception: - logging.exception("api startup failed") - reset_logs() - raise - yield - logging.info("api stopping") - reset_logs() - - -async def _start_services(): - global temporal_client, minio_client, MINIO_BUCKET - - validate_network_provider_config() - - auth_cfg = load_auth_config() - validate_auth_config(auth_cfg) - if auth_cfg.disabled: - logging.warning( - "WARNING: AUTH_DISABLED=true — every caller is treated as synthetic " - "master_admin with unrestricted instance access. Do not expose this " - "API beyond the internal network or set AUTH_DISABLED=false until " - "the maintainer UI sends Bearer tokens." - ) - else: - logging.info( - "Auth enabled: issuer=%s audience=%s jwks=%s", - auth_cfg.keycloak_issuer, - auth_cfg.keycloak_audience or "(none)", - auth_cfg.keycloak_jwks_url, - ) - - logging.info("api startup: initializing PostgreSQL database") - db.init_db() - - # Temporal - temporal_host = os.environ.get("TEMPORAL_HOST", "localhost:7233") - logging.info("api startup: connecting to Temporal at %s", temporal_host) - temporal_client = await Client.connect(temporal_host) - - # MinIO - credentials required via environment variables - minio_endpoint = os.environ.get("MINIO_ENDPOINT", "localhost:9000") - minio_access_key = os.environ.get("MINIO_ACCESS_KEY") - minio_secret_key = os.environ.get("MINIO_SECRET_KEY") - MINIO_BUCKET = os.environ.get("MINIO_BUCKET", "documents") - - if not minio_access_key or not minio_secret_key: - raise RuntimeError("MINIO_ACCESS_KEY and MINIO_SECRET_KEY environment variables are required") +object_store: Optional[ObjectStore] = None +OBJECT_STORE_BUCKET = "documents" - logging.info("api startup: connecting to MinIO at %s", minio_endpoint) - minio_client = Minio( - minio_endpoint, - access_key=minio_access_key, - secret_key=minio_secret_key, - secure=False - ) - # Ensure bucket exists - if not minio_client.bucket_exists(MINIO_BUCKET): - minio_client.make_bucket(MINIO_BUCKET) - logging.info("api startup: created MinIO bucket %s", MINIO_BUCKET) - - logging.info( - "api started auth=%s temporal=%s minio=%s bucket=%s", - "disabled" if auth_cfg.disabled else "enabled", - temporal_host, - minio_endpoint, - MINIO_BUCKET, - ) - - -app = FastAPI( - title="Document Ingestion Pipeline API", - description=""" -REST API for the Temporal-based document OCR pipeline with translation support. - -## Workflow Stages - -1. `registered` - Document registered -2. `ocr_processing` - OCR in progress -3. `ocr_review` - **Waiting for OCR review/approval** -4. `translation_processing` - Translating non-English content -5. `translation_review` - **Waiting for translation review/approval** -6. `chunking` - Chunking in progress -7. `chunk_review` - **Waiting for chunk review/approval** -8. `ready_for_ingestion` - **Waiting for final approval before DEV ingest** -9. `ingesting` - Ingesting to DEV Qdrant -10. `approval_for_prod` - **Waiting for superadmin prod approval** -11. `ingesting_prod` - Promoting vectors to PROD Qdrant -12. `completed` - Done -13. `failed` - Error occurred - -## Review Flow - -1. Start workflow with `POST /upload` or `POST /documents` -2. Wait for `ocr_review` stage -3. Review/edit pages with `GET/PATCH /documents/{id}/pages/{num}` -4. Approve with `POST /documents/{id}/approve-ocr` -5. Wait for `translation_review` stage -6. Review/edit translations with `PATCH /documents/{id}/pages/{num}` -7. Approve with `POST /documents/{id}/approve-translation` -8. Wait for `chunk_review` stage -9. Review/edit chunks with `GET/PATCH /documents/{id}/chunks/{num}` -10. Approve with `POST /documents/{id}/approve-chunks` -11. Wait for `ready_for_ingestion` stage -12. Approve DEV ingest with `POST /documents/{id}/approve-ingestion` -13. Wait for `approval_for_prod` stage -14. Superadmin promote with `POST /documents/{id}/approve-prod` → `ingesting_prod` → `completed` - """, - version="2.0.0", - lifespan=lifespan +from .file_helpers import ( # noqa: E402 + validate_file_path, + get_filename_from_path, + get_workflow_id, + _rerun_workflow_id, ) - -# CORS configuration - explicit origins for security -ALLOWED_ORIGINS = os.environ.get("CORS_ORIGINS", "http://localhost:3000").split(",") - -app.add_middleware( - CORSMiddleware, - allow_origins=ALLOWED_ORIGINS, - allow_credentials=True, - allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"], - allow_headers=["*"], +from .document_helpers import ( # noqa: E402 + _instance_scope_for_user, + _document_cohorts_payload, + _resolve_create_instance, + _document_title, + _assert_can_replace, + _assert_upload_start_resolved, + _resolve_document_identity, + _require_document_for_user, + _document_for_user_or_none, + _document_summary_from_row, + _provenance_base_urls, + _build_provenance_links, + _list_available_actions, + _mark_reindex_required, ) -request_logging.install(app) - -# Rate limiting configuration -# Default: 100 requests/minute for general endpoints, 10/minute for uploads -RATE_LIMIT_DEFAULT = os.environ.get("RATE_LIMIT_DEFAULT", "100/minute") -RATE_LIMIT_UPLOAD = os.environ.get("RATE_LIMIT_UPLOAD", "10/minute") -# Login OTP endpoints are unauthenticated and mail real people, so they get their -# own much tighter budget than the general default. -RATE_LIMIT_OTP = os.environ.get("RATE_LIMIT_OTP", "10/minute") - -_DOCUMENT_PATH_RE = re.compile(r"^/documents/([^/]+)(/.*)?$") -# Only identifier shapes with an indexed lookup are resolved before auth runs: -# the document UUID, the 32-hex fingerprint, or a ``doc-`` workflow id. -_RESOLVABLE_IDENTIFIER_RE = re.compile( - r"^(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|[0-9a-f]{32}|doc-[0-9a-f]{32}.*)$" +from .audit_helpers import ( # noqa: E402 + _parse_roles_field, + _actor_from_user, + _log_audit, +) +from .index_helpers import ( # noqa: E402 + delete_single_chunk_from_index, + delete_chunks_from_index, +) +from .runtime import _start_services, lifespan # noqa: E402 +from .application import ( # noqa: E402 + _static_document_path_segments, canonicalize_document_identifier, create_app, ) -_static_document_segments: Optional[set[str]] = None - - -def _static_document_path_segments() -> set[str]: - """Literal second segments of ``/documents/...`` routes (``upload``, ``batch``...).""" - global _static_document_segments - if _static_document_segments is None: - segments = set() - for route in app.routes: - match = _DOCUMENT_PATH_RE.match(getattr(route, "path", "")) - if match and "{" not in match.group(1): - segments.add(match.group(1)) - _static_document_segments = segments - return _static_document_segments - - -@app.middleware("http") -async def canonicalize_document_identifier(request: Request, call_next): - """Let ``/documents/{id}/...`` take the document UUID or any legacy identifier. - - Handlers address a document by its Temporal workflow id (they signal and - query Temporal with it), so the path is rewritten to that id before routing. - An identifier of another shape, or one that matches no document, is left alone and - 404s in the handler; the lookup runs before auth, so it is limited to cheap shapes. - """ - match = _DOCUMENT_PATH_RE.match(request.scope["path"]) - if ( - match - and _RESOLVABLE_IDENTIFIER_RE.match(match.group(1)) - and match.group(1) not in _static_document_path_segments() - ): - document = await asyncio.to_thread(db.find_document_by_doc_identifier, match.group(1)) - if document and document["workflow_id"] and document["workflow_id"] != match.group(1): - new_path = f"/documents/{document['workflow_id']}{match.group(2) or ''}" - request.scope["path"] = new_path - request.scope["raw_path"] = quote(new_path, safe="/:").encode("ascii") - return await call_next(request) - - -limiter = Limiter(key_func=get_remote_address) -app.state.limiter = limiter -app.add_exception_handler(RateLimitExceeded, request_logging.recording(_rate_limit_exceeded_handler)) -# Routers split out of this module, one per resource. Register them here. -from .documents import router as documents_router # noqa: E402 -from .search import router as search_router # noqa: E402 -from .upload import router as upload_router # noqa: E402 +ALLOWED_ORIGINS = os.environ.get("CORS_ORIGINS", "http://localhost:3000").split(",") +RATE_LIMIT_UPLOAD = os.environ.get("RATE_LIMIT_UPLOAD", "10/minute") +app = create_app(lifespan) + +from .routers.health import health, router as health_router # noqa: E402 +# Register routes after establishing the runtime dependency boundary. +from .routers.audit import router as audit_router # noqa: E402 +from .routers.auth_sessions import router as auth_sessions_router # noqa: E402 +from .routers.catalog import router as catalog_router # noqa: E402 +from .routers.chunks import router as chunks_router # noqa: E402 +from .routers.crawl import router as crawl_router # noqa: E402 +from .routers.document_content import router as document_content_router # noqa: E402 +from .routers.document_detail import router as document_detail_router # noqa: E402 +from .routers.document_lifecycle import router as document_lifecycle_router # noqa: E402 +from .routers.document_review import router as document_review_router # noqa: E402 +from .routers.documents import router as documents_router # noqa: E402 +from .routers.indexes import router as indexes_router # noqa: E402 +from .routers.operations import router as operations_router # noqa: E402 +from .routers.pages import router as pages_router # noqa: E402 +from .routers.providers import router as providers_router # noqa: E402 +from .routers.registry_proposals import router as registry_proposals_router # noqa: E402 +from .routers.search import router as search_router # noqa: E402 +from .routers.search_settings import router as search_settings_router # noqa: E402 +from .routers.upload import router as upload_router # noqa: E402 +from .routers.user_admin import router as user_admin_router # noqa: E402 +from .routers.webhook_sources import knowledge_router # noqa: E402 +from .routers.webhook_sources import router as webhook_sources_router + +app.include_router(health_router) app.include_router(search_router) app.include_router(documents_router) +app.include_router(crawl_router) +app.include_router(providers_router) app.include_router(upload_router) -# TODO(api-split): extract the remaining resources into their own routers, -# one file each under pipeline/api/, following search.py: the per-document -# /documents/{workflow_id} routes, pages, chunks, artifacts, vector indexes, -# catalog, audit, settings, auth, admin. -# tests/test_api_routes.py pins the route table so a move cannot drop one. - - -# Allowed base directories for file access (configurable via env) -ALLOWED_FILE_PATHS = os.environ.get("ALLOWED_FILE_PATHS", "/app/books,/data/documents").split(",") -ALLOWED_EXTENSIONS = { - ".pdf", ".doc", ".docx", ".ppt", ".pptx", ".xls", ".xlsx", ".csv", ".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff" -} - - -def validate_file_path(filepath: str) -> str: - """ - Validate that a file path is within allowed directories. - Prevents path traversal attacks. - - Returns the validated filepath as a string. - Raises HTTPException if path is not allowed. - """ - # Handle minio:// URIs - these are validated by MinIO access - if filepath.startswith("minio://"): - suffix = Path(filepath).suffix.lower() - if suffix not in ALLOWED_EXTENSIONS: - raise HTTPException(400, f"Unsupported file type: {suffix}") - return filepath # Return string as-is, MinIO handles validation - - path = Path(filepath).resolve() # Resolve to absolute, canonical path - - # Check if path is within any allowed directory - for allowed_base in ALLOWED_FILE_PATHS: - allowed_path = Path(allowed_base.strip()).resolve() - try: - path.relative_to(allowed_path) - # Path is within allowed directory - if not path.exists(): - raise HTTPException(404, "File not found") - if not path.is_file(): - raise HTTPException(400, "Path is not a file") - if path.suffix.lower() not in ALLOWED_EXTENSIONS: - raise HTTPException(400, f"Unsupported file type: {path.suffix.lower()}") - return str(path) - except ValueError: - continue # Not within this allowed path, try next - - # Path not within any allowed directory - raise HTTPException(403, "Access to this file path is not allowed") - - -def get_filename_from_path(filepath: str) -> str: - """Extract filename from a filepath string (works for both local and minio:// paths).""" - if filepath.startswith("minio://"): - # minio://bucket/path/to/file.pdf -> file.pdf - return filepath.split("/")[-1] - return Path(filepath).name - - -def get_workflow_id(filepath: str) -> str: - """Generate consistent workflow ID from filepath.""" - return f"doc-{hashlib.md5(filepath.encode()).hexdigest()[:12]}" - - -def _rerun_workflow_id(base_workflow_id: str) -> str: - """Generate a fresh workflow ID for explicit reruns of the same source.""" - return f"{base_workflow_id}-rerun-{int(time.time())}" - - -def _compute_file_fingerprint(filepath: Path) -> str: - md5 = hashlib.md5() - with filepath.open("rb") as f: - for chunk in iter(lambda: f.read(1024 * 1024), b""): - md5.update(chunk) - return md5.hexdigest() - - -def _instance_scope_for_user(user: AuthUser) -> Optional[list[str]]: - """None = unrestricted; otherwise only these instance codes.""" - allowed = allowed_instances(user) - if allowed is None: - return None - return sorted(allowed) - - -def _document_cohorts_payload(summary: dict, by_instance: Optional[list[dict]] = None) -> dict: - """Shape Postgres aggregate counts for DocumentCohortsResponse (all ints).""" - def _i(key: str, default: int = 0) -> int: - val = summary.get(key, default) - return int(val or 0) - - return { - "total_documents": _i("total_documents"), - "authoritative_documents": _i("authoritative_documents"), - "legacy_documents": _i("legacy_documents"), - "completed_documents": _i("completed_documents"), - "review_queue": _i("review_queue"), - "failed_documents": _i("failed_documents"), - "needs_reindex": _i("needs_reindex"), - "running_jobs": _i("running_jobs"), - "by_stage": { - "ocr_review": _i("ocr_review_documents"), - "translation_review": _i("translation_review_documents"), - "chunk_review": _i("chunk_review_documents"), - "translation_processing": _i("translation_processing_documents"), - "chunking": _i("chunking_documents"), - "ready_for_ingestion": _i("ready_for_ingestion_documents"), - "failed": _i("failed_documents"), - }, - "by_instance": by_instance or [], - } - - -def _resolve_create_instance(user: AuthUser, requested: Optional[str] = None) -> str: - """Resolve the upload owner using the existing Keycloak tenancy rules.""" - return resolve_create_instance(user, requested) - - -def _document_title(filename: str) -> str: - """Human title for a new document: the file name without extension, separators as spaces.""" - stem = Path(filename or "").stem - return re.sub(r"[_\-\s]+", " ", stem).strip() or "Untitled document" - - -def _assert_can_replace(user: AuthUser, existing: dict) -> None: - """Replacing a document's file clears its pages and chunks, so guard it. - - Needs review rights in the document's instance; a disabled document or one with - a run in progress cannot be replaced; a document already promoted to PROD - can only be replaced by someone who may approve ingestion. - """ - doc_instance = existing["instance"] - if not user.has_permission_for_instance(Permission.REVIEW, doc_instance): - raise HTTPException(403, f"No permission to replace documents in instance: {doc_instance}") - if existing.get("is_disabled"): - raise HTTPException(409, "This document is disabled; restore it before replacing its file") - latest_job = db.get_latest_document_job(existing["workflow_id"]) - if latest_job and latest_job.get("status") == "running": - raise HTTPException(409, "A run is still in progress for this document") - if (existing.get("index_status") or {}).get("prod") and not user.has_permission_for_instance( - Permission.APPROVE_INGESTION, doc_instance - ): - raise HTTPException( - 403, "Replacing a document that is live in PROD needs approval rights in its instance" - ) - - -def _resolve_document_identity( - user: AuthUser, - instance: str, - kind: str, - code: str, - filename: str, -) -> tuple[str, str, Optional[dict]]: - """Kind, code and (when ``code`` already exists) the document an upload replaces. - - An explicit ``code`` that is already held in the tenant's catalog of that - kind means "update this document": the caller reuses it instead of inserting. - Without a code one is derived from the file name. - - Permissions are checked against the target instance, not just globally: a user - who may upload in one state must not be able to add to, or replace a - document of, a state where they only hold a view role. Replacing clears the - document's pages and chunks, so it also needs review rights in that instance. - """ - kind = (kind or "").strip().lower() - try: - catalog_type_for_kind(kind) - except ValueError as exc: - raise HTTPException(400, str(exc)) from exc - if not user.has_permission_for_instance(Permission.UPLOAD, instance): - raise HTTPException(403, f"No upload permission in instance: {instance}") - code = (code or "").strip() - if code: - try: - code = scheme_catalog.validate_scheme_code(code) - except ValueError as exc: - raise HTTPException(400, str(exc)) from exc - existing = db.find_document_by_code(instance, kind, code) - if existing: - existing = assert_document_instance_access(user, existing) - _assert_can_replace(user, existing) - return kind, code, existing - return kind, scheme_catalog.derive_scheme_code(_document_title(filename), kind=kind), None - - -def _require_document_for_user(workflow_id: str, user: AuthUser) -> dict: - """Load a document or 404 if missing / outside the caller's instance scope.""" - return assert_document_instance_access(user, db.get_document(workflow_id)) - - -def _document_for_user_or_none(workflow_id: str, user: AuthUser) -> Optional[dict]: - """Like _require_document_for_user but returns None instead of raising (bulk paths).""" - doc = db.get_document(workflow_id) - if not doc or not user_can_access_instance(user, doc.get("instance")): - return None - return doc - - -def _parse_roles_field(value) -> list[str]: - if value is None: - return [] - if isinstance(value, list): - return [str(v).strip() for v in value if str(v).strip()] - text = str(value).strip() - if not text: - return [] - if text.startswith("["): - try: - parsed = json.loads(text) - if isinstance(parsed, list): - return [str(v).strip() for v in parsed if str(v).strip()] - except Exception: - pass - return [part.strip() for part in text.split(",") if part.strip()] - - -def _actor_from_user(user) -> dict: - """Normalize AuthUser into serializable actor identity fields.""" - if user is None: - return { - "user_id": "", - "username": "", - "email": "", - "roles": [], - "roles_csv": "", - } - roles = [str(r).strip() for r in (getattr(user, "roles", None) or []) if str(r).strip()] - return { - "user_id": (getattr(user, "user_id", None) or "").strip(), - "username": (getattr(user, "username", None) or "").strip(), - "email": (getattr(user, "email", None) or "").strip(), - "roles": roles, - "roles_csv": ",".join(roles), - } - - -def _document_summary_from_row(doc: dict, current_job: Optional[dict] = None) -> DocumentSummary: - return DocumentSummary( - id=doc.get("id"), - document_id=doc["document_id"], - canonical_document_id=doc.get("canonical_document_id"), - workflow_id=doc["workflow_id"], - filename=doc["filename"], - name=doc.get("name"), - source_filename=doc.get("source_filename"), - source_manifest_name=doc.get("source_manifest_name"), - source_file_fingerprint=doc.get("source_file_fingerprint"), - authoritative=bool(doc.get("source_manifest_name")), - instance=normalize_instance(doc.get("instance")), - stage=DocumentStage(doc["stage"]), - page_count=doc.get("page_count") or 0, - chunk_count=doc.get("chunk_count") or 0, - error_message=doc.get("error_message"), - created_at=doc.get("created_at"), - updated_at=doc.get("updated_at"), - reindex_required=bool(doc.get("reindex_required")), - reindex_reason=doc.get("reindex_reason"), - available_actions=_list_available_actions( - doc, - current_job if current_job is not None else db.get_latest_document_job(doc["workflow_id"]), - ), - uploaded_by_user_id=doc.get("uploaded_by_user_id"), - uploaded_by_username=doc.get("uploaded_by_username"), - uploaded_by_email=doc.get("uploaded_by_email"), - uploaded_by_roles=_parse_roles_field(doc.get("uploaded_by_roles")), - kind=doc.get("kind"), - code=doc.get("code"), - aliases=list(doc.get("aliases") or []), - tool_routing=doc.get("tool_routing"), - catalog_visible=bool(int(doc["catalog_visible"])) if doc.get("catalog_visible") is not None else True, - network_visible=bool(int(doc["network_visible"])) if doc.get("network_visible") is not None else True, - prod_ready_requested_at=doc.get("prod_ready_requested_at"), - prod_ready_requested_by_username=doc.get("prod_ready_requested_by_username"), - valid_from=doc.get("valid_from"), - valid_to=doc.get("valid_to"), - ) - - -def _provenance_base_urls(request: Request) -> tuple[str, str]: - api_base = (os.environ.get("DOCS_PIPELINE_API_URL") or str(request.base_url)).rstrip("/") - ui_base = (os.environ.get("DOCS_PIPELINE_UI_URL") or "http://localhost:3000").rstrip("/") - return api_base, ui_base - - -def _build_provenance_links(workflow_id: str, chunk_num: int, request: Request) -> dict[str, str]: - api_base, ui_base = _provenance_base_urls(request) - return { - "pdf_url": f"{api_base}/documents/{workflow_id}/pdf", - "document_url": f"{ui_base}/documents/{workflow_id}", - "chunk_url": f"{ui_base}/documents/{workflow_id}?tab=chunks&chunk={chunk_num}", - } - - -def _list_available_actions(doc: dict, current_job: Optional[dict] = None) -> list[str]: - if not doc: - return [] - if doc.get("is_disabled"): - return ["restore_document"] - - stage = doc.get("stage") - actions = ["disable_document", "reconcile_document"] - if stage == "ocr_review": - actions.append("approve_ocr") - elif stage == "translation_review": - actions.append("approve_translation") - elif stage == "chunk_review": - actions.append("approve_chunks") - elif stage == "ready_for_ingestion": - actions.append("approve_ingestion") - elif stage == "approval_for_prod": - # A document already parked at this gate when DISABLE_PROD_SETTING was - # turned on keeps its approve action — otherwise it would be stranded - # with no way forward. - actions.append("approve_prod") - if not doc.get("prod_ready_requested_at"): - actions.append("request_prod_ready") - elif stage == "completed": - actions.append("reingest_document") - if not prod_stage_disabled(): - actions.append("approve_prod") - elif stage == "failed": - if not doc.get("ocr_completed_at"): - actions.append("retry_ocr") - if doc.get("ocr_completed_at") and not doc.get("translation_completed_at"): - actions.append("retry_translation") - if doc.get("translation_completed_at"): - actions.append("retry_chunking") - - if doc.get("reindex_required"): - actions.extend(["reingest_document", "clear_reindex_required"]) - else: - actions.append("mark_reindex_required") - - if current_job and current_job.get("status") == "running": - actions.append("inspect_runtime") - - return sorted(set(actions)) - - -def _mark_reindex_required(workflow_id: str, reason: str, metadata: Optional[dict] = None) -> Optional[dict]: - doc = db.mark_document_reindex_required(workflow_id, True, reason) - if doc: - db.log_audit( - workflow_id=workflow_id, - document_id=doc.get("document_id", workflow_id), - action_type="mark_reindex_required", - field_name="reindex_required", - old_value="false", - new_value="true", - metadata={"reason": reason, **(metadata or {})}, - ) - return doc - - -def delete_single_chunk_from_index(document_id: str, chunk_num: int, index_name: str = "documents-index") -> dict: - """Delete a single chunk from the vector index by doc_id + chunk_num.""" - from ..vector_store import get_default_index_name, get_vector_store - - try: - store = get_vector_store() - resolved_index = index_name or get_default_index_name() - return store.delete_chunk(resolved_index, document_id, chunk_num) - except Exception as e: - logging.exception("vector chunk delete failed doc_id=%s chunk=%s", document_id, chunk_num) - return {"deleted": False, "error": str(e)} - - -def delete_chunks_from_index(document_id: str, index_name: str = "documents-index") -> dict: - """Delete all chunks for a document from the vector index.""" - from ..vector_store import get_default_index_name, get_vector_store - - try: - store = get_vector_store() - resolved_index = index_name or get_default_index_name() - result = store.delete_by_doc_id(resolved_index, document_id) - if "doc_id" not in result: - result["doc_id"] = document_id - return result - except Exception as e: - logging.exception("vector document delete failed doc_id=%s", document_id) - return {"deleted": 0, "doc_id": document_id, "error": str(e)} - - -# ============================================================================= -# Login (unauthenticated — the session does not exist yet) -# -# Both SSO and email-OTP run their Keycloak token calls here rather than in the -# browser. The client is confidential, so only the server can present the secret -# the token endpoint requires — and that is precisely what lets both paths share -# a single client id. See pipeline/auth/email_otp.py. -# ============================================================================= - - -class OtpRequestBody(BaseModel): - email: str = Field(..., description="Email address to mail the login code to") - - -class OtpVerifyBody(BaseModel): - email: str = Field(..., description="Same address the code was sent to") - code: str = Field(..., description="The 6-digit code from the email") - - -class OtpRefreshBody(BaseModel): - refresh_token: str = Field(..., description="Refresh token from a prior verify") - - -@app.post("/auth/otp/request") -@limiter.limit(RATE_LIMIT_OTP) -async def auth_otp_request(request: Request, body: OtpRequestBody): - """Mail a one-time login code. - - Answers the same way for unknown addresses so this cannot be used to test - which emails have accounts. The client secret stays server-side, which is - the whole reason this is proxied rather than called from the browser. - """ - email = normalize_email(body.email) - result = await asyncio.to_thread(send_otp, email) - logging.info("email-otp: login code requested for %s", email) - return result - - -@app.post("/auth/otp/verify") -@limiter.limit(RATE_LIMIT_OTP) -async def auth_otp_verify(request: Request, body: OtpVerifyBody): - """Exchange a code for real Keycloak tokens (access / refresh / id). - - Returns the same token shape as the Google SSO flow so the UI stores and - refreshes an OTP session exactly like an SSO one. - """ - email = normalize_email(body.email) - tokens = await asyncio.to_thread(verify_otp, email, body.code) - logging.info("email-otp: login succeeded for %s", email) - return tokens - - -class SsoExchangeBody(BaseModel): - code: str = Field(..., description="Authorization code from the Keycloak redirect") - redirect_uri: str = Field(..., description="Must match the redirect_uri used to start login") - code_verifier: str | None = Field(None, description="PKCE verifier the UI generated") - - -@app.post("/auth/sso/exchange") -@limiter.limit(RATE_LIMIT_OTP) -async def auth_sso_exchange(request: Request, body: SsoExchangeBody): - """Complete a Google/Keycloak SSO login by exchanging the code for tokens. - - The browser hands us the code it received on /auth/sso-callback; we add the - client secret and do the exchange. Returns the same token shape as - /auth/otp/verify, so the UI stores and refreshes both session kinds alike. - """ - tokens = await asyncio.to_thread( - exchange_authorization_code, body.code, body.redirect_uri, body.code_verifier - ) - logging.info("sso: code exchange succeeded") - return tokens - - -@app.post("/auth/session/refresh") -@app.post("/auth/otp/refresh") # pre-SSO-exchange name; kept so older UI builds keep working -@limiter.limit(RATE_LIMIT_DEFAULT) -async def auth_session_refresh(request: Request, body: OtpRefreshBody): - """Renew a session — SSO or OTP, they are the same confidential client. - - Unauthenticated by design — the refresh token *is* the credential, and the - caller's access token has usually expired by the time this runs. - """ - return await asyncio.to_thread(refresh_tokens, body.refresh_token) - - -# ============================================================================= -# Document Routes -# ============================================================================= - -@app.get("/auth/me") -async def auth_me(user: CurrentUser): - """Return the authenticated caller (local bypass user when AUTH_DISABLED=true). - - Identity fields (username/email/roles) come from the JWT when a Bearer token - is present, including in AUTH_DISABLED mode for UI display. - """ - display_name = (user.username or "").strip() - return { - "user_id": user.user_id, - "username": user.username, - "name": display_name, - "email": user.email, - "roles": user.roles, - "permissions": sorted(p.value for p in user.permissions), - "instances": user.instances, - "envs": user.envs, - # Keycloak group paths (/states/MH/contributor, /global/super-admin) - # and derived per-state role map for tenant-aware UI. - "groups": list(user.groups or []), - "state_roles": dict(user.state_roles or {}), - "is_super_admin": bool(user.is_superadmin), - # Portal instance code for BV / platform docs (not a state). - "portal_instance": PORTAL_INSTANCE, - "auth_disabled": user.token_disabled_mode, - } - - -# ============================================================================= -# Super-admin user provisioning (Keycloak — no app DB access tables) -# ============================================================================= - - -class ProvisionUserRequest(BaseModel): - """Fields a super admin must fill to create / update access.""" - - email: str = Field(..., description="Google SSO email (required)") - first_name: str = "" - last_name: str = "" - username: str = "" - access_type: str = Field( - ..., - description="super_admin | state", - ) - state: str = Field("", description="State code e.g. MH (required for state access)") - role: str = Field( - "", - description="state_admin | state_view (required for state access; aliases: admin, view, contributor, reviewer)", - ) - enabled: bool = True - - -class SetUserAccessRequest(BaseModel): - """Change an existing user's single product role.""" - - access_type: str = Field(..., description="super_admin | bh_viewer | state") - state: str = Field("", description="State/centre code e.g. MH (required when access_type=state)") - role: str = Field( - "", - description=( - "state_admin | state_approver | state_contributor | state_view " - "(required when access_type=state)" - ), - ) - - -@app.get("/admin/access-options") -async def admin_access_options(user: RequireManageUsers): - """Form options + required fields for the Users admin UI.""" - return list_access_options() - - -@app.get("/admin/users") -async def admin_list_users( - user: RequireManageUsers, - search: str = "", - max_results: int = Query(100, ge=1, le=200), -): - """List Keycloak users with groups/access labels (no delete).""" - return await asyncio.to_thread(list_realm_users, search=search, max_results=max_results) - - -@app.post("/admin/users") -async def admin_provision_user(data: ProvisionUserRequest, user: RequireManageUsers): - """Create or update a Keycloak user and assign group/role. Returns share text.""" - return await asyncio.to_thread( - provision_user, - email=data.email, - first_name=data.first_name, - last_name=data.last_name, - username=data.username or None, - access_type=data.access_type, - state=data.state or None, - role=data.role or None, - enabled=data.enabled, - ) - - -@app.put("/admin/users/{user_id}/access") -async def admin_set_user_access( - user_id: str, data: SetUserAccessRequest, user: RequireManageUsers -): - """Change an existing user's role (super admin only). - - A user holds exactly one product role: the new group replaces whatever - product groups they had. They must re-login for the new role to apply. - """ - return await asyncio.to_thread( - set_user_access, - user_id=user_id, - access_type=data.access_type, - state=data.state or None, - role=data.role or None, - ) - - -@app.get("/operations/queue", response_model=OperationQueueResponse) -async def get_operations_queue( - user: RequireSearch, - limit: int = Query(100, le=500), - offset: int = Query(0, ge=0), - x_include_disabled: Optional[str] = Header(None, alias="X-Include-Disabled"), -): - """Return documents that currently need operator or agent action. - - Results are limited to instances the caller can access (state users never - see portal ``bv`` or other states' documents). - """ - include_disabled = x_include_disabled and x_include_disabled.lower() == "true" - rows, total = db.list_operations_queue( - limit=limit, - offset=offset, - include_disabled=include_disabled, - instances=_instance_scope_for_user(user), - ) - items = [ - OperationQueueEntry( - workflow_id=row["workflow_id"], - filename=row["filename"], - stage=row["stage"], - job_id=row.get("job_id"), - job_type=row.get("job_type"), - job_status=row.get("job_status"), - started_at=row.get("started_at"), - error_message=row.get("error_message") or row.get("reindex_reason"), - available_actions=_list_available_actions(row, row), - ) - for row in rows - ] - return OperationQueueResponse(items=items, total=total) - - -@app.get("/runs") -async def list_runs( - user: RequireSearch, - limit: int = Query(100, le=500), - offset: int = Query(0, ge=0), - status: Optional[str] = None, -): - """List recent document jobs visible to the caller (tenant/role scoped). - - Restricted users only see jobs whose document ``instance`` is in their - allowed states. Super-admin / bypass users see all tenants. - """ - return db.list_runs( - limit=limit, - offset=offset, - status=status, - instances=_instance_scope_for_user(user), - ) - - - -@app.get("/runs/{job_id}") -async def get_run(job_id: str, user: RequireSearch): - """Get a specific document job/run (404 if missing or outside tenant scope).""" - run = db.get_document_job(job_id) - if not run: - raise HTTPException(404, f"Run not found: {job_id}") - # Enforce same instance scope as list: hide other tenants' runs as 404. - workflow_id = run.get("workflow_id") - if workflow_id: - _require_document_for_user(str(workflow_id), user) - elif not unrestricted(user): - raise HTTPException(404, f"Run not found: {job_id}") - return run - - -NETWORK_PUBLISH_ARTIFACT_TYPES = {"network_publish_payload", "network_publish_skipped"} - - -def _network_publish_outcome(artifacts: list[dict]) -> NetworkPublishOutcome | None: - latest = next((a for a in artifacts if a.get("artifact_type") in NETWORK_PUBLISH_ARTIFACT_TYPES), None) - if latest is None: - return None - try: - exchange = json.loads(latest.get("metadata_json") or "{}") - except ValueError: - exchange = {} - if latest["artifact_type"] == "network_publish_skipped": - return NetworkPublishOutcome( - status="skipped", - reason_code=exchange.get("reason_code") or PUBLISH_SKIPPED_UNMAPPED_KIND, - reason=exchange.get("reason") or "No network catalog is mapped to this document kind.", - created_at=latest.get("created_at"), - ) - if exchange.get("result_status") == RESULT_REJECTED: - return NetworkPublishOutcome( - status="rejected", - reason=f"The network rejected the catalog: {exchange.get('errors') or []}", - created_at=latest.get("created_at"), - ) - return NetworkPublishOutcome(status="published", created_at=latest.get("created_at")) - - -def _build_document_detail(doc: dict) -> DocumentDetail: - workflow_id = doc["workflow_id"] - current_job = db.get_latest_document_job(workflow_id) - artifacts = db.list_document_artifacts(workflow_id) - return DocumentDetail( - id=doc.get("id"), - document_id=doc["document_id"], - canonical_document_id=doc.get("canonical_document_id"), - workflow_id=workflow_id, - filename=doc["filename"], - name=doc.get("name"), - source_filename=doc.get("source_filename"), - source_manifest_name=doc.get("source_manifest_name"), - source_file_fingerprint=doc.get("source_file_fingerprint"), - authoritative=bool(doc.get("source_manifest_name")), - instance=normalize_instance(doc.get("instance")), - filepath=doc["filepath"], - stage=DocumentStage(doc["stage"]), - page_count=doc.get("page_count", 0), - chunk_count=doc.get("chunk_count", 0), - error_message=doc.get("error_message"), - reindex_required=bool(doc.get("reindex_required")), - reindex_reason=doc.get("reindex_reason"), - available_actions=_list_available_actions(doc, current_job), - translated_count=db.count_translated_pages(workflow_id), - created_at=doc.get("created_at"), - updated_at=doc.get("updated_at"), - ocr_completed_at=doc.get("ocr_completed_at"), - translation_completed_at=doc.get("translation_completed_at"), - chunks_completed_at=doc.get("chunks_completed_at"), - ingested_at=doc.get("ingested_at"), - source_type=doc.get("source_type"), - canonical_input_type=doc.get("canonical_input_type"), - stop_after_ocr=bool(doc.get("stop_after_ocr")), - original_artifact_id=doc.get("original_artifact_id"), - normalized_artifact_id=doc.get("normalized_artifact_id"), - latest_job_id=doc.get("latest_job_id"), - current_job=current_job, - artifacts=artifacts, - index_status=db.list_document_index_status(workflow_id), - network_publish=_network_publish_outcome(artifacts), - uploaded_by_user_id=doc.get("uploaded_by_user_id"), - uploaded_by_username=doc.get("uploaded_by_username"), - uploaded_by_email=doc.get("uploaded_by_email"), - uploaded_by_roles=_parse_roles_field(doc.get("uploaded_by_roles")), - kind=doc.get("kind"), - code=doc.get("code"), - aliases=list(doc.get("aliases") or []), - tool_routing=doc.get("tool_routing"), - catalog_visible=bool(int(doc["catalog_visible"])) if doc.get("catalog_visible") is not None else True, - network_visible=bool(int(doc["network_visible"])) if doc.get("network_visible") is not None else True, - prod_ready_requested_at=doc.get("prod_ready_requested_at"), - prod_ready_requested_by_username=doc.get("prod_ready_requested_by_username"), - valid_from=doc.get("valid_from"), - valid_to=doc.get("valid_to"), - ) - - -def _catalog_service_keys() -> list[str]: - raw = (os.environ.get("CATALOG_SERVICE_API_KEYS") or "").strip() - if not raw: - return [] - return [k.strip() for k in raw.split(",") if k.strip()] - - -def _auth_disabled() -> bool: - return (os.environ.get("AUTH_DISABLED") or "").strip().lower() in ("1", "true", "yes") - - -def _service_key_ok(request: Request) -> bool: - import hmac - - keys = _catalog_service_keys() - if not keys: - return False - header = ( - request.headers.get("X-Catalog-Service-Key") - or request.headers.get("x-catalog-service-key") - or "" - ).strip() - if not header: - return False - return any(hmac.compare_digest(header, k) for k in keys) - - -def _require_catalog_read(request: Request, user: AuthUser) -> None: - """Service key, search/admin JWT, or AUTH_DISABLED without keys.""" - from ..auth.permissions import Permission - - if _service_key_ok(request): - return - if user.has_permission(Permission.ADMIN) or user.has_permission(Permission.SEARCH): - return - if _auth_disabled() and not _catalog_service_keys(): - return - if _catalog_service_keys(): - raise HTTPException(401, "Missing or invalid X-Catalog-Service-Key") - raise HTTPException(403, "Catalog access requires search or admin permission") - - -def _vector_doc_ids(doc: dict, env: str) -> list[str]: - """Index keys to delete for a document: its current key plus keys of earlier file versions in ``env``. - - A key another live document is indexed under in ``env`` (the same file - uploaded twice) is left out, since deleting it would remove that document's - vectors too. - """ - ids = [doc.get("document_id")] - ids.extend((doc.get("superseded_doc_ids") or {}).get(env, [])) - return [ - i - for i in dict.fromkeys(i for i in ids if i) - if not db.doc_id_used_by_other_document(i, doc.get("id"), env) - ] - - -def _delete_all_doc_ids(store, collection: str, doc_ids: list[str]) -> dict: - results = [store.delete_by_doc_id(collection, doc_id) for doc_id in doc_ids] - if len(results) == 1: - return results[0] - errors = [r["error"] for r in results if r.get("error")] - summary = {"deleted": 0 if errors else -1, "doc_ids": doc_ids, "results": results} - if errors: - summary["error"] = "; ".join(errors) - return summary - - -def _delete_document_points_from_prod(doc: dict) -> dict: - """Best-effort delete of a promoted (non-scheme) document's vectors from PROD documents-index.""" - doc_ids = _vector_doc_ids(doc, "prod") - if not doc_ids: - return {"deleted": False, "reason": "no_doc_id"} - prod_url = (os.environ.get("PROD_VECTOR_DB_URL") or "").strip() - if not prod_url: - return {"deleted": False, "reason": "no_prod_qdrant_url"} - prod_key = (os.environ.get("PROD_VECTOR_DB_API_KEY") or "").strip() or None - collection = ( - os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME") - or os.environ.get("VECTOR_DB_COLLECTION_NAME") - or "documents-index" - ).strip() - try: - from ..vector_store.qdrant_store import QdrantVectorStore, get_qdrant_client - - client = get_qdrant_client(url=prod_url, api_key=prod_key) - store = QdrantVectorStore(client=client) - return _delete_all_doc_ids(store, collection, doc_ids) - except Exception as exc: - logging.exception("prod vector delete failed doc_ids=%s collection=%s", doc_ids, collection) - return {"deleted": False, "error": str(exc), "collection": collection} - - -def _delete_scheme_points_from_prod(doc: dict) -> dict: - """Best-effort delete of scheme vectors from PROD schemes-index.""" - doc_ids = _vector_doc_ids(doc, "prod") - if not doc_ids: - return {"deleted": False, "reason": "no_doc_id"} - prod_url = ( - (os.environ.get("PROD_SCHEME_VECTOR_DB_URL") or "").strip() - or (os.environ.get("PROD_VECTOR_DB_URL") or "").strip() - ) - if not prod_url: - return {"deleted": False, "reason": "no_prod_qdrant_url"} - prod_key = ( - (os.environ.get("PROD_SCHEME_VECTOR_DB_API_KEY") or "").strip() - or (os.environ.get("PROD_VECTOR_DB_API_KEY") or "").strip() - or None - ) - collection = ( - os.environ.get("PROD_SCHEME_VECTOR_DB_COLLECTION_NAME") or "schemes-index" - ).strip() - try: - from ..vector_store.qdrant_store import QdrantVectorStore, get_qdrant_client - - client = get_qdrant_client(url=prod_url, api_key=prod_key) - store = QdrantVectorStore(client=client) - return _delete_all_doc_ids(store, collection, doc_ids) - except Exception as exc: - logging.exception("prod vector delete failed doc_ids=%s collection=%s", doc_ids, collection) - return {"deleted": False, "error": str(exc), "collection": collection} - - -def _build_stage_io_payload(workflow_id: str, current_stage: Optional[str] = None) -> dict: - artifacts = db.list_document_artifacts(workflow_id) - grouped: dict[str, dict] = {} - for stage_id, label, description in PIPELINE_STAGES: - grouped[stage_id] = { - "stage": stage_id, - "label": label, - "description": description, - "input_artifacts": [], - "output_artifacts": [], - } - - input_types = {"original_upload", "normalized_pdf", "normalized_spreadsheet"} - for artifact in artifacts: - stage = artifact.get("stage") or "registered" - if stage not in grouped: - grouped[stage] = { - "stage": stage, - "label": stage.replace("_", " ").title(), - "description": "", - "input_artifacts": [], - "output_artifacts": [], - } - bucket = "input_artifacts" if artifact["artifact_type"] in input_types else "output_artifacts" - grouped[stage][bucket].append(artifact) - - return { - "workflow_id": workflow_id, - "current_stage": current_stage, - "stages": list(grouped.values()), - } - - -async def _get_runtime_payload(workflow_id: str, doc: Optional[dict] = None) -> dict: - doc = doc or db.get_document(workflow_id) - if not doc: - raise HTTPException(404, f"Document not found: {workflow_id}") - current_job = db.get_latest_document_job(workflow_id) - runtime_workflow_id = ( - current_job.get("temporal_workflow_id") - if current_job and current_job.get("status") == "running" and current_job.get("temporal_workflow_id") - else workflow_id - ) - - chunking_progress = None - if current_job and current_job.get("config_json"): - try: - parsed_config = json.loads(current_job["config_json"]) if isinstance(current_job["config_json"], str) else current_job["config_json"] - if isinstance(parsed_config, dict): - chunking_progress = parsed_config.get("chunking_progress") - except Exception: - chunking_progress = None - - runtime = { - "workflow_id": workflow_id, - "sqlite_stage": doc.get("stage"), - "sqlite_error_message": doc.get("error_message"), - "temporal_connected": temporal_client is not None, - "job": current_job, - "chunking_progress": chunking_progress, - "temporal": None, - } - - if temporal_client is None: - return runtime - - try: - handle = temporal_client.get_workflow_handle(runtime_workflow_id) - description = await handle.describe() - temporal_state = None - query_error = None - try: - temporal_state = await handle.query("get_state") - except Exception as exc: - query_error = str(exc) - - runtime["temporal"] = { - "workflow_id": runtime_workflow_id, - "run_id": description.run_id, - "status": description.status.name, - "close_time": description.close_time.isoformat() if description.close_time else None, - "execution_time": description.execution_time.isoformat() if description.execution_time else None, - "state": temporal_state, - "query_error": query_error, - } - except Exception as exc: - runtime["temporal"] = { - "workflow_id": workflow_id, - "status": "UNAVAILABLE", - "error": str(exc), - } - - return runtime - - -@app.get("/documents/{workflow_id}", response_model=DocumentDetail) -async def get_document(workflow_id: str, user: CurrentUser): - """Get document workflow state with artifacts and indexing metadata.""" - doc = _require_document_for_user(workflow_id, user) - return _build_document_detail(doc) - - -@app.get("/documents/{workflow_id}/error-details") -async def get_workflow_error_details(workflow_id: str, user: RequireSearch): - """ - Get detailed error information from Temporal for a failed workflow. - - Returns comprehensive error details including: - - Error message - - Stack trace (if available) - - Failure type - - Workflow execution status - - This endpoint queries Temporal directly to get the most detailed - error information available, which may be more complete than what's - stored in Postgres. - """ - import traceback - - # Enforce tenant scope before touching Temporal (404 hides other tenants). - _require_document_for_user(workflow_id, user) - - try: - handle = temporal_client.get_workflow_handle(workflow_id) - description = await handle.describe() - - result = { - "workflow_id": workflow_id, - "run_id": description.run_id, - "status": description.status.name, - "error_message": None, - "error_type": None, - "stack_trace": None, - "has_error": False - } - - # If workflow is failed, try to get detailed error information - if description.status.name == "FAILED": - result["has_error"] = True - - # Try to get error from workflow result (this raises WorkflowFailureError for failed workflows) - try: - await handle.result() - except WorkflowFailureError as wf_err: - # Extract error details from the failure - result["error_message"] = str(wf_err) - result["error_type"] = type(wf_err).__name__ - - # Try to get the underlying cause - if hasattr(wf_err, 'cause') and wf_err.cause: - cause = wf_err.cause - result["error_message"] = str(cause) - result["error_type"] = type(cause).__name__ - - # Get stack trace if available - if hasattr(cause, '__traceback__') and cause.__traceback__: - result["stack_trace"] = ''.join(traceback.format_tb(cause.__traceback__)) - elif hasattr(wf_err, '__traceback__') and wf_err.__traceback__: - result["stack_trace"] = ''.join(traceback.format_tb(wf_err.__traceback__)) - - # Also try to get failure details from the exception itself - if hasattr(wf_err, 'failure') and wf_err.failure: - failure = wf_err.failure - if hasattr(failure, 'message') and failure.message: - result["error_message"] = failure.message - if hasattr(failure, 'stack_trace') and failure.stack_trace: - result["stack_trace"] = failure.stack_trace - except Exception as e: - # If result() doesn't work, try other methods - result["error_message"] = f"Could not retrieve error details: {str(e)}" - - # Also try to get error from workflow state query (fallback) - if not result["error_message"]: - try: - state = await handle.query("get_state") - if state and state.get("error_message"): - result["error_message"] = state.get("error_message") - result["has_error"] = True - except Exception: - pass # Workflow might not support queries or be in wrong state - - return result - - except Exception as e: - # If workflow doesn't exist or can't be accessed - error_msg = str(e) - if "not found" in error_msg.lower() or "workflow" in error_msg.lower(): - raise HTTPException(404, f"Workflow not found: {workflow_id}") - raise HTTPException(500, f"Error fetching workflow details: {error_msg}") - - -@app.get("/documents/{workflow_id}/runtime") -async def get_document_runtime(workflow_id: str, user: RequireSearch): - """Return live runtime status by combining Postgres state and Temporal workflow state.""" - doc = _require_document_for_user(workflow_id, user) - return await _get_runtime_payload(workflow_id, doc=doc) - - -@app.get("/documents/{workflow_id}/artifacts") -async def list_document_artifacts(workflow_id: str, user: RequireSearch): - _require_document_for_user(workflow_id, user) - return db.list_document_artifacts(workflow_id) - - -@app.get("/documents/{workflow_id}/artifacts/{artifact_id}") -async def get_document_artifact(workflow_id: str, user: RequireSearch, artifact_id: str): - _require_document_for_user(workflow_id, user) - artifact = db.get_document_artifact(workflow_id, artifact_id) - if not artifact: - raise HTTPException(404, f"Artifact not found: {artifact_id}") - return artifact - - -@app.get("/documents/{workflow_id}/artifacts/{artifact_id}/content") -async def get_document_artifact_content(workflow_id: str, user: RequireSearch, artifact_id: str): - _require_document_for_user(workflow_id, user) - artifact = db.get_document_artifact(workflow_id, artifact_id) - if not artifact: - raise HTTPException(404, f"Artifact not found: {artifact_id}") - - storage_uri = artifact["storage_uri"] - if storage_uri.startswith("minio://"): - path = storage_uri.replace("minio://", "") - bucket, object_name = path.split("/", 1) - response = minio_client.get_object(bucket, object_name) - return StreamingResponse( - response, - media_type=artifact.get("mime_type") or "application/octet-stream", - headers={"Content-Disposition": f'inline; filename="{artifact.get("filename") or "artifact"}"'}, - ) - - file_path = Path(storage_uri) - if not file_path.exists(): - raise HTTPException(404, "Artifact content not found") - return StreamingResponse( - open(file_path, "rb"), - media_type=artifact.get("mime_type") or "application/octet-stream", - headers={"Content-Disposition": f'inline; filename="{artifact.get("filename") or file_path.name}"'}, - ) - - -@app.get("/documents/{workflow_id}/jobs") -async def list_document_jobs(workflow_id: str, user: RequireSearch, limit: int = Query(20, le=100)): - _require_document_for_user(workflow_id, user) - return db.list_document_jobs(workflow_id, limit=limit) - - -@app.get("/documents/{workflow_id}/stage-io") -async def get_document_stage_io(workflow_id: str, user: RequireSearch): - doc = _require_document_for_user(workflow_id, user) - return _build_stage_io_payload(workflow_id, current_stage=doc.get("stage")) - - -@app.get("/documents/{workflow_id}/allowed-actions") -async def get_document_allowed_actions(workflow_id: str, user: RequireSearch): - """Return the currently valid machine-facing actions for a document.""" - doc = _require_document_for_user(workflow_id, user) - return { - "workflow_id": workflow_id, - "stage": doc.get("stage"), - "reindex_required": bool(doc.get("reindex_required")), - "available_actions": _list_available_actions(doc, db.get_latest_document_job(workflow_id)), - } - - -@app.get("/documents/{workflow_id}/graph", response_model=DocumentGraph) -async def get_document_graph(workflow_id: str, user: RequireSearch): - """Return a document-centric graph of state, jobs, artifacts, index status, and runtime.""" - doc = _require_document_for_user(workflow_id, user) - detail = _build_document_detail(doc) - return DocumentGraph( - workflow_id=workflow_id, - document=detail, - jobs=db.list_document_jobs(workflow_id, limit=100), - artifacts=detail.artifacts, - index_status=detail.index_status, - stage_io=_build_stage_io_payload(workflow_id, current_stage=doc.get("stage")), - runtime=await _get_runtime_payload(workflow_id, doc=doc), - ) - - -def _delete_artifacts_from_minio(workflow_id: str) -> dict: - """Remove every MinIO object backing a document's artifacts. - - Artifacts are the only part of a document that lives outside Postgres and the - vector store — the original upload, the normalized PDF, and the OCR / - translation / chunk JSON snapshots. Nothing else reads them once the - document is gone, so a hard delete has to sweep them explicitly. - """ - outcome = {"deleted": 0, "missing": 0, "errors": []} - try: - artifacts = db.list_document_artifacts(workflow_id) - except Exception as exc: - outcome["errors"].append(f"list artifacts: {exc}") - return outcome - - for artifact in artifacts: - storage_uri = (artifact.get("storage_uri") or "").strip() - if not storage_uri: - continue - try: - if storage_uri.startswith("minio://"): - bucket, object_name = storage_uri.replace("minio://", "").split("/", 1) - minio_client.remove_object(bucket, object_name) - outcome["deleted"] += 1 - else: - path = Path(storage_uri) - if path.exists(): - path.unlink() - outcome["deleted"] += 1 - else: - outcome["missing"] += 1 - except Exception as exc: - outcome["errors"].append(f"{storage_uri}: {exc}") - return outcome - - -@app.delete("/documents/{workflow_id}") -async def disable_document( - workflow_id: str, - user: CurrentUser, - remove_from_search: bool = Query(True), - purge: bool = Query(False), - keep_audit: bool = Query(True), -): - """ - Remove a document — soft delete by default, hard delete with ``purge=true``. - - Both modes: - - Remove all chunks from the DEV vector index - - Delete PROD Qdrant points too, if this document was ever promoted - (schemes-index for scheme documents, documents-index otherwise) - - For scheme documents, also rebuild the scheme catalog - - Stop the workflow if it is still running - - Soft delete (default) then marks the document disabled in Postgres and keeps - every row and artifact, so ``POST /documents/{id}/restore`` can bring it - back. Use X-Include-Disabled: true in list_documents to see it. - - ``purge=true`` additionally deletes the MinIO artifacts (original upload, - normalized PDF, OCR/translation/chunk JSON) and every Postgres row (pages, - chunks, artifacts, jobs, index status, the document itself). This is - irreversible — restore has nothing left to restore. Audit rows are kept - unless ``keep_audit=false``, since they record the deletion itself. - - Args: - workflow_id: The document workflow ID - remove_from_search: If True (default), removes chunks from the index - purge: If True, hard delete — also drops MinIO artifacts and Postgres rows - keep_audit: If False (and purge=True), also deletes the audit trail - - Requires ``admin`` (super admin — any document), or ``delete_own`` - (state admin — only documents they uploaded, inside their own state). - A purge is irreversible, so it stays super-admin only. - """ - doc = _require_document_for_user(workflow_id, user) - - if not user.has_permission(Permission.ADMIN): - if not user.has_permission_for_instance(Permission.DELETE_OWN, doc.get("instance")): - raise HTTPException(403, "Missing permission: delete_own") - if purge: - raise HTTPException( - 403, "purge=true is restricted to super admins; delete without purge instead" - ) - owner = (doc.get("uploaded_by_user_id") or "").strip() - if not owner or owner != (user.user_id or "").strip(): - raise HTTPException(403, "You may only delete documents you uploaded") - - # A purge that skipped the index would strand vectors no row points at any - # more, so "remove from everywhere" always includes the search cleanup. - if purge: - remove_from_search = True - - result = { - "workflow_id": workflow_id, - "disabled": True, - "purged": purge, - "workflow_cancelled": False, - "workflow_terminated": False, - "vector_deleted": 0, - "scheme_qdrant_deleted": None, - "prod_qdrant_deleted": None, - "catalog_version": None, - "minio_artifacts_deleted": None, - "sqlite_rows_deleted": None, - } - - # Stop the workflow if still running. A purge terminates rather than - # cancels: cancellation lets the workflow run its next activity, which - # would write rows back into the tables we are about to delete. - try: - handle = temporal_client.get_workflow_handle(workflow_id) - if purge: - await handle.terminate(reason=f"Document purged by {user.username or user.user_id}") - result["workflow_terminated"] = True - else: - await handle.cancel() - result["workflow_cancelled"] = True - except Exception: - pass # Workflow already completed/cancelled - - # Mark as disabled in Postgres (a purge deletes the row outright below, but - # this keeps the document hidden if a later step fails partway through) - db.set_document_disabled(workflow_id, True) - - # Remove from the DEV vector index if requested - if remove_from_search: - # The current index key plus keys of earlier file versions still in DEV - # (see metadata.superseded_doc_ids). - for index, doc_id in enumerate(_vector_doc_ids(doc, "dev")): - vector_result = delete_chunks_from_index(doc_id) - if index == 0: - result["vector_deleted"] = vector_result.get("deleted", 0) - if "error" in vector_result: - result["vector_error"] = vector_result["error"] - - # PROD Qdrant cleanup, if this document was ever promoted. A document - # promoted as a scheme lives in the scheme index even if its kind was - # changed since, so the PROD index entry decides, not only the kind. - kind = (doc.get("kind") or "").strip().lower() - prod_entry = (doc.get("index_status") or {}).get("prod") or {} - prod_is_scheme = ( - kind == "scheme" - or prod_entry.get("index_name") == scheme_catalog.scheme_collection_name() - ) - if prod_is_scheme: - q_result = _delete_scheme_points_from_prod(doc) - result["scheme_qdrant_deleted"] = q_result - try: - result["catalog_version"] = scheme_catalog.on_scheme_document_disabled( - workflow_id, code=doc.get("code") - ) - except Exception as exc: - result["catalog_error"] = str(exc) - else: - result["prod_qdrant_deleted"] = _delete_document_points_from_prod(doc) - - # Hard delete: drop the artifacts and the Postgres rows themselves. Ordered - # storage-then-database so a failure mid-way leaves rows pointing at - # already-deleted objects (visible, fixable) rather than orphaned objects - # no row references any more (invisible, unreclaimable). - if purge: - result["minio_artifacts_deleted"] = await asyncio.to_thread( - _delete_artifacts_from_minio, workflow_id - ) - # Audit is written before the purge so the trail survives keep_audit=True. - _log_audit( - workflow_id=workflow_id, - action_type="purge_document", - entity_type="document", - metadata={ - "filename": doc.get("filename"), - "code": doc.get("code"), - "stage": doc.get("stage"), - "minio_artifacts_deleted": result["minio_artifacts_deleted"], - "keep_audit": keep_audit, - }, - user=user, - ) - result["sqlite_rows_deleted"] = db.purge_document(workflow_id, keep_audit=keep_audit) - result["disabled"] = False # nothing left to be disabled - return result - - # Log audit - _log_audit( - workflow_id=workflow_id, - action_type="disable_document", - entity_type="document", - metadata={ - "remove_from_search": remove_from_search, - "vector_deleted": result["vector_deleted"], - "scheme_qdrant_deleted": result.get("scheme_qdrant_deleted"), - "prod_qdrant_deleted": result.get("prod_qdrant_deleted"), - "code": doc.get("code"), - "catalog_version": result.get("catalog_version"), - }, - user=user, - ) - - return result - - -@app.patch("/documents/{workflow_id}/scheme-metadata") -async def patch_scheme_metadata( - workflow_id: str, - body: SchemeMetadataUpdate, - user: RequireReview, -): - """ - Set code / name / aliases / validity used by Master Catalog and Qdrant scheme promote. - - On completed docs, renaming the code sets pending_reindex (excluded from live - vector_schemes until re-promote). Other metadata edits bump catalog version only. - """ - _require_document_for_user(workflow_id, user) - try: - result = scheme_catalog.apply_scheme_metadata( - workflow_id, - code=body.code, - name=body.name, - aliases=body.aliases, - tool_routing=body.tool_routing, - catalog_visible=body.catalog_visible, - network_visible=body.network_visible, - valid_from=body.valid_from, - valid_to=body.valid_to, - display_name=body.display_name, - ) - except LookupError as exc: - raise HTTPException(404, str(exc)) from exc - except display_name_rules.DisplayNameLocked as exc: - raise HTTPException(409, str(exc)) from exc - except ValueError as exc: - raise HTTPException(400, str(exc)) from exc - - _log_audit( - workflow_id=workflow_id, - action_type="scheme_metadata_update", - entity_type="document", - metadata={ - "body": body.model_dump(exclude_none=True), - "catalog_version": result.get("catalog_version"), - "pending_reindex": result.get("pending_reindex"), - }, - user=user, - ) - doc = result["document"] - return { - "workflow_id": workflow_id, - "kind": doc.get("kind"), - "code": doc.get("code"), - "name": doc.get("name"), - "aliases": list(doc.get("aliases") or []), - "tool_routing": doc.get("tool_routing"), - "catalog_visible": bool(int(doc["catalog_visible"])) if doc.get("catalog_visible") is not None else True, - "network_visible": bool(int(doc["network_visible"])) if doc.get("network_visible") is not None else True, - "valid_from": doc.get("valid_from"), - "valid_to": doc.get("valid_to"), - "catalog_version": result.get("catalog_version"), - "requires_reindex": result.get("requires_reindex"), - "pending_reindex": result.get("pending_reindex"), - "reindex_reason": doc.get("reindex_reason"), - } - - -# --------------------------------------------------------------------------- -# Master Catalog API (AI tool / prompt sync) -# --------------------------------------------------------------------------- - - -@app.get("/catalog/v1/version") -async def catalog_version(request: Request, user: CurrentUser): - """Cheap poll for cache warmers: { version, updated_at }.""" - _require_catalog_read(request, user) - meta = db.get_catalog_meta() - return { - "version": int(meta.get("version") or 0), - "updated_at": meta.get("updated_at"), - "notes": meta.get("notes"), - } - - -@app.get("/catalog/v1/schemes") -async def catalog_schemes( - request: Request, - user: CurrentUser, - instance: Optional[str] = Query(None), - status: str = Query("live"), - include_pending: bool = Query(False), - network_visible: Optional[bool] = Query(None), -): - """List schemes from the master catalog (vector-indexed + filters).""" - _require_catalog_read(request, user) - snap = scheme_catalog.build_snapshot( - instance=instance, - status=status, - include_pending=include_pending, - network_visible=network_visible, - ) - return { - "catalog_version": snap["catalog_version"], - "updated_at": snap["updated_at"], - "vector_schemes": snap["vector_schemes"], - "legacy_schemes": snap["legacy_schemes"], - "routing_exceptions": snap["routing_exceptions"], - } - - -@app.get("/catalog/v1/snapshot") -async def catalog_snapshot( - request: Request, - user: CurrentUser, - instance: Optional[str] = Query(None), - status: str = Query("live"), - include_pending: bool = Query(False), - network_visible: Optional[bool] = Query(None), -): - """ - Full catalog snapshot for OAN / provider warmers. - - Includes vector_schemes, legacy_schemes, tool_prompts, routing_exceptions, - collections metadata, and monotonic catalog_version. - """ - _require_catalog_read(request, user) - return scheme_catalog.build_snapshot( - instance=instance, - status=status, - include_pending=include_pending, - network_visible=network_visible, - ) - - -@app.get("/catalog/v1/tool-prompt") -async def catalog_tool_prompt(request: Request, user: CurrentUser): - """Prompt fragments only (for AI system prompt assembly).""" - _require_catalog_read(request, user) - snap = scheme_catalog.build_snapshot() - return { - "catalog_version": snap["catalog_version"], - "updated_at": snap["updated_at"], - "tool_prompts": snap["tool_prompts"], - "routing_exceptions": snap["routing_exceptions"], - "vector_scheme_codes": [s["scheme_code"] for s in snap["vector_schemes"]], - "legacy_scheme_codes": [s["scheme_code"] for s in snap["legacy_schemes"]], - } - - -@app.post("/catalog/v1/rebuild") -async def catalog_rebuild(request: Request, user: RequireAdmin): - """Admin: bump the catalog version (entries are computed from documents).""" - from ..auth.permissions import Permission - - # Admin JWT always OK; service key also accepted for automation - if not _service_key_ok(request) and not user.has_permission(Permission.ADMIN): - raise HTTPException(403, "Rebuild requires admin") - version = scheme_catalog.rebuild_catalog_from_documents(reason="admin_rebuild") - snap = scheme_catalog.build_snapshot() - _log_audit( - workflow_id="__catalog__", - action_type="catalog_rebuild", - entity_type="catalog", - metadata={"catalog_version": version, "vector_count": len(snap["vector_schemes"])}, - user=user, - ) - return { - "ok": True, - "catalog_version": version, - "vector_scheme_count": len(snap["vector_schemes"]), - "updated_at": snap["updated_at"], - } - - -@app.post("/documents/{workflow_id}/restore") -async def restore_document(workflow_id: str, user: RequireAdmin): - """ - Restore a soft-deleted (disabled) document. - - Note: This only restores the document in Postgres. Chunks that were removed - from the vector index will NOT be automatically re-indexed. To re-index, - you would need to re-run the ingestion process. - """ - _require_document_for_user(workflow_id, user) - - db.set_document_disabled(workflow_id, False) - - # Log audit - _log_audit( - workflow_id=workflow_id, - action_type="restore_document", - entity_type="document", - user=user, - ) - - return {"workflow_id": workflow_id, "restored": True} - - -@app.post("/documents/{workflow_id}/reingest") -async def reingest_document( - workflow_id: str, - user: RequirePipeline, - index_name: str = "documents-index", -): - """ - Re-ingest a completed document to the vector index. - - Use this to re-ingest documents that completed but weren't properly - indexed (e.g., due to index schema changes). This starts a lightweight - workflow that uses chunks already stored in Postgres. - - The document must have chunks stored in Postgres (typically from a - completed or previously ingested document). - """ - # Get document from Postgres - doc = _require_document_for_user(workflow_id, user) - - # Get chunks from Postgres - chunks = db.get_chunks(workflow_id, include_excluded=False) - if not chunks: - raise HTTPException(400, "No chunks found for document. The document may need to be reprocessed from scratch.") - - document_id = doc.get("document_id", "") - filename = doc.get("filename", "") - page_count = doc.get("page_count", 0) - - # Generate unique workflow ID for re-ingestion - import time - reingest_workflow_id = f"{workflow_id}-reingest-{int(time.time())}" - - # Start re-ingestion workflow - await temporal_client.start_workflow( - ReingestionWorkflow.run, - args=[ - document_id, - filename, - workflow_id, # original workflow_id for Postgres updates - page_count, - len(chunks), - index_name - ], - id=reingest_workflow_id, - task_queue=TASK_QUEUE, - ) - db.create_document_job( - workflow_id=workflow_id, - job_type="reingest", - temporal_workflow_id=reingest_workflow_id, - status="running", - current_stage="ingesting", - config={"index_name": index_name, "chunk_count": len(chunks)}, - actor=_actor_from_user(user), - ) - - # Log audit - _log_audit( - workflow_id=workflow_id, - action_type="reingest_started", - entity_type="document", - metadata={"reingest_workflow_id": reingest_workflow_id, "chunk_count": len(chunks)}, - user=user, - ) - - return { - "workflow_id": workflow_id, - "reingest_workflow_id": reingest_workflow_id, - "chunk_count": len(chunks), - "status": "started" - } - - -@app.post("/documents/{workflow_id}/retry-ingestion") -async def retry_ingestion( - workflow_id: str, - user: RequirePipeline, - index_name: str = "documents-index", -): - """Alias for reingesting a document when search is stale or missing.""" - return await reingest_document( - workflow_id, - user=user, - index_name=index_name, - ) - - -@app.post("/documents/{workflow_id}/retry-ocr") -async def retry_ocr(workflow_id: str, user: RequirePipeline): - """Retry OCR for an existing document and stop at OCR review.""" - doc = _require_document_for_user(workflow_id, user) - filepath = doc.get("filepath") - if not filepath: - raise HTTPException(400, "Document has no source filepath for OCR retry") - temporal_workflow_id = f"{workflow_id}-retry-ocr-{int(datetime.utcnow().timestamp())}" - await temporal_client.start_workflow( - OcrOnlyWorkflow.run, - args=[workflow_id, doc["document_id"], doc["filename"], filepath], - id=temporal_workflow_id, - task_queue=TASK_QUEUE, - ) - db.create_document_job( - workflow_id=workflow_id, - job_type="ocr_retry", - temporal_workflow_id=temporal_workflow_id, - status="running", - current_stage="ocr_processing", - config={"source": "api_retry_ocr"}, - actor=_actor_from_user(user), - ) - db.update_document_fields(workflow_id, error_message=None) - _log_audit( - workflow_id=workflow_id, - action_type="retry_ocr", - entity_type="document", - metadata={"temporal_workflow_id": temporal_workflow_id}, - user=user, - ) - return {"workflow_id": workflow_id, "status": "started", "retry_workflow_id": temporal_workflow_id} - - -@app.post("/documents/{workflow_id}/retry-translation") -async def retry_translation(workflow_id: str, user: RequirePipeline): - """Retry translation for an existing document and stop at translation review.""" - doc = _require_document_for_user(workflow_id, user) - if not db.get_pages(workflow_id): - raise HTTPException(400, "No OCR pages found for translation retry") - temporal_workflow_id = f"{workflow_id}-retry-translation-{int(datetime.utcnow().timestamp())}" - await temporal_client.start_workflow( - TranslationOnlyWorkflow.run, - args=[workflow_id, doc["document_id"], doc["filename"]], - id=temporal_workflow_id, - task_queue=TASK_QUEUE, - ) - db.create_document_job( - workflow_id=workflow_id, - job_type="translation_retry", - temporal_workflow_id=temporal_workflow_id, - status="running", - current_stage="translation_processing", - config={"source": "api_retry_translation"}, - actor=_actor_from_user(user), - ) - db.update_document_fields(workflow_id, error_message=None) - _log_audit( - workflow_id=workflow_id, - action_type="retry_translation", - entity_type="document", - metadata={"temporal_workflow_id": temporal_workflow_id}, - user=user, - ) - return {"workflow_id": workflow_id, "status": "started", "retry_workflow_id": temporal_workflow_id} - - -@app.post("/documents/{workflow_id}/retry-chunking") -async def retry_chunking( - workflow_id: str, - user: RequirePipeline, - chunk_size: int = 450, - chunk_overlap: int = 128, - min_tokens: int = 100, -): - """Retry chunking for an existing document and stop at chunk review.""" - doc = _require_document_for_user(workflow_id, user) - if not db.get_pages(workflow_id): - raise HTTPException(400, "No page content found for chunking retry") - temporal_workflow_id = f"{workflow_id}-retry-chunking-{int(datetime.utcnow().timestamp())}" - await temporal_client.start_workflow( - ChunkingOnlyWorkflow.run, - args=[ - workflow_id, - doc["document_id"], - doc["filename"], - doc.get("page_count", 0), - chunk_size, - chunk_overlap, - min_tokens, - ], - id=temporal_workflow_id, - task_queue=TASK_QUEUE, - ) - db.create_document_job( - workflow_id=workflow_id, - job_type="chunking_retry", - temporal_workflow_id=temporal_workflow_id, - status="running", - current_stage="chunking", - config={ - "source": "api_retry_chunking", - "chunk_size": chunk_size, - "chunk_overlap": chunk_overlap, - "min_tokens": min_tokens, - }, - actor=_actor_from_user(user), - ) - db.update_document_fields(workflow_id, error_message=None) - _log_audit( - workflow_id=workflow_id, - action_type="retry_chunking", - entity_type="document", - metadata={"temporal_workflow_id": temporal_workflow_id}, - user=user, - ) - return {"workflow_id": workflow_id, "status": "started", "retry_workflow_id": temporal_workflow_id} - - -@app.post("/documents/{workflow_id}/mark-reindex-required") -async def mark_reindex_required(workflow_id: str, payload: ReindexStateRequest, user: RequirePipeline): - """Mark a document as needing reindex after chunk edits or operational drift.""" - _require_document_for_user(workflow_id, user) - updated = _mark_reindex_required( - workflow_id, - payload.reason or "Marked manually for reindex", - metadata={"source": "api"}, - ) - return { - "workflow_id": workflow_id, - "reindex_required": bool(updated.get("reindex_required")) if updated else True, - "reindex_reason": updated.get("reindex_reason") if updated else payload.reason, - } - - -@app.post("/documents/{workflow_id}/clear-reindex-required") -async def clear_reindex_required(workflow_id: str, user: RequirePipeline): - """Clear the reindex-required flag after verification or reingestion.""" - doc = _require_document_for_user(workflow_id, user) - old_reason = doc.get("reindex_reason") - updated = db.mark_document_reindex_required(workflow_id, False) - _log_audit( - workflow_id=workflow_id, - action_type="clear_reindex_required", - entity_type="document", - field_name="reindex_required", - old_value="true", - new_value="false", - metadata={"reason": old_reason}, - user=user, - ) - return { - "workflow_id": workflow_id, - "reindex_required": bool(updated.get("reindex_required")) if updated else False, - "reindex_reason": updated.get("reindex_reason") if updated else None, - } - - -async def _reconcile_single_document(doc: dict) -> dict: - workflow_id = doc.get("workflow_id") - current_stage = doc.get("stage") - materialized = db.reconcile_materialized_state(workflow_id) - if materialized and materialized.get("updated"): - doc = db.get_document(workflow_id) or doc - current_stage = doc.get("stage") - return { - "workflow_id": workflow_id, - "action": "materialized_state_reconciled", - "to": current_stage, - "page_count": doc.get("page_count", 0), - "chunk_count": doc.get("chunk_count", 0), - "job_status": materialized.get("job_status"), - "job_stage": materialized.get("job_stage"), - } - - current_job = db.get_latest_document_job(workflow_id) - runtime_workflow_id = ( - current_job.get("temporal_workflow_id") - if current_job and current_job.get("status") == "running" and current_job.get("temporal_workflow_id") - else workflow_id - ) - - try: - handle = temporal_client.get_workflow_handle(runtime_workflow_id) - state = await asyncio.wait_for( - handle.query("get_state"), - timeout=5.0, - ) - temporal_stage = state.get("stage") if state else None - if temporal_stage and temporal_stage != current_stage: - db.update_document_stage(workflow_id, temporal_stage) - return { - "workflow_id": workflow_id, - "action": "stage_synced", - "from": current_stage, - "to": temporal_stage, - "temporal_workflow_id": runtime_workflow_id, - } - return { - "workflow_id": workflow_id, - "action": "no_change", - "stage": current_stage, - "temporal_workflow_id": runtime_workflow_id, - } - except asyncio.TimeoutError: - db.update_document_stage(workflow_id, "failed", error_message="Workflow query timed out") - return { - "workflow_id": workflow_id, - "action": "marked_failed", - "from": current_stage, - "reason": "query_timeout", - } - except Exception as exc: - error_msg = str(exc) - if "not found" in error_msg.lower() or "workflow task" in error_msg.lower(): - db.update_document_stage(workflow_id, "failed", error_message="Workflow terminated or lost") - db.log_audit( - workflow_id=workflow_id, - document_id=doc.get("document_id", ""), - action_type="reconcile_failed", - metadata={"from_stage": current_stage, "reason": "workflow_not_found"}, - ) - return { - "workflow_id": workflow_id, - "action": "marked_failed", - "from": current_stage, - "reason": "workflow_not_found", - } - return { - "workflow_id": workflow_id, - "action": "error", - "from": current_stage, - "reason": error_msg, - } - - -@app.post("/documents/{workflow_id}/reconcile") -async def reconcile_single_document(workflow_id: str, user: RequirePipeline): - """Reconcile Postgres stage with Temporal state for one document.""" - doc = _require_document_for_user(workflow_id, user) - return await _reconcile_single_document(doc) - - -# ============================================================================= -# Approval Routes -# ============================================================================= - -_STAGE_APPROVAL_HINTS = { - "ocr_review": "POST /documents/{id}/approve-ocr", - "translation_review": "POST /documents/{id}/approve-translation", - "chunk_review": "POST /documents/{id}/approve-chunks", - "ready_for_ingestion": "POST /documents/{id}/approve-ingestion", - "approval_for_prod": "POST /documents/{id}/approve-prod", -} - - -async def _validate_approval_stage(workflow_id: str, expected_stage: str): - """Validate that workflow is in the expected stage before approval.""" - try: - handle = temporal_client.get_workflow_handle(workflow_id) - state = await handle.query("get_state") - current_stage = state.get("stage") if isinstance(state, dict) else getattr(state, "stage", None) - if current_stage != expected_stage: - hint = _STAGE_APPROVAL_HINTS.get(str(current_stage or "")) - extra = f" Use {hint.replace('{id}', workflow_id)} instead." if hint else "" - raise HTTPException( - 400, - ( - f"Cannot approve: workflow is in '{current_stage}' stage, " - f"expected '{expected_stage}'.{extra}" - ), - ) - return handle - except HTTPException: - raise - except Exception: - # Try Postgres fallback to check if workflow exists but is completed/failed - doc = db.get_document(workflow_id) - if doc: - current = doc.get("stage") - hint = _STAGE_APPROVAL_HINTS.get(str(current or "")) - extra = f" Use {hint.replace('{id}', workflow_id)} instead." if hint else "" - raise HTTPException( - 400, - ( - f"Cannot approve: workflow is in '{current}' stage " - f"(completed/failed workflows cannot be approved).{extra}" - ), - ) - raise HTTPException(404, f"Workflow not found: {workflow_id}") - - -async def _execute_bulk_approval_action( - request: BulkWorkflowActionRequest, - action: str, - expected_stage: str, - signal_method, - user: AuthUser, -) -> BulkWorkflowActionResponse: - results: list[BulkWorkflowActionResult] = [] - for workflow_id in request.workflow_ids: - doc = _document_for_user_or_none(workflow_id, user) - if not doc: - results.append(BulkWorkflowActionResult( - workflow_id=workflow_id, - ok=False, - action=action, - message="document_not_found", - )) - continue - current_stage = doc.get("stage") - if current_stage != expected_stage: - results.append(BulkWorkflowActionResult( - workflow_id=workflow_id, - ok=False, - action=action, - message=f"invalid_stage:{current_stage}", - )) - continue - if request.dry_run: - results.append(BulkWorkflowActionResult( - workflow_id=workflow_id, - ok=True, - action=action, - message="would_execute", - )) - continue - try: - handle = await _validate_approval_stage(workflow_id, expected_stage) - await handle.signal(signal_method) - results.append(BulkWorkflowActionResult( - workflow_id=workflow_id, - ok=True, - action=action, - message="queued", - )) - except Exception as exc: - results.append(BulkWorkflowActionResult( - workflow_id=workflow_id, - ok=False, - action=action, - message=str(exc), - )) - - return BulkWorkflowActionResponse( - action=action, - dry_run=request.dry_run, - requested=len(request.workflow_ids), - succeeded=sum(1 for result in results if result.ok), - failed=sum(1 for result in results if not result.ok), - results=results, - ) - - -@app.post("/documents/bulk/approve-ocr", response_model=BulkWorkflowActionResponse) -async def bulk_approve_ocr(request: BulkWorkflowActionRequest, user: RequireReview): - """Bulk-approve documents waiting in OCR review.""" - return await _execute_bulk_approval_action( - request, - action="approve_ocr", - expected_stage="ocr_review", - signal_method=DocumentPipelineWorkflow.approve_ocr, - user=user, - ) - - -@app.post("/documents/bulk/approve-translation", response_model=BulkWorkflowActionResponse) -async def bulk_approve_translation(request: BulkWorkflowActionRequest, user: RequireReview): - """Bulk-approve documents waiting in translation review.""" - return await _execute_bulk_approval_action( - request, - action="approve_translation", - expected_stage="translation_review", - signal_method=DocumentPipelineWorkflow.approve_translation, - user=user, - ) - - -@app.post("/documents/bulk/approve-chunks", response_model=BulkWorkflowActionResponse) -async def bulk_approve_chunks(request: BulkWorkflowActionRequest, user: RequireReview): - """Bulk-approve documents waiting in chunk review.""" - return await _execute_bulk_approval_action( - request, - action="approve_chunks", - expected_stage="chunk_review", - signal_method=DocumentPipelineWorkflow.approve_chunks, - user=user, - ) - - -@app.post("/documents/bulk/reindex", response_model=BulkWorkflowActionResponse) -async def bulk_reindex_documents( - request: BulkWorkflowActionRequest, - user: RequirePipeline, - index_name: str = "documents-index", -): - """Bulk queue reingestion for completed or dirty documents.""" - results: list[BulkWorkflowActionResult] = [] - for workflow_id in request.workflow_ids: - doc = _document_for_user_or_none(workflow_id, user) - if not doc: - results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=False, action="reindex", message="document_not_found")) - continue - if doc.get("stage") not in {"completed", "ready_for_ingestion", "chunk_review"} and not doc.get("reindex_required"): - results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=False, action="reindex", message=f"invalid_stage:{doc.get('stage')}")) - continue - if request.dry_run: - results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=True, action="reindex", message="would_execute")) - continue - try: - await reingest_document(workflow_id, user=user, index_name=index_name) - results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=True, action="reindex", message="queued")) - except Exception as exc: - results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=False, action="reindex", message=str(exc))) - - return BulkWorkflowActionResponse( - action="reindex", - dry_run=request.dry_run, - requested=len(request.workflow_ids), - succeeded=sum(1 for result in results if result.ok), - failed=sum(1 for result in results if not result.ok), - results=results, - ) - - -@app.post("/documents/{workflow_id}/approve-ocr") -async def approve_ocr(workflow_id: str, user: RequireReview): - """Approve OCR results and continue to chunking. Requires permission: review.""" - _require_document_for_user(workflow_id, user) - handle = await _validate_approval_stage(workflow_id, "ocr_review") - await handle.signal(DocumentPipelineWorkflow.approve_ocr) - - # Log approval - _log_audit( - workflow_id=workflow_id, - action_type="approval", - entity_type="document", - field_name="ocr_approved", - new_value=True, - metadata={"stage": "ocr_review", "next_stage": "translation_processing"}, - user=user, - ) - - return {"approved": "ocr", "workflow_id": workflow_id} - - -@app.post("/documents/{workflow_id}/approve-chunks") -async def approve_chunks(workflow_id: str, user: RequireReview): - """Approve chunks and continue to prepare for ingestion.""" - _require_document_for_user(workflow_id, user) - handle = await _validate_approval_stage(workflow_id, "chunk_review") - await handle.signal(DocumentPipelineWorkflow.approve_chunks) - - # Log approval - _log_audit( - workflow_id=workflow_id, - action_type="approval", - entity_type="document", - field_name="chunks_approved", - new_value=True, - metadata={"stage": "chunk_review", "next_stage": "ready_for_ingestion"}, - user=user, - ) - - return {"approved": "chunks", "workflow_id": workflow_id} - - -@app.post("/documents/{workflow_id}/approve-translation") -async def approve_translation(workflow_id: str, user: RequireReview): - """Approve translations and continue to chunking.""" - _require_document_for_user(workflow_id, user) - handle = await _validate_approval_stage(workflow_id, "translation_review") - await handle.signal(DocumentPipelineWorkflow.approve_translation) - - # Log approval - _log_audit( - workflow_id=workflow_id, - action_type="approval", - entity_type="document", - field_name="translation_approved", - new_value=True, - metadata={"stage": "translation_review", "next_stage": "chunking"}, - user=user, - ) - - return {"approved": "translation", "workflow_id": workflow_id} - - -@app.post("/documents/{workflow_id}/approve-ingestion") -async def approve_ingestion(workflow_id: str, user: RequireApproveIngestion): - """Approve ingestion and continue to vector-index ingestion. - - Requires APPROVE_INGESTION, not REVIEW: a ``state_contributor`` may approve - the OCR / translation / chunking gates but must not publish to DEV. - """ - _require_document_for_user(workflow_id, user) - handle = await _validate_approval_stage(workflow_id, "ready_for_ingestion") - await handle.signal(DocumentPipelineWorkflow.approve_ingestion) - - # Log approval - _log_audit( - workflow_id=workflow_id, - action_type="approval", - entity_type="document", - field_name="ingestion_approved", - new_value=True, - metadata={"stage": "ready_for_ingestion", "next_stage": "ingesting"}, - user=user, - ) - - return {"approved": "ingestion", "workflow_id": workflow_id} - - -@app.post("/documents/{workflow_id}/request-prod-ready") -async def request_prod_ready(workflow_id: str, user: RequireReview): - """ - state_admin: flag this document as ready for a super_admin to review for - PROD promotion. Purely informational — does not itself fire approve_prod - or promote anything; the super_admin still makes that call via - POST /documents/{id}/approve-prod. Mirrors the "request ingest" pattern - used elsewhere in this pipeline: a request is a separate, weaker signal - from the actual privileged action. - """ - doc = _require_document_for_user(workflow_id, user) - stage = doc.get("stage") - if stage != "approval_for_prod": - raise HTTPException( - 400, - f"Cannot request prod-ready: document is in '{stage}' stage " - "(expected 'approval_for_prod').", - ) - - updated = db.mark_prod_ready_requested( - workflow_id, - user_id=getattr(user, "user_id", None), - username=getattr(user, "username", None), - ) - - _log_audit( - workflow_id=workflow_id, - action_type="request_prod_ready", - entity_type="document", - field_name="prod_ready_requested_at", - new_value=updated.get("prod_ready_requested_at") if updated else None, - metadata={"stage": stage}, - user=user, - ) - - return { - "requested": True, - "workflow_id": workflow_id, - "prod_ready_requested_at": updated.get("prod_ready_requested_at") if updated else None, - "prod_ready_requested_by_username": updated.get("prod_ready_requested_by_username") if updated else None, - } - - -@app.post("/documents/{workflow_id}/approve-prod") -async def approve_prod( - workflow_id: str, - user: RequireAdmin, -): - """Superadmin-only: promote DEV-ingested vectors into PROD Qdrant.""" - doc = _require_document_for_user(workflow_id, user) - stage = doc.get("stage") - # Promoting an already-completed document is a fresh PROD write, so it is - # blocked outright when PROD is disabled. A document still parked at the - # approval gate (started before the flag was set) is allowed through, or it - # would be stranded there forever. - if prod_stage_disabled() and stage != "approval_for_prod": - raise HTTPException( - 400, - "PROD promotion is disabled (DISABLE_PROD_SETTING=true). " - "Documents finish at DEV ingest.", - ) - if stage not in {"approval_for_prod", "completed"}: - raise HTTPException( - 400, - f"Cannot approve for prod: document is in '{stage}' stage " - "(expected 'approval_for_prod' or 'completed').", - ) - - # Prefer signaling the running main workflow when it is waiting at approval_for_prod. - signaled = False - if stage == "approval_for_prod": - try: - handle = await _validate_approval_stage(workflow_id, "approval_for_prod") - await handle.signal(DocumentPipelineWorkflow.approve_prod) - signaled = True - except HTTPException: - signaled = False - except Exception: - signaled = False - - promote_workflow_id = None - if not signaled: - import time - - promote_workflow_id = f"{workflow_id}-promote-prod-{int(time.time())}" - await temporal_client.start_workflow( - PromoteToProdWorkflow.run, - args=[ - doc.get("document_id") or workflow_id, - doc.get("filename") or workflow_id, - workflow_id, - int(doc.get("page_count") or 0), - int(doc.get("chunk_count") or 0), - ], - id=promote_workflow_id, - task_queue=TASK_QUEUE, - ) - db.create_document_job( - workflow_id=workflow_id, - job_type="promote_prod", - temporal_workflow_id=promote_workflow_id, - status="running", - current_stage="ingesting_prod", - actor=_actor_from_user(user), - ) - db.update_document_stage( - workflow_id=workflow_id, - stage="ingesting_prod", - page_count=doc.get("page_count"), - chunk_count=doc.get("chunk_count"), - ) - - if doc.get("prod_ready_requested_at"): - db.clear_prod_ready_requested(workflow_id) - - _log_audit( - workflow_id=workflow_id, - action_type="approval", - entity_type="document", - field_name="prod_approved", - new_value=True, - metadata={ - "stage": stage, - "next_stage": "ingesting_prod", - "signaled": signaled, - "promote_workflow_id": promote_workflow_id, - }, - user=user, - ) - return { - "approved": "prod", - "workflow_id": workflow_id, - "signaled": signaled, - "promote_workflow_id": promote_workflow_id, - "next_stage": "ingesting_prod", - } - - -# ============================================================================= -# Audit Log Helper and Routes -# ============================================================================= - -def _log_audit( - workflow_id: str, - action_type: str, - entity_type: str = None, - entity_id: int = None, - field_name: str = None, - old_value=None, - new_value=None, - metadata: dict = None, - user=None, -): - """ - Helper to log audit entries with JSON serialization and optional actor identity. - - Args: - workflow_id: The Temporal workflow ID - action_type: Type of action (page_edit, chunk_edit, approval, etc.) - entity_type: Type of entity (page, chunk) - entity_id: Entity identifier (page_number, chunk_number) - field_name: Name of the field changed - old_value: Previous value (will be JSON serialized if not string) - new_value: New value (will be JSON serialized if not string) - metadata: Additional context as dict - user: Optional AuthUser whose email/roles are stored on the audit row - """ - # Get document_id from Postgres - doc = db.get_document(workflow_id) - document_id = doc["document_id"] if doc else workflow_id - - # Serialize values to JSON if needed - old_str = json.dumps(old_value) if old_value is not None and not isinstance(old_value, str) else old_value - new_str = json.dumps(new_value) if new_value is not None and not isinstance(new_value, str) else new_value - - actor = _actor_from_user(user) - meta = dict(metadata or {}) - if actor.get("email") or actor.get("username") or actor.get("roles"): - meta.setdefault("actor", { - "user_id": actor.get("user_id"), - "username": actor.get("username"), - "email": actor.get("email"), - "roles": actor.get("roles"), - }) - - db.log_audit( - workflow_id=workflow_id, - document_id=document_id, - action_type=action_type, - entity_type=entity_type, - entity_id=entity_id, - field_name=field_name, - old_value=old_str, - new_value=new_str, - metadata=meta or None, - actor_user_id=actor.get("user_id") or None, - actor_username=actor.get("username") or None, - actor_email=actor.get("email") or None, - actor_roles=actor.get("roles_csv") or None, - ) - - -@app.get("/audit", response_model=AuditLogResponse) -async def get_all_audit_logs( - user: RequireSearch, - action_type: str = None, - limit: int = Query(50, le=200), - offset: int = 0 -): - """ - Get global audit trail across all documents. - - Returns a list of all changes including: - - Stage transitions - - Page edits - - Chunk edits - - Approvals - - Resets - - Each entry includes the document filename for context. - """ - scope = _instance_scope_for_user(user) - logs = db.get_all_audit_logs( - action_type=action_type, - limit=limit, - offset=offset, - instances=scope, - ) - total = db.get_all_audit_log_count(action_type, instances=scope) - - return AuditLogResponse( - logs=logs, - total=total, - limit=limit, - offset=offset - ) - - -@app.get("/documents/{workflow_id}/audit", response_model=AuditLogResponse) -async def get_document_audit_log( - workflow_id: str, - user: RequireSearch, - action_type: str = None, - limit: int = Query(50, le=200), - offset: int = 0 -): - """ - Get audit trail for a document. - - Returns a list of all changes made to the document including: - - Stage transitions - - Page edits - - Chunk edits - - Approvals - - Resets - """ - _require_document_for_user(workflow_id, user) - logs = db.get_audit_logs( - workflow_id=workflow_id, - action_type=action_type, - limit=limit, - offset=offset - ) - total = db.get_audit_log_count(workflow_id, action_type) - - return AuditLogResponse( - logs=logs, - total=total, - limit=limit, - offset=offset - ) - - -# ============================================================================= -# Page Routes (OCR Review) -# ============================================================================= - -@app.get("/documents/{workflow_id}/pages") -async def list_pages(workflow_id: str, user: RequireSearch): - """Get all pages for a document. Postgres-first for speed.""" - # Enforce tenant scope up front (404 hides other tenants' documents). - _require_document_for_user(workflow_id, user) - return db.get_pages(workflow_id) - - -@app.get("/documents/{workflow_id}/pages/{page_num}") -async def get_page(workflow_id: str, user: RequireSearch, page_num: int = PathParam(..., ge=1, le=10000, description="Page number (1-indexed)")): - """Get a specific page. Postgres-first for speed.""" - _require_document_for_user(workflow_id, user) - page = db.get_page(workflow_id, page_num) - if page: - return page - - raise HTTPException(404, f"Page {page_num} not found") - - -@app.patch("/documents/{workflow_id}/pages/{page_num}") -async def update_page( - workflow_id: str, - data: PageUpdate, - user: RequireReview, - page_num: int = PathParam(..., ge=1, le=10000, description="Page number (1-indexed)"), -): - """Update a page (edit markdown, mark reviewed).""" - doc = _require_document_for_user(workflow_id, user) - old_page = db.get_page(workflow_id, page_num) - if not old_page: - raise HTTPException(404, f"Page {page_num} not found") - - updated = db.update_page( - workflow_id, - page_num, - edited_markdown=data.edited_markdown, - is_reviewed=data.is_reviewed, - reviewer_notes=data.reviewer_notes, - edited_translation=data.edited_translation, - translation_reviewed=data.translation_reviewed, - translation_notes=data.translation_notes, - ) - if not updated: - raise HTTPException(404, f"Page {page_num} not found") - - # Log audits for changed fields - if data.edited_markdown is not None: - old_text = old_page.get("edited_markdown") or old_page.get("original_markdown", "") - _log_audit( - workflow_id=workflow_id, - action_type="page_edit", - entity_type="page", - entity_id=page_num, - field_name="edited_markdown", - old_value=old_text, - new_value=data.edited_markdown, - user=user, - ) - - if data.is_reviewed is not None: - _log_audit( - workflow_id=workflow_id, - action_type="page_edit", - entity_type="page", - entity_id=page_num, - field_name="is_reviewed", - old_value=old_page.get("is_reviewed", False), - new_value=data.is_reviewed, - user=user, - ) - - if data.reviewer_notes is not None: - _log_audit( - workflow_id=workflow_id, - action_type="page_edit", - entity_type="page", - entity_id=page_num, - field_name="reviewer_notes", - old_value=old_page.get("reviewer_notes"), - new_value=data.reviewer_notes, - user=user, - ) - - if data.edited_translation is not None: - old_translation = old_page.get("edited_translation") or old_page.get("translated_markdown", "") - _log_audit( - workflow_id=workflow_id, - action_type="translation_edit", - entity_type="page", - entity_id=page_num, - field_name="edited_translation", - old_value=old_translation, - new_value=data.edited_translation, - user=user, - ) - - if data.translation_reviewed is not None: - _log_audit( - workflow_id=workflow_id, - action_type="translation_edit", - entity_type="page", - entity_id=page_num, - field_name="translation_reviewed", - old_value=old_page.get("translation_reviewed", False), - new_value=data.translation_reviewed, - user=user, - ) - - if data.translation_notes is not None: - _log_audit( - workflow_id=workflow_id, - action_type="translation_edit", - entity_type="page", - entity_id=page_num, - field_name="translation_notes", - old_value=old_page.get("translation_notes"), - new_value=data.translation_notes, - user=user, - ) - - # Review flags/notes alone must not dirty the search index — only content edits. - content_changed = data.edited_markdown is not None or data.edited_translation is not None - if doc and content_changed and ( - doc.get("chunk_count", 0) > 0 - or doc.get("stage") in {"chunking", "chunk_review", "ready_for_ingestion", "ingesting", "approval_for_prod", "ingesting_prod", "completed"} - ): - _mark_reindex_required( - workflow_id, - "Page content changed after chunk generation; rechunk and reindex required", - metadata={"page_number": page_num}, - ) - - return db.get_page(workflow_id, page_num) - - -@app.post("/documents/{workflow_id}/pages/{page_num}/reset") -async def reset_page( - workflow_id: str, - user: RequireReview, - page_num: int = PathParam(..., ge=1, le=10000, description="Page number (1-indexed)"), -): - """Reset page to original OCR output.""" - doc = _require_document_for_user(workflow_id, user) - old_page = db.get_page(workflow_id, page_num) - if not old_page: - raise HTTPException(404, f"Page {page_num} not found") - db.reset_page(workflow_id, page_num) - - # Log reset action - _log_audit( - workflow_id=workflow_id, - action_type="page_reset", - entity_type="page", - entity_id=page_num, - field_name="edited_markdown", - old_value=old_page.get("edited_markdown") if old_page else None, - new_value=None, - metadata={"reset_to": "original_markdown"}, - user=user, - ) - - if doc and (doc.get("chunk_count", 0) > 0 or doc.get("stage") in {"chunking", "chunk_review", "ready_for_ingestion", "ingesting", "approval_for_prod", "ingesting_prod", "completed"}): - _mark_reindex_required( - workflow_id, - "Page reset after chunk generation; rechunk and reindex required", - metadata={"page_number": page_num}, - ) - - return db.get_page(workflow_id, page_num) - - -# ============================================================================= -# Chunk Routes (Chunk Review) -# ============================================================================= - -@app.get("/chunks/search") -async def search_chunks_across_documents( - user: RequireSearch, - q: str = Query("", description="Keyword search within chunk text"), - limit: int = Query(100, ge=1, le=500), - offset: int = Query(0, ge=0), - include_excluded: bool = Query(False, description="Include excluded chunks"), - stage: Optional[DocumentStage] = Query(None, description="Optional document stage filter"), -): - """Search chunks across all documents for KB maintainer workflows.""" - # Tenant-scope via the owning document's instance (None = unrestricted admin). - chunks, total = db.search_chunks( - query=q, - limit=limit, - offset=offset, - include_excluded=include_excluded, - stage=stage.value if stage else None, - instances=_instance_scope_for_user(user), - ) - return { - "items": chunks, - "total": total, - "limit": limit, - "offset": offset, - "query": q, - "include_excluded": include_excluded, - "stage": stage.value if stage else None, - } - - -@app.get("/documents/{workflow_id}/chunks") -async def list_chunks(workflow_id: str, user: RequireSearch, include_excluded: bool = False): - """Get all chunks for a document. Postgres-first for speed.""" - _require_document_for_user(workflow_id, user) - return db.get_chunks(workflow_id, include_excluded=include_excluded) - - -@app.get("/documents/{workflow_id}/chunks/{chunk_num}") -async def get_chunk(workflow_id: str, user: RequireSearch, chunk_num: int = PathParam(..., ge=1, le=10000, description="Chunk number (1-indexed)")): - """Get a specific chunk. Postgres-first for speed.""" - _require_document_for_user(workflow_id, user) - chunk = db.get_chunk(workflow_id, chunk_num) - if chunk: - return chunk - - raise HTTPException(404, f"Chunk {chunk_num} not found") - - -@app.patch("/documents/{workflow_id}/chunks/{chunk_num}") -async def update_chunk( - workflow_id: str, - data: ChunkUpdate, - user: RequireReview, - chunk_num: int = PathParam(..., ge=1, le=10000, description="Chunk number (1-indexed)"), -): - """Update a chunk (edit text, mark reviewed, exclude).""" - doc = _require_document_for_user(workflow_id, user) - old_chunk = db.get_chunk(workflow_id, chunk_num) - if not old_chunk: - raise HTTPException(404, f"Chunk {chunk_num} not found") - - updated = db.update_chunk( - workflow_id, - chunk_num, - edited_text=data.edited_text, - is_reviewed=data.is_reviewed, - is_excluded=data.is_excluded, - reviewer_notes=data.reviewer_notes, - ) - if not updated: - raise HTTPException(404, f"Chunk {chunk_num} not found") - - # Log audits for changed fields - if data.edited_text is not None: - old_text = old_chunk.get("edited_text") or old_chunk.get("original_text", "") - _log_audit( - workflow_id=workflow_id, - action_type="chunk_edit", - entity_type="chunk", - entity_id=chunk_num, - field_name="edited_text", - old_value=old_text, - new_value=data.edited_text, - user=user, - ) - - if data.is_reviewed is not None: - _log_audit( - workflow_id=workflow_id, - action_type="chunk_edit", - entity_type="chunk", - entity_id=chunk_num, - field_name="is_reviewed", - old_value=old_chunk.get("is_reviewed", False), - new_value=data.is_reviewed, - user=user, - ) - - if data.is_excluded is not None: - _log_audit( - workflow_id=workflow_id, - action_type="chunk_edit", - entity_type="chunk", - entity_id=chunk_num, - field_name="is_excluded", - old_value=old_chunk.get("is_excluded", False), - new_value=data.is_excluded, - user=user, - ) - - # If excluding a chunk and document is completed (already ingested), remove from the vector index - if data.is_excluded and not old_chunk.get("is_excluded", False): - if doc and doc.get("stage") == "completed": - doc_id = doc.get("document_id") - if doc_id: - vector_result = delete_single_chunk_from_index(doc_id, chunk_num) - if vector_result.get("deleted"): - _log_audit( - workflow_id=workflow_id, - action_type="chunk_removed_from_search", - entity_type="chunk", - entity_id=chunk_num, - metadata={"vector_id": vector_result.get("chunk_id")}, - user=user, - ) - - if data.reviewer_notes is not None: - _log_audit( - workflow_id=workflow_id, - action_type="chunk_edit", - entity_type="chunk", - entity_id=chunk_num, - field_name="reviewer_notes", - old_value=old_chunk.get("reviewer_notes"), - new_value=data.reviewer_notes, - user=user, - ) - - if data.edited_text is not None or data.is_excluded is not None: - _mark_reindex_required( - workflow_id, - "Chunk content changed; search index is out of sync", - metadata={"chunk_number": chunk_num}, - ) - - return db.get_chunk(workflow_id, chunk_num) - - -@app.post("/documents/{workflow_id}/chunks/{chunk_num}/reset") -async def reset_chunk( - workflow_id: str, - user: RequireReview, - chunk_num: int = PathParam(..., ge=1, le=10000, description="Chunk number (1-indexed)"), -): - """Reset chunk to original text.""" - _require_document_for_user(workflow_id, user) - old_chunk = db.get_chunk(workflow_id, chunk_num) - if not old_chunk: - raise HTTPException(404, f"Chunk {chunk_num} not found") - db.reset_chunk(workflow_id, chunk_num) - - # Log reset action - _log_audit( - workflow_id=workflow_id, - action_type="chunk_reset", - entity_type="chunk", - entity_id=chunk_num, - field_name="edited_text", - old_value=old_chunk.get("edited_text") if old_chunk else None, - new_value=None, - metadata={"reset_to": "original_text"}, - user=user, - ) - _mark_reindex_required( - workflow_id, - "Chunk reset; search index is out of sync", - metadata={"chunk_number": chunk_num}, - ) - - return db.get_chunk(workflow_id, chunk_num) - - -# ============================================================================= -# Export Routes -# ============================================================================= - -@app.get("/documents/{workflow_id}/export/markdown") -async def export_markdown(workflow_id: str, user: RequireSearch): - """Export document as combined markdown.""" - doc = _require_document_for_user(workflow_id, user) - - pages = db.get_pages(workflow_id) - content = [] - for page in pages: - md = ( - page.get("edited_translation") - or page.get("translated_markdown") - or page.get("edited_markdown") - or page.get("original_markdown", "") - ) - content.append(f"\n\n{md}") - - return { - "filename": doc.get("filename", "").replace(".pdf", ".md"), - "content": "\n\n---\n\n".join(content) - } - - -@app.get("/documents/{workflow_id}/export/chunks") -async def export_chunks(workflow_id: str, user: RequireSearch, include_excluded: bool = False): - """Export chunks as JSON for vector-index ingestion.""" - doc = _require_document_for_user(workflow_id, user) - - chunks = db.get_chunks(workflow_id, include_excluded=include_excluded) - doc_id = doc.get("document_id", "") - filename = doc.get("filename", "") - name = filename.replace(".pdf", "") - - records = [] - for chunk in chunks: - text = chunk.get("edited_text") or chunk.get("original_text", "") - chunk_num = chunk.get("chunk_number", 0) - - records.append({ - "_id": hashlib.md5(f"{doc_id}_{chunk_num}_{text[:50]}".encode()).hexdigest(), - "doc_id": doc_id, - "name": name, - "text": text, - "chunk_num": chunk_num, - "token_count": chunk.get("token_count", 0), - "source": "docs-pipeline" - }) - - return records - - -# ============================================================================= -# PDF Serving -# ============================================================================= - -def _inline_content_disposition(filename: str) -> str: - """Build a latin-1-safe Content-Disposition header for inline file display.""" - # Strip CR/LF/NUL so a crafted filename cannot inject response headers. - safe_name = (filename or "document.pdf").replace('"', "'") - safe_name = "".join(ch for ch in safe_name if ch not in "\r\n\0").strip() or "document.pdf" - try: - safe_name.encode("latin-1") - return f'inline; filename="{safe_name}"' - except UnicodeEncodeError: - ascii_name = safe_name.encode("ascii", "ignore").decode("ascii").strip() or "document.pdf" - encoded_name = quote(safe_name) - return f"inline; filename=\"{ascii_name}\"; filename*=UTF-8''{encoded_name}" - - -@app.get("/documents/{workflow_id}/pdf") -async def get_document_pdf(workflow_id: str, user: RequireSearch): - """ - Get the original PDF file for a document. - Returns the PDF as a streaming response. Postgres-first for speed. - """ - # Postgres-first - instant lookup, tenant-scoped (404 hides other tenants). - doc = _require_document_for_user(workflow_id, user) - - filepath = doc.get("filepath", "") - filename = doc.get("filename", "document.pdf") - - if not filepath: - raise HTTPException(404, f"Document has no PDF path: {workflow_id}") - - try: - if filepath.startswith("minio://"): - # Parse minio://bucket/object/path - path = filepath.replace("minio://", "") - parts = path.split("/", 1) - bucket = parts[0] - object_name = parts[1] if len(parts) > 1 else "" - - # Get object from MinIO - response = minio_client.get_object(bucket, object_name) - - return StreamingResponse( - response, - media_type="application/pdf", - headers={ - "Content-Disposition": _inline_content_disposition(filename) - } - ) - else: - # Local file - file_path = Path(filepath) - if not file_path.exists(): - raise HTTPException(404, f"PDF file not found: {filepath}") - - def file_iterator(): - with open(file_path, "rb") as f: - yield from f - - return StreamingResponse( - file_iterator(), - media_type="application/pdf", - headers={ - "Content-Disposition": _inline_content_disposition(filename) - } - ) - except HTTPException: - raise - except Exception as e: - # Log the actual error server-side but don't expose details to client - logging.error(f"PDF serving error for {workflow_id}: {str(e)}") - raise HTTPException(500, "Error serving PDF file") - - -@app.get("/provenance/chunk") -async def resolve_provenance_chunk( - request: Request, - user: RequireSearch, - doc_id: Optional[str] = Query(None, description="workflow slug, Postgres document_id, or legacy vector-index doc_id"), - chunk_num: Optional[int] = Query(None, alias="chunk_num"), -): - """ - Resolve a retrieved chunk to workflow metadata and maintainer URLs. - - Used by chat/retrieval clients when search hits lack workflow_id (legacy rows) or for enrichment. - """ - from ..activities import _infer_section - - resolved_doc_id = doc_id - resolved_chunk_num = chunk_num - - if not resolved_doc_id or resolved_chunk_num is None: - raise HTTPException(400, "Provide doc_id and chunk_num") - - provenance = db.resolve_chunk_provenance(doc_id=resolved_doc_id, chunk_num=int(resolved_chunk_num)) - if not provenance: - raise HTTPException(404, "Chunk provenance not found") - - workflow_id = provenance["workflow_id"] - # Tenant scope: a restricted caller may not resolve another tenant's chunk. - _require_document_for_user(workflow_id, user) - chunk = db.get_chunk(workflow_id, int(resolved_chunk_num)) - if chunk: - text = chunk.get("edited_text") or chunk.get("original_text") or "" - provenance["section"] = _infer_section(text, chunk.get("section_title")) - provenance["excerpt"] = text[:320] + ("..." if len(text) > 320 else "") - - links = _build_provenance_links(workflow_id, int(resolved_chunk_num), request) - return {**provenance, **links} - - -@app.get("/documents/{workflow_id}/qdrant") -async def get_document_qdrant_status( - workflow_id: str, - user: RequireSearch, - index_name: str = Query("documents-index"), -): - """Return vector-index status/chunks for a document.""" - from ..vector_store import get_default_index_name, get_vector_store - - doc = _require_document_for_user(workflow_id, user) - - index_doc_id = doc["document_id"] - resolved_index = index_name if index_name and index_name != "documents-index" else get_default_index_name() - store = get_vector_store() - backend = store.backend - - try: - raw_hits = store.list_by_doc_id(resolved_index, index_doc_id, limit=1000) - except Exception as exc: - raise HTTPException( - 503, - f"Vector backend '{backend}' unavailable for index status: {exc}", - ) from exc - - hits = [] - for hit in raw_hits: - normalized_hit = dict(hit) - chunk_num = normalized_hit.get("chunk_num") - normalized_hit.setdefault( - "_id", - f"{normalized_hit.get('doc_id', index_doc_id)}:{chunk_num if chunk_num is not None else 'unknown'}", - ) - normalized_hit.setdefault("chunk_number", chunk_num) - hits.append(normalized_hit) - db_chunks = db.get_chunks(workflow_id, include_excluded=True) - status = { - "workflow_id": workflow_id, - "index_name": resolved_index, - "backend": backend, - "index_doc_id": index_doc_id, - "doc_id": index_doc_id, - "sqlite_chunk_count": len([c for c in db_chunks if not c.get("is_excluded")]), - "indexed_chunk_count": len(hits), - "status": "indexed" if hits else "missing", - "hits": hits, - } - db.upsert_document_index_status( - workflow_id=workflow_id, - index_name=resolved_index, - doc_id=index_doc_id, - chunk_count_indexed=len(hits), - last_verified_at=datetime.utcnow().isoformat(), - status=status["status"], - details={"sqlite_chunk_count": status["sqlite_chunk_count"], "backend": backend}, - ) - return status - - -@app.get("/documents/{workflow_id}/qdrant/chunks") -async def list_document_qdrant_chunks( - workflow_id: str, - user: RequireSearch, - index_name: str = Query("documents-index"), -): - result = await get_document_qdrant_status(workflow_id, user, index_name=index_name) - return result["hits"] - - -@app.get("/indexes/{index_name}/settings") -async def get_vector_index_settings(index_name: str, user: RequireSearch): - from ..vector_store import get_vector_store - - try: - return get_vector_store().get_settings(index_name) - except Exception as exc: - raise HTTPException(404, f"Index '{index_name}' settings unavailable: {exc}") from exc - - -@app.get("/indexes/{index_name}/stats") -async def get_vector_index_stats(index_name: str, user: RequireSearch): - from ..vector_store import get_vector_store - - try: - return get_vector_store().get_stats(index_name) - except Exception as exc: - raise HTTPException(404, f"Index '{index_name}' stats unavailable: {exc}") from exc - - -@app.get("/indexes/summary") -async def get_vector_indexes_summary( - user: RequireSearch, - x_include_disabled: Optional[str] = Header(None, alias="X-Include-Disabled"), -): - """Summarize index coverage from Postgres-backed index status plus live vector-store stats.""" - from ..vector_store import get_vector_store - - include_disabled = x_include_disabled and x_include_disabled.lower() == "true" - summaries = db.list_index_summaries( - include_disabled=include_disabled, - ) - if not summaries: - return [] - - store = get_vector_store() - backend = store.backend - - results = [] - for summary in summaries: - live_stats = None - live_error = None - try: - live_stats = store.get_stats(summary["index_name"]) - except Exception as exc: - live_error = str(exc) - results.append({ - **summary, - "backend": backend, - "live_stats": live_stats, - "live_error": live_error, - }) - return results - - - - -# ============================================================================= -# Health -# ============================================================================= - -@app.get("/health") -async def health(): - """Health check.""" - return { - "status": "ok", - "temporal_connected": temporal_client is not None, - "database": await asyncio.to_thread(db.check_health), - } - - -# ============================================================================= -# Webhook (inbound pushes from external systems, x-api-key protected) -# ============================================================================= - -WEBHOOK_MAX_RECORDS = int(os.environ.get("WEBHOOK_MAX_RECORDS", "500")) - - -def require_webhook_key(request: Request) -> None: - """Dependency: x-api-key must equal WEBHOOK_API_KEY. Fails closed when unset.""" - import hmac - - expected = (os.environ.get("WEBHOOK_API_KEY") or "").strip() - if not expected: - raise HTTPException(503, "WEBHOOK_API_KEY is not configured") - provided = (request.headers.get("x-api-key") or "").strip() - if not provided or not hmac.compare_digest(provided, expected): - raise HTTPException(401, "Missing or invalid x-api-key") - - -@app.post("/webhook") -@limiter.limit(RATE_LIMIT_UPLOAD) -async def webhook( - request: Request, - records: List[Dict[str, Any]], - _: None = Depends(require_webhook_key), -): - """Accept a JSON array of records and upsert them into webhook_records.""" - if len(records) > WEBHOOK_MAX_RECORDS: - raise HTTPException(413, f"Too many records (max {WEBHOOK_MAX_RECORDS})") - try: - count = db.upsert_webhook_records("webhook", records) - except ValueError as e: - raise HTTPException(422, str(e)) - await _start_advisory_index_for(records) - return { - "success": True, - "message": "Webhook processed successfully.", - "recordCount": count, - } - - -async def _start_advisory_index_for(records: list) -> int: - """Start one advisory index workflow per record of this call that is not indexed yet. - - Best effort: the records are already saved, so a Temporal failure never fails the - webhook; the rows stay unindexed and the reindex endpoint (or worker startup backfill) - picks them up. - """ - if temporal_client is None: - logger.warning("Temporal client not ready; advisory indexing skipped for this webhook call") - return 0 - try: - external_ids = [db.webhook_external_id(r) for r in records if isinstance(r, dict)] - record_ids = db.get_unindexed_webhook_record_ids(external_ids=external_ids) - return await start_advisory_index(temporal_client, record_ids) - except Exception: - logger.exception("Could not start advisory indexing for this webhook call") - return 0 - - -class AdvisoryReindexBody(BaseModel): - record_ids: Optional[List[str]] = None - force: bool = False - - -class AdvisoryPromoteBody(BaseModel): - record_ids: Optional[List[str]] = None - - -def _require_temporal() -> Client: - if temporal_client is None: - raise HTTPException(503, "Temporal client is not ready") - return temporal_client - - -@app.post("/webhook/advisory/reindex") -async def advisory_reindex(body: AdvisoryReindexBody, user: RequireAdmin): - """Queue advisory indexing (dev collection): given record ids, else all unindexed (all with force).""" - client = _require_temporal() - if body.record_ids is not None: - record_ids = body.record_ids - elif body.force: - record_ids = db.get_webhook_record_ids() - else: - record_ids = db.get_unindexed_webhook_record_ids() - queued = await start_advisory_index(client, record_ids) - return {"success": True, "queued": queued} - - -@app.post("/webhook/advisory/promote") -async def advisory_promote(body: AdvisoryPromoteBody, user: RequireAdmin): - """Queue promotion to the PROD collection: given record ids, else all published records already indexed in dev.""" - client = _require_temporal() - if body.record_ids is not None: - record_ids = body.record_ids - else: - record_ids = db.get_webhook_record_ids(published_only=True, indexed_only=True) - queued = await start_advisory_promote(client, record_ids) - return {"success": True, "queued": queued} - - -@app.get("/webhook/advisory/status") -async def advisory_status(user: RequireAdmin): - """Counts of webhook advisory records: total, published, indexed, unindexed, promoted.""" - return db.advisory_status_counts() - - -# ============================================================================= -# Vector index (passage schema) -# ============================================================================= - -@app.get("/admin/index/schema") -async def get_vector_index_schema( - user: RequireAdmin, - index_name: str = Query("documents-index", description="Vector index / collection name"), -): - """Report the live vector index's field schema vs. the canonical passage schema.""" - from ..activities import _passage_schema_field_names - from ..vector_store import get_default_index_name, get_vector_store - - resolved = index_name or get_default_index_name() - store = get_vector_store() - try: - index_settings = store.get_settings(resolved) - except Exception as exc: - raise HTTPException(404, f"Index '{resolved}' not found: {exc}") from exc - - field_names = sorted( - f.get("name") - for f in (index_settings.get("allFields") or []) - if isinstance(f, dict) and f.get("name") - ) - canonical_fields = sorted(_passage_schema_field_names()) - missing_fields = sorted(set(canonical_fields) - set(field_names)) - - return { - "index_name": resolved, - "backend": store.backend, - "fields": field_names, - "canonical_passage_fields": canonical_fields, - "missing_canonical_fields": missing_fields, - } - - -@app.post("/admin/index/create") -async def create_vector_index( - user: RequireAdmin, - index_name: str = Query("documents-index", description="Vector index / collection name"), - recreate_if_exists: bool = Query(False, description="If true, delete existing index and create with passage schema"), -): - """Create the vector index/collection — a cosine Qdrant collection sized from EMBEDDING_VECTOR_SIZE.""" - _ = user - from ..vector_store import get_default_index_name, get_vector_store - - resolved = index_name or get_default_index_name() - store = get_vector_store() - result = store.ensure_collection(resolved, recreate=recreate_if_exists) - return { - "index_name": resolved, - "backend": store.backend, - **result, - } - - -@app.get("/admin/ingest-info") -async def get_ingest_info(user: RequireAdmin): - """ - Return what the running container's ingest code would send to the vector index. - Use this to verify the API/worker image has the passage schema (text_for_embedding, etc.). - """ - from ..activities import _passage_schema_field_names, _prepare_records - - passage_fields = sorted(_passage_schema_field_names()) - has_text_for_embedding = "text_for_embedding" in set(passage_fields) - # One fake chunk to see exact record shape the worker would send - fake_chunk = { - "chunk_number": 0, - "original_text": "Sample text.", - "edited_text": None, - "is_excluded": False, - "token_count": 2, - "page_start": 1, - "page_end": 1, - } - sample_records = _prepare_records( - document_id="debug-document-id", - filename="debug.pdf", - chunks=[fake_chunk], - workflow_id="doc-debugsample12", - ) - sample_record_keys = sorted(sample_records[0].keys()) if sample_records else [] - return { - "passage_schema_fields": passage_fields, - "has_text_for_embedding": has_text_for_embedding, - "sample_record_keys": sample_record_keys, - "sample_has_passage_prefix": ( - sample_records[0].get("text_for_embedding", "").startswith("passage:") - if sample_records else False - ), - } - - -@app.post("/documents/reconcile") -async def reconcile_document_states(user: RequirePipeline): - """ - Reconcile Postgres document states with Temporal workflow states. - - This endpoint checks all documents in processing/review stages and updates - Postgres if the Temporal workflow has terminated or failed. This fixes - inconsistencies caused by external workflow termination or worker crashes. - - Returns a summary of documents checked and updated. - """ - # Stages that indicate an active workflow (not terminal states) - active_stages = [ - 'ocr_processing', 'ocr_review', - 'translation_processing', 'translation_review', - 'chunking', 'chunk_review', - 'ready_for_ingestion', 'ingesting' - ] - - # Scope to caller's instances (None = unrestricted bypass / all tenants). - docs = db.list_documents( - limit=1000, - include_disabled=True, - instances=_instance_scope_for_user(user), - ) - active_docs = [d for d in docs if d.get('stage') in active_stages] - - results = { - "checked": len(active_docs), - "updated": 0, - "still_running": 0, - "details": [] - } - - for doc in active_docs: - detail = await _reconcile_single_document(doc) - results["details"].append(detail) - if detail.get("action") == "stage_synced" or detail.get("action") == "marked_failed": - results["updated"] += 1 - elif detail.get("action") == "no_change": - results["still_running"] += 1 - - return results - - -@app.get("/pipeline/stages") -async def get_pipeline_stages(user: RequireSearch): - """Get the pipeline stages for UI stepper display. - - The PROD stages are omitted when DISABLE_PROD_SETTING is on, so the stepper - shows the pipeline the documents will actually follow. - """ - prod_disabled = prod_stage_disabled() - return [ - {"id": stage[0], "label": stage[1], "description": stage[2]} - for stage in PIPELINE_STAGES - if not (prod_disabled and stage[0] in PROD_ONLY_STAGES) - ] - - -# ============================================================================= -# Settings Routes -# ============================================================================= - -@app.get("/settings/search", response_model=SearchSettings) -async def get_search_settings(user: RequireSearch): - """ - Get current search settings. - - Read-only search configuration. Gated with ``RequireSearch`` (not - ``RequireAdmin``) because any search-capable user needs these defaults - (index name, method, limits) to drive the search UI; mutating the - settings (PUT / reset) still requires ``RequireAdmin``. - - Returns the current search configuration including: - - searchMethod: TENSOR, LEXICAL, or HYBRID - - limit: Number of results to return - - alpha: Balance between lexical (0) and semantic (1) for hybrid search - - rankingMethod: rrf or normalize_linear for hybrid search - - showHighlights: Whether to show highlighted matches - - efSearch: HNSW search accuracy parameter - """ - return db.get_search_settings() - - -@app.put("/settings/search", response_model=SearchSettings) -async def update_search_settings_endpoint(settings: SearchSettingsUpdate, user: RequireAdmin): - """ - Update search settings. - - Only provided fields will be updated. Changes are logged to the audit trail. - """ - _ = user - # Convert to dict, excluding None values - updates = {k: v for k, v in settings.model_dump().items() if v is not None} - - if not updates: - raise HTTPException(400, "No settings provided to update") - - try: - return db.update_search_settings(updates) - except ValueError as exc: - raise HTTPException(400, str(exc)) from exc - - -@app.get("/settings/search/audit", response_model=SettingsAuditResponse) -async def get_search_settings_audit( - user: RequireAdmin, - limit: int = Query(50, le=200), - offset: int = 0 -): - """ - Get audit trail for search settings changes. - - Shows all historical changes to search settings with old/new values. - """ - logs = db.get_settings_audit_logs(limit=limit, offset=offset) - total = db.get_settings_audit_count() - - return SettingsAuditResponse( - logs=logs, - total=total, - limit=limit, - offset=offset - ) - - -@app.post("/settings/search/reset", response_model=SearchSettings) -async def reset_search_settings(user: RequireAdmin): - """ - Reset search settings to defaults. - - Resets all search settings to their default values: - - searchMethod: HYBRID - - limit: 10 - - alpha: 0.7 - - rankingMethod: rrf - - showHighlights: true - - efSearch: 256 - """ - _ = user - defaults = { - "searchMethod": "HYBRID", - "limit": 12, - "alpha": 0.6, - "rankingMethod": "rrf", - "showHighlights": True, - "efSearch": 256, - "indexName": "documents-index", - "candidateCap": 120, - "candidateMultiplier": 10, - "maxChunksPerDoc": 2, - "useE5Prefix": True, - "excludeReference": True, - "queryExpansionProfile": "gu-v1", - "rerankMode": "none", - "hybridRrfK": 60, - } - return db.update_search_settings(defaults) +app.include_router(webhook_sources_router) +app.include_router(knowledge_router) +app.include_router(registry_proposals_router) +app.include_router(auth_sessions_router) +app.include_router(user_admin_router) +app.include_router(catalog_router) +app.include_router(pages_router) +app.include_router(chunks_router) +app.include_router(search_settings_router) +app.include_router(operations_router) +app.include_router(document_detail_router) +app.include_router(document_lifecycle_router) +app.include_router(document_review_router) +app.include_router(audit_router) +app.include_router(document_content_router) +app.include_router(indexes_router) diff --git a/pipeline/api/application.py b/pipeline/api/application.py new file mode 100644 index 0000000..8355487 --- /dev/null +++ b/pipeline/api/application.py @@ -0,0 +1,120 @@ +"""FastAPI construction, middleware, and exception handling.""" + +import asyncio +import re +from typing import Optional +from urllib.parse import quote + +from fastapi import FastAPI, Request +from fastapi.middleware.cors import CORSMiddleware +from slowapi import _rate_limit_exceeded_handler +from slowapi.errors import RateLimitExceeded + +from . import api as _api, request_logging +from .rate_limiting import limiter + + +_DOCUMENT_PATH_RE = re.compile(r"^/documents/([^/]+)(/.*)?$") +# Only identifier shapes with an indexed lookup are resolved before auth runs: +# the document UUID, the 32-hex fingerprint, or a ``doc-`` workflow id. +_RESOLVABLE_IDENTIFIER_RE = re.compile( + r"^(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|[0-9a-f]{32}|doc-[0-9a-f]{32}.*)$" +) +_static_document_segments: Optional[set[str]] = None + + +def _static_document_path_segments() -> set[str]: + """Literal second segments of ``/documents/...`` routes (``upload``, ``batch``...).""" + global _static_document_segments + if _static_document_segments is None: + segments = set() + for route in _api.app.routes: + match = _DOCUMENT_PATH_RE.match(getattr(route, "path", "")) + if match and "{" not in match.group(1): + segments.add(match.group(1)) + _static_document_segments = segments + return _static_document_segments + + +async def canonicalize_document_identifier(request: Request, call_next): + """Let ``/documents/{id}/...`` take the document UUID or any legacy identifier. + + Handlers address a document by its Temporal workflow id (they signal and + query Temporal with it), so the path is rewritten to that id before routing. + An identifier of another shape, or one that matches no document, is left alone and + 404s in the handler; the lookup runs before auth, so it is limited to cheap shapes. + """ + match = _DOCUMENT_PATH_RE.match(request.scope["path"]) + if ( + match + and _RESOLVABLE_IDENTIFIER_RE.match(match.group(1)) + and match.group(1) not in _api._static_document_path_segments() + ): + document = await asyncio.to_thread(_api.db.find_document_by_doc_identifier, match.group(1)) + if document and document["workflow_id"] and document["workflow_id"] != match.group(1): + new_path = f"/documents/{document['workflow_id']}{match.group(2) or ''}" + request.scope["path"] = new_path + request.scope["raw_path"] = quote(new_path, safe="/:").encode("ascii") + return await call_next(request) + + +API_DESCRIPTION = """ +REST API for the Temporal-based document OCR pipeline with translation support. + +## Workflow Stages + +1. `registered` - Document registered +2. `ocr_processing` - OCR in progress +3. `ocr_review` - **Waiting for OCR review/approval** +4. `translation_processing` - Translating non-English content +5. `translation_review` - **Waiting for translation review/approval** +6. `chunking` - Chunking in progress +7. `chunk_review` - **Waiting for chunk review/approval** +8. `ready_for_ingestion` - **Waiting for final approval before DEV ingest** +9. `ingesting` - Ingesting to DEV Qdrant +10. `approval_for_prod` - **Waiting for superadmin prod approval** +11. `ingesting_prod` - Promoting vectors to PROD Qdrant +12. `completed` - Done +13. `failed` - Error occurred + +## Review Flow + +1. Start workflow with `POST /upload` or `POST /documents` +2. Wait for `ocr_review` stage +3. Review/edit pages with `GET/PATCH /documents/{id}/pages/{num}` +4. Approve with `POST /documents/{id}/approve-ocr` +5. Wait for `translation_review` stage +6. Review/edit translations with `PATCH /documents/{id}/pages/{num}` +7. Approve with `POST /documents/{id}/approve-translation` +8. Wait for `chunk_review` stage +9. Review/edit chunks with `GET/PATCH /documents/{id}/chunks/{num}` +10. Approve with `POST /documents/{id}/approve-chunks` +11. Wait for `ready_for_ingestion` stage +12. Approve DEV ingest with `POST /documents/{id}/approve-ingestion` +13. Wait for `approval_for_prod` stage +14. Superadmin promote with `POST /documents/{id}/approve-prod` → `ingesting_prod` → `completed` + """ + + +def create_app(lifespan): + app = FastAPI( + title="Document Ingestion Pipeline API", + description=API_DESCRIPTION, + version="2.0.0", + lifespan=lifespan + ) + + # CORS configuration - explicit origins for security + app.add_middleware( + CORSMiddleware, + allow_origins=_api.ALLOWED_ORIGINS, + allow_credentials=True, + allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"], + allow_headers=["*"], + ) + request_logging.install(app) + + app.middleware("http")(canonicalize_document_identifier) + app.state.limiter = limiter + app.add_exception_handler(RateLimitExceeded, request_logging.recording(_rate_limit_exceeded_handler)) + return app diff --git a/pipeline/api/audit_helpers.py b/pipeline/api/audit_helpers.py new file mode 100644 index 0000000..aecb755 --- /dev/null +++ b/pipeline/api/audit_helpers.py @@ -0,0 +1,106 @@ +"""Actor normalization and audit serialization for API mutations.""" + +import json + +from . import api as _api + + +def _parse_roles_field(value) -> list[str]: + if value is None: + return [] + if isinstance(value, list): + return [str(v).strip() for v in value if str(v).strip()] + text = str(value).strip() + if not text: + return [] + if text.startswith("["): + try: + parsed = json.loads(text) + if isinstance(parsed, list): + return [str(v).strip() for v in parsed if str(v).strip()] + except Exception: + pass + return [part.strip() for part in text.split(",") if part.strip()] + + +def _actor_from_user(user) -> dict: + """Normalize AuthUser into serializable actor identity fields.""" + if user is None: + return { + "user_id": "", + "username": "", + "email": "", + "roles": [], + "roles_csv": "", + } + roles = [str(r).strip() for r in (getattr(user, "roles", None) or []) if str(r).strip()] + return { + "user_id": (getattr(user, "user_id", None) or "").strip(), + "username": (getattr(user, "username", None) or "").strip(), + "email": (getattr(user, "email", None) or "").strip(), + "roles": roles, + "roles_csv": ",".join(roles), + } + + +def _log_audit( + workflow_id: str, + action_type: str, + entity_type: str = None, + entity_id: int = None, + field_name: str = None, + old_value=None, + new_value=None, + metadata: dict = None, + user=None, + *, + _conn=None, +): + """ + Helper to log audit entries with JSON serialization and optional actor identity. + + Args: + workflow_id: The Temporal workflow ID + action_type: Type of action (page_edit, chunk_edit, approval, etc.) + entity_type: Type of entity (page, chunk) + entity_id: Entity identifier (page_number, chunk_number) + field_name: Name of the field changed + old_value: Previous value (will be JSON serialized if not string) + new_value: New value (will be JSON serialized if not string) + metadata: Additional context as dict + user: Optional AuthUser whose email/roles are stored on the audit row + """ + # Get document_id from Postgres + doc = _api.db.get_document(workflow_id) if _conn is None else None + document_id = doc["document_id"] if doc else workflow_id + + # Serialize values to JSON if needed + old_str = json.dumps(old_value) if old_value is not None and not isinstance(old_value, str) else old_value + new_str = json.dumps(new_value) if new_value is not None and not isinstance(new_value, str) else new_value + + actor = _api._actor_from_user(user) + meta = dict(metadata or {}) + if actor.get("email") or actor.get("username") or actor.get("roles"): + meta.setdefault("actor", { + "user_id": actor.get("user_id"), + "username": actor.get("username"), + "email": actor.get("email"), + "roles": actor.get("roles"), + }) + + _api.db.log_audit( + workflow_id=workflow_id, + document_id=document_id, + action_type=action_type, + entity_type=entity_type, + entity_id=entity_id, + field_name=field_name, + old_value=old_str, + new_value=new_str, + metadata=meta or None, + actor_user_id=actor.get("user_id") or None, + actor_username=actor.get("username") or None, + actor_email=actor.get("email") or None, + actor_roles=actor.get("roles_csv") or None, + **({"_conn": _conn} if _conn is not None else {}), + ) diff --git a/pipeline/api/document_helpers.py b/pipeline/api/document_helpers.py new file mode 100644 index 0000000..0a9f900 --- /dev/null +++ b/pipeline/api/document_helpers.py @@ -0,0 +1,277 @@ +"""Shared document authorization, presentation, and review helpers.""" + +import os +import re +from pathlib import Path +from typing import Optional + +from fastapi import HTTPException, Request + +from pipeline.network.catalog import scheme_catalog +from ..auth.models import AuthUser +from ..auth.permissions import Permission +from ..auth.tenancy import allowed_instances, assert_document_instance_access, normalize_instance, resolve_create_instance +from pipeline.documents.document_kinds import validate_document_kind +from pipeline.network.release_policy import prod_stage_disabled +from pipeline.documents.models import DocumentStage, DocumentSummary +from . import api as _api + + +def _instance_scope_for_user(user: AuthUser, permission: Permission | None = None) -> Optional[list[str]]: + """None = unrestricted; otherwise only these instance codes.""" + allowed = allowed_instances(user) + if permission is None or user.is_superadmin or user.token_disabled_mode: + return sorted(allowed) if allowed is not None else None + # A global viewer can still have a write role in one provider. + candidates = allowed if allowed is not None else user.instances + return sorted(code for code in candidates if user.has_permission_for_instance(permission, code)) + + +def _document_cohorts_payload(summary: dict, by_instance: Optional[list[dict]] = None) -> dict: + """Shape Postgres aggregate counts for DocumentCohortsResponse (all ints).""" + def _i(key: str, default: int = 0) -> int: + val = summary.get(key, default) + return int(val or 0) + + return { + "total_documents": _i("total_documents"), + "authoritative_documents": _i("authoritative_documents"), + "legacy_documents": _i("legacy_documents"), + "completed_documents": _i("completed_documents"), + "review_queue": _i("review_queue"), + "failed_documents": _i("failed_documents"), + "needs_reindex": _i("needs_reindex"), + "running_jobs": _i("running_jobs"), + "by_stage": { + "ocr_review": _i("ocr_review_documents"), + "translation_review": _i("translation_review_documents"), + "chunk_review": _i("chunk_review_documents"), + "translation_processing": _i("translation_processing_documents"), + "chunking": _i("chunking_documents"), + "ready_for_ingestion": _i("ready_for_ingestion_documents"), + "failed": _i("failed_documents"), + }, + "by_instance": by_instance or [], + } + + +def _resolve_create_instance(user: AuthUser, requested: Optional[str] = None) -> str: + """Resolve the upload owner using the existing Keycloak tenancy rules.""" + return resolve_create_instance(user, requested) + + +def _document_title(filename: str) -> str: + """Human title for a new document: the file name without extension, separators as spaces.""" + stem = Path(filename or "").stem + return re.sub(r"[_\-\s]+", " ", stem).strip() or "Untitled document" + + +def _assert_can_replace(user: AuthUser, existing: dict) -> None: + """Replacing a document's file clears its pages and chunks, so guard it. + + Needs review rights in the document's instance; a disabled document or one with + a run in progress cannot be replaced; a document already promoted to PROD + can only be replaced by someone who may approve ingestion. + """ + doc_instance = existing["instance"] + if not user.has_permission_for_instance(Permission.REVIEW, doc_instance): + raise HTTPException(403, f"No permission to replace documents in instance: {doc_instance}") + if existing.get("is_disabled"): + raise HTTPException(409, "This document is disabled; restore it before replacing its file") + latest_job = _api.db.get_latest_document_job(existing["workflow_id"]) + if latest_job and latest_job.get("status") in {"running", "start_pending"}: + raise HTTPException(409, "A run is still in progress for this document") + if (existing.get("index_status") or {}).get("prod") and not user.has_permission_for_instance( + Permission.APPROVE_INGESTION, doc_instance + ): + raise HTTPException( + 403, "Replacing a document that is live in PROD needs approval rights in its instance" + ) + + +def _assert_upload_start_resolved(workflow_id: str) -> None: + job = _api.db.get_latest_document_job(workflow_id) + if job and job.get("status") == "start_pending": + raise HTTPException(409, "Resolve the pending upload start first") + + +def _resolve_document_identity( + user: AuthUser, + instance: str, + kind: str, + code: str, + filename: str, +) -> tuple[str, str, Optional[dict]]: + """Kind, code and (when ``code`` already exists) the document an upload replaces. + + An explicit ``code`` that is already held in the tenant's catalog of that + kind means "update this document": the caller reuses it instead of inserting. + Without a code one is derived from the file name. + + Permissions are checked against the target instance, not just globally: a user + who may upload in one state must not be able to add to, or replace a + document of, a state where they only hold a view role. Replacing clears the + document's pages and chunks, so it also needs review rights in that instance. + """ + try: + kind = validate_document_kind(kind or "") + except ValueError as exc: + raise HTTPException(400, str(exc)) from exc + if not user.has_permission_for_instance(Permission.UPLOAD, instance): + raise HTTPException(403, f"No upload permission in instance: {instance}") + code = (code or "").strip() + if code: + try: + code = scheme_catalog.validate_scheme_code(code) + except ValueError as exc: + raise HTTPException(400, str(exc)) from exc + existing = _api.db.find_document_by_code(instance, kind, code) + if existing: + existing = assert_document_instance_access(user, existing) + _api._assert_can_replace(user, existing) + return kind, code, existing + return kind, scheme_catalog.derive_scheme_code(_api._document_title(filename), kind=kind), None + + +def _require_document_for_user(workflow_id: str, user: AuthUser, permission: Permission | None = None) -> dict: + """Load a document or 404 if missing / outside the caller's instance scope.""" + doc = assert_document_instance_access(user, _api.db.get_document(workflow_id)) + if permission is not None and not user.has_permission_for_instance(permission, doc.get("instance")): + raise HTTPException(403, f"Missing provider permission: {permission.value}") + return doc + + +def _document_for_user_or_none( + workflow_id: str, user: AuthUser, permission: Permission | None = None, +) -> Optional[dict]: + """Like _require_document_for_user but returns None instead of raising (bulk paths).""" + try: + return _api._require_document_for_user(workflow_id, user, permission) + except HTTPException as exc: + if exc.status_code not in (403, 404): + raise + return None + + +def _document_summary_from_row(doc: dict, current_job: Optional[dict] = None) -> DocumentSummary: + return DocumentSummary( + crop_subjects=doc.get("crop_subjects") or [], + knowledge_metadata=doc.get("knowledge_metadata") or {}, + id=doc.get("id"), + document_id=doc["document_id"], + canonical_document_id=doc.get("canonical_document_id"), + workflow_id=doc["workflow_id"], + filename=doc["filename"], + name=doc.get("name"), + source_filename=doc.get("source_filename"), + source_manifest_name=doc.get("source_manifest_name"), + source_file_fingerprint=doc.get("source_file_fingerprint"), + authoritative=bool(doc.get("source_manifest_name")), + instance=normalize_instance(doc.get("instance")), + stage=DocumentStage(doc["stage"]), + page_count=doc.get("page_count") or 0, + chunk_count=doc.get("chunk_count") or 0, + error_message=doc.get("error_message"), + created_at=doc.get("created_at"), + updated_at=doc.get("updated_at"), + reindex_required=bool(doc.get("reindex_required")), + reindex_reason=doc.get("reindex_reason"), + available_actions=_api._list_available_actions( + doc, + current_job if current_job is not None else _api.db.get_latest_document_job(doc["workflow_id"]), + ), + uploaded_by_user_id=doc.get("uploaded_by_user_id"), + uploaded_by_username=doc.get("uploaded_by_username"), + uploaded_by_email=doc.get("uploaded_by_email"), + uploaded_by_roles=_api._parse_roles_field(doc.get("uploaded_by_roles")), + kind=doc.get("kind"), + code=doc.get("code"), + aliases=list(doc.get("aliases") or []), + tool_routing=doc.get("tool_routing"), + catalog_visible=bool(int(doc["catalog_visible"])) if doc.get("catalog_visible") is not None else True, + network_visible=bool(int(doc["network_visible"])) if doc.get("network_visible") is not None else True, + prod_ready_requested_at=doc.get("prod_ready_requested_at"), + prod_ready_requested_by_username=doc.get("prod_ready_requested_by_username"), + valid_from=doc.get("valid_from"), + valid_to=doc.get("valid_to"), + ) + + +def _provenance_base_urls(request: Request) -> tuple[str, str]: + api_base = (os.environ.get("DOCS_PIPELINE_API_URL") or str(request.base_url)).rstrip("/") + ui_base = (os.environ.get("DOCS_PIPELINE_UI_URL") or "http://localhost:3000").rstrip("/") + return api_base, ui_base + + +def _build_provenance_links(workflow_id: str, chunk_num: int, request: Request) -> dict[str, str]: + api_base, ui_base = _api._provenance_base_urls(request) + return { + "pdf_url": f"{api_base}/documents/{workflow_id}/pdf", + "document_url": f"{ui_base}/documents/{workflow_id}", + "chunk_url": f"{ui_base}/documents/{workflow_id}?tab=chunks&chunk={chunk_num}", + } + + +def _list_available_actions(doc: dict, current_job: Optional[dict] = None) -> list[str]: + if not doc: + return [] + if doc.get("is_disabled"): + return ["restore_document"] + + if current_job and current_job.get("status") == "start_pending": + return ["retry_upload_start"] + + stage = doc.get("stage") + actions = ["disable_document", "reconcile_document"] + automatic_release = doc.get("uses_network_releases", False) + if stage == "ocr_review" and not automatic_release: + actions.append("approve_ocr") + elif stage == "translation_review" and not automatic_release: + actions.append("approve_translation") + elif stage == "chunk_review" and not automatic_release: + actions.append("approve_chunks") + elif stage == "ready_for_ingestion" and not automatic_release: + actions.append("approve_ingestion") + elif stage == "approval_for_prod": + # A document already parked at this gate when DISABLE_PROD_SETTING was + # turned on keeps its approve action — otherwise it would be stranded + # with no way forward. + actions.append("approve_prod") + if not doc.get("prod_ready_requested_at"): + actions.append("request_prod_ready") + elif stage == "completed": + actions.append("reingest_document") + if not automatic_release and not prod_stage_disabled(): + actions.append("approve_prod") + elif stage == "failed": + if not doc.get("ocr_completed_at"): + actions.append("retry_ocr") + if doc.get("ocr_completed_at") and not doc.get("translation_completed_at"): + actions.append("retry_translation") + if doc.get("translation_completed_at"): + actions.append("retry_chunking") + + if doc.get("reindex_required"): + actions.extend(["reingest_document", "clear_reindex_required"]) + else: + actions.append("mark_reindex_required") + + if current_job and current_job.get("status") == "running": + actions.append("inspect_runtime") + + return sorted(set(actions)) + + +def _mark_reindex_required(workflow_id: str, reason: str, metadata: Optional[dict] = None) -> Optional[dict]: + doc = _api.db.mark_document_reindex_required(workflow_id, True, reason) + if doc: + _api.db.log_audit( + workflow_id=workflow_id, + document_id=doc.get("document_id", workflow_id), + action_type="mark_reindex_required", + field_name="reindex_required", + old_value="false", + new_value="true", + metadata={"reason": reason, **(metadata or {})}, + ) + return doc diff --git a/pipeline/api/file_helpers.py b/pipeline/api/file_helpers.py new file mode 100644 index 0000000..5dfd952 --- /dev/null +++ b/pipeline/api/file_helpers.py @@ -0,0 +1,40 @@ +"""File admission and source identity helpers for API intake.""" + +import hashlib +import time +from pathlib import Path + +from fastapi import HTTPException + +from pipeline.documents.document_formats import allowed_file_extensions +from ..object_storage import is_object_uri, parse_object_uri + +def validate_file_path(filepath: str) -> str: + """Accept a supported object storage reference, never a host path.""" + if not is_object_uri(filepath): + raise HTTPException(403, "Host file registration is no longer supported. Upload the file instead.") + try: + parse_object_uri(filepath) + except ValueError as error: + raise HTTPException(400, str(error)) from None + suffix = Path(filepath).suffix.lower() + if suffix not in allowed_file_extensions(): + raise HTTPException(400, f"Unsupported file type: {suffix}") + return filepath + + +def get_filename_from_path(filepath: str) -> str: + """Extract filename from a filepath string (works for local paths and object references).""" + if is_object_uri(filepath): + return filepath.split("/")[-1] + return Path(filepath).name + + +def get_workflow_id(filepath: str) -> str: + """Generate consistent workflow ID from filepath.""" + return f"doc-{hashlib.md5(filepath.encode()).hexdigest()[:12]}" + + +def _rerun_workflow_id(base_workflow_id: str) -> str: + """Generate a fresh workflow ID for explicit reruns of the same source.""" + return f"{base_workflow_id}-rerun-{int(time.time())}" diff --git a/pipeline/api/index_helpers.py b/pipeline/api/index_helpers.py new file mode 100644 index 0000000..8540925 --- /dev/null +++ b/pipeline/api/index_helpers.py @@ -0,0 +1,32 @@ +"""Vector cleanup helpers used by document lifecycle routes.""" + +import logging + + +def delete_single_chunk_from_index(document_id: str, chunk_num: int, index_name: str | None = None) -> dict: + """Delete a single chunk from the vector index by doc_id + chunk_num.""" + from ..vector_store import get_default_index_name, get_vector_store + + try: + store = get_vector_store() + resolved_index = index_name or get_default_index_name() + return store.delete_chunk(resolved_index, document_id, chunk_num) + except Exception as e: + logging.exception("vector chunk delete failed doc_id=%s chunk=%s", document_id, chunk_num) + return {"deleted": False, "error": str(e)} + + +def delete_chunks_from_index(document_id: str, index_name: str | None = None) -> dict: + """Delete all chunks for a document from the vector index.""" + from ..vector_store import get_default_index_name, get_vector_store + + try: + store = get_vector_store() + resolved_index = index_name or get_default_index_name() + result = store.delete_by_doc_id(resolved_index, document_id) + if "doc_id" not in result: + result["doc_id"] = document_id + return result + except Exception as e: + logging.exception("vector document delete failed doc_id=%s", document_id) + return {"deleted": 0, "doc_id": document_id, "error": str(e)} diff --git a/pipeline/api/rate_limiting.py b/pipeline/api/rate_limiting.py new file mode 100644 index 0000000..38404a8 --- /dev/null +++ b/pipeline/api/rate_limiting.py @@ -0,0 +1,4 @@ +from slowapi import Limiter +from slowapi.util import get_remote_address + +limiter = Limiter(key_func=get_remote_address) diff --git a/pipeline/api/request_logging.py b/pipeline/api/request_logging.py index d5bc3ba..3093f14 100644 --- a/pipeline/api/request_logging.py +++ b/pipeline/api/request_logging.py @@ -12,12 +12,25 @@ request_validation_exception_handler, ) from fastapi.exceptions import RequestValidationError +from fastapi.responses import JSONResponse from starlette.exceptions import HTTPException as StarletteHTTPException from starlette.types import ASGIApp, Message, Receive, Scope, Send logger = logging.getLogger("pipeline.api.requests") QUIET_PATHS = frozenset({"/health"}) +GOOGLE_CALLBACK_PATH = "/auth/google/callback" + + +def _redact_google_callback(record: logging.LogRecord) -> bool: + if isinstance(record.args, tuple) and len(record.args) == 5: + path = record.args[2] + if isinstance(path, str) and path.split("?", 1)[0] == GOOGLE_CALLBACK_PATH: + record.args = (*record.args[:2], GOOGLE_CALLBACK_PATH, *record.args[3:]) + return True + + +logging.getLogger("uvicorn.access").addFilter(_redact_google_callback) def install(app: FastAPI) -> None: @@ -30,6 +43,9 @@ def recording(handler: Callable) -> Callable: async def record_then_handle(request: Request, exc: Exception): request.state.error = _detail(exc) request.state.error_cause = exc.__cause__ + if isinstance(exc, RequestValidationError) and getattr(request.scope.get("route"), "name", None) == "create_source_token": + # The default validation response includes input values. + return JSONResponse(status_code=422, content={"detail": "A valid service client secret is required."}) result = handler(request, exc) return await result if inspect.isawaitable(result) else result @@ -99,6 +115,8 @@ def _log_response(scope: Scope, code: int, duration_ms: int) -> None: def _uri(scope: Scope) -> str: + if scope["path"] == GOOGLE_CALLBACK_PATH: + return GOOGLE_CALLBACK_PATH query = scope.get("query_string", b"").decode("latin-1") return f"{scope['path']}?{query}" if query else scope["path"] diff --git a/pipeline/api/routers/__init__.py b/pipeline/api/routers/__init__.py new file mode 100644 index 0000000..db10eff --- /dev/null +++ b/pipeline/api/routers/__init__.py @@ -0,0 +1 @@ +"""Feature routers registered by the API composition module.""" diff --git a/pipeline/api/routers/audit.py b/pipeline/api/routers/audit.py new file mode 100644 index 0000000..44b8808 --- /dev/null +++ b/pipeline/api/routers/audit.py @@ -0,0 +1,81 @@ +"""Provider-scoped and document-scoped audit routes.""" + +from fastapi import APIRouter, Query + +from pipeline import db +from pipeline.auth.deps import RequireSearch +from pipeline.documents.models import AuditLogResponse +from pipeline.api.api import _instance_scope_for_user, _require_document_for_user + +router = APIRouter() + + +@router.get("/audit", response_model=AuditLogResponse) +async def get_all_audit_logs( + user: RequireSearch, + action_type: str = None, + limit: int = Query(50, le=200), + offset: int = 0 +): + """ + Get global audit trail across all documents. + + Returns a list of all changes including: + - Stage transitions + - Page edits + - Chunk edits + - Approvals + - Resets + + Each entry includes the document filename for context. + """ + scope = _instance_scope_for_user(user) + logs = db.get_all_audit_logs( + action_type=action_type, + limit=limit, + offset=offset, + instances=scope, + ) + total = db.get_all_audit_log_count(action_type, instances=scope) + + return AuditLogResponse( + logs=logs, + total=total, + limit=limit, + offset=offset + ) + + +@router.get("/documents/{workflow_id}/audit", response_model=AuditLogResponse) +async def get_document_audit_log( + workflow_id: str, + user: RequireSearch, + action_type: str = None, + limit: int = Query(50, le=200), + offset: int = 0 +): + """ + Get audit trail for a document. + + Returns a list of all changes made to the document including: + - Stage transitions + - Page edits + - Chunk edits + - Approvals + - Resets + """ + _require_document_for_user(workflow_id, user) + logs = db.get_audit_logs( + workflow_id=workflow_id, + action_type=action_type, + limit=limit, + offset=offset + ) + total = db.get_audit_log_count(workflow_id, action_type) + + return AuditLogResponse( + logs=logs, + total=total, + limit=limit, + offset=offset + ) diff --git a/pipeline/api/routers/auth_sessions.py b/pipeline/api/routers/auth_sessions.py new file mode 100644 index 0000000..5ac58e1 --- /dev/null +++ b/pipeline/api/routers/auth_sessions.py @@ -0,0 +1,168 @@ +import asyncio +import os + +from fastapi import APIRouter, HTTPException, Request, Response +from fastapi.responses import JSONResponse, RedirectResponse +from pydantic import BaseModel, Field, SecretStr + +from pipeline.api.rate_limiting import limiter +from pipeline.auth.config import load_auth_config +from pipeline.auth.deps import CurrentUser +from pipeline.auth.email_sign_in import send_email_code, verify_email_code +from pipeline.auth.google_sign_in import ( + GOOGLE_RESULT_COOKIE, + GOOGLE_RESULT_TTL, + GOOGLE_STATE_COOKIE, + GOOGLE_STATE_TTL, + accept_google_callback, + complete_google_sign_in, + google_console, + require_google_console, + start_google_sign_in, +) +from pipeline.auth.oidc import end_session, refresh_tokens +from pipeline.auth.tenancy import portal_instance + +RATE_LIMIT_DEFAULT = os.environ.get("RATE_LIMIT_DEFAULT", "100/minute") +RATE_LIMIT_SIGN_IN = os.environ.get("RATE_LIMIT_SIGN_IN", os.environ.get("RATE_LIMIT_OTP", "10/minute")) + +router = APIRouter() + + +class SessionRefreshBody(BaseModel): + refresh_token: str = Field(..., description="Refresh token from a Keycloak session") + + +class EmailCodeRequest(BaseModel): + email: SecretStr + + +class EmailCodeVerification(BaseModel): + challenge: SecretStr + code: SecretStr + + +@router.post("/auth/email/send", status_code=202) +@limiter.limit(RATE_LIMIT_SIGN_IN) +async def auth_send_email(request: Request, response: Response, body: EmailCodeRequest): + challenge = await asyncio.to_thread(send_email_code, body.email.get_secret_value()) + response.headers["Cache-Control"] = "no-store" + return {"challenge": challenge} + + +@router.post("/auth/email/verify") +@limiter.limit(RATE_LIMIT_SIGN_IN) +async def auth_verify_email(request: Request, body: EmailCodeVerification): + tokens, challenge = await asyncio.to_thread( + verify_email_code, body.challenge.get_secret_value(), body.code.get_secret_value(), + ) + headers = {"Cache-Control": "no-store"} + if tokens is None: + return JSONResponse( + {"detail": "That code is invalid or expired.", "challenge": challenge}, + status_code=401, headers=headers, + ) + return JSONResponse(tokens, headers=headers) + + +@router.get("/auth/google") +async def google_status(): + return {"enabled": google_console() is not None} + + +@router.post("/auth/google/start") +@limiter.limit(RATE_LIMIT_SIGN_IN) +async def google_start(request: Request, response: Response): + console = require_google_console() + location, state = await asyncio.to_thread(start_google_sign_in, console) + response.headers["Cache-Control"] = "no-store" + response.delete_cookie(GOOGLE_RESULT_COOKIE) + response.set_cookie(GOOGLE_STATE_COOKIE, state, max_age=GOOGLE_STATE_TTL, secure=console.secure, httponly=True) + return {"location": location} + + +@router.get("/auth/google/callback", include_in_schema=False) +@limiter.limit(RATE_LIMIT_SIGN_IN) +async def google_callback(request: Request): + console = require_google_console() + response = RedirectResponse(f"{console.login_path}#google-error", status_code=303, headers={ + "Cache-Control": "no-store", "Referrer-Policy": "no-referrer", + }) + if request.query_params.get("error"): + return response + codes = request.query_params.getlist("code") + states = request.query_params.getlist("state") + if len(codes) != 1 or len(states) != 1: + return response + try: + result = await asyncio.to_thread( + accept_google_callback, request.cookies.get(GOOGLE_STATE_COOKIE), codes[0], states[0], + ) + except HTTPException: + return response + response.delete_cookie(GOOGLE_STATE_COOKIE) + response.headers["Location"] = f"{console.login_path}#google-complete" + response.set_cookie(GOOGLE_RESULT_COOKIE, result, max_age=GOOGLE_RESULT_TTL, secure=console.secure, httponly=True) + return response + + +@router.post("/auth/google/complete") +@limiter.limit(RATE_LIMIT_SIGN_IN) +async def google_complete(request: Request): + require_google_console() + try: + tokens = await asyncio.to_thread(complete_google_sign_in, request.cookies.get(GOOGLE_RESULT_COOKIE)) + except HTTPException: + response = JSONResponse({"detail": "Google sign-in could not be completed."}, status_code=401) + else: + response = JSONResponse(tokens) + response.headers["Cache-Control"] = "no-store" + response.delete_cookie(GOOGLE_RESULT_COOKIE) + return response + + +@router.post("/auth/session/logout", status_code=204) +@limiter.limit(RATE_LIMIT_DEFAULT) +async def auth_session_logout(request: Request, body: SessionRefreshBody): + await asyncio.to_thread(end_session, body.refresh_token) + return Response(status_code=204) + + +@router.post("/auth/session/refresh") +@router.post("/auth/otp/refresh") # pre-SSO-exchange name; kept so older UI builds keep working +@limiter.limit(RATE_LIMIT_DEFAULT) +async def auth_session_refresh(request: Request, response: Response, body: SessionRefreshBody): + """Renew a session. The refresh token authenticates the request.""" + response.headers["Cache-Control"] = "no-store" + return await asyncio.to_thread(refresh_tokens, body.refresh_token) + + +@router.get("/auth/me") +async def auth_me(user: CurrentUser): + """Return the authenticated caller (local bypass user when AUTH_DISABLED=true). + + Identity fields (username/email/roles) come from the JWT when a Bearer token + is present, including in AUTH_DISABLED mode for UI display. + """ + display_name = (user.username or "").strip() + return { + "user_id": user.user_id, + "username": user.username, + "name": display_name, + "email": user.email, + "roles": user.roles, + "permissions": sorted(p.value for p in user.permissions), + "instances": user.instances, + "envs": user.envs, + # Keycloak group paths (/states/MH/contributor, /global/super-admin) + # and derived per-state role map for tenant-aware UI. + "groups": list(user.groups or []), + "state_roles": dict(user.state_roles or {}), + "is_super_admin": bool(user.is_superadmin), + "is_global_viewer": user.is_global_viewer, + "is_bh_viewer": user.is_global_viewer, # compatibility for older consoles + "live_provider_access": not user.token_disabled_mode and load_auth_config().verify_live_groups, + # Portal instance code for BV / platform docs (not a state). + "portal_instance": portal_instance(), + "auth_disabled": user.token_disabled_mode, + } diff --git a/pipeline/api/routers/catalog.py b/pipeline/api/routers/catalog.py new file mode 100644 index 0000000..d68ad0f --- /dev/null +++ b/pipeline/api/routers/catalog.py @@ -0,0 +1,163 @@ +import os +from typing import Optional + +from fastapi import APIRouter, HTTPException, Query, Request + +from pipeline import db +from pipeline.network.catalog import scheme_catalog +from pipeline.auth.deps import CurrentUser, RequireAdmin +from pipeline.auth.models import AuthUser +from pipeline.auth.tenancy import allowed_instances, normalize_instance +from pipeline.api import api as _api + +router = APIRouter() + + +def _catalog_service_keys() -> list[str]: + raw = (os.environ.get("CATALOG_SERVICE_API_KEYS") or "").strip() + if not raw: + return [] + return [k.strip() for k in raw.split(",") if k.strip()] + + +def _service_key_ok(request: Request) -> bool: + import hmac + + keys = _catalog_service_keys() + if not keys: + return False + header = ( + request.headers.get("X-Catalog-Service-Key") + or request.headers.get("x-catalog-service-key") + or "" + ).strip() + if not header: + return False + return any(hmac.compare_digest(header, k) for k in keys) + + +def _catalog_read_scope(request: Request, user: AuthUser, instance: str | None = None) -> set[str] | None: + """Return the provider scope for a catalog read after checking access.""" + from pipeline.auth.permissions import Permission + + service_key = _service_key_ok(request) + if not service_key and not (user.has_permission(Permission.ADMIN) or user.has_permission(Permission.SEARCH)): + if _catalog_service_keys(): + raise HTTPException(401, "Missing or invalid X-Catalog-Service-Key") + raise HTTPException(403, "Catalog access requires search or admin permission") + scope = allowed_instances(user) + if service_key and not user.token_disabled_mode: + machine_scope = { + normalize_instance(code) for code in os.environ.get("CATALOG_SERVICE_PROVIDER_SCOPE", "").split(",") + if code.strip() + } + if not machine_scope: + raise HTTPException(403, "No provider scope is configured for the catalog service") + scope = machine_scope if scope is None else scope & machine_scope + if instance and scope is not None and normalize_instance(instance) not in scope: + raise HTTPException(403, "No access to the requested provider") + return scope + + +@router.get("/catalog/v1/version") +async def catalog_version(request: Request, user: CurrentUser): + """Cheap poll for cache warmers: { version, updated_at }.""" + _catalog_read_scope(request, user) + meta = db.get_catalog_meta() + return { + "version": int(meta.get("version") or 0), + "updated_at": meta.get("updated_at"), + "notes": meta.get("notes"), + } + + +@router.get("/catalog/v1/schemes") +async def catalog_schemes( + request: Request, + user: CurrentUser, + instance: Optional[str] = Query(None), + status: str = Query("live"), + include_pending: bool = Query(False), + network_visible: Optional[bool] = Query(None), +): + """List schemes from the master catalog (vector-indexed + filters).""" + scope = _catalog_read_scope(request, user, instance) + snap = scheme_catalog.build_snapshot( + instance=instance, + allowed_providers=scope, + status=status, + include_pending=include_pending, + network_visible=network_visible, + ) + return { + "catalog_version": snap["catalog_version"], + "updated_at": snap["updated_at"], + "vector_schemes": snap["vector_schemes"], + "legacy_schemes": snap["legacy_schemes"], + "routing_exceptions": snap["routing_exceptions"], + } + + +@router.get("/catalog/v1/snapshot") +async def catalog_snapshot( + request: Request, + user: CurrentUser, + instance: Optional[str] = Query(None), + status: str = Query("live"), + include_pending: bool = Query(False), + network_visible: Optional[bool] = Query(None), +): + """ + Full catalog snapshot for OAN / provider warmers. + + Includes vector_schemes, legacy_schemes, tool_prompts, routing_exceptions, + collections metadata, and monotonic catalog_version. + """ + scope = _catalog_read_scope(request, user, instance) + return scheme_catalog.build_snapshot( + instance=instance, + allowed_providers=scope, + status=status, + include_pending=include_pending, + network_visible=network_visible, + ) + + +@router.get("/catalog/v1/tool-prompt") +async def catalog_tool_prompt(request: Request, user: CurrentUser): + """Prompt fragments only (for AI system prompt assembly).""" + scope = _catalog_read_scope(request, user) + snap = scheme_catalog.build_snapshot(allowed_providers=scope) + return { + "catalog_version": snap["catalog_version"], + "updated_at": snap["updated_at"], + "tool_prompts": snap["tool_prompts"], + "routing_exceptions": snap["routing_exceptions"], + "vector_scheme_codes": [s["scheme_code"] for s in snap["vector_schemes"]], + "legacy_scheme_codes": [s["scheme_code"] for s in snap["legacy_schemes"]], + } + + +@router.post("/catalog/v1/rebuild") +async def catalog_rebuild(request: Request, user: RequireAdmin): + """Admin: bump the catalog version (entries are computed from documents).""" + from pipeline.auth.permissions import Permission + + # Admin JWT always OK; service key also accepted for automation + if not _service_key_ok(request) and not user.has_permission(Permission.ADMIN): + raise HTTPException(403, "Rebuild requires admin") + version = scheme_catalog.rebuild_catalog_from_documents(reason="admin_rebuild") + snap = scheme_catalog.build_snapshot() + _api._log_audit( + workflow_id="__catalog__", + action_type="catalog_rebuild", + entity_type="catalog", + metadata={"catalog_version": version, "vector_count": len(snap["vector_schemes"])}, + user=user, + ) + return { + "ok": True, + "catalog_version": version, + "vector_scheme_count": len(snap["vector_schemes"]), + "updated_at": snap["updated_at"], + } diff --git a/pipeline/api/routers/chunks.py b/pipeline/api/routers/chunks.py new file mode 100644 index 0000000..082ff70 --- /dev/null +++ b/pipeline/api/routers/chunks.py @@ -0,0 +1,204 @@ +from typing import Optional + +from fastapi import APIRouter, HTTPException, Query +from fastapi import Path as PathParam + +from pipeline import db +from pipeline.auth.deps import RequireReview, RequireSearch +from pipeline.auth.permissions import Permission +from pipeline.search.index_identity import projection_doc_id, recorded_doc_id +from pipeline.network.release_policy import index_only_mode +from pipeline.documents.models import ChunkUpdate, DocumentStage +from pipeline.api import api as _api + +router = APIRouter() + + +@router.get("/chunks/search") +async def search_chunks_across_documents( + user: RequireSearch, + q: str = Query("", description="Keyword search within chunk text"), + limit: int = Query(100, ge=1, le=500), + offset: int = Query(0, ge=0), + include_excluded: bool = Query(False, description="Include excluded chunks"), + stage: Optional[DocumentStage] = Query(None, description="Optional document stage filter"), +): + """Search chunks across all documents for KB maintainer workflows.""" + # Tenant-scope via the owning document's instance (None = unrestricted admin). + chunks, total = db.search_chunks( + query=q, + limit=limit, + offset=offset, + include_excluded=include_excluded, + stage=stage.value if stage else None, + instances=_api._instance_scope_for_user(user), + ) + return { + "items": chunks, + "total": total, + "limit": limit, + "offset": offset, + "query": q, + "include_excluded": include_excluded, + "stage": stage.value if stage else None, + } + + +@router.get("/documents/{workflow_id}/chunks") +async def list_chunks(workflow_id: str, user: RequireSearch, include_excluded: bool = False): + """Get all chunks for a document. Postgres-first for speed.""" + _api._require_document_for_user(workflow_id, user) + return db.get_chunks(workflow_id, include_excluded=include_excluded) + + +@router.get("/documents/{workflow_id}/chunks/{chunk_num}") +async def get_chunk(workflow_id: str, user: RequireSearch, chunk_num: int = PathParam(..., ge=1, le=10000, description="Chunk number (1-indexed)")): + """Get a specific chunk. Postgres-first for speed.""" + _api._require_document_for_user(workflow_id, user) + chunk = db.get_chunk(workflow_id, chunk_num) + if chunk: + return chunk + + raise HTTPException(404, f"Chunk {chunk_num} not found") + + +@router.patch("/documents/{workflow_id}/chunks/{chunk_num}") +async def update_chunk( + workflow_id: str, + data: ChunkUpdate, + user: RequireReview, + chunk_num: int = PathParam(..., ge=1, le=10000, description="Chunk number (1-indexed)"), +): + """Update a chunk (edit text, mark reviewed, exclude).""" + doc = _api._require_document_for_user(workflow_id, user, Permission.REVIEW) + old_chunk = db.get_chunk(workflow_id, chunk_num) + if not old_chunk: + raise HTTPException(404, f"Chunk {chunk_num} not found") + + updated = db.update_chunk( + workflow_id, + chunk_num, + edited_text=data.edited_text, + is_reviewed=data.is_reviewed, + is_excluded=data.is_excluded, + reviewer_notes=data.reviewer_notes, + tags=data.tags, + ) + if not updated: + raise HTTPException(404, f"Chunk {chunk_num} not found") + + # Log audits for changed fields + if data.edited_text is not None: + old_text = old_chunk.get("edited_text") or old_chunk.get("original_text", "") + _api._log_audit( + workflow_id=workflow_id, + action_type="chunk_edit", + entity_type="chunk", + entity_id=chunk_num, + field_name="edited_text", + old_value=old_text, + new_value=data.edited_text, + user=user, + ) + + if data.is_reviewed is not None: + _api._log_audit( + workflow_id=workflow_id, + action_type="chunk_edit", + entity_type="chunk", + entity_id=chunk_num, + field_name="is_reviewed", + old_value=old_chunk.get("is_reviewed", False), + new_value=data.is_reviewed, + user=user, + ) + + if data.is_excluded is not None: + _api._log_audit( + workflow_id=workflow_id, + action_type="chunk_edit", + entity_type="chunk", + entity_id=chunk_num, + field_name="is_excluded", + old_value=old_chunk.get("is_excluded", False), + new_value=data.is_excluded, + user=user, + ) + + # If excluding a chunk and document is completed (already ingested), remove from the vector index + if data.is_excluded and not old_chunk.get("is_excluded", False): + if index_only_mode() or doc.get("stage") == "completed": + doc_id = projection_doc_id(doc) if index_only_mode() else recorded_doc_id(doc, "dev") + if doc_id: + vector_result = _api.delete_single_chunk_from_index(doc_id, chunk_num) + if vector_result.get("deleted"): + _api._log_audit( + workflow_id=workflow_id, + action_type="chunk_removed_from_search", + entity_type="chunk", + entity_id=chunk_num, + metadata={"vector_id": vector_result.get("chunk_id")}, + user=user, + ) + + if data.reviewer_notes is not None: + _api._log_audit( + workflow_id=workflow_id, + action_type="chunk_edit", + entity_type="chunk", + entity_id=chunk_num, + field_name="reviewer_notes", + old_value=old_chunk.get("reviewer_notes"), + new_value=data.reviewer_notes, + user=user, + ) + + if data.tags is not None: + _api._log_audit( + workflow_id=workflow_id, action_type="chunk_edit", entity_type="chunk", + entity_id=chunk_num, field_name="tags", old_value=old_chunk.get("tags", []), + new_value=updated.get("tags", []), user=user, + ) + + if data.edited_text is not None or data.is_excluded is not None or data.tags is not None: + _api._mark_reindex_required( + workflow_id, + "Chunk content changed; search index is out of sync", + metadata={"chunk_number": chunk_num}, + ) + + return db.get_chunk(workflow_id, chunk_num) + + +@router.post("/documents/{workflow_id}/chunks/{chunk_num}/reset") +async def reset_chunk( + workflow_id: str, + user: RequireReview, + chunk_num: int = PathParam(..., ge=1, le=10000, description="Chunk number (1-indexed)"), +): + """Reset chunk to original text.""" + _api._require_document_for_user(workflow_id, user, Permission.REVIEW) + old_chunk = db.get_chunk(workflow_id, chunk_num) + if not old_chunk: + raise HTTPException(404, f"Chunk {chunk_num} not found") + db.reset_chunk(workflow_id, chunk_num) + + # Log reset action + _api._log_audit( + workflow_id=workflow_id, + action_type="chunk_reset", + entity_type="chunk", + entity_id=chunk_num, + field_name="edited_text", + old_value=old_chunk.get("edited_text") if old_chunk else None, + new_value=None, + metadata={"reset_to": "original_text"}, + user=user, + ) + _api._mark_reindex_required( + workflow_id, + "Chunk reset; search index is out of sync", + metadata={"chunk_number": chunk_num}, + ) + + return db.get_chunk(workflow_id, chunk_num) diff --git a/pipeline/api/routers/crawl.py b/pipeline/api/routers/crawl.py new file mode 100644 index 0000000..19a0bd4 --- /dev/null +++ b/pipeline/api/routers/crawl.py @@ -0,0 +1,211 @@ +from __future__ import annotations + +import asyncio +import hashlib +import json +import os +import re +from io import BytesIO + +from fastapi import APIRouter, HTTPException, Request, UploadFile +from pydantic import BaseModel, ConfigDict, Field, ValidationError + +from pipeline import db +from pipeline.api import api as _api +from pipeline.api.routers.upload import _process_upload +from pipeline.auth.deps import RequireUpload +from pipeline.auth.permissions import Permission +from pipeline.metadata.crop_metadata import CropSubjects +from pipeline.intake.source_fetch import SourceFile, SourcePageError, fetch_source +from pipeline.network.release_policy import upload_release_target +from pipeline.metadata.knowledge_metadata import KnowledgeMetadata +from pipeline.intake.provider_sources import CrawlProfile +from pipeline.intake.source_intake import ( + IntakeConflict, + NormalizedSourceDocument, + attach_source_response, + canonical_json_sha256, + crawl_record_key, + register_source_document, + start_source_document, + store_source_response, +) +from pipeline.intake.webhook_mapping import ( + CanonicalMapping, + decode_source_json, + load_intake_template, + map_source_submission, +) + +router = APIRouter() + + +class CrawlRequest(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + url: str = Field(min_length=1, max_length=4096) + kind: str = "advisory" + external_id: str | None = Field(default=None, max_length=512) + crop_subjects: CropSubjects = Field(default_factory=list) + knowledge_metadata: KnowledgeMetadata | None = None + valid_from: str | None = None + valid_to: str | None = None + + + +def _crawl_provider(user, provider: str) -> str: + instance = _api._resolve_create_instance(user, provider) + if not user.has_permission_for_instance(Permission.UPLOAD, instance): + raise HTTPException(403, "Upload permission is required for this provider") + return instance + + +def _crawl_profiles(instance: str) -> dict[str, CrawlProfile]: + from pipeline.intake.provider_sources import stored_settings + + try: + stored = stored_settings(instance) + if stored: + return {key: CrawlProfile.model_validate(profile) for key, profile in stored["settings"]["crawl"].items()} + providers = json.loads(os.environ.get("CRAWL_SOURCE_PROFILES") or "{}") + return { + key: CrawlProfile.model_validate(profile) + for key, profile in providers.get(instance, {}).items() + if re.fullmatch(r"[a-z0-9][a-z0-9-]{0,63}", key) + } + except (AttributeError, ValueError) as exc: + raise HTTPException(503, "The crawl source configuration is invalid") from exc + + +@router.get("/providers/{provider}/crawl") +def crawl_sources(provider: str, user: RequireUpload): + profiles = _crawl_profiles(_crawl_provider(user, provider)) + return {"sources": [{"id": key, "host": profile.host} for key, profile in sorted(profiles.items())]} + + +@router.post("/providers/{provider}/crawl", status_code=202) +@_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) +async def crawl_public_source(request: Request, provider: str, data: CrawlRequest, user: RequireUpload): + instance = _crawl_provider(user, provider) + # No saved profile means no outbound credentials. A URL-specific identity + # keeps unrelated public feeds from sharing external IDs. + source_id = "public-" + canonical_json_sha256(data.url)[:32] + return await _ingest_url(instance, source_id, data, user) + + +@router.post("/providers/{provider}/crawl/{source_id}", status_code=202) +@_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) +async def crawl_source(request: Request, provider: str, source_id: str, data: CrawlRequest, user: RequireUpload): + instance = _crawl_provider(user, provider) + profile = _crawl_profiles(instance).get(source_id) + if profile is None: + raise HTTPException(404, "Crawl source not found") + return await _ingest_url(instance, source_id, data, user, profile) + + +async def _ingest_url(instance: str, source_id: str, data: CrawlRequest, user, profile: CrawlProfile | None = None): + authorization = os.environ.get(profile.authorization_env) if profile and profile.authorization_env else None + if profile and profile.authorization_env and not authorization: + raise HTTPException(503, "The crawl source credential is unavailable") + try: + source = await fetch_source(data.url, {profile.host.lower()} if profile else None, authorization) + except SourcePageError as exc: + raise HTTPException(422, str(exc)) from exc + capture = await asyncio.to_thread( + store_source_response, _api.object_store, _api.OBJECT_STORE_BUCKET, instance, source, + ) + if isinstance(source, SourceFile) and source.media_type != "application/json": + source_key = canonical_json_sha256([source_id, data.external_id or data.url])[:32] + code = f"crawl-{source_key}" + existing = db.find_document_by_code(instance, data.kind.strip().lower(), code) + crops = [crop.model_dump(exclude_none=True) for crop in data.crop_subjects] + metadata = data.knowledge_metadata.storage_value() if data.knowledge_metadata else None + if ( + existing and not existing["is_disabled"] + and existing.get("source_file_fingerprint") == hashlib.md5(source.content).hexdigest() + and ("crop_subjects" not in data.model_fields_set or crops == (existing.get("crop_subjects") or [])) + and (metadata is None or metadata == (existing.get("knowledge_metadata") or {})) + and (data.valid_from is None or data.valid_from == existing.get("valid_from")) + and (data.valid_to is None or data.valid_to == existing.get("valid_to")) + ): + job = db.get_document_job(existing["latest_job_id"]) + if job and (job.get("config") or {}).get("source_url") == source.attribution_url: + with db.get_connection() as conn: + attach_source_response(conn, job["id"], capture) + return { + "document_id": existing["id"], "workflow_id": job["temporal_workflow_id"], + "status": job["status"], "source_url": source.attribution_url, + } + file = UploadFile(BytesIO(source.content), filename=source.filename) + try: + document = await _process_upload( + user, file, instance, data.kind, code, None, + json.dumps(crops) if "crop_subjects" in data.model_fields_set else None, + knowledge_metadata=data.knowledge_metadata.model_dump_json() if data.knowledge_metadata else None, + valid_from=data.valid_from, valid_to=data.valid_to, + parser_provider=profile.parser_profile if profile else None, source_url=source.attribution_url, + source_capture=capture, + ) + finally: + await file.close() + return { + "document_id": document.id, "workflow_id": document.workflow_id, + "status": "start_pending" if "retry_upload_start" in document.available_actions else document.stage.value, "source_url": source.attribution_url, + } + + mapping = CanonicalMapping() + if isinstance(source, SourceFile) and profile and profile.intake_template: + try: + mapping = load_intake_template(profile.intake_template, provider=instance) + except (OSError, ValueError) as exc: + raise HTTPException(503, "The configured JSON intake template is unavailable") from exc + try: + if isinstance(source, SourceFile): + payload = decode_source_json(source.content) + rows = [ + (record, document.model_copy(update={"source_url": source.attribution_url})) + for record, document in await asyncio.to_thread(map_source_submission, payload, mapping) + ] + else: + document = NormalizedSourceDocument( + external_id=data.external_id or canonical_json_sha256(data.url), title=source.title, + text=source.text, language=source.language, kind=data.kind, source_url=source.attribution_url, + crop_subjects=data.crop_subjects, knowledge_metadata=data.knowledge_metadata, + valid_from=data.valid_from, valid_to=data.valid_to, + ) + rows = [(None, document)] + except (ValidationError, ValueError, RecursionError) as exc: + raise HTTPException(422, "The downloaded content does not match the knowledge intake schema") from exc + + try: + with db.get_connection() as conn: + receipts = [] + for record, document in rows: + normalized = document.model_dump(mode="json", exclude_none=True) + normalized_key = canonical_json_sha256(normalized) + target = upload_release_target(document.kind, provider=instance) + receipts.append(register_source_document( + conn, instance=instance, source_kind="crawl", source_id=source_id, + idempotency_key=normalized_key if record is None else crawl_record_key(record, document), + raw_payload=normalized if record is None else record, + legacy_crawl_key=normalized_key if record is not None else None, + document=document, release_target=target, + )) + attach_source_response(conn, receipts[-1]["job_id"], capture) + except IntakeConflict as exc: + raise HTTPException(409, str(exc)) from exc + except LookupError as exc: + raise HTTPException(404, "Provider not found") from exc + except (ValueError, RecursionError) as exc: + raise HTTPException(422, "Invalid knowledge metadata or schema attributes") from exc + if not receipts: + return {"items": []} + items = [] + for receipt in receipts: + started = await start_source_document(receipt) + items.append({ + "receipt_id": receipt["id"], "document_id": receipt["document_id"], "workflow_id": receipt["workflow_id"], + "status": "start_pending" if not started else "running" if receipt["status"] == "start_pending" else receipt["status"], + "source_url": source.attribution_url, + }) + return items[0] if len(items) == 1 else {"items": items} diff --git a/pipeline/api/routers/document_content.py b/pipeline/api/routers/document_content.py new file mode 100644 index 0000000..0d26d63 --- /dev/null +++ b/pipeline/api/routers/document_content.py @@ -0,0 +1,178 @@ +"""Document downloads, exports, and chunk provenance routes.""" + +import hashlib +import logging +from pathlib import Path +from typing import Optional +from urllib.parse import quote + +from fastapi import APIRouter, HTTPException, Query, Request +from fastapi.responses import StreamingResponse + +from pipeline import db +from pipeline.auth.deps import RequireSearch +from pipeline.search.index_identity import projection_doc_id +from pipeline.object_storage import is_object_uri, parse_object_uri +from pipeline.api import api as _api +from pipeline.api.api import _build_provenance_links, _require_document_for_user + +router = APIRouter() + + +@router.get("/documents/{workflow_id}/export/markdown") +async def export_markdown(workflow_id: str, user: RequireSearch): + """Export document as combined markdown.""" + doc = _require_document_for_user(workflow_id, user) + + pages = db.get_pages(workflow_id) + content = [] + for page in pages: + md = ( + page.get("edited_translation") + or page.get("translated_markdown") + or page.get("edited_markdown") + or page.get("original_markdown", "") + ) + content.append(f"\n\n{md}") + + return { + "filename": doc.get("filename", "").replace(".pdf", ".md"), + "content": "\n\n---\n\n".join(content) + } + + +@router.get("/documents/{workflow_id}/export/chunks") +async def export_chunks(workflow_id: str, user: RequireSearch, include_excluded: bool = False): + """Export chunks as JSON for vector-index ingestion.""" + doc = _require_document_for_user(workflow_id, user) + + chunks = db.get_chunks(workflow_id, include_excluded=include_excluded) + doc_id = projection_doc_id(doc) + filename = doc.get("filename", "") + name = filename.replace(".pdf", "") + + records = [] + for chunk in chunks: + text = chunk.get("edited_text") or chunk.get("original_text", "") + chunk_num = chunk.get("chunk_number", 0) + + records.append({ + "_id": hashlib.md5(f"{doc_id}_{chunk_num}_{text[:50]}".encode()).hexdigest(), + "doc_id": doc_id, + "name": name, + "text": text, + "chunk_num": chunk_num, + "token_count": chunk.get("token_count", 0), + "source": "docs-pipeline" + }) + + return records + + +# ============================================================================= +# PDF Serving +# ============================================================================= + +def _inline_content_disposition(filename: str) -> str: + """Build a latin-1-safe Content-Disposition header for inline file display.""" + # Strip CR/LF/NUL so a crafted filename cannot inject response headers. + safe_name = (filename or "document.pdf").replace('"', "'") + safe_name = "".join(ch for ch in safe_name if ch not in "\r\n\0").strip() or "document.pdf" + try: + safe_name.encode("latin-1") + return f'inline; filename="{safe_name}"' + except UnicodeEncodeError: + ascii_name = safe_name.encode("ascii", "ignore").decode("ascii").strip() or "document.pdf" + encoded_name = quote(safe_name) + return f"inline; filename=\"{ascii_name}\"; filename*=UTF-8''{encoded_name}" + + +@router.get("/documents/{workflow_id}/pdf") +async def get_document_pdf(workflow_id: str, user: RequireSearch): + """ + Get the original PDF file for a document. + Returns the PDF as a streaming response. Postgres-first for speed. + """ + # Postgres-first - instant lookup, tenant-scoped (404 hides other tenants). + doc = _require_document_for_user(workflow_id, user) + + if doc.get("canonical_input_type") == "structured_text": + raise HTTPException(404, "This document has no PDF preview") + + filepath = doc.get("filepath", "") + filename = doc.get("filename", "document.pdf") + + if not filepath: + raise HTTPException(404, f"Document has no PDF path: {workflow_id}") + + try: + if is_object_uri(filepath): + response = _api.object_store.open(parse_object_uri(filepath)) + + return StreamingResponse( + response, + media_type="application/pdf", + headers={ + "Content-Disposition": _inline_content_disposition(filename) + } + ) + else: + # Local file + file_path = Path(filepath) + if not file_path.exists(): + raise HTTPException(404, f"PDF file not found: {filepath}") + + def file_iterator(): + with open(file_path, "rb") as f: + yield from f + + return StreamingResponse( + file_iterator(), + media_type="application/pdf", + headers={ + "Content-Disposition": _inline_content_disposition(filename) + } + ) + except HTTPException: + raise + except Exception as e: + # Log the actual error server-side but don't expose details to client + logging.error(f"PDF serving error for {workflow_id}: {str(e)}") + raise HTTPException(500, "Error serving PDF file") + + +@router.get("/provenance/chunk") +async def resolve_provenance_chunk( + request: Request, + user: RequireSearch, + doc_id: Optional[str] = Query(None, description="workflow slug, Postgres document_id, or legacy vector-index doc_id"), + chunk_num: Optional[int] = Query(None, alias="chunk_num"), +): + """ + Resolve a retrieved chunk to workflow metadata and maintainer URLs. + + Used by chat/retrieval clients when search hits lack workflow_id (legacy rows) or for enrichment. + """ + from pipeline.documents.document_text import _infer_section + + resolved_doc_id = doc_id + resolved_chunk_num = chunk_num + + if not resolved_doc_id or resolved_chunk_num is None: + raise HTTPException(400, "Provide doc_id and chunk_num") + + provenance = db.resolve_chunk_provenance(doc_id=resolved_doc_id, chunk_num=int(resolved_chunk_num)) + if not provenance: + raise HTTPException(404, "Chunk provenance not found") + + workflow_id = provenance["workflow_id"] + # Tenant scope: a restricted caller may not resolve another tenant's chunk. + _require_document_for_user(workflow_id, user) + chunk = db.get_chunk(workflow_id, int(resolved_chunk_num)) + if chunk: + text = chunk.get("edited_text") or chunk.get("original_text") or "" + provenance["section"] = _infer_section(text, chunk.get("section_title")) + provenance["excerpt"] = text[:320] + ("..." if len(text) > 320 else "") + + links = _build_provenance_links(workflow_id, int(resolved_chunk_num), request) + return {**provenance, **links} diff --git a/pipeline/api/routers/document_detail.py b/pipeline/api/routers/document_detail.py new file mode 100644 index 0000000..e2ce752 --- /dev/null +++ b/pipeline/api/routers/document_detail.py @@ -0,0 +1,395 @@ +"""Document detail, processing artifacts, and runtime routes.""" + +import json +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, HTTPException, Query +from fastapi.responses import StreamingResponse +from temporalio.client import WorkflowFailureError + +from pipeline import db +from pipeline.api import api as _api +from pipeline.api.api import _list_available_actions, _parse_roles_field, _require_document_for_user +from pipeline.api.routers.document_content import _inline_content_disposition +from pipeline.auth.deps import CurrentUser, RequireSearch +from pipeline.auth.tenancy import normalize_instance +from pipeline.documents.models import ( + PIPELINE_STAGES, + PUBLISH_SKIPPED_UNMAPPED_KIND, + DocumentDetail, + DocumentGraph, + DocumentStage, + NetworkPublishOutcome, +) +from pipeline.network.network_constants import RESULT_REJECTED +from pipeline.object_storage import is_object_uri, parse_object_uri +from pipeline.intake.source_intake import public_source_job + +router = APIRouter() + + +NETWORK_PUBLISH_ARTIFACT_TYPES = {"network_publish_payload", "network_publish_skipped"} + + +def _network_publish_outcome(artifacts: list[dict]) -> NetworkPublishOutcome | None: + latest = next((a for a in artifacts if a.get("artifact_type") in NETWORK_PUBLISH_ARTIFACT_TYPES), None) + if latest is None: + return None + try: + exchange = json.loads(latest.get("metadata_json") or "{}") + except ValueError: + exchange = {} + if latest["artifact_type"] == "network_publish_skipped": + return NetworkPublishOutcome( + status="skipped", + reason_code=exchange.get("reason_code") or PUBLISH_SKIPPED_UNMAPPED_KIND, + reason=exchange.get("reason") or "No network catalog is mapped to this document kind.", + created_at=latest.get("created_at"), + ) + if exchange.get("result_status") == RESULT_REJECTED or exchange.get("failure_reason"): + return NetworkPublishOutcome( + status="rejected", + reason=exchange.get("failure_reason") or f"The network rejected the catalog: {exchange.get('errors') or []}", + created_at=latest.get("created_at"), + ) + return NetworkPublishOutcome(status="published", created_at=latest.get("created_at")) + + +def _build_document_detail(doc: dict) -> DocumentDetail: + workflow_id = doc["workflow_id"] + current_job = db.get_latest_document_job(workflow_id) + artifacts = db.list_document_artifacts(workflow_id) + return DocumentDetail( + id=doc.get("id"), + document_id=doc["document_id"], + canonical_document_id=doc.get("canonical_document_id"), + workflow_id=workflow_id, + filename=doc["filename"], + name=doc.get("name"), + source_filename=doc.get("source_filename"), + source_manifest_name=doc.get("source_manifest_name"), + source_file_fingerprint=doc.get("source_file_fingerprint"), + authoritative=bool(doc.get("source_manifest_name")), + instance=normalize_instance(doc.get("instance")), + filepath=doc["filepath"], + stage=DocumentStage(doc["stage"]), + page_count=doc.get("page_count", 0), + chunk_count=doc.get("chunk_count", 0), + error_message=doc.get("error_message"), + reindex_required=bool(doc.get("reindex_required")), + reindex_reason=doc.get("reindex_reason"), + available_actions=_list_available_actions(doc, current_job), + translated_count=db.count_translated_pages(workflow_id), + created_at=doc.get("created_at"), + updated_at=doc.get("updated_at"), + ocr_completed_at=doc.get("ocr_completed_at"), + translation_completed_at=doc.get("translation_completed_at"), + chunks_completed_at=doc.get("chunks_completed_at"), + ingested_at=doc.get("ingested_at"), + source_type=doc.get("source_type"), + canonical_input_type=doc.get("canonical_input_type"), + stop_after_ocr=bool(doc.get("stop_after_ocr")), + original_artifact_id=doc.get("original_artifact_id"), + normalized_artifact_id=doc.get("normalized_artifact_id"), + latest_job_id=doc.get("latest_job_id"), + current_job=public_source_job(current_job) if current_job else None, + artifacts=artifacts, + index_status=db.list_document_index_status(workflow_id), + network_publish=_network_publish_outcome(artifacts), + crop_subjects=doc.get("crop_subjects") or [], + knowledge_metadata=doc.get("knowledge_metadata") or {}, + uploaded_by_user_id=doc.get("uploaded_by_user_id"), + uploaded_by_username=doc.get("uploaded_by_username"), + uploaded_by_email=doc.get("uploaded_by_email"), + uploaded_by_roles=_parse_roles_field(doc.get("uploaded_by_roles")), + kind=doc.get("kind"), + code=doc.get("code"), + aliases=list(doc.get("aliases") or []), + tool_routing=doc.get("tool_routing"), + catalog_visible=bool(int(doc["catalog_visible"])) if doc.get("catalog_visible") is not None else True, + network_visible=bool(int(doc["network_visible"])) if doc.get("network_visible") is not None else True, + prod_ready_requested_at=doc.get("prod_ready_requested_at"), + prod_ready_requested_by_username=doc.get("prod_ready_requested_by_username"), + valid_from=doc.get("valid_from"), + valid_to=doc.get("valid_to"), + ) + +def _build_stage_io_payload(workflow_id: str, current_stage: Optional[str] = None) -> dict: + artifacts = db.list_document_artifacts(workflow_id) + grouped: dict[str, dict] = {} + for stage_id, label, description in PIPELINE_STAGES: + grouped[stage_id] = { + "stage": stage_id, + "label": label, + "description": description, + "input_artifacts": [], + "output_artifacts": [], + } + + input_types = {"original_upload", "normalized_pdf", "normalized_spreadsheet", "normalized_text"} + for artifact in artifacts: + stage = artifact.get("stage") or "registered" + if stage not in grouped: + grouped[stage] = { + "stage": stage, + "label": stage.replace("_", " ").title(), + "description": "", + "input_artifacts": [], + "output_artifacts": [], + } + bucket = "input_artifacts" if artifact["artifact_type"] in input_types else "output_artifacts" + grouped[stage][bucket].append(artifact) + + return { + "workflow_id": workflow_id, + "current_stage": current_stage, + "stages": list(grouped.values()), + } + + +async def _get_runtime_payload(workflow_id: str, doc: Optional[dict] = None) -> dict: + doc = doc or db.get_document(workflow_id) + if not doc: + raise HTTPException(404, f"Document not found: {workflow_id}") + current_job = db.get_latest_document_job(workflow_id) + runtime_workflow_id = ( + current_job.get("temporal_workflow_id") + if current_job and current_job.get("status") == "running" and current_job.get("temporal_workflow_id") + else workflow_id + ) + + chunking_progress = None + if current_job and current_job.get("config_json"): + try: + parsed_config = json.loads(current_job["config_json"]) if isinstance(current_job["config_json"], str) else current_job["config_json"] + if isinstance(parsed_config, dict): + chunking_progress = parsed_config.get("chunking_progress") + except Exception: + chunking_progress = None + + runtime = { + "workflow_id": workflow_id, + "sqlite_stage": doc.get("stage"), + "sqlite_error_message": doc.get("error_message"), + "temporal_connected": _api.temporal_client is not None, + "job": public_source_job(current_job) if current_job else None, + "chunking_progress": chunking_progress, + "temporal": None, + } + + if _api.temporal_client is None: + return runtime + + try: + handle = _api.temporal_client.get_workflow_handle(runtime_workflow_id) + description = await handle.describe() + temporal_state = None + query_error = None + try: + temporal_state = await handle.query("get_state") + except Exception as exc: + query_error = str(exc) + + runtime["temporal"] = { + "workflow_id": runtime_workflow_id, + "run_id": description.run_id, + "status": description.status.name, + "close_time": description.close_time.isoformat() if description.close_time else None, + "execution_time": description.execution_time.isoformat() if description.execution_time else None, + "state": temporal_state, + "query_error": query_error, + } + except Exception as exc: + runtime["temporal"] = { + "workflow_id": workflow_id, + "status": "UNAVAILABLE", + "error": str(exc), + } + + return runtime + + +@router.get("/documents/{workflow_id}", response_model=DocumentDetail) +async def get_document(workflow_id: str, user: CurrentUser): + """Get document workflow state with artifacts and indexing metadata.""" + doc = _require_document_for_user(workflow_id, user) + return _build_document_detail(doc) + + +@router.get("/documents/{workflow_id}/error-details") +async def get_workflow_error_details(workflow_id: str, user: RequireSearch): + """ + Get detailed error information from Temporal for a failed workflow. + + Returns comprehensive error details including: + - Error message + - Stack trace (if available) + - Failure type + - Workflow execution status + + This endpoint queries Temporal directly to get the most detailed + error information available, which may be more complete than what's + stored in Postgres. + """ + import traceback + + # Enforce tenant scope before touching Temporal (404 hides other tenants). + _require_document_for_user(workflow_id, user) + + try: + handle = _api.temporal_client.get_workflow_handle(workflow_id) + description = await handle.describe() + + result = { + "workflow_id": workflow_id, + "run_id": description.run_id, + "status": description.status.name, + "error_message": None, + "error_type": None, + "stack_trace": None, + "has_error": False + } + + # If workflow is failed, try to get detailed error information + if description.status.name == "FAILED": + result["has_error"] = True + + # Try to get error from workflow result (this raises WorkflowFailureError for failed workflows) + try: + await handle.result() + except WorkflowFailureError as wf_err: + # Extract error details from the failure + result["error_message"] = str(wf_err) + result["error_type"] = type(wf_err).__name__ + + # Try to get the underlying cause + if hasattr(wf_err, 'cause') and wf_err.cause: + cause = wf_err.cause + result["error_message"] = str(cause) + result["error_type"] = type(cause).__name__ + + # Get stack trace if available + if hasattr(cause, '__traceback__') and cause.__traceback__: + result["stack_trace"] = ''.join(traceback.format_tb(cause.__traceback__)) + elif hasattr(wf_err, '__traceback__') and wf_err.__traceback__: + result["stack_trace"] = ''.join(traceback.format_tb(wf_err.__traceback__)) + + # Also try to get failure details from the exception itself + if hasattr(wf_err, 'failure') and wf_err.failure: + failure = wf_err.failure + if hasattr(failure, 'message') and failure.message: + result["error_message"] = failure.message + if hasattr(failure, 'stack_trace') and failure.stack_trace: + result["stack_trace"] = failure.stack_trace + except Exception as e: + # If result() doesn't work, try other methods + result["error_message"] = f"Could not retrieve error details: {str(e)}" + + # Also try to get error from workflow state query (fallback) + if not result["error_message"]: + try: + state = await handle.query("get_state") + if state and state.get("error_message"): + result["error_message"] = state.get("error_message") + result["has_error"] = True + except Exception: + pass # Workflow might not support queries or be in wrong state + + return result + + except Exception as e: + # If workflow doesn't exist or can't be accessed + error_msg = str(e) + if "not found" in error_msg.lower() or "workflow" in error_msg.lower(): + raise HTTPException(404, f"Workflow not found: {workflow_id}") + raise HTTPException(500, f"Error fetching workflow details: {error_msg}") + + +@router.get("/documents/{workflow_id}/runtime") +async def get_document_runtime(workflow_id: str, user: RequireSearch): + """Return live runtime status by combining Postgres state and Temporal workflow state.""" + doc = _require_document_for_user(workflow_id, user) + return await _get_runtime_payload(workflow_id, doc=doc) + + +@router.get("/documents/{workflow_id}/artifacts") +async def list_document_artifacts(workflow_id: str, user: RequireSearch): + _require_document_for_user(workflow_id, user) + return db.list_document_artifacts(workflow_id) + + +@router.get("/documents/{workflow_id}/artifacts/{artifact_id}") +async def get_document_artifact(workflow_id: str, user: RequireSearch, artifact_id: str): + _require_document_for_user(workflow_id, user) + artifact = db.get_document_artifact(workflow_id, artifact_id) + if not artifact: + raise HTTPException(404, f"Artifact not found: {artifact_id}") + return artifact + + +@router.get("/documents/{workflow_id}/artifacts/{artifact_id}/content") +async def get_document_artifact_content(workflow_id: str, user: RequireSearch, artifact_id: str): + _require_document_for_user(workflow_id, user) + artifact = db.get_document_artifact(workflow_id, artifact_id) + if not artifact: + raise HTTPException(404, f"Artifact not found: {artifact_id}") + + storage_uri = artifact["storage_uri"] + filename = artifact.get("filename") or "artifact" + media_type = artifact.get("mime_type") or "application/octet-stream" + disposition = _inline_content_disposition(filename) + if Path(filename).suffix.lower() in {".html", ".htm"} or media_type.split(";")[0].strip().lower() in { + "text/html", "application/xhtml+xml", + }: + media_type = "application/octet-stream" + disposition = disposition.replace("inline", "attachment", 1) + headers = {"Content-Disposition": disposition, "X-Content-Type-Options": "nosniff"} + if is_object_uri(storage_uri): + response = _api.object_store.open(parse_object_uri(storage_uri)) + return StreamingResponse(response, media_type=media_type, headers=headers) + + file_path = Path(storage_uri) + if not file_path.exists(): + raise HTTPException(404, "Artifact content not found") + return StreamingResponse(open(file_path, "rb"), media_type=media_type, headers=headers) + + +@router.get("/documents/{workflow_id}/jobs") +async def list_document_jobs(workflow_id: str, user: RequireSearch, limit: int = Query(20, le=100)): + _require_document_for_user(workflow_id, user) + return [public_source_job(job) for job in db.list_document_jobs(workflow_id, limit=limit)] + + +@router.get("/documents/{workflow_id}/stage-io") +async def get_document_stage_io(workflow_id: str, user: RequireSearch): + doc = _require_document_for_user(workflow_id, user) + return _build_stage_io_payload(workflow_id, current_stage=doc.get("stage")) + + +@router.get("/documents/{workflow_id}/allowed-actions") +async def get_document_allowed_actions(workflow_id: str, user: RequireSearch): + """Return the currently valid machine-facing actions for a document.""" + doc = _require_document_for_user(workflow_id, user) + return { + "workflow_id": workflow_id, + "stage": doc.get("stage"), + "reindex_required": bool(doc.get("reindex_required")), + "available_actions": _list_available_actions(doc, db.get_latest_document_job(workflow_id)), + } + + +@router.get("/documents/{workflow_id}/graph", response_model=DocumentGraph) +async def get_document_graph(workflow_id: str, user: RequireSearch): + """Return a document-centric graph of state, jobs, artifacts, index status, and runtime.""" + doc = _require_document_for_user(workflow_id, user) + detail = _build_document_detail(doc) + return DocumentGraph( + workflow_id=workflow_id, + document=detail, + jobs=[public_source_job(job) for job in db.list_document_jobs(workflow_id, limit=100)], + artifacts=detail.artifacts, + index_status=detail.index_status, + stage_io=_build_stage_io_payload(workflow_id, current_stage=doc.get("stage")), + runtime=await _get_runtime_payload(workflow_id, doc=doc), + ) diff --git a/pipeline/api/routers/document_lifecycle.py b/pipeline/api/routers/document_lifecycle.py new file mode 100644 index 0000000..adbd60a --- /dev/null +++ b/pipeline/api/routers/document_lifecycle.py @@ -0,0 +1,784 @@ +"""Document lifecycle, retries, and reconciliation routes.""" + +import asyncio +import logging +import os +from datetime import datetime +from pathlib import Path + +from fastapi import APIRouter, HTTPException, Query + +from pipeline import db +from pipeline.network.catalog import scheme_catalog +from pipeline.documents import display_name as display_name_rules +from pipeline.auth.deps import CurrentUser, RequireAdmin, RequirePipeline, RequireReview +from pipeline.auth.permissions import Permission +from pipeline.search.index_identity import projection_doc_id, recorded_doc_id +from pipeline.network.release_policy import index_only_mode, intake_index_options +from pipeline.documents.models import ReindexStateRequest, SchemeMetadataUpdate +from pipeline.object_storage import is_object_uri, parse_object_uri +from pipeline.temporal.workflows import ( + ChunkingOnlyWorkflow, + OcrOnlyWorkflow, + ReingestionWorkflow, + TranslationOnlyWorkflow, +) +from pipeline.api import api as _api +from pipeline.api.api import ( + _actor_from_user, + _assert_upload_start_resolved, + _instance_scope_for_user, + _log_audit, + _mark_reindex_required, + _require_document_for_user, + delete_chunks_from_index, +) + +router = APIRouter() + + +def _vector_doc_ids(doc: dict, env: str) -> list[str]: + """Index keys to delete for a document: its current key plus keys of earlier file versions in ``env``. + + A key another live document is indexed under in ``env`` (the same file + uploaded twice) is left out, since deleting it would remove that document's + vectors too. + """ + ids = [recorded_doc_id(doc, env)] + if env == "dev" and index_only_mode() and doc.get("id") and doc.get("document_id"): + ids.append(projection_doc_id(doc)) + ids.extend((doc.get("superseded_doc_ids") or {}).get(env, [])) + return [ + i + for i in dict.fromkeys(i for i in ids if i) + if not db.doc_id_used_by_other_document(i, doc.get("id"), env) + ] + + +def _delete_all_doc_ids(store, collection: str, doc_ids: list[str]) -> dict: + results = [store.delete_by_doc_id(collection, doc_id) for doc_id in doc_ids] + if len(results) == 1: + return results[0] + errors = [r["error"] for r in results if r.get("error")] + summary = {"deleted": 0 if errors else -1, "doc_ids": doc_ids, "results": results} + if errors: + summary["error"] = "; ".join(errors) + return summary + + +def _delete_document_points_from_prod(doc: dict) -> dict: + """Best-effort delete of a promoted (non-scheme) document's vectors from PROD documents-index.""" + doc_ids = _vector_doc_ids(doc, "prod") + if not doc_ids: + return {"deleted": False, "reason": "no_doc_id"} + prod_url = (os.environ.get("PROD_VECTOR_DB_URL") or "").strip() + if not prod_url: + return {"deleted": False, "reason": "no_prod_qdrant_url"} + prod_key = (os.environ.get("PROD_VECTOR_DB_API_KEY") or "").strip() or None + collection = ( + os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME") + or os.environ.get("VECTOR_DB_COLLECTION_NAME") + or "documents-index" + ).strip() + try: + from pipeline.vector_store.qdrant_store import QdrantVectorStore, get_qdrant_client + + client = get_qdrant_client(url=prod_url, api_key=prod_key) + store = QdrantVectorStore(client=client) + return _delete_all_doc_ids(store, collection, doc_ids) + except Exception as exc: + logging.exception("prod vector delete failed doc_ids=%s collection=%s", doc_ids, collection) + return {"deleted": False, "error": str(exc), "collection": collection} + + +def _delete_scheme_points_from_prod(doc: dict) -> dict: + """Best-effort delete of scheme vectors from PROD schemes-index.""" + doc_ids = _vector_doc_ids(doc, "prod") + if not doc_ids: + return {"deleted": False, "reason": "no_doc_id"} + prod_url = ( + (os.environ.get("PROD_SCHEME_VECTOR_DB_URL") or "").strip() + or (os.environ.get("PROD_VECTOR_DB_URL") or "").strip() + ) + if not prod_url: + return {"deleted": False, "reason": "no_prod_qdrant_url"} + prod_key = ( + (os.environ.get("PROD_SCHEME_VECTOR_DB_API_KEY") or "").strip() + or (os.environ.get("PROD_VECTOR_DB_API_KEY") or "").strip() + or None + ) + collection = ( + os.environ.get("PROD_SCHEME_VECTOR_DB_COLLECTION_NAME") or "schemes-index" + ).strip() + try: + from pipeline.vector_store.qdrant_store import QdrantVectorStore, get_qdrant_client + + client = get_qdrant_client(url=prod_url, api_key=prod_key) + store = QdrantVectorStore(client=client) + return _delete_all_doc_ids(store, collection, doc_ids) + except Exception as exc: + logging.exception("prod vector delete failed doc_ids=%s collection=%s", doc_ids, collection) + return {"deleted": False, "error": str(exc), "collection": collection} + +def _delete_stored_artifacts(workflow_id: str) -> dict: + """Remove every stored object backing a document's artifacts. + + Artifacts are the only part of a document that lives outside Postgres and the + vector store — the original upload, the normalized PDF, and the OCR / + translation / chunk JSON snapshots. Nothing else reads them once the + document is gone, so a hard delete has to sweep them explicitly. + """ + outcome = {"deleted": 0, "missing": 0, "errors": []} + try: + artifacts = db.list_document_artifacts(workflow_id) + except Exception as exc: + outcome["errors"].append(f"list artifacts: {exc}") + return outcome + + for artifact in artifacts: + storage_uri = (artifact.get("storage_uri") or "").strip() + if not storage_uri: + continue + try: + if is_object_uri(storage_uri): + _api.object_store.delete(parse_object_uri(storage_uri)) + outcome["deleted"] += 1 + else: + path = Path(storage_uri) + if path.exists(): + path.unlink() + outcome["deleted"] += 1 + else: + outcome["missing"] += 1 + except Exception as exc: + outcome["errors"].append(f"{storage_uri}: {exc}") + return outcome + + +@router.delete("/documents/{workflow_id}") +async def disable_document( + workflow_id: str, + user: CurrentUser, + remove_from_search: bool = Query(True), + purge: bool = Query(False), + keep_audit: bool = Query(True), +): + """ + Remove a document — soft delete by default, hard delete with ``purge=true``. + + Both modes: + - Remove all chunks from the DEV vector index + - Delete PROD Qdrant points too, if this document was ever promoted + (schemes-index for scheme documents, documents-index otherwise) + - For scheme documents, also rebuild the scheme catalog + - Stop the workflow if it is still running + + Soft delete (default) then marks the document disabled in Postgres and keeps + every row and artifact, so ``POST /documents/{id}/restore`` can bring it + back. Use X-Include-Disabled: true in list_documents to see it. + + ``purge=true`` additionally deletes the MinIO artifacts (original upload, + normalized PDF, OCR/translation/chunk JSON) and every Postgres row (pages, + chunks, artifacts, jobs, index status, the document itself). This is + irreversible — restore has nothing left to restore. Audit rows are kept + unless ``keep_audit=false``, since they record the deletion itself. + + Args: + workflow_id: The document workflow ID + remove_from_search: If True (default), removes chunks from the index + purge: If True, hard delete — also drops MinIO artifacts and Postgres rows + keep_audit: If False (and purge=True), also deletes the audit trail + + Requires ``admin`` (super admin — any document), or ``delete_own`` + (state admin — only documents they uploaded, inside their own state). + A purge is irreversible, so it stays super-admin only. + """ + doc = _require_document_for_user(workflow_id, user) + + if not user.has_permission(Permission.ADMIN): + if not user.has_permission_for_instance(Permission.DELETE_OWN, doc.get("instance")): + raise HTTPException(403, "Missing permission: delete_own") + if purge: + raise HTTPException( + 403, "purge=true is restricted to super admins; delete without purge instead" + ) + owner = (doc.get("uploaded_by_user_id") or "").strip() + if not owner or owner != (user.user_id or "").strip(): + raise HTTPException(403, "You may only delete documents you uploaded") + + _assert_upload_start_resolved(workflow_id) + + # A purge that skipped the index would strand vectors no row points at any + # more, so "remove from everywhere" always includes the search cleanup. + if purge: + remove_from_search = True + + result = { + "workflow_id": workflow_id, + "disabled": True, + "purged": purge, + "workflow_cancelled": False, + "workflow_terminated": False, + "vector_deleted": 0, + "scheme_qdrant_deleted": None, + "prod_qdrant_deleted": None, + "catalog_version": None, + "stored_artifacts_deleted": None, + "sqlite_rows_deleted": None, + } + + # Stop the workflow if still running. A purge terminates rather than + # cancels: cancellation lets the workflow run its next activity, which + # would write rows back into the tables we are about to delete. + try: + handle = _api.temporal_client.get_workflow_handle(workflow_id) + if purge: + await handle.terminate(reason=f"Document purged by {user.username or user.user_id}") + result["workflow_terminated"] = True + else: + await handle.cancel() + result["workflow_cancelled"] = True + except Exception: + pass # Workflow already completed/cancelled + + # Mark as disabled in Postgres (a purge deletes the row outright below, but + # this keeps the document hidden if a later step fails partway through) + db.set_document_disabled(workflow_id, True) + + # Remove from the DEV vector index if requested + if remove_from_search: + # The current index key plus keys of earlier file versions still in DEV + # (see metadata.superseded_doc_ids). + for index, doc_id in enumerate(_vector_doc_ids(doc, "dev")): + vector_result = delete_chunks_from_index(doc_id) + if index == 0: + result["vector_deleted"] = vector_result.get("deleted", 0) + if "error" in vector_result: + result["vector_error"] = vector_result["error"] + + # PROD Qdrant cleanup, if this document was ever promoted. A document + # promoted as a scheme lives in the scheme index even if its kind was + # changed since, so the PROD index entry decides, not only the kind. + kind = (doc.get("kind") or "").strip().lower() + prod_entry = (doc.get("index_status") or {}).get("prod") or {} + prod_is_scheme = ( + kind == "scheme" + or prod_entry.get("index_name") == scheme_catalog.scheme_collection_name() + ) + if prod_is_scheme: + q_result = _delete_scheme_points_from_prod(doc) + result["scheme_qdrant_deleted"] = q_result + try: + result["catalog_version"] = scheme_catalog.on_scheme_document_disabled( + workflow_id, code=doc.get("code") + ) + except Exception as exc: + result["catalog_error"] = str(exc) + else: + result["prod_qdrant_deleted"] = _delete_document_points_from_prod(doc) + + # Hard delete: drop the artifacts and the Postgres rows themselves. Ordered + # storage-then-database so a failure mid-way leaves rows pointing at + # already-deleted objects (visible, fixable) rather than orphaned objects + # no row references any more (invisible, unreclaimable). + if purge: + result["stored_artifacts_deleted"] = await asyncio.to_thread( + _delete_stored_artifacts, workflow_id + ) + # Audit is written before the purge so the trail survives keep_audit=True. + _log_audit( + workflow_id=workflow_id, + action_type="purge_document", + entity_type="document", + metadata={ + "filename": doc.get("filename"), + "code": doc.get("code"), + "stage": doc.get("stage"), + "stored_artifacts_deleted": result["stored_artifacts_deleted"], + "keep_audit": keep_audit, + }, + user=user, + ) + result["sqlite_rows_deleted"] = db.purge_document(workflow_id, keep_audit=keep_audit) + result["disabled"] = False # nothing left to be disabled + return result + + # Log audit + _log_audit( + workflow_id=workflow_id, + action_type="disable_document", + entity_type="document", + metadata={ + "remove_from_search": remove_from_search, + "vector_deleted": result["vector_deleted"], + "scheme_qdrant_deleted": result.get("scheme_qdrant_deleted"), + "prod_qdrant_deleted": result.get("prod_qdrant_deleted"), + "code": doc.get("code"), + "catalog_version": result.get("catalog_version"), + }, + user=user, + ) + + return result + + +@router.patch("/documents/{workflow_id}/scheme-metadata") +async def patch_scheme_metadata( + workflow_id: str, + body: SchemeMetadataUpdate, + user: RequireReview, +): + """ + Set code / name / aliases / validity used by Master Catalog and Qdrant scheme promote. + + On completed docs, renaming the code sets pending_reindex (excluded from live + vector_schemes until re-promote). Other metadata edits bump catalog version only. + """ + _require_document_for_user(workflow_id, user, Permission.REVIEW) + try: + result = scheme_catalog.apply_scheme_metadata( + workflow_id, + code=body.code, + name=body.name, + aliases=body.aliases, + tool_routing=body.tool_routing, + catalog_visible=body.catalog_visible, + network_visible=body.network_visible, + valid_from=body.valid_from, + valid_to=body.valid_to, + display_name=body.display_name, + ) + except LookupError as exc: + raise HTTPException(404, str(exc)) from exc + except display_name_rules.DisplayNameLocked as exc: + raise HTTPException(409, str(exc)) from exc + except ValueError as exc: + raise HTTPException(400, str(exc)) from exc + + _log_audit( + workflow_id=workflow_id, + action_type="scheme_metadata_update", + entity_type="document", + metadata={ + "body": body.model_dump(exclude_none=True), + "catalog_version": result.get("catalog_version"), + "pending_reindex": result.get("pending_reindex"), + }, + user=user, + ) + doc = result["document"] + return { + "workflow_id": workflow_id, + "kind": doc.get("kind"), + "code": doc.get("code"), + "name": doc.get("name"), + "aliases": list(doc.get("aliases") or []), + "tool_routing": doc.get("tool_routing"), + "catalog_visible": bool(int(doc["catalog_visible"])) if doc.get("catalog_visible") is not None else True, + "network_visible": bool(int(doc["network_visible"])) if doc.get("network_visible") is not None else True, + "valid_from": doc.get("valid_from"), + "valid_to": doc.get("valid_to"), + "catalog_version": result.get("catalog_version"), + "requires_reindex": result.get("requires_reindex"), + "pending_reindex": result.get("pending_reindex"), + "reindex_reason": doc.get("reindex_reason"), + } + + +@router.post("/documents/{workflow_id}/restore") +async def restore_document(workflow_id: str, user: RequireAdmin): + """ + Restore a soft-deleted (disabled) document. + + Note: This only restores the document in Postgres. Chunks that were removed + from the vector index will NOT be automatically re-indexed. To re-index, + you would need to re-run the ingestion process. + """ + _require_document_for_user(workflow_id, user) + + db.set_document_disabled(workflow_id, False) + + # Log audit + _log_audit( + workflow_id=workflow_id, + action_type="restore_document", + entity_type="document", + user=user, + ) + + return {"workflow_id": workflow_id, "restored": True} + + +@router.post("/documents/{workflow_id}/reingest") +async def reingest_document( + workflow_id: str, + user: RequirePipeline, + index_name: str = "documents-index", +): + """ + Re-ingest a completed document to the vector index. + + Use this to re-ingest documents that completed but weren't properly + indexed (e.g., due to index schema changes). This starts a lightweight + workflow that uses chunks already stored in Postgres. + + The document must have chunks stored in Postgres (typically from a + completed or previously ingested document). + """ + index_name, _ = intake_index_options(index_name, False) + doc = _require_document_for_user(workflow_id, user, Permission.PIPELINE) + _assert_upload_start_resolved(workflow_id) + + # Get chunks from Postgres + chunks = db.get_chunks(workflow_id, include_excluded=False) + if not chunks: + raise HTTPException(400, "No chunks found for document. The document may need to be reprocessed from scratch.") + + document_id = doc.get("document_id", "") + filename = doc.get("filename", "") + page_count = doc.get("page_count", 0) + + # Generate unique workflow ID for re-ingestion + import time + reingest_workflow_id = f"{workflow_id}-reingest-{int(time.time())}" + + # Pass the deployment mode in the workflow input to keep replay deterministic. + args = [document_id, filename, workflow_id, page_count, len(chunks), index_name] + if index_only_mode(): + args.append(True) + job_id = db.create_document_job( + workflow_id=workflow_id, + job_type="reingest", + temporal_workflow_id=reingest_workflow_id, + status="running", + current_stage="ingesting", + config={"index_name": index_name, "chunk_count": len(chunks)}, + actor=_actor_from_user(user), + ) + try: + await _api.temporal_client.start_workflow( + ReingestionWorkflow.run, args=args, id=reingest_workflow_id, task_queue=_api.TASK_QUEUE, + ) + except Exception: + db.update_document_job(job_id, status="failed", current_stage="failed", error_message="Could not start re-ingestion") + raise + + # Log audit + _log_audit( + workflow_id=workflow_id, + action_type="reingest_started", + entity_type="document", + metadata={"reingest_workflow_id": reingest_workflow_id, "chunk_count": len(chunks)}, + user=user, + ) + + return { + "workflow_id": workflow_id, + "reingest_workflow_id": reingest_workflow_id, + "chunk_count": len(chunks), + "status": "started" + } + + +@router.post("/documents/{workflow_id}/retry-ingestion") +async def retry_ingestion( + workflow_id: str, + user: RequirePipeline, + index_name: str = "documents-index", +): + """Alias for reingesting a document when search is stale or missing.""" + return await reingest_document( + workflow_id, + user=user, + index_name=index_name, + ) + + +@router.post("/documents/{workflow_id}/retry-ocr") +async def retry_ocr(workflow_id: str, user: RequirePipeline): + """Retry OCR for an existing document and stop at OCR review.""" + doc = _require_document_for_user(workflow_id, user, Permission.PIPELINE) + _assert_upload_start_resolved(workflow_id) + filepath = doc.get("filepath") + if not filepath: + raise HTTPException(400, "Document has no source filepath for OCR retry") + temporal_workflow_id = f"{workflow_id}-retry-ocr-{int(datetime.utcnow().timestamp())}" + await _api.temporal_client.start_workflow( + OcrOnlyWorkflow.run, + args=[workflow_id, doc["document_id"], doc["filename"], filepath], + id=temporal_workflow_id, + task_queue=_api.TASK_QUEUE, + ) + db.create_document_job( + workflow_id=workflow_id, + job_type="ocr_retry", + temporal_workflow_id=temporal_workflow_id, + status="running", + current_stage="ocr_processing", + config={"source": "api_retry_ocr"}, + actor=_actor_from_user(user), + ) + db.update_document_fields(workflow_id, error_message=None) + _log_audit( + workflow_id=workflow_id, + action_type="retry_ocr", + entity_type="document", + metadata={"temporal_workflow_id": temporal_workflow_id}, + user=user, + ) + return {"workflow_id": workflow_id, "status": "started", "retry_workflow_id": temporal_workflow_id} + + +@router.post("/documents/{workflow_id}/retry-translation") +async def retry_translation(workflow_id: str, user: RequirePipeline): + """Retry translation for an existing document and stop at translation review.""" + doc = _require_document_for_user(workflow_id, user, Permission.PIPELINE) + _assert_upload_start_resolved(workflow_id) + if not db.get_pages(workflow_id): + raise HTTPException(400, "No OCR pages found for translation retry") + temporal_workflow_id = f"{workflow_id}-retry-translation-{int(datetime.utcnow().timestamp())}" + await _api.temporal_client.start_workflow( + TranslationOnlyWorkflow.run, + args=[workflow_id, doc["document_id"], doc["filename"]], + id=temporal_workflow_id, + task_queue=_api.TASK_QUEUE, + ) + db.create_document_job( + workflow_id=workflow_id, + job_type="translation_retry", + temporal_workflow_id=temporal_workflow_id, + status="running", + current_stage="translation_processing", + config={"source": "api_retry_translation"}, + actor=_actor_from_user(user), + ) + db.update_document_fields(workflow_id, error_message=None) + _log_audit( + workflow_id=workflow_id, + action_type="retry_translation", + entity_type="document", + metadata={"temporal_workflow_id": temporal_workflow_id}, + user=user, + ) + return {"workflow_id": workflow_id, "status": "started", "retry_workflow_id": temporal_workflow_id} + + +@router.post("/documents/{workflow_id}/retry-chunking") +async def retry_chunking( + workflow_id: str, + user: RequirePipeline, + chunk_size: int = 450, + chunk_overlap: int = 128, + min_tokens: int = 100, +): + """Retry chunking for an existing document and stop at chunk review.""" + doc = _require_document_for_user(workflow_id, user, Permission.PIPELINE) + _assert_upload_start_resolved(workflow_id) + if not db.get_pages(workflow_id): + raise HTTPException(400, "No page content found for chunking retry") + temporal_workflow_id = f"{workflow_id}-retry-chunking-{int(datetime.utcnow().timestamp())}" + await _api.temporal_client.start_workflow( + ChunkingOnlyWorkflow.run, + args=[ + workflow_id, + doc["document_id"], + doc["filename"], + doc.get("page_count", 0), + chunk_size, + chunk_overlap, + min_tokens, + ], + id=temporal_workflow_id, + task_queue=_api.TASK_QUEUE, + ) + db.create_document_job( + workflow_id=workflow_id, + job_type="chunking_retry", + temporal_workflow_id=temporal_workflow_id, + status="running", + current_stage="chunking", + config={ + "source": "api_retry_chunking", + "chunk_size": chunk_size, + "chunk_overlap": chunk_overlap, + "min_tokens": min_tokens, + }, + actor=_actor_from_user(user), + ) + db.update_document_fields(workflow_id, error_message=None) + _log_audit( + workflow_id=workflow_id, + action_type="retry_chunking", + entity_type="document", + metadata={"temporal_workflow_id": temporal_workflow_id}, + user=user, + ) + return {"workflow_id": workflow_id, "status": "started", "retry_workflow_id": temporal_workflow_id} + + +@router.post("/documents/{workflow_id}/mark-reindex-required") +async def mark_reindex_required(workflow_id: str, payload: ReindexStateRequest, user: RequirePipeline): + """Mark a document as needing reindex after chunk edits or operational drift.""" + _require_document_for_user(workflow_id, user, Permission.PIPELINE) + updated = _mark_reindex_required( + workflow_id, + payload.reason or "Marked manually for reindex", + metadata={"source": "api"}, + ) + return { + "workflow_id": workflow_id, + "reindex_required": bool(updated.get("reindex_required")) if updated else True, + "reindex_reason": updated.get("reindex_reason") if updated else payload.reason, + } + + +@router.post("/documents/{workflow_id}/clear-reindex-required") +async def clear_reindex_required(workflow_id: str, user: RequirePipeline): + """Clear the reindex-required flag after verification or reingestion.""" + doc = _require_document_for_user(workflow_id, user, Permission.PIPELINE) + old_reason = doc.get("reindex_reason") + updated = db.mark_document_reindex_required(workflow_id, False) + _log_audit( + workflow_id=workflow_id, + action_type="clear_reindex_required", + entity_type="document", + field_name="reindex_required", + old_value="true", + new_value="false", + metadata={"reason": old_reason}, + user=user, + ) + return { + "workflow_id": workflow_id, + "reindex_required": bool(updated.get("reindex_required")) if updated else False, + "reindex_reason": updated.get("reindex_reason") if updated else None, + } + + +async def _reconcile_single_document(doc: dict) -> dict: + workflow_id = doc.get("workflow_id") + current_stage = doc.get("stage") + materialized = db.reconcile_materialized_state(workflow_id) + if materialized and materialized.get("updated"): + doc = db.get_document(workflow_id) or doc + current_stage = doc.get("stage") + return { + "workflow_id": workflow_id, + "action": "materialized_state_reconciled", + "to": current_stage, + "page_count": doc.get("page_count", 0), + "chunk_count": doc.get("chunk_count", 0), + "job_status": materialized.get("job_status"), + "job_stage": materialized.get("job_stage"), + } + + current_job = db.get_latest_document_job(workflow_id) + runtime_workflow_id = ( + current_job.get("temporal_workflow_id") + if current_job and current_job.get("status") == "running" and current_job.get("temporal_workflow_id") + else workflow_id + ) + + try: + handle = _api.temporal_client.get_workflow_handle(runtime_workflow_id) + state = await asyncio.wait_for( + handle.query("get_state"), + timeout=5.0, + ) + temporal_stage = state.get("stage") if state else None + if temporal_stage and temporal_stage != current_stage: + db.update_document_stage(workflow_id, temporal_stage) + return { + "workflow_id": workflow_id, + "action": "stage_synced", + "from": current_stage, + "to": temporal_stage, + "temporal_workflow_id": runtime_workflow_id, + } + return { + "workflow_id": workflow_id, + "action": "no_change", + "stage": current_stage, + "temporal_workflow_id": runtime_workflow_id, + } + except asyncio.TimeoutError: + db.update_document_stage(workflow_id, "failed", error_message="Workflow query timed out") + return { + "workflow_id": workflow_id, + "action": "marked_failed", + "from": current_stage, + "reason": "query_timeout", + } + except Exception as exc: + error_msg = str(exc) + if "not found" in error_msg.lower() or "workflow task" in error_msg.lower(): + db.update_document_stage(workflow_id, "failed", error_message="Workflow terminated or lost") + db.log_audit( + workflow_id=workflow_id, + document_id=doc.get("document_id", ""), + action_type="reconcile_failed", + metadata={"from_stage": current_stage, "reason": "workflow_not_found"}, + ) + return { + "workflow_id": workflow_id, + "action": "marked_failed", + "from": current_stage, + "reason": "workflow_not_found", + } + return { + "workflow_id": workflow_id, + "action": "error", + "from": current_stage, + "reason": error_msg, + } + + +@router.post("/documents/{workflow_id}/reconcile") +async def reconcile_single_document(workflow_id: str, user: RequirePipeline): + """Reconcile Postgres stage with Temporal state for one document.""" + doc = _require_document_for_user(workflow_id, user, Permission.PIPELINE) + return await _reconcile_single_document(doc) + +@router.post("/documents/reconcile") +async def reconcile_document_states(user: RequirePipeline): + """ + Reconcile Postgres document states with Temporal workflow states. + + This endpoint checks all documents in processing/review stages and updates + Postgres if the Temporal workflow has terminated or failed. This fixes + inconsistencies caused by external workflow termination or worker crashes. + + Returns a summary of documents checked and updated. + """ + # Stages that indicate an active workflow (not terminal states) + active_stages = [ + 'ocr_processing', 'ocr_review', + 'translation_processing', 'translation_review', + 'chunking', 'chunk_review', + 'ready_for_ingestion', 'ingesting' + ] + + # Scope to caller's instances (None = unrestricted bypass / all tenants). + docs = db.list_documents( + limit=1000, + include_disabled=True, + instances=_instance_scope_for_user(user, Permission.PIPELINE), + ) + active_docs = [d for d in docs if d.get('stage') in active_stages] + + results = { + "checked": len(active_docs), + "updated": 0, + "still_running": 0, + "details": [] + } + + for doc in active_docs: + detail = await _reconcile_single_document(doc) + results["details"].append(detail) + if detail.get("action") == "stage_synced" or detail.get("action") == "marked_failed": + results["updated"] += 1 + elif detail.get("action") == "no_change": + results["still_running"] += 1 + + return results diff --git a/pipeline/api/routers/document_review.py b/pipeline/api/routers/document_review.py new file mode 100644 index 0000000..6cc32ed --- /dev/null +++ b/pipeline/api/routers/document_review.py @@ -0,0 +1,433 @@ +"""Document review and legacy approval routes.""" + +from fastapi import APIRouter, HTTPException + +from pipeline import db +from pipeline.auth.deps import RequireAdmin, RequireApproveIngestion, RequirePipeline, RequireReview +from pipeline.auth.models import AuthUser +from pipeline.auth.permissions import Permission +from pipeline.network.release_policy import index_only_mode, prod_stage_disabled +from pipeline.documents.models import ( + BulkWorkflowActionRequest, + BulkWorkflowActionResponse, + BulkWorkflowActionResult, +) +from pipeline.temporal.workflows import DocumentPipelineWorkflow, PromoteToProdWorkflow +from pipeline.api import api as _api +from pipeline.api.api import ( + _actor_from_user, + _document_for_user_or_none, + _log_audit, + _require_document_for_user, +) +from pipeline.api.routers.document_lifecycle import reingest_document + +router = APIRouter() + + +_STAGE_APPROVAL_HINTS = { + "ocr_review": "POST /documents/{id}/approve-ocr", + "translation_review": "POST /documents/{id}/approve-translation", + "chunk_review": "POST /documents/{id}/approve-chunks", + "ready_for_ingestion": "POST /documents/{id}/approve-ingestion", + "approval_for_prod": "POST /documents/{id}/approve-prod", +} + + +async def _validate_approval_stage(workflow_id: str, expected_stage: str): + """Validate that workflow is in the expected stage before approval.""" + try: + handle = _api.temporal_client.get_workflow_handle(workflow_id) + state = await handle.query("get_state") + current_stage = state.get("stage") if isinstance(state, dict) else getattr(state, "stage", None) + if current_stage != expected_stage: + hint = _STAGE_APPROVAL_HINTS.get(str(current_stage or "")) + extra = f" Use {hint.replace('{id}', workflow_id)} instead." if hint else "" + raise HTTPException( + 400, + ( + f"Cannot approve: workflow is in '{current_stage}' stage, " + f"expected '{expected_stage}'.{extra}" + ), + ) + return handle + except HTTPException: + raise + except Exception: + # Try Postgres fallback to check if workflow exists but is completed/failed + doc = db.get_document(workflow_id) + if doc: + current = doc.get("stage") + hint = _STAGE_APPROVAL_HINTS.get(str(current or "")) + extra = f" Use {hint.replace('{id}', workflow_id)} instead." if hint else "" + raise HTTPException( + 400, + ( + f"Cannot approve: workflow is in '{current}' stage " + f"(completed/failed workflows cannot be approved).{extra}" + ), + ) + raise HTTPException(404, f"Workflow not found: {workflow_id}") + + +async def _execute_bulk_approval_action( + request: BulkWorkflowActionRequest, + action: str, + expected_stage: str, + signal_method, + user: AuthUser, +) -> BulkWorkflowActionResponse: + results: list[BulkWorkflowActionResult] = [] + for workflow_id in request.workflow_ids: + doc = _document_for_user_or_none(workflow_id, user, Permission.REVIEW) + if not doc: + results.append(BulkWorkflowActionResult( + workflow_id=workflow_id, + ok=False, + action=action, + message="document_not_found", + )) + continue + current_stage = doc.get("stage") + if current_stage != expected_stage: + results.append(BulkWorkflowActionResult( + workflow_id=workflow_id, + ok=False, + action=action, + message=f"invalid_stage:{current_stage}", + )) + continue + if request.dry_run: + results.append(BulkWorkflowActionResult( + workflow_id=workflow_id, + ok=True, + action=action, + message="would_execute", + )) + continue + try: + handle = await _validate_approval_stage(workflow_id, expected_stage) + await handle.signal(signal_method) + results.append(BulkWorkflowActionResult( + workflow_id=workflow_id, + ok=True, + action=action, + message="queued", + )) + except Exception as exc: + results.append(BulkWorkflowActionResult( + workflow_id=workflow_id, + ok=False, + action=action, + message=str(exc), + )) + + return BulkWorkflowActionResponse( + action=action, + dry_run=request.dry_run, + requested=len(request.workflow_ids), + succeeded=sum(1 for result in results if result.ok), + failed=sum(1 for result in results if not result.ok), + results=results, + ) + + +@router.post("/documents/bulk/approve-ocr", response_model=BulkWorkflowActionResponse) +async def bulk_approve_ocr(request: BulkWorkflowActionRequest, user: RequireReview): + """Bulk-approve documents waiting in OCR review.""" + return await _execute_bulk_approval_action( + request, + action="approve_ocr", + expected_stage="ocr_review", + signal_method=DocumentPipelineWorkflow.approve_ocr, + user=user, + ) + + +@router.post("/documents/bulk/approve-translation", response_model=BulkWorkflowActionResponse) +async def bulk_approve_translation(request: BulkWorkflowActionRequest, user: RequireReview): + """Bulk-approve documents waiting in translation review.""" + return await _execute_bulk_approval_action( + request, + action="approve_translation", + expected_stage="translation_review", + signal_method=DocumentPipelineWorkflow.approve_translation, + user=user, + ) + + +@router.post("/documents/bulk/approve-chunks", response_model=BulkWorkflowActionResponse) +async def bulk_approve_chunks(request: BulkWorkflowActionRequest, user: RequireReview): + """Bulk-approve documents waiting in chunk review.""" + return await _execute_bulk_approval_action( + request, + action="approve_chunks", + expected_stage="chunk_review", + signal_method=DocumentPipelineWorkflow.approve_chunks, + user=user, + ) + + +@router.post("/documents/bulk/reindex", response_model=BulkWorkflowActionResponse) +async def bulk_reindex_documents( + request: BulkWorkflowActionRequest, + user: RequirePipeline, + index_name: str = "documents-index", +): + """Bulk queue reingestion for completed or dirty documents.""" + results: list[BulkWorkflowActionResult] = [] + for workflow_id in request.workflow_ids: + doc = _document_for_user_or_none(workflow_id, user, Permission.PIPELINE) + if not doc: + results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=False, action="reindex", message="document_not_found")) + continue + if doc.get("stage") not in {"completed", "ready_for_ingestion", "chunk_review"} and not doc.get("reindex_required"): + results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=False, action="reindex", message=f"invalid_stage:{doc.get('stage')}")) + continue + if request.dry_run: + results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=True, action="reindex", message="would_execute")) + continue + try: + await reingest_document(workflow_id, user=user, index_name=index_name) + results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=True, action="reindex", message="queued")) + except Exception as exc: + results.append(BulkWorkflowActionResult(workflow_id=workflow_id, ok=False, action="reindex", message=str(exc))) + + return BulkWorkflowActionResponse( + action="reindex", + dry_run=request.dry_run, + requested=len(request.workflow_ids), + succeeded=sum(1 for result in results if result.ok), + failed=sum(1 for result in results if not result.ok), + results=results, + ) + + +@router.post("/documents/{workflow_id}/approve-ocr") +async def approve_ocr(workflow_id: str, user: RequireReview): + """Approve OCR results and continue to chunking. Requires permission: review.""" + _require_document_for_user(workflow_id, user, Permission.REVIEW) + handle = await _validate_approval_stage(workflow_id, "ocr_review") + await handle.signal(DocumentPipelineWorkflow.approve_ocr) + + # Log approval + _log_audit( + workflow_id=workflow_id, + action_type="approval", + entity_type="document", + field_name="ocr_approved", + new_value=True, + metadata={"stage": "ocr_review", "next_stage": "translation_processing"}, + user=user, + ) + + return {"approved": "ocr", "workflow_id": workflow_id} + + +@router.post("/documents/{workflow_id}/approve-chunks") +async def approve_chunks(workflow_id: str, user: RequireReview): + """Approve chunks and continue to prepare for ingestion.""" + _require_document_for_user(workflow_id, user, Permission.REVIEW) + handle = await _validate_approval_stage(workflow_id, "chunk_review") + await handle.signal(DocumentPipelineWorkflow.approve_chunks) + + # Log approval + _log_audit( + workflow_id=workflow_id, + action_type="approval", + entity_type="document", + field_name="chunks_approved", + new_value=True, + metadata={"stage": "chunk_review", "next_stage": "ready_for_ingestion"}, + user=user, + ) + + return {"approved": "chunks", "workflow_id": workflow_id} + + +@router.post("/documents/{workflow_id}/approve-translation") +async def approve_translation(workflow_id: str, user: RequireReview): + """Approve translations and continue to chunking.""" + _require_document_for_user(workflow_id, user, Permission.REVIEW) + handle = await _validate_approval_stage(workflow_id, "translation_review") + await handle.signal(DocumentPipelineWorkflow.approve_translation) + + # Log approval + _log_audit( + workflow_id=workflow_id, + action_type="approval", + entity_type="document", + field_name="translation_approved", + new_value=True, + metadata={"stage": "translation_review", "next_stage": "chunking"}, + user=user, + ) + + return {"approved": "translation", "workflow_id": workflow_id} + + +@router.post("/documents/{workflow_id}/approve-ingestion") +async def approve_ingestion(workflow_id: str, user: RequireApproveIngestion): + """Approve ingestion and continue to vector-index ingestion. + + Requires APPROVE_INGESTION, not REVIEW: a ``state_contributor`` may approve + the OCR / translation / chunking gates but must not publish to DEV. + """ + _require_document_for_user(workflow_id, user, Permission.APPROVE_INGESTION) + handle = await _validate_approval_stage(workflow_id, "ready_for_ingestion") + await handle.signal(DocumentPipelineWorkflow.approve_ingestion) + + # Log approval + _log_audit( + workflow_id=workflow_id, + action_type="approval", + entity_type="document", + field_name="ingestion_approved", + new_value=True, + metadata={"stage": "ready_for_ingestion", "next_stage": "ingesting"}, + user=user, + ) + + return {"approved": "ingestion", "workflow_id": workflow_id} + + +@router.post("/documents/{workflow_id}/request-prod-ready") +async def request_prod_ready(workflow_id: str, user: RequireReview): + """ + state_admin: flag this document as ready for a super_admin to review for + PROD promotion. Purely informational — does not itself fire approve_prod + or promote anything; the super_admin still makes that call via + POST /documents/{id}/approve-prod. Mirrors the "request ingest" pattern + used elsewhere in this pipeline: a request is a separate, weaker signal + from the actual privileged action. + """ + doc = _require_document_for_user(workflow_id, user, Permission.REVIEW) + stage = doc.get("stage") + if stage != "approval_for_prod": + raise HTTPException( + 400, + f"Cannot request prod-ready: document is in '{stage}' stage " + "(expected 'approval_for_prod').", + ) + + updated = db.mark_prod_ready_requested( + workflow_id, + user_id=getattr(user, "user_id", None), + username=getattr(user, "username", None), + ) + + _log_audit( + workflow_id=workflow_id, + action_type="request_prod_ready", + entity_type="document", + field_name="prod_ready_requested_at", + new_value=updated.get("prod_ready_requested_at") if updated else None, + metadata={"stage": stage}, + user=user, + ) + + return { + "requested": True, + "workflow_id": workflow_id, + "prod_ready_requested_at": updated.get("prod_ready_requested_at") if updated else None, + "prod_ready_requested_by_username": updated.get("prod_ready_requested_by_username") if updated else None, + } + + +@router.post("/documents/{workflow_id}/approve-prod") +async def approve_prod( + workflow_id: str, + user: RequireAdmin, +): + """Superadmin-only: promote DEV-ingested vectors into PROD Qdrant.""" + if index_only_mode(): + raise HTTPException(409, "Promotion is disabled in index-only mode") + doc = _require_document_for_user(workflow_id, user) + if doc.get("uses_network_releases"): + raise HTTPException(409, "This upload uses automatic release to its configured network") + stage = doc.get("stage") + # Promoting an already-completed document is a fresh PROD write, so it is + # blocked outright when PROD is disabled. A document still parked at the + # approval gate (started before the flag was set) is allowed through, or it + # would be stranded there forever. + if prod_stage_disabled() and stage != "approval_for_prod": + raise HTTPException( + 400, + "PROD promotion is disabled (DISABLE_PROD_SETTING=true). " + "Documents finish at DEV ingest.", + ) + if stage not in {"approval_for_prod", "completed"}: + raise HTTPException( + 400, + f"Cannot approve for prod: document is in '{stage}' stage " + "(expected 'approval_for_prod' or 'completed').", + ) + + # Prefer signaling the running main workflow when it is waiting at approval_for_prod. + signaled = False + if stage == "approval_for_prod": + try: + handle = await _validate_approval_stage(workflow_id, "approval_for_prod") + await handle.signal(DocumentPipelineWorkflow.approve_prod) + signaled = True + except HTTPException: + signaled = False + except Exception: + signaled = False + + promote_workflow_id = None + if not signaled: + import time + + promote_workflow_id = f"{workflow_id}-promote-prod-{int(time.time())}" + await _api.temporal_client.start_workflow( + PromoteToProdWorkflow.run, + args=[ + doc.get("document_id") or workflow_id, + doc.get("filename") or workflow_id, + workflow_id, + int(doc.get("page_count") or 0), + int(doc.get("chunk_count") or 0), + ], + id=promote_workflow_id, + task_queue=_api.TASK_QUEUE, + ) + db.create_document_job( + workflow_id=workflow_id, + job_type="promote_prod", + temporal_workflow_id=promote_workflow_id, + status="running", + current_stage="ingesting_prod", + actor=_actor_from_user(user), + ) + db.update_document_stage( + workflow_id=workflow_id, + stage="ingesting_prod", + page_count=doc.get("page_count"), + chunk_count=doc.get("chunk_count"), + ) + + if doc.get("prod_ready_requested_at"): + db.clear_prod_ready_requested(workflow_id) + + _log_audit( + workflow_id=workflow_id, + action_type="approval", + entity_type="document", + field_name="prod_approved", + new_value=True, + metadata={ + "stage": stage, + "next_stage": "ingesting_prod", + "signaled": signaled, + "promote_workflow_id": promote_workflow_id, + }, + user=user, + ) + return { + "approved": "prod", + "workflow_id": workflow_id, + "signaled": signaled, + "promote_workflow_id": promote_workflow_id, + "next_stage": "ingesting_prod", + } diff --git a/pipeline/api/documents.py b/pipeline/api/routers/documents.py similarity index 64% rename from pipeline/api/documents.py rename to pipeline/api/routers/documents.py index f7d3867..119e95f 100644 --- a/pipeline/api/documents.py +++ b/pipeline/api/routers/documents.py @@ -1,30 +1,116 @@ from __future__ import annotations -import hashlib -import logging -from pathlib import Path from typing import Optional +from uuid import UUID, uuid4 from fastapi import APIRouter, Header, HTTPException, Query, Request +from pydantic import BaseModel -from .. import db -from ..documents import document_validity -from ..auth.deps import CurrentUser, RequireUpload -from ..auth.tenancy import assert_document_instance_access, normalize_instance -from ..instances import prod_stage_disabled -from ..models import ( +from pipeline import db +from pipeline.documents import document_validity +from pipeline.auth.deps import CurrentUser, RequireUpload +from pipeline.auth.permissions import Permission +from pipeline.auth.tenancy import assert_document_instance_access, normalize_instance +from pipeline.network.release_policy import ( + index_only_mode, + intake_index_options, + managed_release_targets, + prod_stage_disabled, +) +from pipeline.documents.models import ( DocumentCohortsResponse, DocumentStage, DocumentSummary, - RegisterFolderRequest, RegisterRequest, ) -from ..workflows import DocumentPipelineWorkflow -from . import api as _api +from pipeline.temporal.workflows import DocumentPipelineWorkflow, SelectNetworkRevisionWorkflow +from pipeline.api import api as _api router = APIRouter(prefix="/documents") +class SelectRevisionRequest(BaseModel): + revision_id: UUID + expected_generation: UUID | None = None + + +def _require_managed_release_network(kind: str, network_id: str) -> None: + try: + targets = managed_release_targets(kind) + except ValueError as exc: + raise HTTPException(503, "Network release is misconfigured") from exc + if network_id not in targets: + raise HTTPException(404, "This deployment does not manage that network") + + +@router.get("/{workflow_id}/revisions") +async def list_revisions(workflow_id: str, user: CurrentUser): + document = _api._require_document_for_user(workflow_id, user) + result = db.list_document_revisions(document["id"]) + try: + targets = list(managed_release_targets(document["kind"])) + except ValueError: + targets = [] + return {**result, "managed_network_ids": targets, "managed_network_id": targets[0] if targets else None} + + +@router.get("/{workflow_id}/revisions/{revision_id}") +async def read_revision(workflow_id: str, revision_id: str, user: CurrentUser): + document = _api._require_document_for_user(workflow_id, user) + revision = db.get_revision_snapshot(document["id"], revision_id) + if revision is None: + raise HTTPException(404, "Revision not found") + snapshot = revision["snapshot"] + return { + "revision_id": revision["id"], + "content_sha256": revision["content_sha256"], + "metadata": snapshot["metadata"], + "filename": snapshot["source"]["filename"], + "source_url": snapshot["source"].get("url"), + "pages": snapshot["pages"], + "passage_count": len(snapshot["passages"]), + } + + +@router.delete("/{workflow_id}/releases/{network_id}", status_code=204) +async def withdraw_release(workflow_id: str, network_id: str, revision_id: str, user: CurrentUser): + document = _api._require_document_for_user(workflow_id, user) + if not user.has_permission_for_instance(Permission.APPROVE_INGESTION, document["instance"]): + raise HTTPException(403, "Network release permission is required for this provider") + _require_managed_release_network(document["kind"], network_id) + with db.get_connection() as conn: + if not db.withdraw_network_release(document["id"], network_id, revision_id, _conn=conn): + raise HTTPException(409, "The selected revision is not the current release on this network") + _api._log_audit( + workflow_id=workflow_id, action_type="withdraw_network_release", entity_type="document", + metadata={"network_id": network_id, "revision_id": revision_id}, user=user, _conn=conn, + ) + + +@router.put("/{workflow_id}/releases/{network_id}", status_code=202) +async def select_release(workflow_id: str, network_id: str, data: SelectRevisionRequest, user: CurrentUser): + document = _api._require_document_for_user(workflow_id, user) + if not user.has_permission_for_instance(Permission.APPROVE_INGESTION, document["instance"]): + raise HTTPException(403, "Network release permission is required for this provider") + _require_managed_release_network(document["kind"], network_id) + if document["is_disabled"] or not document["network_visible"]: + raise HTTPException(409, "The document is unavailable for release") + revision_id = str(data.revision_id) + if db.get_revision_snapshot(document["id"], revision_id) is None: + raise HTTPException(404, "Revision not found") + release_run = f"release-{uuid4()}" + await _api.temporal_client.start_workflow( + SelectNetworkRevisionWorkflow.run, + args=[document["id"], revision_id, network_id, str(data.expected_generation) if data.expected_generation else None], + id=release_run, task_queue=_api.TASK_QUEUE, + ) + _api._log_audit( + workflow_id=workflow_id, action_type="select_network_revision", entity_type="document", + metadata={"network_id": network_id, "revision_id": revision_id, "release_workflow_id": release_run}, user=user, + ) + return {"workflow_id": release_run, "network_id": network_id, "revision_id": revision_id} + + @router.post("", response_model=DocumentSummary) @_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) async def start_document_workflow( @@ -51,18 +137,19 @@ async def start_document_workflow( 4. Wait for approval (unless auto_approve=True) 5. Ingest to the vector index - Note: File path must be within allowed directories (ALLOWED_FILE_PATHS env var). + The object must already be registered to the requesting provider. Rate limited to 10 requests/minute per IP. Requires permission: upload (no-op while AUTH_DISABLED=true). """ create_instance = _api._resolve_create_instance(user, instance) - # Validate file path to prevent path traversal attacks filepath = _api.validate_file_path(data.filepath) source_filename = _api.get_filename_from_path(filepath) doc_kind, doc_code, replace_doc = _api._resolve_document_identity( user, create_instance, kind, code, source_filename ) - source_file_fingerprint = _api._compute_file_fingerprint(filepath) + source_file_fingerprint = db.owned_source_fingerprint(filepath, create_instance) + if not source_file_fingerprint: + raise HTTPException(404, "Source is not registered to this provider") canonical_document_id = source_file_fingerprint workflow_id = _api.get_workflow_id(str(filepath)) @@ -71,7 +158,7 @@ async def start_document_workflow( # Reuse only when Postgres still tracks this workflow. # If Postgres was purged, avoid returning stale Temporal state and create a fresh run ID. existing_doc = db.get_document(workflow_id) - if existing_doc: + if existing_doc and normalize_instance(existing_doc.get("instance")) == create_instance: # Same fingerprint/path must not leak or restart another tenant's doc. existing_doc = assert_document_instance_access(user, existing_doc) try: @@ -96,8 +183,9 @@ async def start_document_workflow( raise except Exception: pass # Workflow doesn't exist or is not queryable; proceed to new run - else: - workflow_id = _api._rerun_workflow_id(workflow_id) + workflow_id = _api._rerun_workflow_id(workflow_id) + + index_name, auto_approve = intake_index_options(index_name, auto_approve) # Start new workflow handle = await _api.temporal_client.start_workflow( @@ -113,6 +201,7 @@ async def start_document_workflow( auto_approve, stop_after_ocr, prod_stage_disabled(), + *([None, None, True] if index_only_mode() else []), ], id=workflow_id, task_queue=_api.TASK_QUEUE, @@ -210,64 +299,6 @@ async def start_document_workflow( ) -@router.post("/batch", response_model=list[DocumentSummary]) -@_api.limiter.limit("5/minute") # Stricter limit for batch operations -async def start_batch_workflows( - request: Request, # Required for rate limiting - data: RegisterFolderRequest, - user: RequireUpload, - auto_approve: bool = False, - chunk_size: int = 450, - chunk_overlap: int = 128, - min_tokens: int = 100, - stop_after_ocr: bool = False, - instance: str = "", - kind: str = Query(..., description="scheme or advisory"), -): - """Start workflows for all supported documents in a directory.""" - create_instance = _api._resolve_create_instance(user, instance) - directory = Path(data.directory) - if not directory.exists(): - raise HTTPException(404, f"Directory not found: {data.directory}") - - candidate_files = [p for p in directory.glob("*") if p.is_file() and p.suffix.lower() in _api.ALLOWED_EXTENSIONS] - if not candidate_files: - raise HTTPException(400, "No supported files found") - - results = [] - for pdf_path in candidate_files: - try: - result = await start_document_workflow( - request, - RegisterRequest(filepath=str(pdf_path)), - user, - auto_approve=auto_approve, - chunk_size=chunk_size, - chunk_overlap=chunk_overlap, - min_tokens=min_tokens, - stop_after_ocr=stop_after_ocr, - instance=create_instance, - kind=kind, - code="", - ) - results.append(result) - except Exception as e: - # Log full error, return sanitized message - logging.error(f"Batch workflow error for {pdf_path.name}: {str(e)}") - results.append(DocumentSummary( - document_id=hashlib.md5(str(pdf_path).encode()).hexdigest(), - workflow_id=_api.get_workflow_id(str(pdf_path)), - filename=pdf_path.name, - authoritative=False, - stage=DocumentStage.FAILED, - page_count=0, - chunk_count=0, - error_message="Failed to start workflow", - )) - - return results - - @router.get("", response_model=list[DocumentSummary]) async def list_documents( user: CurrentUser, diff --git a/pipeline/api/routers/health.py b/pipeline/api/routers/health.py new file mode 100644 index 0000000..4617700 --- /dev/null +++ b/pipeline/api/routers/health.py @@ -0,0 +1,19 @@ +"""Runtime health endpoint.""" + +import asyncio + +from fastapi import APIRouter + +from pipeline.api import api as _api + +router = APIRouter() + + +@router.get("/health") +async def health(): + """Health check.""" + return { + "status": "ok", + "temporal_connected": _api.temporal_client is not None, + "database": await asyncio.to_thread(_api.db.check_health), + } diff --git a/pipeline/api/routers/indexes.py b/pipeline/api/routers/indexes.py new file mode 100644 index 0000000..94d85dd --- /dev/null +++ b/pipeline/api/routers/indexes.py @@ -0,0 +1,242 @@ +"""Document and platform index inspection routes.""" + +from typing import Optional + +from fastapi import APIRouter, Header, HTTPException, Query + +from pipeline import db +from pipeline.auth.deps import RequireAdmin, RequireSearch +from pipeline.auth.tenancy import unrestricted +from pipeline.search.index_identity import projection_doc_id, projection_owner +from pipeline.api import api as _api +from pipeline.api.api import _instance_scope_for_user, _require_document_for_user + +router = APIRouter() + + +@router.get("/documents/{workflow_id}/qdrant") +async def get_document_qdrant_status( + workflow_id: str, + user: RequireSearch, + index_name: str | None = Query(None), +): + """Return vector-index status/chunks for a document.""" + from pipeline.vector_store import get_default_index_name, get_vector_store + + doc = _require_document_for_user(workflow_id, user) + + entries = doc.get("index_status") or {} + default_index = get_default_index_name() + if index_name: + resolved_index = default_index if index_name == "documents-index" else index_name + else: + preferred = entries.get("dev") or entries.get("prod") or {} + resolved_index = preferred.get("index_name") or default_index + matching = next((entry for entry in entries.values() if entry.get("index_name") == resolved_index), None) + if not matching and resolved_index != default_index: + raise HTTPException(404, "Index not found for this document") + index_doc_id = (matching or {}).get("doc_id") or projection_doc_id(doc) + store = get_vector_store() + backend = store.backend + + try: + raw_hits = store.list_by_doc_id(resolved_index, index_doc_id, limit=1000) + except Exception as exc: + _api.logger.warning("Document index diagnostics unavailable backend=%s error=%s", backend, type(exc).__name__) + raise HTTPException( + 503, + "Vector diagnostics are unavailable", + ) from exc + + hits = [] + for hit in raw_hits: + hit_doc_id = hit.get("doc_id") or "" + if hit_doc_id != index_doc_id or hit.get("instance") != doc.get("instance"): + continue + if projection_owner(hit_doc_id) != str(doc["id"]) and hit.get("workflow_id") != doc["workflow_id"]: + continue + normalized_hit = dict(hit) + chunk_num = normalized_hit.get("chunk_num") + normalized_hit.setdefault( + "_id", + f"{normalized_hit.get('doc_id', index_doc_id)}:{chunk_num if chunk_num is not None else 'unknown'}", + ) + normalized_hit.setdefault("chunk_number", chunk_num) + hits.append(normalized_hit) + db_chunks = db.get_chunks(workflow_id, include_excluded=True) + status = { + "workflow_id": workflow_id, + "index_name": resolved_index, + "backend": backend, + "index_doc_id": index_doc_id, + "doc_id": index_doc_id, + "sqlite_chunk_count": len([c for c in db_chunks if not c.get("is_excluded")]), + "indexed_chunk_count": len(hits), + "status": "indexed" if hits else "missing", + "hits": hits, + } + return status + + +@router.get("/documents/{workflow_id}/qdrant/chunks") +async def list_document_qdrant_chunks( + workflow_id: str, + user: RequireSearch, + index_name: str | None = Query(None), +): + result = await get_document_qdrant_status(workflow_id, user, index_name=index_name) + return result["hits"] + + +@router.get("/indexes/{index_name}/settings") +async def get_vector_index_settings(index_name: str, user: RequireSearch): + from pipeline.vector_store import get_vector_store + + if not unrestricted(user): + raise HTTPException(403, "Global index inspection requires platform access") + try: + return get_vector_store().get_settings(index_name) + except Exception as exc: + raise HTTPException(404, f"Index '{index_name}' settings unavailable: {exc}") from exc + + +@router.get("/indexes/{index_name}/stats") +async def get_vector_index_stats(index_name: str, user: RequireSearch): + from pipeline.vector_store import get_vector_store + + if not unrestricted(user): + raise HTTPException(403, "Global index inspection requires platform access") + try: + return get_vector_store().get_stats(index_name) + except Exception as exc: + raise HTTPException(404, f"Index '{index_name}' stats unavailable: {exc}") from exc + + +@router.get("/indexes/summary") +async def get_vector_indexes_summary( + user: RequireSearch, + x_include_disabled: Optional[str] = Header(None, alias="X-Include-Disabled"), +): + """Summarize index coverage from Postgres-backed index status plus live vector-store stats.""" + from pipeline.vector_store import get_vector_store + + include_disabled = x_include_disabled and x_include_disabled.lower() == "true" + scope = _instance_scope_for_user(user) + summaries = db.list_index_summaries( + include_disabled=include_disabled, + instances=scope, + ) + if not summaries: + return [] + + store = get_vector_store() + backend = store.backend + + results = [] + for summary in summaries: + live_stats = None + live_error = None + if scope is None: + try: + live_stats = store.get_stats(summary["index_name"]) + except Exception as exc: + _api.logger.warning("Index summary unavailable error=%s", type(exc).__name__) + live_error = "Vector index statistics are unavailable" + results.append({ + **summary, + "backend": backend, + "live_stats": live_stats, + "live_error": live_error, + }) + return results + +@router.get("/admin/index/schema") +async def get_vector_index_schema( + user: RequireAdmin, + index_name: str = Query("documents-index", description="Vector index / collection name"), +): + """Report the live vector index's field schema vs. the canonical passage schema.""" + from pipeline.search.passage_projection import _passage_schema_field_names + from pipeline.vector_store import get_default_index_name, get_vector_store + + resolved = index_name or get_default_index_name() + store = get_vector_store() + try: + index_settings = store.get_settings(resolved) + except Exception as exc: + raise HTTPException(404, f"Index '{resolved}' not found: {exc}") from exc + + field_names = sorted( + f.get("name") + for f in (index_settings.get("allFields") or []) + if isinstance(f, dict) and f.get("name") + ) + canonical_fields = sorted(_passage_schema_field_names()) + missing_fields = sorted(set(canonical_fields) - set(field_names)) + + return { + "index_name": resolved, + "backend": store.backend, + "fields": field_names, + "canonical_passage_fields": canonical_fields, + "missing_canonical_fields": missing_fields, + } + + +@router.post("/admin/index/create") +async def create_vector_index( + user: RequireAdmin, + index_name: str = Query("documents-index", description="Vector index / collection name"), + recreate_if_exists: bool = Query(False, description="If true, delete existing index and create with passage schema"), +): + """Create the vector index/collection — a cosine Qdrant collection sized from EMBEDDING_VECTOR_SIZE.""" + _ = user + from pipeline.network.release_policy import index_only_mode + from pipeline.vector_store import get_default_index_name, get_vector_store + + resolved = get_default_index_name() if index_only_mode() else index_name or get_default_index_name() + store = get_vector_store() + result = store.ensure_collection(resolved, recreate=recreate_if_exists) + return { + "index_name": resolved, + "backend": store.backend, + **result, + } + + +@router.get("/admin/ingest-info") +async def get_ingest_info(user: RequireAdmin): + """ + Return what the running container's ingest code would send to the vector index. + Use this to verify the API/worker image has the passage schema (text_for_embedding, etc.). + """ + from pipeline.search.passage_projection import _passage_schema_field_names, _prepare_records + + passage_fields = sorted(_passage_schema_field_names()) + has_text_for_embedding = "text_for_embedding" in set(passage_fields) + # One fake chunk to see exact record shape the worker would send + fake_chunk = { + "chunk_number": 0, + "original_text": "Sample text.", + "edited_text": None, + "is_excluded": False, + "token_count": 2, + "page_start": 1, + "page_end": 1, + } + sample_records = _prepare_records( + document_id="debug-document-id", + filename="debug.pdf", + chunks=[fake_chunk], + workflow_id="doc-debugsample12", + ) + sample_record_keys = sorted(sample_records[0].keys()) if sample_records else [] + return { + "passage_schema_fields": passage_fields, + "has_text_for_embedding": has_text_for_embedding, + "sample_record_keys": sample_record_keys, + "sample_has_passage_prefix": ( + sample_records[0].get("text_for_embedding", "").startswith("passage:") + if sample_records else False + ), + } diff --git a/pipeline/api/routers/operations.py b/pipeline/api/routers/operations.py new file mode 100644 index 0000000..6ea3355 --- /dev/null +++ b/pipeline/api/routers/operations.py @@ -0,0 +1,110 @@ +"""Operator queue, runs, and workflow stage routes.""" + +from typing import Optional + +from fastapi import APIRouter, Header, HTTPException, Query + +from pipeline import db +from pipeline.api.api import ( + _instance_scope_for_user, + _list_available_actions, + _require_document_for_user, +) +from pipeline.auth.deps import RequireSearch +from pipeline.auth.tenancy import unrestricted +from pipeline.network.release_policy import prod_stage_disabled +from pipeline.documents.models import ( + PIPELINE_STAGES, + PROD_ONLY_STAGES, + OperationQueueEntry, + OperationQueueResponse, +) +from pipeline.intake.source_intake import public_source_job + +router = APIRouter() + + +@router.get("/operations/queue", response_model=OperationQueueResponse) +async def get_operations_queue( + user: RequireSearch, + limit: int = Query(100, le=500), + offset: int = Query(0, ge=0), + x_include_disabled: Optional[str] = Header(None, alias="X-Include-Disabled"), +): + """Return documents that currently need operator or agent action. + + Results are limited to instances the caller can access (state users never + see portal ``bv`` or other states' documents). + """ + include_disabled = x_include_disabled and x_include_disabled.lower() == "true" + rows, total = db.list_operations_queue( + limit=limit, + offset=offset, + include_disabled=include_disabled, + instances=_instance_scope_for_user(user), + ) + items = [ + OperationQueueEntry( + workflow_id=row["workflow_id"], + filename=row["filename"], + stage=row["stage"], + job_id=row.get("job_id"), + job_type=row.get("job_type"), + job_status=row.get("job_status"), + started_at=row.get("started_at"), + error_message=row.get("error_message") or row.get("reindex_reason"), + available_actions=_list_available_actions(row, row), + ) + for row in rows + ] + return OperationQueueResponse(items=items, total=total) + + +@router.get("/runs") +async def list_runs( + user: RequireSearch, + limit: int = Query(100, le=500), + offset: int = Query(0, ge=0), + status: Optional[str] = None, +): + """List recent document jobs visible to the caller (tenant/role scoped). + + Restricted users only see jobs whose document ``instance`` is in their + allowed states. Super-admin / bypass users see all tenants. + """ + return [public_source_job(job) for job in db.list_runs( + limit=limit, + offset=offset, + status=status, + instances=_instance_scope_for_user(user), + )] + + + +@router.get("/runs/{job_id}") +async def get_run(job_id: str, user: RequireSearch): + """Get a specific document job/run (404 if missing or outside tenant scope).""" + run = db.get_document_job(job_id) + if not run: + raise HTTPException(404, f"Run not found: {job_id}") + # Enforce same instance scope as list: hide other tenants' runs as 404. + workflow_id = run.get("workflow_id") + if workflow_id: + _require_document_for_user(str(workflow_id), user) + elif not unrestricted(user): + raise HTTPException(404, f"Run not found: {job_id}") + return public_source_job(run) + +@router.get("/pipeline/stages") +async def get_pipeline_stages(user: RequireSearch): + """Get the pipeline stages for UI stepper display. + + The PROD stages are omitted when DISABLE_PROD_SETTING is on, so the stepper + shows the pipeline the documents will actually follow. + """ + prod_disabled = prod_stage_disabled() + return [ + {"id": stage[0], "label": stage[1], "description": stage[2]} + for stage in PIPELINE_STAGES + if not (prod_disabled and stage[0] in PROD_ONLY_STAGES) + ] diff --git a/pipeline/api/routers/pages.py b/pipeline/api/routers/pages.py new file mode 100644 index 0000000..f550967 --- /dev/null +++ b/pipeline/api/routers/pages.py @@ -0,0 +1,172 @@ +from fastapi import APIRouter, HTTPException +from fastapi import Path as PathParam + +from pipeline import db +from pipeline.auth.deps import RequireReview, RequireSearch +from pipeline.auth.permissions import Permission +from pipeline.documents.models import PageUpdate +from pipeline.api import api as _api + +router = APIRouter() + + +@router.get("/documents/{workflow_id}/pages") +async def list_pages(workflow_id: str, user: RequireSearch): + """Get all pages for a document. Postgres-first for speed.""" + # Enforce tenant scope up front (404 hides other tenants' documents). + _api._require_document_for_user(workflow_id, user) + return db.get_pages(workflow_id) + + +@router.get("/documents/{workflow_id}/pages/{page_num}") +async def get_page(workflow_id: str, user: RequireSearch, page_num: int = PathParam(..., ge=1, le=10000, description="Page number (1-indexed)")): + """Get a specific page. Postgres-first for speed.""" + _api._require_document_for_user(workflow_id, user) + page = db.get_page(workflow_id, page_num) + if page: + return page + + raise HTTPException(404, f"Page {page_num} not found") + + +@router.patch("/documents/{workflow_id}/pages/{page_num}") +async def update_page( + workflow_id: str, + data: PageUpdate, + user: RequireReview, + page_num: int = PathParam(..., ge=1, le=10000, description="Page number (1-indexed)"), +): + """Update a page (edit markdown, mark reviewed).""" + doc = _api._require_document_for_user(workflow_id, user, Permission.REVIEW) + old_page = db.get_page(workflow_id, page_num) + if not old_page: + raise HTTPException(404, f"Page {page_num} not found") + + try: + updated = db.update_page( + workflow_id, + page_num, + edited_markdown=data.edited_markdown, + is_reviewed=data.is_reviewed, + reviewer_notes=data.reviewer_notes, + edited_translation=data.edited_translation, + translation_reviewed=data.translation_reviewed, + translation_notes=data.translation_notes, + translation_language=data.translation_language, + ) + except db.PageTranslationMissing as exc: + raise HTTPException(409, "This translation language is unavailable or ambiguous. Reload the page.") from exc + if not updated: + raise HTTPException(404, f"Page {page_num} not found") + + # Log audits for changed fields + if data.edited_markdown is not None: + old_text = old_page.get("edited_markdown") or old_page.get("original_markdown", "") + _api._log_audit( + workflow_id=workflow_id, + action_type="page_edit", + entity_type="page", + entity_id=page_num, + field_name="edited_markdown", + old_value=old_text, + new_value=data.edited_markdown, + user=user, + ) + + if data.is_reviewed is not None: + _api._log_audit( + workflow_id=workflow_id, + action_type="page_edit", + entity_type="page", + entity_id=page_num, + field_name="is_reviewed", + old_value=old_page.get("is_reviewed", False), + new_value=data.is_reviewed, + user=user, + ) + + if data.reviewer_notes is not None: + _api._log_audit( + workflow_id=workflow_id, + action_type="page_edit", + entity_type="page", + entity_id=page_num, + field_name="reviewer_notes", + old_value=old_page.get("reviewer_notes"), + new_value=data.reviewer_notes, + user=user, + ) + + translation_language = data.translation_language or old_page.get("translation_target_language") + translations = old_page.get("translations") or {} + translation_key = ( + db.page_translation_key(translations, translation_language) if data.translation_language else translation_language + ) + translation = translations.get(translation_key, {}) + for field, old_value, new_value in ( + ("edited_translation", translation.get("edited_translation") or translation.get("translated_markdown"), data.edited_translation), + ("translation_reviewed", bool(translation.get("is_reviewed")), data.translation_reviewed), + ("translation_notes", translation.get("reviewer_notes"), data.translation_notes), + ): + if new_value is not None: + _api._log_audit( + workflow_id=workflow_id, + action_type="translation_edit", + entity_type="page", + entity_id=page_num, + field_name=field, + old_value=old_value, + new_value=new_value, + metadata={"language": translation_language} if translation_language else None, + user=user, + ) + + # Review flags/notes alone must not dirty the search index — only content edits. + content_changed = data.edited_markdown is not None or data.edited_translation is not None + if doc and content_changed and ( + doc.get("chunk_count", 0) > 0 + or doc.get("stage") in {"chunking", "chunk_review", "ready_for_ingestion", "ingesting", "approval_for_prod", "ingesting_prod", "completed"} + ): + _api._mark_reindex_required( + workflow_id, + "Page content changed after chunk generation; rechunk and reindex required", + metadata={"page_number": page_num}, + ) + + return db.get_page(workflow_id, page_num) + + +@router.post("/documents/{workflow_id}/pages/{page_num}/reset") +async def reset_page( + workflow_id: str, + user: RequireReview, + page_num: int = PathParam(..., ge=1, le=10000, description="Page number (1-indexed)"), +): + """Reset page to original OCR output.""" + doc = _api._require_document_for_user(workflow_id, user, Permission.REVIEW) + old_page = db.get_page(workflow_id, page_num) + if not old_page: + raise HTTPException(404, f"Page {page_num} not found") + db.reset_page(workflow_id, page_num) + + # Log reset action + _api._log_audit( + workflow_id=workflow_id, + action_type="page_reset", + entity_type="page", + entity_id=page_num, + field_name="edited_markdown", + old_value=old_page.get("edited_markdown") if old_page else None, + new_value=None, + metadata={"reset_to": "original_markdown"}, + user=user, + ) + + if doc and (doc.get("chunk_count", 0) > 0 or doc.get("stage") in {"chunking", "chunk_review", "ready_for_ingestion", "ingesting", "approval_for_prod", "ingesting_prod", "completed"}): + _api._mark_reindex_required( + workflow_id, + "Page reset after chunk generation; rechunk and reindex required", + metadata={"page_number": page_num}, + ) + + return db.get_page(workflow_id, page_num) diff --git a/pipeline/api/routers/providers.py b/pipeline/api/routers/providers.py new file mode 100644 index 0000000..5518e6c --- /dev/null +++ b/pipeline/api/routers/providers.py @@ -0,0 +1,158 @@ +"""Provider identity and scoped Keycloak membership management.""" + +import logging +from datetime import datetime, timezone +from typing import Annotated, Literal + +from fastapi import APIRouter, HTTPException, Path +from pydantic import BaseModel, StringConstraints + +from pipeline import db +from pipeline.intake import provider_sources, scheduled_intake +from pipeline.auth import keycloak_admin +from pipeline.auth.config import load_auth_config +from pipeline.auth.deps import CurrentUser, RequireAdmin +from pipeline.auth.groups import ROLE_STATE_ADMIN +from pipeline.auth.models import AuthUser +from pipeline.auth.permissions import Permission +from pipeline.auth.tenancy import allowed_instances, default_instance, portal_instance + +router = APIRouter(prefix="/providers", tags=["providers"]) +logger = logging.getLogger(__name__) + +ProviderCode = Annotated[str, Path(pattern=r"^[a-z0-9](?:[a-z0-9_-]{0,61}[a-z0-9])?$")] +UserId = Annotated[str, Path(pattern=r"^[A-Za-z0-9:_-]{1,200}$")] +ProviderRole = Literal["state_admin", "state_approver", "state_contributor", "state_view"] +Email = Annotated[str, StringConstraints(strip_whitespace=True, max_length=254, pattern=r"^[^@\s]+@[^@\s]+$")] + + +class OnboardProvider(BaseModel): + admin_email: Email + + +class ChangeProviderRole(BaseModel): + role: ProviderRole + + +class GrantProviderRole(ChangeProviderRole): + email: Email + + +def _require_live_management(user: AuthUser) -> None: + if user.token_disabled_mode or not load_auth_config().verify_live_groups: + raise HTTPException(503, "Provider management requires live Keycloak authorization") + + +def _require_provider_manager(user: AuthUser, provider: str) -> None: + _require_live_management(user) + if not user.is_superadmin and user.role_for_instance(provider) != ROLE_STATE_ADMIN: + raise HTTPException(403, "Provider administrator access is required") + + +def _set_member(provider: str, user_id: str, role: ProviderRole | None, actor: AuthUser) -> dict: + with db.get_connection() as conn: + # Serialize scoped updates to a user's group set across API replicas. + conn.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", ( + f"provider-membership:{provider}:{user_id}", + )) + result = keycloak_admin.set_provider_membership( + provider, user_id, role, allow_admin_changes=actor.is_superadmin, + ) + logger.info("provider_membership_changed provider=%s member=%s role=%s actor=%s", provider, user_id, role, actor.user_id) + return result + + +@router.get("") +def list_providers(user: CurrentUser): + _require_live_management(user) + scope = None if user.is_superadmin else sorted( + code for code, role in user.state_roles.items() if role == ROLE_STATE_ADMIN + ) + if scope == []: + raise HTTPException(403, "Provider administrator access is required") + return {"providers": db.list_provider_instances(scope)} + + +@router.get("/readable") +def list_readable_providers(user: CurrentUser): + """List provider choices for document and index reads.""" + if not user.has_permission(Permission.SEARCH): + raise HTTPException(403, "Search permission is required") + scope = allowed_instances(user) + if scope is not None: + return {"providers": sorted(scope)} + return {"providers": [row["provider"] for row in db.list_provider_instances(None)]} + + +@router.put("/{provider}") +def onboard_provider(provider: ProviderCode, body: OnboardProvider, user: RequireAdmin): + """Create a provider identity and appoint an existing Keycloak account as administrator.""" + _require_provider_manager(user, provider) + if provider in {default_instance(), portal_instance()}: + raise HTTPException(409, "This provider code is reserved") + if "_" in provider: + raise HTTPException(422, "Use letters, numbers and hyphens for a new provider code") + admin_id = keycloak_admin.find_verified_user_id(body.admin_email) + group = keycloak_admin.create_provider_group(provider) + instance_id = db.upsert_instance(group["path"], keycloak_group_id=group["id"]) + _set_member(provider, admin_id, ROLE_STATE_ADMIN, user) + return {"id": instance_id, "provider": provider, "group": group["path"], "admin_user_id": admin_id} + + +@router.get("/{provider}/members") +def list_members(provider: ProviderCode, user: CurrentUser): + _require_provider_manager(user, provider) + return {"provider": provider, "members": keycloak_admin.list_provider_memberships(provider)} + + +@router.get("/{provider}/scheduled-sources") +def list_scheduled_sources(provider: ProviderCode, user: CurrentUser): + _require_provider_manager(user, provider) + try: + sources = scheduled_intake.provider_pull_status(provider) + except (OSError, RecursionError, ValueError) as exc: + raise HTTPException(503, "Scheduled intake configuration is unavailable") from exc + return {"provider": provider, "checked_at": datetime.now(timezone.utc), "sources": sources} + + +@router.post("/{provider}/members") +def grant_member(provider: ProviderCode, body: GrantProviderRole, user: CurrentUser): + _require_provider_manager(user, provider) + if body.role == ROLE_STATE_ADMIN and not user.is_superadmin: + raise HTTPException(403, "Only a platform administrator can grant provider administrator access") + user_id = keycloak_admin.find_verified_user_id(body.email) + return _set_member(provider, user_id, body.role, user) + + +@router.put("/{provider}/members/{user_id}") +def change_member(provider: ProviderCode, user_id: UserId, body: ChangeProviderRole, user: CurrentUser): + _require_provider_manager(user, provider) + return _set_member(provider, user_id, body.role, user) + + +@router.delete("/{provider}/members/{user_id}") +def revoke_member(provider: ProviderCode, user_id: UserId, user: CurrentUser): + _require_provider_manager(user, provider) + return _set_member(provider, user_id, None, user) + + +@router.get("/{provider}/source-settings") +def get_source_settings(provider: ProviderCode, user: CurrentUser): + _require_provider_manager(user, provider) + try: + return provider_sources.public_settings(provider_sources.source_settings(provider)) + except (OSError, RecursionError, ValueError) as exc: + raise HTTPException(503, "Source settings are unavailable") from exc + + +@router.put("/{provider}/source-settings") +def update_source_settings(provider: ProviderCode, body: provider_sources.SourceSettingsUpdate, user: CurrentUser): + _require_provider_manager(user, provider) + try: + return provider_sources.save_settings(provider, body, user.user_id) + except FileExistsError as exc: + raise HTTPException(409, str(exc)) from exc + except LookupError as exc: + raise HTTPException(404, "Provider not found") from exc + except (OSError, RecursionError, ValueError) as exc: + raise HTTPException(422, "Invalid source settings. Check source hosts, schedules, templates and access policies.") from exc diff --git a/pipeline/api/routers/registry_proposals.py b/pipeline/api/routers/registry_proposals.py new file mode 100644 index 0000000..3066739 --- /dev/null +++ b/pipeline/api/routers/registry_proposals.py @@ -0,0 +1,156 @@ +"""Provider submissions and network-scoped decisions for local Registry proposals.""" + +from dataclasses import asdict +from typing import Annotated +from uuid import UUID + +from fastapi import APIRouter, HTTPException, Path, Query +from pydantic import BaseModel, ConfigDict, Field + +from pipeline import db +from pipeline.auth.deps import CurrentUser +from pipeline.auth.groups import ROLE_STATE_ADMIN, provider_group_root +from pipeline.network.network_connections import PublishConfigError +from pipeline.network.network_registry import RegistryLookupError +from pipeline.network.registry_operator import ( + AdmissionChange, + DecisionOutcome, + RegistryActor, + RegistryProposalConflict, + RegistryRecordVersion, + RouteChange, + decide_registry_proposal, + list_network_proposals, + list_provider_registry_networks, + read_registry_context, + read_registry_proposal, + submit_registry_proposal, +) +from pipeline.api.routers.providers import ProviderCode, _require_live_management, _require_provider_manager + +router = APIRouter(tags=["Registry proposals"]) +NetworkId = Annotated[str, Path(pattern=r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,252}$")] +Capability = Annotated[str | None, Query(max_length=128, pattern=r"^openagrinet:[A-Z][A-Za-z0-9]*$")] + + +class SubmitProposal(BaseModel): + model_config = ConfigDict(extra="forbid") + + request_id: UUID + expected_version: int = Field(ge=0) + change: AdmissionChange | RouteChange + observed_record: RegistryRecordVersion | None = None + reason: str = Field(min_length=1, max_length=1000) + + +class DecideProposal(BaseModel): + model_config = ConfigDict(extra="forbid") + + expected_version: int = Field(ge=1) + outcome: DecisionOutcome + reason: str = Field(min_length=1, max_length=1000) + + +def _operator_networks(user, network_id: str) -> frozenset[str]: + if f"/network-operators/{network_id}" in user.groups: + return frozenset({network_id}) + return frozenset() + + +def _proposal_response(operation, **arguments): + try: + result = operation(**arguments) + return [asdict(item) for item in result] if isinstance(result, list) else asdict(result) + except LookupError as exc: + raise HTTPException(404, str(exc)) from exc + except PermissionError as exc: + raise HTTPException(403, str(exc)) from exc + except RegistryProposalConflict as exc: + raise HTTPException(409, str(exc)) from exc + except RegistryLookupError as exc: + raise HTTPException(503, "The current Registry record could not be verified") from exc + except (ValueError, PublishConfigError) as exc: + raise HTTPException(422, str(exc)) from exc + + +def _managed_provider(user, provider: str) -> UUID: + _require_provider_manager(user, provider) + group_path = f"/{provider_group_root()}/{provider}" + with db.get_connection() as conn: + providers = conn.execute( + "SELECT id FROM instances WHERE lower(path) = lower(%s) LIMIT 2", + (group_path,), + ).fetchall() + if len(providers) != 1: + raise HTTPException(404, "Provider not found") + return providers[0]["id"] + + +@router.get("/providers/{provider}/registry-networks") +def list_registry_networks(provider: ProviderCode, user: CurrentUser): + provider_id = _managed_provider(user, provider) + return {"networks": _proposal_response( + list_provider_registry_networks, + actor=RegistryActor(user.user_id, managed_provider_ids=frozenset({provider_id})), + provider_id=provider_id, + )} + + +@router.get("/providers/{provider}/networks/{network_id}/registry-context") +def get_registry_context(provider: ProviderCode, network_id: NetworkId, user: CurrentUser, capability: Capability = None): + provider_id = _managed_provider(user, provider) + return _proposal_response( + read_registry_context, + actor=RegistryActor(user.user_id, managed_provider_ids=frozenset({provider_id})), + network_id=network_id, provider_id=provider_id, capability=capability, + ) + + +@router.post("/providers/{provider}/networks/{network_id}/registry-proposals", status_code=201) +def submit_proposal(provider: ProviderCode, network_id: NetworkId, body: SubmitProposal, user: CurrentUser): + provider_id = _managed_provider(user, provider) + return _proposal_response( + submit_registry_proposal, + actor=RegistryActor(user.user_id, managed_provider_ids=frozenset({provider_id})), + provider_id=provider_id, network_id=network_id, request_id=body.request_id, + expected_version=body.expected_version, change=body.change, observed_record=body.observed_record, + reason=body.reason, + ) + + +@router.get("/networks/{network_id}/registry-proposals") +def list_proposals(network_id: NetworkId, user: CurrentUser, pending_only: bool = True, limit: int = Query(50, ge=1, le=100)): + _require_live_management(user) + return _proposal_response( + list_network_proposals, + actor=RegistryActor(user.user_id, operator_network_ids=_operator_networks(user, network_id)), + network_id=network_id, pending_only=pending_only, limit=limit, + ) + + +@router.get("/networks/{network_id}/registry-proposals/{proposal_id}") +def read_proposal(network_id: NetworkId, proposal_id: UUID, user: CurrentUser): + _require_live_management(user) + managed = [name for name, role in user.state_roles.items() if role == ROLE_STATE_ADMIN] + with db.get_connection() as conn: + providers = conn.execute( + "SELECT id FROM instances WHERE %s OR lower(name) = ANY(%s)", (user.is_superadmin, managed), + ).fetchall() + return _proposal_response( + read_registry_proposal, + actor=RegistryActor( + user.user_id, managed_provider_ids=frozenset(row["id"] for row in providers), + operator_network_ids=_operator_networks(user, network_id), + ), + network_id=network_id, proposal_id=proposal_id, + ) + + +@router.put("/networks/{network_id}/registry-proposals/{proposal_id}/decision") +def decide_proposal(network_id: NetworkId, proposal_id: UUID, body: DecideProposal, user: CurrentUser): + _require_live_management(user) + return _proposal_response( + decide_registry_proposal, actor=RegistryActor(user.user_id, operator_network_ids=_operator_networks(user, network_id)), + network_id=network_id, proposal_id=proposal_id, expected_version=body.expected_version, + outcome=body.outcome, reason=body.reason, + ) diff --git a/pipeline/api/routers/search.py b/pipeline/api/routers/search.py new file mode 100644 index 0000000..1944ed0 --- /dev/null +++ b/pipeline/api/routers/search.py @@ -0,0 +1,459 @@ +"""Search API: ``POST /search`` and the ranking helpers it owns. + +First router split out of ``pipeline/api/api.py``. The vector store is +imported inside the route (not at module top) on purpose: tests patch +``pipeline.vector_store.get_vector_store`` and that only takes effect when the +name is looked up at call time. +""" + +from __future__ import annotations + +import asyncio +import logging +import math +import re +import unicodedata +from collections import Counter + +from fastapi import APIRouter, HTTPException + +from pipeline import db +from pipeline.documents import document_validity +from pipeline.auth.deps import RequireSearch +from pipeline.auth.tenancy import allowed_instances, assert_instance_access +from pipeline.metadata.crop_metadata import parse_crop_subject_ids +from pipeline.network.release_policy import managed_release_collections, managed_release_targets +from pipeline.metadata.knowledge_metadata import KnowledgeFilters +from pipeline.metadata.localization import localize_search_hit, normalize_language_tag +from pipeline.search.query_expansion import expand_query + +router = APIRouter() + +def _normalize_text(value: str) -> str: + return re.sub(r"\s+", " ", (value or "").strip().lower()) + + +def _tokenize(value: str) -> list[str]: + text = unicodedata.normalize("NFC", _normalize_text(value)) + return "".join( + character if unicodedata.category(character)[0] in "LMN" or character == "-" else " " + for character in text + ).split() + + +def _token_overlap_score(query: str, text: str) -> float: + query_tokens = set(_tokenize(query)) + text_tokens = set(_tokenize(text)) + if not query_tokens or not text_tokens: + return 0.0 + return len(query_tokens & text_tokens) / len(query_tokens) + + +def _metadata_blob(hit: dict) -> str: + metadata = hit.get("knowledge_metadata") or {} + labels = [*metadata.get("topics", []), *metadata.get("tags", []), *metadata.get("seasons", [])] + fields = ( + str(hit.get(key) or "") + for key in ( + "name", + "name_en", + "name_gu", + "filename", + "title_en", + "title_gu", + "category_tags", + "description", + "doc_short_description", + "doc_llm_description", + ) + ) + return " ".join([*fields, *labels]) + + +def _display_search_hit(hit: dict, requested_language: str | None) -> dict: + """Present localized document labels and the legacy search response aliases.""" + name = hit.get("name") + if not name: + return hit + metadata = hit.get("knowledge_metadata") or {} + variants = metadata.get("localized") or {} + selected = variants.get(hit.get("text_language"), {}) if requested_language else {} + result = dict(hit) + result.setdefault("name_en", name) + result.setdefault("name_gu", variants.get("gu", {}).get("title") or name) + result.setdefault("title_en", variants.get("en", {}).get("title") or "") + result.setdefault("title_gu", variants.get("gu", {}).get("title") or "") + result.setdefault("doc_language", hit.get("primary_language") or "") + result.setdefault("category_tags", ", ".join(metadata.get("tags", []))) + result.setdefault("doc_short_description", metadata.get("description") or "") + result.setdefault("doc_llm_description", "") + result.setdefault("ingestion_status", "") + result.setdefault("quality_score", 0.0) + result.setdefault("priority_rank", 0.0) + if selected.get("title"): + result["name"] = selected["title"] + if selected.get("description"): + result["description"] = selected["description"] + return result + + +def _rank_desc(values: list[float]) -> list[int]: + order = sorted(range(len(values)), key=lambda idx: values[idx], reverse=True) + ranks = [0] * len(values) + for pos, idx in enumerate(order, start=1): + ranks[idx] = pos + return ranks + + +def _bm25lite_scores(query: str, docs: list[str]) -> list[float]: + query_tokens = _tokenize(query) + if not query_tokens or not docs: + return [0.0] * len(docs) + + doc_tokens = [_tokenize(doc) for doc in docs] + avg_len = max(1.0, sum(len(tokens) for tokens in doc_tokens) / max(1, len(doc_tokens))) + df: Counter[str] = Counter() + for tokens in doc_tokens: + for token in set(tokens): + df[token] += 1 + + k1 = 1.2 + b = 0.75 + scores: list[float] = [] + for tokens in doc_tokens: + tf = Counter(tokens) + dl = len(tokens) + norm = k1 * (1 - b + b * dl / avg_len) + score = 0.0 + for term in query_tokens: + if term not in tf: + continue + idf = math.log(1.0 + (len(doc_tokens) - df[term] + 0.5) / (df[term] + 0.5)) + score += idf * (tf[term] * (k1 + 1.0)) / (tf[term] + norm) + scores.append(score) + return scores + + +def _rank_hybrid_candidates(query: str, hits: list[dict], alpha: float, method: str, rank_offset: int) -> list[dict]: + if len(hits) < 2 or alpha == 1: + return hits + lexical = _bm25lite_scores(query, [f"{hit.get('text', '')} {_metadata_blob(hit)}" for hit in hits]) + if not any(lexical): + return hits + dense = [float(hit.get("_score") or 0) for hit in hits] + if method == "normalize_linear": + dense_min, dense_max = min(dense), max(dense) + dense = [(score - dense_min) / (dense_max - dense_min or 1) for score in dense] + lexical = [score / max(lexical) for score in lexical] + else: + dense = [1 / (rank_offset + rank) for rank in _rank_desc(dense)] + lexical = [1 / (rank_offset + rank) if score else 0 for rank, score in zip(_rank_desc(lexical), lexical)] + scored = [ + {**hit, "_hybrid_score": alpha * semantic + (1 - alpha) * keyword} + for hit, semantic, keyword in zip(hits, dense, lexical) + ] + return sorted(scored, key=lambda hit: hit["_hybrid_score"], reverse=True) + + +def _rerank_hits(query: str, hits: list[dict], rerank_mode: str) -> list[dict]: + mode = (rerank_mode or "none").strip().lower() + if mode in {"", "none"} or not hits: + return hits + + raw_scores = [float(hit.get("_score", hit.get("score", 0.0)) or 0.0) for hit in hits] + min_score = min(raw_scores) + max_score = max(raw_scores) + denom = (max_score - min_score) if max_score > min_score else 1.0 + semantic_scores = [(raw - min_score) / denom for raw in raw_scores] + text_scores = [_token_overlap_score(query, str(hit.get("text") or "")) for hit in hits] + meta_scores = [_token_overlap_score(query, _metadata_blob(hit)) for hit in hits] + + rescored: list[dict] = [] + if mode == "bm25lite": + docs = [f"{str(hit.get('text') or '')} {_metadata_blob(hit)}".strip() for hit in hits] + bm_scores = _bm25lite_scores(query, docs) + bm_min = min(bm_scores) if bm_scores else 0.0 + bm_max = max(bm_scores) if bm_scores else 1.0 + bm_denom = (bm_max - bm_min) if bm_max > bm_min else 1.0 + bm_norm = [(score - bm_min) / bm_denom for score in bm_scores] + for hit, semantic, bm25, meta in zip(hits, semantic_scores, bm_norm, meta_scores): + enriched = dict(hit) + enriched["_rerank_score"] = (0.50 * semantic) + (0.40 * bm25) + (0.10 * meta) + (-0.10 if bool(hit.get("is_reference", False)) else 0.0) + rescored.append(enriched) + elif mode in {"rrf-lite", "rrf_lite", "rrf"}: + sem_rank = _rank_desc(semantic_scores) + text_rank = _rank_desc(text_scores) + meta_rank = _rank_desc(meta_scores) + k = 30 + for idx, hit in enumerate(hits): + enriched = dict(hit) + enriched["_rerank_score"] = (1.0 / (k + sem_rank[idx])) + (1.0 / (k + text_rank[idx])) + (1.0 / (k + meta_rank[idx])) + rescored.append(enriched) + else: + for hit, semantic, text_score, meta in zip(hits, semantic_scores, text_scores, meta_scores): + enriched = dict(hit) + enriched["_rerank_score"] = (0.60 * semantic) + (0.30 * max(text_score, meta)) + (0.10 * meta) + rescored.append(enriched) + + rescored.sort(key=lambda hit: float(hit.get("_rerank_score", 0.0)), reverse=True) + return rescored + + + + +def _requested_group(payload: dict) -> str | None: + raw = payload.get("group") + if raw is not None and not isinstance(raw, str): + raise HTTPException(400, "group must be a string") + return (raw or "").strip().lower() or None + + +def _requested_provider_id(payload: dict) -> str | None: + raw = payload.get("provider_id") + if raw is not None and not isinstance(raw, str): + raise HTTPException(400, "provider_id must be a string") + return (raw or "").strip().lower() or None + + +@router.get("/search/collections") +async def search_collections(user: RequireSearch): + from pipeline.vector_store import get_default_index_name + + try: + scopes = managed_release_collections() + advisories = managed_release_targets("advisory") + schemes = managed_release_targets("scheme") + except ValueError as exc: + raise HTTPException(503, "Network collections are misconfigured") from exc + if not scopes: + return [{"label": "Documents", "name": get_default_index_name()}] + labels = {collection: "Published schemes" for collection in schemes.values()} + labels.update({collection: "Published advisories" for collection in advisories.values()}) + return [ + {"label": labels.get(collection, "Published documents"), "name": collection, "network_id": network_id} + for collection, network_id in scopes.items() + ] + + +@router.post("/search/", include_in_schema=False) +@router.post("/search") +async def run_search(payload: dict, user: RequireSearch): + """Search passages visible to the caller, optionally within one provider.""" + from pipeline.vector_store import get_default_index_name, get_vector_store + + settings = db.get_search_settings() + from pipeline.network.release_policy import index_only_mode + + index_name = get_default_index_name() + if not index_only_mode(): + index_name = payload.get("index_name") or settings.get("indexName") or index_name + raw_query = payload.get("query") + if raw_query is not None and not isinstance(raw_query, str): + raise HTTPException(422, "query must be text") + query = (raw_query or "").strip() + if not query: + raise HTTPException(400, "query is required") + if len(query) > 4096: + raise HTTPException(422, "query must be at most 4096 characters") + + requested_language = payload.get("language") + language = normalize_language_tag(requested_language) + if requested_language is not None and language is None: + raise HTTPException(422, "language must be a BCP 47 tag") + strict_language = payload.get("strict_language", False) + if type(strict_language) is not bool or (strict_language and not language): + raise HTTPException(422, "strict_language requires a boolean and a language when true") + + logging.info( + "search request user_id=%s index=%s query_length=%s top_k=%s", + user.user_id, + index_name, + len(query), + payload.get("top_k"), + ) + + search_mode = (payload.get("search_mode") or settings.get("searchMethod") or "HYBRID").upper() + top_k = max(1, min(int(payload.get("top_k") or settings.get("limit") or 12), 50)) + candidate_multiplier = max(1, int(payload.get("candidate_multiplier") or settings.get("candidateMultiplier") or 10)) + requested_candidate_cap = payload.get("candidate_cap") + if requested_candidate_cap is None: + candidate_cap = min(max(top_k * candidate_multiplier, top_k), int(settings.get("candidateCap") or 120)) + else: + candidate_cap = int(requested_candidate_cap) + candidate_cap = max(top_k, min(candidate_cap, 200)) + max_chunks_per_doc = max(1, int(payload.get("max_chunks_per_doc") or settings.get("maxChunksPerDoc") or 2)) + use_e5_prefix = bool(payload.get("use_e5_prefix", settings.get("useE5Prefix", True))) + exclude_reference = bool(payload.get("exclude_reference", settings.get("excludeReference", True))) + alpha = float(payload.get("hybrid_alpha", settings.get("alpha", 0.6))) + if not 0 <= alpha <= 1: + raise HTTPException(422, "hybrid_alpha must be between 0 and 1") + ranking_method = payload.get("ranking_method") or settings.get("rankingMethod") or "rrf" + ef_search = int(payload.get("ef_search") or settings.get("efSearch") or 256) + query_expansion_profile = payload.get("query_expansion_profile") or settings.get("queryExpansionProfile") or "none" + rerank_mode = payload.get("rerank_mode") or settings.get("rerankMode") or "none" + hybrid_rrf_k = max(1, min(int(payload.get("hybrid_rrf_k") or settings.get("hybridRrfK") or 60), 1000)) + # Validity is on by default: a search serving a farmer must not answer from + # a document that has expired or has not started. Both overrides are + # operator tools - `include_expired` to inspect what an expired document + # still holds, `valid_on` to ask what search would have returned on another + # day - and the response echoes which day was actually applied. + include_expired = bool(payload.get("include_expired", False)) + valid_on = (payload.get("valid_on") or "").strip() or None + if valid_on: + try: + # Reassigned, not just checked: parse_day returns the canonical + # zero-padded form, and the vector store's date range rejects the + # `2026-9-3` spelling that strptime happily accepts. + valid_on = document_validity.parse_day(valid_on, "valid_on") + except document_validity.DocumentValidityError as exc: + raise HTTPException(400, str(exc)) from None + group = _requested_group(payload) + permitted = allowed_instances(user) + if permitted is not None and not permitted: + raise HTTPException(403, "No provider access") + if group is not None: + assert_instance_access(user, group) + scope_filter = {"allowed_instances": sorted(permitted)} if group is None and permitted is not None else {} + provider_id = _requested_provider_id(payload) + selected_workflow = payload.get("document_workflow_id") + if selected_workflow is not None and ( + not isinstance(selected_workflow, str) or not 1 <= len(selected_workflow) <= 256 + ): + raise HTTPException(422, "document_workflow_id must be a workflow identifier") + try: + crop_subject_ids = parse_crop_subject_ids(payload.get("crop_subject_ids")) + except ValueError as exc: + raise HTTPException(400, "crop_subject_ids must contain at most 32 absolute URI strings") from exc + try: + applicability = KnowledgeFilters.model_validate(payload.get("applicability", {})).qdrant_values() + except ValueError as exc: + raise HTTPException(422, "Invalid agriculture applicability filters") from exc + try: + expanded_query = expand_query(query, query_expansion_profile) + except (TypeError, ValueError) as exc: + raise HTTPException(503, "Search expansion is misconfigured") from exc + # Qdrant embeddings apply E5 prefixes internally; don't pre-prefix here. + search_query = expanded_query + + store = get_vector_store() + backend = store.backend + # HYBRID on Qdrant currently maps to dense tensor search (lexical is separate mode). + store_mode = "TENSOR" if backend == "qdrant" and search_mode == "HYBRID" else search_mode + + try: + release_network = managed_release_collections().get(index_name) + exclusions = await asyncio.to_thread(db.inactive_network_revision_ids, release_network, index_name) if release_network else [] + search_arguments = dict( + name=index_name, + query=search_query, + search_mode=store_mode, + exclude_reference=exclude_reference, + use_e5_prefix=use_e5_prefix, + hybrid_alpha=alpha, + ef_search=ef_search, + apply_validity=not include_expired, + valid_on=valid_on, + instance=group, + provider_id=provider_id, + **({"strict_language": language} if strict_language else {}), + **scope_filter, + **({"document_workflow_id": selected_workflow} if selected_workflow else {}), + **({"crop_subject_ids": crop_subject_ids} if crop_subject_ids else {}), + **({"applicability": applicability} if applicability else {}), + **({"excluded_revision_ids": exclusions} if exclusions else {}), + ) + result = await asyncio.to_thread(store.search, limit=candidate_cap, **search_arguments) + candidates = result.get("hits", []) + validate_hits = db.visible_document_hits if index_only_mode() else db.visible_release_hits + hits = await asyncio.to_thread(validate_hits, candidates) + if (index_only_mode() or release_network) and len(candidates) == candidate_cap and len(hits) < top_k: + result = await asyncio.to_thread(store.search, limit=min(candidate_cap * 3, 600), **search_arguments) + hits = await asyncio.to_thread(validate_hits, result.get("hits", [])) + except Exception as error: + logging.warning("vector search failed backend=%s error_type=%s", backend, type(error).__name__) + raise HTTPException(503, "Vector search is temporarily unavailable") from error + + if language: + hits = [ + localized for hit in hits + if (localized := localize_search_hit(hit, language, strict=strict_language)) is not None + ] + hits = [_display_search_hit(hit, language) for hit in hits] + if search_mode == "HYBRID" and backend == "qdrant": + hits = _rank_hybrid_candidates(query, hits, alpha, ranking_method, hybrid_rrf_k) + hits = _rerank_hits(query, hits, rerank_mode) + final_hits = [] + per_doc_counts: dict[str, int] = {} + for hit in hits: + doc_key = hit.get("doc_id") or hit.get("filename") or "__unknown__" + if per_doc_counts.get(doc_key, 0) >= max_chunks_per_doc: + continue + per_doc_counts[doc_key] = per_doc_counts.get(doc_key, 0) + 1 + final_hits.append(hit) + if len(final_hits) >= top_k: + break + + scheme_candidates = {} + query_tokens = set(_tokenize(query)) + for hit in hits: + code = hit.get("scheme_code") + workflow_id = hit.get("workflow_id") + name = hit.get("scheme_name") or hit.get("name") + aliases = hit.get("scheme_aliases") or [] + if not all(isinstance(item, str) and item for item in (code, workflow_id, name)): + continue + labels = [code, name, *(alias for alias in aliases if isinstance(alias, str))] + # A broad topical query can return many related schemes. Ask the user + # to choose when their words refer to more than one actual scheme label. + if any((tokens := set(_tokenize(label))) and tokens <= query_tokens for label in labels): + scheme_candidates.setdefault(workflow_id, { + "workflow_id": workflow_id, "name": name, "code": code, "provider": hit.get("instance"), + }) + if len(scheme_candidates) == 5: + break + choices = list(scheme_candidates.values()) if len(scheme_candidates) > 1 and not selected_workflow else [] + + logging.info( + "search response user_id=%s index=%s candidate_count=%s final_count=%s", + user.user_id, + index_name, + len(hits), + len(final_hits), + ) + + return { + "effective_config": { + "index_name": index_name, + "backend": backend, + "query": query, + "search_mode": search_mode, + "store_search_mode": store_mode, + "top_k": top_k, + "candidate_cap": candidate_cap, + "candidate_multiplier": candidate_multiplier, + "max_chunks_per_doc": max_chunks_per_doc, + "use_e5_prefix": use_e5_prefix, + "exclude_reference": exclude_reference, + "hybrid_alpha": alpha, + "ranking_method": ranking_method, + "hybrid_rrf_k": hybrid_rrf_k, + "ef_search": ef_search, + "query_expansion_profile": query_expansion_profile, + "query_expansion_applied": expanded_query != query, + "rerank_mode": rerank_mode, + "apply_validity": not include_expired, + "valid_on": result.get("valid_on"), + "group": group, + "provider_id": provider_id, + "crop_subject_ids": crop_subject_ids, + "applicability": applicability, + "language": language, + "strict_language": strict_language, + "filter_string": None, + }, + "candidate_count": len(hits), + "final_count": len(final_hits), + "hits": final_hits, + "scheme_choices": choices, + "raw_hits": hits if payload.get("include_raw_hits") else None, + } diff --git a/pipeline/api/routers/search_settings.py b/pipeline/api/routers/search_settings.py new file mode 100644 index 0000000..a53484b --- /dev/null +++ b/pipeline/api/routers/search_settings.py @@ -0,0 +1,104 @@ +from fastapi import APIRouter, HTTPException, Query + +from pipeline import db +from pipeline.auth.deps import RequireAdmin, RequireSearch +from pipeline.documents.models import SearchSettings, SearchSettingsUpdate, SettingsAuditResponse + +router = APIRouter() + + +@router.get("/settings/search", response_model=SearchSettings) +async def get_search_settings(user: RequireSearch): + """ + Get current search settings. + + Read-only search configuration. Gated with ``RequireSearch`` (not + ``RequireAdmin``) because any search-capable user needs these defaults + (index name, method, limits) to drive the search UI; mutating the + settings (PUT / reset) still requires ``RequireAdmin``. + + Returns the current search configuration including: + - searchMethod: TENSOR, LEXICAL, or HYBRID + - limit: Number of results to return + - alpha: Balance between lexical (0) and semantic (1) for hybrid search + - rankingMethod: rrf or normalize_linear for hybrid search + - showHighlights: Whether to show highlighted matches + - efSearch: HNSW search accuracy parameter + """ + return db.get_search_settings() + + +@router.put("/settings/search", response_model=SearchSettings) +async def update_search_settings_endpoint(settings: SearchSettingsUpdate, user: RequireAdmin): + """ + Update search settings. + + Only provided fields will be updated. Changes are logged to the audit trail. + """ + _ = user + # Convert to dict, excluding None values + updates = {k: v for k, v in settings.model_dump().items() if v is not None} + + if not updates: + raise HTTPException(400, "No settings provided to update") + + try: + return db.update_search_settings(updates) + except ValueError as exc: + raise HTTPException(400, str(exc)) from exc + + +@router.get("/settings/search/audit", response_model=SettingsAuditResponse) +async def get_search_settings_audit( + user: RequireAdmin, + limit: int = Query(50, le=200), + offset: int = 0 +): + """ + Get audit trail for search settings changes. + + Shows all historical changes to search settings with old/new values. + """ + logs = db.get_settings_audit_logs(limit=limit, offset=offset) + total = db.get_settings_audit_count() + + return SettingsAuditResponse( + logs=logs, + total=total, + limit=limit, + offset=offset + ) + + +@router.post("/settings/search/reset", response_model=SearchSettings) +async def reset_search_settings(user: RequireAdmin): + """ + Reset search settings to defaults. + + Resets all search settings to their default values: + - searchMethod: HYBRID + - limit: 10 + - alpha: 0.7 + - rankingMethod: rrf + - showHighlights: true + - efSearch: 256 + """ + _ = user + defaults = { + "searchMethod": "HYBRID", + "limit": 12, + "alpha": 0.6, + "rankingMethod": "rrf", + "showHighlights": True, + "efSearch": 256, + "indexName": "documents-index", + "candidateCap": 120, + "candidateMultiplier": 10, + "maxChunksPerDoc": 2, + "useE5Prefix": True, + "excludeReference": True, + "queryExpansionProfile": "none", + "rerankMode": "none", + "hybridRrfK": 60, + } + return db.update_search_settings(defaults) diff --git a/pipeline/api/routers/upload.py b/pipeline/api/routers/upload.py new file mode 100644 index 0000000..fecdcb9 --- /dev/null +++ b/pipeline/api/routers/upload.py @@ -0,0 +1,596 @@ +from __future__ import annotations + +import hashlib +from io import BytesIO +from pathlib import Path +from typing import Optional +from uuid import uuid4 + +from fastapi import APIRouter, File, Form, HTTPException, Query, Request, UploadFile +from pydantic import BaseModel + +from pipeline import db +from pipeline.documents import document_validity +from pipeline.documents import display_name as display_name_rules +from pipeline.auth.deps import RequireUpload +from pipeline.auth.models import AuthUser +from pipeline.auth.permissions import Permission +from pipeline.auth.tenancy import ( + assert_document_instance_access, + configured_portal_instance, + normalize_instance, + portal_instance, +) +from pipeline.metadata.crop_metadata import parse_crop_subjects +from pipeline.documents.document_dispatch import start_registered_document +from pipeline.documents.document_formats import TEXT_UPLOAD_TYPES, allowed_file_extensions +from pipeline.documents.document_kinds import configured_document_kinds +from pipeline.intake.source_fetch import MAX_HTML_BYTES, extract_text_source +from pipeline.network.release_policy import ( + index_only_mode, + intake_index_options, + prod_stage_disabled, + upload_release_target, +) +from pipeline.metadata.knowledge_metadata import KnowledgeMetadata, parse_knowledge_metadata +from pipeline.documents.models import DocumentStage, DocumentSummary +from pipeline.object_storage import ObjectReference +from pipeline.ocr.service import allowed_ocr_providers, default_ocr_profile +from pipeline.metadata.schema_packs import validate_schema_selection +from pipeline.intake.source_intake import attach_source_response +from pipeline.intake.webhook_mapping import parse_canonical_document +from pipeline.temporal.workflows import DocumentPipelineWorkflow +from pipeline.api import api as _api + +router = APIRouter() +MAX_BATCH_FILES = 10 +MAX_UPLOAD_BYTES = 100 * 1024 * 1024 +PENDING_START = "The file was saved, but processing could not be confirmed. Retry starting the document." + + +class BatchUploadItem(BaseModel): + filename: str + status: int + document: DocumentSummary | None = None + detail: str | None = None + + +class BatchUploadResult(BaseModel): + items: list[BatchUploadItem] + + +@router.get("/upload/options") +def upload_options(user: RequireUpload): + try: + return { + "parser_providers": allowed_ocr_providers(), + "default_parser": default_ocr_profile(), + "document_kinds": configured_document_kinds(), + "file_extensions": sorted(allowed_file_extensions()), + } + except ValueError as exc: + _api.logger.error("Invalid document processing configuration: %s", exc) + raise HTTPException(503, "Parser selection is misconfigured") from exc + + +@router.get("/upload/providers") +def upload_providers(user: RequireUpload): + """List the provider choices available to file and URL intake.""" + providers = { + normalize_instance(code) + for code in user.instances + if code.strip() and user.has_permission_for_instance(Permission.UPLOAD, code) + } + preferred = None + if user.is_superadmin or user.token_disabled_mode: + providers.update(row["provider"] for row in db.list_provider_instances(None)) + preferred = portal_instance() if user.token_disabled_mode else configured_portal_instance() + if preferred: + providers.add(preferred) + return {"providers": sorted(providers), "default_provider": preferred} + + +@router.get("/upload/scheme-matches") +@_api.limiter.limit("30/minute") +def find_similar_schemes( + request: Request, user: RequireUpload, instance: str, name: str = Query(min_length=2, max_length=300), +): + """Find candidates without changing or merging a document.""" + from difflib import SequenceMatcher + from unicodedata import normalize + + owner = _api._resolve_create_instance(user, instance) + if not user.has_permission_for_instance(Permission.UPLOAD, owner): + raise HTTPException(403, "Upload permission is required for this provider") + + def key(value: str) -> str: + return " ".join("".join(c if c.isalnum() else " " for c in normalize("NFKC", value).casefold()).split()) + + needle = key(name) + if not needle: + return {"matches": []} + with db.get_connection() as conn: + candidates = conn.execute( + "SELECT d.id::text, d.name, d.code, d.aliases FROM documents d " + "JOIN instances i ON d.provider_id = i.id " + "WHERE lower(i.name) = %s AND d.kind = 'scheme' AND NOT d.is_disabled " + "ORDER BY (lower(d.code) = lower(%s) OR EXISTS (" + "SELECT 1 FROM unnest(d.aliases) AS alias WHERE lower(alias) = lower(%s))) DESC, " + "similarity(lower(d.name), lower(%s)) DESC, d.code LIMIT 200", + (owner, name, name, name), + ).fetchall() + matches = [] + for candidate in candidates: + labels = (candidate["name"], candidate["code"], *(candidate["aliases"] or [])) + score = max(SequenceMatcher(None, needle, key(label)).ratio() for label in labels if label) + if score >= 0.68: + matches.append({"id": candidate["id"], "name": candidate["name"], "code": candidate["code"], "score": score}) + matches = sorted(matches, key=lambda row: (-row["score"], row["code"]))[:8] + if len(matches) > 1: + from math import sqrt + + from pipeline.vector_store.embeddings import embed_passages, embed_query + + def cosine(left, right): + length = sqrt(sum(x * x for x in left) * sum(x * x for x in right)) + return sum(a * b for a, b in zip(left, right)) / length if length else 0.0 + + try: + query = embed_query(name) + passages = embed_passages([f'{row["name"]} ({row["code"]})' for row in matches]) + for row, vector in zip(matches, passages): + row["semantic_score"] = cosine(query, vector) + matches.sort(key=lambda row: (row["score"] < 1, -row.get("semantic_score", 0), -row["score"])) + except Exception: + _api.logger.warning("Scheme similarity model unavailable; using name and code matches") + return {"matches": matches} + + +@router.post("/upload", response_model=DocumentSummary) +@_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) +async def upload_and_process( + request: Request, # Required for rate limiting + user: RequireUpload, + file: UploadFile = File(...), + auto_approve: bool = False, + chunk_size: int = 450, + chunk_overlap: int = 128, + min_tokens: int = 100, + index_name: str = "documents-index", + stop_after_ocr: bool = False, + instance: str = "", + kind: str = Query(..., description="A configured document kind"), + code: str = "", + display_name: Optional[str] = Form(None), + crop_subjects: Optional[str] = Form(None, description="JSON array of document-wide crop subject references"), + knowledge_metadata: Optional[str] = Form(None, description="JSON object with agriculture metadata"), + valid_from: Optional[str] = Form(None), + valid_to: Optional[str] = Form(None), + parser_provider: Optional[str] = Form(None), +): + """ + Upload a supported file and start processing workflow. + + The pipeline stores the original file in object storage and queues processing. + Validates file extensions and checks the header of PDF uploads. + Rate limited to 10 requests/minute per IP. + Requires permission: upload (no-op while AUTH_DISABLED=true). + """ + return await _process_upload( + user, file, instance, kind, code, display_name, crop_subjects, + auto_approve=auto_approve, chunk_size=chunk_size, chunk_overlap=chunk_overlap, + min_tokens=min_tokens, index_name=index_name, stop_after_ocr=stop_after_ocr, + knowledge_metadata=knowledge_metadata, valid_from=valid_from, valid_to=valid_to, + parser_provider=parser_provider, + ) + + +@router.post("/upload/batch", response_model=BatchUploadResult, status_code=207) +@_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) +async def upload_batch( + request: Request, + user: RequireUpload, + files: list[UploadFile] = File(...), + instance: str = "", + kind: str = Query(..., description="Knowledge kind shared by this batch"), + parser_provider: Optional[str] = Form(None), + crop_subjects: Optional[str] = Form(None), + knowledge_metadata: Optional[str] = Form(None), + valid_from: Optional[str] = Form(None), + valid_to: Optional[str] = Form(None), +): + """Submit independent supported files in one request. Return the outcome for each file.""" + if not 1 <= len(files) <= MAX_BATCH_FILES: + raise HTTPException(413, f"Send between 1 and {MAX_BATCH_FILES} files per batch") + if any(file.size is None for file in files): + raise HTTPException(400, "Could not determine the size of every file") + if sum(file.size for file in files) > MAX_UPLOAD_BYTES: + raise HTTPException(413, "Batch files must total at most 100 MiB") + instance = _api._resolve_create_instance(user, instance) + if not user.has_permission_for_instance(Permission.UPLOAD, instance): + raise HTTPException(403, f"No upload permission in instance: {instance}") + items = [] + for file in files: + filename = file.filename or "" + try: + document = await _process_upload( + user, file, instance, kind, "", None, crop_subjects, + parser_provider=parser_provider, knowledge_metadata=knowledge_metadata, + valid_from=valid_from, valid_to=valid_to, + ) + except HTTPException as exc: + items.append(BatchUploadItem(filename=filename, status=exc.status_code, detail=str(exc.detail))) + except Exception: + _api.logger.exception("Batch file intake failed") + items.append(BatchUploadItem(filename=filename, status=500, detail="Could not start this document")) + else: + items.append(BatchUploadItem( + filename=filename, status=503 if "retry_upload_start" in document.available_actions else 202, + document=document, detail=document.error_message, + )) + return BatchUploadResult(items=items) + + +async def _start_registered_upload(job: dict) -> None: + await start_registered_document(_api.temporal_client, _api.TASK_QUEUE, job) + + +@router.post("/upload/{workflow_id}/start", status_code=202) +@_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) +async def retry_registered_upload(request: Request, workflow_id: str, user: RequireUpload): + doc = _api._require_document_for_user(workflow_id, user) + if doc.get("is_disabled"): + raise HTTPException(409, "Restore the document before retrying its upload") + if not user.has_permission_for_instance(Permission.UPLOAD, doc["instance"]): + raise HTTPException(403, "Upload permission is required for this provider") + job = db.get_latest_document_job(workflow_id) + if not job or job["temporal_workflow_id"] != workflow_id or not job["config"].get("upload_start_args"): + raise HTTPException(404, "Registered upload not found") + if job["status"] == "start_pending": + try: + await _start_registered_upload(job) + except Exception as exc: + _api.logger.exception("Could not confirm registered upload start workflow_id=%s", workflow_id) + raise HTTPException(503, PENDING_START) from exc + return {"workflow_id": workflow_id, "status": "started"} + + +async def _process_upload( + user: AuthUser, + file: UploadFile, + instance: str, + kind: str, + code: str, + display_name: str | None, + crop_subjects: str | None, + *, + auto_approve: bool = False, + chunk_size: int = 450, + chunk_overlap: int = 128, + min_tokens: int = 100, + index_name: str = "documents-index", + stop_after_ocr: bool = False, + knowledge_metadata: str | None = None, + valid_from: str | None = None, + valid_to: str | None = None, + parser_provider: str | None = None, + source_url: str | None = None, + source_capture: dict | None = None, +) -> DocumentSummary: + if parser_provider is not None: + parser_provider = parser_provider.strip().lower() + try: + allowed_parsers = allowed_ocr_providers() + except ValueError as exc: + _api.logger.error("Invalid OCR provider configuration: %s", exc) + raise HTTPException(503, "Parser selection is misconfigured") from exc + if parser_provider not in allowed_parsers: + raise HTTPException(422, "Parser provider is not enabled for this deployment") + try: + crops = parse_crop_subjects(crop_subjects) + except ValueError as exc: + raise HTTPException(422, "Invalid crop_subjects: provide up to 32 unique subject references with a URI or a code/name descriptor") from exc + try: + metadata = parse_knowledge_metadata(knowledge_metadata) + if metadata and "schema_pack" in metadata: + with db.get_connection() as conn: + validate_schema_selection(conn, metadata) + except ValueError as exc: + raise HTTPException(422, "Invalid knowledge metadata or schema attributes") from exc + create_instance = _api._resolve_create_instance(user, instance) + doc_kind, doc_code, replace_doc = _api._resolve_document_identity( + user, create_instance, kind, code, file.filename or "" + ) + index_name, auto_approve = intake_index_options(index_name, auto_approve) + try: + release_target = upload_release_target(doc_kind, provider=create_instance) if not stop_after_ocr else None + except ValueError as exc: + _api.logger.error("Invalid automatic upload release configuration: %s", exc) + raise HTTPException(503, "Automatic upload release is misconfigured") from exc + if release_target: + release_network_id, index_name = release_target + auto_approve = True + else: + release_network_id = None + filename = file.filename or "" + if ( + not filename or len(filename) > 255 or "/" in filename or chr(92) in filename + or any(ord(character) < 32 or ord(character) == 127 for character in filename) + ): + raise HTTPException(400, "Invalid filename") + suffix = Path(filename).suffix.lower() + if suffix not in allowed_file_extensions(): + raise HTTPException(400, f"Unsupported file type: {suffix}") + try: + display_name = display_name_rules.normalize(display_name) + except ValueError as exc: + raise HTTPException(400, str(exc)) from exc + + # Read file content + limit = MAX_HTML_BYTES if suffix in TEXT_UPLOAD_TYPES else MAX_UPLOAD_BYTES + content = await file.read(limit + 1) + file_size = len(content) + if file_size > limit: + raise HTTPException(413, f"Files of this type must be at most {limit // (1024 * 1024)} MiB") + attribution_url = source_url + if suffix in TEXT_UPLOAD_TYPES: + try: + if suffix == ".json": + submission = parse_canonical_document(content) + if "kind" in submission.model_fields_set and submission.kind != doc_kind: + raise ValueError("The JSON document kind must match the upload kind") + if display_name is None: + display_name = display_name_rules.normalize(submission.title) + if crops is None and submission.crop_subjects: + crops = [crop.model_dump(exclude_none=True) for crop in submission.crop_subjects] + submitted_metadata = ( + submission.knowledge_metadata if metadata is None + else submission.metadata_with_translation_labels(KnowledgeMetadata.model_validate(metadata)) + ) + if submitted_metadata is not None: + metadata = submitted_metadata.storage_value() + with db.get_connection() as conn: + validate_schema_selection(conn, metadata) + valid_from = valid_from if valid_from is not None else submission.valid_from + valid_to = valid_to if valid_to is not None else submission.valid_to + attribution_url = submission.source_url + else: + extract_text_source(content, TEXT_UPLOAD_TYPES[suffix], filename) + except (ValueError, RecursionError) as exc: + raise HTTPException(422, "Invalid text file or canonical JSON document") from exc + + try: + if replace_doc: + start = valid_from if valid_from is not None else replace_doc.get("valid_from") + end = valid_to if valid_to is not None else replace_doc.get("valid_to") + else: + start, end = valid_from, valid_to + validity = document_validity.parse_period(start, end) + except document_validity.DocumentValidityError as exc: + raise HTTPException(422, str(exc)) from exc + # Validate PDF magic bytes (%PDF-) only for PDF uploads + if suffix == ".pdf": + pdf_magic = b"%PDF-" + if len(content) < 5 or content[:5] != pdf_magic: + raise HTTPException(400, "Invalid PDF file: file does not have valid PDF header") + + file_hash = hashlib.md5(content).hexdigest() + document_id = file_hash + canonical_document_id = file_hash + legacy_path = f"minio://{_api.OBJECT_STORE_BUCKET}/{file_hash}/{filename}" + workflow_id = _api.get_workflow_id(f"{legacy_path}:{doc_code}" if source_url else legacy_path) + + # Reuse only when Postgres still tracks this workflow. + # If Postgres was purged, avoid returning stale Temporal state and create a fresh run ID. + existing_doc = db.get_document(workflow_id) + if existing_doc and normalize_instance(existing_doc.get("instance")) == create_instance: + existing_doc = assert_document_instance_access(user, existing_doc) + if crops is not None and crops != (existing_doc.get("crop_subjects") or []): + raise HTTPException(409, "This upload already exists with different crop metadata; use a new source file to reupload") + if metadata is not None and metadata != (existing_doc.get("knowledge_metadata") or {}): + raise HTTPException(409, "This upload already exists with different agriculture metadata") + try: + handle = _api.temporal_client.get_workflow_handle(workflow_id) + state = await handle.query("get_state") + if state: + return DocumentSummary( + document_id=document_id, + canonical_document_id=canonical_document_id, + workflow_id=workflow_id, + filename=file.filename, + source_filename=file.filename, + source_file_fingerprint=file_hash, + authoritative=bool(existing_doc.get("source_manifest_name")) if existing_doc else False, + instance=normalize_instance(existing_doc.get("instance")), + stage=DocumentStage(state.get("stage", "registered")), + page_count=state.get("page_count", 0), + chunk_count=state.get("chunk_count", 0), + error_message=state.get("error_message"), + crop_subjects=existing_doc.get("crop_subjects") or [], + knowledge_metadata=existing_doc.get("knowledge_metadata") or {}, + ) + except HTTPException: + raise + except Exception: + pass + upload_id = uuid4().hex + workflow_id = f"{workflow_id}-rerun-{upload_id}" + object_name = f"uploads/{upload_id}/{filename}" + ref = ObjectReference(_api.OBJECT_STORE_BUCKET, object_name) + storage_uri = ref.to_uri() + content_type = "application/pdf" if suffix == ".pdf" else "application/octet-stream" + _api.object_store.put(ref, BytesIO(content), file_size, content_type) + + start_args = [ + document_id, filename, storage_uri, chunk_size, chunk_overlap, + min_tokens, index_name, auto_approve, stop_after_ocr, prod_stage_disabled(), + ] + if release_network_id or parser_provider or index_only_mode(): + start_args.append(release_network_id) + if parser_provider or index_only_mode(): + start_args.append(parser_provider) + if index_only_mode(): + start_args.append(True) + registered_start = index_only_mode() or not replace_doc or bool(release_network_id) + if not registered_start: + # The legacy replacement mutates existing content. Keep its start-before-reset order. + await _api.temporal_client.start_workflow( + DocumentPipelineWorkflow.run, args=start_args, id=workflow_id, task_queue=_api.TASK_QUEUE, + ) + + # Save to Postgres for visibility during processing + actor = _api._actor_from_user(user) + with db.get_connection() as conn: + if replace_doc: + document_uuid = replace_doc["id"] + try: + db.reset_document_for_reupload( + document_uuid, + filename=file.filename, + filepath=storage_uri, + fingerprint=file_hash, + crop_subjects=crops, + knowledge_metadata=metadata, + expected_job_id=replace_doc.get("latest_job_id") if release_network_id else None, + _conn=conn, + ) + except db.DocumentReplacementConflict as exc: + raise HTTPException(409, str(exc)) from exc + if valid_from is not None or valid_to is not None: + db.update_document_fields( + document_uuid, valid_from=validity.start_date, valid_to=validity.end_date, _conn=conn, + ) + else: + try: + document_uuid = db.upsert_document( + workflow_id=workflow_id, + document_id=document_id, + canonical_document_id=canonical_document_id, + filename=file.filename, + source_filename=file.filename, + source_file_fingerprint=file_hash, + filepath=storage_uri, + stage="registered", + stop_after_ocr=stop_after_ocr, + instance=create_instance, + uploaded_by_user_id=actor.get("user_id") or None, + uploaded_by_username=actor.get("username") or None, + uploaded_by_email=actor.get("email") or None, + uploaded_by_roles=actor.get("roles_csv") or None, + valid_from=validity.start_date, + valid_to=validity.end_date, + with_workflow=False, + kind=doc_kind, + code=doc_code, + # v2: the contributor's Display Name is documents.name. + name=display_name or _api._document_title(file.filename or ""), + crop_subjects=crops, + knowledge_metadata=metadata, + _conn=conn, + ) + except db.DuplicateDocumentCode as exc: + raise HTTPException(409, str(exc)) from exc + job_id = db.create_document_job( + workflow_id=document_uuid, + job_type="ocr_only" if stop_after_ocr else "pipeline", + temporal_workflow_id=workflow_id, + status="start_pending" if registered_start else "running", + current_stage="registered", + config={ + **({"upload_start_args": start_args} if registered_start else {}), + "auto_approve": auto_approve, + "release_network_id": release_network_id, + **({"source_url": attribution_url} if attribution_url else {}), + **({"parser_provider": parser_provider} if parser_provider else {}), + "chunk_size": chunk_size, + "chunk_overlap": chunk_overlap, + "min_tokens": min_tokens, + "index_name": index_name, + "stop_after_ocr": stop_after_ocr, + "uploaded_by": actor, + }, + actor=actor, + _conn=conn, + ) + if source_capture is not None: + attach_source_response(conn, job_id, source_capture) + db.add_document_artifact( + workflow_id=workflow_id, + job_id=job_id, + artifact_type="original_upload", + stage="registered", + storage_uri=storage_uri, + mime_type=content_type, + filename=file.filename, + size_bytes=file_size, + metadata={"uploaded_via": "url_intake" if source_url else "upload_endpoint", "uploaded_by": actor}, + _conn=conn, + ) + if suffix in TEXT_UPLOAD_TYPES: + source_type, canonical_input_type = "document", "structured_text" + elif suffix in {".csv", ".xlsx"}: + source_type = canonical_input_type = "spreadsheet" + else: + source_type, canonical_input_type = "document", "pdf" + db.update_document_fields( + workflow_id, + source_type=source_type, + canonical_input_type=canonical_input_type, + stop_after_ocr=1 if stop_after_ocr else 0, + _conn=conn, + ) + + _api._log_audit( + workflow_id=workflow_id, + action_type="document_upload", + entity_type="document", + field_name="registered", + new_value=file.filename, + metadata={ + "source": "url_intake" if source_url else "upload_endpoint", + "instance": create_instance, + "kind": doc_kind, + "code": doc_code, + "replaced_existing": bool(replace_doc), + "size_bytes": file_size, + "auto_approve": auto_approve, + "release_network_id": release_network_id, + }, + user=user, + _conn=conn, + ) + + start_error = None + if registered_start: + try: + await _start_registered_upload({ + "id": job_id, + "config": {"upload_start_args": start_args}, + "temporal_workflow_id": workflow_id, + }) + except Exception: + _api.logger.exception("Could not confirm registered upload start workflow_id=%s", workflow_id) + start_error = PENDING_START + + return DocumentSummary( + id=document_uuid, + document_id=document_id, + canonical_document_id=canonical_document_id, + workflow_id=workflow_id, + filename=file.filename, + source_filename=file.filename, + source_file_fingerprint=file_hash, + authoritative=False, + instance=create_instance, + stage=DocumentStage.REGISTERED, + page_count=0, + chunk_count=0, + uploaded_by_user_id=actor.get("user_id") or None, + uploaded_by_username=actor.get("username") or None, + uploaded_by_email=actor.get("email") or None, + uploaded_by_roles=actor.get("roles") or [], + error_message=start_error, + available_actions=["retry_upload_start"] if start_error else [], + crop_subjects=crops if crops is not None else (replace_doc or {}).get("crop_subjects") or [], + knowledge_metadata=metadata if metadata is not None else (replace_doc or {}).get("knowledge_metadata") or {}, + ) diff --git a/pipeline/api/routers/user_admin.py b/pipeline/api/routers/user_admin.py new file mode 100644 index 0000000..9e4e1db --- /dev/null +++ b/pipeline/api/routers/user_admin.py @@ -0,0 +1,93 @@ +import asyncio + +from fastapi import APIRouter, Query +from pydantic import BaseModel, Field + +from pipeline.auth.deps import RequireManageUsers +from pipeline.auth.keycloak_admin import ( + list_access_options, + list_realm_users, + provision_user, + set_user_access, +) + +router = APIRouter() + + +class ProvisionUserRequest(BaseModel): + """Fields a super admin must fill to create / update access.""" + + email: str = Field(..., description="Google SSO email (required)") + first_name: str = "" + last_name: str = "" + username: str = "" + access_type: str = Field( + ..., + description="super_admin | global_viewer | state", + ) + state: str = Field("", description="State code e.g. MH (required for state access)") + role: str = Field( + "", + description="state_admin | state_view (required for state access; aliases: admin, view, contributor, reviewer)", + ) + enabled: bool = True + + +class SetUserAccessRequest(BaseModel): + """Change an existing user's role in the selected scope.""" + + access_type: str = Field(..., description="super_admin | global_viewer | state") + state: str = Field("", description="State/centre code e.g. MH (required when access_type=state)") + role: str = Field( + "", + description=( + "state_admin | state_approver | state_contributor | state_view " + "(required when access_type=state)" + ), + ) + + +@router.get("/admin/access-options") +async def admin_access_options(user: RequireManageUsers): + """Form options + required fields for the Users admin UI.""" + return list_access_options() + + +@router.get("/admin/users") +async def admin_list_users( + user: RequireManageUsers, + search: str = "", + max_results: int = Query(100, ge=1, le=200), +): + """List Keycloak users with groups/access labels (no delete).""" + return await asyncio.to_thread(list_realm_users, search=search, max_results=max_results) + + +@router.post("/admin/users") +async def admin_provision_user(data: ProvisionUserRequest, user: RequireManageUsers): + """Create or update a Keycloak user and assign group/role. Returns share text.""" + return await asyncio.to_thread( + provision_user, + email=data.email, + first_name=data.first_name, + last_name=data.last_name, + username=data.username or None, + access_type=data.access_type, + state=data.state or None, + role=data.role or None, + enabled=data.enabled, + ) + + +@router.put("/admin/users/{user_id}/access") +async def admin_set_user_access( + user_id: str, data: SetUserAccessRequest, user: RequireManageUsers +): + """Change an existing user's role in one scope (super admin only).""" + return await asyncio.to_thread( + set_user_access, + user_id=user_id, + access_type=data.access_type, + state=data.state or None, + role=data.role or None, + ) diff --git a/pipeline/api/routers/webhook_sources.py b/pipeline/api/routers/webhook_sources.py new file mode 100644 index 0000000..3169be6 --- /dev/null +++ b/pipeline/api/routers/webhook_sources.py @@ -0,0 +1,364 @@ +"""Provider webhook source policies and authenticated structured document intake.""" + +import asyncio +from dataclasses import replace +from hashlib import sha256 +from io import BytesIO +from typing import Annotated + +from fastapi import APIRouter, Depends, Header, HTTPException, Path, Query, Request, Response +from fastapi.responses import StreamingResponse +from psycopg.types.json import Jsonb +from pydantic import BaseModel, ConfigDict, Field, SecretStr, ValidationError + +from pipeline import db +from pipeline.api import api as _api +from pipeline.api.routers.providers import ProviderCode, _require_provider_manager +from pipeline.auth import keycloak_admin +from pipeline.auth.config import load_auth_config +from pipeline.auth.deps import CurrentUser, _bearer_token +from pipeline.auth.jwt import decode_verified_claims +from pipeline.auth.oidc import issue_service_token +from pipeline.network.release_policy import index_only_mode, source_release_target +from pipeline.object_storage import ObjectReference, parse_object_uri +from pipeline.intake.source_intake import ( + IntakeConflict, + canonical_json_sha256, + get_source_receipt, + register_source_document, + start_source_document, +) +from pipeline.intake.webhook_mapping import ( + CANONICAL_SUBMISSION, + SOURCE_MAPPING, + CanonicalMapping, + SourceMapping, + decode_source_json, + load_intake_template, + map_source_submission, +) + +router = APIRouter(prefix="/webhook/sources", tags=["webhooks"]) +knowledge_router = APIRouter(prefix="/knowledge/intake", tags=["knowledge intake"]) + +_MAX_BODY_BYTES = 2 * 1024 * 1024 +SourceKey = Annotated[str, Path(pattern=r"^[a-z0-9](?:[a-z0-9_-]{0,61}[a-z0-9])?$")] +_KEY_PATTERN = r"^[A-Za-z0-9._:-]{1,128}$" +IdempotencyKey = Annotated[str, Header(alias="Idempotency-Key", pattern=_KEY_PATTERN)] +DeliveryKey = Annotated[str, Query(pattern=_KEY_PATTERN)] + + +class SourcePolicy(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + expected_version: int = Field(default=0, ge=0) + service_client: str = Field(min_length=1, max_length=256) + service_subject: str | None = Field(default=None, min_length=1, max_length=256) + mapping: SourceMapping | None = None + template: str | None = Field(default=None, pattern=r"^[a-z][a-z0-9-]{0,63}$") + fixture: dict | list + enabled: bool = True + auto_release: bool = False + release_network_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,252}$") + + +class MappingPreview(BaseModel): + model_config = ConfigDict(extra="forbid") + + mapping: SourceMapping + fixture: dict | list + + +def _resolve_mapping(policy: SourcePolicy, provider: str | None = None) -> SourceMapping: + if policy.template is None: + return policy.mapping or CanonicalMapping() + if policy.mapping is not None: + raise ValueError("Choose a template or supply a mapping") + return load_intake_template(policy.template, provider=provider) + + +@knowledge_router.get("/schema") +def knowledge_intake_schema(user: CurrentUser): + return CANONICAL_SUBMISSION.json_schema() + + +def _select_source(conn, provider: str, source: str, *, lock: bool = False) -> dict | None: + return conn.execute( + "SELECT s.id::text, s.enabled, s.auto_release, s.release_network_id, s.service_client, s.service_subject, " + "m.version, m.id::text AS mapping_id, m.mapping, m.mapping_sha256 FROM webhook_sources s " + "JOIN instances i ON i.id = s.instance_id JOIN webhook_mapping_versions m " + "ON m.source_id = s.id AND m.version = s.mapping_version " + "WHERE lower(i.name) = %s AND s.source_key = %s" + (" FOR UPDATE OF s" if lock else ""), + (provider, source), + ).fetchone() + + +@knowledge_router.get("/{provider}") +def list_source_policies(provider: ProviderCode, user: CurrentUser): + _require_provider_manager(user, provider) + with db.get_connection() as conn: + sources = conn.execute( + "SELECT s.source_key, s.enabled, s.mapping_version AS version FROM webhook_sources s " + "JOIN instances i ON i.id = s.instance_id WHERE lower(i.name) = %s ORDER BY s.source_key", + (provider,), + ).fetchall() + return {"sources": sources, "index_only": index_only_mode()} + + +@knowledge_router.post("/{provider}/mapping-preview") +@_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) +async def preview_mapping(provider: ProviderCode, request: Request, user: CurrentUser): + _require_provider_manager(user, provider) + try: + _, payload = await _read_payload(request) + preview = MappingPreview.model_validate(payload) + documents = await asyncio.to_thread(map_source_submission, preview.fixture, preview.mapping) + except (ValueError, RecursionError) as exc: + if isinstance(exc, ValidationError): + issue = exc.errors(include_input=False, include_url=False, include_context=False)[0] + field = ".".join(map(str, issue["loc"])) + raise HTTPException(422, f"{field or 'Mapping output'}: {issue['msg']}") from exc + raise HTTPException(422, "The mapping could not transform this sample.") from exc + return {"documents": [document.model_dump(mode="json", exclude_none=True) for _, document in documents]} + + +@knowledge_router.get("/{provider}/{source}/deliveries") +def source_deliveries(provider: ProviderCode, source: SourceKey, user: CurrentUser): + _require_provider_manager(user, provider) + with db.get_connection() as conn: + policy = _select_source(conn, provider, source) + if not policy: + raise HTTPException(404, "Webhook source not found") + deliveries = conn.execute( + "SELECT d.idempotency_key, d.created_at, m.version, d.raw_storage_uri IS NOT NULL AS has_original, " + "COALESCE((SELECT jsonb_agg(jsonb_build_object('workflow_id', w.temporal_workflow_id, " + "'title', r.normalized_document->>'title', 'status', w.status, 'stage', w.current_stage) " + "ORDER BY receipt.ordinality) " + "FROM unnest(d.receipt_ids) WITH ORDINALITY receipt(id, ordinality) " + "JOIN source_intake_receipts r ON r.id = receipt.id " + "JOIN workflows w ON w.id = r.job_id), '[]'::jsonb) AS documents " + "FROM webhook_deliveries d JOIN webhook_mapping_versions m ON m.id = d.mapping_version_id " + "WHERE d.source_id = %s::uuid ORDER BY d.created_at DESC, d.idempotency_key DESC LIMIT 25", + (policy["id"],), + ).fetchall() + return {"deliveries": deliveries} + + +@knowledge_router.get("/{provider}/{source}/deliveries/original") +def download_delivery(provider: ProviderCode, source: SourceKey, delivery_key: DeliveryKey, user: CurrentUser): + _require_provider_manager(user, provider) + with db.get_connection() as conn: + row = conn.execute( + "SELECT d.raw_storage_uri FROM webhook_deliveries d " + "JOIN webhook_sources s ON s.id = d.source_id JOIN instances i ON i.id = s.instance_id " + "WHERE lower(i.name) = %s AND s.source_key = %s AND d.idempotency_key = %s", + (provider, source, delivery_key), + ).fetchone() + if not row or not row["raw_storage_uri"]: + raise HTTPException(404, "Original delivery body is unavailable") + response = _api.object_store.open(parse_object_uri(row["raw_storage_uri"])) + return StreamingResponse(response, media_type="application/json", headers={ + "Content-Disposition": 'attachment; filename="webhook-original.json"', + "Cache-Control": "no-store", + "X-Content-Type-Options": "nosniff", + }) + + +@knowledge_router.get("/{provider}/{source}") +@router.get("/{provider}/{source}") +def get_source_policy(provider: ProviderCode, source: SourceKey, user: CurrentUser): + _require_provider_manager(user, provider) + with db.get_connection() as conn: + policy = _select_source(conn, provider, source) + if not policy: + raise HTTPException(404, "Webhook source not found") + return policy + + +@knowledge_router.put("/{provider}/{source}") +@router.put("/{provider}/{source}") +def put_source_policy(provider: ProviderCode, source: SourceKey, body: SourcePolicy, user: CurrentUser): + _require_provider_manager(user, provider) + if body.release_network_id is not None and not body.auto_release: + raise HTTPException(422, "A release network requires automatic release") + try: + selected_mapping = _resolve_mapping(body, provider) + examples = map_source_submission(body.fixture, selected_mapping) + if not examples: + raise ValueError("Fixture must map at least one document") + if body.enabled and body.auto_release and not index_only_mode(): + for kind in {document.kind for _, document in examples}: + if not source_release_target(kind, body.release_network_id, provider=provider): + raise ValueError("The source release network is unavailable for this document kind") + except (OSError, TypeError, ValueError) as exc: + raise HTTPException(422, "The mapping fixture or release policy is invalid") from exc + + mapping = selected_mapping.model_dump(mode="json", exclude_none=True) + mapping_hash = canonical_json_sha256(mapping) + service_subject = keycloak_admin.find_webhook_service_subject(body.service_client) if body.enabled else None + if service_subject and body.service_subject and service_subject != body.service_subject: + raise HTTPException(422, "The supplied subject does not belong to this Keycloak client") + with db.get_connection() as conn: + instance = conn.execute("SELECT id FROM instances WHERE lower(name) = %s", (provider,)).fetchone() + if not instance: + raise HTTPException(404, "Provider not found") + lock = canonical_json_sha256([provider, source]) + conn.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", (lock,)) + prior = _select_source(conn, provider, source, lock=True) + if (prior["version"] if prior else 0) != body.expected_version: + raise HTTPException(409, "Webhook source policy changed. Load the current version and retry.") + if service_subject is None: + if not prior or body.service_client != prior["service_client"]: + raise HTTPException(422, "An active webhook source is required before disabling it") + service_subject = prior["service_subject"] + version = body.expected_version + 1 + source_id = conn.execute( + "INSERT INTO webhook_sources (instance_id, source_key, service_client, service_subject, enabled, " + "auto_release, release_network_id, mapping_version) VALUES (%s, %s, %s, %s, %s, %s, %s, %s) " + "ON CONFLICT (instance_id, source_key) DO UPDATE SET service_client = excluded.service_client, " + "service_subject = excluded.service_subject, enabled = excluded.enabled, " + "auto_release = excluded.auto_release, release_network_id = excluded.release_network_id, " + "mapping_version = excluded.mapping_version RETURNING id", + (instance["id"], source, body.service_client, service_subject, body.enabled, + body.auto_release, body.release_network_id, version), + ).fetchone()["id"] + conn.execute( + "INSERT INTO webhook_mapping_versions (source_id, version, mapping, mapping_sha256, created_by) " + "VALUES (%s, %s, %s, %s, %s)", (source_id, version, Jsonb(mapping), mapping_hash, user.user_id), + ) + return _select_source(conn, provider, source) + + +def _verified_service_sender(token: str) -> tuple[str, str]: + config = load_auth_config() + if not config.keycloak_audience or not config.keycloak_issuer or not config.keycloak_jwks_url: + raise HTTPException(503, "Webhook Keycloak token validation is not configured") + claims = decode_verified_claims(token, replace(config, disabled=False)) + client, subject = claims.get("azp"), claims.get("sub") + if claims.get("typ") != "Bearer" or not isinstance(client, str) or not isinstance(subject, str): + raise HTTPException(403, "Webhook sender is not authorized") + return client, subject + + +async def require_webhook_sender(request: Request) -> tuple[str, str]: + return await asyncio.to_thread(_verified_service_sender, _bearer_token(request)) + + +class ServiceTokenRequest(BaseModel): + client_secret: Annotated[SecretStr, Field(min_length=1, max_length=4096)] + + +@knowledge_router.post("/{provider}/{source}/token") +@_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) +def create_source_token( + request: Request, response: Response, provider: ProviderCode, source: SourceKey, + body: ServiceTokenRequest, user: CurrentUser, +): + _require_provider_manager(user, provider) + with db.get_connection() as conn: + policy = _select_source(conn, provider, source) + if not policy or not policy["enabled"]: + raise HTTPException(404, "Active webhook source not found") + token = issue_service_token(policy["service_client"], body.client_secret.get_secret_value()) + if _verified_service_sender(token["access_token"]) != (policy["service_client"], policy["service_subject"]): + raise HTTPException(403, "Webhook sender is not authorized") + response.headers["Cache-Control"] = "no-store" + return token + + +async def _read_payload(request: Request) -> tuple[bytes, object]: + if request.headers.get("content-type", "").split(";", 1)[0].strip().lower() != "application/json": + raise HTTPException(415, "Webhook events require application/json") + data = bytearray() + async for chunk in request.stream(): + if len(data) + len(chunk) > _MAX_BODY_BYTES: + raise HTTPException(413, "Webhook body exceeds 2 MiB") + data.extend(chunk) + try: + return bytes(data), decode_source_json(data) + except (UnicodeError, ValueError, RecursionError) as exc: + raise HTTPException(422, "Invalid webhook JSON") from exc + + +def _register_delivery( + provider: str, source: str, key: str, payload: object, raw: bytes, sender: tuple[str, str], +) -> tuple[str, list[dict]]: + with db.get_connection() as conn: + policy = _select_source(conn, provider, source, lock=True) + if not policy or not policy["enabled"] or sender != (policy["service_client"], policy["service_subject"]): + raise HTTPException(403, "Webhook sender is not authorized") + digest = canonical_json_sha256(payload) + prior = conn.execute( + "SELECT body_sha256, receipt_ids, mapping_version_id::text FROM webhook_deliveries " + "WHERE source_id = %s::uuid AND idempotency_key = %s", (policy["id"], key), + ).fetchone() + if prior: + if prior["body_sha256"] != digest: + raise IntakeConflict("Idempotency key was already used for different content") + receipts = [get_source_receipt(conn, receipt_id) for receipt_id in prior["receipt_ids"]] + if any(receipt is None for receipt in receipts): + raise IntakeConflict("The original intake document has been removed") + return prior["mapping_version_id"], receipts + + if canonical_json_sha256(policy["mapping"]) != policy["mapping_sha256"]: + raise HTTPException(503, "Webhook mapping integrity check failed") + mapping = SOURCE_MAPPING.validate_python(policy["mapping"]) + receipts = [] + targets: dict[str, tuple[str, str]] = {} + release = policy["auto_release"] and not index_only_mode() + network = policy["release_network_id"] if release else None + for record, document in map_source_submission(payload, mapping): + if release and document.kind not in targets: + try: + target = source_release_target(document.kind, network, provider=provider) + except ValueError as exc: + raise HTTPException(503, "Automatic release is unavailable for this source") from exc + if not target: + raise HTTPException(503, "Automatic release is unavailable for this source") + targets[document.kind] = target + receipts.append(register_source_document( + conn, instance=provider, source_kind="webhook", source_id=policy["id"], + idempotency_key=canonical_json_sha256([key, document.external_id]), raw_payload=record, + document=document, release_target=targets.get(document.kind), mapping_version_id=policy["mapping_id"], + explicit_release_network=network is not None, + )) + object_key = canonical_json_sha256([key, sha256(raw).hexdigest()]) + original = ObjectReference(_api.OBJECT_STORE_BUCKET, f"webhook-deliveries/{policy['id']}/{object_key}.json") + try: + _api.object_store.put(original, BytesIO(raw), len(raw), "application/json") + except Exception as exc: + raise HTTPException(503, "The original webhook body could not be stored. Retry with the same key.") from exc + conn.execute( + "INSERT INTO webhook_deliveries (source_id, idempotency_key, body_sha256, raw_payload, " + "mapping_version_id, receipt_ids, raw_storage_uri) " + "VALUES (%s::uuid, %s, %s, %s, %s::uuid, %s::uuid[], %s)", + (policy["id"], key, digest, Jsonb(payload), policy["mapping_id"], + [receipt["id"] for receipt in receipts], original.to_uri()), + ) + return policy["mapping_id"], receipts + + +@knowledge_router.post("/{provider}/{source}/submissions", status_code=202) +@router.post("/{provider}/{source}/events", status_code=202) +@_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) +async def receive_event( + request: Request, provider: ProviderCode, source: SourceKey, idempotency_key: IdempotencyKey, + sender: Annotated[tuple[str, str], Depends(require_webhook_sender)], +): + raw, payload = await _read_payload(request) + try: + mapping_id, receipts = await asyncio.to_thread(_register_delivery, provider, source, idempotency_key, payload, raw, sender) + except IntakeConflict as exc: + raise HTTPException(409, str(exc)) from exc + except (ValueError, RecursionError) as exc: + raise HTTPException(422, "The webhook payload does not match the source mapping") from exc + + items = [] + for receipt in receipts: + started = await start_source_document(receipt) + items.append({ + "receipt_id": receipt["id"], "document_id": receipt["document_id"], + "workflow_id": receipt["workflow_id"], "status": "accepted" if started else "start_pending", + "source_url": receipt["source_url"], + }) + return {"mapping_version_id": mapping_id, "items": items} diff --git a/pipeline/api/runtime.py b/pipeline/api/runtime.py new file mode 100644 index 0000000..ff68668 --- /dev/null +++ b/pipeline/api/runtime.py @@ -0,0 +1,70 @@ +"""API service startup and shutdown, using the composition module's dependencies.""" + +import asyncio +import logging +import os +from contextlib import asynccontextmanager + +from fastapi import FastAPI + +from . import api as _api + + +@asynccontextmanager +async def lifespan(app: FastAPI): + """Initialize Temporal and object storage clients on startup.""" + _api.configure_logs() + try: + await _api._start_services() + except Exception: + logging.exception("api startup failed") + _api.reset_logs() + raise + yield + logging.info("api stopping") + _api.reset_logs() + + +async def _start_services(): + _api.validate_network_adapters() + _api.installed_formats() + + auth_cfg = _api.load_auth_config() + _api.validate_auth_config(auth_cfg) + if auth_cfg.disabled: + logging.warning( + "WARNING: AUTH_DISABLED=true — every caller is treated as synthetic " + "master_admin with unrestricted instance access. Do not expose this " + "API beyond the internal network or set AUTH_DISABLED=false until " + "the maintainer UI sends Bearer tokens." + ) + else: + logging.info( + "Auth enabled: issuer=%s audience=%s jwks=%s", + auth_cfg.keycloak_issuer, + auth_cfg.keycloak_audience or "(none)", + auth_cfg.keycloak_jwks_url, + ) + + logging.info("api startup: initializing PostgreSQL database") + _api.db.init_db() + + # Temporal + temporal_host = os.environ.get("TEMPORAL_HOST", "localhost:7233") + logging.info("api startup: connecting to Temporal at %s", temporal_host) + _api.temporal_client = await _api.Client.connect(temporal_host) + + storage = _api.load_object_store_config() + _api.OBJECT_STORE_BUCKET = storage.bucket + _api.object_store = _api.open_object_store(storage) + if storage.create_bucket: + await asyncio.to_thread(_api.object_store.ensure_bucket, _api.OBJECT_STORE_BUCKET) + logging.info("api startup: ensured object storage bucket %s", _api.OBJECT_STORE_BUCKET) + + logging.info( + "api started auth=%s temporal=%s object_storage=%s bucket=%s", + "disabled" if auth_cfg.disabled else "enabled", + temporal_host, + storage.provider, + _api.OBJECT_STORE_BUCKET, + ) diff --git a/pipeline/api/search.py b/pipeline/api/search.py deleted file mode 100644 index 45f01e7..0000000 --- a/pipeline/api/search.py +++ /dev/null @@ -1,332 +0,0 @@ -"""Search API: ``POST /search`` and the ranking helpers it owns. - -First router split out of ``pipeline/api/api.py``. The vector store is -imported inside the route (not at module top) on purpose: tests patch -``pipeline.vector_store.get_vector_store`` and that only takes effect when the -name is looked up at call time. -""" - -from __future__ import annotations - -import json -import logging -import math -import re -from collections import Counter - -from fastapi import APIRouter, HTTPException - -from .. import db -from ..documents import document_validity -from ..auth.deps import RequireSearch - -router = APIRouter() - -_TOKEN_RE = re.compile(r"[\w\-]+", re.UNICODE) - - -def _normalize_text(value: str) -> str: - return re.sub(r"\s+", " ", (value or "").strip().lower()) - - -def _tokenize(value: str) -> list[str]: - return _TOKEN_RE.findall(_normalize_text(value)) - - -def _token_overlap_score(query: str, text: str) -> float: - query_tokens = set(_tokenize(query)) - text_tokens = set(_tokenize(text)) - if not query_tokens or not text_tokens: - return 0.0 - return len(query_tokens & text_tokens) / len(query_tokens) - - -def _metadata_blob(hit: dict) -> str: - return " ".join( - str(hit.get(key) or "") - for key in ( - "name", - "name_en", - "name_gu", - "filename", - "title_en", - "title_gu", - "category_tags", - "description", - "doc_short_description", - "doc_llm_description", - ) - ) - - -def _rank_desc(values: list[float]) -> list[int]: - order = sorted(range(len(values)), key=lambda idx: values[idx], reverse=True) - ranks = [0] * len(values) - for pos, idx in enumerate(order, start=1): - ranks[idx] = pos - return ranks - - -def _expand_query(query: str, profile: str) -> str: - q = (query or "").strip() - mode = (profile or "none").strip().lower() - if not q or mode in {"none", ""}: - return q - if mode not in {"gu-v1", "gu_v1"}: - return q - - rules = [ - (r"ખરવા|મોવાસા|fmd", "foot and mouth disease FMD blisters lesions mouth ulcer"), - (r"આફરો|bloat", "ruminal bloat tympany frothy bloat"), - (r"તાવ|fever", "pyrexia febrile infection"), - (r"કબજ|constipation", "constipation bowel obstruction laxative"), - (r"ગળિયો|ગળાની", "throat infection pharyngitis upper respiratory"), - (r"કૃમિ|કરમિયા|deworm", "deworming helminth anthelmintic dose"), - (r"ગર્ભપાત|ગાભણ", "abortion pregnancy gestation prenatal feeding"), - (r"ચરમિયા|ચામડી|ખંજવાળ|hair fall", "dermatitis skin disease mange ectoparasite tick"), - ] - - additions: list[str] = [] - query_lower = q.lower() - for pattern, terms in rules: - if re.search(pattern, query_lower, flags=re.IGNORECASE): - additions.append(terms) - if not additions: - return q - return f"{q} {' '.join(additions)}".strip() - - -def _bm25lite_scores(query: str, docs: list[str]) -> list[float]: - query_tokens = _tokenize(query) - if not query_tokens or not docs: - return [0.0] * len(docs) - - doc_tokens = [_tokenize(doc) for doc in docs] - avg_len = max(1.0, sum(len(tokens) for tokens in doc_tokens) / max(1, len(doc_tokens))) - df: Counter[str] = Counter() - for tokens in doc_tokens: - for token in set(tokens): - df[token] += 1 - - k1 = 1.2 - b = 0.75 - scores: list[float] = [] - for tokens in doc_tokens: - tf = Counter(tokens) - dl = len(tokens) - norm = k1 * (1 - b + b * dl / avg_len) - score = 0.0 - for term in query_tokens: - if term not in tf: - continue - idf = math.log(1.0 + (len(doc_tokens) - df[term] + 0.5) / (df[term] + 0.5)) - score += idf * (tf[term] * (k1 + 1.0)) / (tf[term] + norm) - scores.append(score) - return scores - - -def _rerank_hits(query: str, hits: list[dict], rerank_mode: str) -> list[dict]: - mode = (rerank_mode or "none").strip().lower() - if mode in {"", "none"} or not hits: - return hits - - raw_scores = [float(hit.get("_score", hit.get("score", 0.0)) or 0.0) for hit in hits] - min_score = min(raw_scores) - max_score = max(raw_scores) - denom = (max_score - min_score) if max_score > min_score else 1.0 - semantic_scores = [(raw - min_score) / denom for raw in raw_scores] - text_scores = [_token_overlap_score(query, str(hit.get("text") or "")) for hit in hits] - meta_scores = [_token_overlap_score(query, _metadata_blob(hit)) for hit in hits] - - rescored: list[dict] = [] - if mode == "bm25lite": - docs = [f"{str(hit.get('text') or '')} {_metadata_blob(hit)}".strip() for hit in hits] - bm_scores = _bm25lite_scores(query, docs) - bm_min = min(bm_scores) if bm_scores else 0.0 - bm_max = max(bm_scores) if bm_scores else 1.0 - bm_denom = (bm_max - bm_min) if bm_max > bm_min else 1.0 - bm_norm = [(score - bm_min) / bm_denom for score in bm_scores] - for hit, semantic, bm25, meta in zip(hits, semantic_scores, bm_norm, meta_scores): - enriched = dict(hit) - enriched["_rerank_score"] = (0.50 * semantic) + (0.40 * bm25) + (0.10 * meta) + (-0.10 if bool(hit.get("is_reference", False)) else 0.0) - rescored.append(enriched) - elif mode in {"rrf-lite", "rrf_lite", "rrf"}: - sem_rank = _rank_desc(semantic_scores) - text_rank = _rank_desc(text_scores) - meta_rank = _rank_desc(meta_scores) - k = 30 - for idx, hit in enumerate(hits): - enriched = dict(hit) - enriched["_rerank_score"] = (1.0 / (k + sem_rank[idx])) + (1.0 / (k + text_rank[idx])) + (1.0 / (k + meta_rank[idx])) - rescored.append(enriched) - else: - for hit, semantic, text_score, meta in zip(hits, semantic_scores, text_scores, meta_scores): - enriched = dict(hit) - enriched["_rerank_score"] = (0.60 * semantic) + (0.30 * max(text_score, meta)) + (0.10 * meta) - rescored.append(enriched) - - rescored.sort(key=lambda hit: float(hit.get("_rerank_score", 0.0)), reverse=True) - return rescored - - - - -def _requested_group(payload: dict) -> str | None: - raw = payload.get("group") - if raw is not None and not isinstance(raw, str): - raise HTTPException(400, "group must be a string") - return (raw or "").strip().lower() or None - - -def _requested_provider_id(payload: dict) -> str | None: - raw = payload.get("provider_id") - if raw is not None and not isinstance(raw, str): - raise HTTPException(400, "provider_id must be a string") - return (raw or "").strip().lower() or None - - -@router.post("/search") -async def run_search(payload: dict, user: RequireSearch): - """Search the vector index. - - `group` is a filter, not an access check: this endpoint applies no tenant - scoping to its results (unlike `GET /chunks/search`), so any - search-capable caller may ask for any group. - """ - from ..vector_store import get_default_index_name, get_vector_store - - logging.info( - "search payload user=%s payload=%s", - user.username or user.email or user.user_id, - json.dumps(payload, ensure_ascii=False, default=str), - ) - - settings = db.get_search_settings() - index_name = payload.get("index_name") or settings.get("indexName") or get_default_index_name() - query = (payload.get("query") or "").strip() - if not query: - raise HTTPException(400, "query is required") - - logging.info( - "search request user=%s index=%s query=%r top_k=%s", - user.username or user.email or user.user_id, - index_name, - query, - payload.get("top_k"), - ) - - search_mode = (payload.get("search_mode") or settings.get("searchMethod") or "HYBRID").upper() - top_k = max(1, min(int(payload.get("top_k") or settings.get("limit") or 12), 50)) - candidate_multiplier = max(1, int(payload.get("candidate_multiplier") or settings.get("candidateMultiplier") or 10)) - requested_candidate_cap = payload.get("candidate_cap") - if requested_candidate_cap is None: - candidate_cap = min(max(top_k * candidate_multiplier, top_k), int(settings.get("candidateCap") or 120)) - else: - candidate_cap = int(requested_candidate_cap) - candidate_cap = max(top_k, min(candidate_cap, 200)) - max_chunks_per_doc = max(1, int(payload.get("max_chunks_per_doc") or settings.get("maxChunksPerDoc") or 2)) - use_e5_prefix = bool(payload.get("use_e5_prefix", settings.get("useE5Prefix", True))) - exclude_reference = bool(payload.get("exclude_reference", settings.get("excludeReference", True))) - alpha = float(payload.get("hybrid_alpha") or settings.get("alpha") or 0.6) - ranking_method = payload.get("ranking_method") or settings.get("rankingMethod") or "rrf" - ef_search = int(payload.get("ef_search") or settings.get("efSearch") or 256) - query_expansion_profile = payload.get("query_expansion_profile") or settings.get("queryExpansionProfile") or "gu-v1" - rerank_mode = payload.get("rerank_mode") or settings.get("rerankMode") or "none" - hybrid_rrf_k = int(payload.get("hybrid_rrf_k") or settings.get("hybridRrfK") or 60) - # Validity is on by default: a search serving a farmer must not answer from - # a document that has expired or has not started. Both overrides are - # operator tools - `include_expired` to inspect what an expired document - # still holds, `valid_on` to ask what search would have returned on another - # day - and the response echoes which day was actually applied. - include_expired = bool(payload.get("include_expired", False)) - valid_on = (payload.get("valid_on") or "").strip() or None - if valid_on: - try: - # Reassigned, not just checked: parse_day returns the canonical - # zero-padded form, and the vector store's date range rejects the - # `2026-9-3` spelling that strptime happily accepts. - valid_on = document_validity.parse_day(valid_on, "valid_on") - except document_validity.DocumentValidityError as exc: - raise HTTPException(400, str(exc)) from None - group = _requested_group(payload) - provider_id = _requested_provider_id(payload) - expanded_query = _expand_query(query, query_expansion_profile) - # Qdrant embeddings apply E5 prefixes internally; don't pre-prefix here. - search_query = expanded_query - - store = get_vector_store() - backend = store.backend - # HYBRID on Qdrant currently maps to dense tensor search (lexical is separate mode). - store_mode = "TENSOR" if backend == "qdrant" and search_mode == "HYBRID" else search_mode - - try: - result = store.search( - name=index_name, - query=search_query, - limit=candidate_cap, - search_mode=store_mode, - exclude_reference=exclude_reference, - use_e5_prefix=use_e5_prefix, - hybrid_alpha=alpha, - ef_search=ef_search, - apply_validity=not include_expired, - valid_on=valid_on, - instance=group, - provider_id=provider_id, - ) - except Exception as error: - raise HTTPException(400, f"Vector search failed ({backend}): {error}") from error - - hits = result.get("hits", []) - hits = _rerank_hits(query, hits, rerank_mode) - final_hits = [] - per_doc_counts: dict[str, int] = {} - for hit in hits: - doc_key = hit.get("doc_id") or hit.get("filename") or "__unknown__" - if per_doc_counts.get(doc_key, 0) >= max_chunks_per_doc: - continue - per_doc_counts[doc_key] = per_doc_counts.get(doc_key, 0) + 1 - final_hits.append(hit) - if len(final_hits) >= top_k: - break - - logging.info( - "search response user=%s index=%s query=%r candidate_count=%s final_count=%s", - user.username or user.email or user.user_id, - index_name, - query, - len(hits), - len(final_hits), - ) - - return { - "effective_config": { - "index_name": index_name, - "backend": backend, - "query": query, - "search_mode": search_mode, - "store_search_mode": store_mode, - "top_k": top_k, - "candidate_cap": candidate_cap, - "candidate_multiplier": candidate_multiplier, - "max_chunks_per_doc": max_chunks_per_doc, - "use_e5_prefix": use_e5_prefix, - "exclude_reference": exclude_reference, - "hybrid_alpha": alpha, - "ranking_method": ranking_method, - "hybrid_rrf_k": hybrid_rrf_k, - "ef_search": ef_search, - "query_expansion_profile": query_expansion_profile, - "query_expansion_applied": expanded_query != query, - "rerank_mode": rerank_mode, - "apply_validity": not include_expired, - "valid_on": result.get("valid_on"), - "group": group, - "provider_id": provider_id, - "filter_string": None, - }, - "candidate_count": len(hits), - "final_count": len(final_hits), - "hits": final_hits, - "raw_hits": hits if payload.get("include_raw_hits") else None, - } diff --git a/pipeline/api/upload.py b/pipeline/api/upload.py deleted file mode 100644 index 2cd8f03..0000000 --- a/pipeline/api/upload.py +++ /dev/null @@ -1,249 +0,0 @@ -from __future__ import annotations - -import hashlib -from io import BytesIO -from pathlib import Path -from typing import Optional - -from fastapi import APIRouter, File, Form, HTTPException, Query, Request, UploadFile - -from .. import db -from ..documents import document_validity -from ..documents import display_name as display_name_rules -from ..auth.deps import RequireUpload -from ..auth.tenancy import assert_document_instance_access, normalize_instance -from ..instances import prod_stage_disabled -from ..models import DocumentStage, DocumentSummary -from ..workflows import DocumentPipelineWorkflow -from . import api as _api - -router = APIRouter() - - -@router.post("/upload", response_model=DocumentSummary) -@_api.limiter.limit(_api.RATE_LIMIT_UPLOAD) -async def upload_and_process( - request: Request, # Required for rate limiting - user: RequireUpload, - file: UploadFile = File(...), - auto_approve: bool = False, - chunk_size: int = 450, - chunk_overlap: int = 128, - min_tokens: int = 100, - index_name: str = "documents-index", - stop_after_ocr: bool = False, - instance: str = "", - kind: str = Query(..., description="scheme or advisory"), - code: str = "", - display_name: Optional[str] = Form(None), -): - """ - Upload a supported file and start processing workflow. - - The file is stored in MinIO and then processed through the pipeline. - Validates both file extension and PDF magic bytes for security. - Rate limited to 10 requests/minute per IP. - Requires permission: upload (no-op while AUTH_DISABLED=true). - """ - create_instance = _api._resolve_create_instance(user, instance) - doc_kind, doc_code, replace_doc = _api._resolve_document_identity( - user, create_instance, kind, code, file.filename or "" - ) - suffix = Path(file.filename or "").suffix.lower() - if suffix not in _api.ALLOWED_EXTENSIONS: - raise HTTPException(400, f"Unsupported file type: {suffix}") - try: - display_name = display_name_rules.normalize(display_name) - except ValueError as exc: - raise HTTPException(400, str(exc)) from exc - - # Read file content - content = await file.read() - file_size = len(content) - - # Validate PDF magic bytes (%PDF-) only for PDF uploads - if suffix == ".pdf": - pdf_magic = b"%PDF-" - if len(content) < 5 or content[:5] != pdf_magic: - raise HTTPException(400, "Invalid PDF file: file does not have valid PDF header") - - # Generate unique object name - file_hash = hashlib.md5(content).hexdigest() - object_name = f"{file_hash}/{file.filename}" - - # Upload to MinIO - content_type = "application/pdf" if suffix == ".pdf" else "application/octet-stream" - _api.minio_client.put_object( - _api.MINIO_BUCKET, - object_name, - BytesIO(content), - length=file_size, - content_type=content_type - ) - - # Use minio:// URI as filepath - minio_path = f"minio://{_api.MINIO_BUCKET}/{object_name}" - - workflow_id = _api.get_workflow_id(minio_path) - document_id = file_hash - canonical_document_id = file_hash - - # Reuse only when Postgres still tracks this workflow. - # If Postgres was purged, avoid returning stale Temporal state and create a fresh run ID. - existing_doc = db.get_document(workflow_id) - if existing_doc: - existing_doc = assert_document_instance_access(user, existing_doc) - try: - handle = _api.temporal_client.get_workflow_handle(workflow_id) - state = await handle.query("get_state") - if state: - return DocumentSummary( - document_id=document_id, - canonical_document_id=canonical_document_id, - workflow_id=workflow_id, - filename=file.filename, - source_filename=file.filename, - source_file_fingerprint=file_hash, - authoritative=bool(existing_doc.get("source_manifest_name")) if existing_doc else False, - instance=normalize_instance(existing_doc.get("instance")), - stage=DocumentStage(state.get("stage", "registered")), - page_count=state.get("page_count", 0), - chunk_count=state.get("chunk_count", 0), - error_message=state.get("error_message"), - ) - except HTTPException: - raise - except Exception: - pass - else: - workflow_id = _api._rerun_workflow_id(workflow_id) - - # Start new workflow - handle = await _api.temporal_client.start_workflow( - DocumentPipelineWorkflow.run, - args=[ - document_id, - file.filename, - minio_path, - chunk_size, - chunk_overlap, - min_tokens, - index_name, - auto_approve, - stop_after_ocr, - prod_stage_disabled(), - ], - id=workflow_id, - task_queue=_api.TASK_QUEUE, - ) - - # Save to Postgres for visibility during processing - actor = _api._actor_from_user(user) - # See the manifest upload path above: the period starts at the upload. - validity = document_validity.default_period() - if replace_doc: - document_uuid = replace_doc["id"] - db.reset_document_for_reupload( - document_uuid, - filename=file.filename, - filepath=minio_path, - fingerprint=file_hash, - ) - else: - try: - document_uuid = db.upsert_document( - workflow_id=workflow_id, - document_id=document_id, - canonical_document_id=canonical_document_id, - filename=file.filename, - source_filename=file.filename, - source_file_fingerprint=file_hash, - filepath=minio_path, - stage="registered", - stop_after_ocr=stop_after_ocr, - instance=create_instance, - uploaded_by_user_id=actor.get("user_id") or None, - uploaded_by_username=actor.get("username") or None, - uploaded_by_email=actor.get("email") or None, - uploaded_by_roles=actor.get("roles_csv") or None, - valid_from=validity.start_date, - valid_to=validity.end_date, - with_workflow=False, - kind=doc_kind, - code=doc_code, - # v2: the contributor's Display Name is documents.name. - name=display_name or _api._document_title(file.filename or ""), - ) - except db.DuplicateDocumentCode as exc: - raise HTTPException(409, str(exc)) from exc - job_id = db.create_document_job( - workflow_id=document_uuid, - job_type="ocr_only" if stop_after_ocr else "pipeline", - temporal_workflow_id=workflow_id, - status="running", - current_stage="registered", - config={ - "auto_approve": auto_approve, - "chunk_size": chunk_size, - "chunk_overlap": chunk_overlap, - "min_tokens": min_tokens, - "index_name": index_name, - "stop_after_ocr": stop_after_ocr, - "uploaded_by": actor, - }, - actor=actor, - ) - db.add_document_artifact( - workflow_id=workflow_id, - job_id=job_id, - artifact_type="original_upload", - stage="registered", - storage_uri=minio_path, - mime_type=content_type, - filename=file.filename, - size_bytes=file_size, - metadata={"uploaded_via": "upload_endpoint", "uploaded_by": actor}, - ) - _api._log_audit( - workflow_id=workflow_id, - action_type="document_upload", - entity_type="document", - field_name="registered", - new_value=file.filename, - metadata={ - "source": "upload_endpoint", - "instance": create_instance, - "kind": doc_kind, - "code": doc_code, - "replaced_existing": bool(replace_doc), - "size_bytes": file_size, - "auto_approve": auto_approve, - }, - user=user, - ) - source_type = "spreadsheet" if suffix in {".csv", ".xlsx"} else "document" - canonical_input_type = "spreadsheet" if suffix in {".csv", ".xlsx"} else "pdf" - db.update_document_fields( - workflow_id, - source_type=source_type, - canonical_input_type=canonical_input_type, - stop_after_ocr=1 if stop_after_ocr else 0, - ) - - return DocumentSummary( - document_id=document_id, - canonical_document_id=canonical_document_id, - workflow_id=workflow_id, - filename=file.filename, - source_filename=file.filename, - source_file_fingerprint=file_hash, - authoritative=False, - instance=create_instance, - stage=DocumentStage.REGISTERED, - page_count=0, - chunk_count=0, - uploaded_by_user_id=actor.get("user_id") or None, - uploaded_by_username=actor.get("username") or None, - uploaded_by_email=actor.get("email") or None, - uploaded_by_roles=actor.get("roles") or [], - ) diff --git a/pipeline/auth/config.py b/pipeline/auth/config.py index 16b1a3c..a2f2a9b 100644 --- a/pipeline/auth/config.py +++ b/pipeline/auth/config.py @@ -16,11 +16,8 @@ def _env_bool(name: str, default: bool) -> bool: @dataclass(frozen=True) class AuthConfig: """ - AUTH_DISABLED=true (default): accept a synthetic local admin user so - existing deploys and tests keep working until Keycloak is wired. - - AUTH_DISABLED=false: require a valid Bearer JWT from Keycloak. - Do not flip this until the maintainer UI sends Authorization headers. + Production requires a Keycloak bearer token addressed to this API. + Local fixtures can opt into a synthetic admin explicitly. """ disabled: bool @@ -29,6 +26,8 @@ class AuthConfig: keycloak_jwks_url: str # Clock-skew tolerance when validating exp/nbf (seconds). jwt_leeway_seconds: int = 30 + verify_live_groups: bool = False + runtime_profile: str = "production" def load_auth_config() -> AuthConfig: @@ -43,27 +42,24 @@ def load_auth_config() -> AuthConfig: except ValueError: leeway = 30 - # Empty KEYCLOAK_AUDIENCE disables audience checks (common for public SPA - # clients where `aud` is "account" or a multi-value claim). Only set it when - # tokens always include a stable audience you want to enforce. - raw_audience = os.environ.get("KEYCLOAK_AUDIENCE") - if raw_audience is None: - audience = "" - else: - audience = raw_audience.strip() + audience = os.environ.get("KEYCLOAK_AUDIENCE", "").strip() return AuthConfig( - disabled=_env_bool("AUTH_DISABLED", True), + disabled=_env_bool("AUTH_DISABLED", False), keycloak_issuer=issuer, keycloak_audience=audience, keycloak_jwks_url=jwks_url, jwt_leeway_seconds=leeway, + verify_live_groups=_env_bool("KEYCLOAK_VERIFY_LIVE_GROUPS", False), + runtime_profile=os.environ.get("KP_RUNTIME_PROFILE", "production").strip().lower(), ) def validate_auth_config(config: AuthConfig) -> None: """Fail startup early when enabled auth cannot validate Keycloak tokens.""" if config.disabled: + if config.runtime_profile != "local": + raise RuntimeError("AUTH_DISABLED=true requires KP_RUNTIME_PROFILE=local") return missing: list[str] = [] @@ -71,8 +67,15 @@ def validate_auth_config(config: AuthConfig) -> None: missing.append("KEYCLOAK_ISSUER") if not config.keycloak_jwks_url: missing.append("KEYCLOAK_JWKS_URL") + if not config.keycloak_audience: + missing.append("KEYCLOAK_AUDIENCE") if missing: raise RuntimeError( "AUTH_DISABLED=false requires Keycloak configuration: " + ", ".join(missing) ) + if config.verify_live_groups: + from .keycloak_admin import load_keycloak_admin_config + + if not load_keycloak_admin_config().configured: + raise RuntimeError("KEYCLOAK_VERIFY_LIVE_GROUPS=true requires Keycloak admin access") diff --git a/pipeline/auth/deps.py b/pipeline/auth/deps.py index 5a3af81..117a898 100644 --- a/pipeline/auth/deps.py +++ b/pipeline/auth/deps.py @@ -14,6 +14,7 @@ from .. import db from .config import load_auth_config from .jwt import claims_to_user, decode_and_validate_token +from .keycloak_admin import current_user_group_paths from .models import AuthUser, local_bypass_user from .permissions import Permission @@ -114,6 +115,15 @@ async def get_current_user(request: Request) -> AuthUser: raise HTTPException(401, "Missing Bearer token") # JWKS fetch/refresh is sync and can block; keep the event loop free. user = await asyncio.to_thread(decode_and_validate_token, token, config) + if config.verify_live_groups: + groups = await asyncio.to_thread(current_user_group_paths, user.user_id) + user = claims_to_user({ + "sub": user.user_id, + "preferred_username": user.username, + "email": user.email, + "envs": user.envs, + "groups": groups, + }, authoritative_groups=True) if request.method in _WRITE_METHODS and user.user_id: await asyncio.to_thread(_sync_user_row, user) diff --git a/pipeline/auth/email_otp.py b/pipeline/auth/email_otp.py deleted file mode 100644 index 944abee..0000000 --- a/pipeline/auth/email_otp.py +++ /dev/null @@ -1,547 +0,0 @@ -"""Server-side email-OTP login plus Keycloak SSO code exchange for the -confidential ``knowledge-engine`` client. - -Both login paths live here because both need the client secret, and both use -the *same* client id. That is forced, not chosen: Keycloak has no "public to -the browser, confidential to the backend" mode, so the browser can never touch -the token endpoint directly — see ``exchange_authorization_code``. - -Email-OTP is fully custom, not delegated to Keycloak's email-authenticator -plugin: that plugin registers only browser-flow SPIs (no RealmResourceProvider -REST endpoint), and its Email OTP authenticator can't identify a user without -a preceding password step — it's a second-factor plugin, not a standalone -passwordless one. Neither fact is fixable from config, so the API owns the -whole OTP lifecycle instead: - -1. ``send_otp`` generates a code, stores its hash (see ``pipeline.db``), and - emails it directly via SMTP (``KC_SMTP_*`` — the same creds the Keycloak - container uses, already present in every service's env via ``env_file``). -2. ``verify_otp`` checks the code, then bridges to a real Keycloak session: - the Admin API resets the user's Keycloak password to a random one-time - secret, and a plain password grant (client's Direct Grant Flow must be - Keycloak's *default* — no OTP override — since correctness was already - checked here) mints access/refresh/id tokens identical in shape to the - Google SSO ones. -""" - -from __future__ import annotations - -import hashlib -import hmac -import json -import logging -import os -import secrets -import smtplib -import ssl as ssl_lib -import time -import urllib.error -import urllib.parse -import urllib.request -from dataclasses import dataclass -from datetime import datetime, timedelta, timezone -from email.message import EmailMessage -from typing import Any - -from fastapi import HTTPException - -from .. import db -from . import keycloak_admin - -OTP_LENGTH = 6 -OTP_TTL_SECONDS = 300 -OTP_RESEND_COOLDOWN_SECONDS = 30 -OTP_MAX_ATTEMPTS = 5 -# Approximates a typical SMTP round-trip, so the "no such account" path takes -# about as long as the "account exists, email sent" path. -_FAKE_SEND_DELAY_SECONDS = 0.35 - - -@dataclass(frozen=True) -class EmailOtpConfig: - """Realm URL plus the confidential client used for both OTP steps.""" - - issuer: str - client_id: str - client_secret: str - - @property - def configured(self) -> bool: - return bool(self.issuer and self.client_id and self.client_secret) - - @property - def token_url(self) -> str: - return f"{self.issuer}/protocol/openid-connect/token" - - -def load_email_otp_config() -> EmailOtpConfig: - issuer = (os.environ.get("KEYCLOAK_ISSUER") or "").strip().rstrip("/") - # The OTP client is the same one the browser uses for SSO unless overridden; - # it just needs a secret here, which the browser never sees. - client_id = ( - os.environ.get("KEYCLOAK_OTP_CLIENT_ID") - or os.environ.get("KEYCLOAK_CLIENT_ID") - or os.environ.get("VITE_KEYCLOAK_CLIENT_ID") - or "" - ).strip() - client_secret = (os.environ.get("KEYCLOAK_CLIENT_SECRET") or "").strip() - return EmailOtpConfig(issuer=issuer, client_id=client_id, client_secret=client_secret) - - -def require_email_otp_config() -> EmailOtpConfig: - cfg = load_email_otp_config() - if not cfg.configured: - missing = [ - name - for name, value in ( - ("KEYCLOAK_ISSUER", cfg.issuer), - ("KEYCLOAK_CLIENT_ID", cfg.client_id), - ("KEYCLOAK_CLIENT_SECRET", cfg.client_secret), - ) - if not value - ] - raise HTTPException( - 503, - "Email OTP login is not configured. Set " + ", ".join(missing) + ".", - ) - return cfg - - -def _post(url: str, *, body: Any = None, form: dict | None = None) -> tuple[int, Any]: - """POST JSON or form-encoded; returns (status, parsed body) and never raises on 4xx.""" - headers: dict[str, str] = {"Accept": "application/json"} - if form is not None: - data = urllib.parse.urlencode(form).encode() - headers["Content-Type"] = "application/x-www-form-urlencoded" - else: - data = json.dumps(body or {}).encode() - headers["Content-Type"] = "application/json" - - request = urllib.request.Request(url, data=data, method="POST", headers=headers) - try: - with urllib.request.urlopen(request, timeout=30) as response: - raw = response.read() - return response.status, (json.loads(raw.decode()) if raw else None) - except urllib.error.HTTPError as exc: - raw = exc.read().decode("utf-8", errors="replace") - try: - return exc.code, json.loads(raw) - except Exception: - return exc.code, {"error_description": raw} - except urllib.error.URLError as exc: - raise HTTPException(502, f"Keycloak is unreachable: {exc.reason}") from exc - - -def normalize_email(raw: str) -> str: - email = (raw or "").strip().lower() - if not email or "@" not in email: - raise HTTPException(400, "Enter a valid email address.") - return email - - -# -------------------------------------------------------------------------- -# SMTP — sends the OTP code directly, independent of Keycloak entirely. -# -------------------------------------------------------------------------- - - -@dataclass(frozen=True) -class SmtpConfig: - host: str - port: int - username: str - password: str - from_addr: str - from_name: str - use_auth: bool - use_starttls: bool - use_ssl: bool - - @property - def configured(self) -> bool: - return bool(self.host and self.from_addr) - - -def _env_bool(name: str, default: bool) -> bool: - raw = os.environ.get(name) - if raw is None or raw == "": - return default - return raw.strip().lower() in ("1", "true", "yes", "on") - - -def load_smtp_config() -> SmtpConfig: - return SmtpConfig( - host=(os.environ.get("KC_SMTP_HOST") or "").strip(), - port=int(os.environ.get("KC_SMTP_PORT") or 587), - username=(os.environ.get("KC_SMTP_USERNAME") or "").strip(), - password=os.environ.get("KC_SMTP_PASSWORD") or "", - from_addr=(os.environ.get("KC_SMTP_FROM") or "").strip(), - from_name=(os.environ.get("KC_SMTP_FROM_DISPLAY_NAME") or "Bharat Vistaar").strip(), - use_auth=_env_bool("KC_SMTP_AUTH", True), - use_starttls=_env_bool("KC_SMTP_STARTTLS", True), - use_ssl=_env_bool("KC_SMTP_SSL", False), - ) - - -def _send_email(to_addr: str, subject: str, body: str) -> None: - cfg = load_smtp_config() - if not cfg.configured: - raise HTTPException(503, "Email sending is not configured (KC_SMTP_* missing).") - - message = EmailMessage() - message["Subject"] = subject - message["From"] = f"{cfg.from_name} <{cfg.from_addr}>" if cfg.from_name else cfg.from_addr - message["To"] = to_addr - message.set_content(body) - - try: - if cfg.use_ssl: - with smtplib.SMTP_SSL( - cfg.host, cfg.port, timeout=20, context=ssl_lib.create_default_context() - ) as smtp: - if cfg.use_auth: - smtp.login(cfg.username, cfg.password) - smtp.send_message(message) - else: - with smtplib.SMTP(cfg.host, cfg.port, timeout=20) as smtp: - if cfg.use_starttls: - smtp.starttls(context=ssl_lib.create_default_context()) - if cfg.use_auth: - smtp.login(cfg.username, cfg.password) - smtp.send_message(message) - except (smtplib.SMTPException, OSError) as exc: - logging.error("email-otp: failed to send code to %s: %s", to_addr, exc) - raise HTTPException( - 503, "Could not send the email right now. Please try again shortly." - ) from exc - - -# -------------------------------------------------------------------------- -# OTP generation / verification -# -------------------------------------------------------------------------- - - -def _generate_code() -> str: - return f"{secrets.randbelow(10 ** OTP_LENGTH):0{OTP_LENGTH}d}" - - -def _otp_pepper() -> bytes: - """Purpose-specific subkey derived from the confidential client secret. - - A 6-digit code is only 1M possibilities, so hashing it with a public salt - (the email) and a fast, unkeyed hash would let anyone with read access to - the email_otps table brute-force a live code offline in milliseconds. HMAC - with a server-only key closes that: cracking a stored hash now also - requires the key, not just DB access. Derived (not reused directly) to - keep this key separate from the client secret's actual purpose. - """ - client_secret = require_email_otp_config().client_secret - return hmac.new(client_secret.encode(), b"email-otp-code-hash-v1", hashlib.sha256).digest() - - -def _hash_code(email: str, code: str) -> str: - # Keyed with the email too: a leaked hash can't be replayed against a - # different account, and two users with the same code hash differently. - return hmac.new(_otp_pepper(), f"{email}:{code}".encode(), hashlib.sha256).hexdigest() - - -def _find_keycloak_user(email: str) -> dict[str, Any] | None: - cfg = keycloak_admin.require_admin_config() - token = keycloak_admin._admin_token(cfg) - admin = keycloak_admin._admin_root(cfg) - status, found = keycloak_admin._req( - "GET", - f"{admin}/users?{urllib.parse.urlencode({'email': email, 'exact': 'true'})}", - token=token, - ) - if isinstance(found, list) and found: - return found[0] - return None - - -def send_otp(email: str) -> dict[str, Any]: - """Mail a login code. Enumeration-safe: identical response either way. - - Only accounts that actually exist in Keycloak get a real email sent — this - both avoids using the API's SMTP relay to spam arbitrary addresses, and - means a mistyped/unknown email produces no visible difference in the - response the caller sees. - """ - now = datetime.now(timezone.utc) - generic_response = { - "status": "ok", - "message": "If the account exists, a one-time code has been emailed to it", - "resend_after_seconds": OTP_RESEND_COOLDOWN_SECONDS, - } - - # A cooldown row is written on every request below regardless of account - # existence, so this branch (and the countdown it reveals) fires - # identically for real and unknown addresses — it must never be gated on - # whether an account was actually found, or the countdown becomes a - # response oracle for account existence. - existing = db.get_email_otp(email) - if existing is not None: - last_sent = datetime.fromisoformat(existing["last_sent_at"]) - elapsed = (now - last_sent).total_seconds() - if elapsed < OTP_RESEND_COOLDOWN_SECONDS: - # Cooldown — silent no-op, but still looks identical to a fresh send. - remaining = max(0, int(OTP_RESEND_COOLDOWN_SECONDS - elapsed)) - return {**generic_response, "resend_after_seconds": remaining} - - # Always do the same lookup + storage work whether or not the account - # exists, so neither the response nor its timing can be used to enumerate - # which emails have accounts. - user = _find_keycloak_user(email) - expires_at = now + timedelta(seconds=OTP_TTL_SECONDS) - if user is not None: - code = _generate_code() - code_hash = _hash_code(email, code) - else: - # Unreachable by any real 6-digit code — verify_otp always reports - # "incorrect" for this address, same as a genuine wrong guess. - code = None - code_hash = secrets.token_hex(32) - - db.store_email_otp( - email=email, - code_hash=code_hash, - expires_at=expires_at.isoformat(), - created_at=now.isoformat(), - last_sent_at=now.isoformat(), - ) - - if user is not None: - try: - _send_email( - email, - "Your sign-in code", - f"Your one-time sign-in code is: {code}\n\n" - f"This code expires in {OTP_TTL_SECONDS // 60} minutes. " - "If you did not request this, you can safely ignore this email.", - ) - logging.info("email-otp: login code sent to %s", email) - except HTTPException: - # Never let an SMTP failure surface as a different response/status - # than the "no such account" path — that would itself be an - # enumeration oracle. The stored code is now simply unreachable; - # the user's next resend attempt (after the cooldown) tries again. - logging.error("email-otp: send failed for %s, response withheld", email) - else: - # Matches the typical SMTP round-trip _send_email would otherwise - # take, so response latency doesn't leak account existence either. - time.sleep(_FAKE_SEND_DELAY_SECONDS) - - return generic_response - - -def verify_otp(email: str, code: str) -> dict[str, Any]: - """Check a code and, if correct, mint real Keycloak tokens for the account.""" - otp = (code or "").strip() - if not otp: - raise HTTPException(400, "Enter the code from your email.") - - row = db.get_email_otp(email) - if row is None: - raise HTTPException(401, "That code is no longer valid. Request a new one.") - - now = datetime.now(timezone.utc) - if now > datetime.fromisoformat(row["expires_at"]): - db.delete_email_otp(email) - raise HTTPException(401, "That code has expired. Request a new one.") - - if row["attempts"] >= OTP_MAX_ATTEMPTS: - db.delete_email_otp(email) - raise HTTPException(401, "Too many incorrect attempts. Request a new code.") - - if row["code_hash"] != _hash_code(email, otp): - db.increment_email_otp_attempts(email) - raise HTTPException(401, "That code is incorrect.") - - db.delete_email_otp(email) - return _mint_tokens_for_verified_email(email) - - -def _mint_tokens_for_verified_email(email: str) -> dict[str, Any]: - """Bridge a verified OTP to a real Keycloak session. - - Resets the user's Keycloak password to a random one-time secret, spends - it on a plain password grant, then immediately rotates it again — so the - password this account holds is never a value that stays valid or - memorable, it exists as a real credential only for the instant it's - used. - - IMPORTANT residual exposure: this relies on the knowledge-engine client's - Direct Grant Flow being Keycloak's unmodified default (no OTP step), so - that this plain password grant succeeds. That also means *any* caller who - knows the client secret and a user's *current* Keycloak password can get - a token straight from Keycloak's token endpoint, bypassing this API (and - its rate limiting / enumeration protections) entirely — the bridge only - adds a path, it doesn't remove the plain one. The immediate rotation here - keeps that window as small as possible for passwords *this code* sets, - but does not protect against a password set some other way (e.g. by an - admin, or during testing). A fully closed version of this would need a - second Keycloak client — direct-grant-only, no browser flows, ideally - reachable only from the internal network — dedicated solely to this - bridge, so the bypass surface doesn't overlap with the client the browser - also uses for SSO. - """ - user = _find_keycloak_user(email) - if user is None: - # Shouldn't happen — send_otp only emails codes to real accounts — - # but the account could have been deleted in between. - raise HTTPException(401, "No account found for this email.") - - admin_cfg = keycloak_admin.require_admin_config() - admin_token = keycloak_admin._admin_token(admin_cfg) - admin_root = keycloak_admin._admin_root(admin_cfg) - - def _rotate_password() -> str: - new_password = secrets.token_urlsafe(32) - keycloak_admin._req( - "PUT", - f"{admin_root}/users/{user['id']}/reset-password", - token=admin_token, - body={"type": "password", "value": new_password, "temporary": False}, - ) - return new_password - - ephemeral_password = _rotate_password() - - cfg = require_email_otp_config() - status, body = _post( - cfg.token_url, - form={ - "grant_type": "password", - "client_id": cfg.client_id, - "client_secret": cfg.client_secret, - "username": email, - "password": ephemeral_password, - "scope": "openid", - }, - ) - payload = body if isinstance(body, dict) else {} - - # Rotate again regardless of outcome — the just-used password must not - # remain a live credential either way. - try: - _rotate_password() - except HTTPException: - logging.error("email-otp: post-grant password rotation failed for %s", email) - - if status == 200 and payload.get("access_token"): - return _token_response(payload) - - logging.error( - "email-otp: password-grant bridge failed for %s: status=%s body=%s", - email, status, payload, - ) - - error = str(payload.get("error") or "") - if error in ("invalid_client", "unauthorized_client"): - # The client rejected the password grant itself — almost always - # Direct Access Grants being disabled for it, not anything the user - # did. Surface this as an operator-actionable misconfiguration - # rather than a generic try-again, matching exchange_authorization_code's - # handling of the same Keycloak error vocabulary. - raise HTTPException( - 503, - "Email OTP sign-in is misconfigured (Keycloak rejected the password grant). " - "Confirm Direct Access Grants is enabled for the client tied to KEYCLOAK_CLIENT_ID.", - ) - raise HTTPException(502, "Could not complete sign-in. Please try again.") - - -def _token_response(payload: dict[str, Any]) -> dict[str, Any]: - return { - "access_token": payload["access_token"], - "refresh_token": payload.get("refresh_token"), - "id_token": payload.get("id_token"), - "token_type": payload.get("token_type", "Bearer"), - "expires_in": payload.get("expires_in"), - "refresh_expires_in": payload.get("refresh_expires_in"), - } - - -def refresh_tokens(refresh_token: str) -> dict[str, Any]: - """Renew an OTP or SSO session. - - Has to live on the server: the client is confidential, so the browser cannot - present the secret the token endpoint demands for a refresh_token grant. - """ - cfg = require_email_otp_config() - token = (refresh_token or "").strip() - if not token: - raise HTTPException(400, "Missing refresh token.") - - status, body = _post( - cfg.token_url, - form={ - "grant_type": "refresh_token", - "client_id": cfg.client_id, - "client_secret": cfg.client_secret, - "refresh_token": token, - }, - ) - payload = body if isinstance(body, dict) else {} - - if status == 200 and payload.get("access_token"): - return _token_response(payload) - - # An expired or revoked refresh token is a normal end-of-session, not a fault. - raise HTTPException(401, "Your session has expired. Please sign in again.") - - -def exchange_authorization_code( - code: str, redirect_uri: str, code_verifier: str | None = None -) -> dict[str, Any]: - """Trade an SSO authorization code for tokens, on behalf of the browser. - - The browser cannot run this exchange itself: the client is confidential, so - the token endpoint answers ``unauthorized_client`` to anyone who cannot - present the secret. Doing it here is what lets Google SSO and email-OTP - share a single client id, and it keeps the refresh token off the wire to - Keycloak from the browser entirely. - - ``code_verifier`` is the PKCE verifier the UI generated before redirecting; - it is not a secret the server holds, just a value passed through. - """ - cfg = require_email_otp_config() - authorization_code = (code or "").strip() - if not authorization_code: - raise HTTPException(400, "Missing authorization code.") - if not (redirect_uri or "").strip(): - raise HTTPException(400, "Missing redirect_uri.") - - form = { - "grant_type": "authorization_code", - "client_id": cfg.client_id, - "client_secret": cfg.client_secret, - "code": authorization_code, - "redirect_uri": redirect_uri, - } - if code_verifier: - form["code_verifier"] = code_verifier - - status, body = _post(cfg.token_url, form=form) - payload = body if isinstance(body, dict) else {} - - if status == 200 and payload.get("access_token"): - return _token_response(payload) - - error = str(payload.get("error") or "") - description = str(payload.get("error_description") or "") - - if error == "invalid_client" or error == "unauthorized_client": - raise HTTPException( - 503, - "SSO is misconfigured (Keycloak rejected the client credentials). Check " - "KEYCLOAK_CLIENT_ID / KEYCLOAK_CLIENT_SECRET.", - ) - if error == "invalid_grant": - # A code is single-use and short-lived; a reload of the callback URL - # replays a spent one, which is the common way to land here. - raise HTTPException(401, "That sign-in link has already been used or expired. Try again.") - - raise HTTPException( - 502 if status >= 500 else 400, - description or "Could not complete sign-in.", - ) diff --git a/pipeline/auth/email_sign_in.py b/pipeline/auth/email_sign_in.py new file mode 100644 index 0000000..2846ad6 --- /dev/null +++ b/pipeline/auth/email_sign_in.py @@ -0,0 +1,170 @@ +"""Use the private Keycloak email flow to issue the console's normal session.""" + +from __future__ import annotations + +import base64 +import hashlib +import hmac +import json +import os +import re +import secrets +from html.parser import HTMLParser +from urllib.parse import parse_qs, urljoin, urlsplit + +import httpx +from cryptography.fernet import Fernet, InvalidToken +from fastapi import HTTPException + +from pipeline import db + +from .oidc import OidcConfig, exchange_authorization_code, require_oidc_config +from .private_browser import private_client, private_realm, submit_private_form + +CODE_TTL = 300 +REDIRECT_DEFAULT = "http://127.0.0.1/knowledge-provider/auth/sso-callback" +_EMAIL = re.compile(r"[^\s@]+@[^\s@]+\.[^\s@]+") + + +class _LoginForm(HTMLParser): + def __init__(self, html: str, field: str): + super().__init__() + self.field = field + self.form_action = None + self.action = None + self.feed(html) + + def handle_starttag(self, tag, attrs): + attributes = dict(attrs) + if tag == "form" and attributes.get("method", "").lower() == "post": + self.form_action = attributes.get("action") + if tag == "input" and attributes.get("name") == self.field: + if "disabled" not in attributes: + self.action = self.form_action + + def handle_endtag(self, tag): + if tag == "form": + self.form_action = None + + +def _key(config: OidcConfig, purpose: bytes) -> bytes: + return hmac.digest(config.client_secret.encode(), purpose, "sha256") + + +def _cipher(config: OidcConfig) -> Fernet: + return Fernet(base64.urlsafe_b64encode(_key(config, b"kp-email-sign-in-challenge-v1"))) + + +def _throttle(config: OidcConfig, email: str) -> None: + address = hmac.digest(_key(config, b"kp-email-sign-in-throttle-v1"), email.encode(), "sha256") + with db.get_connection() as connection: + connection.execute("DELETE FROM email_sign_in_limits WHERE available_at <= now()") + inserted = connection.execute( + """ + INSERT INTO email_sign_in_limits (address_digest, available_at) + VALUES (%s, now() + interval '60 seconds') + ON CONFLICT (address_digest) DO UPDATE SET available_at = EXCLUDED.available_at + WHERE email_sign_in_limits.available_at <= now() + RETURNING 1 + """, + (address,), + ).fetchone() + if not inserted: + raise HTTPException(429, "Wait a minute before requesting another code.", headers={"Retry-After": "60"}) + + +def _form_action(config: OidcConfig, response: httpx.Response, field: str) -> str | None: + if response.status_code != 200: + return None + action = _LoginForm(response.text, field).action + if not action: + return None + target = urlsplit(urljoin(str(response.request.url), action)) + private = urlsplit(private_realm(config)) + public = urlsplit(config.issuer) + origins = {(item.scheme, item.netloc) for item in (private, public)} + if (target.scheme, target.netloc) not in origins: + raise HTTPException(502, "Sign-in is temporarily unavailable.") + if target.path != f"{private.path}/login-actions/authenticate": + raise HTTPException(502, "Sign-in is temporarily unavailable.") + return f"{private.scheme}://{private.netloc}{target.path}?{target.query}" + + +def _sealed_challenge(config: OidcConfig, context: dict, *, issued_at: int | None = None) -> str: + # Unknown and disabled accounts get the same response shape and size. + context.pop("padding", None) + context["padding"] = " " * max(0, 4096 - len(json.dumps(context, separators=(",", ":")))) + payload = json.dumps(context, separators=(",", ":")).encode() + cipher = _cipher(config) + sealed = cipher.encrypt(payload) if issued_at is None else cipher.encrypt_at_time(payload, issued_at) + return sealed.decode() + + +def send_email_code(raw_email: str) -> str: + email = raw_email.strip().lower() + if len(email) > 254 or not _EMAIL.fullmatch(email): + raise HTTPException(422, "Enter a valid email address.") + config = require_oidc_config() + _throttle(config, email) + verifier, state = secrets.token_urlsafe(48), secrets.token_urlsafe(32) + redirect = os.environ.get("KEYCLOAK_CONSOLE_REDIRECT_URI") or REDIRECT_DEFAULT + challenge = base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).decode().rstrip("=") + + try: + with private_client() as client: + login = client.get(f"{private_realm(config)}/protocol/openid-connect/auth", params={ + "client_id": config.client_id, "response_type": "code", "scope": "openid profile email", + "redirect_uri": redirect, "state": state, "code_challenge": challenge, + "code_challenge_method": "S256", "prompt": "login", "kc_idp_hint": "", + }) + username_action = _form_action(config, login, "username") + if not username_action: + raise HTTPException(503, "Email sign-in is temporarily unavailable.") + code_page = submit_private_form(client, username_action, {"username": email}) + if code_page.status_code >= 500: + raise HTTPException(503, "Email sign-in is temporarily unavailable.") + code_action = _form_action(config, code_page, "emailCode") + context = { + "action": code_action, "cookies": dict(client.cookies.items()) if code_action else {}, + "verifier": verifier, "state": state, "redirect": redirect, + } + except httpx.HTTPError: + raise HTTPException(503, "Email sign-in is temporarily unavailable.") from None + + return _sealed_challenge(config, context) + + +def verify_email_code(challenge: str, code: str) -> tuple[dict | None, str]: + invalid = HTTPException(401, "That code is invalid or expired.") + if not re.fullmatch(r"[0-9]{6}", code) or not 1 <= len(challenge) <= 12000: + raise invalid + config = require_oidc_config() + cipher = _cipher(config) + try: + context = json.loads(cipher.decrypt(challenge.encode(), ttl=CODE_TTL)) + except InvalidToken: + raise invalid from None + if not context["action"]: + return None, challenge + + try: + with private_client(context["cookies"]) as client: + result = submit_private_form(client, context["action"], {"emailCode": code}) + if result.status_code == 200: + context["action"] = _form_action(config, result, "emailCode") + context["cookies"] = dict(client.cookies.items()) if context["action"] else {} + return None, _sealed_challenge(config, context, issued_at=cipher.extract_timestamp(challenge.encode())) + except httpx.HTTPError: + raise HTTPException(503, "Email sign-in is temporarily unavailable.") from None + if result.status_code >= 500: + raise HTTPException(503, "Email sign-in is temporarily unavailable.") + if not result.is_redirect: + raise invalid + target = urlsplit(result.headers.get("Location", "")) + callback = urlsplit(context["redirect"]) + if (target.scheme, target.netloc, target.path) != (callback.scheme, callback.netloc, callback.path): + raise invalid + query = parse_qs(target.query) + if query.get("state") != [context["state"]] or len(query.get("code", [])) != 1: + raise invalid + return exchange_authorization_code(query["code"][0], context["redirect"], context["verifier"]), "" diff --git a/pipeline/auth/google_sign_in.py b/pipeline/auth/google_sign_in.py new file mode 100644 index 0000000..24954ab --- /dev/null +++ b/pipeline/auth/google_sign_in.py @@ -0,0 +1,159 @@ +"""Broker Google through Keycloak while keeping the Keycloak session private.""" + +import base64 +import hashlib +import hmac +import json +import os +import secrets +from dataclasses import dataclass +from urllib.parse import parse_qs, urljoin, urlsplit + +import httpx +from cryptography.fernet import Fernet, InvalidToken +from fastapi import HTTPException + +from .email_sign_in import REDIRECT_DEFAULT +from .oidc import OidcConfig, exchange_authorization_code, require_oidc_config +from .private_browser import private_client, private_cookie_header, private_realm + +GOOGLE_STATE_TTL = 600 +GOOGLE_RESULT_TTL = 60 +GOOGLE_STATE_COOKIE = "kp_google_state" +GOOGLE_RESULT_COOKIE = "kp_google_result" + + +@dataclass(frozen=True) +class GoogleConsole: + url: str + + @property + def secure(self) -> bool: + return urlsplit(self.url).scheme == "https" + + @property + def login_path(self) -> str: + return urlsplit(self.url).path + + +def google_console() -> GoogleConsole | None: + value = os.environ.get("KEYCLOAK_GOOGLE_CONSOLE_URL", "").strip() + if not value: + return None + url = urlsplit(value) + loopback = url.scheme == "http" and url.hostname in {"127.0.0.1", "localhost", "::1"} + if (not (url.scheme == "https" or loopback) or not url.netloc or url.username is not None + or not url.path.startswith("/") or url.path.startswith("//") or url.query or url.fragment): + raise HTTPException(503, "Google sign-in is temporarily unavailable.") + return GoogleConsole(value) + + +def require_google_console() -> GoogleConsole: + console = google_console() + if console is None: + raise HTTPException(404, "Google sign-in is unavailable.") + return console + + +def _cipher(config: OidcConfig, purpose: str) -> Fernet: + key = hmac.digest(config.client_secret.encode(), f"kp-google-{purpose}-v1".encode(), "sha256") + return Fernet(base64.urlsafe_b64encode(key)) + + +def _seal(config: OidcConfig, purpose: str, data: dict) -> str: + return _cipher(config, purpose).encrypt(json.dumps(data, separators=(",", ":")).encode()).decode() + + +def _open(config: OidcConfig, purpose: str, data: str | None, ttl: int) -> dict: + if data is None or len(data) > 4000: + raise HTTPException(401, "Start Google sign-in again.") + try: + return json.loads(_cipher(config, purpose).decrypt(data.encode(), ttl=ttl)) + except InvalidToken: + raise HTTPException(401, "Start Google sign-in again.") from None + + +def _redirect(response: httpx.Response) -> str: + if not response.is_redirect or not response.headers.get("Location"): + raise HTTPException(502, "Google sign-in is temporarily unavailable.") + return urljoin(str(response.request.url), response.headers["Location"]) + + +def _private_redirect(config: OidcConfig, location: str, suffix: str) -> str: + target = urlsplit(location) + private = urlsplit(private_realm(config)) + realms = (urlsplit(config.issuer), private) + if not any( + (target.scheme, target.netloc, target.path) == (realm.scheme, realm.netloc, realm.path + suffix) + for realm in realms + ): + raise HTTPException(502, "Google sign-in is temporarily unavailable.") + return f"{private_realm(config)}{suffix}?{target.query}" + + +def _console_code(location: str, context: dict) -> str: + target = urlsplit(location) + callback = urlsplit(context["redirect"]) + if (target.scheme, target.netloc, target.path) != (callback.scheme, callback.netloc, callback.path): + raise HTTPException(401, "Google sign-in requires a linked account. Use email to sign in.") + params = parse_qs(target.query) + if params.get("state") != [context["state"]] or len(params.get("code", ())) != 1: + raise HTTPException(401, "Google sign-in could not be completed.") + return params["code"][0] + + +def start_google_sign_in(console: GoogleConsole) -> tuple[str, str]: + config = require_oidc_config() + verifier, state = secrets.token_urlsafe(48), secrets.token_urlsafe(32) + redirect = os.environ.get("KEYCLOAK_CONSOLE_REDIRECT_URI") or REDIRECT_DEFAULT + challenge = base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).decode().rstrip("=") + try: + with private_client() as client: + start = client.get(f"{private_realm(config)}/protocol/openid-connect/auth", params={ + "client_id": config.client_id, "response_type": "code", "scope": "openid profile email", + "redirect_uri": redirect, "state": state, "code_challenge": challenge, + "code_challenge_method": "S256", "kc_idp_hint": "google", "prompt": "select_account", + }) + broker = _private_redirect(config, _redirect(start), "/broker/google/login") + google = _redirect(client.get(broker, headers=private_cookie_header(client))) + cookies = dict(client.cookies.items()) + except httpx.HTTPError: + raise HTTPException(502, "Google sign-in is temporarily unavailable.") from None + + target = urlsplit(google) + query = parse_qs(target.query) + issuer = urlsplit(config.issuer) + origin = urlsplit(console.url) + callback = f"{origin.scheme}://{origin.netloc}{issuer.path}/broker/google/endpoint" + if ((target.scheme, target.netloc, target.path) != ("https", "accounts.google.com", "/o/oauth2/v2/auth") + or query.get("redirect_uri") != [callback] or len(query.get("state", ())) != 1): + raise HTTPException(502, "Google sign-in is temporarily unavailable.") + return google, _seal(config, "state", { + "state": state, "verifier": verifier, "redirect": redirect, "cookies": cookies, + "google_state": query["state"][0], + }) + + +def accept_google_callback(cookie: str | None, code: str, state: str) -> str: + config = require_oidc_config() + context = _open(config, "state", cookie, GOOGLE_STATE_TTL) + if not code or len(code) > 4096 or not state or len(state) > 4096: + raise HTTPException(401, "Google sign-in could not be completed.") + if not hmac.compare_digest(state.encode(), context["google_state"].encode()): + raise HTTPException(401, "Start Google sign-in again.") + try: + with private_client(context["cookies"]) as client: + result = client.get(f"{private_realm(config)}/broker/google/endpoint", params={ + "code": code, "state": state, + }) + except httpx.HTTPError: + raise HTTPException(502, "Google sign-in is temporarily unavailable.") from None + console_code = _console_code(_redirect(result), context) + return _seal(config, "result", { + "code": console_code, "verifier": context["verifier"], "redirect": context["redirect"], + }) + + +def complete_google_sign_in(cookie: str | None) -> dict: + context = _open(require_oidc_config(), "result", cookie, GOOGLE_RESULT_TTL) + return exchange_authorization_code(context["code"], context["redirect"], context["verifier"]) diff --git a/pipeline/auth/groups.py b/pipeline/auth/groups.py index f907c02..d584dd4 100644 --- a/pipeline/auth/groups.py +++ b/pipeline/auth/groups.py @@ -3,7 +3,7 @@ Expected Keycloak group layout (full paths in the JWT ``groups`` claim): - ``/global/super-admin`` → super_admin (all states, full access) -- ``/global/bh-viewer`` → bh_viewer (all states, view-only) +- ``/global/global-viewer`` → global_viewer (all providers, read-only) - ``/states/{CODE}/admin`` → state_admin - ``/states/{CODE}/approver`` → state_approver - ``/states/{CODE}/contributor`` → state_contributor @@ -17,13 +17,14 @@ from __future__ import annotations +import os import re from dataclasses import dataclass, field from typing import Any, Iterable # Canonical product roles (JWT / Keycloak group leaves + realm roles). ROLE_SUPER_ADMIN = "super_admin" -ROLE_BH_VIEWER = "bh_viewer" +ROLE_GLOBAL_VIEWER = "global_viewer" ROLE_STATE_ADMIN = "state_admin" ROLE_STATE_APPROVER = "state_approver" ROLE_STATE_CONTRIBUTOR = "state_contributor" @@ -37,8 +38,13 @@ ROLE_STATE_VIEW, ) +GLOBAL_VIEWER_ROLE_ALIASES = frozenset({ + ROLE_GLOBAL_VIEWER, "global-viewer", "bh_viewer", "bh-viewer", "bh_view", "bh-view", +}) +GLOBAL_VIEWER_GROUP_PATH = "/global/global-viewer" + # Roles that apply across every tenant. -GLOBAL_ROLES = (ROLE_SUPER_ADMIN, ROLE_BH_VIEWER) +GLOBAL_ROLES = (ROLE_SUPER_ADMIN, ROLE_GLOBAL_VIEWER) CANONICAL_ROLES = frozenset(GLOBAL_ROLES + STATE_ROLES) @@ -49,7 +55,7 @@ ROLE_STATE_APPROVER: 40, ROLE_STATE_CONTRIBUTOR: 20, ROLE_STATE_VIEW: 10, - ROLE_BH_VIEWER: 5, + ROLE_GLOBAL_VIEWER: 5, } # Leaf / realm-role spellings → canonical role. @@ -58,13 +64,11 @@ # Before the six-role model it aliased to state_admin, so never reintroduce # that mapping — it would silently re-grant edit/approve/delete. _ROLE_ALIASES: dict[str, str] = { - # Global — Bharat Vistaar platform level + # Global platform roles "super_admin": ROLE_SUPER_ADMIN, "super-admin": ROLE_SUPER_ADMIN, "superadmin": ROLE_SUPER_ADMIN, - "bh_viewer": ROLE_BH_VIEWER, - "bh-viewer": ROLE_BH_VIEWER, - "bh_view": ROLE_BH_VIEWER, + **dict.fromkeys(GLOBAL_VIEWER_ROLE_ALIASES, ROLE_GLOBAL_VIEWER), # Per state / centre "state_admin": ROLE_STATE_ADMIN, "state-admin": ROLE_STATE_ADMIN, @@ -81,15 +85,22 @@ } _STATE_GROUP_RE = re.compile( - r"^/states/(?P[A-Za-z0-9_-]+)/(?P[A-Za-z0-9_-]+)/?$", + r"^/(?P[A-Za-z0-9_-]+)/(?P[A-Za-z0-9_-]+)/(?P[A-Za-z0-9_-]+)/?$", re.IGNORECASE, ) + + +def provider_group_root() -> str: + root = os.environ.get("KEYCLOAK_PROVIDER_GROUP_ROOT", "providers").strip() + if not re.fullmatch(r"[A-Za-z0-9_-]+", root) or root.casefold() == "global": + raise ValueError("KEYCLOAK_PROVIDER_GROUP_ROOT must name one provider group root") + return root _GLOBAL_SUPER_RE = re.compile( r"^/global/(?Psuper[_-]?admin|superadmin)/?$", re.IGNORECASE, ) _GLOBAL_VIEWER_RE = re.compile( - r"^/global/(?Pbh[_-]?viewer|bh[_-]?view)/?$", + r"^/global/(?Pglobal[_-]viewer|bh[_-]?viewer|bh[_-]?view)/?$", re.IGNORECASE, ) @@ -120,8 +131,7 @@ class GroupAccess: """Access derived from Keycloak ``groups`` claim paths.""" is_super_admin: bool = False - # Bharat Vistaar viewer: every state, read-only. - is_bh_viewer: bool = False + is_global_viewer: bool = False # instance (state) → highest role in that state state_roles: dict[str, str] = field(default_factory=dict) # flat list of group paths as received @@ -146,6 +156,7 @@ def parse_group_paths(groups: Iterable[str] | None) -> GroupAccess: access = GroupAccess() if not groups: return access + group_root = provider_group_root().casefold() paths: list[str] = [] for raw in groups: @@ -165,14 +176,14 @@ def parse_group_paths(groups: Iterable[str] | None) -> GroupAccess: continue if _GLOBAL_VIEWER_RE.match(path): - access.is_bh_viewer = True + access.is_global_viewer = True continue state_match = _STATE_GROUP_RE.match(path) - if not state_match: - # Also accept /states/{STATE} alone as "member of state" with view + if not state_match or state_match.group("root").casefold() != group_root: + # A membership in the provider group itself grants read access. parts = [p for p in path.split("/") if p] - if len(parts) == 2 and parts[0].lower() == "states": + if len(parts) == 2 and parts[0].casefold() == group_root: state = normalize_state_code(parts[1]) if state: access.state_roles.setdefault(state, ROLE_STATE_VIEW) @@ -180,14 +191,7 @@ def parse_group_paths(groups: Iterable[str] | None) -> GroupAccess: state = normalize_state_code(state_match.group("state")) role = normalize_role(state_match.group("role")) - if not state or not role: - continue - if role == ROLE_SUPER_ADMIN: - # Super-admin under a state still means full platform access. - access.is_super_admin = True - continue - if role == ROLE_BH_VIEWER: - access.is_bh_viewer = True + if not state or role not in STATE_ROLES: continue access.state_roles[state] = _prefer_role(access.state_roles.get(state), role) @@ -195,8 +199,8 @@ def parse_group_paths(groups: Iterable[str] | None) -> GroupAccess: role_set: set[str] = set() if access.is_super_admin: role_set.add(ROLE_SUPER_ADMIN) - if access.is_bh_viewer: - role_set.add(ROLE_BH_VIEWER) + if access.is_global_viewer: + role_set.add(ROLE_GLOBAL_VIEWER) role_set.update(access.state_roles.values()) access.roles = sorted(role_set, key=lambda r: (-_ROLE_RANK.get(r, 0), r)) return access @@ -220,7 +224,7 @@ def extract_groups_claim(claims: dict[str, Any]) -> list[str]: def role_for_instance(access: GroupAccess, instance: str | None) -> str | None: """Return the user's role for a state instance, or SUPER_ADMIN when global. - A ``bh_viewer`` has no per-state group, but reads every state — so any + A ``global_viewer`` reads every provider, so any instance resolves to STATE_VIEW unless a stronger state role is held. """ if access.is_super_admin: @@ -230,13 +234,13 @@ def role_for_instance(access: GroupAccess, instance: str | None) -> str | None: role = access.state_roles.get(normalize_state_code(instance)) if role: return role - return ROLE_STATE_VIEW if access.is_bh_viewer else None + return ROLE_STATE_VIEW if access.is_global_viewer else None # Canonical role → Keycloak group leaf name. _ROLE_LEAF: dict[str, str] = { ROLE_SUPER_ADMIN: "super-admin", - ROLE_BH_VIEWER: "bh-viewer", + ROLE_GLOBAL_VIEWER: "global-viewer", ROLE_STATE_ADMIN: "admin", ROLE_STATE_APPROVER: "approver", ROLE_STATE_CONTRIBUTOR: "contributor", diff --git a/pipeline/auth/jwt.py b/pipeline/auth/jwt.py index 3bd7176..26cc69e 100644 --- a/pipeline/auth/jwt.py +++ b/pipeline/auth/jwt.py @@ -28,7 +28,7 @@ def clear_jwks_cache() -> None: _jwks_clients.clear() -def _extract_roles(claims: dict[str, Any]) -> list[str]: +def _extract_roles(claims: dict[str, Any], client_id: str | None) -> list[str]: roles: set[str] = set() realm = claims.get("realm_access") or {} for role in realm.get("roles") or []: @@ -36,18 +36,13 @@ def _extract_roles(claims: dict[str, Any]) -> list[str]: roles.add(role.strip()) resource_access = claims.get("resource_access") or {} - if isinstance(resource_access, dict): - for client_data in resource_access.values(): - if not isinstance(client_data, dict): - continue + if client_id and isinstance(resource_access, dict): + client_data = resource_access.get(client_id) + if isinstance(client_data, dict): for role in client_data.get("roles") or []: if isinstance(role, str) and role.strip(): roles.add(role.strip()) - for role in claims.get("roles") or []: - if isinstance(role, str) and role.strip(): - roles.add(role.strip()) - return sorted(roles) @@ -65,7 +60,9 @@ def _extract_string_list(claims: dict[str, Any], *keys: str) -> list[str]: return [] -def claims_to_user(claims: dict[str, Any]) -> AuthUser: +def claims_to_user( + claims: dict[str, Any], *, authoritative_groups: bool = False, client_id: str | None = None, +) -> AuthUser: """Map Keycloak JWT claims → AuthUser. Preferred source of tenant + role is the ``groups`` claim (full paths): @@ -74,7 +71,7 @@ def claims_to_user(claims: dict[str, Any]) -> AuthUser: Fallback (legacy tokens): realm roles + multivalued ``instances`` / ``tenants`` user attributes. """ - raw_realm_roles = _extract_roles(claims) + raw_realm_roles = [] if authoritative_groups else _extract_roles(claims, client_id) group_paths = extract_groups_claim(claims) group_access = parse_group_paths(group_paths) @@ -96,7 +93,7 @@ def claims_to_user(claims: dict[str, Any]) -> AuthUser: # Instances: prefer group-derived states; fall back to attribute claims. if group_access.is_super_admin: instances: list[str] = [] - elif group_access.state_roles: + elif authoritative_groups or group_access.state_roles: instances = group_access.instances else: instances = [ @@ -119,11 +116,11 @@ def claims_to_user(claims: dict[str, Any]) -> AuthUser: ) -def decode_and_validate_token(token: str, config: AuthConfig) -> AuthUser: - if not config.keycloak_issuer or not config.keycloak_jwks_url: +def decode_verified_claims(token: str, config: AuthConfig) -> dict[str, Any]: + if not config.keycloak_issuer or not config.keycloak_jwks_url or not config.keycloak_audience: raise HTTPException( 401, - "Auth is enabled but KEYCLOAK_ISSUER / KEYCLOAK_JWKS_URL are not configured", + "Keycloak token verification is not configured", ) try: @@ -134,19 +131,21 @@ def decode_and_validate_token(token: str, config: AuthConfig) -> AuthUser: "issuer": config.keycloak_issuer, "leeway": config.jwt_leeway_seconds, "options": options, + "audience": config.keycloak_audience, } - if config.keycloak_audience: - decode_kwargs["audience"] = config.keycloak_audience - else: - options["verify_aud"] = False claims = jwt.decode(token, signing_key.key, **decode_kwargs) except jwt.PyJWTError as exc: - raise HTTPException(401, f"Invalid token: {exc}") from exc + raise HTTPException(401, "Invalid authentication token") from exc except Exception as exc: - raise HTTPException(401, f"Unable to validate token: {exc}") from exc + raise HTTPException(401, "Unable to validate authentication token") from exc - user = claims_to_user(claims) + return claims + + +def decode_and_validate_token(token: str, config: AuthConfig) -> AuthUser: + claims = decode_verified_claims(token, config) + user = claims_to_user(claims, client_id=config.keycloak_audience) if not user.user_id: raise HTTPException(401, "Token missing subject") return user diff --git a/pipeline/auth/keycloak_admin.py b/pipeline/auth/keycloak_admin.py index 0b67380..0484745 100644 --- a/pipeline/auth/keycloak_admin.py +++ b/pipeline/auth/keycloak_admin.py @@ -12,13 +12,15 @@ import urllib.error import urllib.parse import urllib.request -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import Any from fastapi import HTTPException from .groups import ( - ROLE_BH_VIEWER, + GLOBAL_VIEWER_GROUP_PATH, + GLOBAL_VIEWER_ROLE_ALIASES, + ROLE_GLOBAL_VIEWER, ROLE_STATE_ADMIN, ROLE_STATE_APPROVER, ROLE_STATE_CONTRIBUTOR, @@ -27,8 +29,10 @@ STATE_ROLES, group_leaf_for_role, normalize_role, + parse_group_paths, + provider_group_root, ) -from .tenancy import PORTAL_INSTANCE +from .tenancy import portal_instance # Realm-role descriptions written when a role is auto-created. STATE_ROLE_DESCRIPTIONS: dict[str, str] = { @@ -39,49 +43,19 @@ } # Global access types accepted by ``ensure_access_group``. -_SUPER_ADMIN_ACCESS_TYPES = ("super_admin", "super-admin", "bv", "portal") -_BH_VIEWER_ACCESS_TYPES = ("bh_viewer", "bh-viewer", "bh_view") - -# Common state codes for the admin UI picker (Keycloak groups created on demand). -DEFAULT_STATE_CODES = [ - "MH", - "BH", - "UP", - "GJ", - "RJ", - "MP", - "KA", - "TS", - "AP", - "TN", - "WB", - "OR", - "PB", - "HR", - "KL", - "AS", - "JH", - "CG", - "UK", - "HP", - "GA", - "DL", - "JK", - "LA", -] - +_SUPER_ADMIN_ACCESS_TYPES = ("super_admin", "super-admin", "portal") @dataclass class KeycloakAdminConfig: base_url: str realm: str - admin_username: str - admin_password: str - token_realm: str = "master" + client_id: str + client_secret: str = field(repr=False) + token_realm: str = "" @property def configured(self) -> bool: - return bool(self.base_url and self.realm and self.admin_username and self.admin_password) + return bool(self.base_url and self.realm and self.client_id and self.client_secret) def load_keycloak_admin_config() -> KeycloakAdminConfig: @@ -98,20 +72,12 @@ def load_keycloak_admin_config() -> KeycloakAdminConfig: if not realm: realm = realm_part.split("/")[0] - username = ( - os.environ.get("KEYCLOAK_ADMIN_USERNAME") - or os.environ.get("KEYCLOAK_ADMIN") - or "" - ).strip() - password = (os.environ.get("KEYCLOAK_ADMIN_PASSWORD") or "").strip() - token_realm = (os.environ.get("KEYCLOAK_ADMIN_TOKEN_REALM") or "master").strip() or "master" - return KeycloakAdminConfig( base_url=base, - realm=realm or "bharat-vistaar", - admin_username=username, - admin_password=password, - token_realm=token_realm, + realm=realm, + token_realm=(os.environ.get("KEYCLOAK_ADMIN_TOKEN_REALM") or realm).strip(), + client_id=(os.environ.get("KEYCLOAK_ADMIN_CLIENT_ID") or "").strip(), + client_secret=os.environ.get("KEYCLOAK_ADMIN_CLIENT_SECRET") or "", ) @@ -152,15 +118,12 @@ def _req( def _admin_token(cfg: KeycloakAdminConfig) -> str: + token_realm = cfg.token_realm or cfg.realm + form = {"grant_type": "client_credentials", "client_id": cfg.client_id, "client_secret": cfg.client_secret} status, body = _req( "POST", - f"{cfg.base_url}/realms/{cfg.token_realm}/protocol/openid-connect/token", - form={ - "grant_type": "password", - "client_id": "admin-cli", - "username": cfg.admin_username, - "password": cfg.admin_password, - }, + f"{cfg.base_url}/realms/{token_realm}/protocol/openid-connect/token", + form=form, ) if status != 200 or not isinstance(body, dict) or not body.get("access_token"): raise HTTPException(502, "Unable to obtain Keycloak admin token") @@ -176,13 +139,46 @@ def require_admin_config() -> KeycloakAdminConfig: if not cfg.configured: raise HTTPException( 503, - "Keycloak admin is not configured. Set KEYCLOAK_ADMIN_USERNAME and " - "KEYCLOAK_ADMIN_PASSWORD (and KEYCLOAK_ISSUER or KEYCLOAK_ADMIN_BASE_URL / " - "KEYCLOAK_ADMIN_REALM).", + "Keycloak admin is not configured. Configure a realm service account.", ) return cfg +def _user_admin_url(admin: str, user_id: str) -> str: + if not re.fullmatch(r"[A-Za-z0-9:_-]{1,200}", user_id): + raise HTTPException(400, "Invalid Keycloak user ID") + return f"{admin}/users/{urllib.parse.quote(user_id, safe='')}" + + +def _list_keycloak(admin_url: str, token: str) -> list[dict]: + page_size = 100 + result = [] + while True: + _, page = _req("GET", f"{admin_url}?first={len(result)}&max={page_size}", token=token) + if not isinstance(page, list) or any(not isinstance(item, dict) for item in page): + raise HTTPException(502, "Invalid Keycloak list response") + result.extend(page) + if len(page) < page_size: + return result + + +def current_user_group_paths(user_id: str) -> list[str]: + """Read current memberships, including removal since the JWT was issued.""" + cfg = require_admin_config() + try: + user_url = _user_admin_url(_admin_root(cfg), user_id) + token = _admin_token(cfg) + _, identity = _req("GET", user_url, token=token) + if not isinstance(identity, dict) or identity.get("enabled") is not True: + raise HTTPException(401, "Keycloak user is inactive") + groups = _list_keycloak(f"{user_url}/groups", token) + except HTTPException as exc: + if exc.status_code in (400, 401, 404): + raise HTTPException(401, "Keycloak user access could not be verified") from exc + raise HTTPException(503, "Unable to verify current Keycloak membership") from exc + return [group["path"] for group in groups if isinstance(group.get("path"), str)] + + def _ensure_realm_role(admin: str, token: str, name: str, description: str = "") -> dict: try: status, role = _req("GET", f"{admin}/roles/{urllib.parse.quote(name)}", token=token) @@ -231,8 +227,7 @@ def _get_children(admin: str, token: str, parent_id: str) -> list[dict]: same way — surfacing as "Failed to create child group " even though the group was there all along. """ - status, children = _req("GET", f"{admin}/groups/{parent_id}/children?max=200", token=token) - return children if isinstance(children, list) else [] + return _list_keycloak(f"{admin}/groups/{parent_id}/children", token) def _find_child(admin: str, token: str, parent_id: str, name: str) -> dict | None: @@ -244,9 +239,7 @@ def _find_child(admin: str, token: str, parent_id: str, name: str) -> dict | Non def _ensure_child_group(admin: str, token: str, parent_id: str | None, name: str) -> dict: if parent_id is None: - status, top = _req("GET", f"{admin}/groups?max=200", token=token) - groups = top if isinstance(top, list) else [] - for g in groups: + for g in _list_keycloak(f"{admin}/groups", token): if g.get("name") == name: return _get_group(admin, token, g["id"]) try: @@ -254,8 +247,7 @@ def _ensure_child_group(admin: str, token: str, parent_id: str | None, name: str except HTTPException as exc: if exc.status_code not in (409,): raise - status, top = _req("GET", f"{admin}/groups?max=200", token=token) - for g in top or []: + for g in _list_keycloak(f"{admin}/groups", token): if g.get("name") == name: return _get_group(admin, token, g["id"]) raise HTTPException(502, f"Failed to create group {name}") @@ -287,6 +279,121 @@ def _map_role_on_group(admin: str, token: str, group_id: str, role: dict) -> Non ) +def _provider_group(admin: str, token: str, provider: str, *, create: bool = False) -> dict: + name = provider.upper() + root = provider_group_root() + if create: + providers = _ensure_child_group(admin, token, None, root) + return _ensure_child_group(admin, token, providers["id"], name) + _, group = _req("GET", f"{admin}/group-by-path/{root}/{name}", token=token) + if not isinstance(group, dict) or group.get("path") != f"/{root}/{name}" or not group.get("id"): + raise HTTPException(502, "Invalid Keycloak provider group") + return group + + +def create_provider_group(provider: str) -> dict: + """Idempotently create the Keycloak group for a provider code.""" + cfg = require_admin_config() + return _provider_group(_admin_root(cfg), _admin_token(cfg), provider, create=True) + + +def find_verified_user_id(email: str) -> str: + """Resolve one existing and verified account for a provider grant.""" + cfg = require_admin_config() + params = urllib.parse.urlencode({"email": email, "exact": "true"}) + _, users = _req("GET", f"{_admin_root(cfg)}/users?{params}", token=_admin_token(cfg)) + if not isinstance(users, list): + raise HTTPException(502, "Invalid Keycloak account lookup response") + verified = [user for user in users if ( + str(user.get("email", "")).lower() == email.lower() + and user.get("emailVerified") is True and user.get("enabled") is True + )] + if len(verified) != 1 or not verified[0].get("id"): + raise HTTPException(404, "An active, verified Keycloak account is required") + return verified[0]["id"] + + +def find_webhook_service_subject(client_id: str) -> str: + """Return the enabled service account for an existing confidential client.""" + cfg = require_admin_config() + admin, token = _admin_root(cfg), _admin_token(cfg) + params = urllib.parse.urlencode({"clientId": client_id, "search": "false", "max": 2}) + _, rows = _req("GET", f"{admin}/clients?{params}", token=token) + if not isinstance(rows, list): + raise HTTPException(502, "Invalid Keycloak client lookup response") + clients = [row for row in rows if isinstance(row, dict) and row.get("clientId") == client_id] + if len(clients) != 1: + raise HTTPException(404, "Keycloak webhook client not found") + client = clients[0] + if (client.get("enabled") is not True or client.get("serviceAccountsEnabled") is not True + or client.get("publicClient") is not False or not client.get("id")): + raise HTTPException(422, "The webhook client must be confidential with an enabled service account") + client_uuid = urllib.parse.quote(str(client["id"]), safe="") + _, account = _req("GET", f"{admin}/clients/{client_uuid}/service-account-user", token=token) + if not isinstance(account, dict) or account.get("enabled") is not True or not account.get("id"): + raise HTTPException(422, "An active Keycloak service account is required") + return str(account["id"]) + + +def set_provider_membership( + provider: str, user_id: str, role: str | None, *, allow_admin_changes: bool, +) -> dict: + """Set or remove one provider role, leaving all other scopes in place.""" + if role is not None: + role = normalize_role(role) + if role not in STATE_ROLES: + raise HTTPException(400, "Invalid provider role") + cfg = require_admin_config() + token, admin = _admin_token(cfg), _admin_root(cfg) + provider_group = _provider_group(admin, token, provider) + user_url = _user_admin_url(admin, user_id) + groups_url = f"{user_url}/groups" + current = _list_keycloak(groups_url, token) + provider_groups = [group for group in current if provider in parse_group_paths([group.get("path")]).state_roles] + if not allow_admin_changes and ( + role == ROLE_STATE_ADMIN + or any(parse_group_paths([group.get("path")]).state_roles[provider] == ROLE_STATE_ADMIN for group in provider_groups) + ): + raise HTTPException(403, "Only a platform administrator can manage provider administrators") + + desired_group_id = None + if role: + group = _ensure_child_group(admin, token, provider_group["id"], group_leaf_for_role(role)) + desired_group_id = group["id"] + if all(existing["id"] != desired_group_id for existing in provider_groups): + _req("PUT", f"{groups_url}/{desired_group_id}", token=token) + for existing in provider_groups: + if existing["id"] != desired_group_id: + _req("DELETE", f"{groups_url}/{existing['id']}", token=token) + final_groups = _list_keycloak(groups_url, token) + final_ids = {group["id"] for group in final_groups if provider in parse_group_paths([group.get("path")]).state_roles} + if final_ids != ({desired_group_id} if desired_group_id else set()): + raise HTTPException(502, "Keycloak provider membership change could not be confirmed") + return {"provider": provider, "user_id": user_id, "role": role} + + +def list_provider_memberships(provider: str) -> list[dict]: + """List only the requested provider's users and provider roles.""" + cfg = require_admin_config() + token, admin = _admin_token(cfg), _admin_root(cfg) + provider_group = _provider_group(admin, token, provider) + groups = [provider_group, *_get_children(admin, token, provider_group["id"])] + members: dict[str, dict] = {} + for group in groups: + role = parse_group_paths([group.get("path")]).state_roles.get(provider) + if not role: + continue + for member in _list_keycloak(f"{admin}/groups/{group['id']}/members", token): + user_id = member["id"] + current = members.get(user_id) + if current is None or STATE_ROLES.index(role) < STATE_ROLES.index(current["role"]): + members[user_id] = { + "user_id": user_id, "role": role, "email": member.get("email") or "", + "username": member.get("username") or "", "enabled": member.get("enabled") is True, + } + return sorted(members.values(), key=lambda member: (member["email"], member["user_id"])) + + def ensure_access_group( cfg: KeycloakAdminConfig, *, @@ -299,15 +406,15 @@ def ensure_access_group( admin = _admin_root(cfg) access_type = (access_type or "").strip().lower() - if access_type in _BH_VIEWER_ACCESS_TYPES: + if access_type in GLOBAL_VIEWER_ROLE_ALIASES: viewer_role = _ensure_realm_role( - admin, token, ROLE_BH_VIEWER, "Bharat Vistaar viewer — all states, read-only" + admin, token, ROLE_GLOBAL_VIEWER, "Viewer with read-only access to all providers" ) global_g = _ensure_child_group(admin, token, None, "global") - leaf = _ensure_child_group(admin, token, global_g["id"], "bh-viewer") + leaf = _ensure_child_group(admin, token, global_g["id"], group_leaf_for_role(ROLE_GLOBAL_VIEWER)) leaf_full = _get_group(admin, token, leaf["id"]) _map_role_on_group(admin, token, leaf_full["id"], viewer_role) - return leaf_full.get("path") or "/global/bh-viewer", leaf_full + return leaf_full.get("path") or GLOBAL_VIEWER_GROUP_PATH, leaf_full if access_type in _SUPER_ADMIN_ACCESS_TYPES: sa_role = _ensure_realm_role( @@ -331,8 +438,8 @@ def ensure_access_group( state_code = (state or "").strip().upper() role_canon = normalize_role(role) or (role or "").strip().lower() - if not state_code or not re.fullmatch(r"[A-Z0-9]{2,5}", state_code): - raise HTTPException(400, "state must be a 2–5 letter code (e.g. MH, UP)") + if not re.fullmatch(r"[A-Z0-9](?:[A-Z0-9_-]{0,61}[A-Z0-9])?", state_code): + raise HTTPException(400, "Invalid provider code") if role_canon not in STATE_ROLES: raise HTTPException( 400, @@ -347,11 +454,10 @@ def ensure_access_group( ) # Group leaf: /states/MH/admin or /states/MH/view leaf_name = group_leaf_for_role(role_canon) - states = _ensure_child_group(admin, token, None, "states") - state_g = _ensure_child_group(admin, token, states["id"], state_code) + state_g = _provider_group(admin, token, state_code, create=False) leaf = _ensure_child_group(admin, token, state_g["id"], leaf_name) _map_role_on_group(admin, token, leaf["id"], realm_role) - path = leaf.get("path") or f"/states/{state_code}/{leaf_name}" + path = leaf.get("path") or f"/{provider_group_root()}/{state_code}/{leaf_name}" return path, leaf @@ -407,6 +513,11 @@ def provision_user( if isinstance(found, list) and found: uid = found[0]["id"] body["username"] = found[0].get("username") or uname + actions = found[0].get("requiredActions") + if isinstance(actions, list): + body["requiredActions"] = [action for action in actions if action != "UPDATE_PASSWORD"] + else: + body.pop("requiredActions") _req("PUT", f"{admin}/users/{uid}", token=token, body=body) else: try: @@ -441,11 +552,11 @@ def provision_user( # JWT through the group. Deliberately no direct user→role mapping here: # direct assignments produce users who carry permissions with no tenant # scope, and they are invisible in the group tree. - access_key = (access_type or "").lower() + access_key = (access_type or "").strip().lower() if access_key in _SUPER_ADMIN_ACCESS_TYPES: role_name = ROLE_SUPER_ADMIN - elif access_key in _BH_VIEWER_ACCESS_TYPES: - role_name = ROLE_BH_VIEWER + elif access_key in GLOBAL_VIEWER_ROLE_ALIASES: + role_name = ROLE_GLOBAL_VIEWER else: role_name = normalize_role(role) or (role or "").strip().lower() @@ -457,9 +568,9 @@ def provision_user( login_url = f"{ui_url}/login" if not ui_url.endswith("/login") else ui_url if role_name == ROLE_SUPER_ADMIN: - access_label = "Super Admin (Bharat Vistaar — all states)" - elif role_name == ROLE_BH_VIEWER: - access_label = "BH Viewer (Bharat Vistaar — all states, read-only)" + access_label = "Platform administrator" + elif role_name == ROLE_GLOBAL_VIEWER: + access_label = "Viewer" else: access_label = f"{(state or '').upper()} · {(role or '').capitalize()}" @@ -495,8 +606,8 @@ def provision_user( "access_type": ( "super_admin" if role_name == ROLE_SUPER_ADMIN - else "bh_viewer" - if role_name == ROLE_BH_VIEWER + else ROLE_GLOBAL_VIEWER + if role_name == ROLE_GLOBAL_VIEWER else "state" ), "state": (state or "").upper() or None, @@ -506,39 +617,50 @@ def provision_user( "roles": [r.get("name") for r in (roles or []) if isinstance(r, dict)], "app_url": ui_url, "login_url": login_url, - "portal_instance": PORTAL_INSTANCE, + "portal_instance": portal_instance(), "share_message": "\n".join(share_lines), } def list_access_options() -> dict[str, Any]: - """Static options for the admin form (no Keycloak call required).""" + """List the provider groups that exist in this Keycloak realm.""" cfg = load_keycloak_admin_config() + root_name = provider_group_root() + providers = [] + if cfg.configured: + admin, token = _admin_root(cfg), _admin_token(cfg) + root = next((group for group in _list_keycloak(f"{admin}/groups", token) if group.get("name") == root_name), None) + if root: + for group in _get_children(admin, token, root["id"]): + name = group.get("name", "") + if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", name): + continue + providers.append({"code": name.upper(), "label": name}) return { "keycloak_admin_configured": cfg.configured, "realm": cfg.realm, - "portal_instance": PORTAL_INSTANCE, + "portal_instance": portal_instance(), "access_types": [ { "id": "super_admin", - "label": "Super Admin (Bharat Vistaar)", + "label": "Platform administrator", "description": "Full access to all states, settings, user management, and PROD approval", "group": "/global/super-admin", }, { - "id": "bh_viewer", - "label": "BH Viewer (Bharat Vistaar)", - "description": "Read-only access across all states. No upload, edit, or approve.", - "group": "/global/bh-viewer", + "id": ROLE_GLOBAL_VIEWER, + "label": "Viewer", + "description": "Read-only access across all providers.", + "group": GLOBAL_VIEWER_GROUP_PATH, }, { "id": "state", - "label": "State / Centre role", + "label": "Provider role", "description": ( "Access limited to one state or centre, as Admin, Approver, " "Contributor, or View" ), - "group_template": "/states/{STATE}/{admin|approver|contributor|view}", + "group_template": f"/{root_name}/{{PROVIDER}}/{{admin|approver|contributor|view}}", }, ], "state_roles": [ @@ -570,21 +692,19 @@ def list_access_options() -> dict[str, Any]: "description": "View-only access within the state (search / browse)", }, ], - "states": [{"code": c, "label": c} for c in DEFAULT_STATE_CODES], + "states": sorted(providers, key=lambda provider: provider["code"]), "required_fields": { "super_admin": ["email", "first_name", "last_name"], - "bh_viewer": ["email", "first_name", "last_name"], + ROLE_GLOBAL_VIEWER: ["email", "first_name", "last_name"], "state": ["email", "first_name", "last_name", "state", "role"], }, "notes": [ "Users sign in with Google SSO using the same email you enter.", "No app password is required for normal SSO login.", - "State codes must match document instance tags (mh, up, …).", - "A centre is modelled as just another state code.", - "Super admin (BV) documents can use portal instance 'bv'.", + "Provider codes match the instance codes used by documents.", "Access comes from group membership only — no direct role assignment.", - "Keycloak groups: /global/super-admin, /global/bh-viewer, " - "/states/{STATE}/{admin|approver|contributor|view}", + f"Keycloak groups: /global/super-admin, {GLOBAL_VIEWER_GROUP_PATH}, " + f"/{root_name}/{{PROVIDER}}/{{admin|approver|contributor|view}}", ], } @@ -592,14 +712,10 @@ def list_access_options() -> dict[str, Any]: def _is_product_group_path(path: str) -> bool: """True when a group path grants a product role. - ``/global/super-admin``, ``/global/bh-viewer``, or ``/states/{CODE}/{leaf}``. + ``/global/super-admin``, ``/global/global-viewer``, or ``/states/{CODE}/{leaf}``. Anything else (org units, ad-hoc groups) is left alone when changing a role. """ - clean = (path or "").rstrip("/") - if clean in ("/global/super-admin", "/global/bh-viewer"): - return True - parts = [p for p in clean.split("/") if p] - return len(parts) == 3 and parts[0].lower() == "states" + return bool(parse_group_paths([path]).roles) def set_user_access( @@ -609,29 +725,22 @@ def set_user_access( state: str | None = None, role: str | None = None, ) -> dict[str, Any]: - """Move an existing user to exactly one product role. - - A user holds at most ONE product group: the target is joined and every - other product group is left, so roles never stack. Direct realm-role - assignments are stripped too — group membership is the only source of - access, and a leftover direct role would silently outrank the new group. - - The change only reaches the user's JWT on their next login. - """ + """Update the user's role in the selected provider or platform scope.""" cfg = require_admin_config() token = _admin_token(cfg) admin = _admin_root(cfg) - status, target_user = _req("GET", f"{admin}/users/{user_id}", token=token) + user_url = _user_admin_url(admin, user_id) + status, target_user = _req("GET", user_url, token=token) if status != 200 or not target_user: raise HTTPException(404, "User not found") path, group = ensure_access_group(cfg, access_type=access_type, state=state, role=role) - _req("PUT", f"{admin}/users/{user_id}/groups/{group['id']}", token=token) + _req("PUT", f"{user_url}/groups/{group['id']}", token=token) - # Exactly one product group. - _, current = _req("GET", f"{admin}/users/{user_id}/groups", token=token) + target_scope = parse_group_paths([path]).state_roles.keys() + current = _list_keycloak(f"{user_url}/groups", token) removed: list[str] = [] for existing in current or []: existing_path = existing.get("path") or "" @@ -639,25 +748,28 @@ def set_user_access( continue if not _is_product_group_path(existing_path): continue - _req("DELETE", f"{admin}/users/{user_id}/groups/{existing['id']}", token=token) + existing_scope = parse_group_paths([existing_path]).state_roles.keys() + if existing_scope != target_scope: + continue + _req("DELETE", f"{user_url}/groups/{existing['id']}", token=token) removed.append(existing_path) # Groups are the only source of roles. - _, mapped = _req("GET", f"{admin}/users/{user_id}/role-mappings/realm", token=token) + _, mapped = _req("GET", f"{user_url}/role-mappings/realm", token=token) stripped: list[str] = [] for realm_role in mapped or []: name = realm_role.get("name") or "" - if name in ("offline_access", "uma_authorization") or name.startswith("default-roles"): + if not normalize_role(name): continue _req( "DELETE", - f"{admin}/users/{user_id}/role-mappings/realm", + f"{user_url}/role-mappings/realm", token=token, body=[{"id": realm_role["id"], "name": name}], ) stripped.append(name) - _, after = _req("GET", f"{admin}/users/{user_id}/groups", token=token) + after = _list_keycloak(f"{user_url}/groups", token) paths = [g.get("path") or "" for g in (after or [])] summary = _access_summary_from_groups(paths) @@ -681,66 +793,30 @@ def _access_summary_from_groups(group_paths: list[str]) -> dict[str, Any]: Users without product groups (super_admin / state roles) are labeled **Dashboard** — baseline SSO access (search/view only), not an error state. """ - paths = [p or "" for p in group_paths] - if any( - "/global/super-admin" in p - or p.rstrip("/").endswith("/super-admin") - or p.rstrip("/").endswith("global/super-admin") - for p in paths - ): + access = parse_group_paths(group_paths) + if access.is_super_admin: return { "access_type": "super_admin", "access_label": "Super Admin", "states": [], "roles": [ROLE_SUPER_ADMIN], } - if any( - p.rstrip("/").endswith("/bh-viewer") or p.rstrip("/").endswith("/bh_viewer") - for p in paths - ): + if access.is_global_viewer: return { - "access_type": "bh_viewer", - "access_label": "BH Viewer", + "access_type": ROLE_GLOBAL_VIEWER, + "access_label": "Viewer", "states": [], - "roles": [ROLE_BH_VIEWER], + "roles": access.roles, + "provider_roles": dict(access.state_roles), } - states: list[str] = [] - roles: list[str] = [] - for p in paths: - m = re.match(r"^/states/([A-Za-z0-9_-]+)/([A-Za-z0-9_-]+)/?$", p, re.I) - if not m: - continue - states.append(m.group(1).upper()) - leaf = m.group(2).lower().replace("-", "_") - roles.append(normalize_role(leaf) or leaf) - # unique preserve order - seen_s: set[str] = set() - uniq_states = [] - for s in states: - if s not in seen_s: - seen_s.add(s) - uniq_states.append(s) - seen_r: set[str] = set() - uniq_roles = [] - for r in roles: - if r not in seen_r: - seen_r.add(r) - uniq_roles.append(r) - - if uniq_states: - # e.g. "MH · contributor" or multi "MH · contributor, UP · reviewer" - pairs = [] - if len(uniq_states) == len(uniq_roles): - pairs = [f"{s} · {r}" for s, r in zip(uniq_states, uniq_roles)] - elif uniq_roles: - pairs = [f"{', '.join(uniq_states)} · {', '.join(uniq_roles)}"] - else: - pairs = list(uniq_states) + if access.state_roles: + memberships = sorted(access.state_roles.items()) return { "access_type": "state", - "access_label": ", ".join(pairs), - "states": uniq_states, - "roles": uniq_roles, + "access_label": ", ".join(f"{provider.upper()} · {role}" for provider, role in memberships), + "states": [provider.upper() for provider, _ in memberships], + "roles": access.roles, + "provider_roles": dict(access.state_roles), } # No product group → baseline dashboard / search-only access after SSO @@ -800,12 +876,13 @@ def list_realm_users(*, search: str = "", max_results: int = 100) -> dict[str, A "access_label": summary["access_label"], "states": summary["states"], "roles": summary["roles"], + "provider_roles": summary.get("provider_roles", {}), } ) # Prefer users with product groups first, then by email def sort_key(row: dict) -> tuple: - rank = 0 if row.get("access_type") in ("super_admin", "bh_viewer", "state") else 1 + rank = 0 if row.get("access_type") in ("super_admin", ROLE_GLOBAL_VIEWER, "state") else 1 return (rank, (row.get("email") or row.get("username") or "").lower()) rows.sort(key=sort_key) diff --git a/pipeline/auth/models.py b/pipeline/auth/models.py index 947d21c..3481b0b 100644 --- a/pipeline/auth/models.py +++ b/pipeline/auth/models.py @@ -4,7 +4,12 @@ from dataclasses import dataclass, field -from .groups import ROLE_STATE_VIEW, ROLE_SUPER_ADMIN, normalize_state_code +from .groups import ( + GLOBAL_VIEWER_ROLE_ALIASES, + ROLE_STATE_VIEW, + ROLE_SUPER_ADMIN, + normalize_state_code, +) from .permissions import Permission # Platform superadmin — full permissions AND every instance. @@ -16,20 +21,9 @@ } ) -# Bharat Vistaar viewer — every instance, but only the permissions the role -# actually grants (search). Deliberately NOT a superadmin: keeping these sets -# separate is what stops a read-only global account from gaining write access. -GLOBAL_READ_ROLES = frozenset( - { - "bh_viewer", - "bh-viewer", - "bh_view", - } -) - # Roles not limited by the JWT ``instances`` claim. # State-level roles are restricted to their claimed instances (tenants/states). -INSTANCE_UNRESTRICTED_ROLES = SUPERADMIN_ROLES | GLOBAL_READ_ROLES +INSTANCE_UNRESTRICTED_ROLES = SUPERADMIN_ROLES | GLOBAL_VIEWER_ROLE_ALIASES @dataclass @@ -61,9 +55,9 @@ def is_superadmin(self) -> bool: return bool(SUPERADMIN_ROLES & self._role_keys) @property - def is_bh_viewer(self) -> bool: - """True for the Bharat Vistaar global read-only role.""" - return bool(GLOBAL_READ_ROLES & self._role_keys) + def is_global_viewer(self) -> bool: + """True for the global read-only role.""" + return bool(GLOBAL_VIEWER_ROLE_ALIASES & self._role_keys) @property def is_admin(self) -> bool: @@ -76,8 +70,8 @@ def is_admin(self) -> bool: def is_instance_unrestricted(self) -> bool: """True when the caller may access every instance (all tenants/states). - Two cases: local bypass mode with no scoped claim, or platform - superadmin even when the token carries a narrow ``instances`` claim. + Applies to unscoped local bypass and to global roles, including + viewers whose write permissions can still be scoped to an instance. """ if self.token_disabled_mode and not self.instances: return True @@ -97,7 +91,7 @@ def has_env(self, env: str) -> bool: def role_for_instance(self, instance: str | None) -> str | None: """Role in a given state; super_admin always returns super_admin. - A ``bh_viewer`` reads every state, so any instance falls back to + A ``global_viewer`` reads every provider, so any instance falls back to STATE_VIEW when no stronger per-state role is held. """ if self.is_superadmin: @@ -107,7 +101,7 @@ def role_for_instance(self, instance: str | None) -> str | None: role = self.state_roles.get(normalize_state_code(instance)) if role: return role - return ROLE_STATE_VIEW if self.is_bh_viewer else None + return ROLE_STATE_VIEW if self.is_global_viewer else None def has_permission_for_instance( self, permission: Permission | str, instance: str | None @@ -124,7 +118,9 @@ def has_permission_for_instance( return needed in self.permissions or needed in set(Permission) if not self.state_roles: - return needed in self.permissions + if self.is_global_viewer and needed == Permission.SEARCH: + return True + return needed in self.permissions and bool(instance and self.has_instance(instance)) role = self.role_for_instance(instance) if not role: diff --git a/pipeline/auth/oidc.py b/pipeline/auth/oidc.py new file mode 100644 index 0000000..708452c --- /dev/null +++ b/pipeline/auth/oidc.py @@ -0,0 +1,169 @@ +"""Keycloak token exchange and refresh for the confidential console client.""" + +from __future__ import annotations + +import json +import os +import re +import urllib.error +import urllib.parse +import urllib.request +from dataclasses import dataclass, field +from typing import Any + +from fastapi import HTTPException + + +@dataclass(frozen=True) +class OidcConfig: + issuer: str + client_id: str + client_secret: str = field(repr=False) + backchannel_url: str = "" + + @property + def token_url(self) -> str: + return f"{self.backchannel_url or self.issuer}/protocol/openid-connect/token" + + +def require_oidc_config() -> OidcConfig: + issuer = os.environ.get("KEYCLOAK_ISSUER", "").strip().rstrip("/") + client_id = os.environ.get("KEYCLOAK_CLIENT_ID", "").strip() + client_secret = os.environ.get("KEYCLOAK_CLIENT_SECRET", "") + if not (issuer and client_id and client_secret): + raise HTTPException( + 503, "Keycloak sign-in requires KEYCLOAK_ISSUER, KEYCLOAK_CLIENT_ID, and KEYCLOAK_CLIENT_SECRET." + ) + return OidcConfig( + issuer=issuer, client_id=client_id, client_secret=client_secret, + backchannel_url=os.environ.get("KEYCLOAK_BACKCHANNEL_URL", "").strip().rstrip("/"), + ) + + +def _post(url: str, *, form: dict[str, str]) -> tuple[int, Any]: + request = urllib.request.Request( + url, + data=urllib.parse.urlencode(form).encode(), + method="POST", + headers={"Accept": "application/json", "Content-Type": "application/x-www-form-urlencoded"}, + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return response.status, json.load(response) if response.status != 204 else None + except urllib.error.HTTPError as exc: + try: + return exc.code, json.load(exc) + except (ValueError, OSError): + return exc.code, None + except (ValueError, OSError) as exc: + raise HTTPException(502, "Keycloak is temporarily unavailable.") from exc + + +def _token_response(payload: dict[str, Any]) -> dict[str, Any]: + return { + "access_token": payload["access_token"], + "refresh_token": payload.get("refresh_token"), + "id_token": payload.get("id_token"), + "token_type": payload.get("token_type", "Bearer"), + "expires_in": payload.get("expires_in"), + "refresh_expires_in": payload.get("refresh_expires_in"), + } + + +def issue_service_token(client_id: str, client_secret: str) -> dict[str, Any]: + """Obtain a short-lived client credentials token through the backchannel.""" + status, body = _post(require_oidc_config().token_url, form={ + "grant_type": "client_credentials", "client_id": client_id, "client_secret": client_secret, + }) + if status == 200 and isinstance(body, dict) and isinstance(body.get("access_token"), str): + return {"access_token": body["access_token"], "token_type": "Bearer", "expires_in": body.get("expires_in")} + if status in (400, 401, 403): + raise HTTPException(401, "Keycloak rejected the service client credentials.") + raise HTTPException(502, "Keycloak could not issue a service token.") + + +def refresh_tokens(refresh_token: str) -> dict[str, Any]: + """Renew a Keycloak session using the confidential client.""" + cfg = require_oidc_config() + token = (refresh_token or "").strip() + if not token: + raise HTTPException(400, "Missing refresh token.") + + status, body = _post( + cfg.token_url, + form={ + "grant_type": "refresh_token", + "client_id": cfg.client_id, + "client_secret": cfg.client_secret, + "refresh_token": token, + }, + ) + payload = body if isinstance(body, dict) else {} + + if status == 200 and payload.get("access_token"): + return _token_response(payload) + + # An expired or revoked refresh token is a normal end-of-session, not a fault. + raise HTTPException(401, "Your session has expired. Please sign in again.") + + +def end_session(refresh_token: str) -> None: + """Revoke the console session through the private Keycloak endpoint.""" + cfg = require_oidc_config() + status, _ = _post( + f"{cfg.backchannel_url or cfg.issuer}/protocol/openid-connect/logout", + form={ + "client_id": cfg.client_id, "client_secret": cfg.client_secret, + "refresh_token": refresh_token, + }, + ) + if status not in (204, 400, 401): + raise HTTPException(502, "Could not end the identity session.") + + +def exchange_authorization_code( + code: str, redirect_uri: str, code_verifier: str | None = None +) -> dict[str, Any]: + """Exchange a PKCE-bound authorization code using the confidential client.""" + cfg = require_oidc_config() + authorization_code = (code or "").strip() + if not authorization_code: + raise HTTPException(400, "Missing authorization code.") + if not (redirect_uri or "").strip(): + raise HTTPException(400, "Missing redirect_uri.") + + if code_verifier is None or not re.fullmatch(r"[A-Za-z0-9._~-]{43,128}", code_verifier): + raise HTTPException(400, "Missing or invalid PKCE verifier. Please start sign-in again.") + + form = { + "grant_type": "authorization_code", + "code_verifier": code_verifier, + "client_id": cfg.client_id, + "client_secret": cfg.client_secret, + "code": authorization_code, + "redirect_uri": redirect_uri, + } + + status, body = _post(cfg.token_url, form=form) + payload = body if isinstance(body, dict) else {} + + if status == 200 and payload.get("access_token"): + return _token_response(payload) + + error = str(payload.get("error") or "") + + if error == "invalid_client" or error == "unauthorized_client": + raise HTTPException( + 503, + "SSO is misconfigured (Keycloak rejected the client credentials). Check " + "KEYCLOAK_CLIENT_ID / KEYCLOAK_CLIENT_SECRET.", + ) + if error == "invalid_grant": + # A code is single-use and short-lived; a reload of the callback URL + # replays a spent one, which is the common way to land here. + raise HTTPException(401, "That sign-in link has already been used or expired. Try again.") + + raise HTTPException( + 502 if status >= 500 else 400, + "Could not complete sign-in.", + ) diff --git a/pipeline/auth/permissions.py b/pipeline/auth/permissions.py index 235bb7b..36fa676 100644 --- a/pipeline/auth/permissions.py +++ b/pipeline/auth/permissions.py @@ -2,10 +2,10 @@ Product roles (Keycloak groups / realm roles): -Global (Bharat Vistaar platform level): +Global platform roles: - ``super_admin`` — all states, full access incl. settings / users / PROD -- ``bh_viewer`` — all states, view-only +- ``global_viewer`` — all states, view-only Per state / centre (a centre is just another tenant code): @@ -23,6 +23,8 @@ from enum import Enum +from .groups import GLOBAL_VIEWER_ROLE_ALIASES, ROLE_GLOBAL_VIEWER + class Permission(str, Enum): """Capability names enforced by API dependencies. @@ -49,7 +51,7 @@ class UserRole(str, Enum): """Canonical product roles (mirror Keycloak group leaves / realm roles).""" SUPER_ADMIN = "super_admin" - BH_VIEWER = "bh_viewer" + GLOBAL_VIEWER = ROLE_GLOBAL_VIEWER STATE_ADMIN = "state_admin" STATE_APPROVER = "state_approver" STATE_CONTRIBUTOR = "state_contributor" @@ -90,14 +92,14 @@ class UserRole(str, Enum): Permission.DELETE_OWN, } -# Bharat Vistaar viewer: read-only, but across every state (see models.py for scope). -BH_VIEWER_PERMISSIONS: frozenset[Permission] = frozenset( +# Global viewer: read-only across every provider. +GLOBAL_VIEWER_PERMISSIONS: frozenset[Permission] = frozenset( { Permission.SEARCH, } ) -# Platform super admin (Bharat Vistaar): full console + settings + users + prod. +# Platform super admin: full console + settings + users + prod. SUPERADMIN_PERMISSIONS: frozenset[Permission] = frozenset(Permission) # Keycloak / realm role names → permissions. @@ -105,7 +107,7 @@ class UserRole(str, Enum): ROLE_PERMISSIONS: dict[str, frozenset[Permission]] = { # Product roles UserRole.SUPER_ADMIN.value: SUPERADMIN_PERMISSIONS, - UserRole.BH_VIEWER.value: BH_VIEWER_PERMISSIONS, + **dict.fromkeys(GLOBAL_VIEWER_ROLE_ALIASES, GLOBAL_VIEWER_PERMISSIONS), UserRole.STATE_ADMIN.value: STATE_ADMIN_PERMISSIONS, UserRole.STATE_APPROVER.value: STATE_APPROVER_PERMISSIONS, UserRole.STATE_CONTRIBUTOR.value: STATE_CONTRIBUTOR_PERMISSIONS, @@ -114,9 +116,6 @@ class UserRole(str, Enum): # and stays mapped until every holder is in /global/super-admin. "superadmin": SUPERADMIN_PERMISSIONS, "super-admin": SUPERADMIN_PERMISSIONS, - # Bharat Vistaar viewer spellings - "bh-viewer": BH_VIEWER_PERMISSIONS, - "bh_view": BH_VIEWER_PERMISSIONS, # Group-leaf spellings for the state roles "admin": STATE_ADMIN_PERMISSIONS, "state-admin": STATE_ADMIN_PERMISSIONS, @@ -143,7 +142,7 @@ def permissions_for_roles(roles: list[str] | set[str] | tuple[str, ...]) -> set[ """Union permissions from known roles. - ``super_admin`` → all permissions - - ``bh_viewer`` → search only (across all states — see models.py) + - ``global_viewer`` → search only (across all states — see models.py) - ``state_admin`` → search, upload, pipeline, review, approve_ingestion, delete_own - ``state_approver`` → search, upload, pipeline, review, approve_ingestion - ``state_contributor`` → search, upload, pipeline, review (no DEV publish) diff --git a/pipeline/auth/private_browser.py b/pipeline/auth/private_browser.py new file mode 100644 index 0000000..1cd460c --- /dev/null +++ b/pipeline/auth/private_browser.py @@ -0,0 +1,45 @@ +"""HTTP used by Python to complete a private Keycloak browser session.""" + +import logging +from contextlib import contextmanager +from contextvars import ContextVar + +import httpx + +from .oidc import OidcConfig + +_private_http = ContextVar("private_sign_in_http", default=False) + + +def _log_public_http(_record: logging.LogRecord) -> bool: + return not _private_http.get() + + +# Keep authentication session URLs and cookies out of HTTP request logs. +for _logger in ("httpx", "httpcore.connection", "httpcore.http11", "httpcore.proxy"): + logging.getLogger(_logger).addFilter(_log_public_http) + + +@contextmanager +def private_client(cookies: dict[str, str] | None = None): + marker = _private_http.set(True) + try: + with httpx.Client(cookies=cookies, timeout=10, trust_env=False, follow_redirects=False) as client: + yield client + finally: + _private_http.reset(marker) + + +def private_cookie_header(client: httpx.Client) -> dict[str, str]: + # Actions are resolved to the configured private realm. Forward its session + # cookies explicitly when TLS terminates before the private Keycloak hop. + cookie = "; ".join(f"{item.name}={item.value}" for item in client.cookies.jar if not item.is_expired()) + return {"Cookie": cookie} + + +def submit_private_form(client: httpx.Client, action: str, fields: dict[str, str]) -> httpx.Response: + return client.post(action, data=fields, headers=private_cookie_header(client)) + + +def private_realm(config: OidcConfig) -> str: + return config.backchannel_url or config.issuer diff --git a/pipeline/auth/tenancy.py b/pipeline/auth/tenancy.py index e507bac..106a4d1 100644 --- a/pipeline/auth/tenancy.py +++ b/pipeline/auth/tenancy.py @@ -1,8 +1,8 @@ """Instance (tenant / state) access helpers — Keycloak groups + document instances. Who-sees-what-state comes from the JWT (Keycloak groups). The database only -stores which instance a *document* belongs to (``instances`` via -``documents.catalog_id -> catalogs.instance_id``); the document dict carries its +stores which provider owns a document through +``documents.provider_id -> instances.id``; the document dict carries its tenant code as ``instance``. Conventions: @@ -19,17 +19,17 @@ from .models import AuthUser -# Bharat Vistaar portal / platform-wide documents (not a state). -PORTAL_INSTANCE = "bv" - - def default_instance() -> str: """Fallback instance code for new docs and legacy rows without a value.""" return (os.environ.get("DEFAULT_INSTANCE") or "default").strip().lower() or "default" def portal_instance() -> str: - return (os.environ.get("PORTAL_INSTANCE") or PORTAL_INSTANCE).strip().lower() or PORTAL_INSTANCE + return configured_portal_instance() or default_instance() + + +def configured_portal_instance() -> str | None: + return os.environ.get("PORTAL_INSTANCE", "").strip().lower() or None def group_for_instance(instance: str | None) -> str | None: @@ -96,7 +96,7 @@ def resolve_create_instance(user: AuthUser, requested: str | None = None) -> str if not allowed: raise HTTPException( 403, - "No state assigned. Join a Keycloak group such as /states/MH/admin or /states/MH/view.", + "No provider access is assigned. Ask an administrator to grant access in Keycloak.", ) if requested_norm: @@ -109,7 +109,7 @@ def resolve_create_instance(user: AuthUser, requested: str | None = None) -> str raise HTTPException( 400, - "instance is required when you have multiple states; " + "instance is required when you have access to multiple providers; " f"choose one of: {', '.join(sorted(allowed))}", ) diff --git a/pipeline/catalog_builder.py b/pipeline/catalog_builder.py deleted file mode 100644 index 4b74978..0000000 --- a/pipeline/catalog_builder.py +++ /dev/null @@ -1,121 +0,0 @@ -""" -Builds the Network Catalog Envelope's catalog for a document's knowledge kind. - -One catalog holding exactly one OnDemand resource per kind, identical for every -document of that kind and Group - a seeker that matches the resource comes back -to ask the actual question, so there is nothing per-document to put on the wire. - -Pure: no env, no I/O. Values live in network_constants.py, the provider in -network_provider.py and the transport in discovery_publish_service.py. See -docs/ADR/0003-static-ondemand-network-catalog-per-knowledge-kind.md and -docs/ADR/0006-publish-to-network-under-per-group-provider.md. -""" - -import copy -from typing import Optional - -from .network_constants import ( - ADVISORY_CATALOG_ID, - ADVISORY_RESOURCE_ID, - ADVISORY_RESOURCE_NAME, - ADVISORY_SUBJECT_CATEGORIES, - ADVISORY_TOPICS, - DEFAULT_DOCUMENT_KIND, - INFORMATION_MODE, - SCHEMA_CONTEXT_BASE, - SCHEME_SUBJECT_CATEGORIES, - SCHEME_TOPICS, - SCHEMES_CATALOG_ID, - SCHEMES_RESOURCE_ID, - SCHEMES_RESOURCE_NAME, - SERVED_LANGUAGES, -) -from .network_provider import Group - -_ADVISORY_RESOURCE_ATTRIBUTES = { - "@context": f"{SCHEMA_CONTEXT_BASE}/KnowledgeAdvisory/v0.1/context.jsonld", - "@type": "openagrinet:KnowledgeAdvisory", - "informationMode": INFORMATION_MODE, - "subjectCategories": ADVISORY_SUBJECT_CATEGORIES, - "topics": ADVISORY_TOPICS, - "languages": SERVED_LANGUAGES, -} - -# No KnowledgeScheme schema exists in the network specs - Scheme is only a -# subjectCategories value - so a scheme document is announced on the same -# KnowledgeAdvisory schema as advisories, distinguished only by -# subjectCategories: ["Scheme"]. -_SCHEME_RESOURCE_ATTRIBUTES = { - "@context": f"{SCHEMA_CONTEXT_BASE}/KnowledgeAdvisory/v0.1/context.jsonld", - "@type": "openagrinet:KnowledgeAdvisory", - "informationMode": INFORMATION_MODE, - "subjectCategories": SCHEME_SUBJECT_CATEGORIES, - "topics": SCHEME_TOPICS, - "languages": SERVED_LANGUAGES, -} - -_KNOWLEDGE_KIND_CATALOGS = { - "advisory": { - "catalog_id": ADVISORY_CATALOG_ID, - "resource_id": ADVISORY_RESOURCE_ID, - "resource_name": ADVISORY_RESOURCE_NAME, - "resource_attributes": _ADVISORY_RESOURCE_ATTRIBUTES, - }, - "scheme": { - "catalog_id": SCHEMES_CATALOG_ID, - "resource_id": SCHEMES_RESOURCE_ID, - "resource_name": SCHEMES_RESOURCE_NAME, - "resource_attributes": _SCHEME_RESOURCE_ATTRIBUTES, - }, -} - -PUBLISHABLE_DOCUMENT_KINDS = tuple(_KNOWLEDGE_KIND_CATALOGS) - - -def normalize_document_kind(document_kind: Optional[str]) -> str: - """Coerce a stored `documents.document_kind` to its lookup form.""" - return (document_kind or DEFAULT_DOCUMENT_KIND).strip().lower() - - -def is_publishable_kind(document_kind: Optional[str]) -> bool: - return normalize_document_kind(document_kind) in _KNOWLEDGE_KIND_CATALOGS - - -def build_catalog( - document_kind: Optional[str], - provider: dict, - catalog_name: str, - group: Optional[Group] = None, -) -> Optional[dict]: - """Return the catalog to announce for `document_kind`, or None. - - None means this kind has nothing to announce - `document`, `video` and any - operator-entered custom slug. Callers must skip the publish rather than send - an empty `catalogs` array, which the spec rejects (`minItems: 1`). - """ - spec = _KNOWLEDGE_KIND_CATALOGS.get(normalize_document_kind(document_kind)) - if spec is None: - return None - - catalog_id = spec["catalog_id"] - resource_id = spec["resource_id"] - provider = copy.deepcopy(provider) - if group is not None: - prefix = f"{group.code}-" - catalog_id = prefix + catalog_id - resource_id = prefix + resource_id - - catalog = { - "id": catalog_id, - "descriptor": {"name": catalog_name}, - "isActive": True, - "provider": provider, - "resources": [ - { - "id": resource_id, - "descriptor": {"name": spec["resource_name"]}, - "resourceAttributes": copy.deepcopy(spec["resource_attributes"]), - } - ], - } - return catalog diff --git a/pipeline/db.py b/pipeline/db.py deleted file mode 100644 index b8a70e8..0000000 --- a/pipeline/db.py +++ /dev/null @@ -1,3430 +0,0 @@ -""" -Postgres persistence for documents, workflows, pages, chunks, audit and settings. - -Schema: ``migrations/`` (see docs/POSTGRES_SCHEMA.md). One function per query. - -A document is content and metadata; a workflow is one processing run of a -document. Callers still address a document by ``workflow_id`` - the Temporal -workflow id of its main run (``doc-``) - and every function here also -accepts the document UUID, the fingerprint or the filename (see -``find_document_by_doc_identifier``). Documents, jobs and artifacts come back -as flat dicts so call sites need not know how the tables are split; ``id`` -carries the UUID. -""" - -import hashlib -import json -import logging -import os -import re -import threading -import uuid -from contextlib import contextmanager -from datetime import date, datetime, timezone -from typing import Any, Optional - -import psycopg -from psycopg.rows import dict_row -from psycopg.types.json import Jsonb -from psycopg_pool import ConnectionPool - -from . import migrate -from .models import ( - CandidateSettings, - ChunkingRun, - DocumentMetadata, - DocumentSource, - IndexState, - PageMetadata, - PageTranslation, - QuerySettings, - ResultSettings, - RetrievalSettings, -) -from .network_constants import KIND_TO_CATALOG_TYPE, catalog_type_for_kind, kind_for_catalog_type - -logger = logging.getLogger(__name__) - -DEFAULT_DATABASE_URL = "postgresql://postgres:postgres@localhost:5432/knowledge_provider" - -_pool: Optional[ConnectionPool] = None -_pool_url: Optional[str] = None -_pool_lock = threading.Lock() - - -def database_url() -> str: - return os.environ.get("DATABASE_URL") or DEFAULT_DATABASE_URL - - -def get_pool() -> ConnectionPool: - global _pool, _pool_url - url = database_url() - if _pool is not None and _pool_url == url: - return _pool - with _pool_lock: - if _pool is not None and _pool_url != url: - _pool.close() - _pool = None - if _pool is None: - pool = ConnectionPool( - conninfo=url, - min_size=int(os.environ.get("DB_POOL_MIN_SIZE", "1")), - max_size=int(os.environ.get("DB_POOL_MAX_SIZE", "10")), - timeout=float(os.environ.get("DB_POOL_TIMEOUT_SECONDS", "30")), - kwargs={"row_factory": dict_row, "options": "-c timezone=UTC"}, - open=False, - ) - pool.open(wait=True) - _pool = pool - _pool_url = url - logger.info("db pool opened: %s", pool.get_stats()) - return _pool - - -def close_pool() -> None: - global _pool, _pool_url - with _pool_lock: - if _pool is not None: - _pool.close() - _pool = None - _pool_url = None - - -@contextmanager -def get_connection(): - """A pooled connection; the transaction commits on exit and rolls back on error.""" - with get_pool().connection() as conn: - yield conn - - -def init_db() -> None: - """Apply pending migrations and open the connection pool.""" - applied = migrate.apply_migrations(database_url()) - current = migrate.current_version(database_url()) - logger.info("db migrations current=%s applied_now=%s", current, applied) - get_pool() - created = sync_catalogs_for_all_instances() - if created: - logger.info("catalog sync created %d missing catalogs", created) - - -def check_health() -> dict: - """Reachability and migration state, for the /health endpoint.""" - try: - with get_connection() as conn: - conn.execute("SELECT 1") - return {"reachable": True, "migrations_current": migrate.is_current(database_url())} - except Exception: - logger.exception("database health check failed") - return {"reachable": False, "migrations_current": False} - - -# ============================================================================= -# Helpers -# ============================================================================= - -_UUID_RE = re.compile( - r"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$" -) -_FINGERPRINT_RE = re.compile(r"^[0-9a-f]{32}$") - -_REVIEW_STAGES = { - "ocr_review", - "translation_review", - "chunk_review", - "ready_for_ingestion", - "approval_for_prod", -} - -PROD_INDEX_KEY = "prod" -DEV_INDEX_KEY = "dev" - - -def _default_instance() -> str: - return (os.environ.get("DEFAULT_INSTANCE") or "default").strip().lower() or "default" - - -def _normalize_instances(instances: list[str]) -> list[str]: - default = _default_instance() - return sorted({(i or "").strip().lower() or default for i in instances}) - - -def _iso_now() -> str: - return datetime.now(timezone.utc).replace(tzinfo=None).isoformat() - - -def _is_uuid(value: Any) -> bool: - return isinstance(value, str) and bool(_UUID_RE.match(value)) - - -def _clean(value: Any) -> Any: - if isinstance(value, uuid.UUID): - return str(value) - if isinstance(value, datetime): - if value.tzinfo is not None: - value = value.astimezone(timezone.utc).replace(tzinfo=None) - return value.isoformat() - if isinstance(value, date): - return value.isoformat() - return value - - -def _clean_row(row: Optional[dict]) -> Optional[dict]: - if row is None: - return None - return {k: _clean(v) for k, v in row.items()} - - -def _csv_roles(roles: Any) -> list[str]: - if roles is None: - return [] - if isinstance(roles, (list, tuple, set)): - return [str(r).strip() for r in roles if str(r).strip()] - return [part.strip() for part in str(roles).split(",") if part.strip()] - - -def _fingerprint_for( - source_file_fingerprint: Optional[str], - document_id: Optional[str], - workflow_id: Optional[str], -) -> str: - for candidate in (source_file_fingerprint, document_id): - if candidate and _FINGERPRINT_RE.match(candidate): - return candidate - return hashlib.md5((document_id or workflow_id or "").encode()).hexdigest() - - -def _prod_collection_names() -> set[str]: - names = { - os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME"), - os.environ.get("PROD_SCHEME_VECTOR_DB_COLLECTION_NAME"), - } - return {n.strip() for n in names if n and n.strip()} - - -def _index_key_for(index_name: str) -> str: - return PROD_INDEX_KEY if index_name in _prod_collection_names() else DEV_INDEX_KEY - - -# ============================================================================= -# Identifier resolution -# ============================================================================= - -def _resolve_document_uuid(conn, identifier: Optional[str]) -> Optional[str]: - """Document UUID for a UUID, Temporal workflow id, fingerprint or filename.""" - if not identifier: - return None - identifier = str(identifier) - - if _is_uuid(identifier): - row = conn.execute( - "SELECT id FROM documents WHERE id = %s::uuid", (identifier,) - ).fetchone() - if row: - return str(row["id"]) - - row = conn.execute( - """ - SELECT document_id FROM workflows - WHERE temporal_workflow_id = %s - ORDER BY started_at DESC LIMIT 1 - """, - (identifier,), - ).fetchone() - if row: - return str(row["document_id"]) - - row = conn.execute( - """ - SELECT id FROM documents - WHERE source->>'fingerprint' = %s OR source->>'original_filename' = %s - ORDER BY created_at DESC LIMIT 1 - """, - (identifier, identifier), - ).fetchone() - if row: - return str(row["id"]) - - row = conn.execute( - """ - SELECT id FROM documents d - WHERE EXISTS ( - SELECT 1 FROM jsonb_each(COALESCE(d.metadata->'index_status', '{}'::jsonb)) e - WHERE e.value->>'doc_id' = %s - ) OR md5(d.source->>'fingerprint') = %s - ORDER BY created_at DESC LIMIT 1 - """, - (identifier, identifier), - ).fetchone() - return str(row["id"]) if row else None - - -def _require_document_uuid(conn, identifier: str) -> str: - doc_uuid = _resolve_document_uuid(conn, identifier) - if doc_uuid is None: - raise LookupError(f"Document not found: {identifier}") - return doc_uuid - - -# ============================================================================= -# Document row <-> flat dict -# ============================================================================= - -_DOC_SELECT = """ - SELECT - d.id, d.catalog_id, d.code, d.name, d.aliases, d.source, d.is_disabled, d.metadata, - d.created_at, d.updated_at, - c.type AS catalog_type, g.name AS instance_name, - (SELECT count(*) FROM pages p WHERE p.document_id = d.id) AS page_total, - (SELECT count(*) FROM chunks ch WHERE ch.document_id = d.id) AS chunk_total, - root.temporal_workflow_id AS root_workflow_id, - root.stop_after_ocr AS root_stop_after_ocr, - cw.id AS current_workflow_pk, - cw.current_stage AS current_stage, - cw.error_message AS current_error_message, - u.keycloak_user_id AS up_user_id, u.username AS up_username, - u.email AS up_email, u.roles AS up_roles, - ts.ocr_completed_at, ts.translation_completed_at, ts.chunks_completed_at, - ts.ingested_at, - req.started_at AS prod_ready_requested_at, - ru.keycloak_user_id AS prod_ready_requested_by_user_id, - ru.username AS prod_ready_requested_by_username, - oa.id AS original_artifact_id, - na.id AS normalized_artifact_id - FROM documents d - JOIN catalogs c ON c.id = d.catalog_id - JOIN instances g ON g.id = c.instance_id - LEFT JOIN document_current_workflow cw ON cw.document_id = d.id - LEFT JOIN LATERAL ( - SELECT w.temporal_workflow_id, w.stop_after_ocr FROM workflows w - WHERE w.document_id = d.id - ORDER BY (w.workflow_type IN ('pipeline', 'ocr_only')) DESC, w.started_at ASC, w.id ASC - LIMIT 1 - ) root ON TRUE - LEFT JOIN users u ON u.id = NULLIF(d.metadata->>'uploaded_by_user_id', '')::uuid - LEFT JOIN LATERAL ( - SELECT min(w.stage_timestamps->>'ocr') AS ocr_completed_at, - min(w.stage_timestamps->>'translation') AS translation_completed_at, - min(w.stage_timestamps->>'chunks') AS chunks_completed_at, - max(COALESCE(w.completed_at, w.updated_at)) - FILTER (WHERE w.current_stage = 'completed') AS ingested_at - FROM workflows w WHERE w.document_id = d.id - ) ts ON TRUE - LEFT JOIN LATERAL ( - SELECT w.started_at, w.started_by_user_id FROM workflows w - WHERE w.document_id = d.id AND w.workflow_type = 'promote_prod' - AND w.status = 'waiting_review' - ORDER BY w.started_at DESC LIMIT 1 - ) req ON TRUE - LEFT JOIN users ru ON ru.id = req.started_by_user_id - LEFT JOIN LATERAL ( - SELECT a.id FROM workflow_artifacts a JOIN workflows w ON w.id = a.workflow_id - WHERE w.document_id = d.id AND a.artifact_type = 'original_upload' - ORDER BY a.created_at DESC, a.id DESC LIMIT 1 - ) oa ON TRUE - LEFT JOIN LATERAL ( - SELECT a.id FROM workflow_artifacts a JOIN workflows w ON w.id = a.workflow_id - WHERE w.document_id = d.id - AND a.artifact_type IN ('normalized_pdf', 'normalized_spreadsheet') - ORDER BY a.created_at DESC, a.id DESC LIMIT 1 - ) na ON TRUE -""" - - -def _kind_of(catalog_type: Optional[str]) -> Optional[str]: - """Knowledge kind a catalog type stands for; an unmapped type is returned as is.""" - if not catalog_type: - return None - try: - return kind_for_catalog_type(catalog_type) - except ValueError: - return catalog_type - - -def _document_from_row(row: Optional[dict]) -> Optional[dict]: - if row is None: - return None - source = row["source"] or {} - meta = row["metadata"] or {} - fingerprint = source.get("fingerprint") - roles = row["up_roles"] or [] - entry_aliases = row["aliases"] - doc = { - "id": row["id"], - "workflow_id": row["root_workflow_id"], - "document_id": fingerprint, - "canonical_document_id": fingerprint, - "filename": source.get("original_filename") or "", - "name": row["name"], - "source_filename": source.get("original_filename"), - "source_manifest_name": None, - "source_file_fingerprint": fingerprint, - "filepath": source.get("storage_uri") or "", - "instance": (row["instance_name"] or "").lower(), - "stage": row["current_stage"] or "registered", - "page_count": int(row["page_total"] or 0), - "chunk_count": int(row["chunk_total"] or 0), - "error_message": row["current_error_message"], - "is_disabled": bool(row["is_disabled"]), - "source_type": source.get("source_type"), - "canonical_input_type": source.get("canonical_input_type"), - "stop_after_ocr": bool(row["root_stop_after_ocr"]), - "reindex_required": bool(meta.get("reindex_required")), - "reindex_reason": meta.get("reindex_reason"), - "kind": _kind_of(row["catalog_type"]), - "code": row["code"], - "aliases": list(entry_aliases or []), - "tool_routing": "qdrant", - "catalog_visible": bool(meta.get("catalog_visible", True)), - "network_visible": bool(meta.get("network_visible", True)), - "valid_from": meta.get("valid_from"), - "valid_to": meta.get("valid_to"), - "uploaded_by_user_id": row["up_user_id"], - "uploaded_by_username": row["up_username"], - "uploaded_by_email": row["up_email"], - "uploaded_by_roles": ",".join(roles) if roles else None, - "prod_ready_requested_at": row["prod_ready_requested_at"], - "prod_ready_requested_by_user_id": row["prod_ready_requested_by_user_id"], - "prod_ready_requested_by_username": row["prod_ready_requested_by_username"], - "ocr_completed_at": row["ocr_completed_at"], - "translation_completed_at": row["translation_completed_at"], - "chunks_completed_at": row["chunks_completed_at"], - "ingested_at": row["ingested_at"], - "original_artifact_id": row["original_artifact_id"], - "normalized_artifact_id": row["normalized_artifact_id"], - "latest_job_id": row["current_workflow_pk"], - "catalog_id": row["catalog_id"], - "index_status": meta.get("index_status") or {}, - "superseded_doc_ids": meta.get("superseded_doc_ids") or {}, - "created_at": row["created_at"], - "updated_at": row["updated_at"], - } - return {k: _clean(v) for k, v in doc.items()} - - -def _fetch_document(conn, doc_uuid: Optional[str]) -> Optional[dict]: - if doc_uuid is None: - return None - row = conn.execute(_DOC_SELECT + " WHERE d.id = %s::uuid", (doc_uuid,)).fetchone() - return _document_from_row(row) - - -def _document_filters( - include_disabled: bool, - instances: Optional[list[str]], - alias: str = "d", - instance_alias: str = "g", -) -> tuple[list[str], list]: - """WHERE clauses for document lists. The query must join ``instances`` as ``instance_alias``.""" - clauses: list[str] = [] - params: list = [] - if not include_disabled: - clauses.append(f"NOT {alias}.is_disabled") - if instances is not None: - clauses.append(f"lower({instance_alias}.name) = ANY(%s)") - params.append(_normalize_instances(instances)) - return clauses, params - - -# ============================================================================= -# Instances -# ============================================================================= - -def _tenant_path(instance: str) -> str: - """Keycloak group path of a tenant code: ``bv`` is ``/global``, others ``/states/``.""" - return "/global" if instance == "bv" else f"/states/{instance.upper()}" - - -def _tenant_code_from_path(path: str) -> str: - return "bv" if path == "/global" else path.rstrip("/").rsplit("/", 1)[-1].lower() - - -def upsert_instance(path: str, keycloak_group_id: Optional[str] = None, conn=None) -> str: - """Create or refresh the ``instances`` row for a Keycloak group path; returns its UUID. - - Rows made by the SQLite move have no Keycloak id yet; the first sync that - sees the path fills it in. The instance's catalogs (one per supported type) are - ensured in the same transaction, so an instance never exists without them. - """ - if conn is None: - with get_connection() as own: - return upsert_instance(path, keycloak_group_id, conn=own) - row = conn.execute( - """ - INSERT INTO instances (keycloak_group_id, name, path) VALUES (%s, %s, %s) - ON CONFLICT (path) DO UPDATE SET - keycloak_group_id = COALESCE(EXCLUDED.keycloak_group_id, instances.keycloak_group_id) - RETURNING id - """, - (keycloak_group_id, _tenant_code_from_path(path), path), - ).fetchone() - instance_id = str(row["id"]) - ensure_instance_catalogs(instance_id, conn=conn) - return instance_id - - -def get_or_create_instance(instance: Optional[str], conn=None) -> str: - """UUID of the instance for a tenant code (``bv``, ``mh``, ...), created when missing.""" - code = (instance or _default_instance()).strip().lower() or _default_instance() - return upsert_instance(_tenant_path(code), conn=conn) - - -def _catalog_seed_rows() -> tuple[list[str], list[str]]: - """Parallel ``(types, kinds)`` lists for every supported catalog type.""" - kinds = list(KIND_TO_CATALOG_TYPE) - return [KIND_TO_CATALOG_TYPE[k] for k in kinds], kinds - - -def ensure_instance_catalogs(instance_id: str, conn=None) -> None: - """Create the instance's missing catalogs, one per supported type; existing rows are untouched.""" - if conn is None: - with get_connection() as own: - return ensure_instance_catalogs(instance_id, conn=own) - types, kinds = _catalog_seed_rows() - conn.execute( - """ - INSERT INTO catalogs (instance_id, type, name) - SELECT g.id, t.type, upper(g.name) || ' ' || t.kind || ' catalog' - FROM instances g, unnest(%s::text[], %s::text[]) AS t(type, kind) - WHERE g.id = %s::uuid - ON CONFLICT (instance_id, type) DO NOTHING - """, - (types, kinds, instance_id), - ) - - -def sync_catalogs_for_all_instances(conn=None) -> int: - """Add every missing ``(instance, type)`` catalog; returns how many rows were created. - - Idempotent. Run at startup so a type added to the kind map reaches instances - that already exist. - """ - if conn is None: - with get_connection() as own: - return sync_catalogs_for_all_instances(conn=own) - types, kinds = _catalog_seed_rows() - cur = conn.execute( - """ - INSERT INTO catalogs (instance_id, type, name) - SELECT g.id, t.type, upper(g.name) || ' ' || t.kind || ' catalog' - FROM instances g, unnest(%s::text[], %s::text[]) AS t(type, kind) - ON CONFLICT (instance_id, type) DO NOTHING - """, - (types, kinds), - ) - return cur.rowcount - - -def get_or_create_catalog(instance_id: str, kind: str, conn=None) -> dict: - """The ``(instance, kind)`` catalog row; created when missing (normally seeded with the instance).""" - if conn is None: - with get_connection() as own: - return get_or_create_catalog(instance_id, kind, conn=own) - catalog_type = catalog_type_for_kind(kind) - row = conn.execute( - """ - INSERT INTO catalogs (instance_id, type, name) - SELECT g.id, %s, upper(g.name) || ' ' || %s || ' catalog' FROM instances g WHERE g.id = %s::uuid - ON CONFLICT (instance_id, type) DO UPDATE SET type = EXCLUDED.type - RETURNING id, instance_id, type, name, metadata - """, - (catalog_type, kind, instance_id), - ).fetchone() - return _clean_row(row) - - -# ============================================================================= -# Users -# ============================================================================= - -def upsert_user( - keycloak_user_id: str, - username: Optional[str] = None, - email: Optional[str] = None, - roles: Any = None, - conn=None, - tenants: Optional[list[str]] = None, -) -> Optional[str]: - """Create or refresh the ``users`` row for a Keycloak subject; returns its UUID. - - ``tenants`` are the tenant codes in the user's token. A user has one - ``instance_id``, so the first code (sorted) wins and a warning is logged when - there are several. Authorization still reads the token, not this column. - """ - keycloak_user_id = (keycloak_user_id or "").strip() - if not keycloak_user_id: - return None - if conn is None: - with get_connection() as own: - return upsert_user(keycloak_user_id, username, email, roles, conn=own, tenants=tenants) - - role_list = _csv_roles(roles) - tenant_codes = sorted({(t or "").strip().lower() for t in (tenants or []) if (t or "").strip()}) - if len(tenant_codes) > 1: - logger.warning( - "user %s has several tenants %s; users.instance_id uses %s", - keycloak_user_id, tenant_codes, tenant_codes[0], - ) - instance_uuid = get_or_create_instance(tenant_codes[0], conn=conn) if tenant_codes else None - email = (email or "").strip() or None - username = (username or "").strip() or None - for candidate_email in ([email, None] if email else [None]): - try: - with conn.transaction(): - row = conn.execute( - """ - INSERT INTO users (keycloak_user_id, username, email, roles, instance_id) - VALUES (%s, %s, %s, %s, %s::uuid) - ON CONFLICT (keycloak_user_id) DO UPDATE SET - username = COALESCE(EXCLUDED.username, users.username), - email = COALESCE(EXCLUDED.email, users.email), - roles = CASE WHEN cardinality(EXCLUDED.roles) > 0 - THEN EXCLUDED.roles ELSE users.roles END, - instance_id = COALESCE(EXCLUDED.instance_id, users.instance_id), - last_seen_at = now() - RETURNING id - """, - (keycloak_user_id, username, candidate_email, role_list, instance_uuid), - ).fetchone() - return str(row["id"]) - except psycopg.errors.UniqueViolation: - continue - return None - - -def get_user_by_keycloak_id(keycloak_user_id: str) -> Optional[dict]: - with get_connection() as conn: - return _clean_row( - conn.execute( - "SELECT * FROM users WHERE keycloak_user_id = %s", (keycloak_user_id,) - ).fetchone() - ) - - -# ============================================================================= -# Webhook records -# ============================================================================= - -def _parse_timestamp(value: Any) -> Optional[datetime]: - if not value: - return None - try: - parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00")) - except ValueError: - return None - if parsed.tzinfo is None: - parsed = parsed.replace(tzinfo=timezone.utc) - return parsed - - -def _webhook_payload_and_hash(rec: dict) -> tuple[str, str]: - payload = json.dumps(rec, sort_keys=True, ensure_ascii=False, default=str) - return payload, hashlib.sha256(payload.encode("utf-8")).hexdigest() - - -def webhook_external_id(rec: dict) -> str: - """The (source, external_id) key of a webhook record: contentID, else id, else slug, else payload hash.""" - _, payload_hash = _webhook_payload_and_hash(rec) - for key in ("contentID", "id", "slug"): - value = rec.get(key) - if value not in (None, ""): - return str(value) - return payload_hash - - -def upsert_webhook_records(source: str, records: list) -> int: - """Upsert records by (source, external_id) in one transaction; returns len(records). - - external_id comes from contentID, else id, else slug, else a hash of the payload. - Unchanged payloads (same payload_hash) are not rewritten but still counted. - """ - now = datetime.now(timezone.utc) - rows = [] - for rec in records: - if not isinstance(rec, dict): - raise ValueError("Each record must be a JSON object") - payload_text, payload_hash = _webhook_payload_and_hash(rec) - ext = webhook_external_id(rec) - pub = rec.get("isPublished") - rows.append(( - source, str(ext), rec.get("title"), rec.get("slug"), rec.get("status"), - bool(pub) if isinstance(pub, (bool, int)) else None, - rec.get("domainName"), Jsonb(json.loads(payload_text)), payload_hash, - _parse_timestamp(rec.get("createdAt")) or now, - _parse_timestamp(rec.get("updatedAt")) or now, - )) - with get_connection() as conn: - for r in rows: - conn.execute( - """ - INSERT INTO webhook_records - (source, external_id, title, slug, status, is_published, category, - payload, payload_hash, created_at, updated_at) - VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) - ON CONFLICT (source, external_id) DO UPDATE SET - title = EXCLUDED.title, - slug = EXCLUDED.slug, - status = EXCLUDED.status, - is_published = EXCLUDED.is_published, - category = EXCLUDED.category, - payload = EXCLUDED.payload, - payload_hash = EXCLUDED.payload_hash, - created_at = EXCLUDED.created_at, - updated_at = EXCLUDED.updated_at - WHERE webhook_records.payload_hash IS DISTINCT FROM EXCLUDED.payload_hash - """, - r, - ) - return len(rows) - - -_WEBHOOK_RECORD_COLUMNS = ( - "id, source, external_id, title, slug, status, is_published, category, payload, " - "payload_hash, created_at, updated_at, indexed_hash, promoted_hash" -) - - -def get_webhook_record(record_id: str) -> Optional[dict]: - """One webhook_records row as a dict (``payload`` already parsed from JSONB); None if missing.""" - if not _is_uuid(record_id): - return None - with get_connection() as conn: - row = conn.execute( - f"SELECT {_WEBHOOK_RECORD_COLUMNS} FROM webhook_records WHERE id = %s", (record_id,) - ).fetchone() - if row is None: - return None - record = dict(row) - record["id"] = str(record["id"]) - if not isinstance(record["payload"], dict): - record["payload"] = {} - return record - - -def get_unindexed_webhook_record_ids( - source: str = "webhook", external_ids: Optional[list] = None -) -> list[str]: - """Ids of rows whose payload changed since they were last indexed (or never indexed). - - ``external_ids`` limits the check to the records of one webhook call. - """ - sql = ( - "SELECT id FROM webhook_records WHERE source = %s " - "AND indexed_hash IS DISTINCT FROM payload_hash" - ) - params: list = [source] - if external_ids is not None: - if not external_ids: - return [] - sql += " AND external_id = ANY(%s)" - params.append([str(e) for e in external_ids]) - with get_connection() as conn: - return [str(r["id"]) for r in conn.execute(sql + " ORDER BY created_at, id", params)] - - -def get_webhook_record_ids( - source: str = "webhook", published_only: bool = False, indexed_only: bool = False -) -> list[str]: - """All record ids for a source, optionally only published and/or dev-indexed ones. - - ``indexed_only`` keeps rows whose current payload is what was last written to the dev - collection (``indexed_hash = payload_hash``), so a bulk promote never pushes a payload - that has not been indexed in dev. - """ - sql = "SELECT id FROM webhook_records WHERE source = %s" - if published_only: - sql += " AND is_published AND status = 'Published'" - if indexed_only: - sql += " AND indexed_hash = payload_hash" - with get_connection() as conn: - return [str(r["id"]) for r in conn.execute(sql + " ORDER BY created_at, id", (source,))] - - -def _mark_webhook_record(column: str, record_id: str, payload_hash: str) -> None: - with get_connection() as conn: - conn.execute(f"UPDATE webhook_records SET {column} = %s WHERE id = %s", (payload_hash, record_id)) - - -def mark_webhook_record_indexed(record_id: str, payload_hash: str) -> None: - """Record the payload_hash that was written to the dev collection.""" - _mark_webhook_record("indexed_hash", record_id, payload_hash) - - -def mark_webhook_record_promoted(record_id: str, payload_hash: str) -> None: - """Record the payload_hash that was written to the PROD collection.""" - _mark_webhook_record("promoted_hash", record_id, payload_hash) - - -def advisory_status_counts(source: str = "webhook") -> dict: - """Counts for the advisory status endpoint.""" - with get_connection() as conn: - row = conn.execute( - """ - SELECT - COUNT(*) AS total, - COUNT(*) FILTER (WHERE is_published AND status = 'Published') AS published, - COUNT(*) FILTER (WHERE indexed_hash = payload_hash) AS indexed, - COUNT(*) FILTER (WHERE promoted_hash = payload_hash) AS promoted - FROM webhook_records WHERE source = %s - """, - (source,), - ).fetchone() - total = int(row["total"]) - indexed = int(row["indexed"]) - return { - "total": total, - "published": int(row["published"]), - "indexed": indexed, - "unindexed": total - indexed, - "promoted": int(row["promoted"]), - } - - -# ============================================================================= -# Email OTPs -# ============================================================================= - -def store_email_otp( - *, email: str, code_hash: str, expires_at: str, created_at: str, last_sent_at: str -) -> None: - """Upsert the one live OTP for an email, resetting attempts on resend.""" - with get_connection() as conn: - conn.execute( - """ - INSERT INTO email_otps (email, code_hash, expires_at, attempts, last_sent_at) - VALUES (%s, %s, %s, 0, %s) - ON CONFLICT (email) DO UPDATE SET - code_hash = EXCLUDED.code_hash, - expires_at = EXCLUDED.expires_at, - attempts = 0, - last_sent_at = EXCLUDED.last_sent_at - """, - (email.strip().lower(), code_hash, expires_at, last_sent_at), - ) - - -def get_email_otp(email: str) -> Optional[dict]: - """The live OTP row; timestamps are timezone-aware ISO strings (UTC).""" - with get_connection() as conn: - row = conn.execute( - "SELECT * FROM email_otps WHERE email = %s", (email.strip().lower(),) - ).fetchone() - if row is None: - return None - row = _clean_row(row) | { - "expires_at": row["expires_at"].astimezone(timezone.utc).isoformat(), - "last_sent_at": row["last_sent_at"].astimezone(timezone.utc).isoformat(), - } - return row - - -def increment_email_otp_attempts(email: str) -> Optional[int]: - """Atomically count one failed attempt; returns the new total.""" - with get_connection() as conn: - row = conn.execute( - "UPDATE email_otps SET attempts = attempts + 1 WHERE email = %s RETURNING attempts", - (email.strip().lower(),), - ).fetchone() - return int(row["attempts"]) if row else None - - -def delete_email_otp(email: str) -> None: - with get_connection() as conn: - conn.execute("DELETE FROM email_otps WHERE email = %s", (email.strip().lower(),)) - - -# ============================================================================= -# Catalog -# ============================================================================= - -CATALOG_VERSION_ACTION = "catalog_version_bump" - - -def get_catalog_meta() -> dict: - """Catalog version: the number of version bumps recorded in ``audit_logs``.""" - with get_connection() as conn: - row = conn.execute( - """ - SELECT count(*) AS version, max(created_at) AS updated_at, - (array_agg(metadata->>'reason' ORDER BY created_at DESC))[1] AS notes - FROM audit_logs - WHERE entity_type = 'catalog' AND action_type = %s - """, - (CATALOG_VERSION_ACTION,), - ).fetchone() - return { - "id": 1, - "version": int(row["version"] or 0), - "updated_at": _clean(row["updated_at"]), - "notes": row["notes"], - } - - -def bump_catalog_version(reason: str = "") -> int: - """Monotonic catalog version bump. Returns new version.""" - with get_connection() as conn: - conn.execute( - """ - INSERT INTO audit_logs (action_type, entity_type, metadata) - VALUES (%s, 'catalog', %s) - """, - (CATALOG_VERSION_ACTION, Jsonb({"reason": (reason or "")[:500]})), - ) - row = conn.execute( - "SELECT count(*) AS version FROM audit_logs " - "WHERE entity_type = 'catalog' AND action_type = %s", - (CATALOG_VERSION_ACTION,), - ).fetchone() - return int(row["version"]) - - -def list_document_codes(kind: str) -> set[str]: - """Every document code in catalogs of one knowledge kind.""" - with get_connection() as conn: - rows = conn.execute( - "SELECT d.code FROM documents d JOIN catalogs c ON c.id = d.catalog_id WHERE c.type = %s", - (catalog_type_for_kind(kind),), - ).fetchall() - return {r["code"] for r in rows} - - -def _catalog_row(row: dict) -> dict: - cleaned = _clean_row(row) - cleaned.pop("instance_name", None) - cleaned["aliases"] = list(row["aliases"] or []) - cleaned["instances"] = [(row["instance_name"] or "").lower()] - cleaned["workflow_ids"] = [row["workflow_id"]] if row["workflow_id"] else [] - cleaned["source"] = "pipeline" - return cleaned - - -def list_catalog_entries(status: Optional[str] = None, kind: str = "scheme") -> list[dict]: - """Rows of the ``catalog_status`` view (one per document) for one knowledge kind.""" - catalog_type = catalog_type_for_kind(kind) - with get_connection() as conn: - if status: - rows = conn.execute( - "SELECT * FROM catalog_status WHERE type = %s AND status = %s ORDER BY code", - (catalog_type, status), - ).fetchall() - else: - rows = conn.execute( - "SELECT * FROM catalog_status WHERE type = %s ORDER BY code", (catalog_type,) - ).fetchall() - return [_catalog_row(r) for r in rows] - - -def get_catalog_entry(code: str, kind: str = "scheme") -> Optional[dict]: - with get_connection() as conn: - row = conn.execute( - "SELECT * FROM catalog_status WHERE type = %s AND code = %s LIMIT 1", - (catalog_type_for_kind(kind), code), - ).fetchone() - return _catalog_row(row) if row else None - - -def list_used_scheme_codes(exclude_workflow_id: Optional[str] = None, kind: str = "scheme") -> set[str]: - """Every document code already taken in a kind - the collision domain for auto-generated codes. - - The excluded document's own code is not counted, so re-deriving a - document's code does not collide with itself. - """ - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, exclude_workflow_id) if exclude_workflow_id else None - rows = conn.execute( - """ - SELECT d.code FROM documents d JOIN catalogs c ON c.id = d.catalog_id - WHERE c.type = %s AND (%s::uuid IS NULL OR d.id <> %s::uuid) - """, - (catalog_type_for_kind(kind), doc_uuid, doc_uuid), - ).fetchall() - return {r["code"] for r in rows} - - -def list_scheme_catalog_entries_for_workflow(workflow_id: str) -> list[dict]: - """Catalog rows whose documents still include this workflow.""" - with get_connection() as conn: - rows = conn.execute( - "SELECT code, workflow_id FROM catalog_status WHERE type = %s AND workflow_id = %s", - (catalog_type_for_kind("scheme"), workflow_id), - ).fetchall() - return [{"code": r["code"], "workflow_ids": [r["workflow_id"]]} for r in rows] - - -def find_document_by_code(instance: str, kind: str, code: str) -> Optional[dict]: - """The document holding ``code`` in the instance's ``kind`` catalog, if any.""" - tenant = (instance or _default_instance()).strip().lower() or _default_instance() - with get_connection() as conn: - row = conn.execute( - """ - SELECT d.id FROM documents d - JOIN catalogs c ON c.id = d.catalog_id - JOIN instances g ON g.id = c.instance_id - WHERE lower(g.name) = %s AND c.type = %s AND d.code = %s - """, - (tenant, catalog_type_for_kind(kind), code), - ).fetchone() - return _fetch_document(conn, str(row["id"])) if row else None - - -class DuplicateDocumentCode(ValueError): - """The ``(catalog, code)`` pair is already held by another document.""" - - -# ============================================================================= -# Documents -# ============================================================================= - -def upsert_document( - workflow_id: str, - document_id: str, - filename: str, - filepath: str, - canonical_document_id: Optional[str] = None, - name: Optional[str] = None, - source_filename: Optional[str] = None, - source_manifest_name: Optional[str] = None, - source_file_fingerprint: Optional[str] = None, - stage: str = "registered", - page_count: int = 0, - chunk_count: int = 0, - error_message: Optional[str] = None, - ocr_completed_at: Optional[str] = None, - translation_completed_at: Optional[str] = None, - chunks_completed_at: Optional[str] = None, - ingested_at: Optional[str] = None, - source_type: Optional[str] = None, - canonical_input_type: Optional[str] = None, - stop_after_ocr: bool = False, - original_artifact_id: Optional[str] = None, - normalized_artifact_id: Optional[str] = None, - latest_job_id: Optional[str] = None, - instance: Optional[str] = None, - uploaded_by_user_id: Optional[str] = None, - uploaded_by_username: Optional[str] = None, - uploaded_by_email: Optional[str] = None, - uploaded_by_roles: Optional[str] = None, - valid_from: Optional[str] = None, - valid_to: Optional[str] = None, - with_workflow: bool = True, - kind: Optional[str] = None, - code: Optional[str] = None, - aliases: Optional[list[str]] = None, -) -> str: - """Insert or update a document record; returns the document UUID. - - A new document needs ``kind`` (``scheme`` / ``advisory``) and ``code``: - ``instance`` (a tenant code) and the kind pick its ``(instance, kind)`` catalog, - which is created when missing. ``name`` becomes ``documents.name`` (the - code when absent). ``DuplicateDocumentCode`` is raised when another - document already holds ``code`` in that catalog. - - ``instance`` and the catalog are applied on INSERT only. Updates leave them - alone so a re-upload / restart cannot silently reassign tenants. - Uploader identity is set on INSERT, and filled on UPDATE only when currently null. - - ``valid_from``/``valid_to`` are the document's validity period, stored as - given and likewise INSERT-only, so an approver's later edit survives a - restart that re-registers the same workflow. What the period should default - to is the caller's call (see ``pipeline/document_validity.py``); omitting - both stores NULL, which search reads as "no stated period". - - ``page_count`` and ``chunk_count`` are accepted for call-site compatibility - and ignored: both are counted from ``pages`` and ``chunks``. - - A new document also gets its first workflow row (``temporal_workflow_id`` = - ``workflow_id``) unless ``with_workflow`` is False, for callers that create - it themselves with ``create_document_job`` (addressing the document by the - returned UUID, since no workflow id resolves to it yet). - """ - instance_value = (instance or _default_instance()).strip().lower() or _default_instance() - fingerprint = _fingerprint_for( - source_file_fingerprint or canonical_document_id, document_id, workflow_id - ) - source_patch = { - "fingerprint": fingerprint, - "original_filename": filename, - "storage_uri": filepath, - "source_type": source_type, - "canonical_input_type": canonical_input_type, - } - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) if workflow_id else None - if doc_uuid: - patch = {k: v for k, v in source_patch.items() if v is not None} - DocumentSource.model_validate({**_load(conn, doc_uuid, "source"), **patch}) - uploader_uuid = ( - upsert_user(uploaded_by_user_id, uploaded_by_username, uploaded_by_email, - uploaded_by_roles, conn=conn, tenants=[instance_value]) - if uploaded_by_user_id else None - ) - conn.execute( - """ - UPDATE documents SET - source = source || %s::jsonb, - name = COALESCE(%s, name), - metadata = CASE - WHEN %s::text IS NULL OR metadata ? 'uploaded_by_user_id' THEN metadata - ELSE metadata || jsonb_build_object('uploaded_by_user_id', %s::text) - END - WHERE id = %s::uuid - """, - (Jsonb(patch), name or None, uploader_uuid, uploader_uuid, doc_uuid), - ) - cw = conn.execute( - "SELECT id FROM document_current_workflow WHERE document_id = %s::uuid", (doc_uuid,) - ).fetchone() - if cw: - conn.execute( - """ - UPDATE workflows SET current_stage = %s, error_message = %s, - stop_after_ocr = %s - WHERE id = %s::uuid - """, - (stage, error_message, bool(stop_after_ocr), str(cw["id"])), - ) - return doc_uuid - - if not kind or not code: - raise ValueError("A new document needs kind and code") - source = DocumentSource.model_validate( - {k: v for k, v in source_patch.items() if v is not None} - ).model_dump(exclude_none=True) - user_uuid = ( - upsert_user(uploaded_by_user_id, uploaded_by_username, uploaded_by_email, - uploaded_by_roles, conn=conn, tenants=[instance_value]) - if uploaded_by_user_id else None - ) - metadata = DocumentMetadata( - valid_from=valid_from, valid_to=valid_to, uploaded_by_user_id=user_uuid - ).model_dump(exclude_none=True) - catalog = get_or_create_catalog(get_or_create_instance(instance_value, conn=conn), kind, conn=conn) - try: - with conn.transaction(): - row = conn.execute( - """ - INSERT INTO documents (catalog_id, code, name, aliases, source, metadata) - VALUES (%s::uuid, %s, %s, %s, %s, %s) - RETURNING id - """, - ( - catalog["id"], code, name or code, - [a for a in (aliases or []) if a], Jsonb(source), Jsonb(metadata), - ), - ).fetchone() - except psycopg.errors.UniqueViolation as exc: - if exc.diag.constraint_name == "documents_catalog_id_code_key": - raise DuplicateDocumentCode( - f"Code '{code}' is already used in the {kind} catalog of {instance_value}" - ) from exc - raise - if with_workflow: - conn.execute( - """ - INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, - current_stage, error_message, stop_after_ocr, - started_by_user_id) - VALUES (%s::uuid, %s, %s, %s, %s, %s, %s::uuid) - """, - ( - str(row["id"]), - "ocr_only" if stop_after_ocr else "pipeline", - workflow_id, stage, error_message, bool(stop_after_ocr), user_uuid, - ), - ) - return str(row["id"]) - - -def reset_document_for_reupload( - doc_uuid: str, - *, - filename: str, - filepath: str, - fingerprint: str, - source_type: Optional[str] = None, - canonical_input_type: Optional[str] = None, -) -> None: - """Point an existing document at a re-uploaded file. - - ``source`` is replaced, pages and chunks are cleared for the new run, and - the document is flagged ``reindex_required`` because its vector index is now - stale. Earlier workflows and artifacts stay as history. - - The fingerprint is the document's key in the vector index, so the old key - is remembered in ``metadata.superseded_doc_ids`` for every index (dev/prod) - that holds it. The next ingest/promote and disable/purge delete those points; - without this they would stay searchable with nothing pointing at them. - """ - source = DocumentSource.model_validate({ - "fingerprint": fingerprint, - "original_filename": filename, - "storage_uri": filepath, - "source_type": source_type, - "canonical_input_type": canonical_input_type, - }).model_dump(exclude_none=True) - with get_connection() as conn: - current = conn.execute( - "SELECT source->>'fingerprint' AS fingerprint, metadata FROM documents " - "WHERE id = %s::uuid FOR UPDATE", - (doc_uuid,), - ).fetchone() - if current is None: - raise LookupError(f"Document not found: {doc_uuid}") - meta = current["metadata"] or {} - superseded = {k: list(v) for k, v in (meta.get("superseded_doc_ids") or {}).items()} - for env, entry in (meta.get("index_status") or {}).items(): - old_key = (entry or {}).get("doc_id") or current["fingerprint"] - if old_key and old_key != fingerprint and old_key not in superseded.get(env, []): - superseded.setdefault(env, []).append(old_key) - conn.execute("DELETE FROM pages WHERE document_id = %s::uuid", (doc_uuid,)) - conn.execute("DELETE FROM chunks WHERE document_id = %s::uuid", (doc_uuid,)) - conn.execute( - """ - UPDATE documents SET - source = %s::jsonb, - metadata = (metadata - 'chunking') - || jsonb_build_object('reindex_required', TRUE, 'reindex_reason', 'source_replaced', - 'superseded_doc_ids', %s::jsonb) - WHERE id = %s::uuid - """, - (Jsonb(source), Jsonb(superseded), doc_uuid), - ) - - -def doc_id_used_by_other_document(doc_id: str, exclude_doc_uuid: Optional[str], env: str) -> bool: - """True when another *live* document is indexed in ``env`` (dev/prod) under ``doc_id``. - - The index key is the file fingerprint, so two documents holding the same file - share points in the vector index. Deleting a key another document still needs - would remove that document's vectors too. Only a document that is not - disabled and whose own ``index_status[env]`` holds this key counts: one that - was merely uploaded, or is disabled, has nothing in the index to protect. - """ - with get_connection() as conn: - row = conn.execute( - """ - SELECT 1 FROM documents d - WHERE (%s::uuid IS NULL OR d.id <> %s::uuid) - AND NOT d.is_disabled - AND d.metadata->'index_status'->%s::text->>'doc_id' = %s - LIMIT 1 - """, - (exclude_doc_uuid, exclude_doc_uuid, env, doc_id), - ).fetchone() - return row is not None - - -def clear_superseded_doc_ids(workflow_id: str, env: str, doc_ids: list[str]) -> None: - """Forget superseded index keys once their points are deleted from ``env`` (dev/prod).""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return - row = conn.execute( - "SELECT metadata->'superseded_doc_ids' AS sup FROM documents WHERE id = %s::uuid FOR UPDATE", - (doc_uuid,), - ).fetchone() - superseded = {k: list(v) for k, v in (row["sup"] or {}).items()} - remaining = [i for i in superseded.get(env, []) if i not in set(doc_ids)] - if remaining: - superseded[env] = remaining - else: - superseded.pop(env, None) - conn.execute( - "UPDATE documents SET metadata = metadata || jsonb_build_object('superseded_doc_ids', %s::jsonb) " - "WHERE id = %s::uuid", - (Jsonb(superseded), doc_uuid), - ) - - -def _load(conn, doc_uuid: str, column: str) -> dict: - row = conn.execute(f"SELECT {column} FROM documents WHERE id = %s::uuid", (doc_uuid,)).fetchone() - return dict(row[column]) if row else {} - - -def update_document_stage( - workflow_id: str, - stage: str, - page_count: Optional[int] = None, - chunk_count: Optional[int] = None, - error_message: Optional[str] = None -): - """Update the stage (of the document's current workflow). - - ``page_count`` and ``chunk_count`` are accepted for call-site compatibility; - both are counted from ``pages`` and ``chunks``, so only the audit entry keeps them. - """ - old_stage = None - audit_workflow_id = workflow_id - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return - cw = conn.execute( - "SELECT id, current_stage FROM document_current_workflow WHERE document_id = %s::uuid", - (doc_uuid,), - ).fetchone() - if cw is None: - return - old_stage = cw["current_stage"] - - assignments = ["current_stage = %s"] - params: list = [stage] - if error_message is not None: - assignments.append("error_message = %s") - params.append(error_message) - stamp_key = { - "ocr_review": "ocr", - "translation_review": "translation", - "chunk_review": "chunks", - }.get(stage) - if stamp_key: - assignments.append( - "stage_timestamps = jsonb_build_object(%s::text, %s::text) || stage_timestamps" - ) - params.extend([stamp_key, _iso_now()]) - if stage == "completed": - assignments.append("completed_at = now()") - params.append(str(cw["id"])) - conn.execute( - f"UPDATE workflows SET {', '.join(assignments)} WHERE id = %s::uuid", params - ) - - conn.execute("UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,)) - audit_workflow_id = _root_workflow_id(conn, doc_uuid) or workflow_id - - if old_stage and old_stage != stage: - log_audit( - workflow_id=audit_workflow_id, - document_id=audit_workflow_id, - action_type="stage_change", - field_name="stage", - old_value=old_stage, - new_value=stage, - metadata={"page_count": page_count, "chunk_count": chunk_count} - ) - - -def _root_workflow_id(conn, doc_uuid: str) -> Optional[str]: - row = conn.execute( - """ - SELECT temporal_workflow_id FROM workflows WHERE document_id = %s::uuid - ORDER BY (workflow_type IN ('pipeline', 'ocr_only')) DESC, started_at ASC, id ASC - LIMIT 1 - """, - (doc_uuid,), - ).fetchone() - return row["temporal_workflow_id"] if row else None - - -def reconcile_materialized_state(workflow_id: str) -> Optional[dict]: - """ - Reconcile a document row against already-persisted page/chunk content. - - This is intentionally conservative: it only moves a document forward into - review states when the corresponding materialized content already exists. - """ - doc = get_document(workflow_id) - if not doc: - return None - workflow_id = doc["workflow_id"] - - with get_connection() as conn: - page_stats = conn.execute( - """ - SELECT COUNT(*) AS page_rows, COALESCE(MAX(page_number), 0) AS max_page_number - FROM pages WHERE document_id = %s::uuid - """, - (doc["id"],), - ).fetchone() - chunk_stats = conn.execute( - "SELECT COUNT(*) AS chunk_rows FROM chunks WHERE document_id = %s::uuid", - (doc["id"],), - ).fetchone() - - page_count = int((page_stats["page_rows"] if page_stats else 0) or 0) - max_page_number = int((page_stats["max_page_number"] if page_stats else 0) or 0) - materialized_pages = max(page_count, max_page_number, int(doc.get("page_count") or 0)) - chunk_count = int((chunk_stats["chunk_rows"] if chunk_stats else 0) or 0) - - current_stage = doc.get("stage") - latest_job = get_latest_document_job(workflow_id) - updated = False - now = _iso_now() - - # Highest-confidence reconciliation first: if chunks exist, the document has - # already reached chunk review materiality even if the workflow stalled. - if chunk_count > 0 and current_stage not in { - "chunk_review", - "ready_for_ingestion", - "ingesting", - "approval_for_prod", - "completed", - }: - update_document_stage( - workflow_id=workflow_id, - stage="chunk_review", - page_count=materialized_pages, - chunk_count=chunk_count, - error_message=None, - ) - if latest_job: - update_document_job( - latest_job["id"], - current_stage="chunk_review", - status="waiting_review", - error_message=None, - ) - doc = get_document(workflow_id) or doc - current_stage = doc.get("stage") - updated = True - - # Lower-confidence but still safe: if OCR pages exist and the document is - # somehow still marked as pre-review, move it into OCR review. - elif materialized_pages > 0 and current_stage in {"registered", "ocr_processing"}: - update_document_stage( - workflow_id=workflow_id, - stage="ocr_review", - page_count=materialized_pages, - chunk_count=int(doc.get("chunk_count") or 0), - error_message=None, - ) - if latest_job: - update_document_job( - latest_job["id"], - current_stage="ocr_review", - status="waiting_review", - error_message=None, - ) - updated = True - - # Even when stage is already correct, keep counts truthful. - elif materialized_pages != int(doc.get("page_count") or 0) or chunk_count != int(doc.get("chunk_count") or 0): - update_document_stage( - workflow_id=workflow_id, - stage=current_stage, - page_count=materialized_pages, - chunk_count=chunk_count, - error_message=doc.get("error_message"), - ) - updated = True - - if not updated: - return { - "workflow_id": workflow_id, - "updated": False, - "stage": current_stage, - "page_count": int(doc.get("page_count") or 0), - "chunk_count": int(doc.get("chunk_count") or 0), - } - - refreshed = get_document(workflow_id) or doc - refreshed_job = get_latest_document_job(workflow_id) - return { - "workflow_id": workflow_id, - "updated": True, - "stage": refreshed.get("stage"), - "page_count": int(refreshed.get("page_count") or 0), - "chunk_count": int(refreshed.get("chunk_count") or 0), - "job_status": refreshed_job.get("status") if refreshed_job else None, - "job_stage": refreshed_job.get("current_stage") if refreshed_job else None, - "reconciled_at": now, - } - - -def get_document(workflow_id: str) -> Optional[dict]: - """Get a single document by workflow id, UUID or any other identifier.""" - with get_connection() as conn: - return _fetch_document(conn, _resolve_document_uuid(conn, workflow_id)) - - -def list_documents( - stage: Optional[str] = None, - limit: int = 100, - offset: int = 0, - include_disabled: bool = False, - instances: Optional[list[str]] = None, -) -> list[dict]: - """List documents with optional stage / instance filters. - - Args: - stage: Filter by document stage - limit: Max documents to return - offset: Pagination offset - include_disabled: If False (default), excludes soft-deleted documents from results - instances: If provided, only return docs whose instance is in this list - """ - if instances is not None and not _normalize_instances(instances): - return [] - clauses, params = _document_filters(include_disabled, instances) - if stage: - clauses.append("cw.current_stage = %s") - params.append(stage) - where = f"WHERE {' AND '.join(clauses)}" if clauses else "" - with get_connection() as conn: - rows = conn.execute( - f"{_DOC_SELECT} {where} ORDER BY d.created_at DESC, d.id LIMIT %s OFFSET %s", - (*params, limit, offset), - ).fetchall() - return [_document_from_row(r) for r in rows] - - -_REVIEW_QUEUE_STAGES = ("ocr_review", "translation_review", "chunk_review", "approval_for_prod") - -_SUMMARY_KEYS = ( - "total_documents", - "authoritative_documents", - "legacy_documents", - "completed_documents", - "review_queue", - "ocr_review_documents", - "translation_review_documents", - "chunk_review_documents", - "translation_processing_documents", - "chunking_documents", - "ready_for_ingestion_documents", - "failed_documents", - "needs_reindex", - "running_jobs", -) - - -def get_document_summary_counts( - include_disabled: bool = False, - instances: Optional[list[str]] = None, -) -> dict: - """Return aggregate document counts for dashboard and migration planning.""" - if instances is not None and not _normalize_instances(instances): - return {key: 0 for key in _SUMMARY_KEYS} - clauses, params = _document_filters(include_disabled, instances) - where = f"WHERE {' AND '.join(clauses)}" if clauses else "" - with get_connection() as conn: - row = conn.execute( - f""" - SELECT - COUNT(*) AS total_documents, - 0 AS authoritative_documents, - COUNT(*) AS legacy_documents, - COUNT(*) FILTER (WHERE cw.current_stage = 'completed') AS completed_documents, - COUNT(*) FILTER (WHERE cw.current_stage IN - ('ocr_review', 'translation_review', 'chunk_review', 'approval_for_prod')) AS review_queue, - COUNT(*) FILTER (WHERE cw.current_stage = 'ocr_review') AS ocr_review_documents, - COUNT(*) FILTER (WHERE cw.current_stage = 'translation_review') AS translation_review_documents, - COUNT(*) FILTER (WHERE cw.current_stage = 'chunk_review') AS chunk_review_documents, - COUNT(*) FILTER (WHERE cw.current_stage = 'translation_processing') AS translation_processing_documents, - COUNT(*) FILTER (WHERE cw.current_stage = 'chunking') AS chunking_documents, - COUNT(*) FILTER (WHERE cw.current_stage = 'ready_for_ingestion') AS ready_for_ingestion_documents, - COUNT(*) FILTER (WHERE cw.current_stage = 'failed') AS failed_documents, - COUNT(*) FILTER (WHERE COALESCE((d.metadata->>'reindex_required')::boolean, FALSE)) AS needs_reindex, - COALESCE(SUM((SELECT COUNT(*) FROM workflows j - WHERE j.document_id = d.id AND j.status = 'running')), 0) AS running_jobs - FROM documents d - JOIN catalogs c ON c.id = d.catalog_id - JOIN instances g ON g.id = c.instance_id - LEFT JOIN document_current_workflow cw ON cw.document_id = d.id - {where} - """, - params, - ).fetchone() - return {key: int(row[key] or 0) for key in _SUMMARY_KEYS} - - -def get_document_counts_by_instance( - include_disabled: bool = False, - instances: Optional[list[str]] = None, -) -> list[dict]: - """Return document counts grouped by instance (state), for dashboard breakdowns.""" - if instances is not None and not _normalize_instances(instances): - return [] - clauses, params = _document_filters(include_disabled, instances) - where = f"WHERE {' AND '.join(clauses)}" if clauses else "" - with get_connection() as conn: - rows = conn.execute( - f""" - SELECT - lower(g.name) AS "instance", - COUNT(*) AS count, - COUNT(*) FILTER (WHERE cw.current_stage = 'completed') AS success, - COUNT(*) FILTER (WHERE cw.current_stage = 'failed') AS failed, - COUNT(*) FILTER (WHERE cw.current_stage IN - ('ocr_review', 'translation_review', 'chunk_review', 'approval_for_prod')) AS dev_approval - FROM documents d - JOIN catalogs c ON c.id = d.catalog_id - JOIN instances g ON g.id = c.instance_id - LEFT JOIN document_current_workflow cw ON cw.document_id = d.id - {where} - GROUP BY lower(g.name) - ORDER BY count DESC, "instance" ASC - """, - params, - ).fetchall() - return [ - { - "instance": row["instance"], - "count": int(row["count"] or 0), - "success": int(row["success"] or 0), - "failed": int(row["failed"] or 0), - "dev_approval": int(row["dev_approval"] or 0), - } - for row in rows - ] - - -def set_document_disabled(workflow_id: str, is_disabled: bool = True): - """Soft delete a document (filtered from all queries by default). - - Unlike hard delete, this preserves the document record for audit purposes. - Use X-Include-Disabled: true header in API to see disabled documents. - """ - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid: - conn.execute( - "UPDATE documents SET is_disabled = %s WHERE id = %s::uuid", - (bool(is_disabled), doc_uuid), - ) - - -def delete_document(workflow_id: str): - """Delete a document record and everything that cascades from it.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid: - conn.execute("DELETE FROM documents WHERE id = %s::uuid", (doc_uuid,)) - - -def purge_document(workflow_id: str, keep_audit: bool = True) -> dict: - """Hard delete every row belonging to a document. - - Unlike :func:`set_document_disabled` this is irreversible - there is - nothing left for ``POST /documents/{id}/restore`` to bring back. - - Audit rows are kept by default: they are the record that the deletion - happened, not part of the document itself. Their ``document_id`` becomes - NULL; the ``workflow_id`` and ``instance`` snapshots stay. - - Returns a per-table count of deleted rows. - """ - deleted: dict[str, int] = {} - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return deleted - root = _root_workflow_id(conn, doc_uuid) or workflow_id - - def _count(sql: str) -> int: - return int(conn.execute(sql, (doc_uuid,)).fetchone()["n"]) - - counts = { - "chunks": _count("SELECT count(*) AS n FROM chunks WHERE document_id = %s::uuid"), - "pages": _count("SELECT count(*) AS n FROM pages WHERE document_id = %s::uuid"), - "document_artifacts": _count( - "SELECT count(*) AS n FROM workflow_artifacts a JOIN workflows w ON w.id = a.workflow_id " - "WHERE w.document_id = %s::uuid" - ), - "document_jobs": _count("SELECT count(*) AS n FROM workflows WHERE document_id = %s::uuid"), - } - if not keep_audit: - row = conn.execute( - "DELETE FROM audit_logs WHERE document_id = %s::uuid OR workflow_id = %s", - (doc_uuid, root), - ) - if row.rowcount > 0: - deleted["audit_logs"] = row.rowcount - conn.execute("DELETE FROM documents WHERE id = %s::uuid", (doc_uuid,)) - for table, count in counts.items(): - if count > 0: - deleted[table] = count - deleted["documents"] = 1 - return deleted - - -def get_document_count(stage: Optional[str] = None) -> int: - """Get count of documents, optionally filtered by stage.""" - with get_connection() as conn: - if stage: - row = conn.execute( - """ - SELECT COUNT(*) AS cnt FROM documents d - JOIN document_current_workflow cw ON cw.document_id = d.id - WHERE cw.current_stage = %s - """, - (stage,), - ).fetchone() - else: - row = conn.execute("SELECT COUNT(*) AS cnt FROM documents").fetchone() - return int(row["cnt"]) if row else 0 - - -def update_document_fields(workflow_id: str, **updates: object) -> Optional[dict]: - """Update selected document fields and return the document. - - Flat names are mapped onto the tables: ``stage``/``error_message`` act on the - current workflow, ``code``/``name``/``aliases`` on the document columns, - a document's kind is fixed by its catalog at upload, and flags, - period and uploader go into ``metadata``. Names derived from other rows (counts, artifact and job - pointers, completion timestamps) are accepted and ignored. - """ - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return None - if not updates: - return _fetch_document(conn, doc_uuid) - - doc_sets: list[str] = [] - doc_params: list = [] - source_patch: dict[str, Any] = {} - meta_patch: dict[str, Any] = {} - meta_remove: list[str] = [] - wf_sets: list[str] = [] - wf_params: list = [] - - for key, value in updates.items(): - if key == "stage": - wf_sets.append("current_stage = %s") - wf_params.append(value) - elif key == "error_message": - wf_sets.append("error_message = %s") - wf_params.append(value) - elif key == "source_type": - source_patch["source_type"] = value - elif key == "canonical_input_type": - source_patch["canonical_input_type"] = value - elif key == "filepath": - source_patch["storage_uri"] = value - elif key in ("filename", "source_filename"): - if key == "filename" or "filename" not in updates: - source_patch["original_filename"] = value - elif key in ("document_id", "canonical_document_id", "source_file_fingerprint"): - if isinstance(value, str) and _FINGERPRINT_RE.match(value): - source_patch["fingerprint"] = value - elif key == "name": - if value: - doc_sets.append("name = %s") - doc_params.append(value) - elif key == "code": - if value: - doc_sets.append("code = %s") - doc_params.append(value) - elif key == "aliases": - doc_sets.append("aliases = %s") - doc_params.append([a for a in (value or []) if a]) - elif key in ("reindex_required", "catalog_visible", "network_visible"): - meta_patch[key] = bool(value) - elif key == "reindex_reason": - if value: - meta_patch["reindex_reason"] = value - else: - meta_remove.append("reindex_reason") - elif key == "stop_after_ocr": - conn.execute( - "UPDATE workflows SET stop_after_ocr = %s WHERE id = (" - "SELECT id FROM workflows WHERE document_id = %s::uuid " - "ORDER BY (workflow_type IN ('pipeline', 'ocr_only')) DESC, started_at ASC, id ASC LIMIT 1)", - (bool(value), doc_uuid), - ) - elif key in ("valid_from", "valid_to"): - meta_patch[key] = value.isoformat() if isinstance(value, date) else value - elif key == "uploaded_by_user_id" and value: - has_uploader = conn.execute( - "SELECT metadata ? 'uploaded_by_user_id' AS has FROM documents WHERE id = %s::uuid", - (doc_uuid,), - ).fetchone()["has"] - if not has_uploader: - meta_patch["uploaded_by_user_id"] = upsert_user( - str(value), - updates.get("uploaded_by_username"), - updates.get("uploaded_by_email"), - updates.get("uploaded_by_roles"), - conn=conn, - ) - - if source_patch: - doc_sets.append("source = source || %s::jsonb") - doc_params.append(Jsonb({k: v for k, v in source_patch.items() if v is not None})) - if meta_patch or meta_remove: - expr = "metadata" - for removed in meta_remove: - expr = f"({expr} - '{removed}')" - if meta_patch: - expr = f"{expr} || %s::jsonb" - doc_params.append(Jsonb(meta_patch)) - doc_sets.append(f"metadata = {expr}") - - if wf_sets: - conn.execute( - f"UPDATE workflows SET {', '.join(wf_sets)} WHERE id = (" - "SELECT id FROM document_current_workflow WHERE document_id = %s::uuid)", - (*wf_params, doc_uuid), - ) - if doc_sets: - try: - with conn.transaction(): - conn.execute( - f"UPDATE documents SET {', '.join(doc_sets)} WHERE id = %s::uuid", - (*doc_params, doc_uuid), - ) - except psycopg.errors.UniqueViolation as exc: - if exc.diag.constraint_name == "documents_catalog_id_code_key": - raise DuplicateDocumentCode( - "Another document in the target catalog already uses this code" - ) from exc - raise - elif wf_sets: - conn.execute("UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,)) - return _fetch_document(conn, doc_uuid) - - -def mark_document_reindex_required( - workflow_id: str, - required: bool = True, - reason: Optional[str] = None, -) -> Optional[dict]: - return update_document_fields( - workflow_id, - reindex_required=bool(required), - reindex_reason=reason if required else None, - ) - - -def set_document_validity( - workflow_id: str, - valid_from: str, - valid_to: str, -) -> Optional[dict]: - """Record the period this document's chunks are searchable in. Both ends - are `YYYY-MM-DD`; validation belongs to the caller - (`pipeline/document_validity.parse_period`), not to this SQL layer.""" - return update_document_fields(workflow_id, valid_from=valid_from, valid_to=valid_to) - - -def mark_prod_ready_requested( - workflow_id: str, - user_id: Optional[str] = None, - username: Optional[str] = None, -) -> Optional[dict]: - """Flag that a state_admin has asked a super_admin to review this document - for prod promotion. Purely informational - does not fire the approve_prod - signal itself; the super_admin still makes that call separately. - - Recorded as a ``promote_prod`` workflow in ``waiting_review``.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return None - root = _root_workflow_id(conn, doc_uuid) or workflow_id - user_uuid = upsert_user(user_id, username, None, None, conn=conn) if user_id else None - conn.execute( - """ - UPDATE workflows SET status = 'running' - WHERE document_id = %s::uuid AND workflow_type = 'promote_prod' AND status = 'waiting_review' - """, - (doc_uuid,), - ) - conn.execute( - """ - INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, status, - current_stage, started_by_user_id) - VALUES (%s::uuid, 'promote_prod', %s, 'waiting_review', 'approval_for_prod', %s::uuid) - """, - (doc_uuid, f"{root}-prod-request", user_uuid), - ) - return _fetch_document(conn, doc_uuid) - - -def clear_prod_ready_requested(workflow_id: str) -> Optional[dict]: - """Reset the request flag once prod approval actually happens (or the - document leaves the approval_for_prod gate for any other reason).""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return None - conn.execute( - """ - UPDATE workflows SET status = 'running' - WHERE document_id = %s::uuid AND workflow_type = 'promote_prod' AND status = 'waiting_review' - """, - (doc_uuid,), - ) - return _fetch_document(conn, doc_uuid) - - -# ============================================================================= -# Workflows (exposed as "jobs") -# ============================================================================= - -_JOB_SELECT = """ - SELECT w.id, w.document_id, w.workflow_type, w.temporal_workflow_id, w.status, - w.current_stage, w.config, w.error_message, w.started_at, w.completed_at, - root.temporal_workflow_id AS doc_workflow_id - FROM workflows w - LEFT JOIN LATERAL ( - SELECT r.temporal_workflow_id FROM workflows r - WHERE r.document_id = w.document_id - ORDER BY (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC - LIMIT 1 - ) root ON TRUE -""" - - -def _job_from_row(row: Optional[dict]) -> Optional[dict]: - if row is None: - return None - config = row["config"] or {} - job = { - "id": row["id"], - "workflow_id": row["doc_workflow_id"], - "job_type": row["workflow_type"], - "temporal_workflow_id": row["temporal_workflow_id"], - "temporal_run_id": config.get("temporal_run_id"), - "status": row["status"], - "current_stage": row["current_stage"], - "started_at": row["started_at"], - "completed_at": row["completed_at"], - "error_message": row["error_message"], - "config": config, - "config_json": json.dumps(config) if config else None, - } - return {k: _clean(v) for k, v in job.items()} - - -def create_document_job( - workflow_id: str, - job_type: str, - temporal_workflow_id: Optional[str] = None, - temporal_run_id: Optional[str] = None, - status: str = "running", - current_stage: Optional[str] = None, - config: Optional[dict] = None, - actor: Optional[dict] = None, -) -> str: - """Record a workflow run for a document; returns the workflow UUID. - - A ``promote_prod`` run adopts the document's pending request row (see - ``mark_prod_ready_requested``) instead of adding a second one. - """ - cfg = dict(config or {}) - if temporal_run_id: - cfg["temporal_run_id"] = temporal_run_id - with get_connection() as conn: - doc_uuid = _require_document_uuid(conn, workflow_id) - root = _root_workflow_id(conn, doc_uuid) or workflow_id - user_uuid = None - if actor and actor.get("user_id"): - user_uuid = upsert_user( - actor["user_id"], actor.get("username"), actor.get("email"), actor.get("roles"), - conn=conn, - ) - if job_type == "promote_prod": - adopted = conn.execute( - """ - UPDATE workflows SET temporal_workflow_id = %s, status = %s, - current_stage = COALESCE(%s, current_stage), - config = config || %s::jsonb - WHERE id = ( - SELECT id FROM workflows - WHERE document_id = %s::uuid AND workflow_type = 'promote_prod' - AND status = 'waiting_review' - ORDER BY started_at DESC LIMIT 1) - RETURNING id - """, - (temporal_workflow_id or root, status, current_stage, Jsonb(cfg), doc_uuid), - ).fetchone() - if adopted: - conn.execute("UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,)) - return str(adopted["id"]) - row = conn.execute( - """ - INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, status, - current_stage, stop_after_ocr, config, started_by_user_id) - VALUES (%s::uuid, %s, %s, %s, - COALESCE(%s, 'registered'), %s, %s, %s::uuid) - RETURNING id - """, - ( - doc_uuid, job_type, temporal_workflow_id or root, status, current_stage, - bool(cfg.get("stop_after_ocr")), Jsonb(cfg), user_uuid, - ), - ).fetchone() - conn.execute("UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,)) - return str(row["id"]) - - -def update_document_job(job_id: str, **updates: object) -> Optional[dict]: - if not updates: - return get_document_job(job_id) - if not _is_uuid(str(job_id)): - return None - allowed_fields = { - "status", "current_stage", "completed_at", "error_message", "temporal_run_id", "config_json" - } - set_clauses = [] - values: list[object] = [] - for key, value in updates.items(): - if key not in allowed_fields: - continue - if key == "status": - set_clauses.append("status = %s") - elif key == "current_stage": - set_clauses.append("current_stage = %s") - elif key == "completed_at": - set_clauses.append("completed_at = %s::timestamptz") - elif key == "temporal_run_id": - set_clauses.append("config = config || jsonb_build_object('temporal_run_id', %s::text)") - elif key == "config_json": - if isinstance(value, str): - value = json.loads(value) - set_clauses.append("config = config || %s::jsonb") - value = Jsonb(value or {}) - else: - set_clauses.append(f"{key} = %s") - values.append(value) - if not set_clauses: - return get_document_job(job_id) - values.append(str(job_id)) - with get_connection() as conn: - conn.execute(f"UPDATE workflows SET {', '.join(set_clauses)} WHERE id = %s::uuid", values) - return get_document_job(job_id) - - -def get_document_job(job_id: str) -> Optional[dict]: - if not _is_uuid(str(job_id)): - return None - with get_connection() as conn: - return _job_from_row( - conn.execute(_JOB_SELECT + " WHERE w.id = %s::uuid", (str(job_id),)).fetchone() - ) - - -def get_latest_document_job(workflow_id: str) -> Optional[dict]: - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return None - return _job_from_row( - conn.execute( - _JOB_SELECT + " WHERE w.document_id = %s::uuid ORDER BY w.started_at DESC, w.id DESC LIMIT 1", - (doc_uuid,), - ).fetchone() - ) - - -def list_document_jobs(workflow_id: str, limit: int = 50) -> list[dict]: - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return [] - rows = conn.execute( - _JOB_SELECT + " WHERE w.document_id = %s::uuid ORDER BY w.started_at DESC, w.id DESC LIMIT %s", - (doc_uuid, limit), - ).fetchall() - return [_job_from_row(r) for r in rows] - - -def list_operations_queue( - limit: int = 100, - offset: int = 0, - include_disabled: bool = False, - instances: Optional[list[str]] = None, -) -> tuple[list[dict], int]: - """List documents needing action, optionally scoped to tenant instances. - - ``instances``: - - ``None`` - unrestricted (super admin / bypass) - - ``[]`` - restricted user with no states -> empty result - - non-empty - only those document.instance values - """ - if instances is not None and not _normalize_instances(instances): - return [], 0 - clauses, params = _document_filters(include_disabled, instances) - clauses.append( - "(cw.current_stage IN ('ocr_review', 'translation_review', 'chunk_review', " - "'ready_for_ingestion', 'approval_for_prod', 'failed') " - "OR COALESCE((d.metadata->>'reindex_required')::boolean, FALSE) " - "OR cw.status = 'running')" - ) - where = f"WHERE {' AND '.join(clauses)}" - from_sql = """ - FROM documents d - JOIN catalogs c ON c.id = d.catalog_id - JOIN instances g ON g.id = c.instance_id - JOIN document_current_workflow cw ON cw.document_id = d.id - """ - with get_connection() as conn: - total_row = conn.execute(f"SELECT COUNT(*) AS count {from_sql} {where}", params).fetchone() - rows = conn.execute( - f""" - SELECT - d.id AS id, - (SELECT r.temporal_workflow_id FROM workflows r WHERE r.document_id = d.id - ORDER BY (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC - LIMIT 1) AS workflow_id, - d.source->>'original_filename' AS filename, - cw.current_stage AS stage, - lower(g.name) AS "instance", - cw.error_message, - COALESCE((d.metadata->>'reindex_required')::boolean, FALSE) AS reindex_required, - d.metadata->>'reindex_reason' AS reindex_reason, - cw.id AS job_id, - cw.workflow_type AS job_type, - cw.status AS job_status, - cw.started_at AS started_at - {from_sql} - {where} - ORDER BY cw.started_at DESC, d.id - LIMIT %s OFFSET %s - """, - (*params, limit, offset), - ).fetchall() - return [_clean_row(r) for r in rows], int(total_row["count"] if total_row else 0) - - -def list_runs( - limit: int = 100, - offset: int = 0, - status: Optional[str] = None, - instances: Optional[list[str]] = None, -) -> list[dict]: - """List workflows with document title fields for the Runs UI. - - Args: - instances: If provided, only return runs whose document instance is in - this list. Empty list returns no rows. None = unrestricted. - """ - where_parts: list[str] = [] - params: list = [] - if status: - where_parts.append("w.status = %s") - params.append(status) - if instances is not None: - normalized = _normalize_instances(instances) - if not normalized: - return [] - where_parts.append("lower(g.name) = ANY(%s)") - params.append(normalized) - where_sql = f"WHERE {' AND '.join(where_parts)}" if where_parts else "" - with get_connection() as conn: - rows = conn.execute( - f""" - SELECT w.id, w.document_id, w.workflow_type, w.temporal_workflow_id, w.status, - w.current_stage, w.config, w.error_message, w.started_at, w.completed_at, - (SELECT r.temporal_workflow_id FROM workflows r WHERE r.document_id = w.document_id - ORDER BY (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC - LIMIT 1) AS doc_workflow_id, - d.source->>'original_filename' AS filename, - d.name AS name, - d.source->>'original_filename' AS source_filename, - cw.current_stage AS document_stage, - lower(g.name) AS "instance" - FROM workflows w - JOIN documents d ON d.id = w.document_id - JOIN catalogs c ON c.id = d.catalog_id - JOIN instances g ON g.id = c.instance_id - LEFT JOIN document_current_workflow cw ON cw.document_id = d.id - {where_sql} - ORDER BY w.started_at DESC, w.id DESC - LIMIT %s OFFSET %s - """, - (*params, limit, offset), - ).fetchall() - result = [] - for row in rows: - job = _job_from_row(row) - job["document_uuid"] = str(row["document_id"]) - for extra in ("filename", "name", "source_filename", "document_stage", "instance"): - job[extra] = row[extra] - result.append(job) - return result - - -# ============================================================================= -# Artifacts -# ============================================================================= - -def _artifact_from_row(row: Optional[dict]) -> Optional[dict]: - if row is None: - return None - metadata = row["metadata"] or {} - artifact = { - "id": row["id"], - "workflow_id": row["doc_workflow_id"], - "job_id": row["workflow_pk"], - "artifact_type": row["artifact_type"], - "stage": row["stage"], - "storage_uri": row["storage_uri"], - "mime_type": row["mime_type"], - "filename": row["filename"], - "size_bytes": row["size_bytes"], - "metadata_json": json.dumps(metadata) if metadata else None, - "created_at": row["created_at"], - } - return {k: _clean(v) for k, v in artifact.items()} - - -_ARTIFACT_SELECT = """ - SELECT a.id, a.workflow_id AS workflow_pk, a.artifact_type, a.stage, a.storage_uri, - a.mime_type, a.filename, a.size_bytes, a.metadata, a.created_at, - (SELECT r.temporal_workflow_id FROM workflows r WHERE r.document_id = w.document_id - ORDER BY (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC - LIMIT 1) AS doc_workflow_id - FROM workflow_artifacts a - JOIN workflows w ON w.id = a.workflow_id -""" - - -def add_document_artifact( - workflow_id: str, - artifact_type: str, - storage_uri: str, - stage: Optional[str] = None, - job_id: Optional[str] = None, - mime_type: Optional[str] = None, - filename: Optional[str] = None, - size_bytes: Optional[int] = None, - metadata: Optional[dict] = None, -) -> str: - """Record a file produced by a workflow; returns the artifact UUID. - - Belongs to ``job_id`` when given, else to the document's current workflow. - """ - with get_connection() as conn: - doc_uuid = _require_document_uuid(conn, workflow_id) - wf_uuid = None - if job_id and _is_uuid(str(job_id)): - row = conn.execute( - "SELECT id FROM workflows WHERE id = %s::uuid AND document_id = %s::uuid", - (str(job_id), doc_uuid), - ).fetchone() - wf_uuid = str(row["id"]) if row else None - if wf_uuid is None: - row = conn.execute( - "SELECT id FROM document_current_workflow WHERE document_id = %s::uuid", (doc_uuid,) - ).fetchone() - if row is None: - row = conn.execute( - """ - INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id) - VALUES (%s::uuid, 'pipeline', %s) RETURNING id - """, - (doc_uuid, workflow_id), - ).fetchone() - wf_uuid = str(row["id"]) - created = conn.execute( - """ - INSERT INTO workflow_artifacts (workflow_id, artifact_type, stage, storage_uri, - mime_type, filename, size_bytes, metadata) - VALUES (%s::uuid, %s, %s, %s, %s, %s, %s, %s) - RETURNING id - """, - ( - wf_uuid, artifact_type, stage, storage_uri, mime_type, filename, size_bytes, - Jsonb(metadata or {}), - ), - ).fetchone() - return str(created["id"]) - - -def list_document_artifacts(workflow_id: str) -> list[dict]: - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return [] - rows = conn.execute( - _ARTIFACT_SELECT + " WHERE w.document_id = %s::uuid ORDER BY a.created_at DESC, a.id DESC", - (doc_uuid,), - ).fetchall() - return [_artifact_from_row(r) for r in rows] - - -def get_document_artifact(workflow_id: str, artifact_id: str) -> Optional[dict]: - if not _is_uuid(str(artifact_id)): - return None - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return None - return _artifact_from_row( - conn.execute( - _ARTIFACT_SELECT + " WHERE w.document_id = %s::uuid AND a.id = %s::uuid", - (doc_uuid, str(artifact_id)), - ).fetchone() - ) - - -# ============================================================================= -# Vector index state (documents.metadata.index_status) -# ============================================================================= - -def _index_entry_to_legacy(workflow_id: str, key: str, entry: dict) -> dict: - details = entry.get("details") or {} - return { - "workflow_id": workflow_id, - "index_name": entry.get("index_name") or key, - "doc_id": entry.get("doc_id"), - "chunk_count_indexed": entry.get("chunk_count_indexed") or 0, - "last_indexed_at": entry.get("last_indexed_at"), - "last_verified_at": entry.get("last_verified_at"), - "schema_version": entry.get("schema_version"), - "status": entry.get("status") or "unknown", - "details_json": json.dumps(details) if details else None, - } - - -def upsert_document_index_status( - workflow_id: str, - index_name: str, - doc_id: Optional[str] = None, - chunk_count_indexed: Optional[int] = None, - last_indexed_at: Optional[str] = None, - last_verified_at: Optional[str] = None, - schema_version: Optional[str] = None, - status: str = "unknown", - details: Optional[dict] = None, -) -> dict: - """Merge one index's state into ``documents.metadata.index_status``. - - The key is ``prod`` for the PROD collection names and ``dev`` otherwise; - the collection name is kept inside the entry. ``None`` arguments leave the - stored value alone, except ``status`` which always replaces it. - """ - key = _index_key_for(index_name) - with get_connection() as conn: - doc_uuid = _require_document_uuid(conn, workflow_id) - row = conn.execute( - "SELECT metadata->'index_status'->%s::text AS entry FROM documents WHERE id = %s::uuid FOR UPDATE", - (key, doc_uuid), - ).fetchone() - existing = dict(row["entry"] or {}) - incoming = { - "index_name": index_name, - "doc_id": doc_id, - "chunk_count_indexed": chunk_count_indexed, - "last_indexed_at": last_indexed_at, - "last_verified_at": last_verified_at, - "schema_version": schema_version, - "details": details or None, - } - merged = {**existing, **{k: v for k, v in incoming.items() if v is not None}, "status": status} - validated = IndexState.model_validate(merged).model_dump() - conn.execute( - "UPDATE documents SET metadata = jsonb_set(" - "metadata || jsonb_build_object('index_status', COALESCE(metadata->'index_status', '{}'::jsonb)), " - "ARRAY['index_status', %s::text], %s::jsonb) " - "WHERE id = %s::uuid", - (key, Jsonb(validated), doc_uuid), - ) - root = _root_workflow_id(conn, doc_uuid) or workflow_id - return _index_entry_to_legacy(root, key, validated) - - -def list_document_index_status(workflow_id: str) -> list[dict]: - doc = get_document(workflow_id) - if not doc: - return [] - entries = [ - _index_entry_to_legacy(doc["workflow_id"], key, entry) - for key, entry in doc["index_status"].items() - ] - entries.sort( - key=lambda e: (e["last_verified_at"] or "", e["last_indexed_at"] or ""), reverse=True - ) - return entries - - -def get_document_index_status(workflow_id: str, index_name: str) -> Optional[dict]: - for entry in list_document_index_status(workflow_id): - if entry["index_name"] == index_name: - return entry - return None - - -def list_index_summaries(include_disabled: bool = False) -> list[dict]: - clauses, params = _document_filters(include_disabled, None) - where = f"AND {' AND '.join(clauses)}" if clauses else "" - with get_connection() as conn: - rows = conn.execute( - f""" - SELECT - COALESCE(e.value->>'index_name', e.key) AS index_name, - COUNT(*) AS documents, - SUM(COALESCE((e.value->>'chunk_count_indexed')::int, 0)) AS indexed_chunks, - SUM(CASE WHEN COALESCE((d.metadata->>'reindex_required')::boolean, FALSE) - THEN 1 ELSE 0 END) AS stale_documents, - MAX(e.value->>'last_indexed_at') AS last_indexed_at, - array_agg(DISTINCT COALESCE(e.value->>'status', 'unknown')) AS statuses - FROM documents d, jsonb_each(COALESCE(d.metadata->'index_status', '{{}}'::jsonb)) e - WHERE TRUE {where} - GROUP BY COALESCE(e.value->>'index_name', e.key) - ORDER BY index_name - """, - params, - ).fetchall() - result = [] - for row in rows: - item = _clean_row(row) - item["indexed_chunks"] = int(item["indexed_chunks"] or 0) - item["stale_documents"] = int(item["stale_documents"] or 0) - item["statuses"] = sorted(s for s in row["statuses"] if s) - result.append(item) - return result - - -def find_document_by_doc_identifier(identifier: str) -> Optional[dict]: - """Resolve a vector-store/chat doc identifier to a document. - - Matches the document UUID, a Temporal workflow id, the fingerprint, the - filename, an index ``doc_id``, or the legacy ``md5(fingerprint)`` doc id. - """ - if not identifier: - return None - with get_connection() as conn: - return _fetch_document(conn, _resolve_document_uuid(conn, identifier)) - - -def resolve_chunk_provenance( - *, - doc_id: Optional[str] = None, - chunk_num: Optional[int] = None, -) -> Optional[dict]: - """Resolve workflow + chunk metadata for provenance links.""" - if not doc_id or chunk_num is None: - return None - - doc = find_document_by_doc_identifier(doc_id) - if not doc: - return None - - workflow_id = doc["workflow_id"] - chunk = get_chunk(workflow_id, int(chunk_num)) - if not chunk: - return None - - text = chunk.get("edited_text") or chunk.get("original_text") or "" - excerpt = text[:320] + ("..." if len(text) > 320 else "") - page_start = int(chunk.get("page_start") or 1) - page_end = int(chunk.get("page_end") or page_start) - doc_name = doc.get("name") or doc.get("filename") or workflow_id - - return { - "workflow_id": workflow_id, - "doc_id": doc.get("document_id"), - "filename": workflow_id, - "doc_name": doc_name, - "chunk_num": int(chunk_num), - "page_start": page_start, - "page_end": page_end, - "page_range": f"{page_start}-{page_end}" if page_end != page_start else str(page_start), - "section": "", - "excerpt": excerpt, - } - - -# ============================================================================= -# Audit Log Functions -# ============================================================================= - -_AUDIT_ENTITY_TYPES = {"document", "page", "chunk", "setting", "catalog"} - - -def log_audit( - workflow_id: str, - document_id: str, - action_type: str, - entity_type: Optional[str] = None, - entity_id: Optional[int] = None, - field_name: Optional[str] = None, - old_value: Optional[str] = None, - new_value: Optional[str] = None, - metadata: Optional[dict] = None, - actor_user_id: Optional[str] = None, - actor_username: Optional[str] = None, - actor_email: Optional[str] = None, - actor_roles: Optional[str] = None, -) -> str: - """ - Log an audit entry. - - Args: - workflow_id: The Temporal workflow ID (or document identifier). Ids that - start with ``__`` mark system-level entries and are not stored. - document_id: Unused; kept for call-site compatibility. The document is - resolved from ``workflow_id``. - action_type: Type of action (stage_change, page_edit, chunk_edit, approval, reset) - entity_type: Type of entity (document, page, chunk, setting, catalog) - entity_id: Page number or chunk number the action applied to - field_name: Name of the field that changed - old_value: Previous value (as string or JSON) - new_value: New value (as string or JSON) - metadata: Additional context as dict - actor_*: Identity of the human/system that performed the action - - Returns: - The UUID of the created audit log entry - """ - entity = entity_type if entity_type in _AUDIT_ENTITY_TYPES else None - system_entry = (workflow_id or "").startswith("__") - roles = _csv_roles(actor_roles) - with get_connection() as conn: - doc_uuid = None if system_entry else _resolve_document_uuid(conn, workflow_id) - instance_uuid = None - stored_workflow_id = None if system_entry else workflow_id - if doc_uuid: - doc_row = conn.execute( - "SELECT c.instance_id FROM documents d JOIN catalogs c ON c.id = d.catalog_id " - "WHERE d.id = %s::uuid", - (doc_uuid,), - ).fetchone() - instance_uuid = str(doc_row["instance_id"]) - if _is_uuid(workflow_id): - stored_workflow_id = _root_workflow_id(conn, doc_uuid) or workflow_id - if stored_workflow_id is None and entity not in ("setting", "catalog"): - stored_workflow_id = workflow_id - actor_uuid = ( - upsert_user(actor_user_id, actor_username, actor_email, roles, conn=conn) - if actor_user_id else None - ) - row = conn.execute( - """ - INSERT INTO audit_logs (document_id, workflow_id, instance_id, action_type, entity_type, - entity_number, field_name, old_value, new_value, metadata, - actor_user_id, actor) - VALUES (%s::uuid, %s, %s::uuid, %s, %s, %s, %s, %s, %s, %s, %s::uuid, %s) - RETURNING id - """, - ( - doc_uuid, stored_workflow_id, instance_uuid, action_type, entity, entity_id, - field_name, old_value, new_value, Jsonb(metadata or {}), actor_uuid, - Jsonb({"username": actor_username or None, "email": actor_email or None, "roles": roles}) - if (actor_user_id or actor_username or actor_email or roles) else Jsonb({}), - ), - ).fetchone() - return str(row["id"]) - - -# Keycloak attaches these to every account regardless of what the user can -# actually do here. Rendering them made the actor column a wall of noise -# ("...manage-account,manage-account-links,offline_access,uma_authorization...") -# that buried the one role a reviewer cares about. -_NOISE_ROLES = { - "manage-account", - "manage-account-links", - "offline_access", - "uma_authorization", - "view-profile", - "account", -} - -# Most-privileged first, so a multi-role account is labelled by what it can do. -_ROLE_PRECEDENCE = ( - "superadmin", - "super_admin", - "master_admin", - "admin", - "content_curator", - "reviewer", - "contributor", - "viewer", -) - - -def meaningful_roles(raw: Optional[str]) -> list[str]: - """Application roles from a stored role list, Keycloak built-ins removed.""" - parts = [p.strip() for p in (raw or "").split(",") if p.strip()] - kept = [p for p in parts if p.lower() not in _NOISE_ROLES and not p.lower().startswith("default-roles")] - ranked = [r for r in _ROLE_PRECEDENCE if r in {k.lower() for k in kept}] - if ranked: - # Preserve original casing of the first matching role. - first = next(k for k in kept if k.lower() == ranked[0]) - return [first] - return kept[:1] - - -def _normalize_audit_row(row) -> dict: - entry = dict(row) - email = (entry.get("actor_email") or "").strip() - username = (entry.get("actor_username") or "").strip() - roles = meaningful_roles(entry.get("actor_roles")) - role_label = roles[0] if roles else "" - - who = email or username - if who and role_label: - entry["actor"] = f"{who} ({role_label})" - elif who: - entry["actor"] = who - else: - entry["actor"] = "system" - - # Split out so the UI can render identity and role separately rather than - # parsing the combined string back apart. - entry["actor_display"] = who or "System" - entry["actor_role"] = role_label - entry["is_system"] = not who - return entry - - -_AUDIT_SELECT = """ - SELECT a.id, a.workflow_id, a.action_type, a.entity_type, a.entity_number, a.field_name, - a.old_value, a.new_value, a.metadata, a.created_at, a.actor, ag.name AS audit_instance, - au.keycloak_user_id AS actor_keycloak_id, - d.source->>'fingerprint' AS doc_fingerprint, - d.source->>'original_filename' AS filename, - d.name AS document_name, - dg.name AS doc_instance, - uu.username AS uploaded_by_username, uu.email AS uploaded_by_email - FROM audit_logs a - LEFT JOIN documents d ON d.id = a.document_id - LEFT JOIN instances ag ON ag.id = a.instance_id - LEFT JOIN catalogs dc ON dc.id = d.catalog_id - LEFT JOIN instances dg ON dg.id = dc.instance_id - LEFT JOIN users au ON au.id = a.actor_user_id - LEFT JOIN users uu ON uu.id = NULLIF(d.metadata->>'uploaded_by_user_id', '')::uuid -""" - - -def _audit_row_to_legacy(row: dict) -> dict: - actor = row["actor"] or {} - metadata = row["metadata"] or {} - entity_type = row["entity_type"] - if entity_type == "setting": - workflow_id, document_id = "__system__", "__settings__" - elif entity_type == "catalog" and not row["workflow_id"]: - workflow_id, document_id = "__catalog__", "__catalog__" - else: - workflow_id = row["workflow_id"] - document_id = row["doc_fingerprint"] or workflow_id - roles = actor.get("roles") or [] - entry = { - "id": row["id"], - "workflow_id": workflow_id, - "document_id": document_id, - "action_type": row["action_type"], - "entity_type": entity_type, - "entity_id": row["entity_number"], - "field_name": row["field_name"], - "old_value": row["old_value"], - "new_value": row["new_value"], - "metadata": json.dumps(metadata) if metadata else None, - "timestamp": row["created_at"], - "actor_user_id": row["actor_keycloak_id"], - "actor_username": actor.get("username"), - "actor_email": actor.get("email"), - "actor_roles": ",".join(roles) if roles else None, - "filename": row["filename"], - "name": row["document_name"], - "instance": row["doc_instance"] or row["audit_instance"], - "uploaded_by_username": row["uploaded_by_username"], - "uploaded_by_email": row["uploaded_by_email"], - } - return _normalize_audit_row({k: _clean(v) for k, v in entry.items()}) - - -def _audit_scope(workflow_id: str, conn) -> tuple[str, list]: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid: - root = _root_workflow_id(conn, doc_uuid) or workflow_id - return "(a.document_id = %s::uuid OR a.workflow_id = %s)", [doc_uuid, root] - return "a.workflow_id = %s", [workflow_id] - - -def get_audit_logs( - workflow_id: str, - action_type: Optional[str] = None, - limit: int = 100, - offset: int = 0 -) -> list[dict]: - """ - Get audit logs for a document. - - Args: - workflow_id: The Temporal workflow ID (or document UUID) - action_type: Optional filter by action type - limit: Maximum number of entries to return - offset: Offset for pagination - - Returns: - List of audit log entries as dicts - """ - with get_connection() as conn: - where, params = _audit_scope(workflow_id, conn) - if action_type: - where += " AND a.action_type = %s" - params.append(action_type) - rows = conn.execute( - f"{_AUDIT_SELECT} WHERE {where} ORDER BY a.created_at DESC, a.id LIMIT %s OFFSET %s", - (*params, limit, offset), - ).fetchall() - return [_audit_row_to_legacy(r) for r in rows] - - -def get_audit_log_count(workflow_id: str, action_type: Optional[str] = None) -> int: - """Get count of audit logs for a document.""" - with get_connection() as conn: - where, params = _audit_scope(workflow_id, conn) - if action_type: - where += " AND a.action_type = %s" - params.append(action_type) - row = conn.execute(f"SELECT COUNT(*) AS cnt FROM audit_logs a WHERE {where}", params).fetchone() - return int(row["cnt"]) if row else 0 - - -def _global_audit_filters( - action_type: Optional[str], instances: Optional[list[str]] -) -> tuple[list[str], list]: - where_parts: list[str] = ["a.action_type <> %s"] - params: list = [CATALOG_VERSION_ACTION] - if action_type: - where_parts.append("a.action_type = %s") - params.append(action_type) - if instances is not None: - where_parts.append("lower(COALESCE(ag.name, dg.name, %s)) = ANY(%s)") - params.extend([_default_instance(), _normalize_instances(instances)]) - return where_parts, params - - -def get_all_audit_logs( - action_type: Optional[str] = None, - limit: int = 100, - offset: int = 0, - instances: Optional[list[str]] = None, -) -> list[dict]: - """ - Get all audit logs across all documents. - - Args: - action_type: Optional filter by action type - limit: Maximum number of entries to return - offset: Offset for pagination - instances: Tenant scope (None = all, [] = none) - - Returns: - List of audit log entries as dicts, with document filename included - """ - if instances is not None and not _normalize_instances(instances): - return [] - where_parts, params = _global_audit_filters(action_type, instances) - with get_connection() as conn: - rows = conn.execute( - f"{_AUDIT_SELECT} WHERE {' AND '.join(where_parts)} " - "ORDER BY a.created_at DESC, a.id LIMIT %s OFFSET %s", - (*params, limit, offset), - ).fetchall() - return [_audit_row_to_legacy(r) for r in rows] - - -def get_all_audit_log_count( - action_type: Optional[str] = None, - instances: Optional[list[str]] = None, -) -> int: - """Get total count of all audit logs (optionally tenant-scoped).""" - if instances is not None and not _normalize_instances(instances): - return 0 - where_parts, params = _global_audit_filters(action_type, instances) - with get_connection() as conn: - row = conn.execute( - "SELECT COUNT(*) AS cnt FROM audit_logs a " - "LEFT JOIN documents d ON d.id = a.document_id " - "LEFT JOIN instances ag ON ag.id = a.instance_id " - "LEFT JOIN catalogs dc ON dc.id = d.catalog_id " - "LEFT JOIN instances dg ON dg.id = dc.instance_id " - f"WHERE {' AND '.join(where_parts)}", - params, - ).fetchone() - return int(row["cnt"]) if row else 0 - - -# ============================================================================= -# Page Functions -# ============================================================================= - -_PAGE_SELECT = """ - SELECT p.id, p.document_id, p.page_number, p.original_markdown, p.edited_markdown, - p.translations, p.metadata, p.created_at, p.updated_at - FROM pages p -""" - - -def _primary_translation(translations: dict) -> tuple[Optional[str], dict]: - if not translations: - return None, {} - lang = max( - translations, - key=lambda k: (translations[k].get("translated_at") or "", k == "en"), - ) - return lang, translations[lang] - - -def _page_from_row(row: Optional[dict], workflow_id: str) -> Optional[dict]: - if row is None: - return None - meta = row["metadata"] or {} - translations = row["translations"] or {} - lang, tr = _primary_translation(translations) - page = { - "id": row["id"], - "workflow_id": workflow_id, - "page_number": row["page_number"], - "original_markdown": row["original_markdown"], - "edited_markdown": row["edited_markdown"], - "is_reviewed": bool(meta.get("is_reviewed")), - "reviewer_notes": meta.get("reviewer_notes"), - "detected_language": meta.get("detected_language"), - "translated_markdown": tr.get("translated_markdown"), - "edited_translation": tr.get("edited_translation"), - "translation_reviewed": bool(tr.get("is_reviewed")), - "translation_notes": tr.get("reviewer_notes"), - "translation_provider": tr.get("provider"), - "translation_model": tr.get("model"), - "translation_target_language": lang, - "translated_at": tr.get("translated_at"), - "translations": translations, - "metadata": meta, - } - return {k: _clean(v) for k, v in page.items()} - - -_TRANSLATION_FLAT_KEYS = ( - "translated_markdown", "edited_translation", "translation_notes", - "translation_provider", "translation_model", "translated_at", -) - - -def _translation_from_flat(page: dict) -> Optional[tuple[str, dict]]: - if not any(page.get(k) not in (None, "") for k in _TRANSLATION_FLAT_KEYS) and not page.get( - "translation_reviewed" - ): - return None - lang = page.get("translation_target_language") or "en" - entry = PageTranslation( - translated_markdown=page.get("translated_markdown"), - edited_translation=page.get("edited_translation"), - is_reviewed=bool(page.get("translation_reviewed")), - reviewer_notes=page.get("translation_notes"), - provider=page.get("translation_provider"), - model=page.get("translation_model"), - translated_at=page.get("translated_at"), - ).model_dump() - return lang, entry - - -def save_pages(workflow_id: str, pages: list[dict]): - """ - Save all pages for a document (bulk upsert). - - Each call replaces the whole row, so the caller passes every field. Page - ``translations`` are kept as stored unless the page carries ``translations`` - (replaces them) or flat ``translated_*`` fields (merged in under - ``translation_target_language``); a page whose OCR text changed loses its - translations because they no longer describe it. - """ - with get_connection() as conn: - doc_uuid = _require_document_uuid(conn, workflow_id) - existing = { - r["page_number"]: r - for r in conn.execute( - "SELECT page_number, original_markdown, translations FROM pages " - "WHERE document_id = %s::uuid", - (doc_uuid,), - ).fetchall() - } - for page in pages: - number = page.get("page_number") - current = existing.get(number) - meta = dict(page.get("metadata") or {}) - for flat_key in ("is_reviewed", "reviewer_notes", "detected_language"): - if flat_key in page: - meta[flat_key] = page[flat_key] - meta["is_reviewed"] = bool(meta.get("is_reviewed")) - page_meta = PageMetadata.model_validate( - {k: v for k, v in meta.items() if v is not None} - ).model_dump(exclude_none=True) - - translations = dict(current["translations"]) if current else {} - if current and current["original_markdown"] != page.get("original_markdown"): - translations = {} - if isinstance(page.get("translations"), dict): - translations = { - lang: PageTranslation.model_validate(entry).model_dump() - for lang, entry in page["translations"].items() - } - else: - flat = _translation_from_flat(page) - if flat: - translations[flat[0]] = flat[1] - - conn.execute( - """ - INSERT INTO pages (document_id, page_number, original_markdown, edited_markdown, - translations, metadata) - VALUES (%s::uuid, %s, %s, %s, %s, %s) - ON CONFLICT (document_id, page_number) DO UPDATE SET - original_markdown = EXCLUDED.original_markdown, - edited_markdown = EXCLUDED.edited_markdown, - translations = EXCLUDED.translations, - metadata = EXCLUDED.metadata - """, - ( - doc_uuid, number, page.get("original_markdown"), page.get("edited_markdown"), - Jsonb(translations), Jsonb(page_meta), - ), - ) - - -def persist_document_content(workflow_id: str, pages: list[dict], chunks: list[dict]): - """Backward-compatible helper to persist pages and chunks in one call.""" - save_pages(workflow_id, pages) - save_chunks(workflow_id, chunks) - - -def get_pages(workflow_id: str) -> list[dict]: - """Get all pages for a document.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return [] - root = _root_workflow_id(conn, doc_uuid) or workflow_id - rows = conn.execute( - _PAGE_SELECT + " WHERE p.document_id = %s::uuid ORDER BY p.page_number", (doc_uuid,) - ).fetchall() - return [_page_from_row(r, root) for r in rows] - - -def count_translated_pages(workflow_id: str) -> int: - """Count pages with translation text without loading full page bodies.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return 0 - row = conn.execute( - """ - SELECT COUNT(*) AS n FROM pages p - WHERE p.document_id = %s::uuid - AND EXISTS ( - SELECT 1 FROM jsonb_each(p.translations) t - WHERE nullif(btrim(t.value->>'translated_markdown'), '') IS NOT NULL - OR nullif(btrim(t.value->>'edited_translation'), '') IS NOT NULL - ) - """, - (doc_uuid,), - ).fetchone() - return int(row["n"] if row else 0) - - -def get_saved_page_numbers(workflow_id: str) -> list[int]: - """Get saved page numbers for a document without loading full page content.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return [] - rows = conn.execute( - "SELECT page_number FROM pages WHERE document_id = %s::uuid ORDER BY page_number", - (doc_uuid,), - ).fetchall() - return [int(row["page_number"]) for row in rows] - - -def get_page(workflow_id: str, page_num: int) -> Optional[dict]: - """Get a specific page.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return None - root = _root_workflow_id(conn, doc_uuid) or workflow_id - row = conn.execute( - _PAGE_SELECT + " WHERE p.document_id = %s::uuid AND p.page_number = %s", - (doc_uuid, page_num), - ).fetchone() - return _page_from_row(row, root) - - -def update_page( - workflow_id: str, - page_num: int, - edited_markdown: Optional[str] = None, - is_reviewed: Optional[bool] = None, - reviewer_notes: Optional[str] = None, - edited_translation: Optional[str] = None, - translation_reviewed: Optional[bool] = None, - translation_notes: Optional[str] = None, -) -> Optional[dict]: - """Update a page. Returns updated page or None if not found.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return None - existing = conn.execute( - "SELECT translations FROM pages WHERE document_id = %s::uuid AND page_number = %s FOR UPDATE", - (doc_uuid, page_num), - ).fetchone() - if not existing: - return None - - assignments: list[str] = [] - params: list = [] - if edited_markdown is not None: - assignments.append("edited_markdown = %s") - params.append(edited_markdown) - - meta_patch: dict[str, Any] = {} - if is_reviewed is not None: - meta_patch["is_reviewed"] = bool(is_reviewed) - if reviewer_notes is not None: - meta_patch["reviewer_notes"] = reviewer_notes - if meta_patch: - assignments.append("metadata = metadata || %s::jsonb") - params.append(Jsonb(meta_patch)) - - tr_patch: dict[str, Any] = {} - if edited_translation is not None: - tr_patch["edited_translation"] = edited_translation - if translation_reviewed is not None: - tr_patch["is_reviewed"] = bool(translation_reviewed) - if translation_notes is not None: - tr_patch["reviewer_notes"] = translation_notes - if tr_patch: - lang, _ = _primary_translation(existing["translations"] or {}) - lang = lang or "en" - assignments.append( - "translations = jsonb_set(translations, ARRAY[%s::text], " - "COALESCE(translations->%s::text, '{}'::jsonb) || %s::jsonb, TRUE)" - ) - params.extend([lang, lang, Jsonb(tr_patch)]) - - if assignments: - conn.execute( - f"UPDATE pages SET {', '.join(assignments)} " - "WHERE document_id = %s::uuid AND page_number = %s", - (*params, doc_uuid, page_num), - ) - return get_page(workflow_id, page_num) - - -def reset_page(workflow_id: str, page_num: int) -> Optional[dict]: - """Reset a page to its original OCR markdown.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is not None: - conn.execute( - """ - UPDATE pages SET - edited_markdown = NULL, - metadata = (metadata - 'reviewer_notes') || '{"is_reviewed": false}'::jsonb - WHERE document_id = %s::uuid AND page_number = %s - """, - (doc_uuid, page_num), - ) - return get_page(workflow_id, page_num) - - -# ============================================================================= -# Chunk Functions -# ============================================================================= - -_CHUNK_SELECT = """ - SELECT c.id, c.document_id, c.chunk_number, c.original_text, c.edited_text, c.token_count, - c.page_start, c.page_end, c.source_spans, c.section_title, c.content_type, - c.is_reference, c.is_reviewed, c.is_excluded, c.reviewer_notes, - c.created_at, c.updated_at, d.metadata->'chunking' AS chunking - FROM chunks c - JOIN documents d ON d.id = c.document_id -""" - - -def _chunk_from_row(row: Optional[dict], workflow_id: str) -> Optional[dict]: - if row is None: - return None - spans = row["source_spans"] or [] - page_numbers = sorted({int(s["page_number"]) for s in spans if "page_number" in s}) or list( - range(int(row["page_start"]), int(row["page_end"]) + 1) - ) - run = row.get("chunking") or {} - chunk = { - "id": row["id"], - "workflow_id": workflow_id, - "chunk_number": row["chunk_number"], - "original_text": row["original_text"], - "edited_text": row["edited_text"], - "token_count": row["token_count"], - "page_start": row["page_start"], - "page_end": row["page_end"], - "source_page_numbers_json": json.dumps(page_numbers), - "source_spans_json": json.dumps(spans), - "section_title": row["section_title"], - "content_type": row["content_type"], - "is_reference": bool(row["is_reference"]), - "chunking_provider": run.get("provider"), - "chunking_model": run.get("model"), - "chunking_config_json": json.dumps(run["config"]) if run.get("config") else None, - "chunking_run_id": f"run-{run['run_number']}" if run.get("run_number") else None, - "chunk_version": run.get("run_number") or 1, - "is_reviewed": bool(row["is_reviewed"]), - "is_excluded": bool(row["is_excluded"]), - "reviewer_notes": row["reviewer_notes"], - } - return {k: _clean(v) for k, v in chunk.items()} - - -def _load_json_value(raw: Any, default: Any) -> Any: - if raw is None or raw == "": - return default - if isinstance(raw, str): - return json.loads(raw) - return raw - - -def save_chunks(workflow_id: str, chunks: list[dict]): - """ - Save all chunks for a document. - - Re-chunking runs in one transaction: the document's chunks are deleted, the - new ones inserted, and ``metadata.chunking`` (with ``run_number`` - incremented) updated together. - """ - with get_connection() as conn: - doc_uuid = _require_document_uuid(conn, workflow_id) - previous = conn.execute( - "SELECT metadata->'chunking'->>'run_number' AS run_number FROM documents " - "WHERE id = %s::uuid FOR UPDATE", - (doc_uuid,), - ).fetchone() - run_number = int(previous["run_number"] or 0) + 1 - conn.execute("DELETE FROM chunks WHERE document_id = %s::uuid", (doc_uuid,)) - for chunk in chunks: - spans = _load_json_value( - chunk.get("source_spans") if chunk.get("source_spans") is not None - else chunk.get("source_spans_json"), - [], - ) - page_start = int(chunk.get("page_start") or 1) - page_end = max(int(chunk.get("page_end") or page_start), page_start) - conn.execute( - """ - INSERT INTO chunks (document_id, chunk_number, original_text, edited_text, - token_count, page_start, page_end, source_spans, - section_title, content_type, is_reference, is_reviewed, - is_excluded, reviewer_notes) - VALUES (%s::uuid, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) - """, - ( - doc_uuid, - chunk.get("chunk_number"), - chunk.get("original_text") or "", - chunk.get("edited_text"), - chunk.get("token_count") or 0, - page_start, - page_end, - Jsonb(spans), - chunk.get("section_title"), - chunk.get("content_type"), - bool(chunk.get("is_reference")), - bool(chunk.get("is_reviewed")), - bool(chunk.get("is_excluded")), - chunk.get("reviewer_notes"), - ), - ) - meta_patch: dict[str, Any] = {} - if chunks: - first = chunks[0] - meta_patch["chunking"] = ChunkingRun( - run_number=run_number, - provider=first.get("chunking_provider"), - model=first.get("chunking_model"), - config=_load_json_value(first.get("chunking_config_json"), {}), - created_at=_iso_now(), - ).model_dump() - if meta_patch: - conn.execute( - "UPDATE documents SET metadata = metadata || %s::jsonb WHERE id = %s::uuid", - (Jsonb(meta_patch), doc_uuid), - ) - else: - conn.execute("UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,)) - - -def get_chunks(workflow_id: str, include_excluded: bool = False) -> list[dict]: - """Get all chunks for a document.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return [] - root = _root_workflow_id(conn, doc_uuid) or workflow_id - excluded = "" if include_excluded else "AND NOT c.is_excluded" - rows = conn.execute( - f"{_CHUNK_SELECT} WHERE c.document_id = %s::uuid {excluded} ORDER BY c.chunk_number", - (doc_uuid,), - ).fetchall() - return [_chunk_from_row(r, root) for r in rows] - - -def search_chunks( - *, - query: str = "", - limit: int = 100, - offset: int = 0, - include_excluded: bool = False, - stage: Optional[str] = None, - instances: Optional[list[str]] = None, -) -> tuple[list[dict], int]: - """Search chunks across documents for maintainer workflows. - - instances: If provided, only return chunks whose owning document's instance - is in this list (tenant scoping). ``None`` means no instance restriction. - """ - where_clauses = ["TRUE"] - params: list = [] - - if not include_excluded: - where_clauses.append("NOT c.is_excluded") - if stage: - where_clauses.append("cw.current_stage = %s") - params.append(stage) - if instances is not None: - normalized = _normalize_instances(instances) - if not normalized: - where_clauses.append("FALSE") - else: - where_clauses.append("lower(g.name) = ANY(%s)") - params.append(normalized) - if query and query.strip(): - where_clauses.append("lower(coalesce(c.edited_text, c.original_text)) LIKE %s") - params.append(f"%{query.strip().lower()}%") - - base_from_sql = f""" - FROM chunks c - JOIN documents d ON d.id = c.document_id - JOIN catalogs cat ON cat.id = d.catalog_id - JOIN instances g ON g.id = cat.instance_id - LEFT JOIN document_current_workflow cw ON cw.document_id = d.id - WHERE {' AND '.join(where_clauses)} - """ - with get_connection() as conn: - total_row = conn.execute(f"SELECT COUNT(*) AS cnt {base_from_sql}", params).fetchone() - total = int(total_row["cnt"] if total_row and total_row["cnt"] is not None else 0) - rows = conn.execute( - f""" - SELECT - c.id, c.chunk_number, c.original_text, c.edited_text, c.token_count, - c.page_start, c.page_end, c.source_spans, c.section_title, c.content_type, - c.is_reference, c.is_reviewed, c.is_excluded, c.reviewer_notes, - d.metadata->'chunking' AS chunking, - d.source->>'fingerprint' AS document_id, - d.source->>'original_filename' AS filename, - d.name AS name, - cw.current_stage AS stage, - d.updated_at AS document_updated_at, - (SELECT r.temporal_workflow_id FROM workflows r WHERE r.document_id = d.id - ORDER BY (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC - LIMIT 1) AS root_workflow_id - {base_from_sql} - ORDER BY d.updated_at DESC, d.id, c.chunk_number - LIMIT %s OFFSET %s - """, - [*params, int(limit), int(offset)], - ).fetchall() - - chunks: list[dict] = [] - for row in rows: - row = dict(row) - chunk = _chunk_from_row(row, row["root_workflow_id"]) - for extra in ("document_id", "filename", "name", "stage", "document_updated_at"): - chunk[extra] = _clean(row[extra]) - chunks.append(chunk) - return chunks, total - - -def get_chunk(workflow_id: str, chunk_num: int) -> Optional[dict]: - """Get a specific chunk.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return None - root = _root_workflow_id(conn, doc_uuid) or workflow_id - row = conn.execute( - _CHUNK_SELECT + " WHERE c.document_id = %s::uuid AND c.chunk_number = %s", - (doc_uuid, chunk_num), - ).fetchone() - return _chunk_from_row(row, root) - - -def update_chunk( - workflow_id: str, - chunk_num: int, - edited_text: Optional[str] = None, - is_reviewed: Optional[bool] = None, - is_excluded: Optional[bool] = None, - reviewer_notes: Optional[str] = None -) -> Optional[dict]: - """Update a chunk. Returns updated chunk or None if not found.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is None: - return None - existing = conn.execute( - "SELECT id FROM chunks WHERE document_id = %s::uuid AND chunk_number = %s", - (doc_uuid, chunk_num), - ).fetchone() - if not existing: - return None - - updates = [] - values: list = [] - if edited_text is not None: - updates.append("edited_text = %s") - values.append(edited_text) - if is_reviewed is not None: - updates.append("is_reviewed = %s") - values.append(bool(is_reviewed)) - if is_excluded is not None: - updates.append("is_excluded = %s") - values.append(bool(is_excluded)) - if reviewer_notes is not None: - updates.append("reviewer_notes = %s") - values.append(reviewer_notes) - if updates: - conn.execute( - f"UPDATE chunks SET {', '.join(updates)} WHERE document_id = %s::uuid AND chunk_number = %s", - (*values, doc_uuid, chunk_num), - ) - return get_chunk(workflow_id, chunk_num) - - -def reset_chunk(workflow_id: str, chunk_num: int) -> Optional[dict]: - """Reset a chunk to its original text.""" - with get_connection() as conn: - doc_uuid = _resolve_document_uuid(conn, workflow_id) - if doc_uuid is not None: - conn.execute( - """ - UPDATE chunks SET edited_text = NULL, is_reviewed = FALSE, is_excluded = FALSE, - reviewer_notes = NULL - WHERE document_id = %s::uuid AND chunk_number = %s - """, - (doc_uuid, chunk_num), - ) - return get_chunk(workflow_id, chunk_num) - - -# ============================================================================= -# Settings Functions -# ============================================================================= - -_SETTING_GROUPS: dict[str, tuple[str, type]] = { - "searchMethod": ("retrieval", RetrievalSettings), - "alpha": ("retrieval", RetrievalSettings), - "rankingMethod": ("retrieval", RetrievalSettings), - "hybridRrfK": ("retrieval", RetrievalSettings), - "indexName": ("candidates", CandidateSettings), - "efSearch": ("candidates", CandidateSettings), - "candidateCap": ("candidates", CandidateSettings), - "candidateMultiplier": ("candidates", CandidateSettings), - "limit": ("results", ResultSettings), - "maxChunksPerDoc": ("results", ResultSettings), - "excludeReference": ("results", ResultSettings), - "showHighlights": ("results", ResultSettings), - "rerankMode": ("results", ResultSettings), - "useE5Prefix": ("query", QuerySettings), - "queryExpansionProfile": ("query", QuerySettings), -} - -_GROUP_MODELS: dict[str, type] = {group: model for group, model in _SETTING_GROUPS.values()} - -_SETTINGS_ROW_SQL = ( - "SELECT retrieval, candidates, results, query FROM search_settings WHERE singleton" -) - - -def _settings_row(conn, lock: bool = False) -> dict: - sql = _SETTINGS_ROW_SQL + (" FOR UPDATE" if lock else "") - row = conn.execute(sql).fetchone() - if row is None: - conn.execute("INSERT INTO search_settings DEFAULT VALUES ON CONFLICT (singleton) DO NOTHING") - row = conn.execute(sql).fetchone() - return row - - -def _flatten_settings(row: dict) -> dict: - flat: dict[str, Any] = {} - for group, model in _GROUP_MODELS.items(): - flat.update(model.model_validate(row[group] or {}).model_dump()) - return flat - - -def get_search_settings() -> dict: - """Get all search settings as a flat dict (stored values merged over defaults).""" - with get_connection() as conn: - return _flatten_settings(_settings_row(conn)) - - -def _setting_text(value: Any) -> str: - if isinstance(value, bool): - return "true" if value else "false" - return str(value) - - -def update_search_settings(settings: dict) -> dict: - """ - Update multiple search settings at once. - - Args: - settings: Dict with keys like searchMethod, limit, alpha, etc. - - Returns: - Updated search settings - - Raises: - ValueError: a value fails the setting's validation rules. - """ - unknown = sorted(set(settings) - set(_SETTING_GROUPS)) - if unknown: - raise ValueError(f"Unknown search settings: {', '.join(unknown)}") - changes: list[tuple[str, str, str]] = [] - with get_connection() as conn: - row = _settings_row(conn, lock=True) - before = _flatten_settings(row) - for group, model in _GROUP_MODELS.items(): - patch = {k: v for k, v in settings.items() if _SETTING_GROUPS[k][0] == group} - if not patch: - continue - try: - validated = model.model_validate({**model.model_validate(row[group] or {}).model_dump(), **patch}) - except Exception as exc: - raise ValueError(str(exc)) from exc - stored = {**(row[group] or {}), **{k: getattr(validated, k) for k in patch}} - conn.execute( - f"UPDATE search_settings SET {group} = {group} || %s::jsonb WHERE singleton", - (Jsonb({k: getattr(validated, k) for k in patch}),), - ) - for key in patch: - if before[key] != stored[key]: - changes.append((f"{group}.{key}", _setting_text(before[key]), _setting_text(stored[key]))) - for field_name, old_value, new_value in changes: - conn.execute( - """ - INSERT INTO audit_logs (action_type, entity_type, field_name, old_value, new_value) - VALUES ('settings_change', 'setting', %s, %s, %s) - """, - (field_name, old_value, new_value), - ) - return _flatten_settings( - conn.execute(_SETTINGS_ROW_SQL).fetchone() - ) - - -def get_settings_audit_logs(limit: int = 50, offset: int = 0) -> list[dict]: - """Get audit logs for settings changes.""" - with get_connection() as conn: - rows = conn.execute( - f"{_AUDIT_SELECT} WHERE a.entity_type = 'setting' " - "ORDER BY a.created_at DESC, a.id LIMIT %s OFFSET %s", - (limit, offset), - ).fetchall() - return [_audit_row_to_legacy(r) for r in rows] - - -def get_settings_audit_count() -> int: - """Get count of settings audit logs.""" - with get_connection() as conn: - row = conn.execute( - "SELECT COUNT(*) AS cnt FROM audit_logs WHERE entity_type = 'setting'" - ).fetchone() - return int(row["cnt"]) if row else 0 diff --git a/pipeline/db/__init__.py b/pipeline/db/__init__.py new file mode 100644 index 0000000..de34bab --- /dev/null +++ b/pipeline/db/__init__.py @@ -0,0 +1,136 @@ +"""PostgreSQL query API for the application and migration tools.""" + +from .artifacts import ( + add_document_artifact as add_document_artifact, + get_document_artifact as get_document_artifact, + list_document_artifacts as list_document_artifacts, +) +from .audit import ( + get_all_audit_log_count as get_all_audit_log_count, + get_all_audit_logs as get_all_audit_logs, + get_audit_log_count as get_audit_log_count, + get_audit_logs as get_audit_logs, + log_audit as log_audit, + meaningful_roles as meaningful_roles, +) +from .catalog import ( + CATALOG_VERSION_ACTION as CATALOG_VERSION_ACTION, + bump_catalog_version as bump_catalog_version, + ensure_instance_catalogs as ensure_instance_catalogs, + get_catalog_entry as get_catalog_entry, + get_catalog_meta as get_catalog_meta, + get_or_create_catalog as get_or_create_catalog, + list_catalog_entries as list_catalog_entries, + list_document_codes as list_document_codes, + list_scheme_catalog_entries_for_workflow as list_scheme_catalog_entries_for_workflow, + list_used_scheme_codes as list_used_scheme_codes, + sync_catalogs_for_all_instances as sync_catalogs_for_all_instances, +) +from .chunks import ( + get_chunk as get_chunk, + get_chunks as get_chunks, + reset_chunk as reset_chunk, + save_chunks as save_chunks, + search_chunks as search_chunks, + update_chunk as update_chunk, +) +from .connection import ( + DEFAULT_DATABASE_URL as DEFAULT_DATABASE_URL, + check_health as check_health, + close_pool as close_pool, + database_url as database_url, + get_connection as get_connection, + get_pool as get_pool, + init_db as init_db, +) +from .document_reads import ( + find_document_by_code as find_document_by_code, + get_document as get_document, + get_document_count as get_document_count, + get_document_counts_by_instance as get_document_counts_by_instance, + get_document_source_url as get_document_source_url, + get_document_summary_counts as get_document_summary_counts, + list_documents as list_documents, + owned_source_fingerprint as owned_source_fingerprint, +) +from .document_state import ( + reconcile_materialized_state as reconcile_materialized_state, + update_document_stage as update_document_stage, +) +from .document_writes import ( + DocumentReplacementConflict as DocumentReplacementConflict, + DuplicateDocumentCode as DuplicateDocumentCode, + clear_prod_ready_requested as clear_prod_ready_requested, + delete_document as delete_document, + mark_document_reindex_required as mark_document_reindex_required, + mark_prod_ready_requested as mark_prod_ready_requested, + purge_document as purge_document, + reset_document_for_reupload as reset_document_for_reupload, + set_document_disabled as set_document_disabled, + set_document_validity as set_document_validity, + update_document_fields as update_document_fields, + upsert_document as upsert_document, +) +from .index_state import ( + DEV_INDEX_KEY as DEV_INDEX_KEY, + PROD_INDEX_KEY as PROD_INDEX_KEY, + clear_superseded_doc_ids as clear_superseded_doc_ids, + doc_id_used_by_other_document as doc_id_used_by_other_document, + find_document_by_doc_identifier as find_document_by_doc_identifier, + get_document_index_status as get_document_index_status, + inactive_network_revision_ids as inactive_network_revision_ids, + list_document_index_status as list_document_index_status, + list_index_summaries as list_index_summaries, + resolve_chunk_provenance as resolve_chunk_provenance, + upsert_document_index_status as upsert_document_index_status, + visible_document_hits as visible_document_hits, + visible_release_hits as visible_release_hits, +) +from .jobs import ( + create_document_job as create_document_job, + get_document_job as get_document_job, + get_latest_document_job as get_latest_document_job, + list_document_jobs as list_document_jobs, + list_operations_queue as list_operations_queue, + list_runs as list_runs, + mark_upload_started as mark_upload_started, + update_document_job as update_document_job, +) +from .pages import ( + PageTranslationMissing as PageTranslationMissing, + count_translated_pages as count_translated_pages, + get_page as get_page, + get_pages as get_pages, + get_saved_page_numbers as get_saved_page_numbers, + page_translation_key as page_translation_key, + persist_document_content as persist_document_content, + reset_page as reset_page, + save_pages as save_pages, + update_page as update_page, +) +from .providers import ( + get_or_create_instance as get_or_create_instance, + get_user_by_keycloak_id as get_user_by_keycloak_id, + list_provider_instances as list_provider_instances, + upsert_instance as upsert_instance, + upsert_user as upsert_user, +) +from .revisions import ( + activate_network_release as activate_network_release, + get_latest_completed_index_revision as get_latest_completed_index_revision, + get_revision_snapshot as get_revision_snapshot, + list_document_revisions as list_document_revisions, + load_revision_material as load_revision_material, + save_draft_revision_snapshot as save_draft_revision_snapshot, + save_legacy_index_revisions as save_legacy_index_revisions, + save_revision_snapshot as save_revision_snapshot, + select_network_revision as select_network_revision, + stage_saved_network_revision as stage_saved_network_revision, + withdraw_network_release as withdraw_network_release, +) +from .settings import ( + get_search_settings as get_search_settings, + get_settings_audit_count as get_settings_audit_count, + get_settings_audit_logs as get_settings_audit_logs, + update_search_settings as update_search_settings, +) diff --git a/pipeline/db/artifacts.py b/pipeline/db/artifacts.py new file mode 100644 index 0000000..fa78a18 --- /dev/null +++ b/pipeline/db/artifacts.py @@ -0,0 +1,135 @@ +"""Store references to workflow artifacts in object storage.""" + +import json +from contextlib import nullcontext +from typing import Optional + +from psycopg.types.json import Jsonb + +from . import common, connection, document_reads + +# ============================================================================= +# Artifacts +# ============================================================================= + + +def _artifact_from_row(row: Optional[dict]) -> Optional[dict]: + if row is None: + return None + metadata = row["metadata"] or {} + artifact = { + "id": row["id"], + "workflow_id": row["doc_workflow_id"], + "job_id": row["workflow_pk"], + "artifact_type": row["artifact_type"], + "stage": row["stage"], + "storage_uri": row["storage_uri"], + "mime_type": row["mime_type"], + "filename": row["filename"], + "size_bytes": row["size_bytes"], + "metadata_json": json.dumps(metadata) if metadata else None, + "created_at": row["created_at"], + } + return {k: common._clean(v) for k, v in artifact.items()} + + +_ARTIFACT_SELECT = """ + SELECT a.id, a.workflow_id AS workflow_pk, a.artifact_type, a.stage, a.storage_uri, + a.mime_type, a.filename, a.size_bytes, a.metadata, a.created_at, + (SELECT r.temporal_workflow_id FROM workflows r WHERE r.document_id = w.document_id + ORDER BY (r.workflow_type = 'legacy_index_import'), (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC + LIMIT 1) AS doc_workflow_id + FROM workflow_artifacts a + JOIN workflows w ON w.id = a.workflow_id +""" + + +def add_document_artifact( + workflow_id: str, + artifact_type: str, + storage_uri: str, + stage: Optional[str] = None, + job_id: Optional[str] = None, + mime_type: Optional[str] = None, + filename: Optional[str] = None, + size_bytes: Optional[int] = None, + metadata: Optional[dict] = None, + *, + artifact_id: str | None = None, + _conn=None, +) -> str: + """Record a file produced by a workflow; returns the artifact UUID. + + Belongs to ``job_id`` when given, else to the document's current workflow. + """ + with connection.get_connection() if _conn is None else nullcontext(_conn) as conn: + doc_uuid = document_reads._require_document_uuid(conn, workflow_id) + wf_uuid = None + if job_id and common._is_uuid(str(job_id)): + row = conn.execute( + "SELECT id FROM workflows WHERE id = %s::uuid AND document_id = %s::uuid", + (str(job_id), doc_uuid), + ).fetchone() + wf_uuid = str(row["id"]) if row else None + if wf_uuid is None: + row = conn.execute( + "SELECT id FROM document_current_workflow WHERE document_id = %s::uuid", (doc_uuid,) + ).fetchone() + if row is None: + row = conn.execute( + """ + INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id) + VALUES (%s::uuid, 'pipeline', %s) RETURNING id + """, + (doc_uuid, workflow_id), + ).fetchone() + wf_uuid = str(row["id"]) + created = conn.execute( + """ + INSERT INTO workflow_artifacts (id, workflow_id, artifact_type, stage, storage_uri, + mime_type, filename, size_bytes, metadata) + VALUES (COALESCE(%s::uuid, gen_random_uuid()), %s::uuid, %s, %s, %s, %s, %s, %s, %s) + ON CONFLICT (id) DO NOTHING + RETURNING id + """, + ( + artifact_id, + wf_uuid, + artifact_type, + stage, + storage_uri, + mime_type, + filename, + size_bytes, + Jsonb(metadata or {}), + ), + ).fetchone() + return str(created["id"]) if created else artifact_id + + +def list_document_artifacts(workflow_id: str) -> list[dict]: + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return [] + rows = conn.execute( + _ARTIFACT_SELECT + + " WHERE w.document_id = %s::uuid ORDER BY a.created_at DESC, a.id DESC", + (doc_uuid,), + ).fetchall() + return [_artifact_from_row(r) for r in rows] + + +def get_document_artifact(workflow_id: str, artifact_id: str) -> Optional[dict]: + if not common._is_uuid(str(artifact_id)): + return None + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return None + return _artifact_from_row( + conn.execute( + _ARTIFACT_SELECT + " WHERE w.document_id = %s::uuid AND a.id = %s::uuid", + (doc_uuid, str(artifact_id)), + ).fetchone() + ) diff --git a/pipeline/db/audit.py b/pipeline/db/audit.py new file mode 100644 index 0000000..030f579 --- /dev/null +++ b/pipeline/db/audit.py @@ -0,0 +1,343 @@ +"""Record and query the audit trail.""" + +import json +from contextlib import nullcontext +from typing import Optional + +from psycopg.types.json import Jsonb + +from . import catalog, common, connection, document_reads, jobs, providers + +# ============================================================================= +# Audit Log Functions +# ============================================================================= + +_AUDIT_ENTITY_TYPES = {"document", "page", "chunk", "setting", "catalog"} + + +def log_audit( + workflow_id: str, + document_id: str, + action_type: str, + entity_type: Optional[str] = None, + entity_id: Optional[int] = None, + field_name: Optional[str] = None, + old_value: Optional[str] = None, + new_value: Optional[str] = None, + metadata: Optional[dict] = None, + actor_user_id: Optional[str] = None, + actor_username: Optional[str] = None, + actor_email: Optional[str] = None, + actor_roles: Optional[str] = None, + *, + _conn=None, +) -> str: + """ + Log an audit entry. + + Args: + workflow_id: The Temporal workflow ID (or document identifier). Ids that + start with ``__`` mark system-level entries and are not stored. + document_id: Unused; kept for call-site compatibility. The document is + resolved from ``workflow_id``. + action_type: Type of action (stage_change, page_edit, chunk_edit, approval, reset) + entity_type: Type of entity (document, page, chunk, setting, catalog) + entity_id: Page number or chunk number the action applied to + field_name: Name of the field that changed + old_value: Previous value (as string or JSON) + new_value: New value (as string or JSON) + metadata: Additional context as dict + actor_*: Identity of the human/system that performed the action + + Returns: + The UUID of the created audit log entry + """ + entity = entity_type if entity_type in _AUDIT_ENTITY_TYPES else None + system_entry = (workflow_id or "").startswith("__") + roles = common._csv_roles(actor_roles) + with connection.get_connection() if _conn is None else nullcontext(_conn) as conn: + doc_uuid = ( + None if system_entry else document_reads._resolve_document_uuid(conn, workflow_id) + ) + instance_uuid = None + stored_workflow_id = None if system_entry else workflow_id + if doc_uuid: + doc_row = conn.execute( + "SELECT provider_id FROM documents WHERE id = %s::uuid", + (doc_uuid,), + ).fetchone() + instance_uuid = str(doc_row["provider_id"]) + if common._is_uuid(workflow_id): + stored_workflow_id = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + if stored_workflow_id is None and entity not in ("setting", "catalog"): + stored_workflow_id = workflow_id + actor_uuid = ( + providers.upsert_user(actor_user_id, actor_username, actor_email, roles, conn=conn) + if actor_user_id + else None + ) + row = conn.execute( + """ + INSERT INTO audit_logs (document_id, workflow_id, instance_id, action_type, entity_type, + entity_number, field_name, old_value, new_value, metadata, + actor_user_id, actor) + VALUES (%s::uuid, %s, %s::uuid, %s, %s, %s, %s, %s, %s, %s, %s::uuid, %s) + RETURNING id + """, + ( + doc_uuid, + stored_workflow_id, + instance_uuid, + action_type, + entity, + entity_id, + field_name, + old_value, + new_value, + Jsonb(metadata or {}), + actor_uuid, + Jsonb( + { + "username": actor_username or None, + "email": actor_email or None, + "roles": roles, + } + ) + if (actor_user_id or actor_username or actor_email or roles) + else Jsonb({}), + ), + ).fetchone() + return str(row["id"]) + + +# Keycloak attaches these to every account regardless of what the user can +# actually do here. Rendering them made the actor column a wall of noise +# ("...manage-account,manage-account-links,offline_access,uma_authorization...") +# that buried the one role a reviewer cares about. +_NOISE_ROLES = { + "manage-account", + "manage-account-links", + "offline_access", + "uma_authorization", + "view-profile", + "account", +} + + +# Most-privileged first, so a multi-role account is labelled by what it can do. +_ROLE_PRECEDENCE = ( + "superadmin", + "super_admin", + "master_admin", + "admin", + "content_curator", + "reviewer", + "contributor", + "viewer", +) + + +def meaningful_roles(raw: Optional[str]) -> list[str]: + """Application roles from a stored role list, Keycloak built-ins removed.""" + parts = [p.strip() for p in (raw or "").split(",") if p.strip()] + kept = [ + p + for p in parts + if p.lower() not in _NOISE_ROLES and not p.lower().startswith("default-roles") + ] + ranked = [r for r in _ROLE_PRECEDENCE if r in {k.lower() for k in kept}] + if ranked: + # Preserve original casing of the first matching role. + first = next(k for k in kept if k.lower() == ranked[0]) + return [first] + return kept[:1] + + +def _normalize_audit_row(row) -> dict: + entry = dict(row) + email = (entry.get("actor_email") or "").strip() + username = (entry.get("actor_username") or "").strip() + roles = meaningful_roles(entry.get("actor_roles")) + role_label = roles[0] if roles else "" + + who = email or username + if who and role_label: + entry["actor"] = f"{who} ({role_label})" + elif who: + entry["actor"] = who + else: + entry["actor"] = "system" + + # Split out so the UI can render identity and role separately rather than + # parsing the combined string back apart. + entry["actor_display"] = who or "System" + entry["actor_role"] = role_label + entry["is_system"] = not who + return entry + + +_AUDIT_SELECT = """ + SELECT a.id, a.workflow_id, a.action_type, a.entity_type, a.entity_number, a.field_name, + a.old_value, a.new_value, a.metadata, a.created_at, a.actor, ag.name AS audit_instance, + au.keycloak_user_id AS actor_keycloak_id, + d.source->>'fingerprint' AS doc_fingerprint, + d.source->>'original_filename' AS filename, + d.name AS document_name, + dg.name AS doc_instance, + uu.username AS uploaded_by_username, uu.email AS uploaded_by_email + FROM audit_logs a + LEFT JOIN documents d ON d.id = a.document_id + LEFT JOIN instances ag ON ag.id = a.instance_id + LEFT JOIN instances dg ON dg.id = d.provider_id + LEFT JOIN users au ON au.id = a.actor_user_id + LEFT JOIN users uu ON uu.id = NULLIF(d.metadata->>'uploaded_by_user_id', '')::uuid +""" + + +def _audit_row_to_legacy(row: dict) -> dict: + actor = row["actor"] or {} + metadata = row["metadata"] or {} + entity_type = row["entity_type"] + if entity_type == "setting": + workflow_id, document_id = "__system__", "__settings__" + elif entity_type == "catalog" and not row["workflow_id"]: + workflow_id, document_id = "__catalog__", "__catalog__" + else: + workflow_id = row["workflow_id"] + document_id = row["doc_fingerprint"] or workflow_id + roles = actor.get("roles") or [] + entry = { + "id": row["id"], + "workflow_id": workflow_id, + "document_id": document_id, + "action_type": row["action_type"], + "entity_type": entity_type, + "entity_id": row["entity_number"], + "field_name": row["field_name"], + "old_value": row["old_value"], + "new_value": row["new_value"], + "metadata": json.dumps(metadata) if metadata else None, + "timestamp": row["created_at"], + "actor_user_id": row["actor_keycloak_id"], + "actor_username": actor.get("username"), + "actor_email": actor.get("email"), + "actor_roles": ",".join(roles) if roles else None, + "filename": row["filename"], + "name": row["document_name"], + "instance": row["doc_instance"] or row["audit_instance"], + "uploaded_by_username": row["uploaded_by_username"], + "uploaded_by_email": row["uploaded_by_email"], + } + return _normalize_audit_row({k: common._clean(v) for k, v in entry.items()}) + + +def _audit_scope(workflow_id: str, conn) -> tuple[str, list]: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid: + root = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + return "(a.document_id = %s::uuid OR a.workflow_id = %s)", [doc_uuid, root] + return "a.workflow_id = %s", [workflow_id] + + +def get_audit_logs( + workflow_id: str, action_type: Optional[str] = None, limit: int = 100, offset: int = 0 +) -> list[dict]: + """ + Get audit logs for a document. + + Args: + workflow_id: The Temporal workflow ID (or document UUID) + action_type: Optional filter by action type + limit: Maximum number of entries to return + offset: Offset for pagination + + Returns: + List of audit log entries as dicts + """ + with connection.get_connection() as conn: + where, params = _audit_scope(workflow_id, conn) + if action_type: + where += " AND a.action_type = %s" + params.append(action_type) + rows = conn.execute( + f"{_AUDIT_SELECT} WHERE {where} ORDER BY a.created_at DESC, a.id LIMIT %s OFFSET %s", + (*params, limit, offset), + ).fetchall() + return [_audit_row_to_legacy(r) for r in rows] + + +def get_audit_log_count(workflow_id: str, action_type: Optional[str] = None) -> int: + """Get count of audit logs for a document.""" + with connection.get_connection() as conn: + where, params = _audit_scope(workflow_id, conn) + if action_type: + where += " AND a.action_type = %s" + params.append(action_type) + row = conn.execute( + f"SELECT COUNT(*) AS cnt FROM audit_logs a WHERE {where}", params + ).fetchone() + return int(row["cnt"]) if row else 0 + + +def _global_audit_filters( + action_type: Optional[str], instances: Optional[list[str]] +) -> tuple[list[str], list]: + where_parts: list[str] = ["a.action_type <> %s"] + params: list = [catalog.CATALOG_VERSION_ACTION] + if action_type: + where_parts.append("a.action_type = %s") + params.append(action_type) + if instances is not None: + where_parts.append("lower(COALESCE(ag.name, dg.name, %s)) = ANY(%s)") + params.extend([common._default_instance(), common._normalize_instances(instances)]) + return where_parts, params + + +def get_all_audit_logs( + action_type: Optional[str] = None, + limit: int = 100, + offset: int = 0, + instances: Optional[list[str]] = None, +) -> list[dict]: + """ + Get all audit logs across all documents. + + Args: + action_type: Optional filter by action type + limit: Maximum number of entries to return + offset: Offset for pagination + instances: Tenant scope (None = all, [] = none) + + Returns: + List of audit log entries as dicts, with document filename included + """ + if instances is not None and not common._normalize_instances(instances): + return [] + where_parts, params = _global_audit_filters(action_type, instances) + with connection.get_connection() as conn: + rows = conn.execute( + f"{_AUDIT_SELECT} WHERE {' AND '.join(where_parts)} " + "ORDER BY a.created_at DESC, a.id LIMIT %s OFFSET %s", + (*params, limit, offset), + ).fetchall() + return [_audit_row_to_legacy(r) for r in rows] + + +def get_all_audit_log_count( + action_type: Optional[str] = None, + instances: Optional[list[str]] = None, +) -> int: + """Get total count of all audit logs (optionally tenant-scoped).""" + if instances is not None and not common._normalize_instances(instances): + return 0 + where_parts, params = _global_audit_filters(action_type, instances) + with connection.get_connection() as conn: + row = conn.execute( + "SELECT COUNT(*) AS cnt FROM audit_logs a " + "LEFT JOIN documents d ON d.id = a.document_id " + "LEFT JOIN instances ag ON ag.id = a.instance_id " + "LEFT JOIN instances dg ON dg.id = d.provider_id " + f"WHERE {' AND '.join(where_parts)}", + params, + ).fetchone() + return int(row["cnt"]) if row else 0 diff --git a/pipeline/db/catalog.py b/pipeline/db/catalog.py new file mode 100644 index 0000000..30bdb08 --- /dev/null +++ b/pipeline/db/catalog.py @@ -0,0 +1,197 @@ +"""Maintain the catalog projection and its change version.""" + +from typing import Optional + +from psycopg.types.json import Jsonb + +from pipeline.network.network_constants import KIND_TO_CATALOG_TYPE, catalog_type_for_kind + +from . import common, connection, document_reads + + +def _catalog_seed_rows() -> tuple[list[str], list[str]]: + """Parallel ``(types, kinds)`` lists for every supported catalog type.""" + kinds = list(KIND_TO_CATALOG_TYPE) + return [KIND_TO_CATALOG_TYPE[k] for k in kinds], kinds + + +def ensure_instance_catalogs(instance_id: str, conn=None) -> None: + """Create the instance's missing catalogs, one per supported type; existing rows are untouched.""" + if conn is None: + with connection.get_connection() as own: + return ensure_instance_catalogs(instance_id, conn=own) + types, kinds = _catalog_seed_rows() + conn.execute( + """ + INSERT INTO catalogs (instance_id, type, name) + SELECT g.id, t.type, upper(g.name) || ' ' || t.kind || ' catalog' + FROM instances g, unnest(%s::text[], %s::text[]) AS t(type, kind) + WHERE g.id = %s::uuid + ON CONFLICT (instance_id, type) DO NOTHING + """, + (types, kinds, instance_id), + ) + + +def sync_catalogs_for_all_instances(conn=None) -> int: + """Add every missing ``(instance, type)`` catalog; returns how many rows were created. + + Idempotent. Run at startup so a type added to the kind map reaches instances + that already exist. + """ + if conn is None: + with connection.get_connection() as own: + return sync_catalogs_for_all_instances(conn=own) + types, kinds = _catalog_seed_rows() + cur = conn.execute( + """ + INSERT INTO catalogs (instance_id, type, name) + SELECT g.id, t.type, upper(g.name) || ' ' || t.kind || ' catalog' + FROM instances g, unnest(%s::text[], %s::text[]) AS t(type, kind) + ON CONFLICT (instance_id, type) DO NOTHING + """, + (types, kinds), + ) + return cur.rowcount + + +def get_or_create_catalog(instance_id: str, kind: str, conn=None) -> dict: + """The ``(instance, kind)`` catalog row; created when missing (normally seeded with the instance).""" + if conn is None: + with connection.get_connection() as own: + return get_or_create_catalog(instance_id, kind, conn=own) + catalog_type = catalog_type_for_kind(kind) + row = conn.execute( + """ + INSERT INTO catalogs (instance_id, type, name) + SELECT g.id, %s, upper(g.name) || ' ' || %s || ' catalog' FROM instances g WHERE g.id = %s::uuid + ON CONFLICT (instance_id, type) DO UPDATE SET type = EXCLUDED.type + RETURNING id, instance_id, type, name, metadata + """, + (catalog_type, kind, instance_id), + ).fetchone() + return common._clean_row(row) + + +# ============================================================================= +# Catalog +# ============================================================================= + +CATALOG_VERSION_ACTION = "catalog_version_bump" + + +def get_catalog_meta() -> dict: + """Catalog version: the number of version bumps recorded in ``audit_logs``.""" + with connection.get_connection() as conn: + row = conn.execute( + """ + SELECT count(*) AS version, max(created_at) AS updated_at, + (array_agg(metadata->>'reason' ORDER BY created_at DESC))[1] AS notes + FROM audit_logs + WHERE entity_type = 'catalog' AND action_type = %s + """, + (CATALOG_VERSION_ACTION,), + ).fetchone() + return { + "id": 1, + "version": int(row["version"] or 0), + "updated_at": common._clean(row["updated_at"]), + "notes": row["notes"], + } + + +def bump_catalog_version(reason: str = "") -> int: + """Monotonic catalog version bump. Returns new version.""" + with connection.get_connection() as conn: + conn.execute( + """ + INSERT INTO audit_logs (action_type, entity_type, metadata) + VALUES (%s, 'catalog', %s) + """, + (CATALOG_VERSION_ACTION, Jsonb({"reason": (reason or "")[:500]})), + ) + row = conn.execute( + "SELECT count(*) AS version FROM audit_logs " + "WHERE entity_type = 'catalog' AND action_type = %s", + (CATALOG_VERSION_ACTION,), + ).fetchone() + return int(row["version"]) + + +def list_document_codes(kind: str) -> set[str]: + """Every document code in catalogs of one knowledge kind.""" + with connection.get_connection() as conn: + rows = conn.execute( + "SELECT code FROM documents WHERE kind = %s", + (kind,), + ).fetchall() + return {r["code"] for r in rows} + + +def _catalog_row(row: dict) -> dict: + cleaned = common._clean_row(row) + cleaned.pop("instance_name", None) + cleaned["aliases"] = list(row["aliases"] or []) + cleaned["instances"] = [(row["instance_name"] or "").lower()] + cleaned["workflow_ids"] = [row["workflow_id"]] if row["workflow_id"] else [] + cleaned["source"] = "pipeline" + return cleaned + + +def list_catalog_entries(status: Optional[str] = None, kind: str = "scheme") -> list[dict]: + """Rows of the ``catalog_status`` view (one per document) for one knowledge kind.""" + catalog_type = catalog_type_for_kind(kind) + with connection.get_connection() as conn: + if status: + rows = conn.execute( + "SELECT * FROM catalog_status WHERE type = %s AND status = %s ORDER BY code", + (catalog_type, status), + ).fetchall() + else: + rows = conn.execute( + "SELECT * FROM catalog_status WHERE type = %s ORDER BY code", (catalog_type,) + ).fetchall() + return [_catalog_row(r) for r in rows] + + +def get_catalog_entry(code: str, kind: str = "scheme") -> Optional[dict]: + with connection.get_connection() as conn: + row = conn.execute( + "SELECT * FROM catalog_status WHERE type = %s AND code = %s LIMIT 1", + (catalog_type_for_kind(kind), code), + ).fetchone() + return _catalog_row(row) if row else None + + +def list_used_scheme_codes( + exclude_workflow_id: Optional[str] = None, kind: str = "scheme" +) -> set[str]: + """Every document code already taken in a kind - the collision domain for auto-generated codes. + + The excluded document's own code is not counted, so re-deriving a + document's code does not collide with itself. + """ + with connection.get_connection() as conn: + doc_uuid = ( + document_reads._resolve_document_uuid(conn, exclude_workflow_id) + if exclude_workflow_id + else None + ) + rows = conn.execute( + """ + SELECT code FROM documents + WHERE kind = %s AND (%s::uuid IS NULL OR id <> %s::uuid) + """, + (kind, doc_uuid, doc_uuid), + ).fetchall() + return {r["code"] for r in rows} + + +def list_scheme_catalog_entries_for_workflow(workflow_id: str) -> list[dict]: + """Catalog rows whose documents still include this workflow.""" + with connection.get_connection() as conn: + rows = conn.execute( + "SELECT code, workflow_id FROM catalog_status WHERE type = %s AND workflow_id = %s", + (catalog_type_for_kind("scheme"), workflow_id), + ).fetchall() + return [{"code": r["code"], "workflow_ids": [r["workflow_id"]]} for r in rows] diff --git a/pipeline/db/chunks.py b/pipeline/db/chunks.py new file mode 100644 index 0000000..5d59306 --- /dev/null +++ b/pipeline/db/chunks.py @@ -0,0 +1,307 @@ +"""Persist, edit and retrieve document passages.""" + +import json +from typing import Any, Optional + +from psycopg.types.json import Jsonb + +from pipeline.documents.models import ( + ChunkingRun, +) + +from . import common, connection, document_reads, jobs + +# ============================================================================= +# Chunk Functions +# ============================================================================= + +_CHUNK_SELECT = """ + SELECT c.id, c.document_id, c.chunk_number, c.original_text, c.edited_text, c.token_count, + c.page_start, c.page_end, c.source_spans, c.section_title, c.content_type, + c.is_reference, c.is_reviewed, c.is_excluded, c.reviewer_notes, c.tags, + c.created_at, c.updated_at, d.metadata->'chunking' AS chunking + FROM chunks c + JOIN documents d ON d.id = c.document_id +""" + + +def _chunk_from_row(row: Optional[dict], workflow_id: str) -> Optional[dict]: + if row is None: + return None + spans = row["source_spans"] or [] + page_numbers = sorted({int(s["page_number"]) for s in spans if "page_number" in s}) or list( + range(int(row["page_start"]), int(row["page_end"]) + 1) + ) + run = row.get("chunking") or {} + chunk = { + "id": row["id"], + "workflow_id": workflow_id, + "chunk_number": row["chunk_number"], + "original_text": row["original_text"], + "edited_text": row["edited_text"], + "token_count": row["token_count"], + "page_start": row["page_start"], + "page_end": row["page_end"], + "source_page_numbers_json": json.dumps(page_numbers), + "source_spans_json": json.dumps(spans), + "section_title": row["section_title"], + "content_type": row["content_type"], + "is_reference": bool(row["is_reference"]), + "chunking_provider": run.get("provider"), + "chunking_model": run.get("model"), + "chunking_config_json": json.dumps(run["config"]) if run.get("config") else None, + "chunking_run_id": f"run-{run['run_number']}" if run.get("run_number") else None, + "chunk_version": run.get("run_number") or 1, + "is_reviewed": bool(row["is_reviewed"]), + "is_excluded": bool(row["is_excluded"]), + "reviewer_notes": row["reviewer_notes"], + "tags": row.get("tags") or [], + } + return {k: common._clean(v) for k, v in chunk.items()} + + +def _load_json_value(raw: Any, default: Any) -> Any: + if raw is None or raw == "": + return default + if isinstance(raw, str): + return json.loads(raw) + return raw + + +def save_chunks(workflow_id: str, chunks: list[dict]): + """ + Save all chunks for a document. + + Re-chunking runs in one transaction: the document's chunks are deleted, the + new ones inserted, and ``metadata.chunking`` (with ``run_number`` + incremented) updated together. + """ + with connection.get_connection() as conn: + doc_uuid = document_reads._require_document_uuid(conn, workflow_id) + previous = conn.execute( + "SELECT metadata->'chunking'->>'run_number' AS run_number FROM documents " + "WHERE id = %s::uuid FOR UPDATE", + (doc_uuid,), + ).fetchone() + run_number = int(previous["run_number"] or 0) + 1 + conn.execute("DELETE FROM chunks WHERE document_id = %s::uuid", (doc_uuid,)) + for chunk in chunks: + spans = _load_json_value( + chunk.get("source_spans") + if chunk.get("source_spans") is not None + else chunk.get("source_spans_json"), + [], + ) + page_start = int(chunk.get("page_start") or 1) + page_end = max(int(chunk.get("page_end") or page_start), page_start) + conn.execute( + """ + INSERT INTO chunks (document_id, chunk_number, original_text, edited_text, + token_count, page_start, page_end, source_spans, + section_title, content_type, is_reference, is_reviewed, + is_excluded, reviewer_notes) + VALUES (%s::uuid, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) + """, + ( + doc_uuid, + chunk.get("chunk_number"), + chunk.get("original_text") or "", + chunk.get("edited_text"), + chunk.get("token_count") or 0, + page_start, + page_end, + Jsonb(spans), + chunk.get("section_title"), + chunk.get("content_type"), + bool(chunk.get("is_reference")), + bool(chunk.get("is_reviewed")), + bool(chunk.get("is_excluded")), + chunk.get("reviewer_notes"), + ), + ) + meta_patch: dict[str, Any] = {} + if chunks: + first = chunks[0] + meta_patch["chunking"] = ChunkingRun( + run_number=run_number, + provider=first.get("chunking_provider"), + model=first.get("chunking_model"), + config=_load_json_value(first.get("chunking_config_json"), {}), + created_at=common._iso_now(), + ).model_dump() + if meta_patch: + conn.execute( + "UPDATE documents SET metadata = metadata || %s::jsonb WHERE id = %s::uuid", + (Jsonb(meta_patch), doc_uuid), + ) + else: + conn.execute("UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,)) + + +def get_chunks(workflow_id: str, include_excluded: bool = False) -> list[dict]: + """Get all chunks for a document.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return [] + root = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + excluded = "" if include_excluded else "AND NOT c.is_excluded" + rows = conn.execute( + f"{_CHUNK_SELECT} WHERE c.document_id = %s::uuid {excluded} ORDER BY c.chunk_number", + (doc_uuid,), + ).fetchall() + return [_chunk_from_row(r, root) for r in rows] + + +def search_chunks( + *, + query: str = "", + limit: int = 100, + offset: int = 0, + include_excluded: bool = False, + stage: Optional[str] = None, + instances: Optional[list[str]] = None, +) -> tuple[list[dict], int]: + """Search chunks across documents for maintainer workflows. + + instances: If provided, only return chunks whose owning document's instance + is in this list (tenant scoping). ``None`` means no instance restriction. + """ + where_clauses = ["TRUE"] + params: list = [] + + if not include_excluded: + where_clauses.append("NOT c.is_excluded") + if stage: + where_clauses.append("cw.current_stage = %s") + params.append(stage) + if instances is not None: + normalized = common._normalize_instances(instances) + if not normalized: + where_clauses.append("FALSE") + else: + where_clauses.append("lower(g.name) = ANY(%s)") + params.append(normalized) + if query and query.strip(): + where_clauses.append("lower(coalesce(c.edited_text, c.original_text)) LIKE %s") + params.append(f"%{query.strip().lower()}%") + + base_from_sql = f""" + FROM chunks c + JOIN documents d ON d.id = c.document_id + JOIN instances g ON g.id = d.provider_id + LEFT JOIN document_current_workflow cw ON cw.document_id = d.id + WHERE {" AND ".join(where_clauses)} + """ + with connection.get_connection() as conn: + total_row = conn.execute(f"SELECT COUNT(*) AS cnt {base_from_sql}", params).fetchone() + total = int(total_row["cnt"] if total_row and total_row["cnt"] is not None else 0) + rows = conn.execute( + f""" + SELECT + c.id, c.chunk_number, c.original_text, c.edited_text, c.token_count, + c.page_start, c.page_end, c.source_spans, c.section_title, c.content_type, + c.is_reference, c.is_reviewed, c.is_excluded, c.reviewer_notes, c.tags, + d.metadata->'chunking' AS chunking, + d.source->>'fingerprint' AS document_id, + d.source->>'original_filename' AS filename, + d.name AS name, + cw.current_stage AS stage, + d.updated_at AS document_updated_at, + (SELECT r.temporal_workflow_id FROM workflows r WHERE r.document_id = d.id + ORDER BY (r.workflow_type = 'legacy_index_import'), (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC + LIMIT 1) AS root_workflow_id + {base_from_sql} + ORDER BY d.updated_at DESC, d.id, c.chunk_number + LIMIT %s OFFSET %s + """, + [*params, int(limit), int(offset)], + ).fetchall() + + chunks: list[dict] = [] + for row in rows: + row = dict(row) + chunk = _chunk_from_row(row, row["root_workflow_id"]) + for extra in ("document_id", "filename", "name", "stage", "document_updated_at"): + chunk[extra] = common._clean(row[extra]) + chunks.append(chunk) + return chunks, total + + +def get_chunk(workflow_id: str, chunk_num: int) -> Optional[dict]: + """Get a specific chunk.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return None + root = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + row = conn.execute( + _CHUNK_SELECT + " WHERE c.document_id = %s::uuid AND c.chunk_number = %s", + (doc_uuid, chunk_num), + ).fetchone() + return _chunk_from_row(row, root) + + +def update_chunk( + workflow_id: str, + chunk_num: int, + edited_text: Optional[str] = None, + is_reviewed: Optional[bool] = None, + is_excluded: Optional[bool] = None, + reviewer_notes: Optional[str] = None, + tags: list[str] | None = None, +) -> Optional[dict]: + """Update a chunk. Returns updated chunk or None if not found.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return None + existing = conn.execute( + "SELECT id FROM chunks WHERE document_id = %s::uuid AND chunk_number = %s", + (doc_uuid, chunk_num), + ).fetchone() + if not existing: + return None + + updates = [] + values: list = [] + if edited_text is not None: + updates.append("edited_text = %s") + values.append(edited_text) + if is_reviewed is not None: + updates.append("is_reviewed = %s") + values.append(bool(is_reviewed)) + if is_excluded is not None: + updates.append("is_excluded = %s") + values.append(bool(is_excluded)) + if reviewer_notes is not None: + updates.append("reviewer_notes = %s") + values.append(reviewer_notes) + if tags is not None: + updates.append("tags = %s") + distinct = {} + for tag in tags: + distinct.setdefault(tag.casefold(), tag) + values.append(list(distinct.values())) + if updates: + conn.execute( + f"UPDATE chunks SET {', '.join(updates)} WHERE document_id = %s::uuid AND chunk_number = %s", + (*values, doc_uuid, chunk_num), + ) + return get_chunk(workflow_id, chunk_num) + + +def reset_chunk(workflow_id: str, chunk_num: int) -> Optional[dict]: + """Reset a chunk to its original text.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is not None: + conn.execute( + """ + UPDATE chunks SET edited_text = NULL, is_reviewed = FALSE, is_excluded = FALSE, + reviewer_notes = NULL + WHERE document_id = %s::uuid AND chunk_number = %s + """, + (doc_uuid, chunk_num), + ) + return get_chunk(workflow_id, chunk_num) diff --git a/pipeline/db/common.py b/pipeline/db/common.py new file mode 100644 index 0000000..77c2937 --- /dev/null +++ b/pipeline/db/common.py @@ -0,0 +1,82 @@ +"""Normalize identifiers and values shared by PostgreSQL queries.""" + +import hashlib +import os +import re +import uuid +from datetime import date, datetime, timezone +from typing import Any, Optional + +# ============================================================================= +# Helpers +# ============================================================================= + +_UUID_RE = re.compile( + r"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$" +) + + +_FINGERPRINT_RE = re.compile(r"^[0-9a-f]{32}$") + + +_REVIEW_STAGES = { + "ocr_review", + "translation_review", + "chunk_review", + "ready_for_ingestion", + "approval_for_prod", +} + + +def _default_instance() -> str: + return (os.environ.get("DEFAULT_INSTANCE") or "default").strip().lower() or "default" + + +def _normalize_instances(instances: list[str]) -> list[str]: + default = _default_instance() + return sorted({(i or "").strip().lower() or default for i in instances}) + + +def _iso_now() -> str: + return datetime.now(timezone.utc).replace(tzinfo=None).isoformat() + + +def _is_uuid(value: Any) -> bool: + return isinstance(value, str) and bool(_UUID_RE.match(value)) + + +def _clean(value: Any) -> Any: + if isinstance(value, uuid.UUID): + return str(value) + if isinstance(value, datetime): + if value.tzinfo is not None: + value = value.astimezone(timezone.utc).replace(tzinfo=None) + return value.isoformat() + if isinstance(value, date): + return value.isoformat() + return value + + +def _clean_row(row: Optional[dict]) -> Optional[dict]: + if row is None: + return None + return {k: _clean(v) for k, v in row.items()} + + +def _csv_roles(roles: Any) -> list[str]: + if roles is None: + return [] + if isinstance(roles, (list, tuple, set)): + return [str(r).strip() for r in roles if str(r).strip()] + return [part.strip() for part in str(roles).split(",") if part.strip()] + + +def _fingerprint_for( + source_file_fingerprint: Optional[str], + document_id: Optional[str], + workflow_id: Optional[str], +) -> str: + for candidate in (source_file_fingerprint, document_id): + if candidate and _FINGERPRINT_RE.match(candidate): + return candidate + return hashlib.md5((document_id or workflow_id or "").encode()).hexdigest() diff --git a/pipeline/db/connection.py b/pipeline/db/connection.py new file mode 100644 index 0000000..b14a7d3 --- /dev/null +++ b/pipeline/db/connection.py @@ -0,0 +1,95 @@ +"""Create the shared Psycopg pool and check database readiness.""" + +import logging +import os +import threading +from contextlib import contextmanager +from typing import Optional + +from psycopg.rows import dict_row +from psycopg_pool import ConnectionPool + +from pipeline import migrate + +logger = logging.getLogger(__name__) + + +DEFAULT_DATABASE_URL = "postgresql://knowledge_provider@localhost:5432/knowledge_provider" + + +_pool: Optional[ConnectionPool] = None + + +_pool_url: Optional[str] = None + + +_pool_lock = threading.Lock() + + +def database_url() -> str: + return os.environ.get("DATABASE_URL") or DEFAULT_DATABASE_URL + + +def get_pool() -> ConnectionPool: + global _pool, _pool_url + url = database_url() + if _pool is not None and _pool_url == url: + return _pool + with _pool_lock: + if _pool is not None and _pool_url != url: + _pool.close() + _pool = None + if _pool is None: + pool = ConnectionPool( + conninfo=url, + min_size=int(os.environ.get("DB_POOL_MIN_SIZE", "1")), + max_size=int(os.environ.get("DB_POOL_MAX_SIZE", "10")), + timeout=float(os.environ.get("DB_POOL_TIMEOUT_SECONDS", "30")), + kwargs={"row_factory": dict_row, "options": "-c timezone=UTC"}, + open=False, + ) + pool.open(wait=True) + _pool = pool + _pool_url = url + logger.info("db pool opened: %s", pool.get_stats()) + return _pool + + +def close_pool() -> None: + global _pool, _pool_url + with _pool_lock: + if _pool is not None: + _pool.close() + _pool = None + _pool_url = None + + +@contextmanager +def get_connection(): + """A pooled connection; the transaction commits on exit and rolls back on error.""" + with get_pool().connection() as conn: + yield conn + + +def init_db() -> None: + """Apply pending migrations and open the connection pool.""" + from . import catalog + + applied = migrate.apply_migrations(database_url()) + current = migrate.current_version(database_url()) + logger.info("db migrations current=%s applied_now=%s", current, applied) + get_pool() + created = catalog.sync_catalogs_for_all_instances() + if created: + logger.info("catalog sync created %d missing catalogs", created) + + +def check_health() -> dict: + """Reachability and migration state, for the /health endpoint.""" + try: + with get_connection() as conn: + conn.execute("SELECT 1") + return {"reachable": True, "migrations_current": migrate.is_current(database_url())} + except Exception: + logger.exception("database health check failed") + return {"reachable": False, "migrations_current": False} diff --git a/pipeline/db/document_reads.py b/pipeline/db/document_reads.py new file mode 100644 index 0000000..19d2f04 --- /dev/null +++ b/pipeline/db/document_reads.py @@ -0,0 +1,418 @@ +"""Read provider-owned documents and resolve their stable identifiers.""" + +from typing import Optional + +from pipeline.search.index_identity import projection_owner + +from . import common, connection + +# ============================================================================= +# Identifier resolution +# ============================================================================= + + +def _resolve_document_uuid(conn, identifier: Optional[str]) -> Optional[str]: + """Document UUID for a UUID, Temporal workflow id, fingerprint or filename.""" + if not identifier: + return None + identifier = str(identifier) + identifier = projection_owner(identifier) or identifier + + if common._is_uuid(identifier): + row = conn.execute("SELECT id FROM documents WHERE id = %s::uuid", (identifier,)).fetchone() + if row: + return str(row["id"]) + + row = conn.execute( + """ + SELECT document_id FROM workflows + WHERE temporal_workflow_id = %s + ORDER BY started_at DESC LIMIT 1 + """, + (identifier,), + ).fetchone() + if row: + return str(row["document_id"]) + + row = conn.execute( + """ + SELECT id FROM documents + WHERE source->>'fingerprint' = %s OR source->>'original_filename' = %s + ORDER BY created_at DESC LIMIT 1 + """, + (identifier, identifier), + ).fetchone() + if row: + return str(row["id"]) + + row = conn.execute( + """ + SELECT id FROM documents d + WHERE EXISTS ( + SELECT 1 FROM jsonb_each(COALESCE(d.metadata->'index_status', '{}'::jsonb)) e + WHERE e.value->>'doc_id' = %s + ) OR md5(d.source->>'fingerprint') = %s + ORDER BY created_at DESC LIMIT 1 + """, + (identifier, identifier), + ).fetchone() + return str(row["id"]) if row else None + + +def _require_document_uuid(conn, identifier: str) -> str: + doc_uuid = _resolve_document_uuid(conn, identifier) + if doc_uuid is None: + raise LookupError(f"Document not found: {identifier}") + return doc_uuid + + +# ============================================================================= +# Document row <-> flat dict +# ============================================================================= + +_DOC_SELECT = """ + SELECT + d.id, c.id AS catalog_id, d.kind, d.code, d.name, d.aliases, d.source, d.is_disabled, d.metadata, + d.created_at, d.updated_at, + g.name AS instance_name, + (SELECT count(*) FROM pages p WHERE p.document_id = d.id) AS page_total, + (SELECT count(*) FROM chunks ch WHERE ch.document_id = d.id) AS chunk_total, + root.temporal_workflow_id AS root_workflow_id, + root.stop_after_ocr AS root_stop_after_ocr, + cw.id AS current_workflow_pk, + cw.current_stage AS current_stage, + cw.error_message AS current_error_message, + EXISTS (SELECT 1 FROM workflows w WHERE w.document_id = d.id + AND w.config->>'release_network_id' IS NOT NULL) AS uses_network_releases, + u.keycloak_user_id AS up_user_id, u.username AS up_username, + u.email AS up_email, u.roles AS up_roles, + ts.ocr_completed_at, ts.translation_completed_at, ts.chunks_completed_at, + ts.ingested_at, + req.started_at AS prod_ready_requested_at, + ru.keycloak_user_id AS prod_ready_requested_by_user_id, + ru.username AS prod_ready_requested_by_username, + oa.id AS original_artifact_id, + na.id AS normalized_artifact_id + FROM documents d + LEFT JOIN legacy_document_catalog_types m ON m.kind = d.kind + LEFT JOIN catalogs c ON c.instance_id = d.provider_id AND c.type = COALESCE(m.catalog_type, d.kind) + JOIN instances g ON g.id = d.provider_id + LEFT JOIN document_current_workflow cw ON cw.document_id = d.id + LEFT JOIN LATERAL ( + SELECT w.temporal_workflow_id, w.stop_after_ocr FROM workflows w + WHERE w.document_id = d.id + ORDER BY (w.workflow_type = 'legacy_index_import'), (w.workflow_type IN ('pipeline', 'ocr_only')) DESC, w.started_at ASC, w.id ASC + LIMIT 1 + ) root ON TRUE + LEFT JOIN users u ON u.id = NULLIF(d.metadata->>'uploaded_by_user_id', '')::uuid + LEFT JOIN LATERAL ( + SELECT min(w.stage_timestamps->>'ocr') AS ocr_completed_at, + min(w.stage_timestamps->>'translation') AS translation_completed_at, + min(w.stage_timestamps->>'chunks') AS chunks_completed_at, + max(COALESCE(w.completed_at, w.updated_at)) + FILTER (WHERE w.current_stage = 'completed') AS ingested_at + FROM workflows w WHERE w.document_id = d.id + ) ts ON TRUE + LEFT JOIN LATERAL ( + SELECT w.started_at, w.started_by_user_id FROM workflows w + WHERE w.document_id = d.id AND w.workflow_type = 'promote_prod' + AND w.status = 'waiting_review' + ORDER BY w.started_at DESC LIMIT 1 + ) req ON TRUE + LEFT JOIN users ru ON ru.id = req.started_by_user_id + LEFT JOIN LATERAL ( + SELECT a.id FROM workflow_artifacts a JOIN workflows w ON w.id = a.workflow_id + WHERE w.document_id = d.id AND a.artifact_type = 'original_upload' + ORDER BY a.created_at DESC, a.id DESC LIMIT 1 + ) oa ON TRUE + LEFT JOIN LATERAL ( + SELECT a.id FROM workflow_artifacts a JOIN workflows w ON w.id = a.workflow_id + WHERE w.document_id = d.id + AND a.artifact_type IN ('normalized_pdf', 'normalized_spreadsheet') + ORDER BY a.created_at DESC, a.id DESC LIMIT 1 + ) na ON TRUE +""" + + +def _document_from_row(row: Optional[dict]) -> Optional[dict]: + if row is None: + return None + source = row["source"] or {} + meta = row["metadata"] or {} + fingerprint = source.get("fingerprint") + roles = row["up_roles"] or [] + entry_aliases = row["aliases"] + doc = { + "id": row["id"], + "workflow_id": row["root_workflow_id"], + "document_id": fingerprint, + "canonical_document_id": fingerprint, + "filename": source.get("original_filename") or "", + "name": row["name"], + "source_filename": source.get("original_filename"), + "source_manifest_name": None, + "source_file_fingerprint": fingerprint, + "filepath": source.get("storage_uri") or "", + "instance": (row["instance_name"] or "").lower(), + "stage": row["current_stage"] or "registered", + "page_count": int(row["page_total"] or 0), + "chunk_count": int(row["chunk_total"] or 0), + "error_message": row["current_error_message"], + "is_disabled": bool(row["is_disabled"]), + "source_type": source.get("source_type"), + "canonical_input_type": source.get("canonical_input_type"), + "stop_after_ocr": bool(row["root_stop_after_ocr"]), + "reindex_required": bool(meta.get("reindex_required")), + "reindex_reason": meta.get("reindex_reason"), + "kind": row["kind"], + "code": row["code"], + "aliases": list(entry_aliases or []), + "tool_routing": "qdrant", + "catalog_visible": bool(meta.get("catalog_visible", True)), + "network_visible": bool(meta.get("network_visible", True)), + "uses_network_releases": bool(row["uses_network_releases"]), + "valid_from": meta.get("valid_from"), + "valid_to": meta.get("valid_to"), + "crop_subjects": meta.get("crop_subjects") or [], + "knowledge_metadata": meta.get("knowledge_metadata") or {}, + "uploaded_by_user_id": row["up_user_id"], + "uploaded_by_username": row["up_username"], + "uploaded_by_email": row["up_email"], + "uploaded_by_roles": ",".join(roles) if roles else None, + "prod_ready_requested_at": row["prod_ready_requested_at"], + "prod_ready_requested_by_user_id": row["prod_ready_requested_by_user_id"], + "prod_ready_requested_by_username": row["prod_ready_requested_by_username"], + "ocr_completed_at": row["ocr_completed_at"], + "translation_completed_at": row["translation_completed_at"], + "chunks_completed_at": row["chunks_completed_at"], + "ingested_at": row["ingested_at"], + "original_artifact_id": row["original_artifact_id"], + "normalized_artifact_id": row["normalized_artifact_id"], + "latest_job_id": row["current_workflow_pk"], + "catalog_id": row["catalog_id"], + "index_status": meta.get("index_status") or {}, + "superseded_doc_ids": meta.get("superseded_doc_ids") or {}, + "created_at": row["created_at"], + "updated_at": row["updated_at"], + } + return {k: common._clean(v) for k, v in doc.items()} + + +def _fetch_document(conn, doc_uuid: Optional[str]) -> Optional[dict]: + if doc_uuid is None: + return None + row = conn.execute(_DOC_SELECT + " WHERE d.id = %s::uuid", (doc_uuid,)).fetchone() + return _document_from_row(row) + + +def _document_filters( + include_disabled: bool, + instances: Optional[list[str]], + alias: str = "d", + instance_alias: str = "g", +) -> tuple[list[str], list]: + """WHERE clauses for document lists. The query must join ``instances`` as ``instance_alias``.""" + clauses: list[str] = [] + params: list = [] + if not include_disabled: + clauses.append(f"NOT {alias}.is_disabled") + if instances is not None: + clauses.append(f"lower({instance_alias}.name) = ANY(%s)") + params.append(common._normalize_instances(instances)) + return clauses, params + + +def owned_source_fingerprint(storage_uri: str, provider: str) -> str | None: + """Return the single known source fingerprint for an active provider-owned object.""" + with connection.get_connection() as conn: + rows = conn.execute( + """ + SELECT DISTINCT d.source->>'fingerprint' AS fingerprint + FROM documents d + JOIN instances g ON g.id = d.provider_id + WHERE d.source->>'storage_uri' = %s AND g.name = %s AND NOT d.is_disabled + LIMIT 2 + """, + (storage_uri, provider), + ).fetchall() + return rows[0]["fingerprint"] if len(rows) == 1 else None + + +def find_document_by_code(instance: str, kind: str, code: str) -> Optional[dict]: + """The document holding ``code`` in the instance's ``kind`` catalog, if any.""" + tenant = (instance or common._default_instance()).strip().lower() or common._default_instance() + with connection.get_connection() as conn: + row = conn.execute( + """ + SELECT d.id FROM documents d + JOIN instances g ON g.id = d.provider_id + WHERE lower(g.name) = %s AND d.kind = %s AND d.code = %s + """, + (tenant, kind, code), + ).fetchone() + return _fetch_document(conn, str(row["id"])) if row else None + + +def get_document(workflow_id: str) -> Optional[dict]: + """Get a single document by workflow id, UUID or any other identifier.""" + with connection.get_connection() as conn: + return _fetch_document(conn, _resolve_document_uuid(conn, workflow_id)) + + +def list_documents( + stage: Optional[str] = None, + limit: int = 100, + offset: int = 0, + include_disabled: bool = False, + instances: Optional[list[str]] = None, +) -> list[dict]: + """List documents with optional stage / instance filters. + + Args: + stage: Filter by document stage + limit: Max documents to return + offset: Pagination offset + include_disabled: If False (default), excludes soft-deleted documents from results + instances: If provided, only return docs whose instance is in this list + """ + if instances is not None and not common._normalize_instances(instances): + return [] + clauses, params = _document_filters(include_disabled, instances) + if stage: + clauses.append("cw.current_stage = %s") + params.append(stage) + where = f"WHERE {' AND '.join(clauses)}" if clauses else "" + with connection.get_connection() as conn: + rows = conn.execute( + f"{_DOC_SELECT} {where} ORDER BY d.created_at DESC, d.id LIMIT %s OFFSET %s", + (*params, limit, offset), + ).fetchall() + return [_document_from_row(r) for r in rows] + + +_REVIEW_QUEUE_STAGES = ("ocr_review", "translation_review", "chunk_review", "approval_for_prod") + + +_SUMMARY_KEYS = ( + "total_documents", + "authoritative_documents", + "legacy_documents", + "completed_documents", + "review_queue", + "ocr_review_documents", + "translation_review_documents", + "chunk_review_documents", + "translation_processing_documents", + "chunking_documents", + "ready_for_ingestion_documents", + "failed_documents", + "needs_reindex", + "running_jobs", +) + + +def get_document_summary_counts( + include_disabled: bool = False, + instances: Optional[list[str]] = None, +) -> dict: + """Return aggregate document counts for dashboard and migration planning.""" + if instances is not None and not common._normalize_instances(instances): + return {key: 0 for key in _SUMMARY_KEYS} + clauses, params = _document_filters(include_disabled, instances) + where = f"WHERE {' AND '.join(clauses)}" if clauses else "" + with connection.get_connection() as conn: + row = conn.execute( + f""" + SELECT + COUNT(*) AS total_documents, + 0 AS authoritative_documents, + COUNT(*) AS legacy_documents, + COUNT(*) FILTER (WHERE cw.current_stage = 'completed') AS completed_documents, + COUNT(*) FILTER (WHERE cw.current_stage IN + ('ocr_review', 'translation_review', 'chunk_review', 'approval_for_prod')) AS review_queue, + COUNT(*) FILTER (WHERE cw.current_stage = 'ocr_review') AS ocr_review_documents, + COUNT(*) FILTER (WHERE cw.current_stage = 'translation_review') AS translation_review_documents, + COUNT(*) FILTER (WHERE cw.current_stage = 'chunk_review') AS chunk_review_documents, + COUNT(*) FILTER (WHERE cw.current_stage = 'translation_processing') AS translation_processing_documents, + COUNT(*) FILTER (WHERE cw.current_stage = 'chunking') AS chunking_documents, + COUNT(*) FILTER (WHERE cw.current_stage = 'ready_for_ingestion') AS ready_for_ingestion_documents, + COUNT(*) FILTER (WHERE cw.current_stage = 'failed') AS failed_documents, + COUNT(*) FILTER (WHERE COALESCE((d.metadata->>'reindex_required')::boolean, FALSE)) AS needs_reindex, + COALESCE(SUM((SELECT COUNT(*) FROM workflows j + WHERE j.document_id = d.id AND j.status = 'running')), 0) AS running_jobs + FROM documents d + JOIN instances g ON g.id = d.provider_id + LEFT JOIN document_current_workflow cw ON cw.document_id = d.id + {where} + """, + params, + ).fetchone() + return {key: int(row[key] or 0) for key in _SUMMARY_KEYS} + + +def get_document_counts_by_instance( + include_disabled: bool = False, + instances: Optional[list[str]] = None, +) -> list[dict]: + """Return document counts grouped by instance (state), for dashboard breakdowns.""" + if instances is not None and not common._normalize_instances(instances): + return [] + clauses, params = _document_filters(include_disabled, instances) + where = f"WHERE {' AND '.join(clauses)}" if clauses else "" + with connection.get_connection() as conn: + rows = conn.execute( + f""" + SELECT + lower(g.name) AS "instance", + COUNT(*) AS count, + COUNT(*) FILTER (WHERE cw.current_stage = 'completed') AS success, + COUNT(*) FILTER (WHERE cw.current_stage = 'failed') AS failed, + COUNT(*) FILTER (WHERE cw.current_stage IN + ('ocr_review', 'translation_review', 'chunk_review', 'approval_for_prod')) AS dev_approval + FROM documents d + JOIN instances g ON g.id = d.provider_id + LEFT JOIN document_current_workflow cw ON cw.document_id = d.id + {where} + GROUP BY lower(g.name) + ORDER BY count DESC, "instance" ASC + """, + params, + ).fetchall() + return [ + { + "instance": row["instance"], + "count": int(row["count"] or 0), + "success": int(row["success"] or 0), + "failed": int(row["failed"] or 0), + "dev_approval": int(row["dev_approval"] or 0), + } + for row in rows + ] + + +def get_document_count(stage: Optional[str] = None) -> int: + """Get count of documents, optionally filtered by stage.""" + with connection.get_connection() as conn: + if stage: + row = conn.execute( + """ + SELECT COUNT(*) AS cnt FROM documents d + JOIN document_current_workflow cw ON cw.document_id = d.id + WHERE cw.current_stage = %s + """, + (stage,), + ).fetchone() + else: + row = conn.execute("SELECT COUNT(*) AS cnt FROM documents").fetchone() + return int(row["cnt"]) if row else 0 + + +def get_document_source_url(document_id: str) -> str | None: + with connection.get_connection() as conn: + row = conn.execute( + "SELECT config->>'source_url' AS url FROM workflows " + "WHERE document_id = %s::uuid AND config ? 'source_url' " + "ORDER BY started_at DESC, id DESC LIMIT 1", + (document_id,), + ).fetchone() + return row["url"] if row else None diff --git a/pipeline/db/document_state.py b/pipeline/db/document_state.py new file mode 100644 index 0000000..0f3dbd8 --- /dev/null +++ b/pipeline/db/document_state.py @@ -0,0 +1,183 @@ +"""Mirror processing stages and reconcile persisted workflow state.""" + +from typing import Optional + +from . import audit, common, connection, document_reads, jobs + + +def update_document_stage( + workflow_id: str, + stage: str, + page_count: Optional[int] = None, + chunk_count: Optional[int] = None, + error_message: Optional[str] = None, +): + """Update the stage (of the document's current workflow). + + ``page_count`` and ``chunk_count`` are accepted for call-site compatibility; + both are counted from ``pages`` and ``chunks``, so only the audit entry keeps them. + """ + old_stage = None + audit_workflow_id = workflow_id + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return + cw = conn.execute( + "SELECT id, current_stage FROM document_current_workflow WHERE document_id = %s::uuid", + (doc_uuid,), + ).fetchone() + if cw is None: + return + old_stage = cw["current_stage"] + + assignments = ["current_stage = %s"] + params: list = [stage] + if error_message is not None: + assignments.append("error_message = %s") + params.append(error_message) + stamp_key = { + "ocr_review": "ocr", + "translation_review": "translation", + "chunk_review": "chunks", + }.get(stage) + if stamp_key: + assignments.append( + "stage_timestamps = jsonb_build_object(%s::text, %s::text) || stage_timestamps" + ) + params.extend([stamp_key, common._iso_now()]) + if stage == "completed": + assignments.append("completed_at = now()") + params.append(str(cw["id"])) + conn.execute(f"UPDATE workflows SET {', '.join(assignments)} WHERE id = %s::uuid", params) + + conn.execute("UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,)) + audit_workflow_id = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + + if old_stage and old_stage != stage: + audit.log_audit( + workflow_id=audit_workflow_id, + document_id=audit_workflow_id, + action_type="stage_change", + field_name="stage", + old_value=old_stage, + new_value=stage, + metadata={"page_count": page_count, "chunk_count": chunk_count}, + ) + + +def reconcile_materialized_state(workflow_id: str) -> Optional[dict]: + """ + Reconcile a document row against already-persisted page/chunk content. + + This is intentionally conservative: it only moves a document forward into + review states when the corresponding materialized content already exists. + """ + doc = document_reads.get_document(workflow_id) + if not doc: + return None + workflow_id = doc["workflow_id"] + + with connection.get_connection() as conn: + page_stats = conn.execute( + """ + SELECT COUNT(*) AS page_rows, COALESCE(MAX(page_number), 0) AS max_page_number + FROM pages WHERE document_id = %s::uuid + """, + (doc["id"],), + ).fetchone() + chunk_stats = conn.execute( + "SELECT COUNT(*) AS chunk_rows FROM chunks WHERE document_id = %s::uuid", + (doc["id"],), + ).fetchone() + + page_count = int((page_stats["page_rows"] if page_stats else 0) or 0) + max_page_number = int((page_stats["max_page_number"] if page_stats else 0) or 0) + materialized_pages = max(page_count, max_page_number, int(doc.get("page_count") or 0)) + chunk_count = int((chunk_stats["chunk_rows"] if chunk_stats else 0) or 0) + + current_stage = doc.get("stage") + latest_job = jobs.get_latest_document_job(workflow_id) + updated = False + now = common._iso_now() + + # Highest-confidence reconciliation first: if chunks exist, the document has + # already reached chunk review materiality even if the workflow stalled. + if chunk_count > 0 and current_stage not in { + "chunk_review", + "ready_for_ingestion", + "ingesting", + "approval_for_prod", + "completed", + }: + update_document_stage( + workflow_id=workflow_id, + stage="chunk_review", + page_count=materialized_pages, + chunk_count=chunk_count, + error_message=None, + ) + if latest_job: + jobs.update_document_job( + latest_job["id"], + current_stage="chunk_review", + status="waiting_review", + error_message=None, + ) + doc = document_reads.get_document(workflow_id) or doc + current_stage = doc.get("stage") + updated = True + + # Lower-confidence but still safe: if OCR pages exist and the document is + # somehow still marked as pre-review, move it into OCR review. + elif materialized_pages > 0 and current_stage in {"registered", "ocr_processing"}: + update_document_stage( + workflow_id=workflow_id, + stage="ocr_review", + page_count=materialized_pages, + chunk_count=int(doc.get("chunk_count") or 0), + error_message=None, + ) + if latest_job: + jobs.update_document_job( + latest_job["id"], + current_stage="ocr_review", + status="waiting_review", + error_message=None, + ) + updated = True + + # Even when stage is already correct, keep counts truthful. + elif materialized_pages != int(doc.get("page_count") or 0) or chunk_count != int( + doc.get("chunk_count") or 0 + ): + update_document_stage( + workflow_id=workflow_id, + stage=current_stage, + page_count=materialized_pages, + chunk_count=chunk_count, + error_message=doc.get("error_message"), + ) + updated = True + + if not updated: + return { + "workflow_id": workflow_id, + "updated": False, + "stage": current_stage, + "page_count": int(doc.get("page_count") or 0), + "chunk_count": int(doc.get("chunk_count") or 0), + } + + refreshed = document_reads.get_document(workflow_id) or doc + refreshed_job = jobs.get_latest_document_job(workflow_id) + return { + "workflow_id": workflow_id, + "updated": True, + "stage": refreshed.get("stage"), + "page_count": int(refreshed.get("page_count") or 0), + "chunk_count": int(refreshed.get("chunk_count") or 0), + "job_status": refreshed_job.get("status") if refreshed_job else None, + "job_stage": refreshed_job.get("current_stage") if refreshed_job else None, + "reconciled_at": now, + } diff --git a/pipeline/db/document_writes.py b/pipeline/db/document_writes.py new file mode 100644 index 0000000..366697d --- /dev/null +++ b/pipeline/db/document_writes.py @@ -0,0 +1,592 @@ +"""Create, replace and update document records.""" + +from contextlib import nullcontext +from datetime import date +from typing import Any, Optional + +import psycopg +from psycopg.types.json import Jsonb + +from pipeline.documents.models import ( + DocumentMetadata, + DocumentSource, +) +from pipeline.metadata.schema_packs import validate_schema_selection +from pipeline.search.index_identity import projection_doc_id + +from . import common, connection, document_reads, jobs, providers + + +class DuplicateDocumentCode(ValueError): + """The ``(catalog, code)`` pair is already held by another document.""" + + +class DocumentReplacementConflict(ValueError): + """The document changed while a replacement upload was being registered.""" + + +# ============================================================================= +# Documents +# ============================================================================= + + +def _stored_source(conn, doc_uuid: str) -> dict: + row = conn.execute("SELECT source FROM documents WHERE id = %s::uuid", (doc_uuid,)).fetchone() + return dict(row["source"]) if row else {} + + +def upsert_document( + workflow_id: str, + document_id: str, + filename: str, + filepath: str, + canonical_document_id: Optional[str] = None, + name: Optional[str] = None, + source_filename: Optional[str] = None, + source_manifest_name: Optional[str] = None, + source_file_fingerprint: Optional[str] = None, + stage: str = "registered", + page_count: int = 0, + chunk_count: int = 0, + error_message: Optional[str] = None, + ocr_completed_at: Optional[str] = None, + translation_completed_at: Optional[str] = None, + chunks_completed_at: Optional[str] = None, + ingested_at: Optional[str] = None, + source_type: Optional[str] = None, + canonical_input_type: Optional[str] = None, + stop_after_ocr: bool = False, + original_artifact_id: Optional[str] = None, + normalized_artifact_id: Optional[str] = None, + latest_job_id: Optional[str] = None, + instance: Optional[str] = None, + uploaded_by_user_id: Optional[str] = None, + uploaded_by_username: Optional[str] = None, + uploaded_by_email: Optional[str] = None, + uploaded_by_roles: Optional[str] = None, + valid_from: Optional[str] = None, + valid_to: Optional[str] = None, + with_workflow: bool = True, + kind: Optional[str] = None, + code: Optional[str] = None, + aliases: Optional[list[str]] = None, + crop_subjects: Optional[list[dict]] = None, + knowledge_metadata: Optional[dict] = None, + *, + _conn=None, +) -> str: + """Insert or update a document record; returns the document UUID. + + A new document needs a kind and a code unique within its provider and kind. + The instance code resolves to the provider UUID. A missing display name + defaults to the code. Existing workflows keep their original provider and + kind when re-registering a document. + Uploader identity is set on INSERT, and filled on UPDATE only when currently null. + + ``valid_from``/``valid_to`` are the document's validity period, stored as + given and likewise INSERT-only, so an approver's later edit survives a + restart that re-registers the same workflow. What the period should default + to is the caller's call (see ``pipeline/document_validity.py``); omitting + both stores NULL, which search reads as "no stated period". + + ``page_count`` and ``chunk_count`` are accepted for call-site compatibility + and ignored: both are counted from ``pages`` and ``chunks``. + + A new document also gets its first workflow row (``temporal_workflow_id`` = + ``workflow_id``) unless ``with_workflow`` is False, for callers that create + it themselves with ``create_document_job`` (addressing the document by the + returned UUID, since no workflow id resolves to it yet). + """ + instance_value = ( + instance or common._default_instance() + ).strip().lower() or common._default_instance() + fingerprint = common._fingerprint_for( + source_file_fingerprint or canonical_document_id, document_id, workflow_id + ) + source_patch = { + "fingerprint": fingerprint, + "original_filename": filename, + "storage_uri": filepath, + "source_type": source_type, + "canonical_input_type": canonical_input_type, + } + with connection.get_connection() if _conn is None else nullcontext(_conn) as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) if workflow_id else None + if doc_uuid: + patch = {k: v for k, v in source_patch.items() if v is not None} + DocumentSource.model_validate({**_stored_source(conn, doc_uuid), **patch}) + uploader_uuid = ( + providers.upsert_user( + uploaded_by_user_id, + uploaded_by_username, + uploaded_by_email, + uploaded_by_roles, + conn=conn, + tenants=[instance_value], + ) + if uploaded_by_user_id + else None + ) + conn.execute( + """ + UPDATE documents SET + source = source || %s::jsonb, + name = COALESCE(%s, name), + metadata = CASE + WHEN %s::text IS NULL OR metadata ? 'uploaded_by_user_id' THEN metadata + ELSE metadata || jsonb_build_object('uploaded_by_user_id', %s::text) + END + WHERE id = %s::uuid + """, + (Jsonb(patch), name or None, uploader_uuid, uploader_uuid, doc_uuid), + ) + cw = conn.execute( + "SELECT id FROM document_current_workflow WHERE document_id = %s::uuid", (doc_uuid,) + ).fetchone() + if cw: + conn.execute( + """ + UPDATE workflows SET current_stage = %s, error_message = %s, + stop_after_ocr = %s + WHERE id = %s::uuid + """, + (stage, error_message, bool(stop_after_ocr), str(cw["id"])), + ) + return doc_uuid + + if not kind or not code: + raise ValueError("A new document needs kind and code") + source = DocumentSource.model_validate( + {k: v for k, v in source_patch.items() if v is not None} + ).model_dump(exclude_none=True) + user_uuid = ( + providers.upsert_user( + uploaded_by_user_id, + uploaded_by_username, + uploaded_by_email, + uploaded_by_roles, + conn=conn, + tenants=[instance_value], + ) + if uploaded_by_user_id + else None + ) + validated_metadata = DocumentMetadata( + valid_from=valid_from, + valid_to=valid_to, + uploaded_by_user_id=user_uuid, + crop_subjects=crop_subjects or [], + knowledge_metadata=knowledge_metadata, + ) + metadata = validated_metadata.model_dump(mode="json", exclude_none=True) + if validated_metadata.knowledge_metadata is not None: + metadata["knowledge_metadata"] = validated_metadata.knowledge_metadata.storage_value() + validate_schema_selection(conn, metadata["knowledge_metadata"]) + provider_id = providers.get_or_create_instance(instance_value, conn=conn) + try: + with conn.transaction(): + row = conn.execute( + """ + INSERT INTO documents (provider_id, kind, code, name, aliases, source, metadata) + VALUES (%s::uuid, %s, %s, %s, %s, %s, %s) + RETURNING id + """, + ( + provider_id, + kind, + code, + name or code, + [a for a in (aliases or []) if a], + Jsonb(source), + Jsonb(metadata), + ), + ).fetchone() + except psycopg.errors.UniqueViolation as exc: + if exc.diag.constraint_name == "documents_provider_kind_code_key": + raise DuplicateDocumentCode( + f"Code '{code}' is already used in the {kind} catalog of {instance_value}" + ) from exc + raise + if with_workflow: + conn.execute( + """ + INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, + current_stage, error_message, stop_after_ocr, + started_by_user_id) + VALUES (%s::uuid, %s, %s, %s, %s, %s, %s::uuid) + """, + ( + str(row["id"]), + "ocr_only" if stop_after_ocr else "pipeline", + workflow_id, + stage, + error_message, + bool(stop_after_ocr), + user_uuid, + ), + ) + return str(row["id"]) + + +def reset_document_for_reupload( + doc_uuid: str, + *, + filename: str, + filepath: str, + fingerprint: str, + source_type: Optional[str] = None, + canonical_input_type: Optional[str] = None, + crop_subjects: Optional[list[dict]] = None, + knowledge_metadata: Optional[dict] = None, + expected_job_id: Optional[str] = None, + _conn=None, +) -> None: + """Point an existing document at a re-uploaded file. + + ``source`` is replaced, pages and chunks are cleared for the new run, and + the document is flagged ``reindex_required`` because its vector index is now + stale. Earlier workflows and artifacts stay as history. + + The fingerprint is the document's key in the vector index, so the old key + is remembered in ``metadata.superseded_doc_ids`` for every index (dev/prod) + that holds it. The next ingest/promote and disable/purge delete those points; + without this they would stay searchable with nothing pointing at them. + """ + source = DocumentSource.model_validate( + { + "fingerprint": fingerprint, + "original_filename": filename, + "storage_uri": filepath, + "source_type": source_type, + "canonical_input_type": canonical_input_type, + } + ).model_dump(exclude_none=True) + crop_patch = ( + { + "crop_subjects": DocumentMetadata(crop_subjects=crop_subjects).model_dump( + exclude_none=True + )["crop_subjects"] + } + if crop_subjects is not None + else {} + ) + if knowledge_metadata is not None: + crop_patch["knowledge_metadata"] = DocumentMetadata( + knowledge_metadata=knowledge_metadata + ).knowledge_metadata.storage_value() + with connection.get_connection() if _conn is None else nullcontext(_conn) as conn: + validate_schema_selection(conn, crop_patch.get("knowledge_metadata")) + current = conn.execute( + "SELECT source->>'fingerprint' AS fingerprint, metadata, is_disabled FROM documents " + "WHERE id = %s::uuid FOR UPDATE", + (doc_uuid,), + ).fetchone() + if current is None: + raise LookupError(f"Document not found: {doc_uuid}") + if expected_job_id: + job = conn.execute( + "SELECT id::text, status FROM document_current_workflow WHERE document_id = %s::uuid", + (doc_uuid,), + ).fetchone() + if ( + current["is_disabled"] + or not job + or job["id"] != expected_job_id + or job["status"] + not in { + "completed", + "failed", + "cancelled", + } + ): + raise DocumentReplacementConflict( + "This document changed before the replacement could start" + ) + meta = current["metadata"] or {} + superseded = {k: list(v) for k, v in (meta.get("superseded_doc_ids") or {}).items()} + for env, entry in (meta.get("index_status") or {}).items(): + old_key = (entry or {}).get("doc_id") or current["fingerprint"] + current_keys = {fingerprint, projection_doc_id({"id": doc_uuid}, fingerprint)} + if old_key and old_key not in current_keys and old_key not in superseded.get(env, []): + superseded.setdefault(env, []).append(old_key) + conn.execute("DELETE FROM pages WHERE document_id = %s::uuid", (doc_uuid,)) + conn.execute("DELETE FROM chunks WHERE document_id = %s::uuid", (doc_uuid,)) + conn.execute( + """ + UPDATE documents SET + source = %s::jsonb, + metadata = (metadata - 'chunking') + || jsonb_build_object('reindex_required', TRUE, 'reindex_reason', 'source_replaced', + 'superseded_doc_ids', %s::jsonb) + || %s::jsonb + WHERE id = %s::uuid + """, + (Jsonb(source), Jsonb(superseded), Jsonb(crop_patch), doc_uuid), + ) + + +def set_document_disabled(workflow_id: str, is_disabled: bool = True): + """Soft delete a document (filtered from all queries by default). + + Unlike hard delete, this preserves the document record for audit purposes. + Use X-Include-Disabled: true header in API to see disabled documents. + """ + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid: + conn.execute( + "UPDATE documents SET is_disabled = %s WHERE id = %s::uuid", + (bool(is_disabled), doc_uuid), + ) + + +def delete_document(workflow_id: str): + """Delete a document record and everything that cascades from it.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid: + conn.execute("DELETE FROM documents WHERE id = %s::uuid", (doc_uuid,)) + + +def purge_document(workflow_id: str, keep_audit: bool = True) -> dict: + """Hard delete every row belonging to a document. + + Unlike :func:`set_document_disabled` this is irreversible - there is + nothing left for ``POST /documents/{id}/restore`` to bring back. + + Audit rows are kept by default: they are the record that the deletion + happened, not part of the document itself. Their ``document_id`` becomes + NULL; the ``workflow_id`` and ``instance`` snapshots stay. + + Returns a per-table count of deleted rows. + """ + deleted: dict[str, int] = {} + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return deleted + root = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + + def _count(sql: str) -> int: + return int(conn.execute(sql, (doc_uuid,)).fetchone()["n"]) + + counts = { + "chunks": _count("SELECT count(*) AS n FROM chunks WHERE document_id = %s::uuid"), + "pages": _count("SELECT count(*) AS n FROM pages WHERE document_id = %s::uuid"), + "document_artifacts": _count( + "SELECT count(*) AS n FROM workflow_artifacts a JOIN workflows w ON w.id = a.workflow_id " + "WHERE w.document_id = %s::uuid" + ), + "document_jobs": _count( + "SELECT count(*) AS n FROM workflows WHERE document_id = %s::uuid" + ), + } + if not keep_audit: + row = conn.execute( + "DELETE FROM audit_logs WHERE document_id = %s::uuid OR workflow_id = %s", + (doc_uuid, root), + ) + if row.rowcount > 0: + deleted["audit_logs"] = row.rowcount + conn.execute("DELETE FROM documents WHERE id = %s::uuid", (doc_uuid,)) + for table, count in counts.items(): + if count > 0: + deleted[table] = count + deleted["documents"] = 1 + return deleted + + +def update_document_fields(workflow_id: str, *, _conn=None, **updates: object) -> Optional[dict]: + """Update selected document fields and return the document. + + Flat names are mapped onto the tables: ``stage``/``error_message`` act on the + current workflow, ``code``/``name``/``aliases`` on the document columns, + a document's kind is fixed by its catalog at upload, and flags, + period and uploader go into ``metadata``. Names derived from other rows (counts, artifact and job + pointers, completion timestamps) are accepted and ignored. + """ + with connection.get_connection() if _conn is None else nullcontext(_conn) as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return None + if not updates: + return document_reads._fetch_document(conn, doc_uuid) + + doc_sets: list[str] = [] + doc_params: list = [] + source_patch: dict[str, Any] = {} + meta_patch: dict[str, Any] = {} + meta_remove: list[str] = [] + wf_sets: list[str] = [] + wf_params: list = [] + + for key, value in updates.items(): + if key == "stage": + wf_sets.append("current_stage = %s") + wf_params.append(value) + elif key == "error_message": + wf_sets.append("error_message = %s") + wf_params.append(value) + elif key == "source_type": + source_patch["source_type"] = value + elif key == "canonical_input_type": + source_patch["canonical_input_type"] = value + elif key == "filepath": + source_patch["storage_uri"] = value + elif key in ("filename", "source_filename"): + if key == "filename" or "filename" not in updates: + source_patch["original_filename"] = value + elif key in ("document_id", "canonical_document_id", "source_file_fingerprint"): + if isinstance(value, str) and common._FINGERPRINT_RE.match(value): + source_patch["fingerprint"] = value + elif key == "name": + if value: + doc_sets.append("name = %s") + doc_params.append(value) + elif key == "code": + if value: + doc_sets.append("code = %s") + doc_params.append(value) + elif key == "aliases": + doc_sets.append("aliases = %s") + doc_params.append([a for a in (value or []) if a]) + elif key in ("reindex_required", "catalog_visible", "network_visible"): + meta_patch[key] = bool(value) + elif key == "reindex_reason": + if value: + meta_patch["reindex_reason"] = value + else: + meta_remove.append("reindex_reason") + elif key == "stop_after_ocr": + conn.execute( + "UPDATE workflows SET stop_after_ocr = %s WHERE id = (" + "SELECT id FROM workflows WHERE document_id = %s::uuid " + "ORDER BY (workflow_type = 'legacy_index_import'), (workflow_type IN ('pipeline', 'ocr_only')) DESC, started_at ASC, id ASC LIMIT 1)", + (bool(value), doc_uuid), + ) + elif key in ("valid_from", "valid_to"): + meta_patch[key] = value.isoformat() if isinstance(value, date) else value + elif key == "uploaded_by_user_id" and value: + has_uploader = conn.execute( + "SELECT metadata ? 'uploaded_by_user_id' AS has FROM documents WHERE id = %s::uuid", + (doc_uuid,), + ).fetchone()["has"] + if not has_uploader: + meta_patch["uploaded_by_user_id"] = providers.upsert_user( + str(value), + updates.get("uploaded_by_username"), + updates.get("uploaded_by_email"), + updates.get("uploaded_by_roles"), + conn=conn, + ) + + if source_patch: + doc_sets.append("source = source || %s::jsonb") + doc_params.append(Jsonb({k: v for k, v in source_patch.items() if v is not None})) + if meta_patch or meta_remove: + expr = "metadata" + for removed in meta_remove: + expr = f"({expr} - '{removed}')" + if meta_patch: + expr = f"{expr} || %s::jsonb" + doc_params.append(Jsonb(meta_patch)) + doc_sets.append(f"metadata = {expr}") + + if wf_sets: + conn.execute( + f"UPDATE workflows SET {', '.join(wf_sets)} WHERE id = (" + "SELECT id FROM document_current_workflow WHERE document_id = %s::uuid)", + (*wf_params, doc_uuid), + ) + if doc_sets: + try: + with conn.transaction(): + conn.execute( + f"UPDATE documents SET {', '.join(doc_sets)} WHERE id = %s::uuid", + (*doc_params, doc_uuid), + ) + except psycopg.errors.UniqueViolation as exc: + if exc.diag.constraint_name == "documents_provider_kind_code_key": + raise DuplicateDocumentCode( + "Another document in the target catalog already uses this code" + ) from exc + raise + elif wf_sets: + conn.execute("UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,)) + return document_reads._fetch_document(conn, doc_uuid) + + +def mark_document_reindex_required( + workflow_id: str, + required: bool = True, + reason: Optional[str] = None, +) -> Optional[dict]: + return update_document_fields( + workflow_id, + reindex_required=bool(required), + reindex_reason=reason if required else None, + ) + + +def set_document_validity( + workflow_id: str, + valid_from: str, + valid_to: str, +) -> Optional[dict]: + """Record the period this document's chunks are searchable in. Both ends + are `YYYY-MM-DD`; validation belongs to the caller + (`pipeline/document_validity.parse_period`), not to this SQL layer.""" + return update_document_fields(workflow_id, valid_from=valid_from, valid_to=valid_to) + + +def mark_prod_ready_requested( + workflow_id: str, + user_id: Optional[str] = None, + username: Optional[str] = None, +) -> Optional[dict]: + """Flag that a state_admin has asked a super_admin to review this document + for prod promotion. Purely informational - does not fire the approve_prod + signal itself; the super_admin still makes that call separately. + + Recorded as a ``promote_prod`` workflow in ``waiting_review``.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return None + root = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + user_uuid = ( + providers.upsert_user(user_id, username, None, None, conn=conn) if user_id else None + ) + conn.execute( + """ + UPDATE workflows SET status = 'running' + WHERE document_id = %s::uuid AND workflow_type = 'promote_prod' AND status = 'waiting_review' + """, + (doc_uuid,), + ) + conn.execute( + """ + INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, status, + current_stage, started_by_user_id) + VALUES (%s::uuid, 'promote_prod', %s, 'waiting_review', 'approval_for_prod', %s::uuid) + """, + (doc_uuid, f"{root}-prod-request", user_uuid), + ) + return document_reads._fetch_document(conn, doc_uuid) + + +def clear_prod_ready_requested(workflow_id: str) -> Optional[dict]: + """Reset the request flag once prod approval actually happens (or the + document leaves the approval_for_prod gate for any other reason).""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return None + conn.execute( + """ + UPDATE workflows SET status = 'running' + WHERE document_id = %s::uuid AND workflow_type = 'promote_prod' AND status = 'waiting_review' + """, + (doc_uuid,), + ) + return document_reads._fetch_document(conn, doc_uuid) diff --git a/pipeline/db/index_state.py b/pipeline/db/index_state.py new file mode 100644 index 0000000..6da5582 --- /dev/null +++ b/pipeline/db/index_state.py @@ -0,0 +1,341 @@ +"""Persist per-document index state and enforce search visibility.""" + +import json +import os +from typing import Optional + +from psycopg.types.json import Jsonb + +from pipeline.documents.models import ( + IndexState, +) +from pipeline.search.index_identity import projection_owner + +from . import chunks as chunk_queries +from . import common, connection, document_reads, jobs + +PROD_INDEX_KEY = "prod" + + +DEV_INDEX_KEY = "dev" + + +def _prod_collection_names() -> set[str]: + names = { + os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME"), + os.environ.get("PROD_SCHEME_VECTOR_DB_COLLECTION_NAME"), + } + return {n.strip() for n in names if n and n.strip()} + + +def index_key_for(index_name: str) -> str: + return PROD_INDEX_KEY if index_name in _prod_collection_names() else DEV_INDEX_KEY + + +def doc_id_used_by_other_document(doc_id: str, exclude_doc_uuid: Optional[str], env: str) -> bool: + """True when another *live* document is indexed in ``env`` (dev/prod) under ``doc_id``. + + The index key is the file fingerprint, so two documents holding the same file + share points in the vector index. Deleting a key another document still needs + would remove that document's vectors too. Only a document that is not + disabled and whose own ``index_status[env]`` holds this key counts: one that + was merely uploaded, or is disabled, has nothing in the index to protect. + """ + with connection.get_connection() as conn: + row = conn.execute( + """ + SELECT 1 FROM documents d + WHERE (%s::uuid IS NULL OR d.id <> %s::uuid) + AND NOT d.is_disabled + AND d.metadata->'index_status'->%s::text->>'doc_id' = %s + LIMIT 1 + """, + (exclude_doc_uuid, exclude_doc_uuid, env, doc_id), + ).fetchone() + return row is not None + + +def clear_superseded_doc_ids(workflow_id: str, env: str, doc_ids: list[str]) -> None: + """Forget superseded index keys once their points are deleted from ``env`` (dev/prod).""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return + row = conn.execute( + "SELECT metadata->'superseded_doc_ids' AS sup FROM documents WHERE id = %s::uuid FOR UPDATE", + (doc_uuid,), + ).fetchone() + superseded = {k: list(v) for k, v in (row["sup"] or {}).items()} + remaining = [i for i in superseded.get(env, []) if i not in set(doc_ids)] + if remaining: + superseded[env] = remaining + else: + superseded.pop(env, None) + conn.execute( + "UPDATE documents SET metadata = metadata || jsonb_build_object('superseded_doc_ids', %s::jsonb) " + "WHERE id = %s::uuid", + (Jsonb(superseded), doc_uuid), + ) + + +def visible_document_hits(hits: list[dict]) -> list[dict]: + """Validate document-owned index-only hits against current Postgres state.""" + owners = [projection_owner(str(hit.get("doc_id") or "")) for hit in hits] + document_ids = list(dict.fromkeys(owner for owner in owners if owner)) + if not document_ids: + return hits + with connection.get_connection() as conn: + documents = conn.execute( + "SELECT d.id::text, i.name AS instance FROM documents d JOIN instances i ON i.id = d.provider_id " + "WHERE d.id = ANY(%s::uuid[]) AND NOT d.is_disabled", + (document_ids,), + ).fetchall() + exclusions = conn.execute( + "SELECT document_id::text, chunk_number FROM chunks " + "WHERE document_id = ANY(%s::uuid[]) AND is_excluded", + (document_ids,), + ).fetchall() + providers = {row["id"]: row["instance"].lower() for row in documents} + excluded = {(row["document_id"], row["chunk_number"]) for row in exclusions} + return [ + hit + for owner, hit in zip(owners, hits) + if owner is None + or ( + owner in providers + and providers[owner] == hit.get("instance") + and (owner, hit.get("chunk_num")) not in excluded + ) + ] + + +def visible_release_hits(hits: list[dict]) -> list[dict]: + """Hide staged, superseded and withdrawn revision points from retrieval.""" + revision_ids = { + hit.get("revision_id") for hit in hits if common._is_uuid(str(hit.get("revision_id"))) + } + if not revision_ids and not any(hit.get("release_network_id") for hit in hits): + return hits + with connection.get_connection() as conn: + rows = conn.execute( + "SELECT n.revision_id::text, n.network_id FROM network_releases n " + "JOIN documents d ON d.id = n.document_id " + "WHERE n.revision_id = ANY(%s::uuid[]) AND n.withdrawn_at IS NULL AND NOT d.is_disabled " + "AND d.metadata->>'network_visible' IS DISTINCT FROM 'false'", + (list(revision_ids),), + ).fetchall() + active = {(row["revision_id"], row["network_id"]) for row in rows} + return [ + hit + for hit in hits + if (not hit.get("revision_id") and not hit.get("release_network_id")) + or (hit.get("revision_id"), hit.get("release_network_id")) in active + ] + + +def inactive_network_revision_ids(network_id: str, collection: str) -> list[str]: + """Read staged and retired revisions to exclude before vector ranking.""" + with connection.get_connection() as conn: + rows = conn.execute( + "SELECT r.id::text FROM network_release_staging s " + "JOIN document_revisions r ON r.id = s.revision_id JOIN documents d ON d.id = r.document_id " + "LEFT JOIN network_releases n ON n.document_id = d.id AND n.network_id = s.network_id " + "WHERE s.collection = %s AND s.network_id = %s " + "AND (n.revision_id IS DISTINCT FROM r.id OR n.withdrawn_at IS NOT NULL " + "OR d.is_disabled OR d.metadata->>'network_visible' = 'false') " + "LIMIT 1001", + (collection, network_id), + ).fetchall() + if len(rows) > 1000: + raise ValueError("This release scope exceeds the complete pre-ranking filter limit") + return [row["id"] for row in rows] + + +# ============================================================================= +# Vector index state (documents.metadata.index_status) +# ============================================================================= + + +def _index_entry_to_legacy(workflow_id: str, key: str, entry: dict) -> dict: + details = entry.get("details") or {} + return { + "workflow_id": workflow_id, + "index_name": entry.get("index_name") or key, + "doc_id": entry.get("doc_id"), + "chunk_count_indexed": entry.get("chunk_count_indexed") or 0, + "last_indexed_at": entry.get("last_indexed_at"), + "last_verified_at": entry.get("last_verified_at"), + "schema_version": entry.get("schema_version"), + "status": entry.get("status") or "unknown", + "details_json": json.dumps(details) if details else None, + } + + +def upsert_document_index_status( + workflow_id: str, + index_name: str, + doc_id: Optional[str] = None, + chunk_count_indexed: Optional[int] = None, + last_indexed_at: Optional[str] = None, + last_verified_at: Optional[str] = None, + schema_version: Optional[str] = None, + status: str = "unknown", + details: Optional[dict] = None, +) -> dict: + """Merge one index's state into ``documents.metadata.index_status``. + + The key is ``prod`` for the PROD collection names and ``dev`` otherwise; + the collection name is kept inside the entry. ``None`` arguments leave the + stored value alone, except ``status`` which always replaces it. + """ + key = index_key_for(index_name) + with connection.get_connection() as conn: + doc_uuid = document_reads._require_document_uuid(conn, workflow_id) + row = conn.execute( + "SELECT metadata->'index_status'->%s::text AS entry, " + "metadata->'superseded_doc_ids' AS superseded FROM documents WHERE id = %s::uuid FOR UPDATE", + (key, doc_uuid), + ).fetchone() + existing = dict(row["entry"] or {}) + old_key = existing.get("doc_id") + if old_key and doc_id and old_key != doc_id: + superseded = {k: list(v) for k, v in (row["superseded"] or {}).items()} + if old_key not in superseded.get(key, []): + superseded.setdefault(key, []).append(old_key) + conn.execute( + "UPDATE documents SET metadata = metadata || " + "jsonb_build_object('superseded_doc_ids', %s::jsonb) WHERE id = %s::uuid", + (Jsonb(superseded), doc_uuid), + ) + incoming = { + "index_name": index_name, + "doc_id": doc_id, + "chunk_count_indexed": chunk_count_indexed, + "last_indexed_at": last_indexed_at, + "last_verified_at": last_verified_at, + "schema_version": schema_version, + "details": details or None, + } + merged = { + **existing, + **{k: v for k, v in incoming.items() if v is not None}, + "status": status, + } + validated = IndexState.model_validate(merged).model_dump() + conn.execute( + "UPDATE documents SET metadata = jsonb_set(" + "metadata || jsonb_build_object('index_status', COALESCE(metadata->'index_status', '{}'::jsonb)), " + "ARRAY['index_status', %s::text], %s::jsonb) " + "WHERE id = %s::uuid", + (key, Jsonb(validated), doc_uuid), + ) + root = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + return _index_entry_to_legacy(root, key, validated) + + +def list_document_index_status(workflow_id: str) -> list[dict]: + doc = document_reads.get_document(workflow_id) + if not doc: + return [] + entries = [ + _index_entry_to_legacy(doc["workflow_id"], key, entry) + for key, entry in doc["index_status"].items() + ] + entries.sort( + key=lambda e: (e["last_verified_at"] or "", e["last_indexed_at"] or ""), reverse=True + ) + return entries + + +def get_document_index_status(workflow_id: str, index_name: str) -> Optional[dict]: + for entry in list_document_index_status(workflow_id): + if entry["index_name"] == index_name: + return entry + return None + + +def list_index_summaries( + include_disabled: bool = False, instances: list[str] | None = None +) -> list[dict]: + clauses, params = document_reads._document_filters(include_disabled, instances) + where = f"AND {' AND '.join(clauses)}" if clauses else "" + with connection.get_connection() as conn: + rows = conn.execute( + f""" + SELECT + COALESCE(e.value->>'index_name', e.key) AS index_name, + COUNT(*) AS documents, + SUM(COALESCE((e.value->>'chunk_count_indexed')::int, 0)) AS indexed_chunks, + SUM(CASE WHEN COALESCE((d.metadata->>'reindex_required')::boolean, FALSE) + THEN 1 ELSE 0 END) AS stale_documents, + MAX(e.value->>'last_indexed_at') AS last_indexed_at, + array_agg(DISTINCT COALESCE(e.value->>'status', 'unknown')) AS statuses + FROM documents d + JOIN instances g ON g.id = d.provider_id + CROSS JOIN LATERAL jsonb_each(COALESCE(d.metadata->'index_status', '{{}}'::jsonb)) e + WHERE TRUE {where} + GROUP BY COALESCE(e.value->>'index_name', e.key) + ORDER BY index_name + """, + params, + ).fetchall() + result = [] + for row in rows: + item = common._clean_row(row) + item["indexed_chunks"] = int(item["indexed_chunks"] or 0) + item["stale_documents"] = int(item["stale_documents"] or 0) + item["statuses"] = sorted(s for s in row["statuses"] if s) + result.append(item) + return result + + +def find_document_by_doc_identifier(identifier: str) -> Optional[dict]: + """Resolve a vector-store/chat doc identifier to a document. + + Matches the document UUID, a Temporal workflow id, the fingerprint, the + filename, an index ``doc_id``, or the legacy ``md5(fingerprint)`` doc id. + """ + if not identifier: + return None + with connection.get_connection() as conn: + return document_reads._fetch_document( + conn, document_reads._resolve_document_uuid(conn, identifier) + ) + + +def resolve_chunk_provenance( + *, + doc_id: Optional[str] = None, + chunk_num: Optional[int] = None, +) -> Optional[dict]: + """Resolve workflow + chunk metadata for provenance links.""" + if not doc_id or chunk_num is None: + return None + + doc = find_document_by_doc_identifier(doc_id) + if not doc: + return None + + workflow_id = doc["workflow_id"] + chunk = chunk_queries.get_chunk(workflow_id, int(chunk_num)) + if not chunk: + return None + + text = chunk.get("edited_text") or chunk.get("original_text") or "" + excerpt = text[:320] + ("..." if len(text) > 320 else "") + page_start = int(chunk.get("page_start") or 1) + page_end = int(chunk.get("page_end") or page_start) + doc_name = doc.get("name") or doc.get("filename") or workflow_id + + return { + "workflow_id": workflow_id, + "doc_id": doc.get("document_id"), + "filename": workflow_id, + "doc_name": doc_name, + "chunk_num": int(chunk_num), + "page_start": page_start, + "page_end": page_end, + "page_range": f"{page_start}-{page_end}" if page_end != page_start else str(page_start), + "section": "", + "excerpt": excerpt, + } diff --git a/pipeline/db/jobs.py b/pipeline/db/jobs.py new file mode 100644 index 0000000..fe6c88d --- /dev/null +++ b/pipeline/db/jobs.py @@ -0,0 +1,335 @@ +"""Record processing runs and query the operator queues.""" + +import json +from contextlib import nullcontext +from typing import Optional + +from psycopg.types.json import Jsonb + +from . import common, connection, document_reads, providers + +# ============================================================================= +# Workflows (exposed as "jobs") +# ============================================================================= + +_JOB_SELECT = """ + SELECT w.id, w.document_id, w.workflow_type, w.temporal_workflow_id, w.status, + w.current_stage, w.config, w.error_message, w.started_at, w.completed_at, + root.temporal_workflow_id AS doc_workflow_id + FROM workflows w + LEFT JOIN LATERAL ( + SELECT r.temporal_workflow_id FROM workflows r + WHERE r.document_id = w.document_id + ORDER BY (r.workflow_type = 'legacy_index_import'), (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC + LIMIT 1 + ) root ON TRUE +""" + + +def _job_from_row(row: Optional[dict]) -> Optional[dict]: + if row is None: + return None + config = row["config"] or {} + job = { + "id": row["id"], + "workflow_id": row["doc_workflow_id"], + "job_type": row["workflow_type"], + "temporal_workflow_id": row["temporal_workflow_id"], + "temporal_run_id": config.get("temporal_run_id"), + "status": row["status"], + "current_stage": row["current_stage"], + "started_at": row["started_at"], + "completed_at": row["completed_at"], + "error_message": row["error_message"], + "config": config, + "config_json": json.dumps(config) if config else None, + } + return {k: common._clean(v) for k, v in job.items()} + + +def root_workflow_id(conn, doc_uuid: str) -> Optional[str]: + row = conn.execute( + """ + SELECT temporal_workflow_id FROM workflows WHERE document_id = %s::uuid + ORDER BY (workflow_type = 'legacy_index_import'), (workflow_type IN ('pipeline', 'ocr_only')) DESC, started_at ASC, id ASC + LIMIT 1 + """, + (doc_uuid,), + ).fetchone() + return row["temporal_workflow_id"] if row else None + + +def create_document_job( + workflow_id: str, + job_type: str, + temporal_workflow_id: Optional[str] = None, + temporal_run_id: Optional[str] = None, + status: str = "running", + current_stage: Optional[str] = None, + config: Optional[dict] = None, + actor: Optional[dict] = None, + *, + _conn=None, +) -> str: + """Record a workflow run for a document; returns the workflow UUID. + + A ``promote_prod`` run adopts the document's pending request row (see + ``mark_prod_ready_requested``) instead of adding a second one. + """ + cfg = dict(config or {}) + if temporal_run_id: + cfg["temporal_run_id"] = temporal_run_id + with connection.get_connection() if _conn is None else nullcontext(_conn) as conn: + doc_uuid = document_reads._require_document_uuid(conn, workflow_id) + root = root_workflow_id(conn, doc_uuid) or workflow_id + user_uuid = None + if actor and actor.get("user_id"): + user_uuid = providers.upsert_user( + actor["user_id"], + actor.get("username"), + actor.get("email"), + actor.get("roles"), + conn=conn, + ) + if job_type == "promote_prod": + adopted = conn.execute( + """ + UPDATE workflows SET temporal_workflow_id = %s, status = %s, + current_stage = COALESCE(%s, current_stage), + config = config || %s::jsonb + WHERE id = ( + SELECT id FROM workflows + WHERE document_id = %s::uuid AND workflow_type = 'promote_prod' + AND status = 'waiting_review' + ORDER BY started_at DESC LIMIT 1) + RETURNING id + """, + (temporal_workflow_id or root, status, current_stage, Jsonb(cfg), doc_uuid), + ).fetchone() + if adopted: + conn.execute( + "UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,) + ) + return str(adopted["id"]) + row = conn.execute( + """ + INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, status, + current_stage, stop_after_ocr, config, started_by_user_id) + VALUES (%s::uuid, %s, %s, %s, + COALESCE(%s, 'registered'), %s, %s, %s::uuid) + RETURNING id + """, + ( + doc_uuid, + job_type, + temporal_workflow_id or root, + status, + current_stage, + bool(cfg.get("stop_after_ocr")), + Jsonb(cfg), + user_uuid, + ), + ).fetchone() + conn.execute("UPDATE documents SET updated_at = now() WHERE id = %s::uuid", (doc_uuid,)) + return str(row["id"]) + + +def update_document_job(job_id: str, **updates: object) -> Optional[dict]: + if not updates: + return get_document_job(job_id) + if not common._is_uuid(str(job_id)): + return None + allowed_fields = { + "status", + "current_stage", + "completed_at", + "error_message", + "temporal_run_id", + "config_json", + } + set_clauses = [] + values: list[object] = [] + for key, value in updates.items(): + if key not in allowed_fields: + continue + if key == "status": + set_clauses.append("status = %s") + elif key == "current_stage": + set_clauses.append("current_stage = %s") + elif key == "completed_at": + set_clauses.append("completed_at = %s::timestamptz") + elif key == "temporal_run_id": + set_clauses.append("config = config || jsonb_build_object('temporal_run_id', %s::text)") + elif key == "config_json": + if isinstance(value, str): + value = json.loads(value) + set_clauses.append("config = config || %s::jsonb") + value = Jsonb(value or {}) + else: + set_clauses.append(f"{key} = %s") + values.append(value) + if not set_clauses: + return get_document_job(job_id) + values.append(str(job_id)) + with connection.get_connection() as conn: + conn.execute(f"UPDATE workflows SET {', '.join(set_clauses)} WHERE id = %s::uuid", values) + return get_document_job(job_id) + + +def mark_upload_started(job_id: str) -> None: + """Acknowledge Temporal start without overwriting a stage already advanced by the worker.""" + with connection.get_connection() as conn: + conn.execute( + "UPDATE workflows SET status = 'running' WHERE id = %s::uuid AND status = 'start_pending'", + (job_id,), + ) + + +def get_document_job(job_id: str) -> Optional[dict]: + if not common._is_uuid(str(job_id)): + return None + with connection.get_connection() as conn: + return _job_from_row( + conn.execute(_JOB_SELECT + " WHERE w.id = %s::uuid", (str(job_id),)).fetchone() + ) + + +def get_latest_document_job(workflow_id: str) -> Optional[dict]: + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return None + return _job_from_row( + conn.execute( + _JOB_SELECT + + " WHERE w.document_id = %s::uuid ORDER BY w.started_at DESC, w.id DESC LIMIT 1", + (doc_uuid,), + ).fetchone() + ) + + +def list_document_jobs(workflow_id: str, limit: int = 50) -> list[dict]: + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return [] + rows = conn.execute( + _JOB_SELECT + + " WHERE w.document_id = %s::uuid ORDER BY w.started_at DESC, w.id DESC LIMIT %s", + (doc_uuid, limit), + ).fetchall() + return [_job_from_row(r) for r in rows] + + +def list_operations_queue( + limit: int = 100, + offset: int = 0, + include_disabled: bool = False, + instances: Optional[list[str]] = None, +) -> tuple[list[dict], int]: + """List documents needing action, optionally scoped to tenant instances. + + ``instances``: + - ``None`` - unrestricted (super admin / bypass) + - ``[]`` - restricted user with no states -> empty result + - non-empty - only those document.instance values + """ + if instances is not None and not common._normalize_instances(instances): + return [], 0 + clauses, params = document_reads._document_filters(include_disabled, instances) + clauses.append( + "(cw.current_stage IN ('ocr_review', 'translation_review', 'chunk_review', " + "'ready_for_ingestion', 'approval_for_prod', 'failed') " + "OR COALESCE((d.metadata->>'reindex_required')::boolean, FALSE) " + "OR cw.status = 'running')" + ) + where = f"WHERE {' AND '.join(clauses)}" + from_sql = """ + FROM documents d + JOIN instances g ON g.id = d.provider_id + JOIN document_current_workflow cw ON cw.document_id = d.id + """ + with connection.get_connection() as conn: + total_row = conn.execute(f"SELECT COUNT(*) AS count {from_sql} {where}", params).fetchone() + rows = conn.execute( + f""" + SELECT + d.id AS id, + (SELECT r.temporal_workflow_id FROM workflows r WHERE r.document_id = d.id + ORDER BY (r.workflow_type = 'legacy_index_import'), (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC + LIMIT 1) AS workflow_id, + d.source->>'original_filename' AS filename, + cw.current_stage AS stage, + lower(g.name) AS "instance", + cw.error_message, + COALESCE((d.metadata->>'reindex_required')::boolean, FALSE) AS reindex_required, + d.metadata->>'reindex_reason' AS reindex_reason, + cw.id AS job_id, + cw.workflow_type AS job_type, + cw.status AS job_status, + cw.started_at AS started_at + {from_sql} + {where} + ORDER BY cw.started_at DESC, d.id + LIMIT %s OFFSET %s + """, + (*params, limit, offset), + ).fetchall() + return [common._clean_row(r) for r in rows], int(total_row["count"] if total_row else 0) + + +def list_runs( + limit: int = 100, + offset: int = 0, + status: Optional[str] = None, + instances: Optional[list[str]] = None, +) -> list[dict]: + """List workflows with document title fields for the Runs UI. + + Args: + instances: If provided, only return runs whose document instance is in + this list. Empty list returns no rows. None = unrestricted. + """ + where_parts: list[str] = [] + params: list = [] + if status: + where_parts.append("w.status = %s") + params.append(status) + if instances is not None: + normalized = common._normalize_instances(instances) + if not normalized: + return [] + where_parts.append("lower(g.name) = ANY(%s)") + params.append(normalized) + where_sql = f"WHERE {' AND '.join(where_parts)}" if where_parts else "" + with connection.get_connection() as conn: + rows = conn.execute( + f""" + SELECT w.id, w.document_id, w.workflow_type, w.temporal_workflow_id, w.status, + w.current_stage, w.config, w.error_message, w.started_at, w.completed_at, + (SELECT r.temporal_workflow_id FROM workflows r WHERE r.document_id = w.document_id + ORDER BY (r.workflow_type = 'legacy_index_import'), (r.workflow_type IN ('pipeline', 'ocr_only')) DESC, r.started_at ASC, r.id ASC + LIMIT 1) AS doc_workflow_id, + d.source->>'original_filename' AS filename, + d.name AS name, + d.source->>'original_filename' AS source_filename, + cw.current_stage AS document_stage, + lower(g.name) AS "instance" + FROM workflows w + JOIN documents d ON d.id = w.document_id + JOIN instances g ON g.id = d.provider_id + LEFT JOIN document_current_workflow cw ON cw.document_id = d.id + {where_sql} + ORDER BY w.started_at DESC, w.id DESC + LIMIT %s OFFSET %s + """, + (*params, limit, offset), + ).fetchall() + result = [] + for row in rows: + job = _job_from_row(row) + job["document_uuid"] = str(row["document_id"]) + for extra in ("filename", "name", "source_filename", "document_stage", "instance"): + job[extra] = row[extra] + result.append(job) + return result diff --git a/pipeline/db/pages.py b/pipeline/db/pages.py new file mode 100644 index 0000000..eb3904a --- /dev/null +++ b/pipeline/db/pages.py @@ -0,0 +1,322 @@ +"""Persist extracted pages and their translations.""" + +from typing import Any, Optional + +from psycopg.types.json import Jsonb + +from pipeline.documents.models import ( + PageMetadata, + PageTranslation, +) +from pipeline.metadata.localization import normalize_language_tag + +from . import chunks as chunk_queries +from . import common, connection, document_reads, jobs + +# ============================================================================= +# Page Functions +# ============================================================================= + +_PAGE_SELECT = """ + SELECT p.id, p.document_id, p.page_number, p.original_markdown, p.edited_markdown, + p.translations, p.metadata, p.created_at, p.updated_at + FROM pages p +""" + + +def _primary_translation(translations: dict) -> tuple[Optional[str], dict]: + if not translations: + return None, {} + lang = max( + translations, + key=lambda k: (translations[k].get("translated_at") or "", k == "en"), + ) + return lang, translations[lang] + + +def _page_from_row(row: Optional[dict], workflow_id: str) -> Optional[dict]: + if row is None: + return None + meta = row["metadata"] or {} + translations = row["translations"] or {} + lang, tr = _primary_translation(translations) + page = { + "id": row["id"], + "workflow_id": workflow_id, + "page_number": row["page_number"], + "original_markdown": row["original_markdown"], + "edited_markdown": row["edited_markdown"], + "is_reviewed": bool(meta.get("is_reviewed")), + "reviewer_notes": meta.get("reviewer_notes"), + "detected_language": meta.get("detected_language"), + "translated_markdown": tr.get("translated_markdown"), + "edited_translation": tr.get("edited_translation"), + "translation_reviewed": bool(tr.get("is_reviewed")), + "translation_notes": tr.get("reviewer_notes"), + "translation_provider": tr.get("provider"), + "translation_model": tr.get("model"), + "translation_target_language": lang, + "translated_at": tr.get("translated_at"), + "translations": translations, + "metadata": meta, + } + return {k: common._clean(v) for k, v in page.items()} + + +_TRANSLATION_FLAT_KEYS = ( + "translated_markdown", + "edited_translation", + "translation_notes", + "translation_provider", + "translation_model", + "translated_at", +) + + +def _translation_from_flat(page: dict) -> Optional[tuple[str, dict]]: + if not any(page.get(k) not in (None, "") for k in _TRANSLATION_FLAT_KEYS) and not page.get( + "translation_reviewed" + ): + return None + lang = page.get("translation_target_language") or "en" + entry = PageTranslation( + translated_markdown=page.get("translated_markdown"), + edited_translation=page.get("edited_translation"), + is_reviewed=bool(page.get("translation_reviewed")), + reviewer_notes=page.get("translation_notes"), + provider=page.get("translation_provider"), + model=page.get("translation_model"), + translated_at=page.get("translated_at"), + ).model_dump() + return lang, entry + + +def save_pages(workflow_id: str, pages: list[dict]): + """ + Save all pages for a document (bulk upsert). + + Each call replaces the whole row, so the caller passes every field. Page + ``translations`` are kept as stored unless the page carries ``translations`` + (replaces them) or flat ``translated_*`` fields (merged in under + ``translation_target_language``); a page whose OCR text changed loses its + translations because they no longer describe it. + """ + with connection.get_connection() as conn: + doc_uuid = document_reads._require_document_uuid(conn, workflow_id) + existing = { + r["page_number"]: r + for r in conn.execute( + "SELECT page_number, original_markdown, translations FROM pages " + "WHERE document_id = %s::uuid", + (doc_uuid,), + ).fetchall() + } + for page in pages: + number = page.get("page_number") + current = existing.get(number) + meta = dict(page.get("metadata") or {}) + for flat_key in ("is_reviewed", "reviewer_notes", "detected_language"): + if flat_key in page: + meta[flat_key] = page[flat_key] + meta["is_reviewed"] = bool(meta.get("is_reviewed")) + page_meta = PageMetadata.model_validate( + {k: v for k, v in meta.items() if v is not None} + ).model_dump(exclude_none=True) + + translations = dict(current["translations"]) if current else {} + if current and current["original_markdown"] != page.get("original_markdown"): + translations = {} + if isinstance(page.get("translations"), dict): + translations = { + lang: PageTranslation.model_validate(entry).model_dump() + for lang, entry in page["translations"].items() + } + else: + flat = _translation_from_flat(page) + if flat: + translations[flat[0]] = flat[1] + + conn.execute( + """ + INSERT INTO pages (document_id, page_number, original_markdown, edited_markdown, + translations, metadata) + VALUES (%s::uuid, %s, %s, %s, %s, %s) + ON CONFLICT (document_id, page_number) DO UPDATE SET + original_markdown = EXCLUDED.original_markdown, + edited_markdown = EXCLUDED.edited_markdown, + translations = EXCLUDED.translations, + metadata = EXCLUDED.metadata + """, + ( + doc_uuid, + number, + page.get("original_markdown"), + page.get("edited_markdown"), + Jsonb(translations), + Jsonb(page_meta), + ), + ) + + +def persist_document_content(workflow_id: str, pages: list[dict], chunks: list[dict]): + """Backward-compatible helper to persist pages and chunks in one call.""" + save_pages(workflow_id, pages) + chunk_queries.save_chunks(workflow_id, chunks) + + +def get_pages(workflow_id: str) -> list[dict]: + """Get all pages for a document.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return [] + root = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + rows = conn.execute( + _PAGE_SELECT + " WHERE p.document_id = %s::uuid ORDER BY p.page_number", (doc_uuid,) + ).fetchall() + return [_page_from_row(r, root) for r in rows] + + +def count_translated_pages(workflow_id: str) -> int: + """Count pages with translation text without loading full page bodies.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return 0 + row = conn.execute( + """ + SELECT COUNT(*) AS n FROM pages p + WHERE p.document_id = %s::uuid + AND EXISTS ( + SELECT 1 FROM jsonb_each(p.translations) t + WHERE nullif(btrim(t.value->>'translated_markdown'), '') IS NOT NULL + OR nullif(btrim(t.value->>'edited_translation'), '') IS NOT NULL + ) + """, + (doc_uuid,), + ).fetchone() + return int(row["n"] if row else 0) + + +def get_saved_page_numbers(workflow_id: str) -> list[int]: + """Get saved page numbers for a document without loading full page content.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return [] + rows = conn.execute( + "SELECT page_number FROM pages WHERE document_id = %s::uuid ORDER BY page_number", + (doc_uuid,), + ).fetchall() + return [int(row["page_number"]) for row in rows] + + +def get_page(workflow_id: str, page_num: int) -> Optional[dict]: + """Get a specific page.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return None + root = jobs.root_workflow_id(conn, doc_uuid) or workflow_id + row = conn.execute( + _PAGE_SELECT + " WHERE p.document_id = %s::uuid AND p.page_number = %s", + (doc_uuid, page_num), + ).fetchone() + return _page_from_row(row, root) + + +class PageTranslationMissing(LookupError): + """The selected translation is no longer on this page.""" + + +def page_translation_key(translations: dict, language: str | None) -> str | None: + normalized = normalize_language_tag(language) + if normalized is None: + return None + matches = [key for key in translations if normalize_language_tag(key) == normalized] + return matches[0] if len(matches) == 1 else None + + +def update_page( + workflow_id: str, + page_num: int, + edited_markdown: Optional[str] = None, + is_reviewed: Optional[bool] = None, + reviewer_notes: Optional[str] = None, + edited_translation: Optional[str] = None, + translation_reviewed: Optional[bool] = None, + translation_notes: Optional[str] = None, + translation_language: Optional[str] = None, +) -> Optional[dict]: + """Update a page. Returns updated page or None if not found.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is None: + return None + existing = conn.execute( + "SELECT translations FROM pages WHERE document_id = %s::uuid AND page_number = %s FOR UPDATE", + (doc_uuid, page_num), + ).fetchone() + if not existing: + return None + + assignments: list[str] = [] + params: list = [] + if edited_markdown is not None: + assignments.append("edited_markdown = %s") + params.append(edited_markdown) + + meta_patch: dict[str, Any] = {} + if is_reviewed is not None: + meta_patch["is_reviewed"] = bool(is_reviewed) + if reviewer_notes is not None: + meta_patch["reviewer_notes"] = reviewer_notes + if meta_patch: + assignments.append("metadata = metadata || %s::jsonb") + params.append(Jsonb(meta_patch)) + + tr_patch: dict[str, Any] = {} + if edited_translation is not None: + tr_patch["edited_translation"] = edited_translation + if translation_reviewed is not None: + tr_patch["is_reviewed"] = bool(translation_reviewed) + if translation_notes is not None: + tr_patch["reviewer_notes"] = translation_notes + if tr_patch: + translations = existing["translations"] or {} + if translation_language is not None: + lang = page_translation_key(translations, translation_language) + if lang is None: + raise PageTranslationMissing(translation_language) + else: + lang = _primary_translation(translations)[0] or "en" + assignments.append( + "translations = jsonb_set(translations, ARRAY[%s::text], " + "COALESCE(translations->%s::text, '{}'::jsonb) || %s::jsonb, TRUE)" + ) + params.extend([lang, lang, Jsonb(tr_patch)]) + + if assignments: + conn.execute( + f"UPDATE pages SET {', '.join(assignments)} " + "WHERE document_id = %s::uuid AND page_number = %s", + (*params, doc_uuid, page_num), + ) + return get_page(workflow_id, page_num) + + +def reset_page(workflow_id: str, page_num: int) -> Optional[dict]: + """Reset a page to its original OCR markdown.""" + with connection.get_connection() as conn: + doc_uuid = document_reads._resolve_document_uuid(conn, workflow_id) + if doc_uuid is not None: + conn.execute( + """ + UPDATE pages SET + edited_markdown = NULL, + metadata = (metadata - 'reviewer_notes') || '{"is_reviewed": false}'::jsonb + WHERE document_id = %s::uuid AND page_number = %s + """, + (doc_uuid, page_num), + ) + return get_page(workflow_id, page_num) diff --git a/pipeline/db/providers.py b/pipeline/db/providers.py new file mode 100644 index 0000000..e4ae0a4 --- /dev/null +++ b/pipeline/db/providers.py @@ -0,0 +1,144 @@ +"""Store instances and attribution records for Keycloak identities.""" + +from typing import Any, Optional + +import psycopg + +from pipeline.auth.groups import provider_group_root +from pipeline.auth.tenancy import configured_portal_instance, portal_instance + +from . import catalog, common, connection + +# ============================================================================= +# Instances +# ============================================================================= + + +def _tenant_path(instance: str) -> str: + """Path for an instance, including a configured legacy portal projection.""" + return ( + "/global" + if instance == configured_portal_instance() + else f"/{provider_group_root()}/{instance.upper()}" + ) + + +def _tenant_code_from_path(path: str) -> str: + return portal_instance() if path == "/global" else path.rstrip("/").rsplit("/", 1)[-1].lower() + + +def upsert_instance(path: str, keycloak_group_id: Optional[str] = None, conn=None) -> str: + """Create or refresh the ``instances`` row for a Keycloak group path; returns its UUID. + + Rows made by the SQLite move have no Keycloak id yet; the first sync that + sees the path fills it in. The instance's catalogs (one per supported type) are + ensured in the same transaction, so an instance never exists without them. + """ + if conn is None: + with connection.get_connection() as own: + return upsert_instance(path, keycloak_group_id, conn=own) + row = conn.execute( + """ + INSERT INTO instances (keycloak_group_id, name, path) VALUES (%s, %s, %s) + ON CONFLICT (path) DO UPDATE SET + keycloak_group_id = COALESCE(EXCLUDED.keycloak_group_id, instances.keycloak_group_id) + RETURNING id + """, + (keycloak_group_id, _tenant_code_from_path(path), path), + ).fetchone() + instance_id = str(row["id"]) + catalog.ensure_instance_catalogs(instance_id, conn=conn) + return instance_id + + +def list_provider_instances(allowed_codes: list[str] | None) -> list[dict]: + """List Keycloak-backed provider identities visible to the caller.""" + with connection.get_connection() as conn: + rows = conn.execute( + """ + SELECT id, name AS provider, path AS "group" + FROM instances + WHERE keycloak_group_id IS NOT NULL AND split_part(path, '/', 2) = %s + AND (%s::text[] IS NULL OR name = ANY(%s::text[])) + ORDER BY name + """, + (provider_group_root(), allowed_codes, allowed_codes), + ).fetchall() + return [common._clean_row(row) for row in rows] + + +def get_or_create_instance(instance: Optional[str], conn=None) -> str: + """UUID of the instance for a tenant code (``bv``, ``mh``, ...), created when missing.""" + code = (instance or common._default_instance()).strip().lower() or common._default_instance() + return upsert_instance(_tenant_path(code), conn=conn) + + +# ============================================================================= +# Users +# ============================================================================= + + +def upsert_user( + keycloak_user_id: str, + username: Optional[str] = None, + email: Optional[str] = None, + roles: Any = None, + conn=None, + tenants: Optional[list[str]] = None, +) -> Optional[str]: + """Create or refresh the ``users`` row for a Keycloak subject; returns its UUID. + + ``tenants`` are the tenant codes in the user's token. A user has one + ``instance_id``, so the first code (sorted) wins and a warning is logged when + there are several. Authorization still reads the token, not this column. + """ + keycloak_user_id = (keycloak_user_id or "").strip() + if not keycloak_user_id: + return None + if conn is None: + with connection.get_connection() as own: + return upsert_user(keycloak_user_id, username, email, roles, conn=own, tenants=tenants) + + role_list = common._csv_roles(roles) + tenant_codes = sorted({(t or "").strip().lower() for t in (tenants or []) if (t or "").strip()}) + if len(tenant_codes) > 1: + connection.logger.warning( + "user %s has several tenants %s; users.instance_id uses %s", + keycloak_user_id, + tenant_codes, + tenant_codes[0], + ) + instance_uuid = get_or_create_instance(tenant_codes[0], conn=conn) if tenant_codes else None + email = (email or "").strip() or None + username = (username or "").strip() or None + for candidate_email in [email, None] if email else [None]: + try: + with conn.transaction(): + row = conn.execute( + """ + INSERT INTO users (keycloak_user_id, username, email, roles, instance_id) + VALUES (%s, %s, %s, %s, %s::uuid) + ON CONFLICT (keycloak_user_id) DO UPDATE SET + username = COALESCE(EXCLUDED.username, users.username), + email = COALESCE(EXCLUDED.email, users.email), + roles = CASE WHEN cardinality(EXCLUDED.roles) > 0 + THEN EXCLUDED.roles ELSE users.roles END, + instance_id = COALESCE(EXCLUDED.instance_id, users.instance_id), + last_seen_at = now() + RETURNING id + """, + (keycloak_user_id, username, candidate_email, role_list, instance_uuid), + ).fetchone() + return str(row["id"]) + except psycopg.errors.UniqueViolation: + continue + return None + + +def get_user_by_keycloak_id(keycloak_user_id: str) -> Optional[dict]: + with connection.get_connection() as conn: + return common._clean_row( + conn.execute( + "SELECT * FROM users WHERE keycloak_user_id = %s", (keycloak_user_id,) + ).fetchone() + ) diff --git a/pipeline/db/revisions.py b/pipeline/db/revisions.py new file mode 100644 index 0000000..c0d4bcf --- /dev/null +++ b/pipeline/db/revisions.py @@ -0,0 +1,324 @@ +"""Read and freeze revision snapshots and activate network releases.""" + +import hashlib +import json +from contextlib import nullcontext + +from psycopg.types.json import Jsonb + +from pipeline.metadata.schema_packs import validate_schema_selection + +from . import chunks as chunk_queries +from . import common, connection, document_reads +from . import pages as page_queries + + +def _check_revision_job(conn, document_id: str, workflow_id: str, network_id: str | None) -> str: + row = conn.execute( + "SELECT d.is_disabled, d.metadata->>'network_visible' AS network_visible, " + "w.id, w.temporal_workflow_id, w.config->>'release_network_id' AS release_network_id " + "FROM documents d JOIN document_current_workflow w ON w.document_id = d.id " + "WHERE d.id = %s::uuid FOR UPDATE OF d", + (document_id,), + ).fetchone() + if ( + not row + or row["is_disabled"] + or (network_id is not None and row["network_visible"] == "false") + or row["temporal_workflow_id"] != workflow_id + or row["release_network_id"] != network_id + ): + raise ValueError("The release is no longer authorized for this document workflow") + return str(row["id"]) + + +def load_revision_material(workflow_id: str) -> tuple[dict, list[dict], list[dict]]: + """Read the document, pages and chunks from one consistent database snapshot.""" + with connection.get_connection() as conn: + conn.execute("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY") + document_id = document_reads._require_document_uuid(conn, workflow_id) + doc = document_reads._fetch_document(conn, document_id) + pages = conn.execute( + page_queries._PAGE_SELECT + " WHERE p.document_id = %s::uuid ORDER BY p.page_number", + (document_id,), + ).fetchall() + chunks = conn.execute( + chunk_queries._CHUNK_SELECT + " WHERE c.document_id = %s::uuid ORDER BY c.chunk_number", + (document_id,), + ).fetchall() + return ( + doc, + [page_queries._page_from_row(row, workflow_id) for row in pages], + [chunk_queries._chunk_from_row(row, workflow_id) for row in chunks], + ) + + +def _insert_revision_snapshot( + conn, document_id: str, job_id: str, revision_id: str, snapshot: dict +) -> dict: + existing = conn.execute( + "SELECT id::text, snapshot FROM document_revisions WHERE workflow_id = %s::uuid", + (job_id,), + ).fetchone() + if existing: + return existing + validate_schema_selection(conn, (snapshot.get("metadata") or {}).get("knowledge_metadata")) + digest = hashlib.sha256( + json.dumps(snapshot, sort_keys=True, ensure_ascii=False).encode() + ).hexdigest() + return conn.execute( + """ + INSERT INTO document_revisions (id, document_id, workflow_id, content_sha256, snapshot) + VALUES (%s::uuid, %s::uuid, %s::uuid, %s, %s) + RETURNING id::text, snapshot + """, + (revision_id, document_id, job_id, digest, Jsonb(snapshot)), + ).fetchone() + + +def save_draft_revision_snapshot( + document_id: str, workflow_id: str, revision_id: str, snapshot: dict +) -> dict: + """Freeze the current workflow's reviewed content independently of publication.""" + with connection.get_connection() as conn: + job_id = _check_revision_job(conn, document_id, workflow_id, None) + return _insert_revision_snapshot(conn, document_id, job_id, revision_id, snapshot) + + +def save_revision_snapshot( + document_id: str, + workflow_id: str, + network_id: str, + revision_id: str, + snapshot: dict, + *, + collection: str | None = None, + provider_id: str | None = None, +) -> dict: + """Freeze content and stage its network projection together. The first snapshot wins.""" + with connection.get_connection() as conn: + job_id = _check_revision_job(conn, document_id, workflow_id, network_id) + current = _locked_network_release(conn, document_id, network_id) + revision = _insert_revision_snapshot(conn, document_id, job_id, revision_id, snapshot) + conn.execute( + "INSERT INTO network_release_staging (revision_id, network_id, collection, provider_id, expected_generation) " + "VALUES (%s::uuid, %s, %s, %s, %s::uuid) ON CONFLICT (revision_id, network_id) DO NOTHING", + ( + revision["id"], + network_id, + collection, + provider_id, + current["generation"] if current else None, + ), + ) + stage = conn.execute( + "SELECT collection, provider_id FROM network_release_staging WHERE revision_id = %s::uuid AND network_id = %s", + (revision["id"], network_id), + ).fetchone() + return {**revision, "stage": stage} + + +def _locked_network_release(conn, document_id: str, network_id: str) -> dict | None: + return conn.execute( + "SELECT generation::text, transaction_id::text, revision_id::text FROM network_releases " + "WHERE document_id = %s::uuid AND network_id = %s FOR UPDATE", + (document_id, network_id), + ).fetchone() + + +def activate_network_release( + document_id: str, workflow_id: str, network_id: str, transaction_id: str +) -> None: + """Switch the network pointer only after its adapter accepts the release.""" + with connection.get_connection() as conn: + job_id = _check_revision_job(conn, document_id, workflow_id, network_id) + revision = conn.execute( + "SELECT r.id::text, s.generation_captured AS has_generation, " + "s.expected_generation::text AS expected_generation " + "FROM document_revisions r JOIN network_release_staging s ON s.revision_id = r.id " + "WHERE r.workflow_id = %s::uuid AND s.network_id = %s", + (job_id, network_id), + ).fetchone() + if not revision: + raise ValueError("The workflow has no revision to release") + current = _locked_network_release(conn, document_id, network_id) + if current and current["revision_id"] == revision["id"]: + return + generation = current["generation"] if current else None + if not revision["has_generation"] or revision["expected_generation"] != generation: + raise ValueError("The network release changed after this revision was staged") + conn.execute( + "INSERT INTO network_releases (document_id, network_id, revision_id, transaction_id) " + "VALUES (%s::uuid, %s, %s::uuid, %s::uuid) " + "ON CONFLICT (document_id, network_id) DO UPDATE SET revision_id = excluded.revision_id, " + "transaction_id = excluded.transaction_id, released_at = now(), withdrawn_at = NULL, generation = gen_random_uuid()", + (document_id, network_id, revision["id"], transaction_id), + ) + + +def list_document_revisions(document_id: str) -> dict: + """Return revision history and current destinations without loading full snapshots.""" + with connection.get_connection() as conn: + revisions = conn.execute( + "SELECT id::text, content_sha256, created_at, snapshot->'metadata' AS metadata, " + "snapshot->'source'->>'filename' AS filename FROM document_revisions " + "WHERE document_id = %s::uuid ORDER BY created_at DESC, id DESC LIMIT 100", + (document_id,), + ).fetchall() + releases = conn.execute( + "SELECT network_id, revision_id::text, generation::text, released_at, withdrawn_at FROM network_releases " + "WHERE document_id = %s::uuid ORDER BY network_id", + (document_id,), + ).fetchall() + return { + "revisions": [common._clean_row(row) for row in revisions], + "releases": [common._clean_row(row) for row in releases], + } + + +def save_legacy_index_revisions(candidates: list[tuple[str, str, dict]]) -> None: + """Record historical search evidence without changing current workflow state.""" + with connection.get_connection() as conn: + for doc_id, revision_id, snapshot in candidates: + document = conn.execute( + "SELECT created_at, is_disabled FROM documents WHERE id = %s::uuid FOR UPDATE", + (doc_id,), + ).fetchone() + if not document or document["is_disabled"]: + raise ValueError("Migrated document became unavailable") + saved = conn.execute( + "SELECT snapshot FROM document_revisions WHERE id = %s::uuid", + (revision_id,), + ).fetchone() + if saved: + if saved["snapshot"] != snapshot: + raise ValueError( + "Existing legacy search revision differs from the supplied evidence" + ) + continue + job_id = conn.execute( + "INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, status, current_stage, " + "config, started_at, completed_at) VALUES (%s::uuid, 'legacy_index_import', %s, " + "'completed', 'completed', %s, %s::timestamptz - interval '1 second', " + "%s::timestamptz - interval '1 second') RETURNING id", + ( + doc_id, + f"legacy-index-{revision_id}", + Jsonb(snapshot["legacy_search_import"]), + document["created_at"], + document["created_at"], + ), + ).fetchone()["id"] + _insert_revision_snapshot(conn, doc_id, str(job_id), revision_id, snapshot) + + +def get_latest_completed_index_revision(document_id: str) -> dict | None: + with connection.get_connection() as conn: + return conn.execute( + "SELECT r.id::text, r.snapshot FROM document_revisions r " + "JOIN workflows w ON w.id = r.workflow_id " + "WHERE r.document_id = %s::uuid AND w.status = 'completed' " + "AND w.config->>'release_network_id' IS NULL " + "ORDER BY (w.workflow_type = 'legacy_index_import'), r.created_at DESC, r.id DESC LIMIT 1", + (document_id,), + ).fetchone() + + +def get_revision_snapshot(document_id: str, revision_id: str) -> dict | None: + if not common._is_uuid(revision_id): + return None + with connection.get_connection() as conn: + return conn.execute( + "SELECT id::text, content_sha256, snapshot FROM document_revisions " + "WHERE document_id = %s::uuid AND id = %s::uuid", + (document_id, revision_id), + ).fetchone() + + +def withdraw_network_release( + document_id: str, network_id: str, revision_id: str, *, _conn=None +) -> bool: + if not common._is_uuid(revision_id): + return False + with connection.get_connection() if _conn is None else nullcontext(_conn) as conn: + result = conn.execute( + "UPDATE network_releases SET withdrawn_at = COALESCE(withdrawn_at, now()), " + "generation = CASE WHEN withdrawn_at IS NULL THEN gen_random_uuid() ELSE generation END " + "WHERE document_id = %s::uuid AND network_id = %s AND revision_id = %s::uuid", + (document_id, network_id, revision_id), + ) + return result.rowcount > 0 + + +def stage_saved_network_revision( + document_id: str, + network_id: str, + revision_id: str, + transaction_id: str, + expected_generation: str | None, + collection: str, + provider_id: str | None, +) -> bool: + """Register the projection before indexing so it remains hidden until selection.""" + with connection.get_connection() as conn: + document = conn.execute( + """ + SELECT d.is_disabled, d.metadata->>'network_visible' AS network_visible + FROM documents d JOIN document_revisions r ON r.document_id = d.id + WHERE d.id = %s::uuid AND r.id = %s::uuid FOR UPDATE OF d + """, + (document_id, revision_id), + ).fetchone() + if not document or document["is_disabled"] or document["network_visible"] == "false": + raise ValueError("The revision is unavailable for release") + current = _locked_network_release(conn, document_id, network_id) + if current and current["transaction_id"] == transaction_id: + return False + if (current["generation"] if current else None) != expected_generation: + raise ValueError("The network release changed. Reload before selecting a revision") + stage = conn.execute( + """ + INSERT INTO network_release_staging + (revision_id, network_id, collection, provider_id, generation_captured) + VALUES (%s::uuid, %s, %s, %s, FALSE) + ON CONFLICT (revision_id, network_id) DO UPDATE + SET collection = COALESCE(network_release_staging.collection, EXCLUDED.collection), + provider_id = COALESCE(network_release_staging.provider_id, EXCLUDED.provider_id) + WHERE (network_release_staging.collection IS NULL OR network_release_staging.collection = EXCLUDED.collection) + AND (network_release_staging.provider_id IS NULL OR network_release_staging.provider_id = EXCLUDED.provider_id) + RETURNING revision_id + """, + (revision_id, network_id, collection, provider_id), + ).fetchone() + if not stage: + raise ValueError("The network projection no longer matches the staged release") + return True + + +def select_network_revision( + document_id: str, + network_id: str, + revision_id: str, + transaction_id: str, + expected_generation: str | None, +) -> None: + with connection.get_connection() as conn: + document = conn.execute( + "SELECT is_disabled, metadata->>'network_visible' AS network_visible FROM documents " + "WHERE id = %s::uuid FOR UPDATE", + (document_id,), + ).fetchone() + if not document or document["is_disabled"] or document["network_visible"] == "false": + raise ValueError("The document is unavailable for release") + current = _locked_network_release(conn, document_id, network_id) + if current and current["transaction_id"] == transaction_id: + return + if (current["generation"] if current else None) != expected_generation: + raise ValueError("The network release changed. Reload before selecting a revision") + conn.execute( + "INSERT INTO network_releases (document_id, network_id, revision_id, transaction_id) " + "VALUES (%s::uuid, %s, %s::uuid, %s::uuid) " + "ON CONFLICT (document_id, network_id) DO UPDATE SET revision_id = excluded.revision_id, " + "transaction_id = excluded.transaction_id, released_at = now(), withdrawn_at = NULL, generation = gen_random_uuid()", + (document_id, network_id, revision_id, transaction_id), + ) diff --git a/pipeline/db/settings.py b/pipeline/db/settings.py new file mode 100644 index 0000000..24c867b --- /dev/null +++ b/pipeline/db/settings.py @@ -0,0 +1,145 @@ +"""Persist retrieval settings and their audit trail.""" + +from typing import Any + +from psycopg.types.json import Jsonb + +from pipeline.documents.models import ( + CandidateSettings, + QuerySettings, + ResultSettings, + RetrievalSettings, +) + +from . import audit, connection + +# ============================================================================= +# Settings Functions +# ============================================================================= + +_SETTING_GROUPS: dict[str, tuple[str, type]] = { + "searchMethod": ("retrieval", RetrievalSettings), + "alpha": ("retrieval", RetrievalSettings), + "rankingMethod": ("retrieval", RetrievalSettings), + "hybridRrfK": ("retrieval", RetrievalSettings), + "indexName": ("candidates", CandidateSettings), + "efSearch": ("candidates", CandidateSettings), + "candidateCap": ("candidates", CandidateSettings), + "candidateMultiplier": ("candidates", CandidateSettings), + "limit": ("results", ResultSettings), + "maxChunksPerDoc": ("results", ResultSettings), + "excludeReference": ("results", ResultSettings), + "showHighlights": ("results", ResultSettings), + "rerankMode": ("results", ResultSettings), + "useE5Prefix": ("query", QuerySettings), + "queryExpansionProfile": ("query", QuerySettings), +} + + +_GROUP_MODELS: dict[str, type] = {group: model for group, model in _SETTING_GROUPS.values()} + + +_SETTINGS_ROW_SQL = ( + "SELECT retrieval, candidates, results, query FROM search_settings WHERE singleton" +) + + +def _settings_row(conn, lock: bool = False) -> dict: + sql = _SETTINGS_ROW_SQL + (" FOR UPDATE" if lock else "") + row = conn.execute(sql).fetchone() + if row is None: + conn.execute( + "INSERT INTO search_settings DEFAULT VALUES ON CONFLICT (singleton) DO NOTHING" + ) + row = conn.execute(sql).fetchone() + return row + + +def _flatten_settings(row: dict) -> dict: + flat: dict[str, Any] = {} + for group, model in _GROUP_MODELS.items(): + flat.update(model.model_validate(row[group] or {}).model_dump()) + return flat + + +def get_search_settings() -> dict: + """Get all search settings as a flat dict (stored values merged over defaults).""" + with connection.get_connection() as conn: + return _flatten_settings(_settings_row(conn)) + + +def _setting_text(value: Any) -> str: + if isinstance(value, bool): + return "true" if value else "false" + return str(value) + + +def update_search_settings(settings: dict) -> dict: + """ + Update multiple search settings at once. + + Args: + settings: Dict with keys like searchMethod, limit, alpha, etc. + + Returns: + Updated search settings + + Raises: + ValueError: a value fails the setting's validation rules. + """ + unknown = sorted(set(settings) - set(_SETTING_GROUPS)) + if unknown: + raise ValueError(f"Unknown search settings: {', '.join(unknown)}") + changes: list[tuple[str, str, str]] = [] + with connection.get_connection() as conn: + row = _settings_row(conn, lock=True) + before = _flatten_settings(row) + for group, model in _GROUP_MODELS.items(): + patch = {k: v for k, v in settings.items() if _SETTING_GROUPS[k][0] == group} + if not patch: + continue + try: + validated = model.model_validate( + {**model.model_validate(row[group] or {}).model_dump(), **patch} + ) + except Exception as exc: + raise ValueError(str(exc)) from exc + stored = {**(row[group] or {}), **{k: getattr(validated, k) for k in patch}} + conn.execute( + f"UPDATE search_settings SET {group} = {group} || %s::jsonb WHERE singleton", + (Jsonb({k: getattr(validated, k) for k in patch}),), + ) + for key in patch: + if before[key] != stored[key]: + changes.append( + (f"{group}.{key}", _setting_text(before[key]), _setting_text(stored[key])) + ) + for field_name, old_value, new_value in changes: + conn.execute( + """ + INSERT INTO audit_logs (action_type, entity_type, field_name, old_value, new_value) + VALUES ('settings_change', 'setting', %s, %s, %s) + """, + (field_name, old_value, new_value), + ) + return _flatten_settings(conn.execute(_SETTINGS_ROW_SQL).fetchone()) + + +def get_settings_audit_logs(limit: int = 50, offset: int = 0) -> list[dict]: + """Get audit logs for settings changes.""" + with connection.get_connection() as conn: + rows = conn.execute( + f"{audit._AUDIT_SELECT} WHERE a.entity_type = 'setting' " + "ORDER BY a.created_at DESC, a.id LIMIT %s OFFSET %s", + (limit, offset), + ).fetchall() + return [audit._audit_row_to_legacy(r) for r in rows] + + +def get_settings_audit_count() -> int: + """Get count of settings audit logs.""" + with connection.get_connection() as conn: + row = conn.execute( + "SELECT COUNT(*) AS cnt FROM audit_logs WHERE entity_type = 'setting'" + ).fetchone() + return int(row["cnt"]) if row else 0 diff --git a/pipeline/documents/__init__.py b/pipeline/documents/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/pipeline/documents/document_dispatch.py b/pipeline/documents/document_dispatch.py new file mode 100644 index 0000000..7169402 --- /dev/null +++ b/pipeline/documents/document_dispatch.py @@ -0,0 +1,24 @@ +"""Start a previously registered document workflow exactly once.""" + +import asyncio + +from temporalio.common import WorkflowIDReusePolicy +from temporalio.exceptions import WorkflowAlreadyStartedError + +from pipeline import db +from pipeline.temporal.workflows import DocumentPipelineWorkflow + + +async def start_registered_document(client, task_queue: str, job: dict) -> None: + try: + await client.start_workflow( + DocumentPipelineWorkflow.run, + args=job["config"]["upload_start_args"], + id=job["temporal_workflow_id"], + task_queue=task_queue, + id_reuse_policy=WorkflowIDReusePolicy.REJECT_DUPLICATE, + ) + except WorkflowAlreadyStartedError: + # A prior dispatch reached Temporal even if its response was lost. + pass + await asyncio.to_thread(db.mark_upload_started, job["id"]) diff --git a/pipeline/documents/document_formats.py b/pipeline/documents/document_formats.py new file mode 100644 index 0000000..cea0bd9 --- /dev/null +++ b/pipeline/documents/document_formats.py @@ -0,0 +1,70 @@ +"""File format admission shared by upload, URL fetch, and document processing.""" + +import os +import re +from collections.abc import Callable +from dataclasses import dataclass +from functools import lru_cache +from importlib.metadata import entry_points + +TEXT_UPLOAD_TYPES = { + ".txt": "text/plain", ".md": "text/markdown", ".markdown": "text/markdown", + ".html": "text/html", ".htm": "text/html", ".json": "application/json", +} +BINARY_MEDIA_TYPES = { + "application/pdf": ".pdf", + "application/msword": ".doc", + "application/vnd.openxmlformats-officedocument.wordprocessingml.document": ".docx", + "application/vnd.ms-powerpoint": ".ppt", + "application/vnd.openxmlformats-officedocument.presentationml.presentation": ".pptx", + "application/vnd.ms-excel": ".xls", + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": ".xlsx", + "text/csv": ".csv", + "image/jpeg": ".jpg", "image/png": ".png", "image/webp": ".webp", "image/tiff": ".tiff", +} +BUILTIN_BINARY_SUFFIXES = frozenset((*BINARY_MEDIA_TYPES.values(), ".jpeg", ".tif")) +FORMAT_ENTRY_POINT_GROUP = "knowledge_provider.document_formats" + + +@dataclass(frozen=True) +class DocumentFormat: + suffix: str + media_type: str + normalize_to_pdf: Callable[[str, str], None] + + +def installed_formats() -> tuple[DocumentFormat, ...]: + configured = os.environ.get("DOCUMENT_FORMAT_EXTENSIONS", "") + names = tuple(name.strip() for name in configured.split(",") if name.strip()) + return _load_formats(names) + + +@lru_cache(maxsize=8) +def _load_formats(names: tuple[str, ...]) -> tuple[DocumentFormat, ...]: + suffixes = set(BUILTIN_BINARY_SUFFIXES) | TEXT_UPLOAD_TYPES.keys() + media_types = BINARY_MEDIA_TYPES.keys() | set(TEXT_UPLOAD_TYPES.values()) | {"application/xhtml+xml"} + formats = [] + for name in names: + matches = tuple(entry_points(group=FORMAT_ENTRY_POINT_GROUP, name=name)) + if len(matches) != 1: + raise ValueError(f"Install exactly one document format extension named {name!r}") + definition = matches[0].load() + if not isinstance(definition, DocumentFormat) or not callable(definition.normalize_to_pdf): + raise ValueError(f"Document format extension {name!r} must export a DocumentFormat") + if not re.fullmatch(r"\.[a-z0-9]{1,16}", definition.suffix) or definition.suffix in suffixes: + raise ValueError(f"Document format extension {name!r} has an invalid or duplicate suffix") + if (not re.fullmatch(r"[a-z0-9-]+/[a-z0-9][a-z0-9.+-]{0,127}", definition.media_type) + or definition.media_type in media_types or definition.media_type == "application/octet-stream"): + raise ValueError(f"Document format extension {name!r} has an invalid or duplicate media type") + suffixes.add(definition.suffix) + media_types.add(definition.media_type) + formats.append(definition) + return tuple(formats) + + +def allowed_file_extensions() -> set[str]: + return BUILTIN_BINARY_SUFFIXES | TEXT_UPLOAD_TYPES.keys() | {item.suffix for item in installed_formats()} + + +def binary_media_types() -> dict[str, str]: + return BINARY_MEDIA_TYPES | {item.media_type: item.suffix for item in installed_formats()} diff --git a/pipeline/documents/document_kinds.py b/pipeline/documents/document_kinds.py new file mode 100644 index 0000000..b7c2108 --- /dev/null +++ b/pipeline/documents/document_kinds.py @@ -0,0 +1,22 @@ +"""Deployment configuration for document classification.""" + +import os +import re + +_KIND = re.compile(r"[a-z][a-z0-9]*(?:-[a-z0-9]+)*") + + +def configured_document_kinds() -> tuple[str, ...]: + raw = os.environ.get("DOCUMENT_KINDS", "scheme,advisory,video") + kinds = tuple(item.strip().lower() for item in raw.split(",")) + if len(set(kinds)) != len(kinds) or any(len(kind) > 64 or not _KIND.fullmatch(kind) for kind in kinds): + raise ValueError("DOCUMENT_KINDS must list distinct, lowercase document kinds") + return kinds + + +def validate_document_kind(value: str) -> str: + kind = value.strip().lower() + allowed = configured_document_kinds() + if kind not in allowed: + raise ValueError(f"Unknown document kind {value!r}. Use one of: {', '.join(allowed)}") + return kind diff --git a/pipeline/document_repository.py b/pipeline/documents/document_repository.py similarity index 93% rename from pipeline/document_repository.py rename to pipeline/documents/document_repository.py index 8c989e3..ebe7426 100644 --- a/pipeline/document_repository.py +++ b/pipeline/documents/document_repository.py @@ -8,8 +8,8 @@ from typing import Optional -from . import db -from .auth.tenancy import group_for_instance +from pipeline import db +from pipeline.auth.tenancy import group_for_instance class DocumentRepository: diff --git a/pipeline/documents/document_text.py b/pipeline/documents/document_text.py new file mode 100644 index 0000000..77ec779 --- /dev/null +++ b/pipeline/documents/document_text.py @@ -0,0 +1,117 @@ +"""Shared document text normalization and passage evidence helpers.""" + +import re + + +def clean_html_tags(text: str) -> str: + if not isinstance(text, str): + return text + text = text.replace("
", "\n").replace("
", "\n").replace("
", "\n") + text = re.sub(r"<[^>]+>", "", text) + return text.strip() + + +def clean_latex_notation(text: str) -> str: + if not isinstance(text, str): + return text + text = re.sub(r"\[\^[0-9]+\]", "", text) + text = re.sub(r"\$\s*\{\s*\}\s*\^\{[0-9]+\}\s*\$", "", text) + text = re.sub(r"\$\s*\^\{[0-9]+\}\s*\$", "", text) + text = re.sub(r"\$\s*\$", "", text) + text = re.sub(r"\\[a-zA-Z]+\{[^}]*\}", "", text) + text = re.sub(r"\\[a-zA-Z]+", "", text) + text = re.sub(r"[\\{}]", "", text) + return text + + +def format_table_content(text: str) -> str: + if not isinstance(text, str): + return text + lines = text.split("\n") + cleaned_lines = [] + for line in lines: + if re.match(r"^[\s\|]*$", line): + continue + if re.match(r"^[\s\|\-\:]*$", line): + continue + line = re.sub(r"\|\s*\|", "|", line) + line = re.sub(r"^\|\s*", "", line) + line = re.sub(r"\s*\|$", "", line) + line = re.sub(r"\s+", " ", line).strip() + if line: + cleaned_lines.append(line) + return "\n".join(cleaned_lines) + + +def clean_text(text: str) -> str: + if not isinstance(text, str): + return text + text = clean_html_tags(text) + text = clean_latex_notation(text) + text = format_table_content(text) + text = re.sub(r"\n{3,}", "\n\n", text) + text = re.sub(r"[ \t]+", " ", text) + lines = [line.strip() for line in text.split("\n")] + text = "\n".join(lines) + text = re.sub(r"\n\s*\n\s*\n", "\n\n", text) + text = re.sub(r"\n ", "\n", text) + text = re.sub(r" \n", "\n", text) + return text.strip() + + +def _infer_section(text: str, section_title: str | None = None) -> str: + """Best-effort section heading for chunk provenance.""" + if section_title and str(section_title).strip(): + return str(section_title).strip() + if not text: + return "" + for line in text.splitlines(): + stripped = line.strip() + if stripped.startswith("## "): + return stripped[3:].strip() + if stripped.startswith("# "): + return stripped[2:].strip() + return "" + + +def is_reference_section(text: str) -> bool: + """ + Detect if text is primarily a reference/bibliography section. + Returns True if the text appears to be citations/references. + """ + if not text or len(text) < 50: + return False + + ref_headers = [ + r"^\s*#{1,3}\s*(?:references|bibliography|citations|works cited|literature cited)\s*$", + r"^\s*\*{1,2}(?:references|bibliography)\*{1,2}\s*$", + ] + for pattern in ref_headers: + if re.search(pattern, text, re.IGNORECASE | re.MULTILINE): + return True + + lines = text.split("\n") + total_lines = len([line for line in lines if line.strip()]) + if total_lines == 0: + return False + + citation_patterns = [ + r"^\s*\d{1,3}[\.\)]\s+[A-Z][a-z]+[\s,].*(?:\d{4}|\(\d{4}\))", + r"doi[:\s]*10\.\d{4,}", + r"(?:J\.|Journal|Int\.|Proceedings|Trans\.).*\d{4}", + r"\(\d{4}\)\s*$", + r"\bet\s+al\b", + r"(?:Vol\.?\s*\d+|\d+\s*\(\d+\)\s*:)", + r"(?:pp?\.?\s*\d+[-–]\d+|:\s*\d+[-–]\d+)", + ] + + citation_line_count = 0 + for line in lines: + if not line.strip(): + continue + for pattern in citation_patterns: + if re.search(pattern, line, re.IGNORECASE): + citation_line_count += 1 + break + + return (citation_line_count / total_lines) > 0.4 diff --git a/pipeline/documents/document_validity.py b/pipeline/documents/document_validity.py index 31fc8e4..70747c7 100644 --- a/pipeline/documents/document_validity.py +++ b/pipeline/documents/document_validity.py @@ -14,9 +14,11 @@ those fields at search time; `pipeline/api/api.py` validates approver input here. """ +import os from dataclasses import dataclass from datetime import date, datetime from typing import Callable, Optional +from zoneinfo import ZoneInfo # Operators think in calendar dates, not instants - a period is stored, # exchanged and filtered as a plain `YYYY-MM-DD` day. @@ -37,9 +39,8 @@ class DocumentValidityError(ValueError): def system_clock() -> datetime: - """The real clock. Local rather than UTC: a period is a calendar date an - operator picked, so "today" should mean their today.""" - return datetime.now() + """Evaluate applicability dates in the deployment's configured timezone.""" + return datetime.now(ZoneInfo(os.environ.get("KNOWLEDGE_TIMEZONE", "UTC"))) @dataclass(frozen=True) diff --git a/pipeline/models.py b/pipeline/documents/models.py similarity index 88% rename from pipeline/models.py rename to pipeline/documents/models.py index 011b101..143c81b 100644 --- a/pipeline/models.py +++ b/pipeline/documents/models.py @@ -5,9 +5,21 @@ from datetime import datetime from enum import Enum -from typing import Any, Optional +from typing import Annotated, Any, Optional -from pydantic import BaseModel, ConfigDict, Field, model_validator +from pydantic import ( + BaseModel, + ConfigDict, + Field, + StringConstraints, + field_serializer, + field_validator, + model_validator, +) + +from pipeline.metadata.crop_metadata import CropSubjects +from pipeline.metadata.knowledge_metadata import KnowledgeMetadata +from pipeline.metadata.localization import normalize_language_tag class DocumentStage(str, Enum): @@ -137,6 +149,7 @@ class ChunkData(BaseModel): is_reviewed: bool = False is_excluded: bool = False reviewer_notes: Optional[str] = None + tags: list[str] = Field(default_factory=list) @property def text(self) -> str: @@ -182,10 +195,6 @@ class RegisterRequest(BaseModel): filepath: str -class RegisterFolderRequest(BaseModel): - directory: str - - class PageUpdate(BaseModel): edited_markdown: Optional[str] = None is_reviewed: Optional[bool] = None @@ -193,9 +202,29 @@ class PageUpdate(BaseModel): edited_translation: Optional[str] = None translation_reviewed: Optional[bool] = None translation_notes: Optional[str] = None + translation_language: Optional[str] = None + + @field_validator("translation_language") + @classmethod + def normalize_translation_language(cls, value: str | None) -> str | None: + if value is None: + return None + tag = normalize_language_tag(value) + if tag is None: + raise ValueError("translation_language must be a BCP 47 language tag") + return tag + + @model_validator(mode="after") + def check_translation_edit(self) -> "PageUpdate": + if self.translation_language and all(value is None for value in ( + self.edited_translation, self.translation_reviewed, self.translation_notes, + )): + raise ValueError("translation_language requires a translation edit") + return self class ChunkUpdate(BaseModel): + tags: Optional[list[Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=128)]]] = Field(default=None, max_length=16) edited_text: Optional[str] = None is_reviewed: Optional[bool] = None is_excluded: Optional[bool] = None @@ -208,6 +237,13 @@ class ApprovalRequest(BaseModel): class DocumentSummary(BaseModel): + crop_subjects: CropSubjects = Field(default_factory=list) + knowledge_metadata: dict = Field(default_factory=dict) + + @field_serializer("crop_subjects") + def serialize_crop_subjects(self, subjects: CropSubjects) -> list[dict]: + return [subject.model_dump(exclude_none=True) for subject in subjects] + id: Optional[str] = None # documents.id (UUID); canonical identifier in API paths document_id: str canonical_document_id: Optional[str] = None @@ -476,7 +512,7 @@ class SearchSettings(BaseModel): maxChunksPerDoc: int = 2 useE5Prefix: bool = True excludeReference: bool = True - queryExpansionProfile: str = "gu-v1" + queryExpansionProfile: str = "none" rerankMode: str = "none" hybridRrfK: int = 60 @@ -563,6 +599,8 @@ class DocumentMetadata(BaseModel): """``documents.metadata``: period, uploader, index state and flag fields.""" model_config = ConfigDict(extra="forbid") + crop_subjects: CropSubjects = Field(default_factory=list) + knowledge_metadata: KnowledgeMetadata | None = None valid_from: Optional[str] = None # YYYY-MM-DD valid_to: Optional[str] = None # YYYY-MM-DD, None = no end uploaded_by_user_id: Optional[str] = None # users.id (UUID string) @@ -603,13 +641,42 @@ class CatalogMetadata(BaseModel): network: Optional[CatalogNetworkState] = None +PAGE_LAYOUT_MAX_ITEMS = 128 +PAGE_LAYOUT_ITEM_MAX_CHARS = 256 + + +class PositionedPageText(BaseModel): + model_config = ConfigDict(extra="forbid", allow_inf_nan=False) + + text: str = Field(max_length=PAGE_LAYOUT_ITEM_MAX_CHARS) + x: float + y: float + width: float = Field(ge=0) + height: float = Field(ge=0) + + +class PageSourceLayout(BaseModel): + """Source PDF excerpts in points, measured from the page's top left.""" + + model_config = ConfigDict(extra="forbid", allow_inf_nan=False) + + parser: str + width: float = Field(gt=0) + height: float = Field(gt=0) + text_items: list[PositionedPageText] = Field(max_length=PAGE_LAYOUT_MAX_ITEMS) + truncated: bool = False + ocr_recommended: bool | None = None + + class PageMetadata(BaseModel): - """``pages.metadata``: OCR review and language fields.""" + """``pages.metadata``: OCR review, language and source layout fields.""" model_config = ConfigDict(extra="forbid") is_reviewed: bool = False reviewer_notes: Optional[str] = None detected_language: Optional[str] = None + declared_language: Optional[str] = None + source_layout: PageSourceLayout | None = None class PageTranslation(BaseModel): @@ -657,4 +724,4 @@ class QuerySettings(BaseModel): model_config = ConfigDict(extra="forbid") useE5Prefix: bool = True - queryExpansionProfile: str = "gu-v1" + queryExpansionProfile: str = "none" diff --git a/pipeline/instances.py b/pipeline/instances.py deleted file mode 100644 index 0bc0777..0000000 --- a/pipeline/instances.py +++ /dev/null @@ -1,105 +0,0 @@ -"""Instance (state / tenant) code → human-readable name. - -An instance code is a short lowercase ``instances.name`` — ``bv`` for the Bharat -Vistaar portal, otherwise the state code matching the Keycloak group path -(``/states/MH/...`` → ``mh``). A document's instance is derived through its -catalog (``documents.catalog_id`` → ``catalogs.instance_id``). That code is what -the pipeline filters on, but it is not what a reader wants to see: consumers of -the vector payload and the master catalog get ``instance_name`` (their field -name, unchanged) alongside the code so they can show "Maharashtra" without -shipping their own lookup table. - -Codes follow the standard two-letter Indian state/UT abbreviations. Unknown -codes fall back to a title-cased version of the code itself rather than raising -— a new state should degrade to a readable label, not break ingestion. -""" - -from __future__ import annotations - -import os - -# Portal / platform-wide documents (not a state). -PORTAL_INSTANCE = "bv" - - -def publish_to_network_enabled() -> bool: - return (os.environ.get("ENABLE_PUBLISH_TO_NETWORK", "false").strip().lower() - in {"true", "1", "yes"}) - - -def prod_stage_disabled() -> bool: - """True when DISABLE_PROD_SETTING turns off the PROD half of the pipeline. - - With this set, a document finishes at DEV ingest instead of waiting for a - superadmin to promote it — the ``approval_for_prod`` and ``ingesting_prod`` - stages are skipped entirely. - - Read only when a workflow is *started*; the value is then passed into the - workflow as an argument. Reading it inside a workflow would break Temporal - replay, because flipping the flag mid-flight would send a replaying - workflow down a different branch than the one recorded in its history. - """ - return (os.environ.get("DISABLE_PROD_SETTING", "false").strip().lower() - in {"true", "1", "yes"}) - -INSTANCE_NAMES: dict[str, str] = { - # Platform - "bv": "Bharat Vistaar", - "default": "Default", - # States - "ap": "Andhra Pradesh", - "ar": "Arunachal Pradesh", - "as": "Assam", - "br": "Bihar", - "cg": "Chhattisgarh", - "ga": "Goa", - "gj": "Gujarat", - "hr": "Haryana", - "hp": "Himachal Pradesh", - "jh": "Jharkhand", - "ka": "Karnataka", - "kl": "Kerala", - "mp": "Madhya Pradesh", - "mh": "Maharashtra", - "mn": "Manipur", - "ml": "Meghalaya", - "mz": "Mizoram", - "nl": "Nagaland", - "od": "Odisha", - "or": "Odisha", # older code still seen in group paths - "pb": "Punjab", - "rj": "Rajasthan", - "sk": "Sikkim", - "tn": "Tamil Nadu", - "tg": "Telangana", - "ts": "Telangana", # older code still seen in group paths - "tr": "Tripura", - "up": "Uttar Pradesh", - "uk": "Uttarakhand", - "ua": "Uttarakhand", # older code still seen in group paths - "wb": "West Bengal", - # Union territories - "an": "Andaman and Nicobar Islands", - "ch": "Chandigarh", - "dn": "Dadra and Nagar Haveli and Daman and Diu", - "dl": "Delhi", - "jk": "Jammu and Kashmir", - "la": "Ladakh", - "ld": "Lakshadweep", - "py": "Puducherry", -} - - -def instance_display_name(code: str | None) -> str: - """Human-readable name for an instance code (``mh`` → ``Maharashtra``). - - Unknown codes are title-cased so the field is never empty. - """ - normalized = (code or "").strip().lower() - if not normalized: - return INSTANCE_NAMES["default"] - known = INSTANCE_NAMES.get(normalized) - if known: - return known - # Unknown code: make it presentable rather than failing the ingest. - return normalized.replace("-", " ").replace("_", " ").title() diff --git a/pipeline/intake/__init__.py b/pipeline/intake/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/pipeline/intake/jsonata_runtime.py b/pipeline/intake/jsonata_runtime.py new file mode 100644 index 0000000..6f48fe1 --- /dev/null +++ b/pipeline/intake/jsonata_runtime.py @@ -0,0 +1,63 @@ +"""Evaluate source mappings in a separate, bounded process.""" + +import json +import os +import subprocess +import sys +import threading + +_MAX_BYTES = 2 * 1024 * 1024 +_SLOTS = threading.BoundedSemaphore(2) + + +def evaluate(expression: str, payload: object) -> object: + data = json.dumps([expression, payload], ensure_ascii=False, allow_nan=False).encode() + if len(data) > _MAX_BYTES + 32768: + raise ValueError("JSONata input is too large") + if not _SLOTS.acquire(timeout=2): + raise ValueError("JSONata evaluation is busy") + try: + result = subprocess.run( + [sys.executable, "-I", os.path.abspath(__file__)], + input=data, capture_output=True, timeout=3, env={}, check=False, + ) + if result.returncode or len(result.stdout) > _MAX_BYTES: + raise ValueError("JSONata mapping failed or exceeded its limits") + return json.loads(result.stdout) + except (OSError, subprocess.TimeoutExpired) as error: + raise ValueError("JSONata mapping could not finish") from error + finally: + _SLOTS.release() + + +def _main() -> None: + import resource + + if sys.platform.startswith("linux"): + resource.setrlimit(resource.RLIMIT_AS, (256 * 1024**2, 256 * 1024**2)) + resource.setrlimit(resource.RLIMIT_CPU, (2, 2)) + + import jsonata + import re2 + + def linear_regex(pattern, flags): + options = re2.Options() + options.case_sensitive = not flags.case_insensitive + options.one_line = not flags.multiline + options.log_errors = False + return re2.compile(pattern, options) + + expression, payload = json.load(sys.stdin) + result = jsonata.Jsonata(expression, linear_regex, timeout=1000, stack=200).evaluate(payload) + # Write only valid bounded JSON; never put sender input or engine errors in server logs. + output = json.dumps(result, ensure_ascii=False, allow_nan=False).encode() + if len(output) > _MAX_BYTES: + raise ValueError("JSONata output is too large") + sys.stdout.buffer.write(output) + + +if __name__ == "__main__": + try: + _main() + except Exception: + sys.exit(1) diff --git a/pipeline/intake/provider_sources.py b/pipeline/intake/provider_sources.py new file mode 100644 index 0000000..5d98418 --- /dev/null +++ b/pipeline/intake/provider_sources.py @@ -0,0 +1,163 @@ +"""Provider-owned source settings with deployment bootstrap fallback.""" + +import json +import os +import re + +import httpx +from psycopg.types.json import Jsonb +from pydantic import BaseModel, ConfigDict, Field + +from pipeline import db + +_CODE = re.compile(r"^[a-z][a-z0-9-]{0,63}$") + + +class CrawlProfile(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + host: str = Field(pattern=r"^[a-zA-Z0-9.-]{1,253}$") + parser_profile: str | None = Field(default=None, max_length=64) + intake_template: str | None = Field(default=None, pattern=r"^[a-z][a-z0-9-]{0,63}$") + authorization_env: str | None = Field(default=None, pattern=r"^CRAWL_AUTH_[A-Z0-9_]{1,100}$") + + +class SourceSettingsUpdate(BaseModel): + model_config = ConfigDict(extra="forbid") + + version: int = Field(ge=0) + release_network_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,252}$") + templates: dict = Field(default_factory=dict, max_length=64) + crawl: dict = Field(default_factory=dict, max_length=32) + schedules: list[dict] = Field(default_factory=list, max_length=32) + + +def stored_settings(provider: str) -> dict | None: + with db.get_connection() as conn: + return conn.execute( + "SELECT s.settings, s.version FROM provider_source_settings s " + "JOIN instances i ON i.id = s.provider_id WHERE lower(i.name) = %s", (provider,), + ).fetchone() + + +def all_stored_settings() -> dict[str, dict]: + with db.get_connection() as conn: + rows = conn.execute( + "SELECT lower(i.name) AS provider, s.settings FROM provider_source_settings s " + "JOIN instances i ON i.id = s.provider_id", + ).fetchall() + return {row["provider"]: row["settings"] for row in rows} + + +def bootstrap_settings(provider: str) -> dict: + from pipeline.intake.scheduled_intake import bootstrap_profiles + + crawl = json.loads(os.environ.get("CRAWL_SOURCE_PROFILES") or "{}").get(provider, {}) + schedules = [ + profile.model_dump(mode="json", exclude={"provider", "intake_template"}) + for profile in bootstrap_profiles() if profile.provider == provider + ] + return {"templates": {}, "crawl": crawl, "schedules": schedules} + + +def source_settings(provider: str) -> dict: + row = stored_settings(provider) + return row if row else {"version": 0, "settings": bootstrap_settings(provider)} + + +def public_settings(row: dict) -> dict: + # Return access mode, never a credential reference or value. + settings = json.loads(json.dumps(row["settings"])) + for profile in [*settings["crawl"].values(), *settings["schedules"]]: + profile["authenticated"] = bool(profile.pop("authorization_env", None)) + return {"version": row["version"], **settings} + + +def validate_settings(provider: str, update: SourceSettingsUpdate, previous: dict) -> dict: + from pipeline.intake.scheduled_intake import PullProfile + from pipeline.intake.webhook_mapping import SOURCE_MAPPING, load_intake_template + + if len(update.model_dump_json().encode()) > 262144: + raise ValueError("Source settings must be at most 256 KiB") + if any(not _CODE.fullmatch(key) for key in [*update.templates, *update.crawl]): + raise ValueError("Template and source names must use lowercase letters, numbers and hyphens") + templates = {key: SOURCE_MAPPING.validate_python(value).model_dump(mode="json") for key, value in update.templates.items()} + + def template(name): + if name in templates: + return templates[name] + # Deployment templates remain available as bootstrap fallback. + return load_intake_template(name).model_dump(mode="json") + + def bind_access(candidate, old): + candidate = dict(candidate) + authenticated = candidate.pop("authenticated", False) + if candidate.get("authorization_env"): + raise ValueError("Credential references must be provisioned by the deployment operator") + candidate.pop("authorization_env", None) + if old and old.get("authorization_env"): + if candidate.get("host", "").lower() != old["host"].lower(): + raise ValueError("An authenticated source cannot be moved to another host") + candidate["authorization_env"] = old["authorization_env"] + elif authenticated: + raise ValueError("Ask the deployment operator to provision access for this source") + return candidate + + crawl = {} + for key, value in update.crawl.items(): + item = CrawlProfile.model_validate(bind_access(value, previous.get("crawl", {}).get(key))) + if item.intake_template: + template(item.intake_template) + crawl[key] = item.model_dump(mode="json", exclude_none=True) + schedules = [] + previous_schedules = {item["source_key"]: item for item in previous.get("schedules", [])} + for value in update.schedules: + if "provider" in value: + raise ValueError("Scheduled sources belong to the selected provider") + candidate = bind_access(value, previous_schedules.get(value.get("source_key"))) + template_name = candidate.pop("intake_template", None) + if template_name: + if "mapping" in candidate: + raise ValueError("Choose an intake template or supply a mapping") + candidate["mapping"] = template(template_name) + item = PullProfile.model_validate({**candidate, "provider": provider}) + url = httpx.URL(item.url) + if (url.scheme != "https" or url.port not in (None, 443) or url.userinfo or url.fragment + or url.host.lower() != item.host.lower() + or any(ch.isspace() or ord(ch) < 32 for ch in item.url)): + raise ValueError("Scheduled URLs must use HTTPS on the saved source host") + stored = item.model_dump(mode="json", exclude={"provider", "intake_template"}, exclude_none=True) + if template_name: + stored.pop("mapping", None) + stored["intake_template"] = template_name + schedules.append(stored) + if len({item["source_key"] for item in schedules}) != len(schedules): + raise ValueError("Scheduled source keys must be unique") + release_network_id = update.release_network_id + if release_network_id: + from pipeline.network.network_connections import configured_network_connections + + if not any(c.network_id == release_network_id for c in configured_network_connections()): + raise ValueError("Choose a configured publication network") + return {"templates": templates, "crawl": crawl, "schedules": schedules, "release_network_id": release_network_id} + + +def save_settings(provider: str, update: SourceSettingsUpdate, actor: str) -> dict: + with db.get_connection() as conn: + row = conn.execute("SELECT id FROM instances WHERE lower(name) = %s FOR UPDATE", (provider,)).fetchone() + if not row: + raise LookupError("Provider not found") + current = conn.execute( + "SELECT settings, version FROM provider_source_settings WHERE provider_id = %s", (row["id"],), + ).fetchone() + version = current["version"] if current else 0 + if version != update.version: + raise FileExistsError("Source settings changed. Reload before saving.") + settings = validate_settings(provider, update, current["settings"] if current else bootstrap_settings(provider)) + conn.execute( + "INSERT INTO provider_source_settings(provider_id, settings, version, updated_by) VALUES (%s, %s, 1, %s) " + "ON CONFLICT(provider_id) DO UPDATE SET settings = excluded.settings, " + "version = provider_source_settings.version + 1, updated_by = excluded.updated_by, updated_at = now()", + (row["id"], Jsonb(settings), actor), + ) + return public_settings({"version": version + 1, "settings": settings}) diff --git a/pipeline/intake/scheduled_intake.py b/pipeline/intake/scheduled_intake.py new file mode 100644 index 0000000..3c638a6 --- /dev/null +++ b/pipeline/intake/scheduled_intake.py @@ -0,0 +1,336 @@ +"""Durable, bounded polling for deployment-approved knowledge sources.""" + +from __future__ import annotations + +import asyncio +import hashlib +import logging +import os +from io import BytesIO +from pathlib import Path +from typing import Annotated +from uuid import uuid4 + +import httpx +from pydantic import BaseModel, ConfigDict, Field, StringConstraints, TypeAdapter, model_validator + +from pipeline import db +from pipeline.intake.source_fetch import SourceFile, fetch_source +from pipeline.network.release_policy import index_only_mode, source_release_target +from pipeline.object_storage import ObjectReference, load_object_store_config, open_object_store +from pipeline.ocr.service import allowed_ocr_providers +from pipeline.intake.source_intake import ( + BinarySource, + CrawlReleasePolicyConflict, + NormalizedSourceDocument, + SourceDocumentMetadata, + attach_source_response, + canonical_json_sha256, + crawl_record_key, + register_source_document, + store_source_response, +) +from pipeline.intake.webhook_mapping import ( + CanonicalMapping, + JsonPointer, + SourceMapping, + _read_path, + decode_source_json, + load_intake_template, + map_source_submission, +) + +logger = logging.getLogger(__name__) +Code = Annotated[str, StringConstraints(pattern=r"^[a-z0-9][a-z0-9-]{0,63}$")] + + +class PullProfile(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + provider: Code + source_key: Code + host: str = Field(pattern=r"^[a-zA-Z0-9.-]{1,253}$") + url: str = Field(min_length=1, max_length=4096) + interval_seconds: int = Field(default=900, ge=60, le=86400) + authorization_env: str | None = Field(default=None, pattern=r"^CRAWL_AUTH_[A-Z0-9_]{1,100}$") + kind: str = "advisory" + mapping: SourceMapping = Field(default_factory=CanonicalMapping) + intake_template: str | None = Field(default=None, pattern=r"^[a-z][a-z0-9-]{0,63}$") + next_cursor_pointer: JsonPointer | None = None + cursor_parameter: str = Field(default="cursor", pattern=r"^[A-Za-z0-9_-]{1,64}$") + auto_release: bool = False + release_network_id: str | None = Field(default=None, pattern=r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,252}$") + parser_profile: str | None = Field(default=None, max_length=64) + + @model_validator(mode="after") + def resolve_template(self) -> PullProfile: + if self.release_network_id and not self.auto_release: + raise ValueError("A release network requires automatic release") + if self.intake_template is not None: + if "mapping" in self.model_fields_set: + raise ValueError("Choose an intake template or supply a mapping") + self.mapping = load_intake_template(self.intake_template) + return self + + @property + def identity(self) -> tuple[str, str, str]: + profile = self.model_dump(mode="json", exclude={"intake_template"}) + if self.release_network_id is None: + # Profiles using the default destination retain their existing cursor checkpoint. + profile.pop("release_network_id") + return self.provider, self.source_key, canonical_json_sha256(profile) + + +_PROFILES = TypeAdapter(Annotated[list[PullProfile], Field(max_length=32)]) + + +def bootstrap_profiles() -> list[PullProfile]: + path = os.environ.get("SCHEDULED_PULL_PROFILES_FILE") + if not path: + return [] + with Path(path).open("rb") as handle: + contents = handle.read(262145) + if len(contents) > 262144: + raise ValueError("Scheduled pull configuration exceeds 256 KiB") + profiles = _PROFILES.validate_python(decode_source_json(contents)) + if len({(profile.provider, profile.source_key) for profile in profiles}) != len(profiles): + raise ValueError("Scheduled pull profiles must have unique provider and source keys") + return profiles + + +def configured_profiles() -> list[PullProfile]: + from pipeline.intake.provider_sources import all_stored_settings + + settings = all_stored_settings() + profiles = [profile for profile in bootstrap_profiles() if profile.provider not in settings] + for provider, stored in settings.items(): + for item in stored["schedules"]: + item = dict(item) + name = item.pop("intake_template", None) + if name: + item["mapping"] = load_intake_template(name, provider=provider) + profiles.append(PullProfile.model_validate({**item, "provider": provider})) + return profiles + + +def provider_pull_status(provider: str) -> list[dict]: + profiles = {profile.identity: profile for profile in configured_profiles() if profile.provider == provider} + if not profiles: + return [] + with db.get_connection() as conn: + checkpoints = conn.execute( + """ + SELECT source_key, profile_sha256, failure_count, next_run_at, last_fetch_at, + last_page_policy_conflicts, + coalesce(lease_expires_at > now(), false) AS polling + FROM scheduled_source_pulls + WHERE provider = %s AND profile_sha256 = ANY(%s) + """, + (provider, [profile_hash for _, _, profile_hash in profiles]), + ).fetchall() + states = {(row["source_key"], row["profile_sha256"]): row for row in checkpoints} + sources = [] + for (_, source_key, profile_hash), profile in sorted(profiles.items()): + row = states.get((source_key, profile_hash), {}) + failures = row.get("failure_count", 0) + status = "awaiting_worker" + if row: + if row["polling"]: + status = "polling" + elif failures: + status = "retrying" + else: + status = "scheduled" + sources.append({ + "source_key": source_key, + "status": status, + "auto_release": profile.auto_release, + "release_network_id": profile.release_network_id, + "consecutive_failures": failures, + "last_page_received_at": row.get("last_fetch_at"), + "last_page_policy_conflicts": row.get("last_page_policy_conflicts", 0), + "next_check_at": None if row.get("polling") else row.get("next_run_at"), + }) + return sources + + +def _claim(profile: PullProfile) -> dict | None: + with db.get_connection() as conn: + conn.execute( + """ + INSERT INTO scheduled_source_pulls (provider, source_key, profile_sha256) + VALUES (%s, %s, %s) ON CONFLICT DO NOTHING + """, profile.identity, + ) + return conn.execute( + """ + UPDATE scheduled_source_pulls + SET lease_id = %s, lease_expires_at = now() + interval '2 minutes' + WHERE provider = %s AND source_key = %s AND profile_sha256 = %s + AND next_run_at <= now() AND (lease_expires_at IS NULL OR lease_expires_at <= now()) + RETURNING cursor, visited_cursors, lease_id, failure_count + """, + (uuid4(), *profile.identity), + ).fetchone() + + +def _register_page( + profile: PullProfile, claim: dict, rows: list, next_cursor: str | None, checksum: str, + binary: BinarySource | None = None, capture: dict | None = None, +) -> bool: + with db.get_connection() as conn: + state = conn.execute( + """ + SELECT lease_id FROM scheduled_source_pulls + WHERE provider = %s AND source_key = %s AND profile_sha256 = %s AND lease_expires_at > now() + FOR UPDATE + """, profile.identity, + ).fetchone() + if not state or state["lease_id"] != claim["lease_id"]: + return False + targets: dict[str, tuple[str, str]] = {} + policy_conflicts = 0 + release = profile.auto_release and not index_only_mode() + for raw, document in rows: + if release and document.kind not in targets: + target = source_release_target(document.kind, profile.release_network_id, provider=profile.provider) + if not target: + raise ValueError("Automatic publication is unavailable for this source document") + targets[document.kind] = target + try: + receipt = register_source_document( + conn, instance=profile.provider, source_kind="crawl", source_id=profile.source_key, + idempotency_key=crawl_record_key(raw, document), + raw_payload=raw, document=document, release_target=targets.get(document.kind), binary=binary, + explicit_release_network=release and profile.release_network_id is not None, + ) + if capture: + attach_source_response(conn, receipt["job_id"], capture) + except CrawlReleasePolicyConflict: + policy_conflicts += 1 + history = [*claim["visited_cursors"], next_cursor] if next_cursor else [] + conn.execute( + """ + UPDATE scheduled_source_pulls + SET cursor = %s, visited_cursors = %s, next_run_at = now() + make_interval(secs => %s), + lease_id = NULL, lease_expires_at = NULL, failure_count = 0, last_error = NULL, + last_fetch_at = now(), last_payload_sha256 = %s, last_page_policy_conflicts = %s + WHERE provider = %s AND source_key = %s AND profile_sha256 = %s + """, + (next_cursor, history, 0 if next_cursor else profile.interval_seconds, checksum, + policy_conflicts, *profile.identity), + ) + return True + + +def _record_failure(profile: PullProfile, claim: dict, error: Exception) -> None: + with db.get_connection() as conn: + conn.execute( + """ + UPDATE scheduled_source_pulls + SET failure_count = failure_count + 1, last_error = %s, lease_id = NULL, + lease_expires_at = NULL, next_run_at = now() + make_interval(secs => %s) + WHERE provider = %s AND source_key = %s AND profile_sha256 = %s AND lease_id = %s + """, + (type(error).__name__, min(3600, 30 * 2 ** min(claim["failure_count"], 7)), *profile.identity, claim["lease_id"]), + ) + + +def _store_binary(profile: PullProfile, source: SourceFile, checksum: str) -> BinarySource: + if source.media_type == "application/pdf" and not source.content.startswith(b"%PDF-"): + raise ValueError("The source did not return a valid PDF header") + if not source.content: + raise ValueError("The source returned an empty file") + if profile.parser_profile and profile.parser_profile not in allowed_ocr_providers(): + raise ValueError("The selected document parser profile is unavailable") + config = load_object_store_config() + ref = ObjectReference( + config.bucket, f"scheduled/{profile.provider}/{profile.source_key}/{checksum}/{source.filename}", + ) + store = open_object_store(config) + if config.create_bucket: + store.ensure_bucket(config.bucket) + stored_media_type = "application/pdf" if source.media_type == "application/pdf" else "application/octet-stream" + store.put(ref, BytesIO(source.content), len(source.content), stored_media_type) + return BinarySource( + source.filename, ref.to_uri(), hashlib.md5(source.content).hexdigest(), + stored_media_type, len(source.content), profile.parser_profile, + ) + + +def _store_response(profile: PullProfile, source) -> dict: + config = load_object_store_config() + store = open_object_store(config) + if config.create_bucket: + store.ensure_bucket(config.bucket) + return store_source_response(store, config.bucket, profile.provider, source) + + +async def poll_source(profile: PullProfile) -> bool: + claim = await asyncio.to_thread(_claim, profile) + if not claim: + return False + try: + url = httpx.URL(profile.url) + if claim["cursor"]: + url = url.copy_set_param(profile.cursor_parameter, claim["cursor"]) + authorization = os.environ.get(profile.authorization_env) if profile.authorization_env else None + if profile.authorization_env and not authorization: + raise ValueError("The scheduled source credential is unavailable") + source = await fetch_source(str(url), {profile.host.lower()}, authorization) + capture = await asyncio.to_thread(_store_response, profile, source) + next_cursor = None + binary = None + json_response = isinstance(source, SourceFile) and source.media_type == "application/json" + if profile.next_cursor_pointer is not None and not json_response: + raise ValueError("A paginated scheduled feed must return JSON") + if json_response: + payload = decode_source_json(source.content) + checksum = hashlib.sha256(source.content).hexdigest() + rows = await asyncio.to_thread(map_source_submission, payload, profile.mapping) + if profile.next_cursor_pointer is not None: + next_cursor = _read_path(payload, profile.next_cursor_pointer) + if next_cursor is not None and ( + not isinstance(next_cursor, str) or not 0 < len(next_cursor) <= 1024 + or any(ord(ch) < 32 for ch in next_cursor) + ): + raise ValueError("The scheduled source returned an invalid cursor") + elif isinstance(source, SourceFile): + checksum = hashlib.sha256(source.content).hexdigest() + document = SourceDocumentMetadata( + external_id=profile.source_key, title=Path(source.filename).stem, + kind=profile.kind, source_url=source.attribution_url, + ) + binary = await asyncio.to_thread(_store_binary, profile, source, checksum) + rows = [({"sha256": checksum, "filename": source.filename, "media_type": source.media_type}, document)] + else: + document = NormalizedSourceDocument( + external_id=profile.source_key, title=source.title, text=source.text, + kind=profile.kind, language=source.language, source_url=source.attribution_url, + ) + payload = document.model_dump(mode="json") + rows = [(payload, document)] + checksum = capture["sha256"] + if next_cursor and (next_cursor in claim["visited_cursors"] or len(claim["visited_cursors"]) >= 100): + raise ValueError("The scheduled source repeated a cursor or exceeded the page limit") + rows = [ + (raw, document.model_copy(update={"source_url": document.source_url or source.attribution_url})) + for raw, document in rows + ] + return await asyncio.to_thread(_register_page, profile, claim, rows, next_cursor, checksum, binary, capture) + except Exception as error: + await asyncio.to_thread(_record_failure, profile, claim, error) + logger.warning("Scheduled pull failed provider=%s source=%s error_type=%s", + profile.provider, profile.source_key, type(error).__name__) + return False + + +async def poll_due_sources() -> int: + profiles = configured_profiles() + slots = asyncio.Semaphore(4) + + async def poll(profile: PullProfile) -> bool: + async with slots: + return await poll_source(profile) + + return sum(await asyncio.gather(*(poll(profile) for profile in profiles))) diff --git a/pipeline/intake/source_fetch.py b/pipeline/intake/source_fetch.py new file mode 100644 index 0000000..49f07ef --- /dev/null +++ b/pipeline/intake/source_fetch.py @@ -0,0 +1,206 @@ +"""Fetch one bounded public HTTPS source and retain its exact response bytes.""" + +import asyncio +import ipaddress +import socket +from contextlib import aclosing +from dataclasses import dataclass +from html.parser import HTMLParser +from pathlib import PurePosixPath +from urllib.parse import unquote + +import httpx + +from pipeline.documents.document_formats import binary_media_types +from pipeline.metadata.localization import normalize_language_tag + +MAX_HTML_BYTES = 2 * 1024 * 1024 +MAX_EXTRACTED_CHARS = 200_000 +MAX_FILE_BYTES = 20 * 1024 * 1024 +_HTML_TYPES = {"text/html", "application/xhtml+xml"} +_TEXT_TYPES = {"text/plain", "text/markdown"} + +_HIDDEN = {"script", "style", "template", "noscript", "svg", "nav", "footer"} +_BREAKS = {"article", "main", "div", "section", "p", "br", "li", "tr", "h1", "h2", "h3", "h4"} + + +class SourcePageError(ValueError): + """A source URL was disallowed or could not provide usable HTML.""" + + +@dataclass(frozen=True) +class SourcePage: + title: str + text: str + language: str | None + attribution_url: str + content: bytes = b"" + media_type: str = "text/plain" + + +@dataclass(frozen=True) +class SourceFile: + filename: str + media_type: str + content: bytes + attribution_url: str + + +class _PageParser(HTMLParser): + def __init__(self): + super().__init__(convert_charrefs=True) + self.language = None + self.title: list[str] = [] + self.body: list[str] = [] + self._hidden: list[str] = [] + self._in_head = False + self._in_title = False + + def handle_starttag(self, tag, attrs): + if tag == "html": + self.language = normalize_language_tag(dict(attrs).get("lang")) + if tag == "head": + self._in_head = True + if tag == "title": + self._in_title = True + if tag in _HIDDEN: + self._hidden.append(tag) + if tag in _BREAKS and not self._hidden and not self._in_head: + self.body.append("\n") + + def handle_endtag(self, tag): + if tag == "head": + self._in_head = False + if tag == "title": + self._in_title = False + if self._hidden and self._hidden[-1] == tag: + self._hidden.pop() + if tag in _BREAKS and not self._hidden and not self._in_head: + self.body.append("\n") + + def handle_data(self, data): + if self._hidden: + return + if self._in_title: + self.title.append(data) + elif not self._in_head: + self.body.append(data) + + +def _is_public_unicast(address: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool: + if isinstance(address, ipaddress.IPv6Address) and address.ipv4_mapped: + return _is_public_unicast(address.ipv4_mapped) + return address.is_global and not address.is_multicast and not address.is_reserved + + +def _resolve_public_address(host: str) -> str: + try: + addresses = { + ipaddress.ip_address(result[4][0]) + for result in socket.getaddrinfo(host, 443, type=socket.SOCK_STREAM) + } + except (OSError, ValueError) as exc: + raise SourcePageError("The source host could not be resolved") from exc + if not addresses or not all(map(_is_public_unicast, addresses)): + raise SourcePageError("The source host must resolve to public addresses") + return str(min(addresses, key=lambda address: (address.version, int(address)))) + + +async def fetch_source(url: str, allowed_hosts: set[str] | None = None, authorization: str | None = None) -> SourcePage | SourceFile: + if len(url) > 4096 or any(character.isspace() or ord(character) < 32 for character in url): + raise SourcePageError("Invalid source URL") + try: + target = httpx.URL(url) + except (httpx.InvalidURL, ValueError) as exc: + raise SourcePageError("Invalid source URL") from exc + if ( + target.scheme != "https" or target.port not in (None, 443) + or target.userinfo or not target.host or target.fragment + ): + raise SourcePageError("Use a public HTTPS source URL without embedded credentials or a fragment") + host = target.raw_host.decode("ascii").lower() + if authorization and allowed_hosts is None: + raise SourcePageError("Authenticated sources require a saved host policy") + if allowed_hosts is not None and host not in allowed_hosts: + raise SourcePageError("This source host is not enabled for the provider") + headers = {"Host": host, "Accept": "*/*", "Accept-Encoding": "identity"} + if authorization: + headers["Authorization"] = authorization + try: + media_type, body, encoding = await asyncio.wait_for(_read_source(target, host, headers), timeout=20) + except asyncio.TimeoutError as exc: + raise SourcePageError("The source did not respond in time") from exc + except (httpx.HTTPError, ValueError) as exc: + if isinstance(exc, SourcePageError): + raise + raise SourcePageError("The source did not return a usable HTTPS response") from exc + + attribution_url = str(target.copy_with(query=None)) + file_types = binary_media_types() + if media_type in file_types or media_type == "application/json": + suffix = file_types.get(media_type, ".json") + raw_name = unquote(PurePosixPath(target.path).name) + stem = "".join(char for char in PurePosixPath(raw_name).stem if char.isalnum() or char in "-_")[:180] + return SourceFile((stem or "source") + suffix, media_type, body, attribution_url) + + title = unquote(PurePosixPath(target.path).name)[:200] or host + return extract_text_source(body, media_type, title, attribution_url, encoding) + + +def extract_text_source( + body: bytes, media_type: str, title: str, attribution_url: str = "", encoding: str = "utf-8", +) -> SourcePage: + if len(body) > MAX_HTML_BYTES: + raise SourcePageError("Text sources must be at most 2 MiB") + try: + content = body.decode(encoding, errors="strict") + except (LookupError, UnicodeError) as exc: + raise SourcePageError("The source does not use a supported text encoding") from exc + language = None + if media_type in _HTML_TYPES: + parser = _PageParser() + parser.feed(content) + parser.close() + lines = [" ".join(line.split()) for line in "".join(parser.body).splitlines()] + text = "\n\n".join(line for line in lines if line) + title = " ".join("".join(parser.title).split())[:200] or title + language = parser.language + else: + text = content.strip() + if not text or len(text) > MAX_EXTRACTED_CHARS or "\x00" in text: + raise SourcePageError("The source must contain between 1 and 200,000 valid text characters") + return SourcePage(title, text, language, attribution_url, body, media_type) + + +async def fetch_html_source(url: str, allowed_hosts: set[str], authorization: str | None = None) -> SourcePage: + source = await fetch_source(url, allowed_hosts, authorization) + if not isinstance(source, SourcePage): + raise SourcePageError("The source must return text") + return source + + +async def _read_source(target: httpx.URL, host: str, headers: dict[str, str]) -> tuple[str, bytes, str]: + address = await asyncio.to_thread(_resolve_public_address, host) + request = httpx.Request( + "GET", target.copy_with(host=address), headers=headers, + extensions={"sni_hostname": host, "timeout": httpx.Timeout(8).as_dict()}, + ) + # Use the transport directly so the request URL never goes through the + # HTTP client's INFO logger. Attribution omits credentials and query values. + async with httpx.AsyncHTTPTransport(trust_env=False) as transport: + async with aclosing(await transport.handle_async_request(request)) as response: + response.request = request + response.raise_for_status() + media_type = response.headers.get("content-type", "").split(";", 1)[0].strip().lower() + file_types = binary_media_types() + if media_type not in (_HTML_TYPES | _TEXT_TYPES | file_types.keys() | {"application/json"}): + raise SourcePageError("The source returned an unsupported media type") + if response.headers.get("content-encoding", "identity").lower() != "identity": + raise SourcePageError("The source must return uncompressed content") + limit = MAX_FILE_BYTES if media_type in file_types else MAX_HTML_BYTES + body = bytearray() + async for block in response.aiter_raw(): + if len(body) + len(block) > limit: + raise SourcePageError(f"The source exceeds the {limit // (1024 * 1024)} MiB download limit") + body.extend(block) + return media_type, bytes(body), response.encoding or "utf-8" diff --git a/pipeline/intake/source_intake.py b/pipeline/intake/source_intake.py new file mode 100644 index 0000000..3ee530d --- /dev/null +++ b/pipeline/intake/source_intake.py @@ -0,0 +1,463 @@ +"""Persist source payloads and register document pipeline jobs atomically.""" + +import asyncio +import hashlib +import json +import logging +import os +from dataclasses import dataclass +from io import BytesIO +from pathlib import Path +from typing import Annotated +from urllib.parse import urlsplit +from uuid import UUID, uuid4 + +from psycopg.types.json import Jsonb +from pydantic import ( + BaseModel, + ConfigDict, + Field, + StringConstraints, + field_validator, + model_validator, +) + +from pipeline import db +from pipeline.documents import document_validity +from pipeline.metadata.crop_metadata import CropSubjects, parse_crop_subjects +from pipeline.documents.document_kinds import validate_document_kind +from pipeline.network.release_policy import index_only_mode +from pipeline.metadata.knowledge_metadata import KnowledgeMetadata, LocalizedDocumentMetadata +from pipeline.metadata.localization import normalize_language_tag +from pipeline.object_storage import ObjectReference +from pipeline.vector_store import get_default_index_name + +logger = logging.getLogger(__name__) + + +class IntakeConflict(ValueError): + """The event conflicts with an existing intake receipt or running document.""" + + +class CrawlReleasePolicyConflict(IntakeConflict): + """An unchanged crawl item already has a receipt under another release policy.""" + + +class SourceDocumentMetadata(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + external_id: str = Field(min_length=1, max_length=512) + title: str = Field(min_length=1, max_length=256) + kind: str = Field(default="advisory", validate_default=True) + source_url: str | None = Field(default=None, max_length=2048) + crop_subjects: CropSubjects = Field(default_factory=list) + knowledge_metadata: KnowledgeMetadata | None = None + valid_from: str | None = None + valid_to: str | None = None + + @field_validator("kind") + @classmethod + def check_kind(cls, value: str) -> str: + return validate_document_kind(value) + + @field_validator("source_url") + @classmethod + def validate_source_url(cls, source_url: str | None) -> str | None: + if source_url is not None: + parsed = urlsplit(source_url) + if parsed.scheme not in {"https", "http"} or not parsed.hostname or parsed.username is not None: + raise ValueError("source_url must be an HTTP(S) URL without credentials") + return source_url + + @model_validator(mode="after") + def validate_metadata(self) -> "SourceDocumentMetadata": + if self.valid_from: + self.valid_from = document_validity.parse_day(self.valid_from, "valid_from") + if self.valid_to: + self.valid_to = document_validity.parse_day(self.valid_to, "valid_to") + if (self.valid_from or document_validity.today()) > self.valid_to: + raise ValueError("valid_to cannot precede valid_from") + parse_crop_subjects(json.dumps([crop.model_dump(exclude_none=True) for crop in self.crop_subjects])) + return self + + +TranslationText = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=250_000)] + + +class SourceTranslation(LocalizedDocumentMetadata): + text: TranslationText + + +class NormalizedSourceDocument(SourceDocumentMetadata): + text: str = Field(min_length=1, max_length=250_000) + language: str | None = None + translations: dict[str, TranslationText | SourceTranslation] = Field(default_factory=dict, max_length=12) + + @field_validator("language") + @classmethod + def validate_language(cls, language: str | None) -> str | None: + normalized = normalize_language_tag(language) + if language is not None and normalized is None: + raise ValueError("language must be a BCP 47 tag") + return normalized + + @field_validator("translations") + @classmethod + def validate_translations(cls, entries: dict[str, str | SourceTranslation]) -> dict[str, str | SourceTranslation]: + variants = {} + for language, translation in entries.items(): + tag = normalize_language_tag(language) + if tag is None or tag in variants: + raise ValueError("translations must have unique language tags") + variants[tag] = translation + return variants + + @model_validator(mode="after") + def validate_translation_metadata(self) -> "NormalizedSourceDocument": + if self.language in self.translations: + raise ValueError("translations must use a different language from the source") + self.knowledge_metadata = self.metadata_with_translation_labels(self.knowledge_metadata) + return self + + def metadata_with_translation_labels(self, metadata: KnowledgeMetadata | None) -> KnowledgeMetadata | None: + labels = { + tag: translation.model_dump(exclude={"text"}, exclude_none=True) + for tag, translation in self.translations.items() + if isinstance(translation, SourceTranslation) and (translation.title or translation.description) + } + if not labels: + return metadata + merged = metadata.storage_value() if metadata else {} + localized = merged.setdefault("localized", {}) + for tag, fields in labels.items(): + existing = localized.setdefault(tag, {}) + if any(field in existing and existing[field] != value for field, value in fields.items()): + raise ValueError("Translation labels conflict with localized knowledge metadata") + existing.update(fields) + return KnowledgeMetadata.model_validate(merged) + + +@dataclass(frozen=True) +class BinarySource: + filename: str + storage_uri: str + fingerprint: str + mime_type: str + size_bytes: int + parser_profile: str | None = None + + +def store_source_response(store, bucket: str, provider: str, source) -> dict: + """Keep fetched bytes private, separate from processed/downloadable artifacts.""" + checksum = hashlib.sha256(source.content).hexdigest() + # Neither a URL, credential, nor a provider-supplied filename enters the key. + provider_key = hashlib.sha256(provider.encode()).hexdigest()[:32] + ref = ObjectReference(bucket, f"private/source-responses/{provider_key}/{checksum}") + store.put(ref, BytesIO(source.content), len(source.content), "application/octet-stream") + return { + "storage_uri": ref.to_uri(), "sha256": checksum, + "media_type": source.media_type, "size_bytes": len(source.content), + } + + +def attach_source_response(conn, job_id: str, capture: dict) -> None: + """Retain all observed byte variants without exposing them as artifacts.""" + conn.execute( + "UPDATE workflows SET config = jsonb_set(config, '{private_source_responses}', " + "COALESCE(config->'private_source_responses', '{}'::jsonb) || %s::jsonb) WHERE id = %s::uuid", + (Jsonb({capture["sha256"]: capture}), job_id), + ) + + +def public_source_job(job: dict) -> dict: + """Remove private intake evidence from both job API representations.""" + result = dict(job) + for key in ("config", "config_json"): + value = result.get(key) + if not value: + continue + parsed = json.loads(value) if isinstance(value, str) else dict(value) + parsed.pop("private_source_responses", None) + result[key] = json.dumps(parsed) if isinstance(value, str) else parsed + return result + + +def canonical_json_sha256(payload: object) -> str: + pending = [payload] + while pending: + value = pending.pop() + if isinstance(value, str) and "\x00" in value: + raise ValueError("Source payload cannot contain null characters") + if isinstance(value, dict): + pending.extend(value.keys()) + pending.extend(value.values()) + elif isinstance(value, list): + pending.extend(value) + canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False, allow_nan=False) + return hashlib.sha256(canonical.encode()).hexdigest() + + +def crawl_record_key(record: object, document: SourceDocumentMetadata) -> str: + return canonical_json_sha256([record, document.model_dump(mode="json")]) + + +def get_source_receipt(conn, receipt_id: str) -> dict | None: + return conn.execute( + "SELECT r.id::text, r.document_id::text, r.job_id::text, r.external_id, " + "r.mapping_version_id::text, r.normalized_document->>'source_url' AS source_url, " + "w.status, w.config, w.temporal_workflow_id AS workflow_id " + "FROM source_intake_receipts r JOIN workflows w ON w.id = r.job_id WHERE r.id = %s::uuid", + (receipt_id,), + ).fetchone() + + +def _replay_source_receipt( + conn, receipt_id: str, source_kind: str, release_target: tuple[str, str] | None, + explicit_release_network: bool, + *, + verified_raw: bool = True, +) -> dict: + receipt = get_source_receipt(conn, receipt_id) + if not receipt: + raise IntakeConflict("The original intake document has been removed") + if source_kind == "crawl": + config = receipt["config"] + network = config.get("release_network_id") + args = config.get("upload_start_args") or [] + collection = config.get("release_collection") or (args[6] if len(args) > 6 else None) + original_target = (network, collection) if network else None + if original_target != release_target or bool(config.get("explicit_release_network")) != explicit_release_network: + conflict = CrawlReleasePolicyConflict if verified_raw else IntakeConflict + raise conflict("The existing crawl uses a different release policy. Select its revision for the intended network.") + return receipt + + +def register_source_document( + conn, *, instance: str, source_kind: str, source_id: str, idempotency_key: str, + raw_payload: object, document: SourceDocumentMetadata, release_target: tuple[str, str] | None, + mapping_version_id: str | None = None, binary: BinarySource | None = None, + legacy_crawl_key: str | None = None, + explicit_release_network: bool = False, +) -> dict: + """Create one document job, or a revision for an existing source record.""" + instance_row = conn.execute("SELECT id FROM instances WHERE lower(name) = %s", (instance,)).fetchone() + if not instance_row: + raise LookupError("Provider not found") + instance_id = instance_row["id"] + # Serialize this source across retries, including two deliveries with different keys. + lock = canonical_json_sha256([str(instance_id), source_kind, source_id]) + conn.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", (lock,)) + raw_sha = canonical_json_sha256(raw_payload) + scope = (instance_id, source_kind, source_id) + previous = conn.execute( + "SELECT id::text, raw_sha256 FROM source_intake_receipts WHERE instance_id = %s AND source_kind = %s " + "AND source_id = %s AND idempotency_key = %s", (*scope, idempotency_key), + ).fetchone() + if previous: + if previous["raw_sha256"] != raw_sha: + raise IntakeConflict("Idempotency key was already used for different content") + return _replay_source_receipt(conn, previous["id"], source_kind, release_target, explicit_release_network) + + if source_kind == "crawl": + observed = conn.execute( + "SELECT id::text, replay_evidence_sha256 FROM source_intake_receipts " + "WHERE instance_id = %s AND source_kind = %s AND source_id = %s AND replay_evidence_key = %s", + (*scope, idempotency_key), + ).fetchone() + if observed: + if observed["replay_evidence_sha256"] != raw_sha: + raise IntakeConflict("Idempotency key was already used for different content") + return _replay_source_receipt(conn, observed["id"], source_kind, release_target, explicit_release_network) + + if source_kind == "crawl" and legacy_crawl_key: + # Retain the first observed item alongside the original receipt payload. + previous = conn.execute( + "SELECT id::text FROM source_intake_receipts WHERE instance_id = %s AND source_kind = %s " + "AND source_id = %s AND idempotency_key = %s AND raw_sha256 = %s " + "AND external_id = %s AND raw_payload = normalized_document " + "AND replay_evidence_key IS NULL", + (*scope, legacy_crawl_key, legacy_crawl_key, document.external_id), + ).fetchone() + if previous: + receipt = _replay_source_receipt( + conn, previous["id"], source_kind, release_target, explicit_release_network, verified_raw=False, + ) + conn.execute( + "UPDATE source_intake_receipts SET replay_evidence_key = %s, replay_evidence_payload = %s, " + "replay_evidence_sha256 = %s, replay_evidence_observed_at = now() WHERE id = %s", + (idempotency_key, Jsonb(raw_payload), raw_sha, previous["id"]), + ) + return receipt + + prior_document = conn.execute( + "SELECT document_id::text, normalized_document->>'kind' AS kind FROM source_intake_receipts " + "WHERE instance_id = %s AND source_kind = %s " + "AND source_id = %s AND external_id = %s ORDER BY created_at DESC, id DESC LIMIT 1", + (*scope, document.external_id), + ).fetchone() + receipt_id = str(uuid4()) + workflow_id = f"intake-{receipt_id}" + if binary is None and not isinstance(document, NormalizedSourceDocument): + raise ValueError("A text submission must contain its normalized text") + filename = binary.filename if binary else f"{source_kind}-{receipt_id}.json" + filepath = binary.storage_uri if binary else f"intake://{receipt_id}" + fingerprint = binary.fingerprint if binary else raw_sha[:32] + canonical_input_type = "structured_text" + if binary: + canonical_input_type = "spreadsheet" if Path(binary.filename).suffix.lower() in {".csv", ".xlsx"} else "pdf" + normalized = document.model_dump(mode="json", exclude_none=True) + crops = [crop.model_dump(exclude_none=True) for crop in document.crop_subjects] + knowledge_metadata = document.knowledge_metadata.storage_value() if document.knowledge_metadata else {} + valid_from = document.valid_from or document_validity.today() + if prior_document: + if (prior_document["kind"] or "advisory") != document.kind: + raise IntakeConflict("An existing source document cannot change its knowledge kind") + document_id = prior_document["document_id"] + job = conn.execute( + "SELECT id::text FROM document_current_workflow WHERE document_id = %s::uuid", (document_id,), + ).fetchone() + if not job: + raise IntakeConflict("Existing document has no workflow to replace") + try: + db.reset_document_for_reupload( + document_id, filename=filename, filepath=filepath, fingerprint=fingerprint, + source_type=source_kind, canonical_input_type=canonical_input_type, crop_subjects=crops, + knowledge_metadata=knowledge_metadata, + expected_job_id=job["id"], _conn=conn, + ) + except db.DocumentReplacementConflict as exc: + raise IntakeConflict(str(exc)) from exc + db.update_document_fields( + document_id, name=document.title, valid_from=valid_from, valid_to=document.valid_to, _conn=conn, + ) + else: + stable_code = canonical_json_sha256([source_id, document.external_id])[:32] + document_id = db.upsert_document( + workflow_id, fingerprint, filename, filepath, name=document.title, source_type=source_kind, + canonical_input_type=canonical_input_type, instance=instance, valid_from=valid_from, + valid_to=document.valid_to, kind=document.kind, code=f"{source_kind}-{stable_code}", + crop_subjects=crops, knowledge_metadata=knowledge_metadata, with_workflow=False, _conn=conn, + ) + + network, collection = release_target or (None, get_default_index_name()) + args = [fingerprint, filename, filepath, 450, 128, 1, collection, True, False, True] + parser = binary.parser_profile if binary else None + if network or parser or index_only_mode(): + args.append(network) + if parser or index_only_mode(): + args.append(parser) + if index_only_mode(): + args.append(True) + job_id = db.create_document_job( + document_id, job_type="pipeline", temporal_workflow_id=workflow_id, status="start_pending", + current_stage="registered", config={ + "upload_start_args": args, "release_network_id": network, "release_collection": collection if network else None, + "explicit_release_network": explicit_release_network, + "auto_approve": True, + "source_kind": source_kind, "source_id": source_id, "source_receipt_id": receipt_id, + "mapping_version_id": mapping_version_id, + "source_url": document.source_url, + "source_task_queue": os.environ.get("TEMPORAL_TASK_QUEUE", "ocr-pipeline"), + }, _conn=conn, + ) + if binary: + db.add_document_artifact( + workflow_id=workflow_id, job_id=job_id, artifact_type="original_upload", + storage_uri=filepath, stage="registered", filename=filename, mime_type=binary.mime_type, + size_bytes=binary.size_bytes, metadata={"source_kind": source_kind}, _conn=conn, + ) + conn.execute( + "INSERT INTO source_intake_receipts (id, instance_id, source_kind, source_id, idempotency_key, " + "external_id, raw_payload, raw_sha256, normalized_document, mapping_version_id, document_id, job_id) " + "VALUES (%s::uuid, %s, %s, %s, %s, %s, %s, %s, %s, %s::uuid, %s::uuid, %s::uuid)", + (receipt_id, *scope, idempotency_key, document.external_id, Jsonb(raw_payload), raw_sha, Jsonb(normalized), + mapping_version_id, document_id, job_id), + ) + return get_source_receipt(conn, receipt_id) + + +async def _dispatch_source_document(receipt: dict, client, task_queue: str) -> bool: + """Start a committed intent using its original idempotent Temporal ID.""" + if receipt["status"] != "start_pending": + return True + if client is None: + return False + from pipeline.documents.document_dispatch import start_registered_document + + try: + await start_registered_document(client, task_queue, { + "id": receipt["job_id"], "config": receipt["config"], "temporal_workflow_id": receipt["workflow_id"], + }) + except Exception as exc: + logger.warning("Source intake start pending workflow_id=%s error=%s", receipt["workflow_id"], type(exc).__name__) + return False + return True + + +async def start_source_document(receipt: dict) -> bool: + from pipeline.api import api + + queue = receipt["config"].get("source_task_queue") or api.TASK_QUEUE + return await _dispatch_source_document(receipt, api.temporal_client, queue) + + +def _claim_pending_sources(batch_size: int, task_queue: str) -> list[dict]: + with db.get_connection() as conn: + return conn.execute( + "WITH pending AS (" + "SELECT w.id, COALESCE((w.config->>'source_dispatch_attempts')::int, 0) AS attempts " + "FROM workflows w JOIN source_intake_receipts r ON r.job_id = w.id " + "WHERE w.status = 'start_pending' AND r.created_at <= now() - interval '5 seconds' " + "AND COALESCE(w.config->>'source_task_queue', 'ocr-pipeline') = %s " + "AND COALESCE((w.config->>'source_next_dispatch_at')::timestamptz, now()) <= now() " + "ORDER BY r.created_at, w.id LIMIT %s FOR UPDATE OF w SKIP LOCKED" + ") UPDATE workflows w SET config = w.config || jsonb_build_object(" + "'source_dispatch_attempts', least(p.attempts + 1, 10), " + "'source_next_dispatch_at', now() + make_interval(secs => least(300, 10 * power(2, p.attempts)))) " + "FROM pending p WHERE w.id = p.id " + "RETURNING w.id::text AS job_id, w.status, w.config, w.temporal_workflow_id AS workflow_id", + (task_queue, batch_size), + ).fetchall() + + +async def reconcile_pending_sources(client, task_queue: str, *, batch_size: int = 16, concurrency: int = 4) -> int: + """Retry due source intents with bounded concurrency and durable backoff.""" + pending = await asyncio.to_thread(_claim_pending_sources, max(1, min(batch_size, 64)), task_queue) + slots = asyncio.Semaphore(max(1, min(concurrency, 8))) + + async def dispatch(receipt: dict) -> bool: + async with slots: + return await _dispatch_source_document(receipt, client, task_queue) + + return sum(await asyncio.gather(*(dispatch(receipt) for receipt in pending))) + + +def materialize_intake_pages(workflow_id: str, filepath: str) -> dict: + """Read a receipt only when it belongs to the workflow that is running it.""" + receipt_id = str(UUID(filepath.removeprefix("intake://"))) + with db.get_connection() as conn: + row = conn.execute( + "SELECT r.normalized_document FROM source_intake_receipts r JOIN workflows w ON w.id = r.job_id " + "JOIN document_current_workflow c ON c.id = w.id " + "WHERE r.id = %s::uuid AND w.temporal_workflow_id = %s", + (receipt_id, workflow_id), + ).fetchone() + if not row: + raise LookupError("No intake receipt for the current workflow") + db.save_pages(workflow_id, [submitted_document_page(row["normalized_document"])]) + return {"page_count": 1, "source_receipt_id": receipt_id} + + +def submitted_document_page(document: dict) -> dict: + return { + "page_number": 1, "original_markdown": document["text"], + "detected_language": document.get("language"), + "metadata": {"declared_language": document.get("language")}, + "translations": { + tag: { + "translated_markdown": variant if isinstance(variant, str) else variant["text"], + "provider": "source_intake", + } + for tag, variant in document.get("translations", {}).items() + }, + } diff --git a/pipeline/intake/webhook_mapping.py b/pipeline/intake/webhook_mapping.py new file mode 100644 index 0000000..e486e27 --- /dev/null +++ b/pipeline/intake/webhook_mapping.py @@ -0,0 +1,201 @@ +"""Bounded JSON Pointer mapping for provider-owned webhook sources.""" + +import json +import os +import re +from pathlib import Path +from typing import Annotated, Literal + +from pydantic import BaseModel, ConfigDict, Field, StringConstraints, TypeAdapter + +from pipeline.intake.source_intake import NormalizedSourceDocument, canonical_json_sha256 + + +def _reject_json_constant(value: str) -> None: + raise ValueError(f"Unsupported JSON constant: {value}") + + +def _unique_object(pairs: list[tuple[str, object]]) -> dict: + result = {} + for key, value in pairs: + if key in result: + raise ValueError("Duplicate JSON object key") + result[key] = value + return result + + +def decode_source_json(data: bytes | bytearray) -> object: + return json.loads(data.decode("utf-8"), parse_constant=_reject_json_constant, object_pairs_hook=_unique_object) + + +def parse_canonical_document(data: bytes) -> NormalizedSourceDocument: + return NormalizedSourceDocument.model_validate(decode_source_json(data)) + + +MAX_RECORDS = 250 +JsonPointer = Annotated[str, StringConstraints(max_length=512, pattern=r"^(?:/(?:[^~/]|~[01])*)*$")] + + +class Selector(BaseModel): + model_config = ConfigDict(extra="forbid") + + scope: Literal["record", "root"] = "record" + path: JsonPointer + + +class EqualsCondition(Selector): + equals: str | bool | int | None + + +class FieldMapping(BaseModel): + model_config = ConfigDict(extra="forbid") + + external_id: JsonPointer | Selector + title: JsonPointer | Selector + text: JsonPointer | Selector + kind: JsonPointer | Selector | None = None + source_url: JsonPointer | Selector | None = None + language: JsonPointer | Selector | None = None + translations: JsonPointer | Selector | None = None + crop_subjects: JsonPointer | Selector | None = None + knowledge_metadata: JsonPointer | Selector | None = None + valid_from: JsonPointer | Selector | None = None + valid_to: JsonPointer | Selector | None = None + + +class WebhookMapping(BaseModel): + model_config = ConfigDict(extra="forbid") + + runtime: Literal["json-pointer-v1"] = "json-pointer-v1" + records: JsonPointer = "" + each_root: bool = False + fields: FieldMapping + filters: list[EqualsCondition] = Field(default_factory=list, max_length=12) + + +class CanonicalMapping(BaseModel): + model_config = ConfigDict(extra="forbid") + + runtime: Literal["canonical-v1"] = "canonical-v1" + + +class JsonataMapping(BaseModel): + model_config = ConfigDict(extra="forbid") + + runtime: Literal["jsonata-v1"] + expression: str = Field(min_length=1, max_length=8192) + + +SourceMapping = CanonicalMapping | WebhookMapping | JsonataMapping +SOURCE_MAPPING = TypeAdapter(SourceMapping) +CanonicalBatch = Annotated[list[NormalizedSourceDocument], Field(min_length=1, max_length=MAX_RECORDS)] +CANONICAL_SUBMISSION = TypeAdapter(NormalizedSourceDocument | CanonicalBatch) + + +def load_intake_template(name: str, *, provider: str | None = None) -> SourceMapping: + if provider is not None: + from pipeline.intake.provider_sources import stored_settings + + row = stored_settings(provider) + if row and name in row["settings"]["templates"]: + return SOURCE_MAPPING.validate_python(row["settings"]["templates"][name]) + path = os.environ.get("KNOWLEDGE_INTAKE_TEMPLATES_FILE") + if not path: + raise ValueError("No intake templates are configured") + limit = 2 * 1024 * 1024 + with Path(path).open("rb") as handle: + data = handle.read(limit + 1) + if len(data) > limit: + raise ValueError("Intake template configuration exceeds its size limit") + try: + templates = decode_source_json(data) + except RecursionError as exc: + raise ValueError("Intake template configuration is too deeply nested") from exc + if not isinstance(templates, dict) or name not in templates: + raise ValueError("Unknown intake template") + return SOURCE_MAPPING.validate_python(templates[name]) + + +def map_source_submission(payload: object, mapping: SourceMapping) -> list[tuple[object, NormalizedSourceDocument]]: + if isinstance(mapping, WebhookMapping): + return map_webhook(payload, mapping) + if isinstance(mapping, JsonataMapping): + from pipeline.intake.jsonata_runtime import evaluate + + transformed = evaluate(mapping.expression, payload) + if transformed is None or transformed == []: + return [] + submitted = CANONICAL_SUBMISSION.validate_python(transformed) + documents = submitted if isinstance(submitted, list) else [submitted] + if len({doc.external_id for doc in documents}) != len(documents): + raise ValueError("Mapped external IDs must be unique within a delivery") + # A general expression can aggregate many inputs. Preserve the original input once + # on the delivery; receipts store its digest and the normalized result. + evidence = {"source_payload_sha256": canonical_json_sha256(payload)} + return [(evidence, document) for document in documents] + submitted = CANONICAL_SUBMISSION.validate_python(payload) + rows = payload if isinstance(payload, list) else [payload] + documents = submitted if isinstance(submitted, list) else [submitted] + if len({doc.external_id for doc in documents}) != len(documents): + raise ValueError("External IDs must be unique within a submission") + return list(zip(rows, documents)) + + +_ARRAY_INDEX = re.compile(r"0|[1-9][0-9]{0,8}") + + +def _read_path(value: object, path: str) -> object: + if not path: + return value + for token in path.split("/")[1:]: + part = token.replace("~1", "/").replace("~0", "~") + if isinstance(value, dict): + value = value.get(part) + elif isinstance(value, list) and _ARRAY_INDEX.fullmatch(part) and int(part) < len(value): + value = value[int(part)] + else: + return None + return value + + +def _select(selector: JsonPointer | Selector, payload: object, record: dict) -> object: + if isinstance(selector, str): + return _read_path(record, selector) + return _read_path(payload if selector.scope == "root" else record, selector.path) + + +def _matches(value: object, expected: object) -> bool: + return type(value) is type(expected) and value == expected + + +def map_webhook(payload: object, mapping: WebhookMapping) -> list[tuple[dict, NormalizedSourceDocument]]: + """Extract and validate the canonical fields, with no executable expressions.""" + roots = payload if mapping.each_root and isinstance(payload, list) else [payload] + if len(roots) > MAX_RECORDS: + raise ValueError(f"At most {MAX_RECORDS} root payloads are allowed") + records = [] + for root in roots: + selected = _read_path(root, mapping.records) + selected = [selected] if isinstance(selected, dict) else selected + if (not isinstance(selected, list) or len(records) + len(selected) > MAX_RECORDS + or not all(isinstance(row, dict) for row in selected)): + raise ValueError(f"The records pointer must select an object or up to {MAX_RECORDS} objects") + records.extend((root, record) for record in selected) + + outputs = [] + identifiers = set() + for root, record in records: + if not all(_matches(_select(rule, root, record), rule.equals) for rule in mapping.filters): + continue + values = { + field: _select(getattr(mapping.fields, field), root, record) + for field in FieldMapping.model_fields if getattr(mapping.fields, field) is not None + } + if type(values.get("external_id")) is int: + values["external_id"] = str(values["external_id"]) + document = NormalizedSourceDocument.model_validate({k: v for k, v in values.items() if v is not None}) + if document.external_id in identifiers: + raise ValueError("Mapped external IDs must be unique within a delivery") + identifiers.add(document.external_id) + outputs.append((record, document)) + return outputs diff --git a/pipeline/metadata/__init__.py b/pipeline/metadata/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/pipeline/metadata/crop_metadata.py b/pipeline/metadata/crop_metadata.py new file mode 100644 index 0000000..45afec5 --- /dev/null +++ b/pipeline/metadata/crop_metadata.py @@ -0,0 +1,84 @@ +"""Contributor-supplied document crop context using agriculture subject references.""" + +from typing import Annotated + +from pydantic import ( + AnyUrl, + BaseModel, + ConfigDict, + Field, + TypeAdapter, + field_validator, + model_validator, +) + + +class CropDescriptor(BaseModel): + """Supported identification fields from the Beckn Descriptor.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + code: str | None = Field(default=None, min_length=1, max_length=256) + name: str | None = Field(default=None, min_length=1, max_length=256) + + @model_validator(mode="after") + def identified(self) -> "CropDescriptor": + if not self.code and not self.name: + raise ValueError("A crop descriptor needs a code or name") + return self + + +class AgricultureSubjectReference(BaseModel): + """AgricultureResource v0.1 reference; does not assert taxonomy verification.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + subjectId: str | None = Field(default=None, min_length=1, max_length=2048) + descriptor: CropDescriptor | None = None + + @field_validator("subjectId") + @classmethod + def absolute_subject_uri(cls, value: str | None) -> str | None: + if value is not None: + if "\\" in value or any(character.isspace() for character in value): + raise ValueError("subjectId must be an absolute URI without spaces or backslashes") + TypeAdapter(AnyUrl).validate_python(value) + return value + + @model_validator(mode="after") + def identified(self) -> "AgricultureSubjectReference": + if not self.subjectId and self.descriptor is None: + raise ValueError("A crop subject needs subjectId or descriptor") + return self + + +CropSubjects = Annotated[list[AgricultureSubjectReference], Field(max_length=32)] + + +def parse_crop_subjects(raw: str | None) -> list[dict] | None: + """Distinguish omitted metadata from an explicitly empty crop list.""" + if raw is None: + return None + if len(raw) > 32768: + raise ValueError("crop_subjects must be at most 32768 characters") + subjects = TypeAdapter(CropSubjects).validate_json(raw) + result = [subject.model_dump(exclude_none=True) for subject in subjects] + if len({subject.model_dump_json(exclude_none=True) for subject in subjects}) != len(result): + raise ValueError("Crop subject references must be unique") + return result + + +def parse_crop_subject_ids(value: object) -> list[str]: + """Validate the bounded OR-list used to filter explicit crop identifiers.""" + if value is None: + return [] + if not isinstance(value, list) or len(value) > 32: + raise ValueError("crop_subject_ids must be a list of at most 32 absolute URI strings") + identifiers = [] + for item in value: + if not isinstance(item, str): + raise ValueError("Each crop_subject_ids entry must be an absolute URI string") + reference = AgricultureSubjectReference(subjectId=item) + if reference.subjectId not in identifiers: + identifiers.append(reference.subjectId) + return identifiers diff --git a/pipeline/metadata/knowledge_metadata.py b/pipeline/metadata/knowledge_metadata.py new file mode 100644 index 0000000..e4f6fdf --- /dev/null +++ b/pipeline/metadata/knowledge_metadata.py @@ -0,0 +1,143 @@ +"""Document metadata shared by intake and retrieval.""" + +import json +from datetime import datetime +from typing import Annotated + +from pydantic import AwareDatetime, BaseModel, ConfigDict, Field, StringConstraints, field_validator + +from pipeline.metadata.crop_metadata import ( + CropSubjects, + parse_crop_subject_ids, + parse_crop_subjects, +) +from pipeline.metadata.localization import normalize_language_tag +from pipeline.metadata.schema_packs import SchemaSelection + +Label = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=128)] +Description = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=4000)] + + +class LocalizedDocumentMetadata(BaseModel): + model_config = ConfigDict(extra="forbid") + + title: Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=300)] | None = None + description: Description | None = None + + +class AdministrativeArea(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + codeScheme: str = Field(pattern=r"^[A-Za-z0-9_.-]{1,64}$") + areaCode: str = Field(min_length=1, max_length=128) + areaLevel: Label | None = None + areaName: Label | None = None + + def lookup_key(self) -> str: + return f"{self.codeScheme.casefold()}:{self.areaCode.casefold()}" + + +class ApplicabilityGroup(BaseModel): + model_config = ConfigDict(extra="forbid") + + coverage_areas: list[AdministrativeArea] = Field(min_length=1, max_length=32) + seasons: list[Label] = Field(min_length=1, max_length=12) + + def index_value(self) -> dict[str, list[str]]: + return { + "coverage_area_keys": list(dict.fromkeys(area.lookup_key() for area in self.coverage_areas)), + "season_keys": list(dict.fromkeys(season.casefold() for season in self.seasons)), + } + + +class KnowledgeMetadata(BaseModel): + model_config = ConfigDict(extra="forbid") + + description: Description | None = None + localized: dict[str, LocalizedDocumentMetadata] = Field(default_factory=dict, max_length=16) + subject_categories: list[Label] = Field(default_factory=list, max_length=32) + agriculture_subjects: CropSubjects = Field(default_factory=list) + coverage_areas: list[AdministrativeArea] = Field(default_factory=list, max_length=32) + excluded_coverage_areas: list[AdministrativeArea] = Field(default_factory=list, max_length=32) + seasons: list[Label] = Field(default_factory=list, max_length=12) + broad_applicability: bool = False + applicability_groups: list[ApplicabilityGroup] = Field(default_factory=list, max_length=32) + topics: list[Label] = Field(default_factory=list, max_length=24) + tags: list[Label] = Field(default_factory=list, max_length=16) + issued_at: AwareDatetime | None = None + schema_pack: SchemaSelection | None = None + + @field_validator("localized") + @classmethod + def localized_languages(cls, entries: dict[str, LocalizedDocumentMetadata]) -> dict[str, LocalizedDocumentMetadata]: + normalized = {} + for language, entry in entries.items(): + tag = normalize_language_tag(language) + if not tag or tag in normalized or not (entry.title or entry.description): + raise ValueError("Localized metadata requires distinct language tags and nonempty content") + normalized[tag] = entry + return normalized + + @field_validator("subject_categories", "seasons", "topics", "tags") + @classmethod + def distinct_labels(cls, labels: list[str]) -> list[str]: + unique = {} + for label in labels: + unique.setdefault(label.casefold(), label) + return list(unique.values()) + + @field_validator("agriculture_subjects") + @classmethod + def distinct_subjects(cls, subjects): + parse_crop_subjects(json.dumps([subject.model_dump(exclude_none=True) for subject in subjects])) + return subjects + + @field_validator("coverage_areas", "excluded_coverage_areas") + @classmethod + def distinct_areas(cls, areas: list[AdministrativeArea]) -> list[AdministrativeArea]: + if len({area.lookup_key() for area in areas}) != len(areas): + raise ValueError("Coverage areas must have distinct scheme and area codes") + return areas + + @field_validator("issued_at", mode="before") + @classmethod + def timestamp_string(cls, value: object) -> object: + if value is not None and not isinstance(value, (str, datetime)): + raise ValueError("issued_at must be an ISO timestamp with timezone") + return value + + def storage_value(self) -> dict: + return self.model_dump(mode="json", exclude_none=True, exclude_defaults=True) + + +def parse_knowledge_metadata(value: str | None) -> dict | None: + if value is None: + return None + if len(value) > 65_536: + raise ValueError("Knowledge metadata must be at most 65,536 characters") + return KnowledgeMetadata.model_validate_json(value).storage_value() + + +class KnowledgeFilters(BaseModel): + model_config = ConfigDict(extra="forbid") + + subject_categories: list[Label] = Field(default_factory=list, max_length=32) + agriculture_subject_ids: list[str] = Field(default_factory=list, max_length=32) + coverage_areas: list[AdministrativeArea] = Field(default_factory=list, max_length=32) + seasons: list[Label] = Field(default_factory=list, max_length=12) + tags: list[Label] = Field(default_factory=list, max_length=16) + + @field_validator("agriculture_subject_ids") + @classmethod + def subject_identifiers(cls, value: list[str]) -> list[str]: + return parse_crop_subject_ids(value) + + def qdrant_values(self) -> dict[str, list[str]]: + filters = { + "subject_categories": self.subject_categories, + "agriculture_subject_ids": self.agriculture_subject_ids, + "coverage_area_keys": [area.lookup_key() for area in self.coverage_areas], + "season_keys": [season.casefold() for season in self.seasons], + "tag_keys": [tag.casefold() for tag in self.tags], + } + return {key: list(dict.fromkeys(values)) for key, values in filters.items() if values} diff --git a/pipeline/metadata/localization.py b/pipeline/metadata/localization.py new file mode 100644 index 0000000..c520d59 --- /dev/null +++ b/pipeline/metadata/localization.py @@ -0,0 +1,167 @@ +"""Language and source-evidence fields for published passages.""" + +import json +import re + +# RFC 5646 syntax. Tags are compared case-insensitively and stored lowercase. +_LANGUAGE_TAG = re.compile( + r"(?:(?:[a-z]{2,3}(?:-[a-z]{3}){0,3}|[a-z]{4,8})" + r"(?:-[a-z]{4})?(?:-(?:[a-z]{2}|[0-9]{3}))?" + r"(?:-(?:[a-z0-9]{5,8}|[0-9][a-z0-9]{3}))*" + r"(?:-[a-wy-z0-9](?:-[a-z0-9]{2,8})+)*(?:-x(?:-[a-z0-9]{1,8})+)?" + r"|x(?:-[a-z0-9]{1,8})+)", + re.ASCII, +) +_GRANDFATHERED_TAGS = frozenset({ + "en-gb-oed", "i-ami", "i-bnn", "i-default", "i-enochian", "i-hak", "i-klingon", + "i-lux", "i-mingo", "i-navajo", "i-pwn", "i-tao", "i-tay", "i-tsu", "sgn-be-fr", + "sgn-be-nl", "sgn-ch-de", "art-lojban", "cel-gaulish", "no-bok", "no-nyn", "zh-guoyu", + "zh-hakka", "zh-min", "zh-min-nan", "zh-xiang", +}) + + +def normalize_language_tag(value: object) -> str | None: + """Accept one language tag, using a bounded case-insensitive lookup key.""" + if not isinstance(value, str) or len(value) > 255 or not value.isascii(): + return None + tag = value.strip().lower() + if tag in _GRANDFATHERED_TAGS or _LANGUAGE_TAG.fullmatch(tag): + return tag + return None + + +def _source_spans(chunk: dict) -> list[dict]: + spans = chunk.get("source_spans_json") or [] + if isinstance(spans, str): + spans = json.loads(spans) + return [ + span for span in spans + if isinstance(span, dict) and type(span.get("page_number")) is int + ] + + +def _page_variants(page: dict, language: str, text: str) -> dict: + variants = {language: {"text": text, "is_reviewed": bool(page.get("is_reviewed"))}} + for key, translation in (page.get("translations") or {}).items(): + tag = normalize_language_tag(key) + content = translation.get("edited_translation") or translation.get("translated_markdown") or "" + if tag and tag not in variants and content.strip(): + variants[tag] = {"text": content.strip(), "is_reviewed": bool(translation.get("is_reviewed"))} + return variants + + +def _matched_spans(spans: list[dict], page_texts: dict[int, str], text: str) -> list[dict]: + excerpts = [] + for span in spans: + page_text = page_texts.get(span["page_number"], "") + start, end = span.get("start_char"), span.get("end_char") + if type(start) is not int or type(end) is not int or not 0 <= start < end <= len(page_text): + return [] + excerpts.append(page_text[start:end]) + return spans if " ".join(excerpts).split() == text.split() else [] + + +def project_chunk_localization(chunk: dict, pages: list[dict], *, source_first: bool = False) -> dict: + """Project language metadata from a chunk created from the source pages. + + Page translations carry page alignment only. A chunk has a complete + translation when its text covers every cited source page in full. + """ + if not source_first or not pages: + return {} + + from pipeline.chunking.recursive_splitter import clean_and_format_text + + spans = _source_spans(chunk) + page_start = chunk.get("page_start") or 1 + page_end = chunk.get("page_end") or page_start + numbers = sorted({span["page_number"] for span in spans}) or list(range(page_start, page_end + 1)) + source_pages = {page["page_number"]: page for page in pages if page["page_number"] in numbers} + primary_text = chunk.get("original_text") or "" + text = chunk.get("edited_text") or primary_text + languages = set() + evidence = [] + comparable_pages = {} + page_variants = [] + recursive = chunk.get("chunking_provider") == "recursive_splitter" + for number in numbers: + page = source_pages.get(number) + if page is None: + languages.add("und") + continue + language = normalize_language_tag(page.get("detected_language")) or "und" + languages.add(language) + page_text = page.get("edited_markdown") or page.get("original_markdown") or "" + variants = _page_variants(page, language, page_text) + page_variants.append(variants) + evidence.append({ + "page_number": number, + "source_language": language, + "translation_languages": sorted(variants.keys() - {language}), + }) + comparable_pages[number] = clean_and_format_text(page_text) if recursive else page_text + + primary_language = next(iter(languages)) if len(languages) == 1 else "mul" + is_edited = bool(chunk.get("edited_text") and chunk["edited_text"] != chunk.get("original_text")) + matched_spans = _matched_spans(spans, comparable_pages, primary_text) + if matched_spans: + for reference in evidence: + reference["span_basis"] = "normalized_source_text" if recursive else "source_markdown" + reference["spans"] = [ + {"start_char": span["start_char"], "end_char": span["end_char"]} + for span in matched_spans if span["page_number"] == reference["page_number"] + ] + translations = {} + covers_full_pages = ( + len(page_variants) == len(numbers) + and bool(text.strip()) + and text.split() == " ".join(comparable_pages.values()).split() + ) + if covers_full_pages: + shared_languages = set(page_variants[0]).intersection(*page_variants[1:]) + for tag in sorted(shared_languages - {primary_language, "und", "mul"}): + translations[tag] = { + "text": "\n\n".join(variants[tag]["text"] for variants in page_variants), + "is_reviewed": all(variants[tag]["is_reviewed"] for variants in page_variants), + } + + text_language = "und" if is_edited and not covers_full_pages else primary_language + return { + "primary_language": primary_language, + "text_language": text_language, + "primary_text": primary_text, + "translations": translations, + "source_evidence": evidence, + "available_languages": sorted({text_language, primary_language, *translations}), + } + + +def localize_search_hit(hit: dict, language: str, *, strict: bool = False) -> dict | None: + """Resolve exact, less-specific, or primary text while reporting its actual language.""" + text_language = normalize_language_tag(hit.get("text_language")) or "und" + primary_language = normalize_language_tag(hit.get("primary_language")) or text_language + primary_text = hit.get("primary_text") or hit.get("text") or "" + variants = { + **(hit.get("translations") or {}), + primary_language: {"text": primary_text}, + text_language: {"text": hit.get("text") or ""}, + } + candidate = language + while candidate: + variant = variants.get(candidate) + if variant: + return { + **hit, "text": variant["text"], "text_language": candidate, + **({"text_is_reviewed": variant["is_reviewed"]} if "is_reviewed" in variant else {}), + "language_match": "exact" if candidate == language else "lookup", + "language_fallback": candidate != language, + } + if strict: + return None + candidate = candidate.rpartition("-")[0] + if len(candidate.rpartition("-")[2]) == 1: + candidate = candidate.rpartition("-")[0] + return { + **hit, "text": primary_text, "text_language": primary_language, + "language_match": "primary", "language_fallback": True, + } diff --git a/pipeline/metadata/schema_packs.py b/pipeline/metadata/schema_packs.py new file mode 100644 index 0000000..437b2f2 --- /dev/null +++ b/pipeline/metadata/schema_packs.py @@ -0,0 +1,182 @@ +"""Validate a schema pack against its approved, offline dependency bundle.""" + +from __future__ import annotations + +import hashlib +import json +import re +from pathlib import Path +from urllib.parse import urldefrag, urljoin, urlsplit + +import yaml +from jsonschema import Draft202012Validator +from jsonschema.exceptions import ValidationError +from pydantic import BaseModel, ConfigDict, Field +from referencing import Registry +from referencing.exceptions import Unresolvable +from referencing.jsonschema import DRAFT202012 + +_MAX_FILE_SIZE = 512 * 1024 +_MAX_BUNDLE_SIZE = 2 * 1024 * 1024 + + +def _encoded(value: object) -> bytes: + encoder = json.JSONEncoder(sort_keys=True, ensure_ascii=False, separators=(",", ":"), allow_nan=False) + encoded = bytearray() + for chunk in encoder.iterencode(value): + encoded.extend(chunk.encode()) + if len(encoded) > _MAX_BUNDLE_SIZE: + raise ValueError("The schema bundle exceeds its size limit") + return bytes(encoded) + + +class SchemaSelection(BaseModel): + model_config = ConfigDict(extra="forbid") + + network_id: str = Field(min_length=1, max_length=128, pattern=r"^\S+$") + capability: str = Field(pattern=r"^[A-Za-z][A-Za-z0-9_-]*:[A-Za-z][A-Za-z0-9_-]*$") + digest: str = Field(pattern=r"^[0-9a-f]{64}$") + attributes: dict + + def validate_registered_pack(self, conn) -> None: + if len(_encoded(self.attributes)) > 65_536: + raise ValueError("Schema attributes exceed the size limit") + get_schema_pack(conn, self.network_id, self.capability, self.digest).validate_attributes(self.attributes) + + +def validate_schema_selection(conn, metadata: dict | None) -> None: + selection = (metadata or {}).get("schema_pack") + if selection is not None: + SchemaSelection.model_validate(selection).validate_registered_pack(conn) + + +class SchemaPackManifest(BaseModel): + model_config = ConfigDict(extra="forbid") + + capability: str = Field(pattern=r"^[A-Za-z][A-Za-z0-9_-]*:[A-Za-z][A-Za-z0-9_-]*$") + version: str = Field(pattern=r"^v[0-9]+\.[0-9]+(?:\.[0-9]+)?$") + entrypoint: str + component: str = Field(pattern=r"^[A-Za-z0-9_-]+$") + files: dict[str, str] = Field(min_length=1, max_length=32) + + +def _load_file(root: Path, relative_path: str) -> dict: + path = (root / relative_path).resolve() + if not path.is_relative_to(root) or path.suffix not in {".yaml", ".yml", ".json"}: + raise ValueError("Each schema file must be a YAML or JSON artifact inside the bundle") + with path.open("rb") as handle: + contents = handle.read(_MAX_FILE_SIZE + 1) + if len(contents) > _MAX_FILE_SIZE: + raise ValueError("A schema artifact exceeds its size limit") + value = yaml.safe_load(contents) + if not isinstance(value, dict): + raise ValueError("A schema artifact must contain an object") + return value + + +def _references(value): + if isinstance(value, dict): + for key, item in value.items(): + if key in {"$ref", "$dynamicRef"}: + yield item + elif isinstance(item, (dict, list)): + yield from _references(item) + elif isinstance(value, list): + for item in value: + yield from _references(item) + + +class SchemaPack(BaseModel): + model_config = ConfigDict(extra="forbid") + + capability: str + version: str + entrypoint: str + component: str + schemas: dict[str, dict] + + @property + def digest(self) -> str: + return hashlib.sha256(_encoded(self.model_dump())).hexdigest() + + def _registry(self) -> Registry: + return Registry().with_resources( + (uri, DRAFT202012.create_resource(schema)) for uri, schema in self.schemas.items() + ) + + def check(self) -> None: + _encoded(self.model_dump()) + if self.entrypoint not in self.schemas or not re.fullmatch(r"[A-Za-z0-9_-]+", self.component): + raise ValueError("The schema entrypoint or component is invalid") + try: + component = self.schemas[self.entrypoint]["components"]["schemas"][self.component] + except (KeyError, TypeError): + raise ValueError("The selected schema component is absent") from None + Draft202012Validator.check_schema(component) + + registry = self._registry() + for uri, artifact in self.schemas.items(): + target = urlsplit(uri) + if target.scheme != "https" or not target.hostname or target.username or target.query or target.fragment: + raise ValueError("Schema artifact identifiers must be absolute HTTPS URLs") + for reference in _references(artifact): + if not isinstance(reference, str) or urldefrag(urljoin(uri, reference)).url not in self.schemas: + raise ValueError("A schema references an artifact outside its approved bundle") + try: + registry.resolver(uri).lookup(reference) + except Unresolvable: + raise ValueError("A schema references an unknown bundled component") from None + + def validate_attributes(self, attributes: dict) -> None: + reference = f"{self.entrypoint}#/components/schemas/{self.component}" + validator = Draft202012Validator( + {"type": "object", "$ref": reference}, + registry=self._registry(), + format_checker=Draft202012Validator.FORMAT_CHECKER, + ) + try: + validator.validate(attributes) + except ValidationError as exc: + field = ".".join(map(str, exc.absolute_path)) or "attributes" + raise ValueError(f"Schema pack {self.capability} rejected {field} ({exc.validator})") from None + + +def load_schema_pack(manifest_path: Path) -> SchemaPack: + root = manifest_path.resolve().parent + manifest = SchemaPackManifest.model_validate(_load_file(root, manifest_path.name)) + pack = SchemaPack( + **manifest.model_dump(exclude={"files"}), + schemas={uri: _load_file(root, path) for uri, path in manifest.files.items()}, + ) + pack.check() + return pack + + +def install_schema_pack(conn, network_id: str, pack: SchemaPack) -> str: + """Store a checked bundle without replacing a prior digest.""" + from psycopg.types.json import Jsonb + + pack.check() + digest = pack.digest + conn.execute( + """ + INSERT INTO schema_pack_versions (network_id, capability, digest, version, bundle) + VALUES (%s, %s, %s, %s, %s) + ON CONFLICT (network_id, capability, digest) DO NOTHING + """, + (network_id, pack.capability, digest, pack.version, Jsonb(pack.model_dump())), + ) + return digest + + +def get_schema_pack(conn, network_id: str, capability: str, digest: str) -> SchemaPack: + row = conn.execute( + "SELECT bundle FROM schema_pack_versions WHERE network_id = %s AND capability = %s AND digest = %s", + (network_id, capability, digest), + ).fetchone() + if not row: + raise ValueError("The selected schema pack is not installed") + pack = SchemaPack.model_validate(row["bundle"]) + if pack.digest != digest: + raise ValueError("The stored schema pack does not match its pinned digest") + return pack diff --git a/pipeline/migrate.py b/pipeline/migrate.py index fe1a870..97909fa 100644 --- a/pipeline/migrate.py +++ b/pipeline/migrate.py @@ -1,9 +1,4 @@ -"""Versioned SQL migration runner. - -Applies ``migrations/NNNN_.sql`` files in order, once each, recording -them in ``schema_migrations``. A session-level advisory lock serialises -concurrent api/worker startups so only one process applies a given file. -""" +"""Install the active schema and apply later numbered migrations once.""" import logging import re @@ -16,6 +11,16 @@ MIGRATIONS_DIR = Path(__file__).resolve().parent.parent / "migrations" _MIGRATION_FILE_RE = re.compile(r"^(\d{4})_[a-z0-9_]+\.sql$") _ADVISORY_LOCK_KEY = 7_204_911_338_105_221 +_BOOTSTRAP_VERSION = "0022" +_BOOTSTRAP_FILES = ( + "00_shared.sql", + "10_instances_access.sql", + "20_documents_processing.sql", + "25_workflows.sql", + "30_intake_source_evidence.sql", + "40_search_audit_catalog.sql", + "50_network_registry.sql", +) def list_migrations(directory: Path = MIGRATIONS_DIR) -> list[tuple[str, Path]]: @@ -24,12 +29,39 @@ def list_migrations(directory: Path = MIGRATIONS_DIR) -> list[tuple[str, Path]]: match = _MIGRATION_FILE_RE.match(path.name) if not match: raise ValueError(f"Migration file name must be NNNN_name.sql: {path.name}") - found.append((match.group(1), path)) + version = match.group(1) + if found and found[-1][0] == version: + raise ValueError(f"Duplicate migration version: {version}") + found.append((version, path)) return found +def _has_bootstrap(directory: Path) -> bool: + return directory == MIGRATIONS_DIR or (directory / "bootstrap").is_dir() + + +def _install_bootstrap(conn: psycopg.Connection, directory: Path, done: set[str]) -> None: + has_documents = conn.execute("SELECT to_regclass('public.documents')").fetchone()[0] is not None + if done or has_documents: + raise RuntimeError("Pre-release schema detected. Run the archived Postgres upgrades from the migration kit first.") + + with conn.transaction(): + for name in _BOOTSTRAP_FILES: + conn.execute((directory / "bootstrap" / name).read_text(encoding="utf-8")) + conn.execute( + "INSERT INTO schema_migrations (version, name) VALUES (%s, %s)", + (_BOOTSTRAP_VERSION, "active_schema_baseline"), + ) + logger.info("installed the grouped application schema at version %s", _BOOTSTRAP_VERSION) + + def apply_migrations(conninfo: str, directory: Path = MIGRATIONS_DIR) -> list[str]: - """Apply every pending migration. Returns the versions applied by this call.""" + """Apply the baseline and pending migrations under one startup lock.""" + migrations = list_migrations(directory) + has_bootstrap = _has_bootstrap(directory) + if has_bootstrap and migrations and migrations[0][0] <= _BOOTSTRAP_VERSION: + raise ValueError("Runtime migrations must follow the active schema baseline") + applied: list[str] = [] with psycopg.connect(conninfo, autocommit=True) as conn: conn.execute("SELECT pg_advisory_lock(%s)", (_ADVISORY_LOCK_KEY,)) @@ -44,7 +76,10 @@ def apply_migrations(conninfo: str, directory: Path = MIGRATIONS_DIR) -> list[st """ ) done = {row[0] for row in conn.execute("SELECT version FROM schema_migrations")} - for version, path in list_migrations(directory): + if has_bootstrap and _BOOTSTRAP_VERSION not in done: + _install_bootstrap(conn, directory, done) + applied.append(_BOOTSTRAP_VERSION) + for version, path in migrations: if version in done: continue with conn.transaction(): @@ -71,6 +106,10 @@ def current_version(conninfo: str) -> str | None: def is_current(conninfo: str, directory: Path = MIGRATIONS_DIR) -> bool: migrations = list_migrations(directory) - if not migrations: + if migrations: + expected = migrations[-1][0] + elif _has_bootstrap(directory): + expected = _BOOTSTRAP_VERSION + else: return True - return current_version(conninfo) == migrations[-1][0] + return current_version(conninfo) == expected diff --git a/pipeline/network/__init__.py b/pipeline/network/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/pipeline/network/adapters/__init__.py b/pipeline/network/adapters/__init__.py new file mode 100644 index 0000000..9980043 --- /dev/null +++ b/pipeline/network/adapters/__init__.py @@ -0,0 +1 @@ +"""Protocol adapters for network publication.""" diff --git a/pipeline/network/adapters/beckn.py b/pipeline/network/adapters/beckn.py new file mode 100644 index 0000000..7a42a3d --- /dev/null +++ b/pipeline/network/adapters/beckn.py @@ -0,0 +1,118 @@ +"""Beckn catalog publication and Registry admission.""" + +import hashlib +import json + +from pipeline.auth.tenancy import group_for_instance +from pipeline.network.catalog.catalog_builder import is_publishable_kind +from pipeline.network.catalog.catalog_profile import load_catalog_profile +from pipeline.network.catalog.discovery_publish_service import ( + DiscoveryPublishService, + PreparedCatalog, +) +from pipeline.network.network_connections import NetworkConnection, PublishConfigError +from pipeline.network.network_constants import RESULT_REJECTED +from pipeline.network.network_group import load_provider_naming, named_group, provider_for +from pipeline.network.network_provider import InvalidProviderIdError +from pipeline.network.publication_admission import publication_admission_evidence +from pipeline.metadata.schema_packs import get_schema_pack +from pipeline.network.adapters.contracts import PublicationReceipt, PublicationRequest + + +class BecknNetworkPublisher: + def __init__(self, connection: NetworkConnection) -> None: + self.connection = connection + self._prepared: tuple[DiscoveryPublishService, PreparedCatalog | None] | None = None + + def validate(self) -> None: + load_provider_naming(self.connection.network_id) + + def supports_kind(self, kind: str) -> bool: + return is_publishable_kind(kind) + + def provider_id(self, kind: str, provider_code: str | None) -> str | None: + naming = load_provider_naming(self.connection.network_id) + provider = provider_for(naming, kind, named_group(group_for_instance(provider_code))) + return provider["id"] if provider else None + + def prepare_delivery(self, request: PublicationRequest) -> str: + try: + service = DiscoveryPublishService(connection=self.connection) + prepared = service.prepare(request.kind, request.group) + except ValueError as exc: + raise PublishConfigError("The Beckn publication policy is invalid") from exc + provider = (prepared.catalog.get("provider") or {}) if prepared else {} + if request.provider_id and provider.get("id") != request.provider_id: + raise PublishConfigError("The Beckn catalog provider no longer matches the staged release") + self._prepared = (service, prepared) + policy = { + "endpoint": service.endpoint.rstrip("/"), + "context": prepared.context if prepared else None, + "catalog": prepared.catalog if prepared else None, + "directive": prepared.directive if prepared else None, + "schema_pin": ( + prepared.projection.schema_pack.model_dump() + if prepared and prepared.projection.schema_pack else None + ), + } + return hashlib.sha256(json.dumps(policy, sort_keys=True, separators=(",", ":")).encode()).hexdigest() + + def publication_evidence(self, conn, request: PublicationRequest) -> str | None: + prepared = self._prepared[1] if self._prepared else None + projection = prepared.projection if prepared else None + if self._prepared is None: + projection = load_catalog_profile().get(request.kind) + if projection and projection.schema_pack: + try: + pin = projection.schema_pack + declared = projection.resource_attributes.get("@type") + declared_types = declared if isinstance(declared, list) else [declared] + if pin.capability not in declared_types: + raise ValueError("The pinned capability does not match the catalog resource type") + pack = get_schema_pack(conn, request.network_id, pin.capability, pin.digest) + pack.validate_attributes(projection.resource_attributes) + except ValueError as exc: + raise PublishConfigError("The network catalog does not match its pinned schema pack") from exc + participant = (prepared.catalog.get("provider") or {}).get("id") if prepared else None + return publication_admission_evidence( + request.network_id, request.kind, request.group, projection, participant_id=participant, + ) + + def publish_revision(self, request: PublicationRequest) -> PublicationReceipt: + return self.publish(request.transaction_id, request.kind, request.workflow_id, request.group) + + def publish(self, transaction_id: str, document_kind: str, workflow_id: str, group: str | None) -> PublicationReceipt: + try: + if self._prepared: + service, prepared = self._prepared + if prepared is None: + return PublicationReceipt() + result = service.publish_prepared(transaction_id, prepared, workflow_id) + else: + result = DiscoveryPublishService(connection=self.connection).publish( + transaction_id, document_kind, workflow_id, group, + ) + except InvalidProviderIdError as exc: + raise PublishConfigError(str(exc)) from exc + + if result["skipped"]: + return PublicationReceipt() + + status = result["result_status"] + rejection = None + if status == RESULT_REJECTED: + rejection = f"Discovery Service rejected catalog for {document_kind}: {result['errors']}" + return PublicationReceipt( + exchange={ + "request": result["envelope"], + "response_status": result["status_code"], + "response_body": result["response_body"], + "result_status": status, + "context_matches": result.get("context_matches", False), + "errors": result["errors"], + }, + message_id=result["envelope"]["context"]["messageId"], + result_status=status, + rejection=rejection, + context_matches=result.get("context_matches", False), + ) diff --git a/pipeline/network/adapters/contracts.py b/pipeline/network/adapters/contracts.py new file mode 100644 index 0000000..53a06af --- /dev/null +++ b/pipeline/network/adapters/contracts.py @@ -0,0 +1,45 @@ +"""Contract for a deployment-installed network adapter.""" + +from dataclasses import dataclass +from typing import Protocol, runtime_checkable + + +@dataclass(frozen=True) +class PublicationReceipt: + """A correlated outcome and a transport record safe to store as an artifact.""" + + exchange: dict | None = None + message_id: str | None = None + result_status: str | None = None + rejection: str | None = None + context_matches: bool = False + + +@dataclass(frozen=True) +class PublicationRequest: + """Stable delivery identity and the staged snapshot for revision-based adapters.""" + + network_id: str + transaction_id: str + document_id: str + workflow_id: str + kind: str + group: str | None + revision_id: str | None + provider_id: str | None + revision: dict | None + + +@runtime_checkable +class NetworkAdapter(Protocol): + """Protocol-specific publication policy, identity, and delivery.""" + + def validate(self) -> None: ... + + def supports_kind(self, kind: str) -> bool: ... + + def provider_id(self, kind: str, provider_code: str | None) -> str | None: ... + + def publication_evidence(self, conn, request: PublicationRequest) -> str | None: ... + + def publish_revision(self, request: PublicationRequest) -> PublicationReceipt: ... diff --git a/pipeline/network/catalog/__init__.py b/pipeline/network/catalog/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/pipeline/network/catalog/catalog_builder.py b/pipeline/network/catalog/catalog_builder.py new file mode 100644 index 0000000..f8836ff --- /dev/null +++ b/pipeline/network/catalog/catalog_builder.py @@ -0,0 +1,58 @@ +"""Build a Beckn catalog from a deployment's resource projections.""" + +import copy +from typing import Optional + +from pipeline.network.catalog.catalog_profile import CatalogProjection, load_catalog_profile +from pipeline.network.network_constants import DEFAULT_DOCUMENT_KIND +from pipeline.network.network_provider import Group + + +def normalize_document_kind(document_kind: Optional[str]) -> str: + """Normalize the stored document kind for lookup.""" + return (document_kind or DEFAULT_DOCUMENT_KIND).strip().lower() + + +def is_publishable_kind(document_kind: Optional[str]) -> bool: + return normalize_document_kind(document_kind) in load_catalog_profile() + + +def build_catalog( + document_kind: Optional[str], + provider: dict, + catalog_name: str, + group: Optional[Group] = None, + *, + spec: CatalogProjection | None = None, +) -> Optional[dict]: + """Return the catalog to announce for `document_kind`, or None. + + None means the profile has no entry for this kind. The transport skips + publication rather than sending an empty catalogs array. + """ + spec = spec or load_catalog_profile().get(normalize_document_kind(document_kind)) + if spec is None: + return None + + catalog_id = spec.catalog_id + resource_id = spec.resource_id + provider = copy.deepcopy(provider) + if group is not None: + prefix = f"{group.code}-" + catalog_id = prefix + catalog_id + resource_id = prefix + resource_id + + catalog = { + "id": catalog_id, + "descriptor": {"name": catalog_name}, + "isActive": True, + "provider": provider, + "resources": [ + { + "id": resource_id, + "descriptor": {"name": spec.resource_name}, + "resourceAttributes": copy.deepcopy(spec.resource_attributes), + } + ], + } + return catalog diff --git a/pipeline/network/catalog/catalog_profile.py b/pipeline/network/catalog/catalog_profile.py new file mode 100644 index 0000000..b0952c5 --- /dev/null +++ b/pipeline/network/catalog/catalog_profile.py @@ -0,0 +1,42 @@ +"""Load the resource descriptions used by the Beckn publisher.""" + +import os +from functools import lru_cache +from pathlib import Path + +from pydantic import BaseModel, ConfigDict, Field, TypeAdapter + +_DEFAULT_PROFILE = Path(__file__).with_name("profiles") / "beckn-catalogs.json" + + +class CatalogSchemaPin(BaseModel): + model_config = ConfigDict(extra="forbid") + + capability: str = Field(pattern=r"^[A-Za-z][A-Za-z0-9_-]*:[A-Za-z][A-Za-z0-9_-]*$") + digest: str = Field(pattern=r"^[0-9a-f]{64}$") + + +class CatalogProjection(BaseModel): + model_config = ConfigDict(extra="forbid") + + catalog_name: str = Field(min_length=1) + domain: str = Field(min_length=1) + catalog_id: str = Field(min_length=1) + resource_id: str = Field(min_length=1) + resource_name: str = Field(min_length=1) + resource_attributes: dict + schema_pack: CatalogSchemaPin | None = None + + +@lru_cache(maxsize=4) +def _read_catalog_profile(path: Path) -> dict[str, CatalogProjection]: + return TypeAdapter(dict[str, CatalogProjection]).validate_json(path.read_bytes()) + + +def load_catalog_profile() -> dict[str, CatalogProjection]: + override = os.environ.get("BECKN_CATALOG_PROFILE_PATH") + return _read_catalog_profile(Path(override) if override else _DEFAULT_PROFILE) + + +def catalog_names() -> dict[str, str]: + return {kind: projection.catalog_name for kind, projection in load_catalog_profile().items()} diff --git a/pipeline/discovery_publish_service.py b/pipeline/network/catalog/discovery_publish_service.py similarity index 61% rename from pipeline/discovery_publish_service.py rename to pipeline/network/catalog/discovery_publish_service.py index 175fdb7..d8f8a3c 100644 --- a/pipeline/discovery_publish_service.py +++ b/pipeline/network/catalog/discovery_publish_service.py @@ -9,22 +9,32 @@ """ import logging -import os import uuid +from dataclasses import dataclass from datetime import datetime from typing import Optional import httpx -from .catalog_builder import build_catalog, is_publishable_kind, normalize_document_kind -from .network_constants import BECKN_VERSION, RESULT_ACCEPTED -from .network_group import load_provider_naming, named_group, provider_for +from pipeline.network.catalog.catalog_builder import build_catalog, normalize_document_kind +from pipeline.network.catalog.catalog_profile import CatalogProjection, load_catalog_profile +from pipeline.network.network_connections import ( + NetworkConnection, + PublishConfigError, + network_connection, +) +from pipeline.network.network_constants import BECKN_VERSION, RESULT_ACCEPTED +from pipeline.network.network_group import load_provider_naming, named_group, provider_for logger = logging.getLogger(__name__) -class PublishConfigError(RuntimeError): - pass +@dataclass(frozen=True) +class PreparedCatalog: + projection: CatalogProjection + context: dict + catalog: dict + directive: dict class DiscoveryPublishService: @@ -34,11 +44,16 @@ def __init__( self, endpoint: Optional[str] = None, timeout: float = 30.0, + *, + connection: NetworkConnection | None = None, ): - self.endpoint = endpoint or os.environ.get("DISCOVERY_SERVICE_ENDPOINT") - self.provider_naming = load_provider_naming() - # Doubles as context.networkId. - self.sender_id = self.provider_naming.network_id + connection = connection or network_connection() + self.endpoint = endpoint or connection.discovery_endpoint + self.sender_id = connection.sender_id + self.network_id = connection.network_id + self.receiver_id = connection.receiver_id + self.visible_to = connection.visible_to + self.provider_naming = load_provider_naming(self.network_id) if not self.endpoint: raise PublishConfigError("DISCOVERY_SERVICE_ENDPOINT must be set to publish to the network.") @@ -65,42 +80,58 @@ def publish( with `skipped=True`: the spec declares `message.catalogs` as `minItems: 1`, so there is no valid "publish nothing" request to send. """ - if not is_publishable_kind(document_kind): + prepared = self.prepare(document_kind, group) + if prepared is None: logger.info( "workflow_id=%s document_kind=%s network_publish_skipped=True", workflow_id, document_kind, ) return self._skipped() + return self.publish_prepared(transaction_id, prepared, workflow_id) + def prepare(self, document_kind: str | None, group: str | None) -> PreparedCatalog | None: + kind = normalize_document_kind(document_kind) + projection = load_catalog_profile().get(kind) + if projection is None: + return None catalog_group = named_group(group) provider = provider_for(self.provider_naming, document_kind, catalog_group) - catalog_name = self.provider_naming.catalog_name(normalize_document_kind(document_kind), catalog_group) - catalog = build_catalog(document_kind, provider, catalog_name, catalog_group) + catalog_name = self.provider_naming.catalog_name(kind, catalog_group) + catalog = build_catalog(document_kind, provider, catalog_name, catalog_group, spec=projection) + context = { + "domain": projection.domain, + "action": "catalog/publish", + "version": BECKN_VERSION, + "senderId": self.sender_id, + "networkId": self.network_id, + } + if self.receiver_id: + context["receiverId"] = self.receiver_id + + directive = { + "catalogId": catalog["id"], + "catalogType": "REGULAR", + "updateMode": "MERGE", + } + if self.visible_to: + directive["visibleTo"] = list(self.visible_to) + return PreparedCatalog(projection, context, catalog, directive) + + def publish_prepared( + self, transaction_id: str, prepared: PreparedCatalog, workflow_id: str | None = None, + ) -> dict: + context = { + **prepared.context, + "messageId": str(uuid.uuid4()), + "transactionId": transaction_id, + "timestamp": datetime.utcnow().isoformat(timespec="milliseconds") + "Z", + } + catalog = prepared.catalog envelope = { - "context": { - "action": "catalog/publish", - "version": BECKN_VERSION, - "messageId": str(uuid.uuid4()), - "transactionId": transaction_id, - "timestamp": datetime.utcnow().isoformat(timespec="milliseconds") + "Z", - "senderId": self.sender_id, - "networkId": self.sender_id, - }, - "message": { - "catalogs": [catalog], - # MERGE is the documented default, but FULL deletes whatever a - # payload omits - too sharp a difference to leave implicit. - "publishDirectives": [ - { - "catalogId": catalog["id"], - "catalogType": "REGULAR", - "updateMode": "MERGE", - } - - ], - }, + "context": context, + "message": {"catalogs": [catalog], "publishDirectives": [prepared.directive]}, } url = f"{self.endpoint.rstrip('/')}/publish" logger.info( @@ -122,12 +153,11 @@ def publish( # the only trace of a transport failure. logger.error( "workflow_id=%s catalog_id=%s network_publish_failed=True transaction_id=%s " - "error=%s response_body=%s", + "status_code=%s", workflow_id, catalog["id"], transaction_id, - exc, - exc.response.text, + exc.response.status_code, ) raise except httpx.HTTPError as exc: @@ -149,12 +179,26 @@ def publish( ) result_status, errors = self._read_result(response, catalog["id"], workflow_id) + try: + reply_context = response.json()["context"] + context_matches = isinstance(reply_context, dict) and all( + reply_context.get(key) == expected for key, expected in { + "action": "catalog/on_publish", + "transactionId": context["transactionId"], + "networkId": context["networkId"], + "receiverId": context["senderId"], + **({"senderId": context["receiverId"]} if context.get("receiverId") else {}), + }.items() + ) + except (KeyError, TypeError, ValueError): + context_matches = False return { "skipped": False, "envelope": envelope, "status_code": response.status_code, "response_body": response.text, "result_status": result_status, + "context_matches": context_matches, "errors": errors, } @@ -188,17 +232,15 @@ def _read_result( raise ValueError("no results") except Exception: logger.warning( - "workflow_id=%s catalog_id=%s network_publish_result=unreadable " - "response_body=%.200s", + "workflow_id=%s catalog_id=%s network_publish_result=unreadable", workflow_id, catalog_id, - response.text, ) return None, [] result = next( (r for r in results if isinstance(r, dict) and r.get("catalogId") == catalog_id), - results[0] if isinstance(results[0], dict) else {}, + {}, ) result_status = result.get("status") errors = result.get("errors") or [] diff --git a/pipeline/network/catalog/profiles/beckn-catalogs.json b/pipeline/network/catalog/profiles/beckn-catalogs.json new file mode 100644 index 0000000..e00a2ed --- /dev/null +++ b/pipeline/network/catalog/profiles/beckn-catalogs.json @@ -0,0 +1,44 @@ +{ + "advisory": { + "catalog_name": "Advisories from documents", + "domain": "agriculture", + "catalog_id": "cat-oan-knowledge-provider-advisories", + "resource_id": "res-oan-knowledge-provider-advisories", + "resource_name": "Agricultural advisory", + "resource_attributes": { + "@context": "https://openagrinet.github.io/network-specs/schema/KnowledgeAdvisory/v0.1/context.jsonld", + "@type": "openagrinet:KnowledgeAdvisory", + "informationMode": "OnDemand", + "subjectCategories": [ + "Crop" + ], + "topics": [ + "Crop production" + ], + "languages": [ + "en" + ] + } + }, + "scheme": { + "catalog_name": "Schemes from documents", + "domain": "agriculture", + "catalog_id": "cat-oan-knowledge-provider-schemes", + "resource_id": "res-oan-knowledge-provider-schemes", + "resource_name": "Government schemes", + "resource_attributes": { + "@context": "https://openagrinet.github.io/network-specs/schema/KnowledgeAdvisory/v0.1/context.jsonld", + "@type": "openagrinet:KnowledgeAdvisory", + "informationMode": "OnDemand", + "subjectCategories": [ + "Scheme" + ], + "topics": [ + "Government schemes" + ], + "languages": [ + "en" + ] + } + } +} diff --git a/pipeline/scheme_catalog.py b/pipeline/network/catalog/scheme_catalog.py similarity index 69% rename from pipeline/scheme_catalog.py rename to pipeline/network/catalog/scheme_catalog.py index 9a203f8..d86331d 100644 --- a/pipeline/scheme_catalog.py +++ b/pipeline/network/catalog/scheme_catalog.py @@ -12,11 +12,15 @@ import os import re from datetime import datetime, timezone +from pathlib import Path from typing import Any, Optional -from . import db -from .documents import document_validity -from .documents import display_name as display_name_rules +from pydantic import BaseModel, ConfigDict, Field + +from pipeline import db +from pipeline.documents import document_validity +from pipeline.documents import display_name as display_name_rules +from pipeline.vector_store.embeddings import get_embedding_model_name SCHEME_CODE_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$") @@ -27,161 +31,34 @@ } _SCHEME_CODE_WORD_RE = re.compile(r"[A-Za-z0-9]+") -# Prompt "vector-indexed" searchable set (13). Exclude nbm (legacy-only). -BOOTSTRAP_VECTOR_SCHEMES: dict[str, dict[str, Any]] = { - "cdp": { - "scheme_name": "Crop Diversification Programme", - "scheme_aliases": [ - "CDP", - "crop diversification", - "crop diversification programme", - "crop diversification program", - ], - }, - "cotton-mission": { - "scheme_name": "Cotton Mission", - "scheme_aliases": ["cotton mission", "cotton scheme", "nfsm cotton", "NFSM Cotton"], - }, - "e-nam": { - "scheme_name": "Electronic National Agriculture Market", - "scheme_aliases": [ - "e-NAM", - "eNAM", - "enam", - "national agriculture market", - "electronic nam", - "electronic national agriculture market", - ], - }, - "makhana": { - "scheme_name": "Central Sector Scheme for Development of Makhana", - "scheme_aliases": [ - "makhana", - "makana", - "foxnut", - "horticulture makhana", - "makhana scheme", - "development of makhana", - ], - }, - "midh": { - "scheme_name": "Mission for Integrated Development of Horticulture", - "scheme_aliases": [ - "MIDH", - "horticulture mission", - "krishonnati horticulture", - "national horticulture mission", - "integrated development of horticulture", - ], - }, - "mif": { - "scheme_name": "Micro Irrigation Fund", - "scheme_aliases": ["MIF", "micro irrigation fund", "Micro Irrigation Fund scheme"], - }, - "nmeo": { - "scheme_name": "National Mission on Edible Oils – Oilseeds", - "scheme_aliases": [ - "NMEO", - "NMEO-OS", - "NMEO OS", - "oilseeds mission", - "nmoe", - "national mission on edible oils", - "edible oils oilseeds", - ], - }, - "pkvy": { - "scheme_name": "Paramparagat Krishi Vikas Yojana", - "scheme_aliases": [ - "PKVY", - "paramparagat krishi vikas yojana", - "paramparagat krishi", - "organic farming scheme", - ], - }, - "pm-ddky": { - "scheme_name": "Prime Minister Dhan–Dhaanya Krishi Yojana", - "scheme_aliases": [ - "PM-DDKY", - "PM DDKY", - "pm ddky", - "dhan-dhaanya krishi yojana", - "dhan dhaanya krishi yojana", - "dhan-dhaanya", - "dhan dhaanya", - "PM Dhan Dhaanya Krishi Yojana", - ], - }, - "pm-kmy": { - "scheme_name": "Pradhan Mantri Kisan Maandhan Yojana", - "scheme_aliases": [ - "PM-KMY", - "PMKMY", - "pm kmy", - "Kisan Maandhan Yojana", - "Kisan Mandhan Yojana", - "kisan mandhan", - "PMKMY Yojana", - ], - }, - "pm-rkvy": { - "scheme_name": "Pradhan Mantri Rashtriya Krishi Vikas Yojana", - "scheme_aliases": [ - "PM-RKVY", - "PM RKVY", - "pm rkvy", - "RKVY", - "rashtriya krishi vikas", - "rashtriya krishi vikas yojana", - ], - }, - "pulses-mission": { - "scheme_name": "Mission for Aatmanirbharta in Pulses", - "scheme_aliases": [ - "pulses mission", - "pulses scheme", - "nfsm pulses", - "aatmanirbharta in pulses", - "mission for aatmanirbharta in pulses", - "aatmanirbharta pulses mission", - "National Food Security Mission – Pulses", - ], - }, - "rwbcis": { - "scheme_name": "Restructured Weather Based Crop Insurance Scheme", - "scheme_aliases": [ - "RWBCIS", - "weather based crop insurance", - "weather insurance", - "weather based crop insurance scheme", - ], - }, -} -# Legacy integrated schemes (get_scheme_info / Hasura). Not zero-redeploy. -BOOTSTRAP_LEGACY_SCHEMES: list[dict[str, str]] = [ - {"scheme_code": "kcc", "scheme_name": "Kisan Credit Card scheme", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "pmkisan", "scheme_name": "Pradhan Mantri Kisan Samman Nidhi scheme", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "pmfby", "scheme_name": "Pradhan Mantri Fasal Bima Yojana scheme", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "shc", "scheme_name": "Soil Health Card scheme", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "pmksy", "scheme_name": "Pradhan Mantri Krishi Sinchayee Yojana", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "sathi", "scheme_name": "SATHI Seed Authentication, Traceability & Holistic Inventory", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "pmasha", "scheme_name": "Pradhan Mantri Annadata Aay Sanrakshan Abhiyan", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "aif", "scheme_name": "Agriculture Infrastructure Fund", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "smam", "scheme_name": "Sub-Mission on Agricultural Mechanization", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "pdmc", "scheme_name": "Per Drop More Crop scheme", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "pkvy", "scheme_name": "Paramparagat Krishi Vikas Yojana", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "nfsm", "scheme_name": "National Food Security Mission", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "rad", "scheme_name": "Rainfed Area Development", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "ffs", "scheme_name": "Framework for Fertilizer Sales", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "nbm", "scheme_name": "National Bamboo Mission", "tool": "get_scheme_info", "tool_routing": "legacy"}, - {"scheme_code": "nbhm", "scheme_name": "National Beekeeping & Honey Mission", "tool": "get_scheme_info", "tool_routing": "legacy"}, -] - -ROUTING_EXCEPTIONS: dict[str, str] = { - "pkvy": "search_schemes", - "nbm": "get_scheme_info", -} +class SchemeAliases(BaseModel): + scheme_name: str + scheme_aliases: list[str] + + +class LegacyScheme(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True, str_min_length=1) + + scheme_code: str + scheme_name: str + tool: str + tool_routing: str + + +class LegacySchemeProfile(BaseModel): + model_config = ConfigDict(extra="forbid") + + aliases_by_code: dict[str, SchemeAliases] = Field(default_factory=dict) + legacy_schemes: list[LegacyScheme] = Field(default_factory=list) + routing_exceptions: dict[str, str] = Field(default_factory=dict) + + +def load_legacy_scheme_profile() -> LegacySchemeProfile: + path = os.environ.get("LEGACY_SCHEME_PROFILE_PATH", "").strip() + if not path: + return LegacySchemeProfile() + return LegacySchemeProfile.model_validate_json(Path(path).read_text(encoding="utf-8")) def utc_now_iso() -> str: @@ -359,11 +236,11 @@ def resolve_scheme_aliases( code = (code or "").strip().lower() merged: list[str] = list(parse_aliases(explicit)) - bootstrap = BOOTSTRAP_VECTOR_SCHEMES.get(code) + bootstrap = load_legacy_scheme_profile().aliases_by_code.get(code) if bootstrap: - merged.extend(bootstrap.get("scheme_aliases") or []) + merged.extend(bootstrap.scheme_aliases) if not (name or "").strip(): - name = bootstrap.get("scheme_name") + name = bootstrap.scheme_name merged.extend(_derive_aliases(code, name or "")) return _dedupe_aliases(merged) @@ -400,7 +277,9 @@ def on_scheme_document_disabled(workflow_id: str, code: Optional[str] = None) -> return rebuild_catalog_from_documents(reason=f"disable_scheme:{workflow_id}") -def build_tool_prompts(vector_schemes: list[dict[str, Any]]) -> dict[str, str]: +def build_tool_prompts( + vector_schemes: list[dict[str, Any]], legacy_schemes: list[LegacyScheme], +) -> dict[str, str]: """Fragments for AI system prompts / tool docstrings.""" codes = [s["scheme_code"] for s in vector_schemes] bullets = [] @@ -414,7 +293,7 @@ def build_tool_prompts(vector_schemes: list[dict[str, Any]]) -> dict[str, str]: aliases = s.get("scheme_aliases") or [] alias_hint = f" / {aliases[0]}" if aliases else "" identifiers.append(f"- `{code}`{alias_hint}") - legacy_codes = ", ".join(f'"{s["scheme_code"]}"' for s in BOOTSTRAP_LEGACY_SCHEMES) + legacy_codes = ", ".join(f'"{scheme.scheme_code}"' for scheme in legacy_schemes) search_doc = ( "Available Qdrant scheme codes: " + ", ".join(f'"{c}"' for c in codes) @@ -427,13 +306,14 @@ def build_tool_prompts(vector_schemes: list[dict[str, Any]]) -> dict[str, str]: "available_schemes_flat_en": "\n".join(flat), "vector_identifiers_block_en": "\n".join(identifiers), "vector_scheme_count": str(len(vector_schemes)), - "legacy_scheme_count": str(len(BOOTSTRAP_LEGACY_SCHEMES)), + "legacy_scheme_count": str(len(legacy_schemes)), } def build_snapshot( *, instance: Optional[str] = None, + allowed_providers: set[str] | None = None, status: str = "live", include_pending: bool = False, network_visible: Optional[bool] = None, @@ -456,6 +336,8 @@ def build_snapshot( if not include_pending and st != "live": continue instances = e["instances"] + if allowed_providers is not None and not allowed_providers.intersection(instances): + continue if instance: inst = instance.strip().lower() if inst not in {g.lower() for g in instances} and instances: @@ -481,9 +363,13 @@ def build_snapshot( ) vector_schemes.sort(key=lambda s: s["scheme_code"]) - etag_src = f"{version}-" + hashlib.sha256( - json.dumps([s["scheme_code"] for s in vector_schemes]).encode() - ).hexdigest()[:8] + profile = load_legacy_scheme_profile() + model = get_embedding_model_name() + etag_src = f"{version}-" + hashlib.sha256(json.dumps({ + "schemes": [s["scheme_code"] for s in vector_schemes], + "profile": profile.model_dump(), + "model": model, + }, sort_keys=True).encode()).hexdigest()[:8] etag = f'"{etag_src}"' return { @@ -494,21 +380,22 @@ def build_snapshot( "collections": { "scheme_qdrant": { "name": scheme_collection_name(), - "embedding_model": "intfloat/multilingual-e5-large", + "embedding_model": model, "filter_type": "scheme", }, "documents": { "name": documents_collection_name(), }, }, - "legacy_schemes": list(BOOTSTRAP_LEGACY_SCHEMES), + "legacy_schemes": [scheme.model_dump() for scheme in profile.legacy_schemes], "vector_schemes": vector_schemes, "tool_prompts": build_tool_prompts( [s for s in vector_schemes if s.get("status", "live") == "live"] if include_pending - else vector_schemes + else vector_schemes, + profile.legacy_schemes, ), - "routing_exceptions": dict(ROUTING_EXCEPTIONS), + "routing_exceptions": profile.routing_exceptions, } diff --git a/pipeline/network/network_connections.py b/pipeline/network/network_connections.py new file mode 100644 index 0000000..2f47cc9 --- /dev/null +++ b/pipeline/network/network_connections.py @@ -0,0 +1,132 @@ +"""Deployment-owned network connections used by publishing and provider identity.""" + +from __future__ import annotations + +import os +import re +from dataclasses import dataclass, field +from urllib.parse import urlsplit + + +class PublishConfigError(RuntimeError): + """The requested network connection is missing or invalid.""" + + +@dataclass(frozen=True) +class NetworkConnection: + adapter: str + network_id: str + discovery_endpoint: str + sender_id: str + receiver_id: str + visible_to: tuple[str, ...] + registry_url: str = "" + registry_token_file: str = "" + release_collection: str = "" + scheme_release_collection: str = "" + approved_routes_file: str = "" + options: dict[str, str] = field(default_factory=dict, repr=False) + + +def _csv(value: str) -> tuple[str, ...]: + return tuple(dict.fromkeys(item.strip() for item in value.split(",") if item.strip())) + + +def _validate_endpoint(endpoint: str, setting: str) -> None: + try: + url = urlsplit(endpoint) + url.port + if (url.scheme not in {"http", "https"} or not url.hostname + or url.username is not None or url.query or url.fragment + or any(char.isspace() or ord(char) < 32 for char in endpoint)): + raise ValueError + except ValueError: + raise PublishConfigError(f"{setting} must be an HTTP(S) endpoint") from None + + +def _named_connections() -> dict[str, NetworkConnection]: + raw = os.environ.get("NETWORK_CONNECTIONS") + if raw is None: + return {} + names = raw.split(",") + names = [name.strip() for name in names] + if len(set(names)) != len(names) or any(not re.fullmatch("[a-z][a-z0-9]*", name) for name in names): + raise PublishConfigError("NETWORK_CONNECTIONS must list distinct lowercase names") + connections = {} + ids = set() + for name in names: + prefix = f"NETWORK_CONNECTION_{name.upper()}_" + + def setting(key: str) -> str: + return os.environ.get(prefix + key, "").strip() + + network_id = setting("ID") + endpoint = setting("DISCOVERY_URL") + sender_id = setting("SENDER_ID") + adapter = setting("TYPE") + if not re.fullmatch(r"[a-z][a-z0-9-]{0,62}", adapter): + raise PublishConfigError(f"{prefix}TYPE must name a network adapter") + if not network_id or network_id in ids: + raise PublishConfigError(f"{prefix}ID must be unique") + registry_url = setting("REGISTRY_URL") + if adapter == "beckn": + if not sender_id: + raise PublishConfigError(f"{prefix}SENDER_ID is required") + _validate_endpoint(endpoint, prefix + "DISCOVERY_URL") + if registry_url: + _validate_endpoint(registry_url, prefix + "REGISTRY_URL") + elif any(setting(key) for key in ( + "DISCOVERY_URL", "SENDER_ID", "RECEIVER_ID", "VISIBLE_TO", + "REGISTRY_URL", "REGISTRY_TOKEN_FILE", "APPROVED_ROUTES_FILE", "SCHEME_RELEASE_COLLECTION", + )): + raise PublishConfigError(f"{prefix}TYPE uses adapter options instead of Beckn settings") + option_prefix = prefix + "OPTION_" + options = {key.removeprefix(option_prefix).lower(): value for key, value in os.environ.items() + if key.startswith(option_prefix)} + ids.add(network_id) + connections[name] = NetworkConnection( + adapter, network_id, endpoint, sender_id, setting("RECEIVER_ID"), _csv(setting("VISIBLE_TO")), + registry_url, setting("REGISTRY_TOKEN_FILE"), + setting("RELEASE_COLLECTION"), setting("SCHEME_RELEASE_COLLECTION"), setting("APPROVED_ROUTES_FILE"), options, + ) + return connections + + +def network_connection(network_id: str | None = None) -> NetworkConnection: + connections = configured_network_connections() + if connections: + if network_id is None: + return connections[0] + for connection in connections: + if connection.network_id == network_id: + return connection + raise PublishConfigError("The requested network has no configured connection") + + sender_id = os.environ.get("NETWORK_SENDER_ID", "").strip() + configured_id = os.environ.get("DISCOVERY_NETWORK_ID", "").strip() + if network_id is not None and network_id != configured_id: + raise PublishConfigError("The requested network does not match the configured publication network") + return NetworkConnection( + os.environ.get("NETWORK_PUBLISHER", "beckn").strip().lower(), + configured_id or sender_id, + os.environ.get("DISCOVERY_SERVICE_ENDPOINT", "").strip(), + sender_id, + os.environ.get("DISCOVERY_RECEIVER_ID", "").strip(), + _csv(os.environ.get("DISCOVERY_VISIBLE_TO", "")), + ) + + +def default_network_id() -> str: + if "NETWORK_CONNECTIONS" in os.environ: + return network_connection().network_id + return os.environ.get("DISCOVERY_NETWORK_ID", "").strip() + + +def configured_network_connections() -> tuple[NetworkConnection, ...]: + connections = _named_connections() + if not connections: + return () + default = os.environ.get("NETWORK_CONNECTION_DEFAULT", "").strip() + if default not in connections: + raise PublishConfigError("NETWORK_CONNECTION_DEFAULT must name a configured connection") + return (connections[default], *(connection for name, connection in connections.items() if name != default)) diff --git a/pipeline/network_constants.py b/pipeline/network/network_constants.py similarity index 60% rename from pipeline/network_constants.py rename to pipeline/network/network_constants.py index c4e6b39..678df03 100644 --- a/pipeline/network_constants.py +++ b/pipeline/network/network_constants.py @@ -1,4 +1,4 @@ -"""Fixed values sent on the network. v2 replaces the per-kind ones with AI-derived values.""" +"""Protocol status codes and the legacy PostgreSQL catalog type mapping.""" BECKN_VERSION = "2.0.0" @@ -6,31 +6,6 @@ RESULT_ACCEPTED = "ACCEPTED" RESULT_REJECTED = "REJECTED" -SCHEMA_CONTEXT_BASE = "https://openagrinet.github.io/network-specs/schema" - -# Permanent: updateMode MERGE upserts resources by id, so a changed id strands -# the old resource on the network instead of replacing it. -ADVISORY_CATALOG_ID = "cat-oan-knowledge-provider-advisories" -ADVISORY_RESOURCE_ID = "res-oan-knowledge-provider-advisories" -SCHEMES_CATALOG_ID = "cat-oan-knowledge-provider-schemes" -SCHEMES_RESOURCE_ID = "res-oan-knowledge-provider-schemes" - -ADVISORY_CATALOG_NAME = "Advisories from documents" -ADVISORY_RESOURCE_NAME = "Agricultural advisory" -SCHEMES_CATALOG_NAME = "Schemes from documents" -SCHEMES_RESOURCE_NAME = "Government schemes" - -# OnDemand means "invoke me for the specifics", and the schema forbids the -# fields that would carry them. -INFORMATION_MODE = "OnDemand" - -SERVED_LANGUAGES = ["en"] - -ADVISORY_SUBJECT_CATEGORIES = ["Crop"] -ADVISORY_TOPICS = ["Crop production"] -SCHEME_SUBJECT_CATEGORIES = ["Scheme"] -SCHEME_TOPICS = ["Government schemes"] - DEFAULT_DOCUMENT_KIND = "document" PROVIDER_ID_TEMPLATE = "{instance}.{network_id}" diff --git a/pipeline/network_group.py b/pipeline/network/network_group.py similarity index 78% rename from pipeline/network_group.py rename to pipeline/network/network_group.py index ce2aad0..42e52b3 100644 --- a/pipeline/network_group.py +++ b/pipeline/network/network_group.py @@ -6,30 +6,26 @@ from pathlib import Path from typing import Optional -from .auth.tenancy import portal_instance -from .catalog_builder import normalize_document_kind -from .instances import INSTANCE_NAMES, instance_display_name -from .network_constants import ( - ADVISORY_CATALOG_NAME, - GROUP_CATALOG_NAME_TEMPLATE, - PROVIDER_ID_TEMPLATE, - SCHEMES_CATALOG_NAME, -) -from .network_provider import ( +from pipeline.network.network_connections import default_network_id +from pipeline.auth.tenancy import portal_instance +from pipeline.network.catalog.catalog_builder import normalize_document_kind +from pipeline.network.catalog.catalog_profile import catalog_names +from pipeline.network.release_policy import instance_display_name +from pipeline.network.network_constants import GROUP_CATALOG_NAME_TEMPLATE, PROVIDER_ID_TEMPLATE +from pipeline.network.network_provider import ( PROVIDER_ID_RULE, - PROVIDER_KINDS, Group, ProviderNaming, is_valid_provider_id, ) PROVIDER_CONFIG_OVERRIDE_PATH_ENV = "PROVIDER_CONFIG_OVERRIDE_PATH" -DEFAULT_PROVIDER_CONFIG_OVERRIDE_PATH = Path(__file__).with_name("network_group_providers.json") PROVIDER_ID_TEMPLATE_ENV = "NETWORK_PROVIDER_ID_TEMPLATE" GROUP_CATALOG_NAME_TEMPLATE_ENV = "NETWORK_GROUP_CATALOG_NAME_TEMPLATE" ADVISORY_CATALOG_NAME_ENV = "NETWORK_ADVISORY_CATALOG_NAME" SCHEMES_CATALOG_NAME_ENV = "NETWORK_SCHEMES_CATALOG_NAME" -NETWORK_ID_ENV = "NETWORK_SENDER_ID" +NETWORK_ID_ENV = "DISCOVERY_NETWORK_ID" +NETWORK_SENDER_ID_ENV = "NETWORK_SENDER_ID" SAMPLE_NETWORK_ID = "provider.oan.dev" @@ -75,13 +71,15 @@ def parse_provider_id_overrides(raw: Optional[str], source: str) -> dict[str, st return overrides -def provider_config_override_path() -> Path: +def provider_config_override_path() -> Path | None: configured = os.environ.get(PROVIDER_CONFIG_OVERRIDE_PATH_ENV, "").strip() - return Path(configured) if configured else DEFAULT_PROVIDER_CONFIG_OVERRIDE_PATH + return Path(configured) if configured else None def load_provider_id_overrides() -> dict[str, str]: path = provider_config_override_path() + if path is None: + return {} try: raw = path.read_text(encoding="utf-8") except OSError as exc: @@ -117,7 +115,7 @@ def _checked_id_template(shared_instance: str, network_id: Optional[str]) -> str instance=shared_instance, network_id=sample_network_id, ) - for instance in dict.fromkeys((shared_instance, *INSTANCE_NAMES)): + for instance in dict.fromkeys((shared_instance, "sample-provider")): _check_provider_id( template.format(instance=instance, network_id=sample_network_id), f"{PROVIDER_ID_TEMPLATE_ENV}={template!r} with {NETWORK_ID_ENV}={sample_network_id!r}", @@ -127,7 +125,16 @@ def _checked_id_template(shared_instance: str, network_id: Optional[str]) -> str def load_provider_naming(network_id: Optional[str] = None) -> ProviderNaming: shared_instance = portal_instance() - network_id = (network_id or os.environ.get(NETWORK_ID_ENV) or "").strip() or None + network_id = ( + (network_id or "").strip() + or default_network_id() + or os.environ.get(NETWORK_SENDER_ID_ENV, "").strip() + or None + ) + names = catalog_names() + for kind, setting in (("advisory", ADVISORY_CATALOG_NAME_ENV), ("scheme", SCHEMES_CATALOG_NAME_ENV)): + if kind in names: + names[kind] = _env(setting, names[kind]) return ProviderNaming( shared_instance=shared_instance, shared_name=instance_display_name(shared_instance), @@ -137,21 +144,14 @@ def load_provider_naming(network_id: Optional[str] = None) -> ProviderNaming: GROUP_CATALOG_NAME_TEMPLATE_ENV, GROUP_CATALOG_NAME_TEMPLATE, frozenset({"group_name"}), - group_name="Maharashtra", - catalog_name="Advisories from documents", + group_name="Sample provider", + catalog_name="Sample catalog", ), - catalog_names={ - "advisory": _env(ADVISORY_CATALOG_NAME_ENV, ADVISORY_CATALOG_NAME), - "scheme": _env(SCHEMES_CATALOG_NAME_ENV, SCHEMES_CATALOG_NAME), - }, + catalog_names=names, provider_id_overrides=load_provider_id_overrides(), ) -def validate_network_provider_config() -> None: - load_provider_naming() - - def named_group(code: Optional[str]) -> Optional[Group]: if code is None: return None @@ -159,6 +159,6 @@ def named_group(code: Optional[str]) -> Optional[Group]: def provider_for(naming: ProviderNaming, document_kind: Optional[str], group: Optional[Group]) -> Optional[dict]: - if normalize_document_kind(document_kind) not in PROVIDER_KINDS or naming.network_id is None: + if normalize_document_kind(document_kind) not in naming.catalog_names or naming.network_id is None: return None return naming.provider(group) diff --git a/pipeline/network_provider.py b/pipeline/network/network_provider.py similarity index 82% rename from pipeline/network_provider.py rename to pipeline/network/network_provider.py index 9c26318..f3c304a 100644 --- a/pipeline/network_provider.py +++ b/pipeline/network/network_provider.py @@ -2,14 +2,9 @@ from dataclasses import dataclass, field from typing import Mapping, Optional -from .network_constants import ( - ADVISORY_CATALOG_NAME, - GROUP_CATALOG_NAME_TEMPLATE, - PROVIDER_ID_TEMPLATE, - SCHEMES_CATALOG_NAME, -) - -PROVIDER_KINDS = ("advisory", "scheme") +from pipeline.network.catalog.catalog_profile import catalog_names +from pipeline.network.network_constants import GROUP_CATALOG_NAME_TEMPLATE, PROVIDER_ID_TEMPLATE + PROVIDER_ID_PATTERN = re.compile(r"[a-z0-9-]+(?:\.[a-z0-9-]+)*") PROVIDER_ID_RULE = "lowercase letters, digits, '-' and '.', with no leading, trailing or repeated '.'" @@ -35,12 +30,7 @@ class ProviderNaming: network_id: Optional[str] id_template: str = PROVIDER_ID_TEMPLATE group_catalog_name_template: str = GROUP_CATALOG_NAME_TEMPLATE - catalog_names: Mapping[str, str] = field( - default_factory=lambda: { - "advisory": ADVISORY_CATALOG_NAME, - "scheme": SCHEMES_CATALOG_NAME, - } - ) + catalog_names: Mapping[str, str] = field(default_factory=catalog_names) provider_id_overrides: Mapping[str, str] = field(default_factory=dict) def provider(self, group: Optional[Group]) -> dict: diff --git a/pipeline/network/network_publisher.py b/pipeline/network/network_publisher.py new file mode 100644 index 0000000..2c97a9d --- /dev/null +++ b/pipeline/network/network_publisher.py @@ -0,0 +1,62 @@ +"""Resolve the deployment-owned adapter for a network connection.""" + +from functools import lru_cache +from importlib.metadata import entry_points + +from pipeline.network.adapters.contracts import NetworkAdapter +from pipeline.network.adapters.contracts import PublicationReceipt as PublicationReceipt +from pipeline.network.network_connections import ( + NetworkConnection, + configured_network_connections, + network_connection, +) +from pipeline.network.network_connections import PublishConfigError as PublishConfigError + +NETWORK_ADAPTER_ENTRY_POINT_GROUP = "knowledge_provider.network_adapters" + + +@lru_cache(maxsize=32) +def _installed_adapter_factory(name: str): + matches = tuple(entry_points(group=NETWORK_ADAPTER_ENTRY_POINT_GROUP, name=name)) + if len(matches) != 1: + raise PublishConfigError( + f"Unsupported NETWORK_PUBLISHER {name!r}. Install exactly one matching network adapter." + ) + factory = matches[0].load() + if not callable(factory): + raise PublishConfigError(f"Network adapter {name!r} must provide a callable factory") + return factory + + +def load_network_adapter(connection: NetworkConnection) -> NetworkAdapter: + if connection.adapter == "beckn": + from pipeline.network.adapters.beckn import BecknNetworkPublisher + + return BecknNetworkPublisher(connection) + + adapter = _installed_adapter_factory(connection.adapter)(connection) + if not isinstance(adapter, NetworkAdapter): + raise PublishConfigError(f"Network adapter {connection.adapter!r} does not implement the publication contract") + return adapter + + +def get_network_adapter(network_id: str | None = None) -> NetworkAdapter: + return load_network_adapter(network_connection(network_id)) + + +def validate_network_adapters() -> None: + from pipeline.network.release_policy import index_only_mode + + if index_only_mode(): + return + for connection in configured_network_connections() or (network_connection(),): + load_network_adapter(connection).validate() + + +def get_network_publisher(network_id: str | None = None): + """Resolve the catalog-only publisher used by older workflow histories.""" + connection = network_connection(network_id) + adapter = load_network_adapter(connection) + if connection.adapter != "beckn": + raise PublishConfigError("Select a document revision to publish through this network adapter") + return adapter diff --git a/pipeline/network/network_registry.py b/pipeline/network/network_registry.py new file mode 100644 index 0000000..bd33e4f --- /dev/null +++ b/pipeline/network/network_registry.py @@ -0,0 +1,395 @@ +"""Read schema, participant, and route records from a configured network Registry.""" + +from __future__ import annotations + +import hashlib +import json +import re +from dataclasses import asdict, dataclass +from pathlib import Path +from time import monotonic +from typing import Literal, Protocol +from urllib.parse import unquote, urlsplit + +import httpx + +from pipeline.network.network_connections import NetworkConnection, network_connection + +_MAX_RESPONSE_BYTES = 128 * 1024 +_ACTIONS = frozenset({"discover", "select", "init", "confirm", "status", "track", "cancel", "update", "rate", "support"}) +_PARTICIPANT_ID = re.compile(r"[a-z0-9][a-z0-9._:-]{2,252}") +_CAPABILITY_CODE = re.compile(r"openagrinet:[A-Z][A-Za-z0-9]*") +_MAPPING_RUNTIME = re.compile( + r"[A-Za-z0-9._-]+@(?:v?[0-9]+(?:\.[0-9]+){1,2}(?:[-+][A-Za-z0-9.+-]+)?|sha256:[0-9a-f]{64})" +) + +Failure = Literal["missing", "duplicate", "inactive", "invalid", "mismatch", "unavailable"] + + +class RegistryLookupError(ValueError): + """A Registry rejection or failure, with a stable classification.""" + + def __init__(self, entity: str, reason: Failure): + super().__init__(f"Registry {entity} lookup: {reason}") + self.entity = entity + self.reason = reason + + +@dataclass(frozen=True) +class RegisteredSchema: + capability: str + version: str + artifact_url: str + + +@dataclass(frozen=True) +class ProviderAdmission: + network_id: str + registry_url: str + participant_id: str + record_id: str + updated_at: str | None + base_url: str + + +@dataclass(frozen=True) +class ParticipantObservation: + record_id: str + revision: str + name: str + base_url: str + status: str + + +@dataclass(frozen=True) +class ActionObservation: + action: str + method: str + path: str + mapping_url: str + timeout_ms: int + retry_max: int + status: str + + +@dataclass(frozen=True) +class BindingObservation: + record_id: str + revision: str + status: str + actions: tuple[ActionObservation, ...] + + +@dataclass(frozen=True) +class ApprovedRoute: + """An operator-approved route and locally verified mapping pin.""" + + network_id: str + participant_id: str + capability: str + action: str + base_url: str + method: str + path: str + mapping_url: str + mapping_sha256: str + mapping_runtime: str + auth_profile: str + timeout_ms: int = 15_000 + retry_max: int = 0 + + def __post_init__(self) -> None: + if (not self.network_id or not _PARTICIPANT_ID.fullmatch(self.participant_id) + or len(self.participant_id) > 64 or not _CAPABILITY_CODE.fullmatch(self.capability) + or self.action not in _ACTIONS): + raise ValueError("Invalid approved route identity") + if not _valid_url(self.base_url) or not _valid_action( + self.action, self.method, self.path, self.mapping_url, self.timeout_ms, self.retry_max, + ): + raise ValueError("Invalid approved route destination or action") + if (not re.fullmatch(r"[0-9a-f]{64}", self.mapping_sha256) + or not _MAPPING_RUNTIME.fullmatch(self.mapping_runtime) or not self.auth_profile.strip()): + raise ValueError("An approved route requires a mapping digest, runtime pin, and auth profile") + + @property + def binding_key(self) -> str: + return f"{self.participant_id}|{self.capability}" + + +@dataclass(frozen=True) +class RouteBindingEvidence: + """An immutable observation of an active binding matching its local approval.""" + + admission: ProviderAdmission + approval: ApprovedRoute + binding_record_id: str + binding_updated_at: str | None + action_record_id: str | None + + @property + def digest(self) -> str: + serialized = json.dumps(asdict(self), sort_keys=True, separators=(",", ":")).encode("utf-8") + return hashlib.sha256(serialized).hexdigest() + + +class NetworkRegistry(Protocol): + def active_schema(self, capability: str) -> RegisteredSchema: ... + + def active_provider(self, participant_id: str) -> ProviderAdmission: ... + + def observe_participant(self, participant_id: str) -> ParticipantObservation | None: ... + + def observe_binding(self, participant_id: str, capability: str) -> BindingObservation | None: ... + + def ready_route(self, approval: ApprovedRoute) -> RouteBindingEvidence: ... + + +def _valid_url(value: object) -> bool: + if not isinstance(value, str) or len(value) > 2000: + return False + if any(char in value for char in "?#\\") or any(char.isspace() or ord(char) < 32 for char in value): + return False + try: + parsed = urlsplit(value) + parsed.port + except ValueError: + return False + return parsed.scheme in {"http", "https"} and bool(parsed.hostname) and parsed.username is None + + +def _valid_action( + action: str, method: object, path: object, mapping_url: object, timeout_ms: object, retry_max: object, +) -> bool: + return ( + method in ("GET", "POST") + and isinstance(path, str) + and bool(re.fullmatch(r"/[A-Za-z0-9._~%/-]{0,511}", path)) + and "//" not in path + and not {".", ".."}.intersection(unquote(path).split("/")) + and isinstance(mapping_url, str) + and _valid_url(mapping_url) + and mapping_url.endswith((f".{action}.yaml", f".{action}.yml")) + and type(timeout_ms) is int and 1_000 <= timeout_ms <= 120_000 + and type(retry_max) is int and 0 <= retry_max <= 5 + ) + + +def _active_record(rows: list[dict], entity: str) -> dict: + if not rows: + raise RegistryLookupError(entity, "missing") + if len(rows) > 1: + raise RegistryLookupError(entity, "duplicate") + status = rows[0].get("status") + if not isinstance(status, str) or not status: + raise RegistryLookupError(entity, "invalid") + if status.casefold() != "active": + raise RegistryLookupError(entity, "inactive") + return rows[0] + + +def _record_string(row: dict, field: str, entity: str, *, required: bool = False) -> str | None: + value = row.get(field) + if value is None and not required: + return None + if not isinstance(value, str) or not value.strip(): + raise RegistryLookupError(entity, "invalid") + return value + + +def _only_record(rows: list[dict], entity: str) -> dict | None: + if len(rows) > 1: + raise RegistryLookupError(entity, "duplicate") + return rows[0] if rows else None + + +def _observed_revision(row: dict, entity: str) -> str: + return _record_string(row, "osUpdatedAt", entity) or _record_string(row, "osCreatedAt", entity, required=True) + + +def _observed_status(row: dict, entity: str) -> str: + status = _record_string(row, "status", entity, required=True).casefold() + if status not in {"active", "inactive"}: + raise RegistryLookupError(entity, "invalid") + return status + + +class SunbirdRegistry: + def __init__(self, connection: NetworkConnection): + if not _valid_url(connection.registry_url): + raise ValueError("The selected network requires a valid Registry URL") + self._connection = connection + self._base = connection.registry_url.rstrip("/") + if not self._base.endswith("/api/v1"): + self._base += "/api/v1" + + def _headers(self) -> dict[str, str]: + if not self._connection.registry_token_file: + return {} + if not self._base.startswith("https://"): + raise ValueError("An authenticated Registry requires HTTPS") + try: + with Path(self._connection.registry_token_file).open(encoding="utf-8") as token_file: + raw = token_file.read(8193) + except (OSError, UnicodeError): + raise RegistryLookupError("credential", "unavailable") from None + token = raw.strip() + if not token or len(raw) > 8192 or any(character.isspace() for character in token): + raise ValueError("The Registry credential file is invalid") + return {"Authorization": f"Bearer {token}"} + + def _search( + self, entity: str, field: str, value: str, *, active_only: bool = False, + ) -> list[dict]: + filters = {field: {"eq": value}} + if active_only: + filters["status"] = {"eq": "active"} + headers = {"Accept-Encoding": "identity", **self._headers()} + deadline = monotonic() + 10 + try: + with httpx.Client(timeout=10, follow_redirects=False, trust_env=False) as client: + with client.stream( + "POST", f"{self._base}/{entity}/search", headers=headers, json={"filters": filters}, + ) as response: + response.raise_for_status() + if response.headers.get("Content-Encoding", "identity").lower() != "identity": + raise RegistryLookupError(entity, "invalid") + data = bytearray() + for chunk in response.iter_bytes(): + if monotonic() >= deadline: + raise RegistryLookupError(entity, "unavailable") + if len(data) + len(chunk) > _MAX_RESPONSE_BYTES: + raise RegistryLookupError(entity, "invalid") + data.extend(chunk) + except httpx.HTTPError: + raise RegistryLookupError(entity, "unavailable") from None + try: + payload = json.loads(data) + except (ValueError, RecursionError): + raise RegistryLookupError(entity, "invalid") from None + if isinstance(payload, dict): + params = payload.get("params", {}) + if not isinstance(params, dict): + raise RegistryLookupError(entity, "invalid") + status = params.get("status") + if status is not None and (not isinstance(status, str) or status.casefold() not in {"success", "successful"}): + raise RegistryLookupError(entity, "unavailable") + payload = payload.get("data") + if (not isinstance(payload, list) + or any(not isinstance(row, dict) or row.get(field) != value for row in payload)): + raise RegistryLookupError(entity, "invalid") + return payload + + def active_schema(self, capability: str) -> RegisteredSchema: + rows = self._search("SchemaRegistry", "capabilityCode", capability, active_only=True) + rows = [row for row in rows if row.get("status") == "active"] + if len(rows) != 1: + raise ValueError("The Registry must return exactly one active registration for the capability") + version, url = rows[0].get("version"), rows[0].get("schemaUrl") + if not isinstance(version, str) or not _valid_url(url) or not url.startswith("https://"): + raise ValueError("The Registry returned an invalid schema version or artifact URL") + return RegisteredSchema(capability, version, url) + + def active_provider(self, participant_id: str) -> ProviderAdmission: + if not _PARTICIPANT_ID.fullmatch(participant_id): + raise ValueError("Invalid Registry participant ID") + row = _active_record(self._search("Participant", "participantId", participant_id), "Participant") + if row.get("type") != "upstream": + raise RegistryLookupError("Participant", "mismatch") + base_url = row.get("baseUrl") + if not _valid_url(base_url): + raise RegistryLookupError("Participant", "invalid") + return ProviderAdmission( + self._connection.network_id, self._base, participant_id, + _record_string(row, "osid", "Participant", required=True), + _record_string(row, "osUpdatedAt", "Participant"), base_url, + ) + + def observe_participant(self, participant_id: str) -> ParticipantObservation | None: + if not _PARTICIPANT_ID.fullmatch(participant_id): + raise ValueError("Invalid Registry participant ID") + entity = "Participant" + row = _only_record(self._search(entity, "participantId", participant_id), entity) + if row is None: + return None + if row.get("type") != "upstream" or not _valid_url(row.get("baseUrl")): + raise RegistryLookupError(entity, "mismatch") + return ParticipantObservation( + _record_string(row, "osid", entity, required=True), _observed_revision(row, entity), + _record_string(row, "name", entity, required=True), row["baseUrl"], _observed_status(row, entity), + ) + + def observe_binding(self, participant_id: str, capability: str) -> BindingObservation | None: + if (not _PARTICIPANT_ID.fullmatch(participant_id) or len(participant_id) > 64 + or not _CAPABILITY_CODE.fullmatch(capability)): + raise ValueError("Invalid Registry binding identity") + entity = "ProviderSchema" + binding = _only_record(self._search(entity, "bindingKey", f"{participant_id}|{capability}"), entity) + if binding is None: + return None + if (binding.get("participantId"), binding.get("capabilityCode")) != (participant_id, capability): + raise RegistryLookupError(entity, "mismatch") + actions = binding.get("actions") + if not isinstance(actions, list) or not 1 <= len(actions) <= 10: + raise RegistryLookupError("ActionBinding", "invalid") + observed = [] + seen = set() + for row in actions: + if not isinstance(row, dict): + raise RegistryLookupError("ActionBinding", "invalid") + action = row.get("action") + fields = ( + row.get("method"), row.get("path"), row.get("mappings"), + row.get("timeoutMs", 15_000), row.get("retryMax", 0), + ) + if not isinstance(action, str) or action not in _ACTIONS or not _valid_action(action, *fields): + raise RegistryLookupError("ActionBinding", "invalid") + if action in seen: + raise RegistryLookupError("ActionBinding", "duplicate") + seen.add(action) + observed.append(ActionObservation(action, *fields, _observed_status(row, "ActionBinding"))) + return BindingObservation( + _record_string(binding, "osid", entity, required=True), _observed_revision(binding, entity), + _observed_status(binding, entity), tuple(observed), + ) + + def ready_route(self, approval: ApprovedRoute) -> RouteBindingEvidence: + """Compare the Registry to an approval. Never call the route or fetch the mapping.""" + if approval.network_id != self._connection.network_id: + raise RegistryLookupError("network", "mismatch") + admission = self.active_provider(approval.participant_id) + if admission.base_url != approval.base_url: + raise RegistryLookupError("Participant", "mismatch") + + rows = self._search("ProviderSchema", "bindingKey", approval.binding_key) + binding = _active_record(rows, "ProviderSchema") + if (binding.get("participantId"), binding.get("capabilityCode")) != ( + approval.participant_id, approval.capability, + ): + raise RegistryLookupError("ProviderSchema", "mismatch") + actions = binding.get("actions") + if not isinstance(actions, list) or any(not isinstance(row, dict) for row in actions): + raise RegistryLookupError("ActionBinding", "invalid") + action = _active_record([row for row in actions if row.get("action") == approval.action], "ActionBinding") + registered_action = ( + action.get("method"), action.get("path"), action.get("mappings"), + action.get("timeoutMs", 15_000), action.get("retryMax", 0), + ) + if not _valid_action(approval.action, *registered_action): + raise RegistryLookupError("ActionBinding", "invalid") + if registered_action != ( + approval.method, approval.path, approval.mapping_url, approval.timeout_ms, approval.retry_max, + ): + raise RegistryLookupError("ActionBinding", "mismatch") + return RouteBindingEvidence( + admission, approval, + _record_string(binding, "osid", "ProviderSchema", required=True), + _record_string(binding, "osUpdatedAt", "ProviderSchema"), + _record_string(action, "osid", "ActionBinding"), + ) + + +def registry_for_network(network_id: str) -> NetworkRegistry: + connection = network_connection(network_id) + if connection.adapter != "beckn": + raise ValueError("The configured network adapter has no Registry integration") + return SunbirdRegistry(connection) diff --git a/pipeline/network/publication_admission.py b/pipeline/network/publication_admission.py new file mode 100644 index 0000000..969fdce --- /dev/null +++ b/pipeline/network/publication_admission.py @@ -0,0 +1,66 @@ +"""Check a managed publication against locally approved Registry bindings.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from pipeline.network.catalog.catalog_profile import CatalogProjection +from pipeline.network.network_connections import PublishConfigError, configured_network_connections +from pipeline.network.network_group import load_provider_naming, named_group, provider_for +from pipeline.network.network_registry import ( + ApprovedRoute, + RegistryLookupError, + registry_for_network, +) + +_MAX_APPROVAL_FILE_BYTES = 256 * 1024 + + +def publication_admission_evidence( + network_id: str, document_kind: str, group: str | None, projection: CatalogProjection | None, + *, participant_id: str | None = None, +) -> str | None: + connection = next((item for item in configured_network_connections() if item.network_id == network_id), None) + if not connection or not connection.approved_routes_file or not projection: + return None + if not projection.schema_pack: + raise PublishConfigError("Registry admission requires a pinned catalog schema pack") + if projection.resource_attributes.get("informationMode") != "OnDemand": + raise PublishConfigError("The approved publication route requires an OnDemand catalog") + if not participant_id: + try: + provider = provider_for(load_provider_naming(network_id), document_kind, named_group(group)) + except ValueError as exc: + raise PublishConfigError("The catalog provider identity is invalid for Registry admission") from exc + if not provider: + raise PublishConfigError("The catalog has no provider identity for Registry admission") + participant_id = provider["id"] + + try: + with Path(connection.approved_routes_file).open("rb") as handle: + raw = handle.read(_MAX_APPROVAL_FILE_BYTES + 1) + if len(raw) > _MAX_APPROVAL_FILE_BYTES: + raise ValueError("The approval file exceeds its size limit") + rows = json.loads(raw) + if not isinstance(rows, list): + raise ValueError("The approval file must contain a list") + approvals = [ + ApprovedRoute(**row) for row in rows + if isinstance(row, dict) + and (row.get("network_id"), row.get("participant_id"), row.get("capability"), row.get("action")) + == (network_id, participant_id, projection.schema_pack.capability, "select") + ] + except (OSError, TypeError, ValueError): + raise PublishConfigError("The configured Registry route approval file is invalid") from None + if len(approvals) != 1: + raise PublishConfigError("The network requires exactly one approved provider select route") + + try: + return registry_for_network(network_id).ready_route(approvals[0]).digest + except RegistryLookupError as exc: + if exc.reason == "unavailable": + raise RuntimeError("The configured Registry is temporarily unavailable") from exc + raise PublishConfigError("The approved provider route is not ready in the Registry") from exc + except ValueError as exc: + raise PublishConfigError("The network Registry configuration is invalid") from exc diff --git a/pipeline/network/publication_delivery.py b/pipeline/network/publication_delivery.py new file mode 100644 index 0000000..e88d8f2 --- /dev/null +++ b/pipeline/network/publication_delivery.py @@ -0,0 +1,124 @@ +"""Persist publication receipts before proceeding to release activation.""" + +import re +from dataclasses import asdict, replace + +from psycopg.types.json import Jsonb + +from pipeline import db +from pipeline.network.adapters.contracts import PublicationReceipt, PublicationRequest +from pipeline.network.network_connections import PublishConfigError, network_connection +from pipeline.network.network_publisher import load_network_adapter + +_POLICY_DIGEST = "publication_policy_sha256" + + +def _validate_receipt(receipt: PublicationReceipt) -> None: + if not isinstance(receipt, PublicationReceipt) or type(receipt.context_matches) is not bool: + raise PublishConfigError("The network adapter did not return a valid publication receipt") + if receipt.exchange is None: + if receipt.context_matches or receipt.result_status or receipt.rejection or receipt.message_id: + raise PublishConfigError("The network adapter returned a receipt without its publication exchange") + elif not isinstance(receipt.exchange, dict): + raise PublishConfigError("The publication exchange must be an object") + + +def deliver_publication( + network_id: str, transaction_id: str, document_id: str, workflow_id: str, + document_kind: str, group: str | None, revision_id: str | None, + *, require_staged_revision: bool = False, +) -> PublicationReceipt: + connection = network_connection(network_id) + adapter = load_network_adapter(connection) + request_identity = { + "workflow_id": workflow_id, "document_kind": document_kind, + "group": group, "revision_id": revision_id, + } + with db.get_connection() as conn: + revision = None + provider_id = None + if require_staged_revision or revision_id is not None or connection.adapter != "beckn": + staged = conn.execute( + """ + SELECT r.id::text, r.snapshot, s.provider_id + FROM document_revisions r + JOIN workflows w ON w.id = r.workflow_id + JOIN network_release_staging s ON s.revision_id = r.id + WHERE r.document_id = %s::uuid AND s.network_id = %s + AND ((%s::uuid IS NOT NULL AND r.id = %s::uuid) + OR (%s::uuid IS NULL AND w.temporal_workflow_id = %s)) + """, + (document_id, network_id, revision_id, revision_id, revision_id, workflow_id), + ).fetchone() + if not staged or not staged["provider_id"]: + raise PublishConfigError("The adapter requires a staged revision with a provider identity") + revision_id = staged["id"] + revision = staged["snapshot"] + provider_id = staged["provider_id"] + if revision.get("metadata", {}).get("kind") != document_kind or not adapter.supports_kind(document_kind): + raise PublishConfigError("The staged revision kind is unavailable for this network adapter") + request_identity.update(adapter=connection.adapter, revision_id=revision_id, provider_id=provider_id) + + request = PublicationRequest( + network_id, transaction_id, document_id, workflow_id, document_kind, group, + revision_id, provider_id, revision, + ) + prepare_delivery = getattr(adapter, "prepare_delivery", None) + if callable(prepare_delivery): + policy_digest = prepare_delivery(request) + if not isinstance(policy_digest, str) or not re.fullmatch("[0-9a-f]{64}", policy_digest): + raise PublishConfigError("The network adapter returned an invalid publication policy digest") + request_identity[_POLICY_DIGEST] = policy_digest + conn.execute( + """ + INSERT INTO publication_deliveries (network_id, transaction_id, document_id, request_identity) + VALUES (%s, %s, %s, %s) + ON CONFLICT (network_id, transaction_id) DO NOTHING + """, + (network_id, transaction_id, document_id, Jsonb(request_identity)), + ) + + # The adapter's bounded remote call runs under the delivery lock. Other + # attempts can reuse the receipt as soon as this transaction commits. + with db.get_connection() as conn: + key = (network_id, transaction_id) + delivery = conn.execute( + """ + SELECT document_id::text, request_identity, receipt + FROM publication_deliveries WHERE network_id = %s AND transaction_id = %s + FOR UPDATE + """, + key, + ).fetchone() + if not delivery or delivery["document_id"] != document_id: + raise PublishConfigError("The publication transaction already identifies a different release") + recorded_identity = dict(delivery["request_identity"]) + current_identity = dict(request_identity) + recorded_policy = recorded_identity.pop(_POLICY_DIGEST, None) + current_policy = current_identity.pop(_POLICY_DIGEST, None) + if recorded_identity != current_identity: + raise PublishConfigError("The publication transaction already identifies a different release") + if recorded_policy != current_policy: + raise PublishConfigError("The network publication policy changed or was not recorded for this transaction") + + evidence = adapter.publication_evidence(conn, request) + if delivery["receipt"] is not None: + recorded = PublicationReceipt(**delivery["receipt"]) + _validate_receipt(recorded) + recorded_evidence = (recorded.exchange or {}).get("binding_evidence_sha256") + if evidence != recorded_evidence: + raise PublishConfigError("The admission binding changed after the publication receipt") + return recorded + + receipt = adapter.publish_revision(request) + _validate_receipt(receipt) + if evidence and receipt.exchange is not None: + receipt = replace(receipt, exchange={**receipt.exchange, "binding_evidence_sha256": evidence}) + conn.execute( + """ + UPDATE publication_deliveries SET receipt = %s, received_at = now() + WHERE network_id = %s AND transaction_id = %s + """, + (Jsonb(asdict(receipt)), *key), + ) + return receipt diff --git a/pipeline/network/registry_operator.py b/pipeline/network/registry_operator.py new file mode 100644 index 0000000..ea0b923 --- /dev/null +++ b/pipeline/network/registry_operator.py @@ -0,0 +1,415 @@ +"""Local Registry proposals and decisions, separate from network admission reads.""" + +from __future__ import annotations + +import re +from dataclasses import asdict, dataclass +from datetime import datetime +from typing import Literal +from uuid import UUID + +from psycopg.types.json import Jsonb + +from pipeline import db +from pipeline.network.network_connections import configured_network_connections, network_connection +from pipeline.network.network_group import load_provider_naming, named_group +from pipeline.network.network_registry import ( + ApprovedRoute, + BindingObservation, + ParticipantObservation, + _valid_url, + registry_for_network, +) + +DesiredStatus = Literal["active", "inactive"] +DecisionOutcome = Literal["approved", "rejected"] + + +class RegistryProposalConflict(RuntimeError): + """A stale proposal or a retry that differs from the original.""" + + +def _bounded_text(value: str, maximum: int, field: str) -> None: + if not isinstance(value, str) or not value.strip() or len(value) > maximum: + raise ValueError(f"Invalid Registry proposal {field}") + + +def _check_status(status: DesiredStatus) -> None: + if status not in ("active", "inactive"): + raise ValueError("A Registry proposal must request active or inactive status") + + +@dataclass(frozen=True) +class RegistryActor: + """Principal and scopes resolved by the caller's trusted authorization layer.""" + + principal_id: str + managed_provider_ids: frozenset[UUID] = frozenset() + operator_network_ids: frozenset[str] = frozenset() + + def __post_init__(self) -> None: + _bounded_text(self.principal_id, 200, "actor") + object.__setattr__(self, "managed_provider_ids", frozenset(self.managed_provider_ids)) + object.__setattr__(self, "operator_network_ids", frozenset(self.operator_network_ids)) + + +@dataclass(frozen=True) +class RegistryRecordVersion: + """Observed remote row and revision to reconcile before any future write.""" + + record_id: str + revision: str + + def __post_init__(self) -> None: + _bounded_text(self.record_id, 253, "record ID") + _bounded_text(self.revision, 253, "record revision") + + +@dataclass(frozen=True) +class AdmissionChange: + name: str + base_url: str + status: DesiredStatus + + def __post_init__(self) -> None: + _bounded_text(self.name, 200, "participant name") + if not _valid_url(self.base_url): + raise ValueError("Invalid Registry participant base URL") + _check_status(self.status) + + +@dataclass(frozen=True) +class RouteChange: + route: ApprovedRoute + schema_pack_digest: str + status: DesiredStatus + + def __post_init__(self) -> None: + if not re.fullmatch(r"[0-9a-f]{64}", self.schema_pack_digest): + raise ValueError("A route proposal requires a schema pack digest") + _check_status(self.status) + + +@dataclass(frozen=True) +class RegistryDecision: + outcome: DecisionOutcome + decided_by: str + reason: str + decided_at: datetime + + +@dataclass(frozen=True) +class RegistryProposal: + id: UUID + request_id: UUID + network_id: str + provider_id: UUID + participant_id: str + version: int + change: AdmissionChange | RouteChange + observed_record: RegistryRecordVersion | None + submitted_by: str + submission_reason: str + submitted_at: datetime + decision: RegistryDecision | None + + +@dataclass(frozen=True) +class RegistryNetwork: + network_id: str + participant_id: str + + +@dataclass(frozen=True) +class RegistryStream: + expected_version: int + current: ParticipantObservation | BindingObservation | None + latest_proposal: RegistryProposal | None + + +@dataclass(frozen=True) +class RegistryContext: + network_id: str + participant_id: str + schema_packs: list[dict] + admission: RegistryStream + route: RegistryStream | None + + +_PROPOSALS = """ + SELECT p.*, s.participant_id, d.outcome, d.decided_by, d.reason AS decision_reason, d.decided_at + FROM registry_operator_proposals p + JOIN registry_operator_subjects s USING (network_id, provider_id) + LEFT JOIN registry_operator_decisions d ON d.proposal_id = p.id +""" + + +def _snapshot(row: dict) -> RegistryProposal: + data = row["change"] + change = ( + RouteChange(ApprovedRoute(**data["route"]), row["schema_pack_digest"], data["status"]) + if row["capability"] else AdmissionChange(**data) + ) + return RegistryProposal( + id=row["id"], request_id=row["request_id"], network_id=row["network_id"], + provider_id=row["provider_id"], participant_id=row["participant_id"], + version=row["version"], change=change, + observed_record=( + RegistryRecordVersion(row["observed_record_id"], row["observed_revision"]) + if row["observed_record_id"] is not None else None + ), + submitted_by=row["submitted_by"], submission_reason=row["submission_reason"], + submitted_at=row["submitted_at"], + decision=( + RegistryDecision(row["outcome"], row["decided_by"], row["decision_reason"], row["decided_at"]) + if row["outcome"] is not None else None + ), + ) + + +def _proposal_row(conn, network_id: str, proposal_id: UUID) -> dict: + row = conn.execute( + _PROPOSALS + " WHERE p.network_id = %s AND p.id = %s", + (network_id, proposal_id), + ).fetchone() + if row is None: + raise LookupError("Registry proposal not found") + return row + + +def _locked_subject(conn, network_id: str, provider_id: UUID) -> dict | None: + return conn.execute( + """ + SELECT participant_id FROM registry_operator_subjects + WHERE network_id = %s AND provider_id = %s FOR UPDATE + """, + (network_id, provider_id), + ).fetchone() + + +def _current_version(conn, network_id: str, provider_id: UUID, capability: str) -> int: + row = conn.execute( + """ + SELECT COALESCE(MAX(version), 0) AS version FROM registry_operator_proposals + WHERE network_id = %s AND provider_id = %s AND capability = %s + """, + (network_id, provider_id, capability), + ).fetchone() + return row["version"] + + +def _provider_name(conn, provider_id: UUID) -> str: + provider = conn.execute("SELECT name FROM instances WHERE id = %s", (provider_id,)).fetchone() + if provider is None: + raise LookupError("Provider not found") + return provider["name"] + + +def _participant_id(network_id: str, provider: str) -> str: + return load_provider_naming(network_id).provider(named_group(provider))["id"] + + +def _require_registry_manager(actor: RegistryActor, provider_id: UUID) -> None: + if provider_id not in actor.managed_provider_ids: + raise PermissionError("Provider manager access is required") + + +def list_provider_registry_networks(*, actor: RegistryActor, provider_id: UUID) -> list[RegistryNetwork]: + _require_registry_manager(actor, provider_id) + with db.get_connection() as conn: + provider = _provider_name(conn, provider_id) + return [ + RegistryNetwork(connection.network_id, _participant_id(connection.network_id, provider)) + for connection in configured_network_connections() + if connection.adapter == "beckn" and connection.registry_url + ] + + +def _latest_proposal(conn, network_id: str, provider_id: UUID, capability: str) -> RegistryProposal | None: + row = conn.execute( + _PROPOSALS + """ + WHERE p.network_id = %s AND p.provider_id = %s AND p.capability = %s + ORDER BY p.version DESC LIMIT 1 + """, + (network_id, provider_id, capability), + ).fetchone() + return _snapshot(row) if row else None + + +def read_registry_context( + *, actor: RegistryActor, network_id: str, provider_id: UUID, capability: str | None, +) -> RegistryContext: + """Pair current Registry observations with the local proposal streams.""" + _require_registry_manager(actor, provider_id) + connection = network_connection(network_id) + if connection.adapter != "beckn" or not connection.registry_url: + raise ValueError("Registry context requires a configured Registry connection") + with db.get_connection() as conn: + participant = _participant_id(network_id, _provider_name(conn, provider_id)) + packs = conn.execute( + """ + SELECT capability, digest, version FROM schema_pack_versions WHERE network_id = %s + ORDER BY capability, installed_at DESC, digest + """, + (network_id,), + ).fetchall() + admission = _latest_proposal(conn, network_id, provider_id, "") + route = _latest_proposal(conn, network_id, provider_id, capability) if capability else None + if any(proposal and proposal.participant_id != participant for proposal in (admission, route)): + raise RegistryProposalConflict("The configured participant differs from the existing proposal stream") + + registry = registry_for_network(network_id) + current_admission = registry.observe_participant(participant) + current_route = registry.observe_binding(participant, capability) if capability else None + return RegistryContext( + network_id, participant, packs, + RegistryStream(admission.version if admission else 0, current_admission, admission), + RegistryStream(route.version if route else 0, current_route, route) if capability else None, + ) + + +def submit_registry_proposal( + *, actor: RegistryActor, network_id: str, provider_id: UUID, request_id: UUID, + expected_version: int, change: AdmissionChange | RouteChange, + observed_record: RegistryRecordVersion | None, reason: str, +) -> RegistryProposal: + """Append an intended change. A null observed record expresses a proposed create.""" + _require_registry_manager(actor, provider_id) + if type(expected_version) is not int or expected_version < 0: + raise ValueError("The expected Registry proposal version must be nonnegative") + _bounded_text(reason, 1000, "submission reason") + reason = reason.strip() + if change.status == "inactive" and observed_record is None: + raise ValueError("An inactive proposal requires an observed Registry record") + connection = network_connection(network_id) + if connection.adapter != "beckn" or not connection.registry_url: + raise ValueError("Registry proposals require a configured Registry connection") + + with db.get_connection() as conn: + participant_id = _participant_id(network_id, _provider_name(conn, provider_id)) + if isinstance(change, RouteChange): + if (change.route.network_id, change.route.participant_id) != (network_id, participant_id): + raise ValueError("The route does not match the configured provider identity") + capability = change.route.capability + pack_digest = change.schema_pack_digest + payload = {"route": asdict(change.route), "status": change.status} + else: + capability, pack_digest, payload = "", None, asdict(change) + + conn.execute( + """ + INSERT INTO registry_operator_subjects (network_id, provider_id, participant_id) + VALUES (%s, %s, %s) ON CONFLICT DO NOTHING + """, + (network_id, provider_id, participant_id), + ) + subject = _locked_subject(conn, network_id, provider_id) + if subject is None or subject["participant_id"] != participant_id: + raise RegistryProposalConflict("The public participant identity belongs to another Registry subject") + + prior = conn.execute( + _PROPOSALS + " WHERE p.network_id = %s AND p.provider_id = %s AND p.request_id = %s", + (network_id, provider_id, request_id), + ).fetchone() + if prior is not None: + proposal = _snapshot(prior) + if (proposal.version - 1, proposal.change, proposal.observed_record, proposal.submitted_by, + proposal.submission_reason) != ( + expected_version, change, observed_record, actor.principal_id, reason, + ): + raise RegistryProposalConflict("The request ID already identifies a different proposal") + return proposal + if _current_version(conn, network_id, provider_id, capability) != expected_version: + raise RegistryProposalConflict("The Registry proposal stream has a newer version") + if pack_digest is not None and conn.execute( + "SELECT 1 FROM schema_pack_versions WHERE network_id = %s AND capability = %s AND digest = %s", + (network_id, capability, pack_digest), + ).fetchone() is None: + raise ValueError("The proposed route's schema pack is not installed for this network") + + row = conn.execute( + """ + INSERT INTO registry_operator_proposals ( + network_id, provider_id, request_id, capability, version, schema_pack_digest, change, + observed_record_id, observed_revision, submitted_by, submission_reason + ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) + RETURNING id + """, + (network_id, provider_id, request_id, capability, expected_version + 1, pack_digest, Jsonb(payload), + observed_record.record_id if observed_record else None, + observed_record.revision if observed_record else None, + actor.principal_id, reason), + ).fetchone() + return _snapshot(_proposal_row(conn, network_id, row["id"])) + + +def read_registry_proposal(*, actor: RegistryActor, network_id: str, proposal_id: UUID) -> RegistryProposal: + with db.get_connection() as conn: + row = _proposal_row(conn, network_id, proposal_id) + if network_id not in actor.operator_network_ids and row["provider_id"] not in actor.managed_provider_ids: + raise LookupError("Registry proposal not found") + return _snapshot(row) + + +def decide_registry_proposal( + *, actor: RegistryActor, network_id: str, proposal_id: UUID, expected_version: int, + outcome: DecisionOutcome, reason: str, +) -> RegistryProposal: + """Record one local operator decision about the latest proposal version.""" + if network_id not in actor.operator_network_ids: + raise PermissionError("Operator access for this network is required") + if type(expected_version) is not int or expected_version < 1: + raise ValueError("The expected Registry proposal version must be positive") + if outcome not in ("approved", "rejected"): + raise ValueError("The Registry proposal decision must be approved or rejected") + _bounded_text(reason, 1000, "decision reason") + reason = reason.strip() + with db.get_connection() as conn: + target = _proposal_row(conn, network_id, proposal_id) + provider_id = target["provider_id"] + # Serialize both decisions and new proposals for this provider/network. + _locked_subject(conn, network_id, provider_id) + row = _proposal_row(conn, network_id, proposal_id) + proposal = _snapshot(row) + if proposal.version != expected_version: + raise RegistryProposalConflict("The decision version does not match the proposal") + if proposal.decision is not None: + decision = proposal.decision + if (decision.outcome, decision.decided_by, decision.reason) != (outcome, actor.principal_id, reason): + raise RegistryProposalConflict("The Registry proposal already has a different decision") + return proposal + if _current_version(conn, network_id, provider_id, row["capability"]) != expected_version: + raise RegistryProposalConflict("The Registry proposal was superseded before the decision") + conn.execute( + """ + INSERT INTO registry_operator_decisions (proposal_id, outcome, decided_by, reason) + VALUES (%s, %s, %s, %s) + """, + (proposal_id, outcome, actor.principal_id, reason), + ) + return _snapshot(_proposal_row(conn, network_id, proposal_id)) + + +def list_network_proposals( + *, actor: RegistryActor, network_id: str, pending_only: bool = True, limit: int = 50, +) -> list[RegistryProposal]: + """Return recent local proposals within the operator's network.""" + if network_id not in actor.operator_network_ids: + raise PermissionError("Operator access for this network is required") + with db.get_connection() as conn: + rows = conn.execute( + _PROPOSALS + """ + WHERE p.network_id = %s AND ( + NOT %s OR ( + d.proposal_id IS NULL AND NOT EXISTS ( + SELECT 1 FROM registry_operator_proposals newer + WHERE newer.network_id = p.network_id AND newer.provider_id = p.provider_id + AND newer.capability = p.capability AND newer.version > p.version + ) + ) + ) + ORDER BY p.submitted_at DESC, p.id DESC LIMIT %s + """, + (network_id, pending_only, max(1, min(limit, 100))), + ).fetchall() + return [_snapshot(row) for row in rows] diff --git a/pipeline/network/release_policy.py b/pipeline/network/release_policy.py new file mode 100644 index 0000000..957a3a0 --- /dev/null +++ b/pipeline/network/release_policy.py @@ -0,0 +1,171 @@ +"""Provider display and network release settings.""" + +from __future__ import annotations + +import json +import os + +from pipeline.network.network_connections import ( + PublishConfigError, + configured_network_connections, + network_connection, +) + + +def _enabled(name: str) -> bool: + return os.environ.get(name, "").strip().lower() in {"true", "1", "yes"} + + +def index_only_mode() -> bool: + """Finish processing at the local search collection without releasing documents.""" + return _enabled("KP_INDEX_ONLY") + + +def intake_index_options(index_name: str, auto_approve: bool) -> tuple[str, bool]: + from pipeline.vector_store import get_default_index_name + + return (get_default_index_name(), True) if index_only_mode() else (index_name, auto_approve) + + +def publish_to_network_enabled() -> bool: + return not index_only_mode() and _enabled("ENABLE_PUBLISH_TO_NETWORK") + + +def _configured_upload_release_target(kind: str | None) -> tuple[str, str] | None: + """Return the deployment's automatic upload network and local collection.""" + if index_only_mode(): + return None + network_id = os.environ.get("AUTO_RELEASE_NETWORK_ID", "").strip() + if not network_id: + return None + try: + connection = network_connection(network_id) + except PublishConfigError as exc: + raise ValueError(str(exc)) from exc + if not publish_to_network_enabled() or network_id != network_connection().network_id: + raise ValueError("Automatic upload release must target the enabled local network") + collection = os.environ.get("AUTO_RELEASE_COLLECTION", "").strip() or connection.release_collection + if kind == "scheme" and connection.adapter == "beckn": + collection = ( + os.environ.get("AUTO_RELEASE_SCHEME_COLLECTION", "").strip() + or connection.scheme_release_collection + or collection + ) + if not collection: + raise ValueError("AUTO_RELEASE_COLLECTION is required for automatic upload release") + draft_collection = os.environ.get("VECTOR_DB_COLLECTION_NAME", "documents-index").strip() or "documents-index" + if collection == draft_collection or collection == "documents-index": + raise ValueError("A network release collection must be separate from the draft index") + return network_id, collection + + +def upload_release_target(kind: str | None = None, *, provider: str | None = None) -> tuple[str, str] | None: + from pipeline.network.network_publisher import get_network_adapter + + if provider and kind and publish_to_network_enabled(): + from pipeline.intake.provider_sources import stored_settings + + row = stored_settings(provider) + network_id = (row or {}).get("settings", {}).get("release_network_id") + if network_id: + collection = managed_release_targets(kind).get(network_id) + if not collection: + raise ValueError("The provider’s default release network cannot publish this document kind") + return network_id, collection + target = _configured_upload_release_target(kind) + if target and kind: + try: + if not get_network_adapter(target[0]).supports_kind(kind): + return None + except PublishConfigError as exc: + raise ValueError(str(exc)) from exc + return target + + +def managed_release_targets(kind: str | None = None) -> dict[str, str]: + """Map explicit local write targets to their independently managed collections.""" + return _managed_release_targets(kind, require_support=True) + + +def source_release_target(kind: str, network_id: str | None, *, provider: str | None = None) -> tuple[str, str] | None: + if network_id is None: + return upload_release_target(kind, provider=provider) + collection = managed_release_targets(kind).get(network_id) + return (network_id, collection) if collection else None + + +def _managed_release_targets(kind: str | None, *, require_support: bool) -> dict[str, str]: + from pipeline.network.network_publisher import load_network_adapter + + automatic = upload_release_target(kind) if require_support else _configured_upload_release_target(kind) + if not publish_to_network_enabled(): + return {} + try: + connections = configured_network_connections() + except PublishConfigError as exc: + raise ValueError(str(exc)) from exc + default_collection = os.environ.get("VECTOR_DB_COLLECTION_NAME", "documents-index").strip() + collections: dict[str, str] = {} + targets = {} + for connection in connections: + general = connection.release_collection + schemes = connection.scheme_release_collection or general + for collection in (general, schemes): + if not collection: + continue + if collection in {default_collection, "documents-index"}: + raise ValueError("A network release collection must be separate from the draft index") + owner = collections.setdefault(collection, connection.network_id) + if owner != connection.network_id: + raise ValueError("Each network must have a separate release collection") + collection = schemes if kind == "scheme" else general + try: + supported = not (require_support and kind) or load_network_adapter(connection).supports_kind(kind) + except PublishConfigError as exc: + raise ValueError(str(exc)) from exc + if collection and supported: + targets[connection.network_id] = collection + if automatic: + network_id, collection = automatic + if collections.get(collection, network_id) != network_id: + raise ValueError("The automatic release collection belongs to a different network") + if targets.get(network_id, collection) != collection: + raise ValueError("The automatic and managed release collections must agree") + targets[network_id] = collection + return targets + + +def managed_release_collections() -> dict[str, str]: + """Map collection to its publication authority for search filtering.""" + collections = {} + for kind in (None, "scheme"): + for network_id, collection in _managed_release_targets(kind, require_support=False).items(): + owner = collections.setdefault(collection, network_id) + if owner != network_id: + raise ValueError("Each network must have a separate release collection") + return collections + + +def prod_stage_disabled() -> bool: + """True when DISABLE_PROD_SETTING turns off the PROD half of the pipeline. + + With this set, a document finishes at DEV ingest instead of waiting for a + superadmin to promote it — the ``approval_for_prod`` and ``ingesting_prod`` + stages are skipped entirely. + + Read only when a workflow is *started*; the value is then passed into the + workflow as an argument. Reading it inside a workflow would break Temporal + replay, because flipping the flag mid-flight would send a replaying + workflow down a different branch than the one recorded in its history. + """ + return index_only_mode() or _enabled("DISABLE_PROD_SETTING") + +def instance_display_name(code: str | None) -> str: + """Resolve a deployment's provider label or display the code itself.""" + normalized = (code or "").strip().lower() + configured = json.loads(os.environ.get("PROVIDER_DISPLAY_NAMES", "{}")) + if not isinstance(configured, dict) or any( + not isinstance(name, str) or not name.strip() for name in configured.values() + ): + raise ValueError("PROVIDER_DISPLAY_NAMES must map provider codes to display names") + return configured.get(normalized) or normalized.replace("-", " ").replace("_", " ").title() or "Default" diff --git a/pipeline/network_group_providers.json b/pipeline/network_group_providers.json deleted file mode 100644 index 0967ef4..0000000 --- a/pipeline/network_group_providers.json +++ /dev/null @@ -1 +0,0 @@ -{} diff --git a/pipeline/ocr/base.py b/pipeline/ocr/base.py index 619e3a5..d135ff3 100644 --- a/pipeline/ocr/base.py +++ b/pipeline/ocr/base.py @@ -13,7 +13,7 @@ from __future__ import annotations from abc import ABC, abstractmethod -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import Callable, Optional @@ -21,7 +21,7 @@ class OcrConfig: provider: str model: str = "" - api_key: str = "" + api_key: str = field(default="", repr=False) endpoint: str = "" api_url: str = "" inference_mode: str = "hf" @@ -31,6 +31,8 @@ class OcrConfig: max_workers: int = 4 image_dpi: int = 192 request_timeout_seconds: float = 300.0 + executable: str = "" + skip_text_layer: bool | None = None PageDict = dict @@ -38,6 +40,8 @@ class OcrConfig: class OcrProvider(ABC): name: str + # Layout parsers need the source page even when it already has text. + requires_all_pages = False @abstractmethod def process_pdf_range( diff --git a/pipeline/ocr/configuration.py b/pipeline/ocr/configuration.py new file mode 100644 index 0000000..b281fba --- /dev/null +++ b/pipeline/ocr/configuration.py @@ -0,0 +1,133 @@ +"""Typed, deployment-owned document parser profiles.""" + +from __future__ import annotations + +import os +import re +from pathlib import Path +from typing import Annotated, Literal + +from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, ValidationError + +from .base import OcrConfig +from .extensions import ExtensionParserConfig + +PositiveInt = Annotated[int, Field(gt=0)] +PositiveSeconds = Annotated[float, Field(gt=0, allow_inf_nan=False)] + + +class ParserOptions(BaseModel): + model_config = ConfigDict(extra="forbid") + + segment_pages: PositiveInt = 20 + skip_text_layer: bool = True + + +class EmbeddedTextOptions(ParserOptions): + type: Literal["pypdf", "mock"] + + +class LiteParseOptions(ParserOptions): + type: Literal["liteparse"] + executable: str = "lit" + request_timeout_seconds: PositiveSeconds = 300.0 + + +class ChandraOptions(ParserOptions): + type: Literal["chandra", "chandra_vllm"] + model: str = "chandra" + endpoint: str = "" + api_url: str = "" + inference_mode: Literal["hf", "vllm"] = "hf" + max_output_tokens: PositiveInt = 12288 + max_workers: PositiveInt = 4 + image_dpi: PositiveInt = 192 + request_timeout_seconds: PositiveSeconds = 300.0 + + +class MistralOptions(ParserOptions): + type: Literal["mistral", "mistral_ocr"] + model: str = "mistral-ocr-latest" + api_url: str = "https://api.mistral.ai/v1/ocr" + api_key_file: str + request_timeout_seconds: PositiveSeconds = 300.0 + + +class ExtensionOptions(ParserOptions): + type: Literal["extension"] + plugin: str = Field(pattern=r"^[A-Za-z0-9][A-Za-z0-9_.-]{0,63}$") + options: dict[str, str] = Field(default_factory=dict, max_length=32) + + +ParserProfile = Annotated[ + EmbeddedTextOptions | LiteParseOptions | ChandraOptions | MistralOptions | ExtensionOptions, + Field(discriminator="type"), +] +_SINGLE_PROFILE = TypeAdapter(ParserProfile) +_PROFILE_NAME = re.compile(r"[a-z0-9][a-z0-9_-]{0,63}") + + +class ParserProfiles(BaseModel): + model_config = ConfigDict(extra="forbid") + + default: str + profiles: dict[str, ParserProfile] + + +def _read_profile(prefix: str) -> ParserProfile: + values = { + key.removeprefix(prefix).lower(): value.strip() + for key, value in os.environ.items() + if key.startswith(prefix) + } + if values.get("type") == "extension": + values["options"] = { + key.removeprefix("option_"): values.pop(key) for key in tuple(values) if key.startswith("option_") + } + try: + return _SINGLE_PROFILE.validate_python(values) + except ValidationError as error: + # Validation errors can include the entire input. Report only the error kind. + error_type = error.errors(include_input=False)[0]["type"] + raise ValueError(f"Invalid {prefix}* settings ({error_type})") from None + + +def read_profiles() -> ParserProfiles | None: + names_value = os.environ.get("DOCUMENT_PARSER_PROFILES") + if names_value is None: + if not any(key.startswith("DOCUMENT_PARSER_") for key in os.environ): + return None + profile = _read_profile("DOCUMENT_PARSER_") + return ParserProfiles(default=profile.type, profiles={profile.type: profile}) + + names = [name.strip() for name in names_value.split(",")] + if len(set(names)) != len(names) or any(not _PROFILE_NAME.fullmatch(name) for name in names): + raise ValueError("DOCUMENT_PARSER_PROFILES must list unique lowercase identifiers") + # Underscores keep profile prefixes readable. Reject overlapping prefixes. + if any(left != right and right.startswith(f"{left}_") for left in names for right in names): + raise ValueError("DOCUMENT_PARSER_PROFILES names must have distinct prefixes") + if "DOCUMENT_PARSER_TYPE" in os.environ: + raise ValueError("Set DOCUMENT_PARSER_TYPE or DOCUMENT_PARSER_PROFILES") + default = os.environ.get("DOCUMENT_PARSER_DEFAULT", names[0]).strip() + if default not in names: + raise ValueError("DOCUMENT_PARSER_DEFAULT must name a configured profile") + profiles = {name: _read_profile(f"DOCUMENT_PARSER_{name.upper()}_") for name in names} + return ParserProfiles(default=default, profiles=profiles) + + +def resolve_profile(config: ParserProfiles, name: str | None) -> OcrConfig: + selected = name or config.default + profile = config.profiles.get(selected) + if profile is None: + raise ValueError("Requested document parser profile is not configured") + + options = profile.model_dump(exclude={"type", "api_key_file"}) + if isinstance(profile, MistralOptions): + try: + options["api_key"] = Path(profile.api_key_file).read_text(encoding="utf-8").strip() + except (OSError, UnicodeError): + raise ValueError("Unable to read the Mistral parser credential file") from None + if not options["api_key"]: + raise ValueError("The Mistral parser credential file is empty") + config_type = ExtensionParserConfig if isinstance(profile, ExtensionOptions) else OcrConfig + return config_type(provider=profile.type, **options) diff --git a/pipeline/ocr/extensions.py b/pipeline/ocr/extensions.py new file mode 100644 index 0000000..862bffe --- /dev/null +++ b/pipeline/ocr/extensions.py @@ -0,0 +1,25 @@ +"""Load deployment-installed OCR providers through a Python entry point.""" + +from dataclasses import dataclass, field +from importlib.metadata import entry_points + +from .base import OcrConfig, OcrProvider + +PARSER_ENTRY_POINT_GROUP = "knowledge_provider.document_parsers" + + +@dataclass +class ExtensionParserConfig(OcrConfig): + plugin: str = "" + options: dict[str, str] = field(default_factory=dict, repr=False) + + +def load_extension(config: ExtensionParserConfig) -> OcrProvider: + matches = tuple(entry_points(group=PARSER_ENTRY_POINT_GROUP, name=config.plugin)) + if len(matches) != 1: + raise ValueError(f"Install exactly one document parser extension named {config.plugin!r}") + factory = matches[0].load() + provider = factory(dict(config.options)) + if not isinstance(provider, OcrProvider): + raise TypeError(f"Document parser extension {config.plugin!r} must return an OcrProvider") + return provider diff --git a/pipeline/ocr/liteparse.py b/pipeline/ocr/liteparse.py new file mode 100644 index 0000000..e6fe0e0 --- /dev/null +++ b/pipeline/ocr/liteparse.py @@ -0,0 +1,126 @@ +"""Local extraction of PDF text and source positions with LiteParse.""" + +from __future__ import annotations + +import json +import math +import os +import subprocess +from pathlib import Path +from tempfile import TemporaryDirectory +from typing import Callable + +from pipeline.documents.models import ( + PAGE_LAYOUT_ITEM_MAX_CHARS, + PAGE_LAYOUT_MAX_ITEMS, + PageSourceLayout, +) +from .base import OcrConfig, OcrProvider, PageDict + +_MAX_OUTPUT_BYTES = 32 * 1024 * 1024 + + +class LiteParseProvider(OcrProvider): + name = "liteparse" + requires_all_pages = True + + def __init__(self, config: OcrConfig) -> None: + self.executable = config.executable or "lit" + self.timeout = config.request_timeout_seconds + if not math.isfinite(self.timeout) or self.timeout <= 0: + raise ValueError("OCR_REQUEST_TIMEOUT_SECONDS must be finite and positive") + + def process_pdf_range( + self, + pdf_path: str, + start_idx: int, + end_idx: int, + *, + log: Callable[..., None] | None = None, + ) -> list[PageDict]: + if start_idx >= end_idx: + return [] + page_range = f"{start_idx + 1}-{end_idx}" + input_path = str(Path(pdf_path).resolve()) + with TemporaryDirectory(prefix="kp-liteparse-") as directory: + output_path = Path(directory) / "pages.json" + command = [ + self.executable, "parse", "--format", "json", "--no-ocr", + "--complexity", "--target-pages", page_range, "--quiet", + "--output", str(output_path), "--", input_path, + ] + try: + subprocess.run( + command, + check=True, + timeout=self.timeout, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + env={"PATH": os.pathsep.join(os.get_exec_path())}, + cwd=directory, + ) + except FileNotFoundError: + raise RuntimeError("LiteParse executable is unavailable on this worker") from None + except subprocess.TimeoutExpired: + raise RuntimeError("LiteParse exceeded the configured parser timeout") from None + except (OSError, subprocess.CalledProcessError): + raise RuntimeError("LiteParse could not parse the requested PDF pages") from None + + try: + with output_path.open("rb") as output: + data = output.read(_MAX_OUTPUT_BYTES + 1) + if len(data) > _MAX_OUTPUT_BYTES: + raise RuntimeError("LiteParse page range output exceeded the size limit") + pages = json.loads(data)["pages"] + if not isinstance(pages, list) or len(pages) != end_idx - start_idx: + raise ValueError("Unexpected PDF page numbers") + for expected, page in enumerate(pages, start=start_idx + 1): + if type(page["page"]) is not int or page["page"] != expected: + raise ValueError("Unexpected PDF page numbers") + return [_pipeline_page(page) for page in pages] + except (OSError, KeyError, TypeError, ValueError): + raise RuntimeError("LiteParse returned invalid page data") from None + + +def _pipeline_page(page: dict) -> PageDict: + text = page["text"] + items = page["text_items"] + if not isinstance(text, str) or not isinstance(items, list): + raise ValueError("Unexpected PDF text data") + + complexity = page.get("complexity") or {} + if not isinstance(complexity, dict): + raise ValueError("Unexpected PDF complexity data") + needs_ocr = complexity.get("needs_ocr") + if needs_ocr and not text.strip() and ( + complexity.get("full_page_image") or complexity.get("has_substantial_images") + ): + raise RuntimeError("LiteParse found a scanned page without text. Select an OCR provider.") + + positioned_text = [] + truncated = len(items) > PAGE_LAYOUT_MAX_ITEMS + for item in items[:PAGE_LAYOUT_MAX_ITEMS]: + excerpt = item["text"][:PAGE_LAYOUT_ITEM_MAX_CHARS] + truncated = truncated or len(excerpt) < len(item["text"]) + positioned_text.append({ + "text": excerpt, + "x": item["x"], + "y": item["y"], + "width": item["width"], + "height": item["height"], + }) + source_layout = PageSourceLayout( + parser=LiteParseProvider.name, + width=page["width"], + height=page["height"], + text_items=positioned_text, + truncated=truncated, + ocr_recommended=needs_ocr, + ) + return { + "page_number": page["page"], + "original_markdown": text, + "edited_markdown": None, + "metadata": {"source_layout": source_layout.model_dump(exclude_none=True)}, + } diff --git a/pipeline/ocr/service.py b/pipeline/ocr/service.py index abe48c7..1366781 100644 --- a/pipeline/ocr/service.py +++ b/pipeline/ocr/service.py @@ -10,8 +10,11 @@ from .base import OcrConfig, OcrProvider, PageDict from .chandra_vllm import ChandraVllmOcrProvider +from .configuration import read_profiles, resolve_profile +from .liteparse import LiteParseProvider from .mistral_ocr import MistralOcrProvider from .mock import MockOcrProvider +from .extensions import ExtensionParserConfig, load_extension from .text_layer import group_contiguous_ranges, split_pdf_range PROVIDERS: dict[str, type[OcrProvider]] = { @@ -22,13 +25,47 @@ # Local dev: embedded PDF text only (no external OCR). "mock": MockOcrProvider, "pypdf": MockOcrProvider, + "liteparse": LiteParseProvider, } logger = logging.getLogger(__name__) -def load_ocr_config() -> OcrConfig: - provider = os.environ.get("OCR_PROVIDER", "mistral").strip().lower() +def default_ocr_profile() -> str: + profiles = read_profiles() + if profiles is not None: + return profiles.default + return load_ocr_config().provider + + +def allowed_ocr_providers() -> tuple[str, ...]: + profiles = read_profiles() + if profiles is not None: + return tuple(profiles.profiles) + configured = os.environ.get("OCR_ALLOWED_PROVIDERS") + names = configured.split(",") if configured is not None else [load_ocr_config().provider] + providers = tuple(dict.fromkeys(name.strip().lower() for name in names if name.strip())) + if not providers or any(name not in PROVIDERS for name in providers): + raise ValueError("OCR_ALLOWED_PROVIDERS must name registered OCR providers") + return providers + + +def load_ocr_config(provider: str | None = None) -> OcrConfig: + profiles = read_profiles() + if profiles is not None: + return resolve_profile(profiles, provider) + provider = (provider or os.environ.get("OCR_PROVIDER", "chandra")).strip().lower() + max_split_pages = int(os.environ.get("OCR_MAX_SPLIT_PAGES", "40")) + segment_pages = int(os.environ.get("OCR_SEGMENT_PAGES", "20")) + if provider == "liteparse": + return OcrConfig( + provider=provider, + max_split_pages=max_split_pages, + segment_pages=segment_pages, + request_timeout_seconds=float(os.environ.get("OCR_REQUEST_TIMEOUT_SECONDS", "300")), + executable=os.environ.get("LITEPARSE_EXECUTABLE", "lit").strip(), + ) + if provider in {"mistral", "mistral_ocr"}: model = ( os.environ.get("OCR_MODEL") @@ -54,8 +91,6 @@ def load_ocr_config() -> OcrConfig: image_dpi = int(os.environ.get("CHANDRA_IMAGE_DPI", "192")) request_timeout_seconds = float(os.environ.get("CHANDRA_REQUEST_TIMEOUT_SECONDS", "300")) - max_split_pages = int(os.environ.get("OCR_MAX_SPLIT_PAGES", "40")) - segment_pages = int(os.environ.get("OCR_SEGMENT_PAGES", "20")) return OcrConfig( provider=provider, model=model, @@ -83,13 +118,19 @@ def _skip_ocr_for_text_layer() -> bool: return os.environ.get("OCR_SKIP_TEXT_LAYER", "true").strip().lower() not in {"0", "false", "no"} +def _provider_type(name: str) -> type[OcrProvider]: + provider_cls = PROVIDERS.get(name) + if provider_cls is None: + supported = ", ".join(sorted(PROVIDERS)) + raise ValueError(f"Unsupported OCR provider '{name}'. Supported: {supported}") + return provider_cls + + def get_ocr_provider(config: Optional[OcrConfig] = None) -> OcrProvider: config = config or load_ocr_config() - provider_cls = PROVIDERS.get(config.provider) - if not provider_cls: - supported = ", ".join(sorted(PROVIDERS)) - raise ValueError(f"Unsupported OCR provider '{config.provider}'. Supported: {supported}") - return provider_cls(config) + if isinstance(config, ExtensionParserConfig): + return load_extension(config) + return _provider_type(config.provider)(config) def _activity_log(level: str, message: str, *args) -> None: @@ -123,9 +164,21 @@ def _process_range( config: OcrConfig, provider_cache: dict[str, OcrProvider], ) -> list[PageDict]: - """OCR pages [start_idx, end_idx), skipping pages with a usable text layer.""" - if not _skip_ocr_for_text_layer(): - provider = provider_cache.setdefault("provider", get_ocr_provider(config)) + """Extract pages [start_idx, end_idx), using embedded text where supported.""" + skip_text_layer = config.skip_text_layer + if skip_text_layer is None: + skip_text_layer = _skip_ocr_for_text_layer() + if isinstance(config, ExtensionParserConfig): + if "provider" not in provider_cache: + provider_cache["provider"] = get_ocr_provider(config) + requires_all_pages = provider_cache["provider"].requires_all_pages + else: + requires_all_pages = _provider_type(config.provider).requires_all_pages + if requires_all_pages or not skip_text_layer: + provider = provider_cache.get("provider") + if provider is None: + provider = get_ocr_provider(config) + provider_cache["provider"] = provider return provider.process_pdf_range(local_pdf_path, start_idx, end_idx, log=_activity_log) split = split_pdf_range(local_pdf_path, start_idx, end_idx, log=_activity_log) @@ -147,8 +200,13 @@ def _process_range( ] -def ocr_pdf(local_pdf_path: str, clean_text: Callable[[str], str]) -> list[PageDict]: - config = load_ocr_config() +def ocr_pdf( + local_pdf_path: str, + clean_text: Callable[[str], str], + *, + provider: str | None = None, +) -> list[PageDict]: + config = load_ocr_config(provider) reader = PdfReader(local_pdf_path) pages = _process_range(local_pdf_path, 0, len(reader.pages), config, {}) pages = _finalize_pages(pages, clean_text) @@ -158,12 +216,14 @@ def ocr_pdf(local_pdf_path: str, clean_text: Callable[[str], str]) -> list[PageD def ocr_pdf_in_segments( local_pdf_path: str, - segment_pages: int, + segment_pages: int | None, clean_text: Callable[[str], str], on_segment_complete=None, completed_page_numbers: set[int] | None = None, + *, + provider: str | None = None, ) -> list[PageDict]: - config = load_ocr_config() + config = load_ocr_config(provider) provider_cache: dict[str, OcrProvider] = {} completed_page_numbers = completed_page_numbers or set() total_pages = len(PdfReader(local_pdf_path).pages) diff --git a/pipeline/search/__init__.py b/pipeline/search/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/pipeline/search/index_identity.py b/pipeline/search/index_identity.py new file mode 100644 index 0000000..a11979d --- /dev/null +++ b/pipeline/search/index_identity.py @@ -0,0 +1,27 @@ +"""Document-owned vector keys, separate from the source file fingerprint.""" + +from uuid import UUID + + +def projection_doc_id(document: dict, fingerprint: str | None = None) -> str: + """A retry keeps its key; another document or source version gets another key.""" + owner = str(UUID(str(document["id"]))) + source = fingerprint or document.get("source_file_fingerprint") or document["document_id"] + return f"document:{owner}:{source}" + + +def projection_owner(identifier: str) -> str | None: + """Recover the local document identity from a new vector key.""" + parts = identifier.split(":", 2) + if len(parts) != 3 or parts[0] != "document" or not parts[2]: + return None + try: + return str(UUID(parts[1])) + except ValueError: + return None + + +def recorded_doc_id(document: dict, environment: str) -> str: + """Read the actual indexed key, including pre-migration fingerprint keys.""" + entry = (document.get("index_status") or {}).get(environment) or {} + return entry.get("doc_id") or document.get("document_id") or "" diff --git a/pipeline/search/passage_projection.py b/pipeline/search/passage_projection.py new file mode 100644 index 0000000..03fda78 --- /dev/null +++ b/pipeline/search/passage_projection.py @@ -0,0 +1,335 @@ +"""Prepare search passages and frozen revision content from stored documents.""" + +import hashlib +import os +import re +from pathlib import Path + +from pipeline.documents import document_validity +from pipeline.network.catalog import scheme_catalog +from pipeline.documents.document_text import _infer_section, is_reference_section +from pipeline.network.release_policy import instance_display_name +from pipeline.metadata.knowledge_metadata import KnowledgeMetadata +from pipeline.metadata.localization import project_chunk_localization + + +def clean_text_for_ingestion(text: str) -> str: + """Clean translation preambles from text before ingestion.""" + if not text: + return text + + result = text + result = re.sub( + r"^Here is the translated text from \*\*[^*]+\*\* to English[^:]*:?\s*\n*-{0,3}\s*\n*", + "", + result, + flags=re.IGNORECASE, + ) + result = re.sub( + r"^Here is the translated text from [^:]+?:\s*\n*-{0,3}\s*\n*", + "", + result, + flags=re.IGNORECASE, + ) + result = re.sub( + r"^Here is the translated text[^:]*:?\s*\n*-{0,3}\s*\n*", + "", + result, + flags=re.IGNORECASE, + ) + result = re.sub( + r"^Here is the (?:English )?translation[^:]*:?\s*\n*", + "", + result, + flags=re.IGNORECASE | re.MULTILINE, + ) + result = re.sub( + r"^Here is the translated text with[^:]+:?\s*\n*-{0,3}\s*\n*", + "", + result, + flags=re.IGNORECASE, + ) + + prefixes = [ + r"^(?:the\s+)?english\s+translation:?\s*\n*", + r"^(?:the\s+)?translation:?\s*\n*", + r"^translated\s+(?:text|content):?\s*\n*", + r"^##?\s*(?:english\s+)?translation\s*\n+", + r"^---+\s*\n+", + r"^\*\*Translation:?\*\*\s*\n*", + ] + for pattern in prefixes: + result = re.sub(pattern, "", result, flags=re.IGNORECASE | re.MULTILINE) + + result = re.sub(r"\n*-{3,}\s*$", "", result) + return result.strip() + + +def _normalize_instance(value: str | None) -> str: + """Ingest-side instance normalizer (mirrors auth.tenancy without importing FastAPI).""" + text = (value or "").strip().lower() + if text: + return text + return (os.environ.get("DEFAULT_INSTANCE") or "default").strip().lower() or "default" + + +def _prepare_records( + document_id: str, + filename: str, + chunks: list[dict], + workflow_id: str | None = None, + description: str | None = None, + include_e5_prefix_field: bool = True, + instance: str | None = None, + kind: str = "document", + code: str | None = None, + name: str | None = None, + aliases: list[str] | None = None, + valid_from: str | None = None, + valid_to: str | None = None, + provider_id: str | None = None, + display_name: str | None = None, + crop_subjects: list[dict] | None = None, + knowledge_metadata: dict | None = None, + source_url: str | None = None, + pages: list[dict] | None = None, + source_first: bool = False, +) -> list[dict]: + resolved_instance = _normalize_instance(instance) + doc_hash = hashlib.md5(document_id.encode()).hexdigest() + external_slug = workflow_id or filename + display_title = display_name or name or Path(filename).stem + + kind = (kind or "document").strip().lower() + is_scheme = kind == "scheme" and bool((code or "").strip()) + scheme_name = name or display_title + # Merge curator-supplied, bootstrap and derived aliases so a scheme is never + # indexed with an empty alias list (it would then only match its exact name). + resolved_aliases = ( + scheme_catalog.resolve_scheme_aliases(code, scheme_name, aliases) + if is_scheme + else [] + ) + instance_name = instance_display_name(resolved_instance) + # One period for the whole document, resolved before the loop: every chunk + # of a document expires together, and re-deriving it per chunk would let a + # midnight boundary split one document across two periods. + validity = document_validity.period_from_row(valid_from, valid_to) + metadata = KnowledgeMetadata.model_validate(knowledge_metadata or {}) + categories = list(metadata.subject_categories) + if crop_subjects and "Crop" not in categories: + categories.append("Crop") + if is_scheme and "Scheme" not in categories: + categories.append("Scheme") + subject_ids = [entry.get("subjectId") for entry in crop_subjects or []] + subject_ids.extend(subject.subjectId for subject in metadata.agriculture_subjects) + applicability = { + "subject_categories": categories, + "agriculture_subject_ids": list(dict.fromkeys(subject_id for subject_id in subject_ids if subject_id)), + "coverage_area_keys": [area.lookup_key() for area in metadata.coverage_areas], + "excluded_coverage_area_keys": [area.lookup_key() for area in metadata.excluded_coverage_areas], + "season_keys": [season.casefold() for season in metadata.seasons], + "applicability_groups": [group.index_value() for group in metadata.applicability_groups], + "tag_keys": [tag.casefold() for tag in metadata.tags], + } + + records = [] + for chunk in chunks: + if chunk.get("is_excluded", False): + continue + + raw_text = chunk.get("edited_text") or chunk.get("original_text", "") + chunk_num = chunk.get("chunk_number", 0) + text = clean_text_for_ingestion(raw_text) + is_ref = is_reference_section(text) + + section = _infer_section(text, chunk.get("section_title")) + point_id = hashlib.md5(f"{doc_hash}_{chunk_num}_{text[:50]}".encode()).hexdigest() + record = { + "_id": point_id, + "doc_id": document_id, + "workflow_id": workflow_id or "", + "instance": resolved_instance, # vector payload key, kept for consumers + "instance_name": instance_name, + "provider_id": provider_id, + "crop_subjects": crop_subjects or [], + "type": kind, + "source": "docs-pipeline", + "filename": external_slug, + "name": display_title, + "description": description or metadata.description or "", + "text": text, + "chunk_num": chunk_num, + "chunk_id": point_id, + "chunk_index": chunk_num, + "section": section, + "token_count": chunk.get("token_count", 0), + "page_start": chunk.get("page_start", 1), + "page_end": chunk.get("page_end", 1), + "is_reference": is_ref, + } + if validity is not None: + record["start_date"] = validity.start_date + # Omitted, not null, when the document never expires: the search + # filter's "end_date missing" branch is what keeps it answerable + # forever, and _record_payload drops None anyway. + if validity.end_date is not None: + record["end_date"] = validity.end_date + record["applicability_scope"] = "broad" if metadata.broad_applicability else "declared" + record.update({key: values for key, values in applicability.items() if values}) + if chunk.get("tags"): + record["chunk_tags"] = chunk["tags"] + record["tag_keys"] = list(dict.fromkeys([ + *applicability["tag_keys"], *(tag.casefold() for tag in chunk["tags"]), + ])) + if knowledge_metadata: + record["knowledge_metadata"] = metadata.storage_value() + if source_url: + record["source_url"] = source_url + if is_scheme: + record["scheme_code"] = (code or "").strip().lower() + record["scheme_name"] = scheme_name + record["scheme_aliases"] = resolved_aliases + if include_e5_prefix_field: + record["text_for_embedding"] = f"passage: {text}" if text else "passage:" + if source_first and pages: + record.update(project_chunk_localization(chunk, pages, source_first=True)) + records.append(record) + + return records + + +def _scheme_fields_from_doc(doc: dict | None) -> dict: + """Extract kind/code/name/aliases kwargs for _prepare_records from a documents row.""" + doc = doc or {} + return { + "kind": (doc.get("kind") or "document"), + "code": doc.get("code"), + "name": doc.get("name"), + "aliases": [str(a) for a in (doc.get("aliases") or [])], + "crop_subjects": doc.get("crop_subjects") or [], + "knowledge_metadata": doc.get("knowledge_metadata") or {}, + } + + +def _validity_fields_from_doc(doc: dict | None) -> dict: + """Extract the validity kwargs for `_prepare_records` from a documents row. + + A row with no stored period is stamped with the default anchored on its + upload day - start there, no end - so every chunk written from here on + carries a start. Old points already in the index keep none until their + document is reingested, which is what keeps them searchable meanwhile. + """ + doc = doc or {} + period = document_validity.period_from_row( + doc.get("valid_from"), doc.get("valid_to") + ) + if period is None: + period = document_validity.period_from_upload_timestamp(doc.get("created_at")) + return {"valid_from": period.start_date, "valid_to": period.end_date} + + +def prepare_ingestion_records( + document_id: str, + filename: str, + chunks: list[dict], + workflow_id: str | None = None, + name: str | None = None, + description: str | None = None, + instance: str | None = None, +) -> list[dict]: + """Public helper used by tests and scripts when preparing vector-index payloads.""" + return _prepare_records( + document_id, + filename, + chunks, + workflow_id=workflow_id, + name=name, + description=description, + instance=instance, + ) + + +def _passage_schema_field_names() -> set[str]: + """Field names for the canonical passage schema (E5 text_for_embedding + full metadata).""" + settings = _passage_schema_definition(use_tensor_prefix_field=True) + return {f.get("name") for f in settings.get("allFields", []) if isinstance(f, dict) and f.get("name")} + + +def _passage_schema_definition(use_tensor_prefix_field: bool = True) -> dict: + tensor_field = "text_for_embedding" if use_tensor_prefix_field else "text" + all_fields = [ + {"name": "doc_id", "type": "text", "features": ["filter"]}, + {"name": "workflow_id", "type": "text", "features": ["filter"]}, + {"name": "instance", "type": "text", "features": ["filter"]}, + {"name": "instance_name", "type": "text", "features": ["filter"]}, + {"name": "provider_id", "type": "text", "features": ["filter"]}, + {"name": "type", "type": "text", "features": ["filter"]}, + {"name": "source", "type": "text", "features": ["filter"]}, + {"name": "filename", "type": "text", "features": ["filter"]}, + {"name": "name", "type": "text", "features": ["lexical_search"]}, + {"name": "description", "type": "text", "features": ["lexical_search"]}, + {"name": "chunk_num", "type": "int", "features": ["filter"]}, + {"name": "section", "type": "text", "features": ["filter"]}, + {"name": "token_count", "type": "int", "features": ["filter"]}, + {"name": "page_start", "type": "int", "features": ["filter"]}, + {"name": "page_end", "type": "int", "features": ["filter"]}, + {"name": "is_reference", "type": "bool", "features": ["filter"]}, + {"name": "text", "type": "text", "features": ["lexical_search"]}, + {"name": "priority", "type": "float", "features": ["score_modifier", "filter"]}, + # Document validity - filtered on at search time, so the day a chunk + # starts and stops answering travels with the chunk itself. + {"name": "start_date", "type": "date", "features": ["filter"]}, + {"name": "end_date", "type": "date", "features": ["filter"]}, + ] + if use_tensor_prefix_field: + all_fields.append({"name": "text_for_embedding", "type": "text"}) + + return { + "type": "structured", + "vectorNumericType": "float", + "model": "hf/multilingual-e5-large", + "normalizeEmbeddings": False, + "textPreprocessing": {"splitLength": 3, "splitOverlap": 1, "splitMethod": "sentence"}, + "allFields": all_fields, + "tensorFields": [tensor_field], + } + + +def _revision_snapshot(doc, pages, records, vector_doc_id, source_url): + return { + "vector_doc_id": vector_doc_id, + "source": { + **{key: doc.get(key) for key in ("filename", "filepath", "source_file_fingerprint")}, + "url": source_url, + }, + "metadata": {key: doc.get(key) for key in ( + "instance", "kind", "code", "name", "aliases", "valid_from", "valid_to", "crop_subjects", "knowledge_metadata", + )}, + "pages": [{key: page.get(key) for key in ( + "page_number", "detected_language", "original_markdown", "edited_markdown", "translations", "metadata", + )} for page in pages], + "passages": records, + } + + +def _draft_revision_passages(records: list[dict], revision_doc_id: str) -> list[dict]: + passages = [] + for record in records: + if not record["text"].strip(): + continue + point_id = hashlib.md5(f"{revision_doc_id}:{record['_id']}".encode()).hexdigest() + passages.append({ + **{key: value for key, value in record.items() if key != "provider_id"}, + "_id": point_id, + "chunk_id": point_id, + "doc_id": revision_doc_id, + }) + return passages + + +def _release_passages(revision: dict, network_id: str, provider_id: str | None) -> list[dict]: + return [ + {**record, "revision_id": revision["id"], "release_network_id": network_id, "provider_id": provider_id} + for record in revision["snapshot"]["passages"] + ] diff --git a/pipeline/search/query_expansion.py b/pipeline/search/query_expansion.py new file mode 100644 index 0000000..335af3d --- /dev/null +++ b/pipeline/search/query_expansion.py @@ -0,0 +1,5 @@ +"""Normalize search queries before ranking.""" + + +def expand_query(query: str, profile: str) -> str: + return query.strip() diff --git a/pipeline/telemetry/__init__.py b/pipeline/telemetry/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/pipeline/temporal/__init__.py b/pipeline/temporal/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/pipeline/temporal/activities/__init__.py b/pipeline/temporal/activities/__init__.py new file mode 100644 index 0000000..876ef6d --- /dev/null +++ b/pipeline/temporal/activities/__init__.py @@ -0,0 +1 @@ +"""Temporal activity implementations, organized by the work they perform.""" diff --git a/pipeline/temporal/activities/artifacts.py b/pipeline/temporal/activities/artifacts.py new file mode 100644 index 0000000..c3bd3e8 --- /dev/null +++ b/pipeline/temporal/activities/artifacts.py @@ -0,0 +1,52 @@ +"""Worker helpers for local copies and processing artifacts.""" + +import json +import mimetypes +import os +import re +import tempfile +from pathlib import Path + +from pipeline.object_storage import ( + ObjectReference, + ObjectStore, + load_object_store_config, + open_object_store, + parse_object_uri, +) + + +def get_object_store() -> ObjectStore: + return open_object_store(load_object_store_config()) + + +def download_object(storage_uri: str) -> str: + ref = parse_object_uri(storage_uri) + temp_path = None + try: + with tempfile.NamedTemporaryFile(delete=False, suffix=Path(ref.key).suffix) as file: + temp_path = file.name + for chunk in get_object_store().open(ref): + file.write(chunk) + return temp_path + except Exception: + if temp_path: + Path(temp_path).unlink(missing_ok=True) + raise + + +def _upload_artifact(local_path: str, workflow_id: str, artifact_type: str, filename: str) -> tuple[str, int, str]: + config = load_object_store_config() + safe_name = re.sub(r"[^A-Za-z0-9._-]+", "_", filename or "artifact") + ref = ObjectReference(config.bucket, f"documents/{workflow_id}/{artifact_type}/{safe_name}") + mime_type = mimetypes.guess_type(filename)[0] or "application/octet-stream" + size_bytes = os.path.getsize(local_path) + with open(local_path, "rb") as file: + open_object_store(config).put(ref, file, size_bytes, mime_type) + return ref.to_uri(), size_bytes, mime_type + + +def _write_json_temp(data: object, suffix: str = ".json") -> str: + with tempfile.NamedTemporaryFile("w", delete=False, suffix=suffix, encoding="utf-8") as f: + json.dump(data, f, ensure_ascii=False, indent=2) + return f.name diff --git a/pipeline/temporal/activities/indexing.py b/pipeline/temporal/activities/indexing.py new file mode 100644 index 0000000..68cbbb5 --- /dev/null +++ b/pipeline/temporal/activities/indexing.py @@ -0,0 +1,431 @@ +"""Activities for writing and maintaining search projections.""" + +import asyncio +import os +from datetime import datetime +from uuid import uuid4 + +from temporalio import activity +from temporalio.exceptions import ApplicationError + +from pipeline.search.index_identity import projection_doc_id +from pipeline.network.release_policy import ( + index_only_mode, + managed_release_targets, + upload_release_target, +) +from pipeline.network.network_provider import InvalidProviderIdError +from pipeline.network.network_publisher import PublishConfigError, get_network_adapter +from pipeline.search.passage_projection import ( + _draft_revision_passages, + _prepare_records, + _release_passages, + _revision_snapshot, + _scheme_fields_from_doc, + _validity_fields_from_doc, +) +from pipeline.temporal.activities.artifacts import _upload_artifact, _write_json_temp + + +def _provider_id_from_doc(doc: dict | None, network_id: str | None = None) -> str | None: + doc = doc or {} + try: + kind = (doc.get("kind") or "document").strip().lower() + provider_id = get_network_adapter(network_id).provider_id(kind, doc.get("instance")) + if provider_id is not None and ( + not isinstance(provider_id, str) or not 1 <= len(provider_id) <= 256 + or provider_id != provider_id.lower() or any(char.isspace() or ord(char) < 32 for char in provider_id) + ): + raise PublishConfigError("The network adapter returned an invalid search provider identity") + except (InvalidProviderIdError, PublishConfigError) as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + return provider_id + + +def _draft_provider_id(doc: dict) -> str | None: + return doc.get("instance") if index_only_mode() else _provider_id_from_doc(doc) + + +@activity.defn +async def prepare_for_ingestion( + document_id: str, + filename: str, + chunks: list[dict], + workflow_id: str | None = None, + legacy_secondary_title: str | None = None, + legacy_primary_title: str | None = None, + description: str = None, +) -> list[dict]: + """Prepare chunks for vector-index ingestion.""" + from pipeline import db + + activity.logger.info(f"Preparing {len(chunks)} chunks for ingestion") + doc = db.get_document(workflow_id) if workflow_id else None + if doc: + document_id = projection_doc_id(doc, document_id) + records = _prepare_records( + document_id, + filename, + chunks, + workflow_id=workflow_id, + description=description, + instance=(doc or {}).get("instance"), + **_validity_fields_from_doc(doc), + provider_id=_provider_id_from_doc(doc), + display_name=legacy_primary_title or (doc or {}).get("name") or legacy_secondary_title, + **_scheme_fields_from_doc(doc), + ) + activity.logger.info(f"Prepared {len(records)} records") + return records + + +@activity.defn +async def ingest_to_vector_db( + records: list[dict], + index_name: str = "documents-index", + batch_size: int = 10, +) -> dict: + """Ingest records into the Qdrant vector index.""" + from pipeline.vector_store import get_default_index_name, get_vector_store + + if not index_name or index_name == "documents-index": + index_name = get_default_index_name() or index_name + + activity.logger.info( + "Ingesting %s records to Qdrant collection %s", + len(records), + index_name, + ) + store = get_vector_store() + # store.upsert() runs CPU-bound embedding synchronously (model.encode()) — + # off the event loop so it doesn't stall Temporal's query/heartbeat handling. + result = await asyncio.to_thread( + store.upsert, index_name, records, batch_size=max(batch_size, 8) + ) + activity.logger.info("Qdrant ingestion complete: %s", result.get("index_stats")) + return result + + +@activity.defn +async def promote_document_to_prod_qdrant( + workflow_id: str, + document_id: str, + filename: str, + batch_size: int = 10, +) -> dict: + """Re-upsert document chunks into the PROD Qdrant collection (from Postgres).""" + from pipeline import db + from pipeline.vector_store.qdrant_store import QdrantVectorStore, get_qdrant_client + + prod_url = (os.environ.get("PROD_VECTOR_DB_URL") or "").strip() + prod_key = (os.environ.get("PROD_VECTOR_DB_API_KEY") or "").strip() or None + prod_collection = ( + os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME") + or os.environ.get("VECTOR_DB_COLLECTION_NAME") + or "documents-index" + ).strip() + if not prod_url: + raise RuntimeError("PROD_VECTOR_DB_URL is required to promote documents to prod") + + timeout = float( + os.environ.get("PROD_VECTOR_DB_TIMEOUT_SECONDS") + or os.environ.get("VECTOR_DB_TIMEOUT_SECONDS") + or "30" + ) + # Temporarily honor prod timeout for this client build via env used by get_qdrant_client + prev_timeout = os.environ.get("VECTOR_DB_TIMEOUT_SECONDS") + os.environ["VECTOR_DB_TIMEOUT_SECONDS"] = str(timeout) + try: + client = get_qdrant_client(url=prod_url, api_key=prod_key) + finally: + if prev_timeout is None: + os.environ.pop("VECTOR_DB_TIMEOUT_SECONDS", None) + else: + os.environ["VECTOR_DB_TIMEOUT_SECONDS"] = prev_timeout + + chunks = db.get_chunks(workflow_id, include_excluded=True) + doc = db.get_document(workflow_id) or {} + document_id = projection_doc_id(doc, document_id) + scheme_kwargs = _scheme_fields_from_doc(doc) + kind = (scheme_kwargs.get("kind") or "document").strip().lower() + is_scheme = kind == "scheme" and bool((scheme_kwargs.get("code") or "").strip()) + + if is_scheme: + prod_url = ( + (os.environ.get("PROD_SCHEME_VECTOR_DB_URL") or "").strip() + or prod_url + ) + prod_key = ( + (os.environ.get("PROD_SCHEME_VECTOR_DB_API_KEY") or "").strip() + or prod_key + ) + prod_collection = ( + os.environ.get("PROD_SCHEME_VECTOR_DB_COLLECTION_NAME") or "schemes-index" + ).strip() + docs_default = ( + os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME") + or os.environ.get("VECTOR_DB_COLLECTION_NAME") + or "documents-index" + ).strip() + if prod_collection == docs_default: + raise RuntimeError( + "Scheme promote refused: PROD_SCHEME_VECTOR_DB_COLLECTION_NAME must not " + f"equal documents collection ({docs_default}). Set schemes-index." + ) + # Rebuild client if scheme URL differs + scheme_url = (os.environ.get("PROD_SCHEME_VECTOR_DB_URL") or "").strip() + if scheme_url and scheme_url != (os.environ.get("PROD_VECTOR_DB_URL") or "").strip(): + client = get_qdrant_client(url=scheme_url, api_key=prod_key) + + records = _prepare_records( + document_id, + filename, + chunks, + workflow_id=workflow_id, + instance=doc.get("instance"), + **scheme_kwargs, + **_validity_fields_from_doc(doc), + provider_id=_provider_id_from_doc(doc), + display_name=doc.get("name"), + ) + activity.logger.info( + "Promoting %s records to PROD Qdrant collection %s (kind=%s)", + len(records), + prod_collection, + kind, + ) + store = QdrantVectorStore(client=client) + result = await asyncio.to_thread( + store.upsert, prod_collection, records, batch_size=max(batch_size, 8) + ) + activity.logger.info("PROD promotion complete: %s", result.get("index_stats")) + db.upsert_document_index_status( + workflow_id=workflow_id, index_name=prod_collection, doc_id=document_id, + chunk_count_indexed=result.get("records_ingested", 0), + last_indexed_at=datetime.utcnow().isoformat(), schema_version="passage-v2", + status="indexed", details=result.get("index_stats"), + ) + await _drop_superseded_points("prod", workflow_id, doc, document_id, prod_collection, store=store) + db.log_audit( + workflow_id=workflow_id, + document_id=document_id, + action_type="promote_to_prod", + entity_type="document", + metadata={ + "collection": prod_collection, + "records_ingested": result.get("records_ingested", len(records)), + "prod_qdrant_url": prod_url, + "kind": kind, + "code": scheme_kwargs.get("code"), + }, + ) + catalog_version = None + if is_scheme: + try: + from pipeline.network.catalog.scheme_catalog import on_scheme_document_promoted + + catalog_version = on_scheme_document_promoted(workflow_id) + except Exception as exc: + activity.logger.warning("Catalog update after promote failed: %s", exc) + return { + **result, + "collection": prod_collection, + "target": "prod", + "document_kind": kind, + "catalog_version": catalog_version, + } + + +async def _drop_superseded_points( + env: str, + workflow_id: str, + doc: dict, + current_doc_id: str, + collection: str, + store=None, +) -> None: + """Delete points of earlier file versions (see ``metadata.superseded_doc_ids``). + + Runs after the new content is indexed, so search never has a gap. A failed + delete keeps the key for the next run and does not fail the ingest. + """ + from pipeline import db + + previous = ((doc.get("index_status") or {}).get(env) or {}).get("doc_id") + stale = [ + doc_id + for doc_id in dict.fromkeys([*(doc.get("superseded_doc_ids") or {}).get(env, []), previous]) + if doc_id and doc_id != current_doc_id + ] + if not stale: + return + if store is None: + from pipeline.vector_store import get_vector_store + + store = get_vector_store() + deleted = [] + for doc_id in stale: + # Two documents holding the same file share one index key. While another + # live document is indexed under it the points stay, and the key stays on + # the list so a later ingest, disable or purge can still remove it. + if await asyncio.to_thread(db.doc_id_used_by_other_document, doc_id, doc.get("id"), env): + activity.logger.info( + "Keeping %s points for %s: another document is indexed under the same key", env, doc_id + ) + continue + result = await asyncio.to_thread(store.delete_by_doc_id, collection, doc_id) + if result.get("error"): + activity.logger.warning( + "Could not delete superseded %s points for %s: %s", env, doc_id, result["error"] + ) + else: + deleted.append(doc_id) + if deleted: + db.clear_superseded_doc_ids(workflow_id, env, deleted) + + +def _release_target_for_job( + doc: dict, job: dict | None, workflow_id: str, release_network_id: str, +) -> tuple[str, str] | None: + if not job or job.get("temporal_workflow_id") != workflow_id: + return None + kind = doc.get("kind") + config = job["config"] + if not config.get("explicit_release_network"): + return upload_release_target(kind, provider=doc.get("instance")) + if config.get("release_network_id") != release_network_id: + return None + collection = managed_release_targets(kind).get(release_network_id) + if not collection or config.get("release_collection") != collection: + return None + return release_network_id, collection + + +@activity.defn +async def ingest_document_from_db( + workflow_id: str, + document_id: str, + filename: str, + index_name: str = "documents-index", + batch_size: int = 10, + release_network_id: str | None = None, + source_first: bool = False, + freeze_draft_revision: bool = False, + revision_workflow_id: str | None = None, +) -> dict: + """Prepare and ingest chunks directly from Postgres by workflow_id.""" + from pipeline import db + + if release_network_id or freeze_draft_revision or source_first: + doc, pages, chunks = db.load_revision_material(workflow_id) + if not doc or doc.get("is_disabled"): + raise ApplicationError("The document is unavailable for indexing", non_retryable=True) + job = db.get_document_job(doc["latest_job_id"]) + source_url = ((job or {}).get("config") or {}).get("source_url") + if source_url is None and revision_workflow_id: + source_url = db.get_document_source_url(doc["id"]) + else: + chunks = db.get_chunks(workflow_id, include_excluded=True) + doc = db.get_document(workflow_id) or {} + pages = None + source_url = None + + if release_network_id: + try: + target = _release_target_for_job(doc, job, workflow_id, release_network_id) + except ValueError as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + if target != (release_network_id, index_name): + raise ApplicationError("The upload release target changed", non_retryable=True) + if not doc.get("network_visible"): + raise ApplicationError("The document is unavailable for release", non_retryable=True) + revision_id = str(uuid4()) + document_id = revision_id + document_id = projection_doc_id(doc, document_id) + records = _prepare_records( + document_id, + filename, + chunks, + workflow_id=workflow_id, + instance=doc.get("instance"), + **_scheme_fields_from_doc(doc), + **_validity_fields_from_doc(doc), + provider_id=None if release_network_id else _draft_provider_id(doc), + display_name=doc.get("name"), + pages=pages, + source_first=source_first, + source_url=source_url, + ) + if release_network_id: + records = [record for record in records if record["text"].strip()] + if not records: + raise ApplicationError("The document has no searchable content", non_retryable=True) + snapshot = _revision_snapshot(doc, pages, records, document_id, source_url) + try: + revision = db.save_revision_snapshot( + doc["id"], workflow_id, release_network_id, revision_id, snapshot, + collection=index_name, provider_id=_provider_id_from_doc(doc, release_network_id), + ) + except ValueError as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + document_id = revision["snapshot"]["vector_doc_id"] + if revision["stage"]["collection"] != index_name: + raise ApplicationError("The staged release collection changed", non_retryable=True) + records = _release_passages(revision, release_network_id, revision["stage"]["provider_id"]) + elif index_only_mode(): + records = [record for record in records if record["text"].strip()] + if not records: + raise ApplicationError("The document has no searchable content", non_retryable=True) + if freeze_draft_revision and not release_network_id: + revision_id = str(uuid4()) + revision_doc_id = projection_doc_id(doc, revision_id) + passages = _draft_revision_passages(records, revision_doc_id) + if passages: + snapshot = _revision_snapshot(doc, pages, passages, revision_doc_id, source_url) + try: + db.save_draft_revision_snapshot(doc["id"], revision_workflow_id or workflow_id, revision_id, snapshot) + except ValueError as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + payload_path = _write_json_temp(records) + try: + payload_uri, payload_size, payload_mime = _upload_artifact( + payload_path, workflow_id, "vector_payload_export", "vector_payload.json" + ) + finally: + if os.path.exists(payload_path): + os.remove(payload_path) + latest_job = db.get_latest_document_job(workflow_id) + db.add_document_artifact( + workflow_id=workflow_id, + job_id=latest_job["id"] if latest_job else None, + artifact_type="vector_payload_export", + stage="ingesting", + storage_uri=payload_uri, + mime_type=payload_mime, + filename="vector_payload.json", + size_bytes=payload_size, + metadata={"record_count": len(records), "index_name": index_name}, + ) + result = await ingest_to_vector_db(records, index_name=index_name, batch_size=batch_size) + if (release_network_id or index_only_mode()) and result.get("records_ingested") != len(records): + raise ApplicationError("The index did not ingest every eligible passage") + if index_only_mode() and not release_network_id and result.get("backend") == "qdrant": + from pipeline.vector_store import get_default_index_name + from pipeline.vector_store.qdrant_store import QdrantVectorStore + + collection = get_default_index_name() if index_name == "documents-index" else index_name + await asyncio.to_thread(QdrantVectorStore().remove_superseded_passages, collection, document_id, records) + db.upsert_document_index_status( + workflow_id=workflow_id, + index_name=index_name, + doc_id=document_id, + chunk_count_indexed=result.get("records_ingested", 0), + last_indexed_at=datetime.utcnow().isoformat(), + last_verified_at=datetime.utcnow().isoformat(), + schema_version="passage-v2", + status="indexed", + details=result.get("index_stats"), + ) + if not release_network_id: + await _drop_superseded_points("dev", workflow_id, doc, document_id, index_name) + return result diff --git a/pipeline/temporal/activities/logging.py b/pipeline/temporal/activities/logging.py index 67ffd9e..192e391 100644 --- a/pipeline/temporal/activities/logging.py +++ b/pipeline/temporal/activities/logging.py @@ -1,3 +1,5 @@ +"""Worker-side activity and workflow logging.""" + from __future__ import annotations import asyncio @@ -15,7 +17,7 @@ WorkflowInterceptorClassInput, ) -logger = logging.getLogger(__name__) +logger = logging.getLogger("pipeline.activity_logging") class ActivityLogging(Interceptor): diff --git a/pipeline/temporal/activities/processing.py b/pipeline/temporal/activities/processing.py new file mode 100644 index 0000000..7757354 --- /dev/null +++ b/pipeline/temporal/activities/processing.py @@ -0,0 +1,256 @@ +"""Activities for chunking and translation.""" + +import json +import os +from dataclasses import replace +from datetime import datetime +from uuid import uuid4 + +from temporalio import activity + +from pipeline.chunking import chunk_pages, load_chunking_config +from pipeline.network.release_policy import index_only_mode +from pipeline.translation import load_translation_config, translate_pages +from pipeline.temporal.activities.artifacts import _upload_artifact, _write_json_temp + + +async def _detect_and_translate_impl( + pages: list[dict], + target_language: str = "en", + source_language: str | None = None, +) -> list[dict]: + del source_language + config = load_translation_config(target_language=target_language) + return await translate_pages( + pages, + target_language=target_language, + config=config, + log=activity.logger.info, + ) + + +@activity.defn +async def create_chunks( + pages: list[dict], + chunk_size: int = 450, + chunk_overlap: int = 128, + min_tokens: int = 100, +) -> list[dict]: + """Create chunks from pages with page range tracking.""" + activity.logger.info(f"Creating chunks from {len(pages)} pages") + config = load_chunking_config(chunk_size=chunk_size, chunk_overlap=chunk_overlap, min_tokens=min_tokens) + result = await chunk_pages(pages, config) + chunks = [] + for idx, chunk in enumerate(result.chunks, 1): + chunks.append( + { + "chunk_number": idx, + "original_text": chunk.text, + "edited_text": None, + "token_count": chunk.token_count, + "page_start": chunk.page_start, + "page_end": chunk.page_end, + "source_page_numbers_json": json.dumps(chunk.source_page_numbers), + "source_spans_json": json.dumps(chunk.source_spans), + "section_title": chunk.section_title, + "content_type": chunk.content_type, + "is_reference": chunk.is_reference, + "chunking_provider": result.provider, + "chunking_model": result.model, + "chunking_config_json": result.config.to_json(), + "chunking_run_id": "", + "chunk_version": 1, + "is_reviewed": False, + "is_excluded": False, + "reviewer_notes": None, + } + ) + activity.logger.info(f"Created {len(chunks)} chunks with page tracking") + return chunks + + +@activity.defn +async def create_chunks_from_db( + workflow_id: str, + chunk_size: int = 450, + chunk_overlap: int = 128, + min_tokens: int = 100, + source_first: bool = False, +) -> dict: + """Create chunks from persisted pages and persist chunks in Postgres.""" + from pipeline import db + + pages = db.get_pages(workflow_id) + activity.logger.info(f"Creating chunks from DB pages for {workflow_id}: {len(pages)} pages") + config = load_chunking_config(chunk_size=chunk_size, chunk_overlap=chunk_overlap, min_tokens=min_tokens) + latest_job = db.get_latest_document_job(workflow_id) + base_job_config = {} + if latest_job and latest_job.get("config_json"): + try: + base_job_config = json.loads(latest_job["config_json"]) or {} + except Exception: + base_job_config = {} + + async def _persist_chunking_progress(event: dict) -> None: + if not latest_job: + return + pages_total = int(event.get("pages_total") or len(pages) or 0) + pages_processed = int(event.get("pages_processed") or 0) + chunks_emitted = int(event.get("chunks_emitted") or 0) + raw_percent = float(event.get("percent") or 0.0) + percent = max(0.0, min(100.0, raw_percent)) + progress = { + "status": "running" if percent < 100.0 else "completed", + "provider": event.get("provider") or config.provider, + "pages_processed": pages_processed, + "pages_total": pages_total, + "chunks_emitted": chunks_emitted, + "percent": round(percent, 2), + "updated_at": datetime.utcnow().isoformat(), + } + next_config = {**base_job_config, "chunking_progress": progress} + db.update_document_job(latest_job["id"], config_json=next_config) + + await _persist_chunking_progress( + { + "provider": config.provider, + "pages_processed": 0, + "pages_total": len(pages), + "chunks_emitted": 0, + "percent": 0.0, + } + ) + + chunking_pages = pages + if source_first: + chunking_pages = [ + {**page, "edited_translation": None, "translated_markdown": None, "translations": {}} + for page in pages + ] + result = await chunk_pages(chunking_pages, config, progress_callback=_persist_chunking_progress) + if (source_first or index_only_mode()) and not result.chunks and config.min_chunk_tokens > 1: + if any((page.get("edited_markdown") or page.get("original_markdown") or "").strip() for page in chunking_pages): + activity.logger.info("Retrying chunking without the size floor for a short source") + result = await chunk_pages( + chunking_pages, replace(config, min_chunk_tokens=1), progress_callback=_persist_chunking_progress, + ) + chunking_run_id = f"chunk-{workflow_id}-{datetime.utcnow().strftime('%Y%m%d%H%M%S')}" + chunks = [] + for idx, chunk in enumerate(result.chunks, 1): + chunks.append( + { + "chunk_number": idx, + "original_text": chunk.text, + "edited_text": None, + "token_count": chunk.token_count, + "page_start": chunk.page_start, + "page_end": chunk.page_end, + "source_page_numbers_json": json.dumps(chunk.source_page_numbers), + "source_spans_json": json.dumps(chunk.source_spans), + "section_title": chunk.section_title, + "content_type": chunk.content_type, + "is_reference": chunk.is_reference, + "chunking_provider": result.provider, + "chunking_model": result.model, + "chunking_config_json": result.config.to_json(), + "chunking_run_id": chunking_run_id, + "is_reviewed": False, + "is_excluded": False, + "reviewer_notes": None, + } + ) + db.save_chunks(workflow_id, chunks) + chunks_json_path = _write_json_temp(chunks) + try: + chunks_uri, chunks_size, chunks_mime = _upload_artifact( + chunks_json_path, workflow_id, "chunk_json_export", "chunks.json" + ) + finally: + if os.path.exists(chunks_json_path): + os.remove(chunks_json_path) + db.add_document_artifact( + workflow_id=workflow_id, + job_id=latest_job["id"] if latest_job else None, + artifact_type="chunk_json_export", + stage="chunk_review", + storage_uri=chunks_uri, + mime_type=chunks_mime, + filename="chunks.json", + size_bytes=chunks_size, + metadata={ + "chunk_count": len(chunks), + "chunking_provider": result.provider, + "chunking_model": result.model, + "chunking_run_id": chunking_run_id, + "chunking_config": json.loads(result.config.to_json()), + "warnings": result.warnings, + "stats": result.stats, + }, + ) + # Reconcile the document row immediately after chunk persistence so Postgres + # remains truthful even if the workflow fails before its final state update. + db.reconcile_materialized_state(workflow_id) + await _persist_chunking_progress( + { + "provider": result.provider, + "pages_processed": len(pages), + "pages_total": len(pages), + "chunks_emitted": len(chunks), + "percent": 100.0, + } + ) + return {"chunk_count": len(chunks)} + + +@activity.defn +async def detect_and_translate_pages( + pages: list[dict], + target_language: str = "en", + source_language: str = None, +) -> list[dict]: + """Detect language and translate non-English pages.""" + return await _detect_and_translate_impl(pages, target_language=target_language, source_language=source_language) + + +@activity.defn +async def detect_and_translate_pages_from_db( + workflow_id: str, + target_language: str = "en", + source_language: str = None, +) -> dict: + """Detect and translate pages loaded from Postgres; persist updated pages back to Postgres.""" + from pipeline import db + + pages = db.get_pages(workflow_id) + translated = await _detect_and_translate_impl(pages, target_language=target_language, source_language=source_language) + db.save_pages(workflow_id, translated) + translated_count = sum(1 for p in translated if p.get("translated_markdown")) + latest_job = db.get_latest_document_job(workflow_id) + translation_config = load_translation_config(target_language=target_language) + translated_json_path = _write_json_temp(translated) + try: + translated_uri, translated_size, translated_mime = _upload_artifact( + translated_json_path, workflow_id, "translation_pages_json", "translated_pages.json" + ) + finally: + if os.path.exists(translated_json_path): + os.remove(translated_json_path) + db.add_document_artifact( + workflow_id=workflow_id, + job_id=latest_job["id"] if latest_job else None, + artifact_type="translation_pages_json", + stage="translation_review", + storage_uri=translated_uri, + mime_type=translated_mime, + filename="translated_pages.json", + size_bytes=translated_size, + metadata={ + "page_count": len(translated), + "translated_count": translated_count, + "translation_provider": translation_config.provider, + "translation_model": translation_config.model, + "translation_target_language": target_language, + "translation_run_id": str(uuid4()), + }, + ) + return {"page_count": len(translated), "translated_count": translated_count} diff --git a/pipeline/temporal/activities/publication.py b/pipeline/temporal/activities/publication.py new file mode 100644 index 0000000..be76af4 --- /dev/null +++ b/pipeline/temporal/activities/publication.py @@ -0,0 +1,215 @@ +"""Activities for network publication and revision selection.""" + +import asyncio +import os +from uuid import ( + NAMESPACE_URL, + uuid5, +) + +from temporalio import activity +from temporalio.exceptions import ApplicationError + +from pipeline.db.index_state import index_key_for +from pipeline.documents.document_repository import DocumentRepository +from pipeline.network.release_policy import managed_release_targets, publish_to_network_enabled +from pipeline.documents.models import PUBLISH_SKIPPED_DISABLED, PUBLISH_SKIPPED_UNMAPPED_KIND +from pipeline.network.network_constants import RESULT_ACCEPTED +from pipeline.network.network_publisher import ( + PublicationReceipt, + PublishConfigError, + get_network_publisher, +) +from pipeline.search.passage_projection import _release_passages +from pipeline.temporal.activities.artifacts import _upload_artifact, _write_json_temp +from pipeline.temporal.activities.indexing import ( + _drop_superseded_points, + _provider_id_from_doc, + _release_target_for_job, + ingest_to_vector_db, +) + + +@activity.defn +async def publish_catalog_to_network( + workflow_id: str, transaction_id: str, release_network_id: str | None = None, + require_matched_receipt: bool = False, + publication_network_id: str | None = None, + publication_revision_id: str | None = None, +) -> dict: + """Publish through the configured network publisher and record the exchange. + + The catalog depends on the document's knowledge kind, read here rather than + passed in as an activity argument - that keeps every workflow call site, and + any workflow already in flight, untouched. + """ + from pipeline import db + + if release_network_id: + doc = db.get_document(workflow_id) + if not doc or doc.get("is_disabled"): + raise ApplicationError("The document is unavailable for release", non_retryable=True) + job = db.get_document_job(doc["latest_job_id"]) + try: + target = _release_target_for_job(doc, job, workflow_id, release_network_id) + except ValueError as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + if not target or target[0] != release_network_id: + raise ApplicationError("The upload release network changed", non_retryable=True) + index_state = db.get_document_index_status(workflow_id, target[1]) + if not index_state or index_state["status"] != "indexed" or not index_state["chunk_count_indexed"]: + raise ApplicationError("The automatic release has no indexed content", non_retryable=True) + + repository = DocumentRepository() + document_kind = (repository.get_document_kind(workflow_id) or "document").strip().lower() + + target_network_id = release_network_id or publication_network_id + if publish_to_network_enabled(): + group = repository.get_document_group(workflow_id) + try: + if target_network_id: + from pipeline.network.publication_delivery import deliver_publication + + publication_document = doc if release_network_id else db.get_document(workflow_id) + if not publication_document: + raise PublishConfigError("The document is unavailable for publication") + receipt = await asyncio.to_thread( + deliver_publication, target_network_id, transaction_id, publication_document["id"], + workflow_id, document_kind, group, publication_revision_id, + require_staged_revision=bool(release_network_id), + ) + else: + publisher = get_network_publisher() + receipt = await asyncio.to_thread( + publisher.publish, transaction_id, document_kind, workflow_id, group + ) + except PublishConfigError as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + skip = None + if receipt.exchange is None: + skip = (PUBLISH_SKIPPED_UNMAPPED_KIND, f"No network catalog is mapped to document kind '{document_kind}'.") + else: + receipt = PublicationReceipt() + skip = (PUBLISH_SKIPPED_DISABLED, "Publish to Network is disabled (ENABLE_PUBLISH_TO_NETWORK is not true).") + + if skip: + artifact_type = "network_publish_skipped" + filename = "network_publish_skipped.json" + exchange = { + "skipped": True, + "document_kind": document_kind, + "reason_code": skip[0], + "reason": skip[1], + } + else: + artifact_type = "network_publish_payload" + filename = "network_publish.json" + exchange = {**receipt.exchange, "result_status": receipt.result_status, "context_matches": receipt.context_matches} + exchange.pop("failure_reason", None) + if receipt.rejection: + exchange["failure_reason"] = receipt.rejection + elif release_network_id or require_matched_receipt: + if not receipt.context_matches: + exchange.update(result_status=None, failure_reason="Receipt context did not match the release") + elif receipt.result_status != RESULT_ACCEPTED: + exchange["failure_reason"] = "The network did not accept the release" + + # Recorded even when skipped: add_document_artifact requires a storage_uri, + # and an empty one 404s the artifact-content endpoint if an operator clicks it. + artifact_identity = {} + if target_network_id: + artifact_id = str(uuid5(NAMESPACE_URL, f"kp-publication:{target_network_id}:{transaction_id}")) + artifact_identity = {"artifact_id": artifact_id} + filename = f"{artifact_type}-{artifact_id}.json" + exchange_path = _write_json_temp(exchange) + try: + exchange_uri, exchange_size, exchange_mime = _upload_artifact( + exchange_path, workflow_id, artifact_type, filename + ) + finally: + if os.path.exists(exchange_path): + os.remove(exchange_path) + + latest_job = db.get_latest_document_job(workflow_id) + db.add_document_artifact( + workflow_id=workflow_id, + job_id=latest_job["id"] if latest_job else None, + artifact_type=artifact_type, + stage="publishing_to_network", + storage_uri=exchange_uri, + mime_type=exchange_mime, + filename=filename, + size_bytes=exchange_size, + metadata=exchange, + **artifact_identity, + ) + + if receipt.rejection: + # Record the exchange before reporting the deterministic rejection. + raise ApplicationError(receipt.rejection, non_retryable=True) + if (release_network_id or require_matched_receipt) and ( + receipt.result_status != RESULT_ACCEPTED or not receipt.context_matches + ): + raise ApplicationError("The network did not accept the automatic release", non_retryable=True) + if release_network_id: + try: + db.activate_network_release(doc["id"], workflow_id, release_network_id, transaction_id) + except ValueError as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + await _drop_superseded_points( + index_key_for(target[1]), workflow_id, doc, index_state["doc_id"], target[1], + ) + + return { + "status": "skipped" if skip else "published", + "reason_code": skip[0] if skip else None, + "document_kind": document_kind, + "transaction_id": transaction_id, + "message_id": receipt.message_id, + "result_status": receipt.result_status, + } + + +@activity.defn +async def release_saved_revision( + document_id: str, revision_id: str, network_id: str, transaction_id: str, expected_generation: str | None, +) -> dict: + """Restore the frozen passages before selecting this revision on its network.""" + from pipeline import db + + document = db.get_document(document_id) + try: + collection = managed_release_targets((document or {}).get("kind")).get(network_id) + except ValueError as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + if not collection: + raise ApplicationError("This deployment cannot release to the selected network", non_retryable=True) + revision = db.get_revision_snapshot(document_id, revision_id) + if not document or document["is_disabled"] or not document["network_visible"] or not revision: + raise ApplicationError("The revision is unavailable for release", non_retryable=True) + + provider_id = _provider_id_from_doc(document, network_id) + outcome = {"revision_id": revision_id, "network_id": network_id, "transaction_id": transaction_id} + try: + staged = db.stage_saved_network_revision( + document_id, network_id, revision_id, transaction_id, expected_generation, collection, provider_id, + ) + except ValueError as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + if not staged: + return outcome + records = _release_passages(revision, network_id, provider_id) + result = await ingest_to_vector_db(records, index_name=collection) + if not records or result.get("records_ingested") != len(records): + raise ApplicationError("The release index did not ingest every eligible passage") + receipt = await publish_catalog_to_network( + document["workflow_id"], transaction_id, require_matched_receipt=True, publication_network_id=network_id, + publication_revision_id=revision_id, + ) + if receipt["result_status"] != RESULT_ACCEPTED: + raise ApplicationError("The network did not accept the selected revision", non_retryable=True) + try: + db.select_network_revision(document_id, network_id, revision_id, transaction_id, expected_generation) + except ValueError as exc: + raise ApplicationError(str(exc), non_retryable=True) from exc + return outcome diff --git a/pipeline/temporal/activities/source.py b/pipeline/temporal/activities/source.py new file mode 100644 index 0000000..e9c5ccd --- /dev/null +++ b/pipeline/temporal/activities/source.py @@ -0,0 +1,508 @@ +"""Activities for loading, normalizing and reading document sources.""" + +import asyncio +import os +import shutil +import subprocess +import tempfile +from pathlib import Path + +from temporalio import activity +from temporalio.exceptions import ApplicationError + +from pipeline.documents.document_formats import ( + BUILTIN_BINARY_SUFFIXES, + TEXT_UPLOAD_TYPES, + installed_formats, +) +from pipeline.documents.document_text import clean_text +from pipeline.intake.source_fetch import MAX_HTML_BYTES, extract_text_source +from pipeline.object_storage import is_object_uri +from pipeline.ocr import ocr_pdf as run_ocr_pdf +from pipeline.ocr import ocr_pdf_in_segments as run_ocr_pdf_in_segments +from pipeline.temporal.activities.artifacts import ( + _upload_artifact, + _write_json_temp, + download_object, +) + +IMAGE_INPUT_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".tif", ".tiff"} + + +OFFICE_INPUT_EXTENSIONS = {".doc", ".docx", ".ppt", ".pptx", ".xls", ".xlsx"} + + +DELIMITED_INPUT_EXTENSIONS = {".csv"} + + +NATIVE_SPREADSHEET_EXTENSIONS = {".xlsx"} + + +def _source_type_from_path(filepath: str) -> tuple[str, str]: + ext = Path(filepath).suffix.lower() + if ext in TEXT_UPLOAD_TYPES: + return ("document", "structured_text") + if ext in {".csv", ".xlsx"}: + return ("spreadsheet", "spreadsheet") + if ext in BUILTIN_BINARY_SUFFIXES or any(item.suffix == ext for item in installed_formats()): + return ("document", "pdf") + return ("unknown", "pdf") + + +def _normalized_filename(original_name: str, canonical_input_type: str) -> str: + stem = Path(original_name).stem or "document" + if canonical_input_type == "spreadsheet": + ext = Path(original_name).suffix.lower() + return f"{stem}{ext if ext in {'.csv', '.xlsx'} else '.csv'}" + return f"{stem}.pdf" + + +def _resolve_local_path(filepath: str) -> tuple[str, bool]: + """ + Resolve path into local file path. + Returns (path, should_cleanup). + """ + if is_object_uri(filepath): + local_path = download_object(filepath) + activity.logger.info(f"Downloaded from MinIO to {local_path}") + return local_path, True + return filepath, False + + +def _convert_image_to_pdf(input_path: str, output_path: str) -> None: + from PIL import Image, ImageOps + + with Image.open(input_path) as img: + rgb_img = ImageOps.exif_transpose(img).convert("RGB") + rgb_img.save(output_path, "PDF", resolution=300.0) + + +def _convert_office_to_pdf(input_path: str, output_dir: str) -> str: + soffice_bin = shutil.which("soffice") + if not soffice_bin: + raise RuntimeError( + "LibreOffice (soffice) is not installed. Install libreoffice to convert office files to PDF." + ) + + cmd = [ + soffice_bin, + "--headless", + "--nologo", + "--nodefault", + "--nolockcheck", + "--nofirststartwizard", + "--convert-to", + "pdf", + "--outdir", + output_dir, + input_path, + ] + result = subprocess.run(cmd, capture_output=True, text=True) + if result.returncode != 0: + raise RuntimeError( + f"Failed to convert office document to PDF: {result.stderr.strip() or result.stdout.strip()}" + ) + + output_pdf = Path(output_dir) / f"{Path(input_path).stem}.pdf" + if not output_pdf.exists(): + raise RuntimeError("Office-to-PDF conversion finished but output PDF was not found") + return str(output_pdf) + + +def _ensure_pdf_input(local_path: str) -> tuple[str, bool]: + """ + Ensure the given file path points to a PDF. + Returns (pdf_path, should_cleanup). + """ + ext = Path(local_path).suffix.lower() + if ext == ".pdf": + return local_path, False + + extension = next((item for item in installed_formats() if item.suffix == ext), None) + if ext not in IMAGE_INPUT_EXTENSIONS | OFFICE_INPUT_EXTENSIONS and extension is None: + raise ValueError(f"Unsupported file extension '{ext}'") + + work_dir = tempfile.mkdtemp(prefix="doc_convert_") + output_pdf = Path(work_dir) / f"{Path(local_path).stem}.pdf" + + try: + if ext in IMAGE_INPUT_EXTENSIONS: + _convert_image_to_pdf(local_path, str(output_pdf)) + elif ext in OFFICE_INPUT_EXTENSIONS: + converted = _convert_office_to_pdf(local_path, work_dir) + if converted != str(output_pdf): + shutil.move(converted, output_pdf) + else: + extension.normalize_to_pdf(local_path, str(output_pdf)) + with output_pdf.open("rb") as result: + if result.read(5) != b"%PDF-": + raise ValueError("The document format extension did not produce a PDF") + except Exception: + shutil.rmtree(work_dir, ignore_errors=True) + raise + + activity.logger.info(f"Converted {local_path} -> {output_pdf}") + return str(output_pdf), True + + +def _csv_to_pages(input_path: str, rows_per_page: int = 80) -> list[dict]: + import csv + + def _open_reader(): + last_err = None + for enc in ("utf-8-sig", "utf-8", "latin-1"): + try: + f = open(input_path, "r", encoding=enc, newline="") + sample = f.read(4096) + f.seek(0) + try: + dialect = csv.Sniffer().sniff(sample, delimiters=",;\t|") + except Exception: + dialect = csv.excel + return f, csv.reader(f, dialect=dialect) + except Exception as e: + last_err = e + raise RuntimeError(f"Could not read CSV file: {last_err}") + + f, reader = _open_reader() + try: + rows = [[(c or "").strip() for c in r] for r in reader if any((c or "").strip() for c in r)] + finally: + f.close() + + if not rows: + return [{ + "page_number": 1, + "original_markdown": f"# {Path(input_path).name}\n\n(Empty CSV file)", + "edited_markdown": None, + "is_reviewed": False, + "reviewer_notes": None, + }] + + header = rows[0] + body = rows[1:] if len(rows) > 1 else [] + + pages: list[dict] = [] + page_num = 1 + for i in range(0, len(body), rows_per_page): + batch = body[i:i + rows_per_page] + lines = [f"# Spreadsheet Data: {Path(input_path).name}", "", f"Columns: {', '.join(header)}", ""] + for row_idx, row in enumerate(batch, start=i + 1): + pairs = [] + for col_i, val in enumerate(row): + col = header[col_i] if col_i < len(header) and header[col_i] else f"col_{col_i+1}" + pairs.append(f"{col}: {val}") + if pairs: + lines.append(f"- Row {row_idx}: " + " | ".join(pairs)) + pages.append({ + "page_number": page_num, + "original_markdown": "\n".join(lines), + "edited_markdown": None, + "is_reviewed": False, + "reviewer_notes": None, + }) + page_num += 1 + return pages + + +def _xlsx_to_pages(input_path: str, rows_per_page: int = 80) -> list[dict]: + from datetime import date, datetime + + from openpyxl import load_workbook + + def _cell_to_str(value) -> str: + if value is None: + return "" + if isinstance(value, (datetime, date)): + return value.isoformat() + return str(value).strip() + + pages: list[dict] = [] + page_num = 1 + wb = load_workbook(filename=input_path, data_only=True, read_only=True) + + for sheet in wb.worksheets: + header: list[str] | None = None + batch: list[tuple[int, list[str]]] = [] + seen_rows = 0 + + def emit_page(rows_batch: list[tuple[int, list[str]]]) -> None: + nonlocal page_num + if not rows_batch: + return + assert header is not None + lines = [ + f"# Spreadsheet Data: {Path(input_path).name}", + "", + f"Sheet: {sheet.title}", + f"Columns: {', '.join(header)}", + "", + ] + for excel_row_num, row in rows_batch: + pairs = [] + for col_i, val in enumerate(row): + col = header[col_i] if col_i < len(header) and header[col_i] else f"col_{col_i+1}" + pairs.append(f"{col}: {val}") + if pairs: + lines.append(f"- Row {excel_row_num}: " + " | ".join(pairs)) + pages.append({ + "page_number": page_num, + "original_markdown": "\n".join(lines), + "edited_markdown": None, + "is_reviewed": False, + "reviewer_notes": None, + }) + page_num += 1 + + for excel_row_num, row in enumerate(sheet.iter_rows(values_only=True), start=1): + values = [_cell_to_str(v) for v in row] + if not any(values): + continue + seen_rows += 1 + + if header is None: + header = values + continue + + batch.append((excel_row_num, values)) + if len(batch) >= rows_per_page: + emit_page(batch) + batch = [] + + if header is not None: + emit_page(batch) + elif seen_rows == 0: + pages.append({ + "page_number": page_num, + "original_markdown": f"# Spreadsheet Data: {Path(input_path).name}\n\nSheet: {sheet.title}\n\n(Empty sheet)", + "edited_markdown": None, + "is_reviewed": False, + "reviewer_notes": None, + }) + page_num += 1 + + wb.close() + + if not pages: + return [{ + "page_number": 1, + "original_markdown": f"# Spreadsheet Data: {Path(input_path).name}\n\n(Empty workbook)", + "edited_markdown": None, + "is_reviewed": False, + "reviewer_notes": None, + }] + return pages + + +def _ocr_pdf(local_pdf_path: str) -> list[dict]: + return run_ocr_pdf(local_pdf_path, clean_text) + + +def _ocr_pdf_in_segments( + local_pdf_path: str, + segment_pages: int | None, + on_segment_complete=None, + completed_page_numbers: set[int] | None = None, + provider: str | None = None, +) -> list[dict]: + return run_ocr_pdf_in_segments( + local_pdf_path, + segment_pages, + clean_text, + on_segment_complete=on_segment_complete, + completed_page_numbers=completed_page_numbers, + **({"provider": provider} if provider else {}), + ) + + +@activity.defn +async def run_ocr(filepath: str) -> list[dict]: + """Run OCR on a supported file and return page dicts.""" + activity.logger.info(f"Running OCR on {filepath}") + + local_path, cleanup_local = _resolve_local_path(filepath) + ext = Path(local_path).suffix.lower() + pdf_path = local_path + cleanup_pdf_dir = False + + try: + if ext in DELIMITED_INPUT_EXTENSIONS: + pages = _csv_to_pages(local_path) + activity.logger.info(f"CSV parsed into {len(pages)} pages") + return pages + if ext in NATIVE_SPREADSHEET_EXTENSIONS: + pages = _xlsx_to_pages(local_path) + activity.logger.info(f"XLSX parsed into {len(pages)} pages") + return pages + pdf_path, cleanup_pdf_dir = await asyncio.to_thread(_ensure_pdf_input, local_path) + return _ocr_pdf(pdf_path) + finally: + if cleanup_pdf_dir: + try: + shutil.rmtree(Path(pdf_path).parent, ignore_errors=True) + except Exception: + pass + if cleanup_local and os.path.exists(local_path): + os.remove(local_path) + + +@activity.defn +async def run_ocr_and_store(workflow_id: str, filepath: str, parser_provider: str | None = None) -> dict: + """Run OCR and persist pages to Postgres to avoid large Temporal payloads.""" + from pipeline import db + + if filepath.startswith("intake://"): + from pipeline.intake.source_intake import materialize_intake_pages + + return materialize_intake_pages(workflow_id, filepath) + + local_path, cleanup_local = _resolve_local_path(filepath) + ext = Path(local_path).suffix.lower() + source_type, canonical_input_type = _source_type_from_path(local_path) + original_filename = Path(local_path).name + normalized_path = local_path + cleanup_normalized = False + + try: + if ext in TEXT_UPLOAD_TYPES: + with open(local_path, "rb") as raw: + contents = raw.read(MAX_HTML_BYTES + 1) + if ext == ".json": + from pipeline.intake.source_intake import submitted_document_page + from pipeline.intake.webhook_mapping import parse_canonical_document + + if len(contents) > MAX_HTML_BYTES: + raise ApplicationError("JSON documents must be at most 2 MiB", non_retryable=True) + document = parse_canonical_document(contents) + pages = [submitted_document_page(document.model_dump())] + else: + source = extract_text_source(contents, TEXT_UPLOAD_TYPES[ext], original_filename) + pages = [{"page_number": 1, "original_markdown": source.text, "detected_language": source.language}] + elif ext in DELIMITED_INPUT_EXTENSIONS: + pages = _csv_to_pages(local_path) + elif ext in NATIVE_SPREADSHEET_EXTENSIONS: + pages = _xlsx_to_pages(local_path) + else: + normalized_path, cleanup_normalized = await asyncio.to_thread(_ensure_pdf_input, local_path) + saved_page_numbers = set(db.get_saved_page_numbers(workflow_id)) + loop = asyncio.get_running_loop() + + def persist_segment(segment_pages_result: list[dict], total_pages: int) -> None: + db.save_pages(workflow_id, segment_pages_result) + current_saved = len(saved_page_numbers.union({p["page_number"] for p in segment_pages_result})) + saved_page_numbers.update(p["page_number"] for p in segment_pages_result) + db.update_document_fields(workflow_id, page_count=current_saved) + # Heartbeat must run on the activity event loop (persist_segment runs in to_thread). + loop.call_soon_threadsafe( + activity.heartbeat, + {"workflow_id": workflow_id, "pages_saved": current_saved, "total_pages": total_pages}, + ) + activity.logger.info( + "Persisted OCR segment for %s: %s/%s pages saved", + workflow_id, + current_saved, + total_pages, + ) + + pages = await asyncio.to_thread( + _ocr_pdf_in_segments, + normalized_path, + segment_pages=None, + on_segment_complete=persist_segment, + completed_page_numbers=saved_page_numbers, + **({"provider": parser_provider} if parser_provider else {}), + ) + + pages = db.get_pages(workflow_id) + + db.save_pages(workflow_id, pages) + + latest_job = db.get_latest_document_job(workflow_id) + job_id = latest_job["id"] if latest_job else None + + artifact_type = { + "structured_text": "normalized_text", "spreadsheet": "normalized_spreadsheet", "pdf": "normalized_pdf", + }[canonical_input_type] + if ext in TEXT_UPLOAD_TYPES: + normalized_filename = f"{Path(original_filename).stem}.txt" + with tempfile.TemporaryDirectory(prefix="kp-normalized-text-") as directory: + text_path = Path(directory) / "document.txt" + text_path.write_text(pages[0]["original_markdown"], encoding="utf-8") + normalized_uri, normalized_size, normalized_mime = _upload_artifact( + str(text_path), workflow_id, artifact_type, normalized_filename, + ) + else: + normalized_filename = _normalized_filename(original_filename, canonical_input_type) + normalized_uri, normalized_size, normalized_mime = _upload_artifact( + normalized_path, workflow_id, artifact_type, normalized_filename, + ) + normalized_artifact_id = db.add_document_artifact( + workflow_id=workflow_id, + job_id=job_id, + artifact_type=artifact_type, + stage="ocr_processing", + storage_uri=normalized_uri, + mime_type=normalized_mime, + filename=normalized_filename, + size_bytes=normalized_size, + metadata={"source_filepath": filepath, "canonical_input_type": canonical_input_type}, + ) + + if is_object_uri(filepath): + original_uri = filepath + original_size = None + original_mime = "application/pdf" if ext == ".pdf" else "application/octet-stream" + else: + original_uri, original_size, original_mime = _upload_artifact( + local_path, + workflow_id, + "original_upload", + original_filename, + ) + original_artifact_id = db.add_document_artifact( + workflow_id=workflow_id, + job_id=job_id, + artifact_type="original_upload", + stage="registered", + storage_uri=original_uri, + mime_type=original_mime, + filename=original_filename, + size_bytes=original_size, + metadata={"source_filepath": filepath}, + ) + + pages_json_path = _write_json_temp(pages) + try: + pages_uri, pages_size, pages_mime = _upload_artifact( + pages_json_path, workflow_id, "ocr_pages_json", "pages.json" + ) + finally: + if os.path.exists(pages_json_path): + os.remove(pages_json_path) + + db.add_document_artifact( + workflow_id=workflow_id, + job_id=job_id, + artifact_type="ocr_pages_json", + stage="ocr_review", + storage_uri=pages_uri, + mime_type=pages_mime, + filename="pages.json", + size_bytes=pages_size, + metadata={"page_count": len(pages)}, + ) + + db.update_document_fields( + workflow_id, + page_count=len(pages), + source_type=source_type, + canonical_input_type=canonical_input_type, + original_artifact_id=original_artifact_id, + normalized_artifact_id=normalized_artifact_id, + ) + return {"page_count": len(pages), "normalized_artifact_id": normalized_artifact_id} + finally: + if cleanup_normalized and os.path.exists(normalized_path): + shutil.rmtree(Path(normalized_path).parent, ignore_errors=True) + if cleanup_local and os.path.exists(local_path): + os.remove(local_path) diff --git a/pipeline/temporal/activities/state.py b/pipeline/temporal/activities/state.py new file mode 100644 index 0000000..eed5313 --- /dev/null +++ b/pipeline/temporal/activities/state.py @@ -0,0 +1,67 @@ +"""Activities for persisting document and workflow state.""" + +from datetime import datetime + +from temporalio import activity + + +@activity.defn +async def update_document_state( + workflow_id: str, + stage: str, + page_count: int = 0, + chunk_count: int = 0, + error_message: str = None, +) -> dict: + """Update document state in Postgres.""" + from pipeline import db + + activity.logger.info(f"Updating state for {workflow_id}: stage={stage}") + db.update_document_stage( + workflow_id=workflow_id, + stage=stage, + page_count=page_count, + chunk_count=chunk_count, + error_message=error_message, + ) + latest_job = db.get_latest_document_job(workflow_id) + if latest_job: + job_updates = {"current_stage": stage} + if stage in { + "ocr_review", + "translation_review", + "chunk_review", + "ready_for_ingestion", + "approval_for_prod", + }: + job_updates["status"] = "waiting_review" + elif stage == "completed": + job_updates["status"] = "completed" + job_updates["completed_at"] = datetime.utcnow().isoformat() + elif stage == "failed": + job_updates["status"] = "failed" + job_updates["completed_at"] = datetime.utcnow().isoformat() + job_updates["error_message"] = error_message + else: + job_updates["status"] = "running" + db.update_document_job(latest_job["id"], **job_updates) + return {"updated": True, "stage": stage} + + +@activity.defn +async def persist_document_content(workflow_id: str, pages: list[dict], chunks: list[dict]) -> dict: + """Persist pages and chunks to Postgres.""" + from pipeline import db + + activity.logger.info(f"Persisting content for {workflow_id}: {len(pages)} pages, {len(chunks)} chunks") + db.save_pages(workflow_id, pages) + db.save_chunks(workflow_id, chunks) + return {"persisted": True, "pages": len(pages), "chunks": len(chunks)} + + +@activity.defn +async def auto_tag_chunks_from_db(workflow_id: str, filename: str = "") -> dict: + """No-op retained so existing workflow histories that scheduled this + activity before the auto-tag-v1 patch continue to replay deterministically. + """ + return {"tagged_chunks": 0, "skipped": True} diff --git a/pipeline/temporal/worker.py b/pipeline/temporal/worker.py new file mode 100644 index 0000000..d204db0 --- /dev/null +++ b/pipeline/temporal/worker.py @@ -0,0 +1,174 @@ +""" +Temporal worker for the OCR pipeline. +""" + +import asyncio +import logging +import os + +from temporalio.client import Client +from temporalio.worker import Worker + +from pipeline.translation.service import ( + PROVIDERS as TRANSLATION_PROVIDERS, + configured_translation_endpoint, + configured_translation_type, +) + +from pipeline import db +from pipeline.temporal.activities.state import ( + auto_tag_chunks_from_db, + persist_document_content, + update_document_state, +) +from pipeline.temporal.activities.processing import ( + create_chunks, + create_chunks_from_db, + detect_and_translate_pages, + detect_and_translate_pages_from_db, +) +from pipeline.temporal.activities.indexing import ( + ingest_document_from_db, + ingest_to_vector_db, + prepare_for_ingestion, + promote_document_to_prod_qdrant, +) +from pipeline.temporal.activities.publication import ( + publish_catalog_to_network, + release_saved_revision, +) +from pipeline.temporal.activities.source import run_ocr, run_ocr_and_store +from pipeline.temporal.activities.logging import ActivityLogging +from pipeline.documents.document_formats import installed_formats +from pipeline.network.network_publisher import validate_network_adapters +from pipeline.telemetry.logs import configure_logs, reset_logs +from pipeline.intake.source_intake import reconcile_pending_sources +from pipeline.intake.scheduled_intake import poll_due_sources +from pipeline.temporal.workflows import ( + ChunkingOnlyWorkflow, + DocumentPipelineWorkflow, + OcrOnlyWorkflow, + PromoteToProdWorkflow, + ReingestionWorkflow, + SelectNetworkRevisionWorkflow, + TranslationOnlyWorkflow, +) + +LOG_LEVEL = getattr(logging, os.environ.get("LOG_LEVEL", "INFO").strip().upper(), logging.INFO) + +# Configure verbose logging +logging.basicConfig( + level=LOG_LEVEL, + format='%(asctime)s [%(levelname)s] %(name)s: %(message)s', + datefmt='%H:%M:%S' +) + +# Set Temporal SDK logging to INFO regardless of LOG_LEVEL - its own DEBUG +# output is noisy and rarely what LOG_LEVEL=DEBUG is set to see. +logging.getLogger("temporalio").setLevel(logging.INFO) + +TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "ocr-pipeline") +INTERCEPTORS = [ActivityLogging()] + + +async def _recover_source_starts(client: Client) -> None: + while True: + try: + started = await reconcile_pending_sources(client, TASK_QUEUE) + if started: + logging.info("Recovered %d source intake starts", started) + except Exception as error: + logging.warning("Source intake start recovery failed error_type=%s", type(error).__name__) + await asyncio.sleep(12) + + +async def _poll_scheduled_sources() -> None: + while True: + try: + await poll_due_sources() + except Exception as error: + logging.warning("Scheduled pull polling failed error_type=%s", type(error).__name__) + await asyncio.sleep(12) + + +async def main(): + """Start the worker.""" + configure_logs() + try: + await _run() + except Exception: + logging.exception("worker crashed") + raise + finally: + logging.info("worker stopping") + reset_logs() + + +async def _run(): + validate_network_adapters() + installed_formats() + + if configured_translation_type() in TRANSLATION_PROVIDERS and not configured_translation_endpoint(): + raise RuntimeError("TRANSLATION_BASE_URL is required for the configured translation provider") + + logging.info("worker startup: initializing PostgreSQL database") + db.init_db() + + temporal_host = os.environ.get("TEMPORAL_HOST", "localhost:7233") + max_concurrent_activities = int(os.environ.get("TEMPORAL_MAX_CONCURRENT_ACTIVITIES", "4")) + logging.info("worker startup: connecting to Temporal at %s", temporal_host) + + client = await Client.connect(temporal_host) + + worker = Worker( + client, + task_queue=TASK_QUEUE, + max_concurrent_activities=max_concurrent_activities, + interceptors=INTERCEPTORS, + workflows=[ + DocumentPipelineWorkflow, + ReingestionWorkflow, + SelectNetworkRevisionWorkflow, + PromoteToProdWorkflow, + TranslationOnlyWorkflow, + OcrOnlyWorkflow, + ChunkingOnlyWorkflow, + ], + activities=[ + run_ocr, + run_ocr_and_store, + create_chunks, + create_chunks_from_db, + auto_tag_chunks_from_db, + prepare_for_ingestion, + ingest_to_vector_db, + ingest_document_from_db, + promote_document_to_prod_qdrant, + publish_catalog_to_network, + release_saved_revision, + update_document_state, + detect_and_translate_pages, + detect_and_translate_pages_from_db, + persist_document_content, + ], + ) + + + logging.info( + "worker started queue=%s temporal=%s activity_concurrency=%s", + TASK_QUEUE, + temporal_host, + max_concurrent_activities, + ) + recovery = asyncio.create_task(_recover_source_starts(client)) + pulls = asyncio.create_task(_poll_scheduled_sources()) + try: + await worker.run() + finally: + recovery.cancel() + pulls.cancel() + await asyncio.gather(recovery, pulls, return_exceptions=True) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/pipeline/temporal/workflows/__init__.py b/pipeline/temporal/workflows/__init__.py new file mode 100644 index 0000000..82be553 --- /dev/null +++ b/pipeline/temporal/workflows/__init__.py @@ -0,0 +1,21 @@ +"""Registered Temporal workflow definitions.""" + +from .document import DocumentPipelineWorkflow as DocumentPipelineWorkflow +from .processing import ( + ChunkingOnlyWorkflow as ChunkingOnlyWorkflow, +) +from .processing import ( + OcrOnlyWorkflow as OcrOnlyWorkflow, +) +from .processing import ( + TranslationOnlyWorkflow as TranslationOnlyWorkflow, +) +from .releases import ( + PromoteToProdWorkflow as PromoteToProdWorkflow, +) +from .releases import ( + ReingestionWorkflow as ReingestionWorkflow, +) +from .releases import ( + SelectNetworkRevisionWorkflow as SelectNetworkRevisionWorkflow, +) diff --git a/pipeline/temporal/workflows/document.py b/pipeline/temporal/workflows/document.py new file mode 100644 index 0000000..6ceb0c3 --- /dev/null +++ b/pipeline/temporal/workflows/document.py @@ -0,0 +1,421 @@ +"""Orchestrate intake and processing of a document.""" + +from dataclasses import dataclass +from datetime import timedelta +from typing import Optional + +from temporalio import workflow + +with workflow.unsafe.imports_passed_through(): + from pipeline.documents.models import DocumentStage + from pipeline.temporal.activities.indexing import ( + ingest_document_from_db, + promote_document_to_prod_qdrant, + ) + from pipeline.temporal.activities.processing import ( + create_chunks_from_db, + detect_and_translate_pages_from_db, + ) + from pipeline.temporal.activities.publication import publish_catalog_to_network + from pipeline.temporal.activities.source import run_ocr_and_store + from pipeline.temporal.activities.state import auto_tag_chunks_from_db + +from .policies import ( + CHUNK_RETRY, + INGEST_RETRY, + NETWORK_PUBLISH_RETRY, + OCR_RETRY, + TRANSLATION_RETRY, + _mirror_state, + _now_iso, +) + + +@dataclass +class DocumentWorkflowState: + """Workflow state - queryable and modifiable via signals.""" + + document_id: str + filename: str + filepath: str + + stage: DocumentStage = DocumentStage.REGISTERED + error_message: Optional[str] = None + + page_count: int = 0 + chunk_count: int = 0 + translated_count: int = 0 + + ocr_approved: bool = False + translation_approved: bool = False + chunks_approved: bool = False + ingestion_approved: bool = False + prod_approved: bool = False + network_published: bool = False + + chunk_size: int = 450 + chunk_overlap: int = 128 + min_tokens: int = 100 + index_name: str = "documents-index" + stop_after_ocr: bool = False + + created_at: str = "" + ocr_completed_at: Optional[str] = None + translation_completed_at: Optional[str] = None + chunks_completed_at: Optional[str] = None + ingested_at: Optional[str] = None + promoted_to_prod_at: Optional[str] = None + + +@workflow.defn +class DocumentPipelineWorkflow: + """Main document processing workflow.""" + + def __init__(self): + self.state = None + + @workflow.run + async def run( + self, + document_id: str, + filename: str, + filepath: str, + chunk_size: int = 450, + chunk_overlap: int = 128, + min_tokens: int = 100, + index_name: str = "documents-index", + auto_approve: bool = False, + stop_after_ocr: bool = False, + skip_prod: bool = False, + release_network_id: str | None = None, + parser_provider: str | None = None, + index_only: bool = False, + ) -> dict: + self.state = DocumentWorkflowState( + document_id=document_id, + filename=filename, + filepath=filepath, + chunk_size=chunk_size, + chunk_overlap=chunk_overlap, + min_tokens=min_tokens, + index_name=index_name, + stop_after_ocr=stop_after_ocr, + created_at=_now_iso(), + ) + + try: + self.state.stage = DocumentStage.OCR_PROCESSING + workflow.logger.info(f"Starting OCR for {filename}") + + await _mirror_state(workflow.info().workflow_id, "ocr_processing", 0, 0, None) + + ocr_args = [workflow.info().workflow_id, filepath] + if parser_provider: + ocr_args.append(parser_provider) + ocr_result = await workflow.execute_activity( + run_ocr_and_store, + args=ocr_args, + start_to_close_timeout=timedelta(minutes=90), + heartbeat_timeout=timedelta(minutes=10), + retry_policy=OCR_RETRY, + ) + self.state.page_count = ocr_result.get("page_count", 0) + self.state.ocr_completed_at = _now_iso() + self.state.stage = DocumentStage.OCR_REVIEW + + await _mirror_state(workflow.info().workflow_id, "ocr_review", self.state.page_count, 0, None) + + if stop_after_ocr: + workflow.logger.info(f"OCR-only run complete for {filename}") + return { + "document_id": document_id, + "filename": filename, + "stage": self.state.stage.value, + "pages": self.state.page_count, + "chunks": 0, + "stop_after_ocr": True, + } + + if not auto_approve: + await workflow.wait_condition(lambda: self.state.ocr_approved) + + self.state.stage = DocumentStage.TRANSLATION_PROCESSING + await _mirror_state(workflow.info().workflow_id, "translation_processing", self.state.page_count, 0, None) + + translation_result = await workflow.execute_activity( + detect_and_translate_pages_from_db, + args=[workflow.info().workflow_id], + start_to_close_timeout=timedelta(minutes=90), + retry_policy=TRANSLATION_RETRY, + ) + self.state.page_count = translation_result.get("page_count", self.state.page_count) + self.state.translated_count = translation_result.get("translated_count", 0) + self.state.translation_completed_at = _now_iso() + self.state.stage = DocumentStage.TRANSLATION_REVIEW + + await _mirror_state(workflow.info().workflow_id, "translation_review", self.state.page_count, 0, None) + + if not auto_approve: + await workflow.wait_condition(lambda: self.state.translation_approved) + + self.state.stage = DocumentStage.CHUNKING + await _mirror_state(workflow.info().workflow_id, "chunking", self.state.page_count, 0, None) + + source_first = index_only or bool(release_network_id and workflow.patched("source-first-release-v1")) + if not index_only and not release_network_id and (filepath.startswith("intake://") or filename.lower().endswith(".json")): + source_first = workflow.patched("source-first-structured-draft-v1") + chunk_args = [workflow.info().workflow_id, chunk_size, chunk_overlap, min_tokens] + if source_first: + chunk_args.append(True) + chunk_result = await workflow.execute_activity( + create_chunks_from_db, + args=chunk_args, + start_to_close_timeout=timedelta(minutes=30), + retry_policy=CHUNK_RETRY, + ) + self.state.chunk_count = chunk_result.get("chunk_count", 0) + self.state.chunks_completed_at = _now_iso() + + if workflow.patched("auto-tag-v1"): + await workflow.execute_activity( + auto_tag_chunks_from_db, + args=[workflow.info().workflow_id, filename], + start_to_close_timeout=timedelta(minutes=45), + retry_policy=CHUNK_RETRY, + ) + + self.state.stage = DocumentStage.CHUNK_REVIEW + + await _mirror_state(workflow.info().workflow_id, "chunk_review", self.state.page_count, self.state.chunk_count, None) + + if not auto_approve: + await workflow.wait_condition(lambda: self.state.chunks_approved) + + self.state.stage = DocumentStage.READY_FOR_INGESTION + await _mirror_state(workflow.info().workflow_id, "ready_for_ingestion", self.state.page_count, self.state.chunk_count, None) + + if not auto_approve: + await workflow.wait_condition(lambda: self.state.ingestion_approved) + + self.state.stage = DocumentStage.INGESTING + await _mirror_state(workflow.info().workflow_id, "ingesting", self.state.page_count, self.state.chunk_count, None) + + ingest_args = [workflow.info().workflow_id, document_id, filename, index_name] + if release_network_id: + ingest_args.extend([10, release_network_id]) + if source_first: + ingest_args.append(True) + elif workflow.patched("freeze-draft-revision-v1"): + ingest_args.extend([10, None, source_first, True]) + result = await workflow.execute_activity( + ingest_document_from_db, + args=ingest_args, + start_to_close_timeout=timedelta(minutes=90), + retry_policy=INGEST_RETRY, + ) + + self.state.ingested_at = _now_iso() + + if release_network_id: + self.state.stage = DocumentStage.PUBLISHING_TO_NETWORK + await _mirror_state( + workflow.info().workflow_id, "publishing_to_network", self.state.page_count, + self.state.chunk_count, None, + ) + await workflow.execute_activity( + publish_catalog_to_network, + args=[workflow.info().workflow_id, str(workflow.uuid4()), release_network_id], + start_to_close_timeout=timedelta(minutes=10), + retry_policy=NETWORK_PUBLISH_RETRY, + ) + self.state.network_published = True + self.state.stage = DocumentStage.COMPLETED + await _mirror_state( + workflow.info().workflow_id, "completed", self.state.page_count, + self.state.chunk_count, None, + ) + return { + "document_id": document_id, + "filename": filename, + "stage": self.state.stage.value, + "pages": self.state.page_count, + "chunks": self.state.chunk_count, + "records_ingested": result.get("records_ingested", 0), + "release_network_id": release_network_id, + } + + if not workflow.patched("publish-after-prod-v1"): + # Old order, preserved for in-flight workflows whose history + # already recorded this activity immediately after DEV ingest. + # New workflows publish only after a successful PROD + # promotion instead (see below). + self.state.stage = DocumentStage.PUBLISHING_TO_NETWORK + await _mirror_state( + workflow.info().workflow_id, + "publishing_to_network", + self.state.page_count, + self.state.chunk_count, + None, + ) + + # transactionId is generated once here (workflow.uuid4() is + # replay-safe) and reused across activity retries; the activity + # generates a fresh messageId on every attempt. + network_transaction_id = str(workflow.uuid4()) + await workflow.execute_activity( + publish_catalog_to_network, + args=[workflow.info().workflow_id, network_transaction_id], + start_to_close_timeout=timedelta(minutes=10), + retry_policy=NETWORK_PUBLISH_RETRY, + ) + self.state.network_published = True + + # DISABLE_PROD_SETTING: finish at DEV ingest. Decided when the + # workflow was started and passed in as an argument, so a replay + # always takes the same branch as the original run. + if skip_prod: + self.state.stage = DocumentStage.COMPLETED + await _mirror_state( + workflow.info().workflow_id, + "completed", + self.state.page_count, + self.state.chunk_count, + None, + ) + workflow.logger.info( + f"Pipeline complete for {filename} (PROD stages disabled)" + ) + return { + "document_id": document_id, + "filename": filename, + "stage": self.state.stage.value, + "pages": self.state.page_count, + "chunks": self.state.chunk_count, + "records_ingested": result.get("records_ingested", 0), + "records_promoted_to_prod": 0, + "prod_skipped": True, + } + + self.state.stage = DocumentStage.APPROVAL_FOR_PROD + await _mirror_state( + workflow.info().workflow_id, + "approval_for_prod", + self.state.page_count, + self.state.chunk_count, + None, + ) + + # Prod promotion is always an explicit superadmin action (even with auto_approve). + await workflow.wait_condition(lambda: self.state.prod_approved) + + self.state.stage = DocumentStage.INGESTING_PROD + await _mirror_state( + workflow.info().workflow_id, + "ingesting_prod", + self.state.page_count, + self.state.chunk_count, + None, + ) + + prod_result = await workflow.execute_activity( + promote_document_to_prod_qdrant, + args=[workflow.info().workflow_id, document_id, filename], + start_to_close_timeout=timedelta(minutes=90), + retry_policy=INGEST_RETRY, + ) + + self.state.promoted_to_prod_at = _now_iso() + + if not self.state.network_published: + # New order: publish to the network only now that PROD + # promotion has actually succeeded. Skipped for workflows + # that already published at the old position above. + self.state.stage = DocumentStage.PUBLISHING_TO_NETWORK + await _mirror_state( + workflow.info().workflow_id, + "publishing_to_network", + self.state.page_count, + self.state.chunk_count, + None, + ) + + network_transaction_id = str(workflow.uuid4()) + await workflow.execute_activity( + publish_catalog_to_network, + args=[workflow.info().workflow_id, network_transaction_id], + start_to_close_timeout=timedelta(minutes=10), + retry_policy=NETWORK_PUBLISH_RETRY, + ) + self.state.network_published = True + + self.state.stage = DocumentStage.COMPLETED + + await _mirror_state(workflow.info().workflow_id, "completed", self.state.page_count, self.state.chunk_count, None) + + workflow.logger.info(f"Pipeline complete for {filename}") + return { + "document_id": document_id, + "filename": filename, + "stage": self.state.stage.value, + "pages": self.state.page_count, + "chunks": self.state.chunk_count, + "records_ingested": result.get("records_ingested", 0), + "records_promoted_to_prod": prod_result.get("records_ingested", 0), + } + + except Exception as e: + self.state.stage = DocumentStage.FAILED + self.state.error_message = str(e) + workflow.logger.error(f"Pipeline failed: {e}") + + try: + await _mirror_state(workflow.info().workflow_id, "failed", self.state.page_count, self.state.chunk_count, str(e)) + except Exception: + pass + raise + + @workflow.query + def get_state(self) -> dict: + if not self.state: + return {} + return { + "document_id": self.state.document_id, + "filename": self.state.filename, + "filepath": self.state.filepath, + "stage": self.state.stage.value, + "error_message": self.state.error_message, + "page_count": self.state.page_count, + "chunk_count": self.state.chunk_count, + "translated_count": self.state.translated_count, + "ocr_approved": self.state.ocr_approved, + "translation_approved": self.state.translation_approved, + "chunks_approved": self.state.chunks_approved, + "ingestion_approved": self.state.ingestion_approved, + "prod_approved": self.state.prod_approved, + "created_at": self.state.created_at, + "ocr_completed_at": self.state.ocr_completed_at, + "translation_completed_at": self.state.translation_completed_at, + "chunks_completed_at": self.state.chunks_completed_at, + "ingested_at": self.state.ingested_at, + "promoted_to_prod_at": self.state.promoted_to_prod_at, + } + + @workflow.signal + def approve_ocr(self): + self.state.ocr_approved = True + + @workflow.signal + def approve_translation(self): + self.state.translation_approved = True + + @workflow.signal + def approve_chunks(self): + self.state.chunks_approved = True + + @workflow.signal + def approve_ingestion(self): + self.state.ingestion_approved = True + + @workflow.signal + def approve_prod(self): + self.state.prod_approved = True diff --git a/pipeline/temporal/workflows/policies.py b/pipeline/temporal/workflows/policies.py new file mode 100644 index 0000000..1b99833 --- /dev/null +++ b/pipeline/temporal/workflows/policies.py @@ -0,0 +1,101 @@ +"""Retry policies, workflow clock and durable state mirroring.""" + +import os +from datetime import timedelta +from typing import Optional + +from temporalio import workflow +from temporalio.common import RetryPolicy + +with workflow.unsafe.imports_passed_through(): + from pipeline.temporal.activities.state import update_document_state + + +def _now_iso() -> str: + """Get current time as ISO string (workflow-safe).""" + return workflow.now().isoformat() + + +OCR_RETRY = RetryPolicy( + initial_interval=timedelta(seconds=15), + backoff_coefficient=2.0, + maximum_interval=timedelta(minutes=15), + maximum_attempts=20, +) + +CHUNK_RETRY = RetryPolicy( + initial_interval=timedelta(seconds=5), + backoff_coefficient=1.5, + maximum_interval=timedelta(minutes=2), + maximum_attempts=10, +) + +INGEST_RETRY = RetryPolicy( + initial_interval=timedelta(seconds=10), + backoff_coefficient=2.0, + maximum_interval=timedelta(minutes=10), + maximum_attempts=15, +) + +STATE_UPDATE_RETRY = RetryPolicy( + initial_interval=timedelta(seconds=3), + backoff_coefficient=1.5, + maximum_interval=timedelta(minutes=1), + maximum_attempts=10, +) + +TRANSLATION_RETRY = RetryPolicy( + initial_interval=timedelta(seconds=15), + backoff_coefficient=2.0, + maximum_interval=timedelta(minutes=15), + maximum_attempts=20, +) + +# Shorter than INGEST_RETRY: the Discovery Service is a new, unproven external +# dependency, so we fail the document faster rather than holding it in-flight +# for hours. Configurable (unlike the other policies above) since this +# dependency's real-world retry behavior is still unproven. +def _network_publish_retry_policy() -> RetryPolicy: + return RetryPolicy( + initial_interval=timedelta( + seconds=int(os.environ.get("DISCOVERY_SERVICE_PUBLISH_INITIAL_INTERVAL_SECONDS", "30")) + ), + backoff_coefficient=float( + os.environ.get("DISCOVERY_SERVICE_PUBLISH_BACKOFF_COEFFICIENT", "2.0") + ), + maximum_interval=timedelta( + seconds=int(os.environ.get("DISCOVERY_SERVICE_PUBLISH_MAX_INTERVAL_SECONDS", "300")) + ), + maximum_attempts=int(os.environ.get("DISCOVERY_SERVICE_PUBLISH_MAX_ATTEMPTS", "5")), + ) + + +NETWORK_PUBLISH_RETRY = _network_publish_retry_policy() + +async def _mirror_state( + workflow_id: str, + stage: str, + page_count: int = 0, + chunk_count: int = 0, + error_message: Optional[str] = None, +) -> dict: + """ + Mirror workflow state into Postgres as a local activity. + + Local activities avoid starving tiny state updates behind long-running OCR, + translation, or chunking activities on the shared task queue. + """ + if workflow.patched("local-state-update-v1"): + return await workflow.execute_local_activity( + update_document_state, + args=[workflow_id, stage, page_count, chunk_count, error_message], + start_to_close_timeout=timedelta(seconds=30), + retry_policy=STATE_UPDATE_RETRY, + ) + + return await workflow.execute_activity( + update_document_state, + args=[workflow_id, stage, page_count, chunk_count, error_message], + start_to_close_timeout=timedelta(seconds=30), + retry_policy=STATE_UPDATE_RETRY, + ) diff --git a/pipeline/temporal/workflows/processing.py b/pipeline/temporal/workflows/processing.py new file mode 100644 index 0000000..89655fe --- /dev/null +++ b/pipeline/temporal/workflows/processing.py @@ -0,0 +1,280 @@ +"""Run one processing stage independently.""" + +from dataclasses import dataclass +from datetime import timedelta +from typing import Optional + +from temporalio import workflow + +with workflow.unsafe.imports_passed_through(): + from pipeline.documents.models import DocumentStage + from pipeline.temporal.activities.processing import ( + create_chunks_from_db, + detect_and_translate_pages_from_db, + ) + from pipeline.temporal.activities.source import run_ocr_and_store + from pipeline.temporal.activities.state import auto_tag_chunks_from_db + +from .policies import ( + CHUNK_RETRY, + OCR_RETRY, + TRANSLATION_RETRY, + _mirror_state, + _now_iso, +) + + +@dataclass +class TranslationOnlyWorkflowState: + workflow_id: str + document_id: str + filename: str + stage: DocumentStage = DocumentStage.OCR_REVIEW + error_message: Optional[str] = None + page_count: int = 0 + translated_count: int = 0 + translation_completed_at: Optional[str] = None + + +@workflow.defn +class TranslationOnlyWorkflow: + """Resume from OCR review, run translation only, and stop at translation review.""" + + def __init__(self): + self.state = None + + @workflow.run + async def run( + self, + original_workflow_id: str, + document_id: str, + filename: str, + ) -> dict: + self.state = TranslationOnlyWorkflowState( + workflow_id=original_workflow_id, + document_id=document_id, + filename=filename, + ) + + try: + await _mirror_state(original_workflow_id, "translation_processing", 0, 0, None) + self.state.stage = DocumentStage.TRANSLATION_PROCESSING + + translation_result = await workflow.execute_activity( + detect_and_translate_pages_from_db, + args=[original_workflow_id], + start_to_close_timeout=timedelta(minutes=90), + retry_policy=TRANSLATION_RETRY, + ) + self.state.page_count = translation_result.get("page_count", 0) + self.state.translated_count = translation_result.get("translated_count", 0) + self.state.translation_completed_at = _now_iso() + self.state.stage = DocumentStage.TRANSLATION_REVIEW + + await _mirror_state(original_workflow_id, "translation_review", self.state.page_count, 0, None) + + return { + "workflow_id": original_workflow_id, + "document_id": document_id, + "filename": filename, + "stage": "translation_review", + "page_count": self.state.page_count, + "translated_count": self.state.translated_count, + } + except Exception as e: + self.state.stage = DocumentStage.FAILED + self.state.error_message = str(e) + try: + await _mirror_state(original_workflow_id, "failed", self.state.page_count, 0, str(e)) + except Exception: + pass + raise + + @workflow.query + def get_state(self) -> dict: + if not self.state: + return {} + return { + "workflow_id": self.state.workflow_id, + "document_id": self.state.document_id, + "filename": self.state.filename, + "stage": self.state.stage.value, + "error_message": self.state.error_message, + "page_count": self.state.page_count, + "translated_count": self.state.translated_count, + "translation_completed_at": self.state.translation_completed_at, + } + + +@dataclass +class OcrOnlyWorkflowState: + workflow_id: str + document_id: str + filename: str + stage: DocumentStage = DocumentStage.REGISTERED + error_message: Optional[str] = None + page_count: int = 0 + ocr_completed_at: Optional[str] = None + + +@workflow.defn +class OcrOnlyWorkflow: + """Resume or retry OCR for an existing document and stop at OCR review.""" + + def __init__(self): + self.state = None + + @workflow.run + async def run( + self, + original_workflow_id: str, + document_id: str, + filename: str, + filepath: str, + ) -> dict: + self.state = OcrOnlyWorkflowState( + workflow_id=original_workflow_id, + document_id=document_id, + filename=filename, + ) + + try: + await _mirror_state(original_workflow_id, "ocr_processing", 0, 0, None) + self.state.stage = DocumentStage.OCR_PROCESSING + + ocr_result = await workflow.execute_activity( + run_ocr_and_store, + args=[original_workflow_id, filepath], + start_to_close_timeout=timedelta(minutes=90), + heartbeat_timeout=timedelta(minutes=10), + retry_policy=OCR_RETRY, + ) + self.state.page_count = ocr_result.get("page_count", 0) + self.state.ocr_completed_at = _now_iso() + self.state.stage = DocumentStage.OCR_REVIEW + + await _mirror_state(original_workflow_id, "ocr_review", self.state.page_count, 0, None) + + return { + "workflow_id": original_workflow_id, + "document_id": document_id, + "filename": filename, + "stage": "ocr_review", + "page_count": self.state.page_count, + } + except Exception as e: + self.state.stage = DocumentStage.FAILED + self.state.error_message = str(e) + try: + await _mirror_state(original_workflow_id, "failed", self.state.page_count, 0, str(e)) + except Exception: + pass + raise + + @workflow.query + def get_state(self) -> dict: + if not self.state: + return {} + return { + "workflow_id": self.state.workflow_id, + "document_id": self.state.document_id, + "filename": self.state.filename, + "stage": self.state.stage.value, + "error_message": self.state.error_message, + "page_count": self.state.page_count, + "ocr_completed_at": self.state.ocr_completed_at, + } + + +@dataclass +class ChunkingOnlyWorkflowState: + workflow_id: str + document_id: str + filename: str + stage: DocumentStage = DocumentStage.TRANSLATION_REVIEW + error_message: Optional[str] = None + page_count: int = 0 + chunk_count: int = 0 + chunks_completed_at: Optional[str] = None + + +@workflow.defn +class ChunkingOnlyWorkflow: + """Resume or retry chunking for an existing document and stop at chunk review.""" + + def __init__(self): + self.state = None + + @workflow.run + async def run( + self, + original_workflow_id: str, + document_id: str, + filename: str, + page_count: int = 0, + chunk_size: int = 450, + chunk_overlap: int = 128, + min_tokens: int = 100, + ) -> dict: + self.state = ChunkingOnlyWorkflowState( + workflow_id=original_workflow_id, + document_id=document_id, + filename=filename, + page_count=page_count, + ) + + try: + await _mirror_state(original_workflow_id, "chunking", page_count, 0, None) + self.state.stage = DocumentStage.CHUNKING + + chunk_result = await workflow.execute_activity( + create_chunks_from_db, + args=[original_workflow_id, chunk_size, chunk_overlap, min_tokens], + start_to_close_timeout=timedelta(minutes=30), + retry_policy=CHUNK_RETRY, + ) + self.state.chunk_count = chunk_result.get("chunk_count", 0) + self.state.chunks_completed_at = _now_iso() + + if workflow.patched("auto-tag-v1"): + await workflow.execute_activity( + auto_tag_chunks_from_db, + args=[original_workflow_id, filename], + start_to_close_timeout=timedelta(minutes=45), + retry_policy=CHUNK_RETRY, + ) + + self.state.stage = DocumentStage.CHUNK_REVIEW + + await _mirror_state(original_workflow_id, "chunk_review", page_count, self.state.chunk_count, None) + + return { + "workflow_id": original_workflow_id, + "document_id": document_id, + "filename": filename, + "stage": "chunk_review", + "chunk_count": self.state.chunk_count, + } + except Exception as e: + self.state.stage = DocumentStage.FAILED + self.state.error_message = str(e) + try: + await _mirror_state(original_workflow_id, "failed", page_count, self.state.chunk_count, str(e)) + except Exception: + pass + raise + + @workflow.query + def get_state(self) -> dict: + if not self.state: + return {} + return { + "workflow_id": self.state.workflow_id, + "document_id": self.state.document_id, + "filename": self.state.filename, + "stage": self.state.stage.value, + "error_message": self.state.error_message, + "page_count": self.state.page_count, + "chunk_count": self.state.chunk_count, + "chunks_completed_at": self.state.chunks_completed_at, + } diff --git a/pipeline/temporal/workflows/releases.py b/pipeline/temporal/workflows/releases.py new file mode 100644 index 0000000..98d9a3c --- /dev/null +++ b/pipeline/temporal/workflows/releases.py @@ -0,0 +1,214 @@ +"""Reindex a document or publish a selected revision.""" + +from dataclasses import dataclass +from datetime import timedelta +from typing import Optional + +from temporalio import workflow + +with workflow.unsafe.imports_passed_through(): + from pipeline.documents.models import DocumentStage + from pipeline.temporal.activities.indexing import ( + ingest_document_from_db, + promote_document_to_prod_qdrant, + ) + from pipeline.temporal.activities.publication import ( + publish_catalog_to_network, + release_saved_revision, + ) + +from .policies import ( + INGEST_RETRY, + NETWORK_PUBLISH_RETRY, + _mirror_state, +) + + +@workflow.defn +class SelectNetworkRevisionWorkflow: + @workflow.run + async def run( + self, document_id: str, revision_id: str, network_id: str, expected_generation: str | None, + ) -> dict: + return await workflow.execute_activity( + release_saved_revision, + args=[document_id, revision_id, network_id, str(workflow.uuid4()), expected_generation], + start_to_close_timeout=timedelta(minutes=30), + retry_policy=NETWORK_PUBLISH_RETRY, + ) + +@dataclass +class ReingestionWorkflowState: + document_id: str + filename: str + workflow_id: str + stage: DocumentStage = DocumentStage.READY_FOR_INGESTION + error_message: Optional[str] = None + chunk_count: int = 0 + records_ingested: int = 0 + + +@workflow.defn +class ReingestionWorkflow: + """Lightweight workflow for re-ingesting completed documents to the vector index.""" + + def __init__(self): + self.state = None + + @workflow.run + async def run( + self, + document_id: str, + filename: str, + original_workflow_id: str, + page_count: int = 0, + chunk_count: int = 0, + index_name: str = "documents-index", + index_only: bool = False, + ) -> dict: + self.state = ReingestionWorkflowState( + document_id=document_id, + filename=filename, + workflow_id=original_workflow_id, + chunk_count=chunk_count, + ) + + try: + await _mirror_state(original_workflow_id, "ingesting", page_count, chunk_count, None) + + self.state.stage = DocumentStage.INGESTING + ingest_args = [original_workflow_id, document_id, filename, index_name] + if index_only: + ingest_args.extend([10, None, True]) + if workflow.patched("freeze-index-only-reingest-v1"): + ingest_args.extend([True, workflow.info().workflow_id]) + result = await workflow.execute_activity( + ingest_document_from_db, + args=ingest_args, + start_to_close_timeout=timedelta(minutes=90), + retry_policy=INGEST_RETRY, + ) + + self.state.records_ingested = result.get("records_ingested", 0) + + if not index_only and not workflow.patched("publish-after-prod-v1"): + # Old order, preserved for in-flight workflows whose history + # already recorded this activity here. New executions publish + # to the network only from PromoteToProdWorkflow, once this + # reingest's own PROD promotion actually succeeds - this + # workflow never performs that promotion itself. + self.state.stage = DocumentStage.PUBLISHING_TO_NETWORK + await _mirror_state(original_workflow_id, "publishing_to_network", page_count, chunk_count, None) + + # Content changed on reingest, so the network gets a fresh publish + # with a new transactionId (same replay-safety rule as above). + network_transaction_id = str(workflow.uuid4()) + await workflow.execute_activity( + publish_catalog_to_network, + args=[original_workflow_id, network_transaction_id], + start_to_close_timeout=timedelta(minutes=10), + retry_policy=NETWORK_PUBLISH_RETRY, + ) + + self.state.stage = DocumentStage.COMPLETED if index_only else DocumentStage.APPROVAL_FOR_PROD + await _mirror_state(original_workflow_id, self.state.stage.value, page_count, chunk_count, None) + + return { + "document_id": document_id, + "filename": filename, + "stage": self.state.stage.value, + "chunks": chunk_count, + "records_ingested": self.state.records_ingested, + } + + except Exception as e: + self.state.stage = DocumentStage.FAILED + self.state.error_message = str(e) + try: + await _mirror_state(original_workflow_id, "failed", page_count, chunk_count, str(e)) + except Exception: + pass + raise + + @workflow.query + def get_state(self) -> dict: + if not self.state: + return {} + return { + "document_id": self.state.document_id, + "filename": self.state.filename, + "workflow_id": self.state.workflow_id, + "stage": self.state.stage.value, + "error_message": self.state.error_message, + "chunk_count": self.state.chunk_count, + "records_ingested": self.state.records_ingested, + } + + +@workflow.defn +class PromoteToProdWorkflow: + """Promote a DEV-ingested document into the PROD Qdrant collection.""" + + def __init__(self): + self.state = None + + @workflow.run + async def run( + self, + document_id: str, + filename: str, + original_workflow_id: str, + page_count: int = 0, + chunk_count: int = 0, + ) -> dict: + self.state = { + "document_id": document_id, + "filename": filename, + "workflow_id": original_workflow_id, + "stage": "ingesting_prod", + } + try: + await _mirror_state(original_workflow_id, "ingesting_prod", page_count, chunk_count, None) + result = await workflow.execute_activity( + promote_document_to_prod_qdrant, + args=[original_workflow_id, document_id, filename], + start_to_close_timeout=timedelta(minutes=90), + retry_policy=INGEST_RETRY, + ) + + if workflow.patched("publish-after-prod-v1"): + # New: this workflow now also announces the network catalog, + # since promotion just succeeded here (previously it never + # published at all - see DocumentPipelineWorkflow/ReingestionWorkflow + # for the equivalent post-promotion publish). + self.state["stage"] = "publishing_to_network" + await _mirror_state(original_workflow_id, "publishing_to_network", page_count, chunk_count, None) + + network_transaction_id = str(workflow.uuid4()) + await workflow.execute_activity( + publish_catalog_to_network, + args=[original_workflow_id, network_transaction_id], + start_to_close_timeout=timedelta(minutes=10), + retry_policy=NETWORK_PUBLISH_RETRY, + ) + + await _mirror_state(original_workflow_id, "completed", page_count, chunk_count, None) + self.state["stage"] = "completed" + return { + "document_id": document_id, + "filename": filename, + "stage": "completed", + "records_promoted_to_prod": result.get("records_ingested", 0), + } + except Exception as e: + self.state["stage"] = "failed" + self.state["error_message"] = str(e) + try: + await _mirror_state(original_workflow_id, "failed", page_count, chunk_count, str(e)) + except Exception: + pass + raise + + @workflow.query + def get_state(self) -> dict: + return self.state or {} diff --git a/pipeline/translation/gemma_vllm.py b/pipeline/translation/gemma_vllm.py index 1c499ee..198f769 100644 --- a/pipeline/translation/gemma_vllm.py +++ b/pipeline/translation/gemma_vllm.py @@ -4,9 +4,10 @@ import httpx +from pipeline.metadata.localization import normalize_language_tag from .base import TranslationConfig, TranslationProvider -TRANSLATE_PROMPT = """Translate the following text to English. +TRANSLATE_PROMPT = """Translate the following text to the language with tag {target_language}. Preserve all formatting, including markdown syntax, tables, and bullet points. Maintain technical terminology accurately. If there are proper nouns or names, keep them as-is or transliterate appropriately. @@ -26,14 +27,17 @@ def __init__(self, config: TranslationConfig): self._endpoint = config.endpoint.rstrip("/") + "/chat/completions" def translate(self, text: str, *, source_lang: str, target_language: str) -> str: - del source_lang, target_language + del source_lang + target_tag = normalize_language_tag(target_language) + if not target_tag: + raise ValueError("Translation target must be a BCP 47 language tag") headers = {"Content-Type": "application/json"} if self.config.api_key: headers["Authorization"] = f"Bearer {self.config.api_key}" payload = { "model": self.config.model, - "messages": [{"role": "user", "content": TRANSLATE_PROMPT.format(text=text)}], + "messages": [{"role": "user", "content": TRANSLATE_PROMPT.format(text=text, target_language=target_tag)}], } if self.config.temperature is not None: payload["temperature"] = self.config.temperature diff --git a/pipeline/translation/service.py b/pipeline/translation/service.py index b5cdd81..9073f44 100644 --- a/pipeline/translation/service.py +++ b/pipeline/translation/service.py @@ -8,11 +8,13 @@ from datetime import datetime from typing import Callable, Optional +from pipeline.metadata.localization import normalize_language_tag from .base import TranslationConfig, TranslationProvider from .gemma_vllm import GemmaVllmTranslationProvider from .script_detect import analyze_script, detect_any, disambiguate PROVIDERS: dict[str, type[TranslationProvider]] = { + "openai_compatible": GemmaVllmTranslationProvider, "gemma_vllm": GemmaVllmTranslationProvider, "gemma4": GemmaVllmTranslationProvider, "gemma": GemmaVllmTranslationProvider, @@ -35,27 +37,31 @@ "zl": "gu", } -def _gemma_endpoint() -> str: - """Resolve OpenAI-compatible Gemma base URL (AGRINET preferred).""" +def configured_translation_endpoint() -> str: return ( - os.environ.get("AGRINET_GEMMA_BASE_URL") + os.environ.get("TRANSLATION_BASE_URL") + or os.environ.get("AGRINET_GEMMA_BASE_URL") or os.environ.get("TRANSLATION_VLLM_BASE_URL") - or "http://localhost:8020/v1" + or "" ).strip() +def configured_translation_type() -> str: + return os.environ.get("TRANSLATION_TYPE", os.environ.get("TRANSLATION_PROVIDER", "gemma_vllm")).strip().lower() + + def _gemma_model() -> str: return ( - os.environ.get("AGRINET_GEMMA_MODEL_NAME") - or os.environ.get("TRANSLATION_MODEL") + os.environ.get("TRANSLATION_MODEL") + or os.environ.get("AGRINET_GEMMA_MODEL_NAME") or "google/gemma-4-31b-it" ).strip() or "google/gemma-4-31b-it" def load_translation_config(target_language: str = "en") -> TranslationConfig: - provider = os.environ.get("TRANSLATION_PROVIDER", "gemma_vllm").strip().lower() + provider = configured_translation_type() model = _gemma_model() - endpoint = _gemma_endpoint() + endpoint = configured_translation_endpoint() or "http://localhost:8020/v1" # Ensure OpenAI-compatible path: base may be .../gemma4 without trailing /v1 if endpoint and not endpoint.rstrip("/").endswith("/v1"): endpoint = endpoint.rstrip("/") + "/v1" @@ -143,23 +149,13 @@ def clean_translation(text: str) -> str: return result.strip() -def _contains_gujarati_script(text: str) -> bool: - if not text: - return False - return any("\u0A80" <= ch <= "\u0AFF" for ch in text) - - def normalize_detected_language(detected_lang: str | None, page_text: str) -> str: - lowered = (detected_lang or "en").lower() - # Fall back to the raw code (already <=3 chars, ISO 639-3) rather than - # truncating to 2 — a blind [:2] slice can collide with a DIFFERENT - # real language's actual ISO 639-1 code for an unmapped input. - normalized = LANG_MAP.get(lowered, lowered[:3] if lowered else "en") - if normalized in {"unknown", "un", "und", "xx", "zl"}: - if _contains_gujarati_script(page_text): - return "gu" - return "en" - return normalized + lowered = (detected_lang or "und").strip().lower() + normalized = normalize_language_tag(LANG_MAP.get(lowered, lowered)) + if normalized and normalized not in {"unknown", "un", "und", "xx", "zl"}: + return normalized + script = analyze_script(page_text, min_chars=1, min_ratio=0.3) + return script.language if script.is_non_english and not script.ambiguous else "und" async def detect_page_languages( @@ -169,11 +165,8 @@ async def detect_page_languages( ) -> dict[int, str]: """Decide each page's language, gating on script regex before any model call. - With the gate on (default), a page is only considered for translation when - it actually contains non-Latin script. pyfranc is then consulted in-process - just for the scripts that map to more than one language (Devanagari, - Bengali), and its answer is accepted only if it names a language in that - script's family — a page of Devanagari cannot be French. + Use script as evidence where possible. Use language detection for Latin + and other scripts without a reliable script-derived language. """ config = config or load_translation_config() @@ -212,9 +205,10 @@ async def _detect_via_script_gate( ) if not analysis.is_non_english: - detected_languages[i] = "en" + language = detect_any(text) if analysis.letter_chars >= 80 else None + detected_languages[i] = normalize_detected_language(language, text) if language else "und" if log: - log("Page %s: regex → %s → SKIP translation", page_no, analysis.summary()) + log("Page %s: language=%s", page_no, detected_languages[i]) continue detected_languages[i] = analysis.language @@ -229,7 +223,7 @@ async def _detect_via_script_gate( if log: log( - "Script gate result: %s/%s page(s) need translation %s; %s skipped as English", + "Script gate result: %s/%s page(s) identified by script %s; %s evaluated separately", len(non_english), len(pages), non_english or "[]", @@ -289,45 +283,15 @@ async def _detect_via_lang_detect( pages: list[dict], log: Optional[Callable[..., None]] = None, ) -> dict[int, str]: - """Legacy per-line detection, kept for TRANSLATION_SCRIPT_GATE_ENABLED=false. - - Runs pyfranc unrestricted (no whitelist) across all 414 languages it - knows, which is the same trade-off the old franc-min-backed lang-detect - service made: short OCR lines can be misdetected as an unrelated Latin - language. That's exactly why the script gate above is the default path — - it never calls pyfranc without a script-derived whitelist. - """ + """Detect the language of each page as a whole, leaving short pages unknown.""" detected_languages: dict[int, str] = {} for i, page in enumerate(pages): text = page.get("edited_markdown") or page.get("original_markdown", "") if not text or len(text.strip()) < 20: - detected_languages[i] = "en" + detected_languages[i] = "und" continue - - lines = [line.strip() for line in text.split("\n") if len(line.strip()) >= 10] - if not lines: - detected_languages[i] = "en" - continue - - non_english_lang = None - for line in lines: - lang = detect_any(line) - if lang and lang not in {"en", "und"}: - non_english_lang = lang - if log: - log( - "Page %s: Found non-English content, detected language: %s", - page.get("page_number"), - lang, - ) - break - - page_text = page.get("edited_markdown") or page.get("original_markdown", "") - detected_languages[i] = normalize_detected_language( - non_english_lang if non_english_lang else "en", - page_text, - ) + detected_languages[i] = normalize_detected_language(detect_any(text), text) return detected_languages @@ -340,7 +304,7 @@ async def translate_pages( log: Optional[Callable[..., None]] = None, ) -> list[dict]: config = config or load_translation_config(target_language=target_language) - provider = get_translation_provider(config) + provider = None if config.provider == "none" else get_translation_provider(config) if log: log("Processing %s pages for translation", len(pages)) @@ -353,7 +317,15 @@ async def translate_pages( config.target_language, ) - detected_languages = await detect_page_languages(pages, log=log, config=config) + declared = { + i: language for i, page in enumerate(pages) + if (language := normalize_language_tag((page.get("metadata") or {}).get("declared_language"))) + } + missing = [i for i in range(len(pages)) if i not in declared] + detected_languages = dict(declared) + if missing: + detected = await detect_page_languages([pages[i] for i in missing], log=log, config=config) + detected_languages.update({missing[i]: language for i, language in detected.items()}) pages_to_translate: list[tuple[int, dict, str]] = [] for i, page in enumerate(pages): @@ -361,7 +333,11 @@ async def translate_pages( continue detected_lang = detected_languages[i] page["detected_language"] = detected_lang - if detected_lang != "en": + target = config.target_language + variant = (page.get("translations") or {}).get(target) or {} + already_available = bool(variant.get("edited_translation") or variant.get("translated_markdown")) + if (provider and not already_available and detected_lang not in {"und", "mul", target} + and not detected_lang.startswith(f"{target}-")): pages_to_translate.append((i, page, detected_lang)) if log: @@ -429,6 +405,14 @@ async def translate_page(idx: int, page: dict, lang: str) -> tuple[int, str | No pages[idx]["translation_model"] = config.model pages[idx]["translation_target_language"] = config.target_language pages[idx]["translated_at"] = translated_at + variants = dict(pages[idx].get("translations") or {}) + variants[config.target_language] = { + "translated_markdown": translation, + "provider": config.provider, + "model": config.model, + "translated_at": translated_at, + } + pages[idx]["translations"] = variants translated_count += 1 elif error: page_no = pages[idx].get("page_number", idx) diff --git a/pipeline/vector_store/__init__.py b/pipeline/vector_store/__init__.py index d17f66f..b96d840 100644 --- a/pipeline/vector_store/__init__.py +++ b/pipeline/vector_store/__init__.py @@ -8,7 +8,7 @@ def get_default_index_name() -> str: - return os.environ.get("VECTOR_DB_COLLECTION_NAME") or "documents-index" + return (os.environ.get("VECTOR_DB_COLLECTION_NAME") or "").strip() or "documents-index" def get_vector_store() -> VectorStore: diff --git a/pipeline/vector_store/base.py b/pipeline/vector_store/base.py index 3c851a6..afc7641 100644 --- a/pipeline/vector_store/base.py +++ b/pipeline/vector_store/base.py @@ -57,8 +57,16 @@ def search( valid_on: Optional[str] = None, instance: Optional[str] = None, provider_id: Optional[str] = None, + crop_subject_ids: Optional[list[str]] = None, + allowed_instances: Optional[list[str]] = None, + strict_language: Optional[str] = None, + applicability: Optional[dict[str, list[str]]] = None, + excluded_revision_ids: Optional[list[str]] = None, + document_workflow_id: Optional[str] = None, ) -> dict[str, Any]: """`apply_validity` keeps only chunks whose validity period covers `valid_on` (default: the store's today). `instance` limits results to - one group's chunks; None means every group.""" + one group's chunks. `allowed_instances` intersects that filter with + the caller's provider scope. `strict_language` requires an exact + available language tag before ranking.""" ... diff --git a/pipeline/vector_store/embeddings.py b/pipeline/vector_store/embeddings.py index bc9feec..240e7d7 100644 --- a/pipeline/vector_store/embeddings.py +++ b/pipeline/vector_store/embeddings.py @@ -4,6 +4,7 @@ import logging import os +from threading import Lock from typing import Iterable import httpx @@ -13,6 +14,7 @@ DEFAULT_EMBEDDING_MODEL = "intfloat/multilingual-e5-large" DEFAULT_VECTOR_SIZE = 1024 _local_embedder = None +_local_embedder_lock = Lock() def get_embedding_model_name() -> str: @@ -38,6 +40,23 @@ def get_embedding_provider() -> str: return "sentence_transformers" +def get_embedding_prompt_style() -> str: + style = os.environ.get("EMBEDDING_PROMPT_STYLE", "e5").strip().lower() + if style not in {"e5", "embeddinggemma", "plain"}: + raise ValueError("EMBEDDING_PROMPT_STYLE must be e5, embeddinggemma, or plain") + return style + + +def format_embedding_input(text: str, kind: str, *, use_e5_prefix: bool = True) -> str: + style = get_embedding_prompt_style() + if style == "embeddinggemma": + prefix = "task: search result | query: " if kind == "query" else "title: none | text: " + return prefix + text.strip() + if style == "plain" or not use_e5_prefix: + return text + return _prefix_query(text) if kind == "query" else _prefix_passage(text) + + def _prefix_passage(text: str) -> str: cleaned = (text or "").strip() if cleaned.lower().startswith("passage:"): @@ -56,7 +75,7 @@ def _embed_openai_compatible(texts: list[str]) -> list[list[float]]: base_url = (os.environ.get("EMBEDDING_BASE_URL") or "").rstrip("/") if not base_url: raise RuntimeError("EMBEDDING_BASE_URL is required for openai_compatible embeddings") - api_key = os.environ.get("EMBEDDING_API_KEY") or os.environ.get("OPENAI_API_KEY") or "" + api_key = os.environ.get("EMBEDDING_API_KEY", "") model = get_embedding_model_name() url = base_url if base_url.endswith("/embeddings") else f"{base_url}/embeddings" headers = {"Content-Type": "application/json"} @@ -78,7 +97,9 @@ def _embed_openai_compatible(texts: list[str]) -> list[list[float]]: def _get_local_embedder(): global _local_embedder - if _local_embedder is None: + with _local_embedder_lock: + if _local_embedder is not None: + return _local_embedder try: from sentence_transformers import SentenceTransformer except ModuleNotFoundError as exc: @@ -92,7 +113,7 @@ def _get_local_embedder(): model_name = get_embedding_model_name() logger.info("Loading SentenceTransformer model %s", model_name) _local_embedder = SentenceTransformer(model_name) - return _local_embedder + return _local_embedder def _embed_local(texts: list[str]) -> list[list[float]]: @@ -107,15 +128,7 @@ def embed_texts( use_e5_prefix: bool = True, ) -> list[list[float]]: """Embed texts as passage or query vectors.""" - prepared: list[str] = [] - for text in texts: - raw = text or "" - if not use_e5_prefix: - prepared.append(raw) - elif kind == "query": - prepared.append(_prefix_query(raw)) - else: - prepared.append(_prefix_passage(raw)) + prepared = [format_embedding_input(text or "", kind, use_e5_prefix=use_e5_prefix) for text in texts] if not prepared: return [] @@ -151,7 +164,7 @@ def count_tokens(text: str) -> int: Uses the local model's tokenizer when embeddings run locally; for a remote (OpenAI-compatible) provider there is no tokenizer here, so it estimates ~4 chars/token. """ - prefixed = _prefix_passage(text) + prefixed = format_embedding_input(text, "passage") if get_embedding_provider() == "openai_compatible": return max(1, len(prefixed) // 4) return len(_get_local_embedder().tokenizer.encode(prefixed)) diff --git a/pipeline/vector_store/qdrant_store.py b/pipeline/vector_store/qdrant_store.py index eaeace2..0dbeda5 100644 --- a/pipeline/vector_store/qdrant_store.py +++ b/pipeline/vector_store/qdrant_store.py @@ -3,6 +3,7 @@ from __future__ import annotations import logging +import math import os import re from typing import Any, Optional @@ -11,8 +12,8 @@ from qdrant_client.http import models as qmodels from qdrant_client.http.exceptions import UnexpectedResponse -from ..documents.document_validity import Clock, system_clock, today -from .embeddings import embed_passages, embed_query, get_vector_size +from pipeline.documents.document_validity import Clock, system_clock, today +from .embeddings import embed_passages, embed_query, get_vector_size, get_embedding_prompt_style logger = logging.getLogger(__name__) @@ -24,14 +25,36 @@ "instance", "instance_name", "provider_id", + "crop_subjects", + "knowledge_metadata", + "subject_categories", + "agriculture_subject_ids", + "coverage_area_keys", + "excluded_coverage_area_keys", + "season_keys", + "applicability_groups", + "applicability_scope", + "tag_keys", + "chunk_tags", + "source_url", "type", "source", "filename", + "name", + # Older frozen revisions and structured advisory imports still carry these fields. "name_gu", "name_en", "title_en", "title_gu", "doc_language", + "primary_language", + "primary_text", + "text_language", + "translations", + "source_evidence", + "available_languages", + "revision_id", + "release_network_id", "category_tags", "doc_short_description", "doc_llm_description", @@ -177,6 +200,19 @@ def _record_payload(record: dict[str, Any]) -> dict[str, Any]: return payload +def _same_cosine_vector(stored: Any, expected: list[float]) -> bool: + if not isinstance(stored, list) or len(stored) != len(expected): + return False + stored_norm = math.hypot(*stored) + expected_norm = math.hypot(*expected) + if not 0 < stored_norm < math.inf or not 0 < expected_norm < math.inf: + return False + return all( + math.isclose(left / stored_norm, right / expected_norm, abs_tol=1e-6, rel_tol=1e-5) + for left, right in zip(stored, expected) + ) + + def _hit_from_point(point: Any) -> dict[str, Any]: payload = dict(point.payload or {}) hit = { @@ -242,6 +278,12 @@ def _build_filter( valid_on: Optional[str] = None, instance: Optional[str] = None, provider_id: Optional[str] = None, + crop_subject_ids: Optional[list[str]] = None, + allowed_instances: Optional[list[str]] = None, + strict_language: Optional[str] = None, + applicability: Optional[dict[str, list[str]]] = None, + excluded_revision_ids: Optional[list[str]] = None, + document_workflow_id: Optional[str] = None, ) -> Optional[qmodels.Filter]: """Assemble a Qdrant filter. `valid_on` (`YYYY-MM-DD`) restricts the result to chunks valid on that day; None leaves validity out entirely, which is @@ -249,10 +291,50 @@ def _build_filter( document's chunk. `instance` keeps only chunks tagged with that group code (the API calls it `group`); None means every group.""" must: list[qmodels.Condition] = [] + must_not: list[qmodels.Condition] = [] if instance is not None: must.append(_exact("instance", instance)) + if allowed_instances is not None: + must.append(qmodels.FieldCondition( + key="instance", match=qmodels.MatchAny(any=allowed_instances), + )) if provider_id is not None: must.append(_exact("provider_id", provider_id)) + if strict_language is not None: + must.append(_exact("available_languages", strict_language)) + if crop_subject_ids: + must.append(qmodels.FieldCondition( + key="crop_subjects[].subjectId", match=qmodels.MatchAny(any=crop_subject_ids), + )) + contextual = [] + for key, values in (applicability or {}).items(): + if not values: + continue + exact = qmodels.FieldCondition(key=key, match=qmodels.MatchAny(any=values)) + if key == "coverage_area_keys": + must_not.append(qmodels.FieldCondition(key="excluded_coverage_area_keys", match=qmodels.MatchAny(any=values))) + if key in {"coverage_area_keys", "season_keys"}: + contextual.append(exact) + else: + must.append(exact) + if contextual: + group_key = "applicability_groups" + legacy_or_broad = qmodels.Filter(should=[ + _exact("applicability_scope", "broad"), + qmodels.IsEmptyCondition(is_empty=qmodels.PayloadField(key="applicability_scope")), + ]) + flat = [ + qmodels.Filter(should=[condition, qmodels.Filter(must=[ + legacy_or_broad, qmodels.IsEmptyCondition(is_empty=qmodels.PayloadField(key=condition.key)), + ])]) + for condition in contextual + ] + must.append(qmodels.Filter(should=[ + qmodels.NestedCondition(nested=qmodels.Nested( + key=group_key, filter=qmodels.Filter(must=contextual), + )), + qmodels.Filter(must=[qmodels.IsEmptyCondition(is_empty=qmodels.PayloadField(key=group_key)), *flat]), + ])) if doc_id is not None: must.append(_exact("doc_id", doc_id)) if chunk_num is not None: @@ -261,9 +343,13 @@ def _build_filter( must.append(_exact("is_reference", False)) if valid_on: must.extend(_validity_conditions(valid_on)) - if not must: + if excluded_revision_ids: + must_not.append(qmodels.FieldCondition(key="revision_id", match=qmodels.MatchAny(any=excluded_revision_ids))) + if document_workflow_id is not None: + must.append(_exact("workflow_id", document_workflow_id)) + if not must and not must_not: return None - return qmodels.Filter(must=must) + return qmodels.Filter(must=must or None, must_not=must_not or None) class QdrantVectorStore: @@ -292,9 +378,21 @@ def today(self) -> str: PAYLOAD_INDEXES = { "doc_id": qmodels.PayloadSchemaType.KEYWORD, "workflow_id": qmodels.PayloadSchemaType.KEYWORD, + "revision_id": qmodels.PayloadSchemaType.KEYWORD, "filename": qmodels.PayloadSchemaType.KEYWORD, "instance": qmodels.PayloadSchemaType.KEYWORD, "provider_id": qmodels.PayloadSchemaType.KEYWORD, + "available_languages": qmodels.PayloadSchemaType.KEYWORD, + "crop_subjects[].subjectId": qmodels.PayloadSchemaType.KEYWORD, + "subject_categories": qmodels.PayloadSchemaType.KEYWORD, + "agriculture_subject_ids": qmodels.PayloadSchemaType.KEYWORD, + "coverage_area_keys": qmodels.PayloadSchemaType.KEYWORD, + "excluded_coverage_area_keys": qmodels.PayloadSchemaType.KEYWORD, + "season_keys": qmodels.PayloadSchemaType.KEYWORD, + "applicability_scope": qmodels.PayloadSchemaType.KEYWORD, + "applicability_groups[].coverage_area_keys": qmodels.PayloadSchemaType.KEYWORD, + "applicability_groups[].season_keys": qmodels.PayloadSchemaType.KEYWORD, + "tag_keys": qmodels.PayloadSchemaType.KEYWORD, "chunk_num": qmodels.PayloadSchemaType.INTEGER, "is_reference": qmodels.PayloadSchemaType.BOOL, "type": qmodels.PayloadSchemaType.KEYWORD, @@ -320,6 +418,15 @@ def _ensure_payload_indexes(self, name: str) -> None: except Exception as exc: logger.debug("Payload index %s skipped/exists: %s", field_name, exc) + def create_new_collection(self, name: str) -> None: + # Qdrant atomically rejects an existing collection. Never delete it. + if not self.client.create_collection( + collection_name=name, + vectors_config=qmodels.VectorParams(size=get_vector_size(), distance=qmodels.Distance.COSINE), + ): + raise RuntimeError("Qdrant did not create the new collection") + self._ensure_payload_indexes(name) + def ensure_collection(self, name: str, recreate: bool = False) -> dict[str, Any]: vector_size = get_vector_size() exists = False @@ -402,8 +509,9 @@ def upsert( } passages: list[str] = [] + use_e5_text = get_embedding_prompt_style() == "e5" for record in records: - text_for_emb = record.get("text_for_embedding") + text_for_emb = record.get("text_for_embedding") if use_e5_text else None if text_for_emb is None: text_for_emb = record.get("text") or "" passages.append(str(text_for_emb)) @@ -411,6 +519,8 @@ def upsert( # If passages already include the E5 "passage:" prefix, do not double-prefix. use_prefix = not all(str(p or "").lower().startswith("passage:") for p in passages) vectors = embed_passages(passages, use_e5_prefix=use_prefix, batch_size=batch_size) + if len(vectors) != len(records): + raise ValueError("The embedding model did not return one vector per passage") points = [ qmodels.PointStruct( @@ -423,7 +533,21 @@ def upsert( for i in range(0, len(points), batch_size): batch = points[i : i + batch_size] - self.client.upsert(collection_name=name, points=batch) + self.client.upsert(collection_name=name, points=batch, wait=True) + release_points = [point for point in batch if (point.payload or {}).get("release_network_id")] + if release_points: + visible = self.client.retrieve( + collection_name=name, ids=[point.id for point in release_points], + with_payload=True, with_vectors=True, consistency="all", + ) + by_id = {str(point.id): point for point in visible} + for expected in release_points: + stored = by_id.get(str(expected.id)) + if ( + stored is None or stored.payload != expected.payload + or not _same_cosine_vector(stored.vector, expected.vector) + ): + raise ValueError("Qdrant could not verify the release projection") stats = self.get_stats(name) return { @@ -433,6 +557,18 @@ def upsert( "backend": self.backend, } + def remove_superseded_passages(self, name: str, doc_id: str, records: list[dict[str, Any]]) -> None: + if not records or any(row["doc_id"] != doc_id for row in records): + raise ValueError("Replacement passages must belong to one document") + self.client.delete( + collection_name=name, + points_selector=qmodels.FilterSelector(filter=qmodels.Filter( + must=[_exact("doc_id", doc_id)], + must_not=[qmodels.HasIdCondition(has_id=[point_id_for_record(row) for row in records])], + )), + wait=True, + ) + def delete_by_doc_id(self, name: str, doc_id: str) -> dict[str, Any]: try: self.client.delete( @@ -539,6 +675,12 @@ def search( valid_on: Optional[str] = None, instance: Optional[str] = None, provider_id: Optional[str] = None, + crop_subject_ids: Optional[list[str]] = None, + allowed_instances: Optional[list[str]] = None, + strict_language: Optional[str] = None, + applicability: Optional[dict[str, list[str]]] = None, + excluded_revision_ids: Optional[list[str]] = None, + document_workflow_id: Optional[str] = None, ) -> dict[str, Any]: """Search `name`, by default returning only chunks valid today. @@ -548,15 +690,23 @@ def search( and neither is what a caller serving an end user should pass. `instance` limits the search to one group's chunks (e.g. `mh`); None - searches every group. The same filter feeds every mode. + searches every permitted group. The same filter feeds every mode. """ mode = (search_mode or "TENSOR").upper() as_of = (valid_on or self.today()) if apply_validity else None + if allowed_instances == []: + return {"hits": [], "backend": self.backend, "search_mode": mode, "valid_on": as_of} query_filter = _build_filter( exclude_reference=exclude_reference, valid_on=as_of, instance=instance, provider_id=provider_id, + crop_subject_ids=crop_subject_ids, + allowed_instances=allowed_instances, + strict_language=strict_language, + applicability=applicability, + excluded_revision_ids=excluded_revision_ids, + document_workflow_id=document_workflow_id, ) if mode == "LEXICAL": @@ -571,9 +721,9 @@ def search( ) scored: list[tuple[float, Any]] = [] for point in points: - text = str((point.payload or {}).get("text") or "") - desc = str((point.payload or {}).get("description") or "") - blob = f"{text} {desc}".lower() + payload = point.payload or {} + translations = (entry.get("text") or "" for entry in (payload.get("translations") or {}).values()) + blob = " ".join([payload.get("text") or "", payload.get("description") or "", *translations]).lower() if not tokens: score = 0.0 else: diff --git a/pipeline/worker.py b/pipeline/worker.py index ed1a504..485ee66 100644 --- a/pipeline/worker.py +++ b/pipeline/worker.py @@ -1,142 +1,8 @@ -""" -Temporal worker for the OCR pipeline. -""" +"""Temporal worker command entry point.""" import asyncio -import logging -import os -from temporalio.client import Client -from temporalio.worker import Worker - -from . import db -from .activities import ( - auto_tag_chunks_from_db, - create_chunks, - create_chunks_from_db, - detect_and_translate_pages, - detect_and_translate_pages_from_db, - index_webhook_advisory, - ingest_document_from_db, - ingest_to_vector_db, - persist_document_content, - prepare_for_ingestion, - promote_document_to_prod_qdrant, - promote_webhook_advisory, - publish_catalog_to_network, - run_ocr, - run_ocr_and_store, - update_document_state, -) -from .temporal.activities.logging import ActivityLogging -from .advisory_start import backfill_on_start -from .network_group import validate_network_provider_config -from .telemetry.logs import configure_logs, reset_logs -from .workflows import ( - AdvisoryIndexWorkflow, - AdvisoryPromoteWorkflow, - ChunkingOnlyWorkflow, - DocumentPipelineWorkflow, - OcrOnlyWorkflow, - PromoteToProdWorkflow, - ReingestionWorkflow, - TranslationOnlyWorkflow, -) - -LOG_LEVEL = getattr(logging, os.environ.get("LOG_LEVEL", "INFO").strip().upper(), logging.INFO) - -# Configure verbose logging -logging.basicConfig( - level=LOG_LEVEL, - format='%(asctime)s [%(levelname)s] %(name)s: %(message)s', - datefmt='%H:%M:%S' -) - -# Set Temporal SDK logging to INFO regardless of LOG_LEVEL - its own DEBUG -# output is noisy and rarely what LOG_LEVEL=DEBUG is set to see. -logging.getLogger("temporalio").setLevel(logging.INFO) - -TASK_QUEUE = "ocr-pipeline" -INTERCEPTORS = [ActivityLogging()] - - -async def main(): - """Start the worker.""" - configure_logs() - try: - await _run() - except Exception: - logging.exception("worker crashed") - raise - finally: - logging.info("worker stopping") - reset_logs() - - -async def _run(): - validate_network_provider_config() - - translation_provider = os.environ.get("TRANSLATION_PROVIDER", "gemma_vllm").strip().lower() - if translation_provider in {"gemma_vllm", "gemma4", "gemma"}: - if not os.environ.get("TRANSLATION_VLLM_BASE_URL", "").strip(): - logging.error("worker startup: TRANSLATION_VLLM_BASE_URL not set, exiting") - return - - logging.info("worker startup: initializing PostgreSQL database") - db.init_db() - - temporal_host = os.environ.get("TEMPORAL_HOST", "localhost:7233") - max_concurrent_activities = int(os.environ.get("TEMPORAL_MAX_CONCURRENT_ACTIVITIES", "4")) - logging.info("worker startup: connecting to Temporal at %s", temporal_host) - - client = await Client.connect(temporal_host) - - worker = Worker( - client, - task_queue=TASK_QUEUE, - max_concurrent_activities=max_concurrent_activities, - interceptors=INTERCEPTORS, - workflows=[ - DocumentPipelineWorkflow, - ReingestionWorkflow, - PromoteToProdWorkflow, - TranslationOnlyWorkflow, - OcrOnlyWorkflow, - ChunkingOnlyWorkflow, - AdvisoryIndexWorkflow, - AdvisoryPromoteWorkflow, - ], - activities=[ - run_ocr, - run_ocr_and_store, - create_chunks, - create_chunks_from_db, - auto_tag_chunks_from_db, - prepare_for_ingestion, - ingest_to_vector_db, - ingest_document_from_db, - promote_document_to_prod_qdrant, - index_webhook_advisory, - promote_webhook_advisory, - publish_catalog_to_network, - update_document_state, - detect_and_translate_pages, - detect_and_translate_pages_from_db, - persist_document_content, - ], - ) - - backfilled = await backfill_on_start(client) - if backfilled: - logging.info("Advisory startup backfill queued %s workflows", backfilled) - - logging.info( - "worker started queue=%s temporal=%s activity_concurrency=%s", - TASK_QUEUE, - temporal_host, - max_concurrent_activities, - ) - await worker.run() +from pipeline.temporal.worker import main if __name__ == "__main__": diff --git a/pipeline/workflows.py b/pipeline/workflows.py deleted file mode 100644 index 4ce63fb..0000000 --- a/pipeline/workflows.py +++ /dev/null @@ -1,947 +0,0 @@ -""" -Temporal workflows for the OCR pipeline. - -The workflow pauses at review stages, waiting for user signals to continue. -""" - -import os -from dataclasses import dataclass -from datetime import timedelta -from typing import Optional - -from temporalio import workflow -from temporalio.common import RetryPolicy - -with workflow.unsafe.imports_passed_through(): - from .activities import ( - auto_tag_chunks_from_db, - create_chunks_from_db, - detect_and_translate_pages_from_db, - index_webhook_advisory, - ingest_document_from_db, - promote_document_to_prod_qdrant, - promote_webhook_advisory, - publish_catalog_to_network, - run_ocr_and_store, - update_document_state, - ) - from .models import DocumentStage - - -def _now_iso() -> str: - """Get current time as ISO string (workflow-safe).""" - return workflow.now().isoformat() - - -OCR_RETRY = RetryPolicy( - initial_interval=timedelta(seconds=15), - backoff_coefficient=2.0, - maximum_interval=timedelta(minutes=15), - maximum_attempts=20, -) - -CHUNK_RETRY = RetryPolicy( - initial_interval=timedelta(seconds=5), - backoff_coefficient=1.5, - maximum_interval=timedelta(minutes=2), - maximum_attempts=10, -) - -INGEST_RETRY = RetryPolicy( - initial_interval=timedelta(seconds=10), - backoff_coefficient=2.0, - maximum_interval=timedelta(minutes=10), - maximum_attempts=15, -) - -STATE_UPDATE_RETRY = RetryPolicy( - initial_interval=timedelta(seconds=3), - backoff_coefficient=1.5, - maximum_interval=timedelta(minutes=1), - maximum_attempts=10, -) - -TRANSLATION_RETRY = RetryPolicy( - initial_interval=timedelta(seconds=15), - backoff_coefficient=2.0, - maximum_interval=timedelta(minutes=15), - maximum_attempts=20, -) - -# Shorter than INGEST_RETRY: the Discovery Service is a new, unproven external -# dependency, so we fail the document faster rather than holding it in-flight -# for hours. Configurable (unlike the other policies above) since this -# dependency's real-world retry behavior is still unproven. -def _network_publish_retry_policy() -> RetryPolicy: - return RetryPolicy( - initial_interval=timedelta( - seconds=int(os.environ.get("DISCOVERY_SERVICE_PUBLISH_INITIAL_INTERVAL_SECONDS", "30")) - ), - backoff_coefficient=float( - os.environ.get("DISCOVERY_SERVICE_PUBLISH_BACKOFF_COEFFICIENT", "2.0") - ), - maximum_interval=timedelta( - seconds=int(os.environ.get("DISCOVERY_SERVICE_PUBLISH_MAX_INTERVAL_SECONDS", "300")) - ), - maximum_attempts=int(os.environ.get("DISCOVERY_SERVICE_PUBLISH_MAX_ATTEMPTS", "5")), - ) - - -NETWORK_PUBLISH_RETRY = _network_publish_retry_policy() - - -async def _mirror_state( - workflow_id: str, - stage: str, - page_count: int = 0, - chunk_count: int = 0, - error_message: Optional[str] = None, -) -> dict: - """ - Mirror workflow state into Postgres as a local activity. - - Local activities avoid starving tiny state updates behind long-running OCR, - translation, or chunking activities on the shared task queue. - """ - if workflow.patched("local-state-update-v1"): - return await workflow.execute_local_activity( - update_document_state, - args=[workflow_id, stage, page_count, chunk_count, error_message], - start_to_close_timeout=timedelta(seconds=30), - retry_policy=STATE_UPDATE_RETRY, - ) - - return await workflow.execute_activity( - update_document_state, - args=[workflow_id, stage, page_count, chunk_count, error_message], - start_to_close_timeout=timedelta(seconds=30), - retry_policy=STATE_UPDATE_RETRY, - ) - - -@dataclass -class DocumentWorkflowState: - """Workflow state - queryable and modifiable via signals.""" - - document_id: str - filename: str - filepath: str - - stage: DocumentStage = DocumentStage.REGISTERED - error_message: Optional[str] = None - - page_count: int = 0 - chunk_count: int = 0 - translated_count: int = 0 - - ocr_approved: bool = False - translation_approved: bool = False - chunks_approved: bool = False - ingestion_approved: bool = False - prod_approved: bool = False - network_published: bool = False - - chunk_size: int = 450 - chunk_overlap: int = 128 - min_tokens: int = 100 - index_name: str = "documents-index" - stop_after_ocr: bool = False - - created_at: str = "" - ocr_completed_at: Optional[str] = None - translation_completed_at: Optional[str] = None - chunks_completed_at: Optional[str] = None - ingested_at: Optional[str] = None - promoted_to_prod_at: Optional[str] = None - - -@workflow.defn -class DocumentPipelineWorkflow: - """Main document processing workflow.""" - - def __init__(self): - self.state = None - - @workflow.run - async def run( - self, - document_id: str, - filename: str, - filepath: str, - chunk_size: int = 450, - chunk_overlap: int = 128, - min_tokens: int = 100, - index_name: str = "documents-index", - auto_approve: bool = False, - stop_after_ocr: bool = False, - skip_prod: bool = False, - ) -> dict: - self.state = DocumentWorkflowState( - document_id=document_id, - filename=filename, - filepath=filepath, - chunk_size=chunk_size, - chunk_overlap=chunk_overlap, - min_tokens=min_tokens, - index_name=index_name, - stop_after_ocr=stop_after_ocr, - created_at=_now_iso(), - ) - - try: - self.state.stage = DocumentStage.OCR_PROCESSING - workflow.logger.info(f"Starting OCR for {filename}") - - await _mirror_state(workflow.info().workflow_id, "ocr_processing", 0, 0, None) - - ocr_result = await workflow.execute_activity( - run_ocr_and_store, - args=[workflow.info().workflow_id, filepath], - start_to_close_timeout=timedelta(minutes=90), - heartbeat_timeout=timedelta(minutes=10), - retry_policy=OCR_RETRY, - ) - self.state.page_count = ocr_result.get("page_count", 0) - self.state.ocr_completed_at = _now_iso() - self.state.stage = DocumentStage.OCR_REVIEW - - await _mirror_state(workflow.info().workflow_id, "ocr_review", self.state.page_count, 0, None) - - if stop_after_ocr: - workflow.logger.info(f"OCR-only run complete for {filename}") - return { - "document_id": document_id, - "filename": filename, - "stage": self.state.stage.value, - "pages": self.state.page_count, - "chunks": 0, - "stop_after_ocr": True, - } - - if not auto_approve: - await workflow.wait_condition(lambda: self.state.ocr_approved) - - self.state.stage = DocumentStage.TRANSLATION_PROCESSING - await _mirror_state(workflow.info().workflow_id, "translation_processing", self.state.page_count, 0, None) - - translation_result = await workflow.execute_activity( - detect_and_translate_pages_from_db, - args=[workflow.info().workflow_id], - start_to_close_timeout=timedelta(minutes=90), - retry_policy=TRANSLATION_RETRY, - ) - self.state.page_count = translation_result.get("page_count", self.state.page_count) - self.state.translated_count = translation_result.get("translated_count", 0) - self.state.translation_completed_at = _now_iso() - self.state.stage = DocumentStage.TRANSLATION_REVIEW - - await _mirror_state(workflow.info().workflow_id, "translation_review", self.state.page_count, 0, None) - - if not auto_approve: - await workflow.wait_condition(lambda: self.state.translation_approved) - - self.state.stage = DocumentStage.CHUNKING - await _mirror_state(workflow.info().workflow_id, "chunking", self.state.page_count, 0, None) - - chunk_result = await workflow.execute_activity( - create_chunks_from_db, - args=[workflow.info().workflow_id, chunk_size, chunk_overlap, min_tokens], - start_to_close_timeout=timedelta(minutes=30), - retry_policy=CHUNK_RETRY, - ) - self.state.chunk_count = chunk_result.get("chunk_count", 0) - self.state.chunks_completed_at = _now_iso() - - if workflow.patched("auto-tag-v1"): - await workflow.execute_activity( - auto_tag_chunks_from_db, - args=[workflow.info().workflow_id, filename], - start_to_close_timeout=timedelta(minutes=45), - retry_policy=CHUNK_RETRY, - ) - - self.state.stage = DocumentStage.CHUNK_REVIEW - - await _mirror_state(workflow.info().workflow_id, "chunk_review", self.state.page_count, self.state.chunk_count, None) - - if not auto_approve: - await workflow.wait_condition(lambda: self.state.chunks_approved) - - self.state.stage = DocumentStage.READY_FOR_INGESTION - await _mirror_state(workflow.info().workflow_id, "ready_for_ingestion", self.state.page_count, self.state.chunk_count, None) - - if not auto_approve: - await workflow.wait_condition(lambda: self.state.ingestion_approved) - - self.state.stage = DocumentStage.INGESTING - await _mirror_state(workflow.info().workflow_id, "ingesting", self.state.page_count, self.state.chunk_count, None) - - result = await workflow.execute_activity( - ingest_document_from_db, - args=[workflow.info().workflow_id, document_id, filename, index_name], - start_to_close_timeout=timedelta(minutes=90), - retry_policy=INGEST_RETRY, - ) - - self.state.ingested_at = _now_iso() - - if not workflow.patched("publish-after-prod-v1"): - # Old order, preserved for in-flight workflows whose history - # already recorded this activity immediately after DEV ingest. - # New workflows publish only after a successful PROD - # promotion instead (see below). - self.state.stage = DocumentStage.PUBLISHING_TO_NETWORK - await _mirror_state( - workflow.info().workflow_id, - "publishing_to_network", - self.state.page_count, - self.state.chunk_count, - None, - ) - - # transactionId is generated once here (workflow.uuid4() is - # replay-safe) and reused across activity retries; the activity - # generates a fresh messageId on every attempt. - network_transaction_id = str(workflow.uuid4()) - await workflow.execute_activity( - publish_catalog_to_network, - args=[workflow.info().workflow_id, network_transaction_id], - start_to_close_timeout=timedelta(minutes=10), - retry_policy=NETWORK_PUBLISH_RETRY, - ) - self.state.network_published = True - - # DISABLE_PROD_SETTING: finish at DEV ingest. Decided when the - # workflow was started and passed in as an argument, so a replay - # always takes the same branch as the original run. - if skip_prod: - self.state.stage = DocumentStage.COMPLETED - await _mirror_state( - workflow.info().workflow_id, - "completed", - self.state.page_count, - self.state.chunk_count, - None, - ) - workflow.logger.info( - f"Pipeline complete for {filename} (PROD stages disabled)" - ) - return { - "document_id": document_id, - "filename": filename, - "stage": self.state.stage.value, - "pages": self.state.page_count, - "chunks": self.state.chunk_count, - "records_ingested": result.get("records_ingested", 0), - "records_promoted_to_prod": 0, - "prod_skipped": True, - } - - self.state.stage = DocumentStage.APPROVAL_FOR_PROD - await _mirror_state( - workflow.info().workflow_id, - "approval_for_prod", - self.state.page_count, - self.state.chunk_count, - None, - ) - - # Prod promotion is always an explicit superadmin action (even with auto_approve). - await workflow.wait_condition(lambda: self.state.prod_approved) - - self.state.stage = DocumentStage.INGESTING_PROD - await _mirror_state( - workflow.info().workflow_id, - "ingesting_prod", - self.state.page_count, - self.state.chunk_count, - None, - ) - - prod_result = await workflow.execute_activity( - promote_document_to_prod_qdrant, - args=[workflow.info().workflow_id, document_id, filename], - start_to_close_timeout=timedelta(minutes=90), - retry_policy=INGEST_RETRY, - ) - - self.state.promoted_to_prod_at = _now_iso() - - if not self.state.network_published: - # New order: publish to the network only now that PROD - # promotion has actually succeeded. Skipped for workflows - # that already published at the old position above. - self.state.stage = DocumentStage.PUBLISHING_TO_NETWORK - await _mirror_state( - workflow.info().workflow_id, - "publishing_to_network", - self.state.page_count, - self.state.chunk_count, - None, - ) - - network_transaction_id = str(workflow.uuid4()) - await workflow.execute_activity( - publish_catalog_to_network, - args=[workflow.info().workflow_id, network_transaction_id], - start_to_close_timeout=timedelta(minutes=10), - retry_policy=NETWORK_PUBLISH_RETRY, - ) - self.state.network_published = True - - self.state.stage = DocumentStage.COMPLETED - - await _mirror_state(workflow.info().workflow_id, "completed", self.state.page_count, self.state.chunk_count, None) - - workflow.logger.info(f"Pipeline complete for {filename}") - return { - "document_id": document_id, - "filename": filename, - "stage": self.state.stage.value, - "pages": self.state.page_count, - "chunks": self.state.chunk_count, - "records_ingested": result.get("records_ingested", 0), - "records_promoted_to_prod": prod_result.get("records_ingested", 0), - } - - except Exception as e: - self.state.stage = DocumentStage.FAILED - self.state.error_message = str(e) - workflow.logger.error(f"Pipeline failed: {e}") - - try: - await _mirror_state(workflow.info().workflow_id, "failed", self.state.page_count, self.state.chunk_count, str(e)) - except Exception: - pass - raise - - @workflow.query - def get_state(self) -> dict: - if not self.state: - return {} - return { - "document_id": self.state.document_id, - "filename": self.state.filename, - "filepath": self.state.filepath, - "stage": self.state.stage.value, - "error_message": self.state.error_message, - "page_count": self.state.page_count, - "chunk_count": self.state.chunk_count, - "translated_count": self.state.translated_count, - "ocr_approved": self.state.ocr_approved, - "translation_approved": self.state.translation_approved, - "chunks_approved": self.state.chunks_approved, - "ingestion_approved": self.state.ingestion_approved, - "prod_approved": self.state.prod_approved, - "created_at": self.state.created_at, - "ocr_completed_at": self.state.ocr_completed_at, - "translation_completed_at": self.state.translation_completed_at, - "chunks_completed_at": self.state.chunks_completed_at, - "ingested_at": self.state.ingested_at, - "promoted_to_prod_at": self.state.promoted_to_prod_at, - } - - @workflow.signal - def approve_ocr(self): - self.state.ocr_approved = True - - @workflow.signal - def approve_translation(self): - self.state.translation_approved = True - - @workflow.signal - def approve_chunks(self): - self.state.chunks_approved = True - - @workflow.signal - def approve_ingestion(self): - self.state.ingestion_approved = True - - @workflow.signal - def approve_prod(self): - self.state.prod_approved = True - - -@dataclass -class ReingestionWorkflowState: - document_id: str - filename: str - workflow_id: str - stage: DocumentStage = DocumentStage.READY_FOR_INGESTION - error_message: Optional[str] = None - chunk_count: int = 0 - records_ingested: int = 0 - - -@workflow.defn -class ReingestionWorkflow: - """Lightweight workflow for re-ingesting completed documents to the vector index.""" - - def __init__(self): - self.state = None - - @workflow.run - async def run( - self, - document_id: str, - filename: str, - original_workflow_id: str, - page_count: int = 0, - chunk_count: int = 0, - index_name: str = "documents-index", - ) -> dict: - self.state = ReingestionWorkflowState( - document_id=document_id, - filename=filename, - workflow_id=original_workflow_id, - chunk_count=chunk_count, - ) - - try: - await _mirror_state(original_workflow_id, "ingesting", page_count, chunk_count, None) - - self.state.stage = DocumentStage.INGESTING - result = await workflow.execute_activity( - ingest_document_from_db, - args=[original_workflow_id, document_id, filename, index_name], - start_to_close_timeout=timedelta(minutes=90), - retry_policy=INGEST_RETRY, - ) - - self.state.records_ingested = result.get("records_ingested", 0) - - if not workflow.patched("publish-after-prod-v1"): - # Old order, preserved for in-flight workflows whose history - # already recorded this activity here. New executions publish - # to the network only from PromoteToProdWorkflow, once this - # reingest's own PROD promotion actually succeeds - this - # workflow never performs that promotion itself. - self.state.stage = DocumentStage.PUBLISHING_TO_NETWORK - await _mirror_state(original_workflow_id, "publishing_to_network", page_count, chunk_count, None) - - # Content changed on reingest, so the network gets a fresh publish - # with a new transactionId (same replay-safety rule as above). - network_transaction_id = str(workflow.uuid4()) - await workflow.execute_activity( - publish_catalog_to_network, - args=[original_workflow_id, network_transaction_id], - start_to_close_timeout=timedelta(minutes=10), - retry_policy=NETWORK_PUBLISH_RETRY, - ) - - # Reingest lands in prod-approval gate; PromoteToProdWorkflow finishes promotion. - self.state.stage = DocumentStage.APPROVAL_FOR_PROD - - await _mirror_state(original_workflow_id, "approval_for_prod", page_count, chunk_count, None) - - return { - "document_id": document_id, - "filename": filename, - "stage": "approval_for_prod", - "chunks": chunk_count, - "records_ingested": self.state.records_ingested, - } - - except Exception as e: - self.state.stage = DocumentStage.FAILED - self.state.error_message = str(e) - try: - await _mirror_state(original_workflow_id, "failed", page_count, chunk_count, str(e)) - except Exception: - pass - raise - - @workflow.query - def get_state(self) -> dict: - if not self.state: - return {} - return { - "document_id": self.state.document_id, - "filename": self.state.filename, - "workflow_id": self.state.workflow_id, - "stage": self.state.stage.value, - "error_message": self.state.error_message, - "chunk_count": self.state.chunk_count, - "records_ingested": self.state.records_ingested, - } - - -@workflow.defn -class PromoteToProdWorkflow: - """Promote a DEV-ingested document into the PROD Qdrant collection.""" - - def __init__(self): - self.state = None - - @workflow.run - async def run( - self, - document_id: str, - filename: str, - original_workflow_id: str, - page_count: int = 0, - chunk_count: int = 0, - ) -> dict: - self.state = { - "document_id": document_id, - "filename": filename, - "workflow_id": original_workflow_id, - "stage": "ingesting_prod", - } - try: - await _mirror_state(original_workflow_id, "ingesting_prod", page_count, chunk_count, None) - result = await workflow.execute_activity( - promote_document_to_prod_qdrant, - args=[original_workflow_id, document_id, filename], - start_to_close_timeout=timedelta(minutes=90), - retry_policy=INGEST_RETRY, - ) - - if workflow.patched("publish-after-prod-v1"): - # New: this workflow now also announces the network catalog, - # since promotion just succeeded here (previously it never - # published at all - see DocumentPipelineWorkflow/ReingestionWorkflow - # for the equivalent post-promotion publish). - self.state["stage"] = "publishing_to_network" - await _mirror_state(original_workflow_id, "publishing_to_network", page_count, chunk_count, None) - - network_transaction_id = str(workflow.uuid4()) - await workflow.execute_activity( - publish_catalog_to_network, - args=[original_workflow_id, network_transaction_id], - start_to_close_timeout=timedelta(minutes=10), - retry_policy=NETWORK_PUBLISH_RETRY, - ) - - await _mirror_state(original_workflow_id, "completed", page_count, chunk_count, None) - self.state["stage"] = "completed" - return { - "document_id": document_id, - "filename": filename, - "stage": "completed", - "records_promoted_to_prod": result.get("records_ingested", 0), - } - except Exception as e: - self.state["stage"] = "failed" - self.state["error_message"] = str(e) - try: - await _mirror_state(original_workflow_id, "failed", page_count, chunk_count, str(e)) - except Exception: - pass - raise - - @workflow.query - def get_state(self) -> dict: - return self.state or {} - - -@dataclass -class TranslationOnlyWorkflowState: - workflow_id: str - document_id: str - filename: str - stage: DocumentStage = DocumentStage.OCR_REVIEW - error_message: Optional[str] = None - page_count: int = 0 - translated_count: int = 0 - translation_completed_at: Optional[str] = None - - -@workflow.defn -class TranslationOnlyWorkflow: - """Resume from OCR review, run translation only, and stop at translation review.""" - - def __init__(self): - self.state = None - - @workflow.run - async def run( - self, - original_workflow_id: str, - document_id: str, - filename: str, - ) -> dict: - self.state = TranslationOnlyWorkflowState( - workflow_id=original_workflow_id, - document_id=document_id, - filename=filename, - ) - - try: - await _mirror_state(original_workflow_id, "translation_processing", 0, 0, None) - self.state.stage = DocumentStage.TRANSLATION_PROCESSING - - translation_result = await workflow.execute_activity( - detect_and_translate_pages_from_db, - args=[original_workflow_id], - start_to_close_timeout=timedelta(minutes=90), - retry_policy=TRANSLATION_RETRY, - ) - self.state.page_count = translation_result.get("page_count", 0) - self.state.translated_count = translation_result.get("translated_count", 0) - self.state.translation_completed_at = _now_iso() - self.state.stage = DocumentStage.TRANSLATION_REVIEW - - await _mirror_state(original_workflow_id, "translation_review", self.state.page_count, 0, None) - - return { - "workflow_id": original_workflow_id, - "document_id": document_id, - "filename": filename, - "stage": "translation_review", - "page_count": self.state.page_count, - "translated_count": self.state.translated_count, - } - except Exception as e: - self.state.stage = DocumentStage.FAILED - self.state.error_message = str(e) - try: - await _mirror_state(original_workflow_id, "failed", self.state.page_count, 0, str(e)) - except Exception: - pass - raise - - @workflow.query - def get_state(self) -> dict: - if not self.state: - return {} - return { - "workflow_id": self.state.workflow_id, - "document_id": self.state.document_id, - "filename": self.state.filename, - "stage": self.state.stage.value, - "error_message": self.state.error_message, - "page_count": self.state.page_count, - "translated_count": self.state.translated_count, - "translation_completed_at": self.state.translation_completed_at, - } - - -@dataclass -class OcrOnlyWorkflowState: - workflow_id: str - document_id: str - filename: str - stage: DocumentStage = DocumentStage.REGISTERED - error_message: Optional[str] = None - page_count: int = 0 - ocr_completed_at: Optional[str] = None - - -@workflow.defn -class OcrOnlyWorkflow: - """Resume or retry OCR for an existing document and stop at OCR review.""" - - def __init__(self): - self.state = None - - @workflow.run - async def run( - self, - original_workflow_id: str, - document_id: str, - filename: str, - filepath: str, - ) -> dict: - self.state = OcrOnlyWorkflowState( - workflow_id=original_workflow_id, - document_id=document_id, - filename=filename, - ) - - try: - await _mirror_state(original_workflow_id, "ocr_processing", 0, 0, None) - self.state.stage = DocumentStage.OCR_PROCESSING - - ocr_result = await workflow.execute_activity( - run_ocr_and_store, - args=[original_workflow_id, filepath], - start_to_close_timeout=timedelta(minutes=90), - heartbeat_timeout=timedelta(minutes=10), - retry_policy=OCR_RETRY, - ) - self.state.page_count = ocr_result.get("page_count", 0) - self.state.ocr_completed_at = _now_iso() - self.state.stage = DocumentStage.OCR_REVIEW - - await _mirror_state(original_workflow_id, "ocr_review", self.state.page_count, 0, None) - - return { - "workflow_id": original_workflow_id, - "document_id": document_id, - "filename": filename, - "stage": "ocr_review", - "page_count": self.state.page_count, - } - except Exception as e: - self.state.stage = DocumentStage.FAILED - self.state.error_message = str(e) - try: - await _mirror_state(original_workflow_id, "failed", self.state.page_count, 0, str(e)) - except Exception: - pass - raise - - @workflow.query - def get_state(self) -> dict: - if not self.state: - return {} - return { - "workflow_id": self.state.workflow_id, - "document_id": self.state.document_id, - "filename": self.state.filename, - "stage": self.state.stage.value, - "error_message": self.state.error_message, - "page_count": self.state.page_count, - "ocr_completed_at": self.state.ocr_completed_at, - } - - -@dataclass -class ChunkingOnlyWorkflowState: - workflow_id: str - document_id: str - filename: str - stage: DocumentStage = DocumentStage.TRANSLATION_REVIEW - error_message: Optional[str] = None - page_count: int = 0 - chunk_count: int = 0 - chunks_completed_at: Optional[str] = None - - -@workflow.defn -class ChunkingOnlyWorkflow: - """Resume or retry chunking for an existing document and stop at chunk review.""" - - def __init__(self): - self.state = None - - @workflow.run - async def run( - self, - original_workflow_id: str, - document_id: str, - filename: str, - page_count: int = 0, - chunk_size: int = 450, - chunk_overlap: int = 128, - min_tokens: int = 100, - ) -> dict: - self.state = ChunkingOnlyWorkflowState( - workflow_id=original_workflow_id, - document_id=document_id, - filename=filename, - page_count=page_count, - ) - - try: - await _mirror_state(original_workflow_id, "chunking", page_count, 0, None) - self.state.stage = DocumentStage.CHUNKING - - chunk_result = await workflow.execute_activity( - create_chunks_from_db, - args=[original_workflow_id, chunk_size, chunk_overlap, min_tokens], - start_to_close_timeout=timedelta(minutes=30), - retry_policy=CHUNK_RETRY, - ) - self.state.chunk_count = chunk_result.get("chunk_count", 0) - self.state.chunks_completed_at = _now_iso() - - if workflow.patched("auto-tag-v1"): - await workflow.execute_activity( - auto_tag_chunks_from_db, - args=[original_workflow_id, filename], - start_to_close_timeout=timedelta(minutes=45), - retry_policy=CHUNK_RETRY, - ) - - self.state.stage = DocumentStage.CHUNK_REVIEW - - await _mirror_state(original_workflow_id, "chunk_review", page_count, self.state.chunk_count, None) - - return { - "workflow_id": original_workflow_id, - "document_id": document_id, - "filename": filename, - "stage": "chunk_review", - "chunk_count": self.state.chunk_count, - } - except Exception as e: - self.state.stage = DocumentStage.FAILED - self.state.error_message = str(e) - try: - await _mirror_state(original_workflow_id, "failed", page_count, self.state.chunk_count, str(e)) - except Exception: - pass - raise - - @workflow.query - def get_state(self) -> dict: - if not self.state: - return {} - return { - "workflow_id": self.state.workflow_id, - "document_id": self.state.document_id, - "filename": self.state.filename, - "stage": self.state.stage.value, - "error_message": self.state.error_message, - "page_count": self.state.page_count, - "chunk_count": self.state.chunk_count, - "chunks_completed_at": self.state.chunks_completed_at, - } - - -@workflow.defn -class AdvisoryIndexWorkflow: - """Index one webhook advisory record into the dev Qdrant collection.""" - - def __init__(self): - self.state = None - - @workflow.run - async def run(self, record_id: str) -> dict: - self.state = {"record_id": record_id, "stage": "indexing"} - try: - result = await workflow.execute_activity( - index_webhook_advisory, - record_id, - start_to_close_timeout=timedelta(minutes=30), - retry_policy=INGEST_RETRY, - ) - except Exception as e: - self.state.update(stage="failed", error_message=str(e)) - raise - self.state.update(stage="completed", status=result.get("status"), chunks=result.get("chunks")) - return result - - @workflow.query - def get_state(self) -> dict: - return self.state or {} - - -@workflow.defn -class AdvisoryPromoteWorkflow: - """Promote one webhook advisory record into the PROD Qdrant collection.""" - - def __init__(self): - self.state = None - - @workflow.run - async def run(self, record_id: str) -> dict: - self.state = {"record_id": record_id, "stage": "promoting"} - try: - result = await workflow.execute_activity( - promote_webhook_advisory, - record_id, - start_to_close_timeout=timedelta(minutes=30), - retry_policy=INGEST_RETRY, - ) - except Exception as e: - self.state.update(stage="failed", error_message=str(e)) - raise - self.state.update(stage="completed", status=result.get("status"), chunks=result.get("chunks")) - return result - - @workflow.query - def get_state(self) -> dict: - return self.state or {} diff --git a/pyproject.toml b/pyproject.toml index 1d4cee7..62ae2ad 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,27 +1,22 @@ [project] name = "knowledge-provider" version = "0.1.0" -description = "Bharat Vistaar Docs Pipeline: review-driven document ingestion pipeline" +description = "OpenAgriNet Knowledge Provider for document ingestion, retrieval, and network publishing" requires-python = ">=3.10" dependencies = [ # Temporal workflow engine "temporalio>=1.0.0", - # FastAPI for REST API "fastapi>=0.100.0", "uvicorn>=0.20.0", - # Chandra OCR 2 (remote vLLM/HF client; no local GPU/torch required in worker image) "chandra-ocr>=0.2.0", - # pdf-inspector: fast local PDF classification + Markdown extraction for # text-based PDFs, so we can skip the OCR provider entirely for those pages. "pdf-inspector>=1.14.0", - # Text processing "tiktoken>=0.5.0", "langchain-text-splitters>=0.0.1", - # Vector search "qdrant-client>=1.12.0", # Local embeddings (default EMBEDDING_PROVIDER=sentence_transformers). @@ -31,18 +26,14 @@ dependencies = [ "transformers==4.56.1", "huggingface-hub==0.35.3", "torch", - # Object storage "minio>=7.0.0", "google-cloud-storage>=3,<4", - # HTTP client "httpx>=0.24.0", - # In-process language detection (replaces the lang-detect HTTP service for # shared-script disambiguation, e.g. Devanagari -> hi/mr) "pyfranc>=0.2.3", - # Utilities "pydantic>=2.0.0", "python-multipart>=0.0.6", @@ -50,19 +41,20 @@ dependencies = [ "openpyxl>=3.1.0", "pypdf>=5.0.0", "PyMuPDF>=1.27.0", - # Postgres driver (sync) + connection pool "psycopg[binary,pool]>=3.2", - # Rate limiting "slowapi>=0.1.9", - # Auth (Keycloak JWT validation) "PyJWT[crypto]>=2.8.0", - "opentelemetry-sdk>=1.44.0", "opentelemetry-exporter-otlp-proto-http>=1.44.0", "opentelemetry-instrumentation-logging>=0.65b0", + "jsonschema>=4.23,<5", + "pyyaml>=6,<7", + "referencing>=0.35,<1", + "jsonata-python==0.7.0", + "google-re2==1.1.20251105", ] [dependency-groups] diff --git a/scripts/backfill_provider_id.py b/scripts/backfill_provider_id.py index bf4327f..21611cc 100644 --- a/scripts/backfill_provider_id.py +++ b/scripts/backfill_provider_id.py @@ -13,8 +13,8 @@ from pipeline import db # noqa: E402 from pipeline.auth.tenancy import group_for_instance # noqa: E402 -from pipeline.network_group import load_provider_naming, named_group, provider_for # noqa: E402 -from pipeline.network_provider import InvalidProviderIdError, ProviderNaming # noqa: E402 +from pipeline.network.network_group import load_provider_naming, named_group, provider_for # noqa: E402 +from pipeline.network.network_provider import InvalidProviderIdError, ProviderNaming # noqa: E402 from pipeline.vector_store import get_default_index_name # noqa: E402 from pipeline.vector_store.qdrant_store import get_qdrant_client # noqa: E402 @@ -41,7 +41,7 @@ def _env(name: str) -> str: def dev_collections() -> list[str]: - indexed = (row["index_name"] for row in db.list_index_summaries(include_demo=True, include_disabled=True)) + indexed = (row["index_name"] for row in db.list_index_summaries(include_disabled=True)) return list(dict.fromkeys(name for name in (get_default_index_name(), *indexed) if name)) @@ -64,7 +64,7 @@ def targets_from_env() -> list[Target]: def all_documents() -> Iterable[dict]: offset = 0 while True: - page = db.list_documents(limit=PAGE_SIZE, offset=offset, include_demo=True, include_disabled=True) + page = db.list_documents(limit=PAGE_SIZE, offset=offset, include_disabled=True) yield from page if len(page) < PAGE_SIZE: return @@ -74,9 +74,13 @@ def all_documents() -> Iterable[dict]: def plan(documents: Iterable[dict], naming: ProviderNaming) -> list[PlannedDocument]: planned = [] for doc in documents: + kind = doc.get("kind") + if not kind: + planned.append(PlannedDocument(doc["workflow_id"], None, "Document kind is missing")) + continue group = named_group(group_for_instance(doc.get("instance"))) try: - provider = provider_for(naming, doc.get("document_kind"), group) + provider = provider_for(naming, kind, group) except InvalidProviderIdError as exc: planned.append(PlannedDocument(doc["workflow_id"], None, str(exc))) continue @@ -86,7 +90,11 @@ def plan(documents: Iterable[dict], naming: ProviderNaming) -> list[PlannedDocum def _workflow_filter(workflow_id: str) -> qmodels.Filter: return qmodels.Filter( - must=[qmodels.FieldCondition(key="workflow_id", match=qmodels.MatchValue(value=workflow_id))] + must=[ + qmodels.FieldCondition(key="workflow_id", match=qmodels.MatchValue(value=workflow_id)), + qmodels.IsEmptyCondition(is_empty=qmodels.PayloadField(key="revision_id")), + qmodels.IsEmptyCondition(is_empty=qmodels.PayloadField(key="release_network_id")), + ], ) diff --git a/scripts/install_schema_pack.py b/scripts/install_schema_pack.py new file mode 100644 index 0000000..58c6176 --- /dev/null +++ b/scripts/install_schema_pack.py @@ -0,0 +1,33 @@ +"""Check an approved local schema bundle and optionally register its digest.""" + +import argparse +from pathlib import Path + +from pipeline.metadata.schema_packs import install_schema_pack, load_schema_pack + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("manifest", type=Path) + parser.add_argument("--install-for-network", metavar="NETWORK_ID") + parser.add_argument("--verify-registry", action="store_true") + args = parser.parse_args() + if args.verify_registry and not args.install_for_network: + parser.error("--verify-registry requires --install-for-network") + pack = load_schema_pack(args.manifest) + if args.verify_registry: + from pipeline.network.network_registry import registry_for_network + + registration = registry_for_network(args.install_for_network).active_schema(pack.capability) + if (registration.version, registration.artifact_url) != (pack.version, pack.entrypoint): + parser.error("The approved bundle does not match the active Registry registration") + if args.install_for_network: + from pipeline.db import get_connection + + with get_connection() as conn: + install_schema_pack(conn, args.install_for_network, pack) + print(pack.digest) + + +if __name__ == "__main__": + main() diff --git a/scripts/keycloak_bootstrap_docs_pipeline.py b/scripts/keycloak_bootstrap_docs_pipeline.py index 184f3b5..b9f0f55 100644 --- a/scripts/keycloak_bootstrap_docs_pipeline.py +++ b/scripts/keycloak_bootstrap_docs_pipeline.py @@ -1,8 +1,7 @@ #!/usr/bin/env python3 -"""Ensure docs-pipeline Keycloak clients/roles/claims and a test curator exist. +"""Ensure docs-pipeline Keycloak clients and claims with a service account. -Safe to re-run. Expects Keycloak already imported with realm docs-pipeline. -Does not print passwords. +Optionally invite a test operator to enroll a passkey. """ from __future__ import annotations @@ -10,7 +9,6 @@ import argparse import json import os -import secrets import sys import urllib.error import urllib.parse @@ -45,41 +43,42 @@ def main() -> int: parser = argparse.ArgumentParser() parser.add_argument("--base-url", default=os.environ.get("KEYCLOAK_BASE_URL", "http://127.0.0.1:8082/auth")) parser.add_argument("--realm", default="docs-pipeline") - parser.add_argument("--admin-user", default=os.environ.get("KEYCLOAK_ADMIN", "admin")) - parser.add_argument("--admin-password", default=os.environ.get("KEYCLOAK_ADMIN_PASSWORD", "")) + parser.add_argument("--redirect-uri", default="http://localhost:3000/auth/sso-callback") parser.add_argument("--test-username", default="docs-test-curator") - parser.add_argument("--test-password", default=os.environ.get("KEYCLOAK_TEST_USER_PASSWORD", "")) + parser.add_argument("--test-email", help="Create a test operator and send a passkey enrollment link") + parser.add_argument("--test-group", help="Optional existing Keycloak group path for the test operator") args = parser.parse_args() - if not args.admin_password: - print("KEYCLOAK_ADMIN_PASSWORD / --admin-password required", file=sys.stderr) + admin_client = os.environ.get("KEYCLOAK_ADMIN_CLIENT_ID") + admin_secret = os.environ.get("KEYCLOAK_ADMIN_CLIENT_SECRET") + if not admin_client or not admin_secret: + print("KEYCLOAK_ADMIN_CLIENT_ID and KEYCLOAK_ADMIN_CLIENT_SECRET are required", file=sys.stderr) return 2 base = args.base_url.rstrip("/") + token_realm = os.environ.get("KEYCLOAK_ADMIN_TOKEN_REALM") or args.realm _, token_body = _req( "POST", - f"{base}/realms/master/protocol/openid-connect/token", + f"{base}/realms/{token_realm}/protocol/openid-connect/token", form={ - "grant_type": "password", - "client_id": "admin-cli", - "username": args.admin_user, - "password": args.admin_password, + "grant_type": "client_credentials", + "client_id": admin_client, + "client_secret": admin_secret, }, ) token = token_body["access_token"] admin = f"{base}/admin/realms/{args.realm}" - for role in ("master_admin", "admin", "content_curator", "viewer"): - _req("POST", f"{admin}/roles", token=token, body={"name": role}) - def ensure_client(client_id: str, rep: dict) -> str: _, found = _req("GET", f"{admin}/clients?{urllib.parse.urlencode({'clientId': client_id})}", token=token) if found: + if rep.get("standardFlowEnabled") and (found[0].get("publicClient") or found[0].get("directAccessGrantsEnabled")): + raise RuntimeError("Configure the existing console client for confidential PKCE sign-in before bootstrapping") return found[0]["id"] _req("POST", f"{admin}/clients", token=token, body=rep) _, found = _req("GET", f"{admin}/clients?{urllib.parse.urlencode({'clientId': client_id})}", token=token) return found[0]["id"] - api_id = ensure_client( + ensure_client( "docs-pipeline-api", { "clientId": "docs-pipeline-api", @@ -95,26 +94,14 @@ def ensure_client(client_id: str, rep: dict) -> str: "clientId": "docs-pipeline-ui", "enabled": True, "protocol": "openid-connect", - "publicClient": True, + "publicClient": False, "standardFlowEnabled": True, "directAccessGrantsEnabled": False, - "redirectUris": ["http://localhost:*/*", "https://search-ui.example.com/*"], - "webOrigins": ["http://localhost:*", "https://search-ui.example.com"], + "implicitFlowEnabled": False, + "redirectUris": [args.redirect_uri], "attributes": {"pkce.code.challenge.method": "S256"}, }, ) - test_id = ensure_client( - "docs-pipeline-test-cli", - { - "clientId": "docs-pipeline-test-cli", - "enabled": True, - "protocol": "openid-connect", - "publicClient": True, - "standardFlowEnabled": False, - "directAccessGrantsEnabled": True, - }, - ) - _ = api_id mappers = [ { @@ -128,97 +115,47 @@ def ensure_client(client_id: str, rep: dict) -> str: }, }, { - "name": "instances-claim", + "name": "groups", "protocol": "openid-connect", - "protocolMapper": "oidc-usermodel-attribute-mapper", + "protocolMapper": "oidc-group-membership-mapper", "config": { - "user.attribute": "instances", - "claim.name": "instances", - "jsonType.label": "String", - "multivalued": "true", + "claim.name": "groups", + "full.path": "true", "access.token.claim": "true", - "id.token.claim": "true", - "userinfo.token.claim": "true", - }, - }, - { - "name": "envs-claim", - "protocol": "openid-connect", - "protocolMapper": "oidc-usermodel-attribute-mapper", - "config": { - "user.attribute": "envs", - "claim.name": "envs", - "jsonType.label": "String", - "multivalued": "true", - "access.token.claim": "true", - "id.token.claim": "true", - "userinfo.token.claim": "true", }, }, ] - for client_uuid in (ui_id, test_id): - _, existing = _req("GET", f"{admin}/clients/{client_uuid}/protocol-mappers/models", token=token) - names = {item.get("name") for item in (existing or [])} - for mapper in mappers: - if mapper["name"] not in names: - _req("POST", f"{admin}/clients/{client_uuid}/protocol-mappers/models", token=token, body=mapper) + mappers_url = f"{admin}/clients/{ui_id}/protocol-mappers/models" + _, existing = _req("GET", mappers_url, token=token) + names = {item.get("name") for item in (existing or [])} + for mapper in mappers: + if mapper["name"] not in names: + _req("POST", mappers_url, token=token, body=mapper) + if args.test_email: + invite_operator(admin, token, args.test_username, args.test_email, args.test_group) + print(f"bootstrap=ok realm={args.realm}") + return 0 - test_password = args.test_password or secrets.token_urlsafe(24) - _, users = _req( - "GET", - f"{admin}/users?{urllib.parse.urlencode({'username': args.test_username, 'exact': 'true'})}", - token=token, - ) - user_body = { - "username": args.test_username, - "enabled": True, - "attributes": {"instances": ["tenant-a"], "envs": ["dev"]}, - } - if users: - uid = users[0]["id"] - _req("PUT", f"{admin}/users/{uid}", token=token, body=user_body) - else: - _req("POST", f"{admin}/users", token=token, body=user_body) - _, users = _req( - "GET", - f"{admin}/users?{urllib.parse.urlencode({'username': args.test_username, 'exact': 'true'})}", - token=token, - ) - uid = users[0]["id"] - _req( - "PUT", - f"{admin}/users/{uid}/reset-password", - token=token, - body={"type": "password", "temporary": False, "value": test_password}, - ) - _, role = _req("GET", f"{admin}/roles/content_curator", token=token) - _req("POST", f"{admin}/users/{uid}/role-mappings/realm", token=token, body=[role]) - # Smoke: password grant - _, tok = _req( - "POST", - f"{base}/realms/{args.realm}/protocol/openid-connect/token", - form={ - "grant_type": "password", - "client_id": "docs-pipeline-test-cli", - "username": args.test_username, - "password": test_password, - }, - ) - print("bootstrap=ok") - print(f"realm={args.realm}") - print(f"test_user={args.test_username}") - print(f"token_acquired={bool(tok.get('access_token'))}") - if not args.test_password: - print("test_password_generated=yes (set KEYCLOAK_TEST_USER_PASSWORD to reuse)") - # Write only if caller opts in via env path - out = os.environ.get("KEYCLOAK_TEST_PASSWORD_FILE") - if out: - with open(out, "w", encoding="utf-8") as handle: - handle.write(test_password) - os.chmod(out, 0o600) - print(f"test_password_file={out}") - return 0 +def invite_operator(admin: str, token: str, username: str, email: str, group_path: str | None) -> None: + group = None + if group_path: + path = urllib.parse.quote(group_path.strip("/"), safe="/") + _, group = _req("GET", f"{admin}/group-by-path/{path}", token=token) + users_url = f"{admin}/users?{urllib.parse.urlencode({'username': username, 'exact': 'true'})}" + _, users = _req("GET", users_url, token=token) + if users and (users[0].get("email") or "").casefold() != email.casefold(): + raise ValueError("The test username already belongs to a different email") + if not users: + _req("POST", f"{admin}/users", token=token, body={ + "username": username, "email": email, "enabled": True, + "firstName": "Test", "lastName": "Operator", + }) + _, users = _req("GET", users_url, token=token) + user_url = f"{admin}/users/{users[0]['id']}" + if group: + _req("PUT", f"{user_url}/groups/{group['id']}", token=token) + _req("PUT", f"{user_url}/execute-actions-email", token=token, body=["webauthn-register-passwordless"]) if __name__ == "__main__": diff --git a/scripts/keycloak_bootstrap_tenant_groups.py b/scripts/keycloak_bootstrap_tenant_groups.py index 980f851..785138d 100755 --- a/scripts/keycloak_bootstrap_tenant_groups.py +++ b/scripts/keycloak_bootstrap_tenant_groups.py @@ -2,9 +2,9 @@ """Idempotently configure Keycloak multi-state tenant groups + roles + mappers. Creates: - - Realm roles: super_admin, bh_viewer, state_admin, state_approver, + - Realm roles: super_admin, global_viewer, state_admin, state_approver, state_contributor, state_view - - Groups: /global/super-admin, /global/bh-viewer, + - Groups: /global/super-admin, /global/global-viewer, /states/{CODE}/{admin|approver|contributor|view} - Group → realm role mappings on leaf groups - Group Membership mapper (claim ``groups``, full path) on SPA clients @@ -70,7 +70,7 @@ PRODUCT_ROLES = ( ("super_admin", "Platform super admin — all states, full access"), - ("bh_viewer", "Bharat Vistaar viewer — all states, read-only"), + ("global_viewer", "Viewer with read-only access to all providers"), ("state_admin", "State admin — upload, edit, review & approve, pipeline, delete own"), ("state_approver", "State approver — upload, edit, review & approve, pipeline"), ("state_contributor", "State contributor — upload and run pipeline only"), @@ -80,7 +80,7 @@ # Global leaf group name → realm role. GLOBAL_LEAVES = ( ("super-admin", "super_admin"), - ("bh-viewer", "bh_viewer"), + ("global-viewer", "global_viewer"), ) # Per-state leaf group name → realm role. diff --git a/scripts/keycloak_fix_email_otp_bindings.py b/scripts/keycloak_fix_email_otp_bindings.py deleted file mode 100755 index c47a668..0000000 --- a/scripts/keycloak_fix_email_otp_bindings.py +++ /dev/null @@ -1,188 +0,0 @@ -#!/usr/bin/env python3 -"""Repair the ``bharat-vistaar`` client's authentication-flow bindings. - -Two things are wrong on the realm as shipped: - -1. ``email-otp-direct-grant`` is bound to the client's **browser** slot. A - direct-grant flow cannot serve a browser authorize request, so - ``/protocol/openid-connect/auth?client_id=bharat-vistaar`` answers - ``401 {"error":"invalid_request","error_description":"Missing parameter: username"}`` - and SSO on that client is dead. -2. Nothing is bound to the **direct_grant** slot, so the realm's built-in - password flow runs instead. ``grant_type=password`` + ``otp=`` is - therefore checked as a *password*, and every real OTP comes back - ``invalid_grant / Invalid user credentials``. - -Fix: clear the browser override (fall back to the realm browser flow, which is -what ``docs-pipeline-ui`` already uses successfully) and bind -``email-otp-direct-grant`` to ``direct_grant``. - -Idempotent — re-running when already correct changes nothing. - - python scripts/keycloak_fix_email_otp_bindings.py [--dry-run] - -Reads KEYCLOAK_ADMIN_* from the repo-root .env. -""" - -from __future__ import annotations - -import argparse -import json -import os -import sys -import urllib.parse -import urllib.request -from pathlib import Path - -OTP_FLOW_ALIAS = "email-otp-direct-grant" - - -def _load_env() -> None: - try: - from dotenv import load_dotenv - except ImportError: - return - env = Path(__file__).resolve().parents[1] / ".env" - if env.is_file(): - load_dotenv(env, override=False) - - -def _request(method: str, url: str, token: str | None = None, body: dict | None = None): - headers = {"Accept": "application/json"} - data = None - if body is not None: - data = json.dumps(body).encode() - headers["Content-Type"] = "application/json" - if token: - headers["Authorization"] = f"Bearer {token}" - req = urllib.request.Request(url, data=data, method=method, headers=headers) - with urllib.request.urlopen(req, timeout=30) as resp: - raw = resp.read() - return json.loads(raw.decode()) if raw else None - - -def _admin_token(base: str, token_realm: str, user: str, password: str) -> str: - form = urllib.parse.urlencode( - { - "grant_type": "password", - "client_id": "admin-cli", - "username": user, - "password": password, - } - ).encode() - req = urllib.request.Request( - f"{base}/realms/{token_realm}/protocol/openid-connect/token", - data=form, - method="POST", - headers={"Content-Type": "application/x-www-form-urlencoded"}, - ) - with urllib.request.urlopen(req, timeout=30) as resp: - return json.loads(resp.read().decode())["access_token"] - - -def main() -> int: - _load_env() - parser = argparse.ArgumentParser() - parser.add_argument("--base-url", default=os.environ.get("KEYCLOAK_ADMIN_BASE_URL")) - parser.add_argument("--realm", default=os.environ.get("KEYCLOAK_ADMIN_REALM", "bharat-vistaar")) - parser.add_argument( - "--token-realm", default=os.environ.get("KEYCLOAK_ADMIN_TOKEN_REALM", "master") - ) - parser.add_argument("--admin-username", default=os.environ.get("KEYCLOAK_ADMIN_USERNAME")) - parser.add_argument("--admin-password", default=os.environ.get("KEYCLOAK_ADMIN_PASSWORD")) - parser.add_argument( - "--client-id", - default=os.environ.get("KEYCLOAK_CLIENT_ID", "bharat-vistaar"), - help="clientId of the confidential client used for email-OTP login", - ) - parser.add_argument("--dry-run", action="store_true") - args = parser.parse_args() - - missing = [ - n - for n, v in ( - ("--base-url / KEYCLOAK_ADMIN_BASE_URL", args.base_url), - ("--admin-username / KEYCLOAK_ADMIN_USERNAME", args.admin_username), - ("--admin-password / KEYCLOAK_ADMIN_PASSWORD", args.admin_password), - ) - if not v - ] - if missing: - print("Missing: " + ", ".join(missing), file=sys.stderr) - return 2 - - base = args.base_url.rstrip("/") - token = _admin_token(base, args.token_realm, args.admin_username, args.admin_password) - admin = f"{base}/admin/realms/{args.realm}" - - flows = _request("GET", f"{admin}/authentication/flows", token) - flow_id = next((f["id"] for f in flows if f["alias"] == OTP_FLOW_ALIAS), None) - if not flow_id: - print( - f"Realm '{args.realm}' has no '{OTP_FLOW_ALIAS}' flow. Install the email-otp\n" - "provider and create the flow before running this.", - file=sys.stderr, - ) - return 1 - - found = _request( - "GET", f"{admin}/clients?clientId={urllib.parse.quote(args.client_id)}", token - ) - if not found: - print(f"Client '{args.client_id}' not found in realm '{args.realm}'.", file=sys.stderr) - return 1 - client = found[0] - - current = dict(client.get("authenticationFlowBindingOverrides") or {}) - desired = {k: v for k, v in current.items() if k not in ("browser", "direct_grant")} - # Explicit null, not omission: Keycloak MERGES this map on PUT, so a key left - # out keeps its old value. Only an explicit null actually clears the binding. - # Browser must end up unbound — the realm's own browser flow is what serves - # Google SSO (proven by docs-pipeline-ui, which has no override). - desired["browser"] = None - desired["direct_grant"] = flow_id - - changes = [] - if current.get("browser"): - changes.append(f"browser: {current['browser']} -> (realm default)") - if current.get("direct_grant") != flow_id: - changes.append( - f"direct_grant: {current.get('direct_grant') or '(realm default)'} -> {OTP_FLOW_ALIAS}" - ) - if not client.get("directAccessGrantsEnabled"): - changes.append("directAccessGrantsEnabled: false -> true") - if client.get("publicClient"): - changes.append( - "publicClient is true — email-OTP needs a CONFIDENTIAL client " - "(turn Client authentication ON); not changed automatically" - ) - - print(f"client : {args.client_id} ({client['id']})") - print(f"realm : {args.realm}") - print(f"otp flow : {OTP_FLOW_ALIAS} ({flow_id})") - if not changes: - print("\nAlready correct — nothing to do.") - return 0 - print("\nChanges:") - for c in changes: - print(" -", c) - - if args.dry_run: - print("\n--dry-run: nothing written.") - return 0 - - _request( - "PUT", - f"{admin}/clients/{client['id']}", - token, - body={ - "authenticationFlowBindingOverrides": desired, - "directAccessGrantsEnabled": True, - }, - ) - print("\nApplied.") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/scripts/keycloak_purge_legacy_roles.py b/scripts/keycloak_purge_legacy_roles.py index a15c35c..7dee95d 100644 --- a/scripts/keycloak_purge_legacy_roles.py +++ b/scripts/keycloak_purge_legacy_roles.py @@ -54,7 +54,8 @@ # Roles that must only ever reach a token through a group. GROUPED_ROLES = ( "super_admin", - "bh_viewer", + "global_viewer", + "bh_viewer", # Existing realms can still use the legacy name. "state_admin", "state_approver", "state_contributor", @@ -113,7 +114,7 @@ def walk(nodes): def _is_product_group(path: str) -> bool: """True when a group path grants a product role in the six-role model.""" clean = (path or "").rstrip("/") - if clean in ("/global/super-admin", "/global/bh-viewer"): + if clean in ("/global/super-admin", "/global/global-viewer", "/global/bh-viewer"): return True parts = [p for p in clean.split("/") if p] return len(parts) == 3 and parts[0] == "states" diff --git a/scripts/list_failed_workflows.py b/scripts/list_failed_workflows.py index 75bbb12..3d0a8f4 100755 --- a/scripts/list_failed_workflows.py +++ b/scripts/list_failed_workflows.py @@ -75,7 +75,7 @@ async def get_temporal_error_details(workflow_id: str, client: Client) -> dict: # Also try to get error from workflow state query (fallback) if not result["temporal_error_message"]: try: - from pipeline.workflows import DocumentPipelineWorkflow + from pipeline.temporal.workflows import DocumentPipelineWorkflow state = await handle.query(DocumentPipelineWorkflow.get_state) if state and state.get("error_message"): result["temporal_error_message"] = state.get("error_message") diff --git a/scripts/migrate_sqlite_to_postgres.py b/scripts/migrate_sqlite_to_postgres.py deleted file mode 100644 index 366e54b..0000000 --- a/scripts/migrate_sqlite_to_postgres.py +++ /dev/null @@ -1,926 +0,0 @@ -#!/usr/bin/env python3 -"""One-off migration of the legacy SQLite store into Postgres. - -Run in the maintenance window, after the Postgres migrations are applied and -with the api and worker stopped. The SQLite file is opened read-only and is -never modified, so rolling back is redeploying the previous images on it. - - python scripts/migrate_sqlite_to_postgres.py --dry-run - python scripts/migrate_sqlite_to_postgres.py - -Every migrated row gets a deterministic UUID derived from its legacy key, so a -second run finds the rows already there and changes nothing. A document is -migrated in one transaction. ``--dry-run`` runs everything and rolls it back. - -Mapping (Schema v2): one legacy ``documents`` row becomes one document plus -its main workflow (``temporal_workflow_id`` = the legacy ``workflow_id``); each -``document_jobs`` row becomes a workflow. The legacy ``instance`` becomes a -``instances`` row (``bv`` is ``/global``, others ``/states/``, Keycloak id -filled in by the first login sync) and the instance's ``catalogs`` row for the -document's kind. ``scheme_code`` / ``scheme_name`` / ``scheme_aliases_json`` -(or the ``scheme_catalog_entries`` registry entry for the code) become the -document's ``code`` / ``name`` / ``aliases``; the validity period, uploader and -vector index state move into ``documents.metadata``. Not migrated, and listed in -the report: documents of kind ``document`` (no catalog), demo documents -(``is_demo``), registry entries no document uses (bootstrap-only), and -``document_manifest_entries``. -""" - -import argparse -import json -import os -import re -import sqlite3 -import sys -import uuid -from collections import Counter -from datetime import date, datetime, timezone -from pathlib import Path -from typing import Any, Optional - -sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) - -import psycopg # noqa: E402 -from psycopg.rows import dict_row # noqa: E402 -from psycopg.types.json import Jsonb # noqa: E402 - -from pipeline import db, migrate # noqa: E402 -from pipeline.models import ( # noqa: E402 - ChunkingRun, - DocumentMetadata, - DocumentSource, - IndexState, - PageMetadata, - PageTranslation, -) -from pipeline.network_constants import KIND_TO_CATALOG_TYPE, catalog_type_for_kind # noqa: E402 - -_NAMESPACE = uuid.UUID("3f2b6c0e-5d1a-4a53-9a77-0c6a5c1d7e10") -_FINGERPRINT_RE = re.compile(r"^[0-9a-f]{32}$") -_CODE_RE = re.compile(r"^[a-z0-9]+(-[a-z0-9]+)*$") -_STAGES = { - "registered", "ocr_processing", "ocr_review", "translation_processing", - "translation_review", "chunking", "chunk_review", "ready_for_ingestion", - "ingesting", "publishing_to_network", "approval_for_prod", "ingesting_prod", - "completed", "failed", -} -_WORKFLOW_TYPES = { - "pipeline", "ocr_only", "ocr_retry", "translation_retry", "chunking_retry", - "reingest", "promote_prod", -} -_WORKFLOW_STATUSES = {"running", "waiting_review", "completed", "failed"} -_REVIEW_STAGES = { - "ocr_review", "translation_review", "chunk_review", "ready_for_ingestion", "approval_for_prod", -} -_ENTITY_TYPES = {"document", "page", "chunk", "setting", "catalog"} - -_LEGACY_SETTING_KEYS = { - "search_method": "searchMethod", - "search_limit": "limit", - "search_alpha": "alpha", - "search_ranking_method": "rankingMethod", - "search_show_highlights": "showHighlights", - "search_ef_search": "efSearch", - "search_index_name": "indexName", - "search_candidate_cap": "candidateCap", - "search_candidate_multiplier": "candidateMultiplier", - "search_max_chunks_per_doc": "maxChunksPerDoc", - "search_use_e5_prefix": "useE5Prefix", - "search_exclude_reference": "excludeReference", - "search_query_expansion_profile": "queryExpansionProfile", - "search_rerank_mode": "rerankMode", - "search_hybrid_rrfk": "hybridRrfK", -} -_BOOL_SETTINGS = {"showHighlights", "useE5Prefix", "excludeReference"} -_INT_SETTINGS = { - "limit", "efSearch", "candidateCap", "candidateMultiplier", "maxChunksPerDoc", "hybridRrfK", -} -_FLOAT_SETTINGS = {"alpha"} - - -class Report: - """Per-table counts: rows read from SQLite and what became of them.""" - - def __init__(self) -> None: - self.tables: dict[str, Counter] = {} - self.skip_reasons: list[dict[str, Any]] = [] - self.notes: list[str] = [] - - def count(self, table: str, outcome: str, n: int = 1) -> None: - self.tables.setdefault(table, Counter())[outcome] += n - - def skip(self, table: str, key: str, reason: str) -> None: - self.count(table, "skipped") - self.skip_reasons.append({"table": table, "key": key, "reason": reason}) - - def mismatches(self) -> list[str]: - problems = [] - for table, counts in sorted(self.tables.items()): - accounted = counts["inserted"] + counts["existing"] + counts["skipped"] + counts["dropped"] - if counts["read"] != accounted: - problems.append(f"{table}: read {counts['read']} but accounted for {accounted}") - return problems - - def as_dict(self) -> dict[str, Any]: - return { - "tables": {t: dict(c) for t, c in sorted(self.tables.items())}, - "skipped": self.skip_reasons, - "notes": self.notes, - "mismatches": self.mismatches(), - } - - def render(self) -> str: - lines = [f"{'table':<28}{'read':>8}{'inserted':>10}{'existing':>10}{'skipped':>9}{'dropped':>9}"] - for table, c in sorted(self.tables.items()): - lines.append( - f"{table:<28}{c['read']:>8}{c['inserted']:>10}{c['existing']:>10}" - f"{c['skipped']:>9}{c['dropped']:>9}" - ) - for item in self.skip_reasons: - lines.append(f" skipped {item['table']} {item['key']}: {item['reason']}") - lines.extend(f" note: {n}" for n in self.notes) - problems = self.mismatches() - lines.append("verification: OK" if not problems else "verification: FAILED") - lines.extend(f" {p}" for p in problems) - return "\n".join(lines) - - -def det_id(kind: str, key: str) -> str: - return str(uuid.uuid5(_NAMESPACE, f"{kind}:{key}")) - - -def parse_ts(value: Any) -> Optional[datetime]: - if not value: - return None - try: - parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00")) - except ValueError: - return None - if parsed.tzinfo is None: - parsed = parsed.replace(tzinfo=timezone.utc) - return parsed.astimezone(timezone.utc) - - -def parse_date(value: Any) -> Optional[date]: - if not value: - return None - try: - return date.fromisoformat(str(value)[:10]) - except ValueError: - return None - - -def load_json(text: Any, default: Any) -> Any: - if text in (None, ""): - return default - try: - return json.loads(text) - except (TypeError, ValueError): - return default - - -def truthy(value: Any, default: bool = False) -> bool: - if value is None: - return default - return bool(int(value)) - - -def _default_instance() -> str: - return (os.environ.get("DEFAULT_INSTANCE") or "default").strip().lower() or "default" - - -def _workflow_status_for_stage(stage: str) -> str: - if stage == "completed": - return "completed" - if stage == "failed": - return "failed" - return "waiting_review" if stage in _REVIEW_STAGES else "running" - - -def _index_key(index_name: str) -> str: - prod_names = { - n.strip() - for n in ( - os.environ.get("PROD_VECTOR_DB_COLLECTION_NAME"), - os.environ.get("PROD_SCHEME_VECTOR_DB_COLLECTION_NAME"), - ) - if n and n.strip() - } - return "prod" if index_name in prod_names else "dev" - - -def _has_table(sq: sqlite3.Connection, name: str) -> bool: - return sq.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?", (name,) - ).fetchone() is not None - - -def _rows(sq: sqlite3.Connection, table: str, where: str = "", params: tuple = ()) -> list[sqlite3.Row]: - if not _has_table(sq, table): - return [] - return sq.execute(f"SELECT * FROM {table} {where}", params).fetchall() - - -def _get(row: sqlite3.Row, key: str, default: Any = None) -> Any: - return row[key] if key in row.keys() else default - - -# ----------------------------------------------------------------------------- -# Catalog -# ----------------------------------------------------------------------------- - -def load_catalog_registry(sq, report: Report) -> dict[str, dict[str, Any]]: - """Legacy ``scheme_catalog_entries`` by code. Their name and aliases are merged - into the documents holding the code; see ``settle_catalog_registry``.""" - registry: dict[str, dict[str, Any]] = {} - for row in _rows(sq, "scheme_catalog_entries"): - report.count("scheme_catalog_entries", "read") - code = (row["scheme_code"] or "").strip().lower() - if not _CODE_RE.match(code): - report.skip("scheme_catalog_entries", str(row["scheme_code"]), "code is not a valid slug") - continue - aliases = [a for a in load_json(_get(row, "scheme_aliases_json"), []) if isinstance(a, str) and a] - registry[code] = {"name": row["scheme_name"] or code, "aliases": aliases} - return registry - - -def settle_catalog_registry(registry: dict[str, dict[str, Any]], consumed: set[str], report: Report) -> None: - """Registry entries a migrated document used count as inserted (merged into the - document); the rest have no document and are dropped.""" - unused = [] - for code in sorted(registry): - if code in consumed: - report.count("scheme_catalog_entries", "inserted") - else: - report.count("scheme_catalog_entries", "dropped") - unused.append(code) - if unused: - report.notes.append( - f"scheme_catalog_entries without a document were not migrated: {', '.join(unused)}" - ) - - -def migrate_catalog_version(sq, conn, report: Report) -> None: - rows = _rows(sq, "scheme_catalog_meta") - legacy_version = int(rows[0]["version"] or 0) if rows else 0 - if rows: - report.count("scheme_catalog_meta", "read") - report.count("scheme_catalog_meta", "inserted") - for n in range(1, legacy_version + 1): - conn.execute( - """ - INSERT INTO audit_logs (id, action_type, entity_type, metadata) - VALUES (%s::uuid, %s, 'catalog', %s) ON CONFLICT (id) DO NOTHING - """, - (det_id("catalog-bump", str(n)), db.CATALOG_VERSION_ACTION, - Jsonb({"reason": "sqlite_migration:carried_over"})), - ) - conn.execute( - """ - INSERT INTO audit_logs (id, action_type, entity_type, metadata) - VALUES (%s::uuid, %s, 'catalog', %s) ON CONFLICT (id) DO NOTHING - """, - (det_id("catalog-bump", "migration"), db.CATALOG_VERSION_ACTION, - Jsonb({"reason": "sqlite_migration"})), - ) - - -def _instance_of(doc: sqlite3.Row) -> str: - return (_get(doc, "instance") or _default_instance()).strip().lower() or _default_instance() - - -def _instance_for(conn, instance: str) -> str: - """The ``instances`` row of a tenant code, created with a deterministic id.""" - path = db._tenant_path(instance) - row = conn.execute( - """ - INSERT INTO instances (id, name, path) VALUES (%s::uuid, %s, %s) - ON CONFLICT (path) DO UPDATE SET name = instances.name RETURNING id - """, - (det_id("instance", path), db._tenant_code_from_path(path), path), - ).fetchone() - return str(row["id"]) - - -def _catalog_for(conn, instance_id: str, kind: str) -> str: - """The ``(instance, kind)`` catalog, created with a deterministic id.""" - catalog_type = catalog_type_for_kind(kind) - row = conn.execute( - """ - INSERT INTO catalogs (id, instance_id, type, name) - SELECT %s::uuid, g.id, %s, g.name || ' ' || %s FROM instances g WHERE g.id = %s::uuid - ON CONFLICT (instance_id, type) DO UPDATE SET type = EXCLUDED.type RETURNING id - """, - (det_id("catalog", f"{instance_id}:{catalog_type}"), catalog_type, kind, instance_id), - ).fetchone() - return str(row["id"]) - - -def _slug(text: str) -> str: - return re.sub(r"[^a-z0-9]+", "-", (text or "").lower()).strip("-")[:40].strip("-") - - -def _unique_code(conn, catalog_id: str, base: str) -> str: - code, n = base, 1 - while conn.execute( - "SELECT 1 FROM documents WHERE catalog_id = %s::uuid AND code = %s", (catalog_id, code) - ).fetchone(): - n += 1 - code = f"{base}-{n}" - return code - - -# ----------------------------------------------------------------------------- -# Documents -# ----------------------------------------------------------------------------- - -def _user_for(conn, user_id, username, email, roles) -> Optional[str]: - if not (user_id or "").strip(): - return None - return db.upsert_user(user_id, username, email, roles, conn=conn) - - -def _build_index_status(sq, workflow_id: str) -> dict[str, dict]: - entries: dict[str, dict] = {} - for row in _rows(sq, "document_index_status", "WHERE workflow_id = ?", (workflow_id,)): - key = _index_key(row["index_name"]) - details = load_json(row["details_json"], {}) - state = IndexState( - status=row["status"] if row["status"] in ("unknown", "indexed", "missing") else "unknown", - index_name=row["index_name"], - doc_id=_get(row, "doc_id") or _get(row, "marqo_doc_id"), - chunk_count_indexed=max(int(row["chunk_count_indexed"] or 0), 0), - last_indexed_at=row["last_indexed_at"], - last_verified_at=row["last_verified_at"], - schema_version=row["schema_version"], - details=details if isinstance(details, dict) else {}, - ).model_dump() - current = entries.get(key) - if current is None or (state["last_indexed_at"] or "") >= (current["last_indexed_at"] or ""): - entries[key] = state - return entries - - -def _build_workflows( - sq, doc: sqlite3.Row, started_by: Optional[str], report: Report -) -> tuple[list[dict], dict[Any, str]]: - workflow_id = doc["workflow_id"] - stage = doc["stage"] if doc["stage"] in _STAGES else "registered" - created_at = parse_ts(doc["created_at"]) or datetime.now(timezone.utc) - stamps = { - key: value - for key, value in ( - ("ocr", _get(doc, "ocr_completed_at")), - ("translation", _get(doc, "translation_completed_at")), - ("chunks", _get(doc, "chunks_completed_at")), - ) - if value - } - - workflows: list[dict] = [] - job_ids: dict[Any, str] = {} - has_root = False - for job in _rows(sq, "document_jobs", "WHERE workflow_id = ? ORDER BY started_at, id", (workflow_id,)): - job_type = job["job_type"] - if job_type not in _WORKFLOW_TYPES: - report.skip("document_jobs", f"{workflow_id}#{job['id']}", f"unknown job_type {job_type!r}") - continue - temporal_id = job["temporal_workflow_id"] or workflow_id - is_root = job_type in ("pipeline", "ocr_only") and temporal_id == workflow_id and not has_root - has_root = has_root or is_root - wf_id = det_id("workflow", f"{workflow_id}:job:{job['id']}") - job_ids[job["id"]] = wf_id - status = job["status"] if job["status"] in _WORKFLOW_STATUSES else "failed" - job_stage = job["current_stage"] if job["current_stage"] in _STAGES else stage - config = load_json(job["config_json"], {}) - config = config if isinstance(config, dict) else {} - if job["temporal_run_id"]: - config["temporal_run_id"] = job["temporal_run_id"] - workflows.append({ - "id": wf_id, "type": job_type, "temporal_id": temporal_id, "status": status, - "stage": job_stage, "stamps": stamps if is_root else {}, - "stop_after_ocr": truthy(_get(doc, "stop_after_ocr")) and is_root, - "error_message": job["error_message"], "config": config, - "started_by": started_by if is_root else None, - "started_at": parse_ts(job["started_at"]) or created_at, - "completed_at": parse_ts(job["completed_at"]), "job_id": job["id"], - "inserted_as": "document_jobs", - }) - - if not has_root: - workflows.append({ - "id": det_id("workflow", f"{workflow_id}:root"), - "type": "ocr_only" if truthy(_get(doc, "stop_after_ocr")) else "pipeline", - "temporal_id": workflow_id, "status": _workflow_status_for_stage(stage), "stage": stage, - "stamps": stamps, "stop_after_ocr": truthy(_get(doc, "stop_after_ocr")), - "error_message": doc["error_message"], "config": {}, "started_by": started_by, - "started_at": created_at, - "completed_at": parse_ts(_get(doc, "ingested_at")) if stage == "completed" else None, - "job_id": None, "inserted_as": None, - }) - - requested_at = parse_ts(_get(doc, "prod_ready_requested_at")) - if requested_at and stage not in ("completed", "ingesting_prod"): - workflows.append({ - "id": det_id("workflow", f"{workflow_id}:prod-request"), "type": "promote_prod", - "temporal_id": f"{workflow_id}-prod-request", "status": "waiting_review", - "stage": "approval_for_prod", "stamps": {}, "stop_after_ocr": False, - "error_message": None, "config": {}, "started_by": None, "started_at": requested_at, - "completed_at": None, "job_id": None, "inserted_as": None, - "requested_by": ( - _get(doc, "prod_ready_requested_by_user_id"), - _get(doc, "prod_ready_requested_by_username"), - ), - }) - - workflows.sort(key=lambda w: w["started_at"]) - latest = workflows[-1] - latest["stage"] = stage - latest["error_message"] = doc["error_message"] - return workflows, job_ids - - -def _skip_document(sq, conn, doc: sqlite3.Row, report: Report, audit_by_workflow: dict[str, list], reason: str) -> None: - """Leave a document behind. Its history rows are counted as skipped; its audit - entries are kept, tied to the tenant's instance but to no document.""" - workflow_id = doc["workflow_id"] - report.skip("documents", workflow_id, reason) - for table in ("document_jobs", "document_artifacts", "pages", "chunks"): - report.count(table, "skipped", len(_rows(sq, table, "WHERE workflow_id = ?", (workflow_id,)))) - for audit in audit_by_workflow.pop(workflow_id, []): - insert_audit(conn, audit, None, _instance_of(doc), report) - - -def migrate_document( - sq, conn, doc: sqlite3.Row, report: Report, audit_by_workflow: dict[str, list], - registry: Optional[dict[str, dict[str, Any]]] = None, consumed: Optional[set[str]] = None, -) -> bool: - registry = registry if registry is not None else {} - consumed = consumed if consumed is not None else set() - workflow_id = doc["workflow_id"] - doc_uuid = det_id("document", workflow_id) - exists = conn.execute("SELECT 1 FROM documents WHERE id = %s::uuid", (doc_uuid,)).fetchone() - if exists: - report.count("documents", "existing") - for table, label in ( - ("document_jobs", "document_jobs"), ("document_artifacts", "document_artifacts"), - ("pages", "pages"), ("chunks", "chunks"), - ): - report.count(label, "existing", len(_rows(sq, table, "WHERE workflow_id = ?", (workflow_id,)))) - return False - - kind = (_get(doc, "document_kind") or "document").strip().lower() - if truthy(_get(doc, "is_demo")): - _skip_document(sq, conn, doc, report, audit_by_workflow, "demo document (is_demo)") - return False - if kind not in KIND_TO_CATALOG_TYPE: - _skip_document(sq, conn, doc, report, audit_by_workflow, f"document_kind {kind!r} has no catalog") - return False - - fingerprint = next( - ( - c for c in ( - _get(doc, "source_file_fingerprint"), _get(doc, "canonical_document_id"), doc["document_id"] - ) if c and _FINGERPRINT_RE.match(c) - ), - None, - ) - if fingerprint is None: - import hashlib - - fingerprint = hashlib.md5((doc["document_id"] or workflow_id).encode()).hexdigest() - report.notes.append(f"{workflow_id}: no hex fingerprint, derived md5 of document_id") - - chunk_rows = _rows(sq, "chunks", "WHERE workflow_id = ? ORDER BY chunk_number", (workflow_id,)) - - chunking = None - if chunk_rows: - first = chunk_rows[0] - chunking = ChunkingRun( - run_number=max(max(int(r["chunk_version"] or 1) for r in chunk_rows), 1), - provider=_get(first, "chunking_provider"), - model=_get(first, "chunking_model"), - config=load_json(_get(first, "chunking_config_json"), {}), - ) - - valid_from, valid_to = parse_date(_get(doc, "valid_from")), parse_date(_get(doc, "valid_to")) - if valid_from and valid_to and valid_from > valid_to: - report.notes.append(f"{workflow_id}: valid_from after valid_to, valid_to cleared") - valid_to = None - - uploader = _user_for( - conn, _get(doc, "uploaded_by_user_id"), _get(doc, "uploaded_by_username"), - _get(doc, "uploaded_by_email"), _get(doc, "uploaded_by_roles"), - ) - - metadata = DocumentMetadata( - valid_from=valid_from.isoformat() if valid_from else None, - valid_to=valid_to.isoformat() if valid_to else None, - uploaded_by_user_id=uploader, - index_status=_build_index_status(sq, workflow_id), - catalog_visible=truthy(_get(doc, "catalog_visible"), True), - network_visible=truthy(_get(doc, "network_visible"), True), - reindex_required=truthy(_get(doc, "reindex_required")), - reindex_reason=_get(doc, "reindex_reason") or None, - chunking=chunking, - ).model_dump(exclude_none=True) - source = DocumentSource( - fingerprint=fingerprint, - original_filename=_get(doc, "source_filename") or doc["filename"], - storage_uri=doc["filepath"], - source_type=_get(doc, "source_type"), - canonical_input_type=_get(doc, "canonical_input_type"), - ).model_dump(exclude_none=True) - - catalog_id = _catalog_for(conn, _instance_for(conn, _instance_of(doc)), kind) - code = (_get(doc, "scheme_code") or "").strip().lower() - entry = registry.get(code) if kind == "scheme" and code else None - if entry is not None: - consumed.add(code) - name = (entry or {}).get("name") or _get(doc, "scheme_name") or _get(doc, "display_name") or None - aliases = (entry or {}).get("aliases") or [ - a for a in load_json(_get(doc, "scheme_aliases_json"), []) if isinstance(a, str) and a - ] - if code and not _CODE_RE.match(code): - report.notes.append(f"{workflow_id}: scheme_code {code!r} is not a valid slug, code derived") - code = "" - if not code: - code = _slug(name or "") or _slug(Path(doc["filename"] or "").stem) - if not _CODE_RE.match(code): - code = f"doc-{fingerprint[:8]}" - report.notes.append(f"{workflow_id}: no usable scheme_code, derived {code!r}") - unique = _unique_code(conn, catalog_id, code) - if unique != code: - report.notes.append(f"{workflow_id}: code {code!r} already used in its catalog, stored as {unique!r}") - code = unique - - created_at = parse_ts(doc["created_at"]) or datetime.now(timezone.utc) - updated_at = parse_ts(doc["updated_at"]) or created_at - conn.execute( - """ - INSERT INTO documents (id, catalog_id, code, name, aliases, source, is_disabled, metadata, - created_at, updated_at) - VALUES (%s::uuid, %s::uuid, %s, %s, %s, %s, %s, %s, %s, %s) - """, - ( - doc_uuid, catalog_id, code, name or code, aliases, Jsonb(source), - truthy(_get(doc, "is_disabled")), Jsonb(metadata), created_at, updated_at, - ), - ) - report.count("documents", "inserted") - - workflows, job_ids = _build_workflows(sq, doc, uploader, report) - root = next((w for w in workflows if w["type"] in ("pipeline", "ocr_only")), workflows[0]) - for w in workflows: - started_by = w["started_by"] - if w.get("requested_by"): - started_by = _user_for(conn, w["requested_by"][0], w["requested_by"][1], None, None) - conn.execute( - """ - INSERT INTO workflows (id, document_id, workflow_type, temporal_workflow_id, status, - current_stage, stage_timestamps, stop_after_ocr, error_message, - config, started_by_user_id, started_at, completed_at, updated_at) - VALUES (%s::uuid, %s::uuid, %s, %s, %s, - %s, %s, %s, %s, %s, %s::uuid, %s, %s, %s) - """, - ( - w["id"], doc_uuid, w["type"], w["temporal_id"], w["status"], w["stage"], - Jsonb(w["stamps"]), w["stop_after_ocr"], w["error_message"], Jsonb(w["config"]), - started_by, w["started_at"], w["completed_at"], w["completed_at"] or w["started_at"], - ), - ) - if w["inserted_as"]: - report.count("document_jobs", "inserted") - - for art in _rows(sq, "document_artifacts", "WHERE workflow_id = ? ORDER BY id", (workflow_id,)): - size = art["size_bytes"] - art_metadata = load_json(art["metadata_json"], {}) - conn.execute( - """ - INSERT INTO workflow_artifacts (id, workflow_id, artifact_type, stage, storage_uri, - mime_type, filename, size_bytes, metadata, created_at) - VALUES (%s::uuid, %s::uuid, %s, %s, %s, %s, %s, %s, %s, %s) - """, - ( - det_id("artifact", f"{workflow_id}:{art['id']}"), - job_ids.get(art["job_id"], root["id"]), - art["artifact_type"], art["stage"] if art["stage"] in _STAGES else None, - art["storage_uri"], art["mime_type"], art["filename"], - size if size is not None and size >= 0 else None, - Jsonb(art_metadata if isinstance(art_metadata, dict) else {}), - parse_ts(art["created_at"]) or created_at, - ), - ) - report.count("document_artifacts", "inserted") - - for page in _rows(sq, "pages", "WHERE workflow_id = ? ORDER BY page_number", (workflow_id,)): - if int(page["page_number"] or 0) < 1: - report.skip("pages", f"{workflow_id}#{page['page_number']}", "page_number < 1") - continue - translations = {} - if any( - _get(page, k) not in (None, "") - for k in ("translated_markdown", "edited_translation", "translation_notes") - ) or truthy(_get(page, "translation_reviewed")): - translations[_get(page, "translation_target_language") or "en"] = PageTranslation( - translated_markdown=_get(page, "translated_markdown"), - edited_translation=_get(page, "edited_translation"), - is_reviewed=truthy(_get(page, "translation_reviewed")), - reviewer_notes=_get(page, "translation_notes"), - provider=_get(page, "translation_provider"), - model=_get(page, "translation_model"), - translated_at=_get(page, "translated_at"), - ).model_dump() - page_meta = PageMetadata( - is_reviewed=truthy(_get(page, "is_reviewed")), - reviewer_notes=_get(page, "reviewer_notes"), - detected_language=_get(page, "detected_language"), - ).model_dump(exclude_none=True) - conn.execute( - """ - INSERT INTO pages (id, document_id, page_number, original_markdown, edited_markdown, - translations, metadata) - VALUES (%s::uuid, %s::uuid, %s, %s, %s, %s, %s) - """, - ( - det_id("page", f"{workflow_id}:{page['page_number']}"), doc_uuid, page["page_number"], - page["original_markdown"], page["edited_markdown"], Jsonb(translations), - Jsonb(page_meta), - ), - ) - report.count("pages", "inserted") - - for chunk in chunk_rows: - if int(chunk["chunk_number"] or 0) < 1: - report.skip("chunks", f"{workflow_id}#{chunk['chunk_number']}", "chunk_number < 1") - continue - page_start = max(int(chunk["page_start"] or 1), 1) - spans = load_json(_get(chunk, "source_spans_json"), []) - conn.execute( - """ - INSERT INTO chunks (id, document_id, chunk_number, original_text, edited_text, - token_count, page_start, page_end, source_spans, section_title, - content_type, is_reference, is_reviewed, is_excluded, reviewer_notes) - VALUES (%s::uuid, %s::uuid, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) - """, - ( - det_id("chunk", f"{workflow_id}:{chunk['chunk_number']}"), doc_uuid, - chunk["chunk_number"], chunk["original_text"] or "", chunk["edited_text"], - max(int(chunk["token_count"] or 0), 0), page_start, - max(int(chunk["page_end"] or page_start), page_start), - Jsonb(spans if isinstance(spans, list) else []), chunk["section_title"], - chunk["content_type"], truthy(_get(chunk, "is_reference")), - truthy(_get(chunk, "is_reviewed")), truthy(_get(chunk, "is_excluded")), - chunk["reviewer_notes"], - ), - ) - report.count("chunks", "inserted") - - for audit in audit_by_workflow.pop(workflow_id, []): - insert_audit(conn, audit, doc_uuid, source_instance=_instance_of(doc), report=report) - return True - - -# ----------------------------------------------------------------------------- -# Audit, settings, OTPs, webhooks -# ----------------------------------------------------------------------------- - -def _legacy_audit_field(field_name: Optional[str], entity: Optional[str]) -> Optional[str]: - if entity == "setting" and field_name in _LEGACY_SETTING_KEYS: - flat = _LEGACY_SETTING_KEYS[field_name] - return f"{db._SETTING_GROUPS[flat][0]}.{flat}" - return field_name - - -def insert_audit(conn, audit: sqlite3.Row, doc_uuid: Optional[str], source_instance: Optional[str], report: Report) -> None: - entity = audit["entity_type"] if audit["entity_type"] in _ENTITY_TYPES else None - workflow_id = audit["workflow_id"] - if audit["document_id"] == "__settings__": - entity, workflow_id = "setting", None - elif workflow_id == "__catalog__": - entity, workflow_id = "catalog", None - metadata = load_json(audit["metadata"], None) - if not isinstance(metadata, dict): - metadata = {"raw": audit["metadata"]} if audit["metadata"] else {} - roles = [r.strip() for r in (_get(audit, "actor_roles") or "").split(",") if r.strip()] - actor_present = any((_get(audit, "actor_user_id"), _get(audit, "actor_username"), _get(audit, "actor_email"), roles)) - actor_uuid = _user_for( - conn, _get(audit, "actor_user_id"), _get(audit, "actor_username"), _get(audit, "actor_email"), roles - ) - inserted = conn.execute( - """ - INSERT INTO audit_logs (id, document_id, workflow_id, instance_id, action_type, entity_type, - entity_number, field_name, old_value, new_value, metadata, - actor_user_id, actor, created_at) - VALUES (%s::uuid, %s::uuid, %s, %s::uuid, %s, %s, %s, %s, %s, %s, %s, - %s::uuid, %s, %s) - ON CONFLICT (id) DO NOTHING RETURNING id - """, - ( - det_id("audit", str(audit["id"])), doc_uuid, workflow_id, - _instance_for(conn, source_instance.strip().lower()) if (source_instance or "").strip() else None, - audit["action_type"], entity, audit["entity_id"], - _legacy_audit_field(audit["field_name"], entity), audit["old_value"], audit["new_value"], - Jsonb(metadata), actor_uuid, - Jsonb({ - "username": _get(audit, "actor_username") or None, - "email": _get(audit, "actor_email") or None, - "roles": roles, - }) if actor_present else Jsonb({}), - parse_ts(audit["timestamp"]) or datetime.now(timezone.utc), - ), - ).fetchone() - report.count("audit_logs", "inserted" if inserted else "existing") - - -def migrate_settings(sq, conn, report: Report) -> None: - rows = _rows(sq, "settings") - for _ in rows: - report.count("settings", "read") - current = conn.execute( - "SELECT retrieval, candidates, results, query FROM search_settings WHERE singleton" - ).fetchone() - if current is None: - conn.execute("INSERT INTO search_settings DEFAULT VALUES") - current = {"retrieval": {}, "candidates": {}, "results": {}, "query": {}} - if any(current[g] for g in ("retrieval", "candidates", "results", "query")): - for _ in rows: - report.count("settings", "existing") - report.notes.append("search_settings already configured in Postgres, legacy settings not applied") - return - - groups: dict[str, dict[str, Any]] = {} - for row in rows: - flat = _LEGACY_SETTING_KEYS.get(row["key"]) - if flat is None: - report.skip("settings", row["key"], "unknown setting key") - continue - raw = row["value"] - try: - value: Any = ( - raw == "true" if flat in _BOOL_SETTINGS - else int(raw) if flat in _INT_SETTINGS - else float(raw) if flat in _FLOAT_SETTINGS - else raw - ) - except ValueError: - report.skip("settings", row["key"], f"unparseable value {raw!r}") - continue - group, model = db._SETTING_GROUPS[flat] - try: - model.model_validate({flat: value}) - except Exception as exc: - report.skip("settings", row["key"], f"invalid value: {exc}") - continue - groups.setdefault(group, {})[flat] = value - report.count("settings", "inserted") - for group, values in groups.items(): - conn.execute( - f"UPDATE search_settings SET {group} = {group} || %s::jsonb WHERE singleton", (Jsonb(values),) - ) - - -def migrate_email_otps(sq, conn, report: Report) -> None: - now = datetime.now(timezone.utc) - for row in _rows(sq, "email_otps"): - report.count("email_otps", "read") - expires = parse_ts(row["expires_at"]) - if expires is None or expires <= now: - report.count("email_otps", "dropped") - continue - inserted = conn.execute( - """ - INSERT INTO email_otps (email, code_hash, expires_at, attempts, last_sent_at) - VALUES (%s, %s, %s, %s, %s) ON CONFLICT (email) DO NOTHING RETURNING id - """, - (row["email"].strip().lower(), row["code_hash"], expires, max(int(row["attempts"] or 0), 0), - parse_ts(row["last_sent_at"]) or now), - ).fetchone() - report.count("email_otps", "inserted" if inserted else "existing") - - -def migrate_webhook_records(sq, conn, report: Report) -> None: - for row in _rows(sq, "webhook_records"): - report.count("webhook_records", "read") - payload = load_json(row["payload"], None) - if not isinstance(payload, dict): - report.skip("webhook_records", f"{row['source']}/{row['external_id']}", "payload is not a JSON object") - continue - try: - record_id = str(uuid.UUID(row["id"])) - except (ValueError, TypeError): - record_id = det_id("webhook", row["id"] or f"{row['source']}:{row['external_id']}") - published = row["is_published"] - inserted = conn.execute( - """ - INSERT INTO webhook_records (id, source, external_id, title, slug, status, is_published, - category, payload, payload_hash, created_at, updated_at) - VALUES (%s::uuid, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) - ON CONFLICT (source, external_id) DO NOTHING RETURNING id - """, - ( - record_id, row["source"], row["external_id"], row["title"], row["slug"], row["status"], - None if published is None else bool(published), row["category"], Jsonb(payload), - row["payload_hash"], parse_ts(row["created_at"]) or datetime.now(timezone.utc), - parse_ts(row["updated_at"]) or datetime.now(timezone.utc), - ), - ).fetchone() - report.count("webhook_records", "inserted" if inserted else "existing") - - -# ----------------------------------------------------------------------------- -# Driver -# ----------------------------------------------------------------------------- - -def run_migration(sqlite_path: str, database_url: str, dry_run: bool = False) -> Report: - report = Report() - sq = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True) - sq.row_factory = sqlite3.Row - try: - if not dry_run: - migrate.apply_migrations(database_url) - elif not migrate.is_current(database_url): - raise SystemExit("Postgres migrations are not applied; apply them before a dry run.") - - with psycopg.connect(database_url, autocommit=not dry_run, row_factory=dict_row) as conn: - conn.execute("SET timezone = 'UTC'") - documents = _rows(sq, "documents", "ORDER BY created_at, workflow_id") - known_workflows = {d["workflow_id"] for d in documents} - for table in ("document_jobs", "document_artifacts", "pages", "chunks"): - if _has_table(sq, table): - total = sq.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0] - stray = sq.execute( - f"SELECT COUNT(*) FROM {table} WHERE workflow_id NOT IN " - "(SELECT workflow_id FROM documents)" - ).fetchone()[0] - report.count(table, "read", total) - if stray: - report.count(table, "skipped", stray) - report.notes.append(f"{table}: {stray} rows belong to no document and were not migrated") - - audit_by_workflow: dict[str, list] = {} - orphan_audit: list = [] - for audit in _rows(sq, "audit_logs", "ORDER BY id"): - report.count("audit_logs", "read") - if audit["workflow_id"] in known_workflows and audit["document_id"] != "__settings__": - audit_by_workflow.setdefault(audit["workflow_id"], []).append(audit) - else: - orphan_audit.append(audit) - - registry = load_catalog_registry(sq, report) - consumed: set[str] = set() - - for doc in documents: - report.count("documents", "read") - with conn.transaction(): - migrate_document(sq, conn, doc, report, audit_by_workflow, registry, consumed) - settle_catalog_registry(registry, consumed, report) - - skipped = Counter(i["reason"] for i in report.skip_reasons if i["table"] == "documents") - for reason, n in sorted(skipped.items()): - report.notes.append(f"documents not migrated: {n} x {reason}") - - with conn.transaction(): - for audits in audit_by_workflow.values(): - orphan_audit.extend(audits) - for audit in orphan_audit: - insert_audit(conn, audit, None, None, report) - migrate_settings(sq, conn, report) - migrate_email_otps(sq, conn, report) - migrate_webhook_records(sq, conn, report) - migrate_catalog_version(sq, conn, report) - - manifest = _rows(sq, "document_manifest_entries") - if manifest: - report.count("document_manifest_entries", "read", len(manifest)) - report.count("document_manifest_entries", "dropped", len(manifest)) - report.notes.append("document_manifest_entries is not part of the schema and was not migrated") - - if dry_run: - conn.rollback() - finally: - sq.close() - return report - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument("--sqlite-path", default=os.environ.get("DOCUMENT_DB_PATH", "/data/documents.db")) - parser.add_argument("--database-url", default=os.environ.get("DATABASE_URL", db.DEFAULT_DATABASE_URL)) - parser.add_argument("--dry-run", action="store_true", help="run everything, then roll back") - parser.add_argument("--report", help="write the report as JSON to this path") - args = parser.parse_args() - - if not os.path.exists(args.sqlite_path): - print(f"SQLite database not found at {args.sqlite_path}", file=sys.stderr) - return 2 - - report = run_migration(args.sqlite_path, args.database_url, dry_run=args.dry_run) - print(("DRY RUN - nothing was written\n" if args.dry_run else "") + report.render()) - if args.report: - Path(args.report).write_text(json.dumps(report.as_dict(), indent=2, default=str)) - return 1 if report.mismatches() else 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/verify_document_validity_e2e.py b/scripts/verify_document_validity_e2e.py index 617672e..be4a9c1 100644 --- a/scripts/verify_document_validity_e2e.py +++ b/scripts/verify_document_validity_e2e.py @@ -36,9 +36,9 @@ from datetime import datetime # noqa: E402 -from pipeline import db # noqa: E402 +from pipeline import db from pipeline.documents import document_validity # noqa: E402 -from pipeline.activities import _prepare_records, _validity_fields_from_doc # noqa: E402 +from pipeline.search.passage_projection import _prepare_records, _validity_fields_from_doc # noqa: E402 from pipeline.vector_store.qdrant_store import QdrantVectorStore # noqa: E402 db.init_db() diff --git a/test_data/synthetic-cotton-gu.pdf b/test_data/synthetic-cotton-gu.pdf new file mode 100644 index 0000000..abee26e Binary files /dev/null and b/test_data/synthetic-cotton-gu.pdf differ diff --git a/test_data/synthetic-cotton-webhook.json b/test_data/synthetic-cotton-webhook.json new file mode 100644 index 0000000..9eea138 --- /dev/null +++ b/test_data/synthetic-cotton-webhook.json @@ -0,0 +1,26 @@ +{ + "advisoryId": "cotton-field-drainage-demo", + "title": "કપાસના ખેતરની સંભાળ", + "content": { + "language": "gu", + "text": "કપાસના ખેતરની સંભાળ\n\nવરસાદ પછી કપાસના ખેતરમાં પાણી ભરાયું છે કે નહીં તે તપાસો. જ્યાં પાણી ઊભું હોય ત્યાં નિકાસનો માર્ગ ખુલ્લો રાખો. જમીનમાં ભેજ હોય તો તરત સિંચાઈ ન કરો. ખેતરની મુલાકાતની તારીખ અને જમીનની સ્થિતિ લખી રાખો.\n\nઆ એક કૃત્રિમ પરીક્ષણ ઉદાહરણ છે.", + "translations": { + "en": "Cotton field care\n\nAfter rainfall, check the cotton field for standing water. Keep drainage channels open wherever water has collected. If the soil is moist, wait before irrigating. Record the date of the field visit and the condition of the soil.\n\nThis is a synthetic test example.", + "hi": "कपास के खेत की देखभाल\n\nबारिश के बाद देखें कि कपास के खेत में पानी जमा तो नहीं है। जहाँ पानी ठहरा है, वहाँ निकासी का रास्ता खुला रखें। मिट्टी में नमी हो तो तुरंत सिंचाई न करें। खेत के निरीक्षण की तारीख और मिट्टी की स्थिति लिख लें।\n\nयह परीक्षण के लिए बनाया गया एक उदाहरण है।" + } + }, + "applicability": { + "cropSubjects": [{"subjectId": "https://taxonomy.openagrinet.global/crops/cotton", "descriptor": {"code": "COTTON", "name": "Cotton"}}], + "validFrom": "2026-01-01", + "validTo": "2027-12-31" + }, + "metadata": { + "subject_categories": ["Crop", "Practice"], + "agriculture_subjects": [{"subjectId": "https://taxonomy.openagrinet.global/crops/cotton", "descriptor": {"code": "COTTON", "name": "Cotton"}}], + "coverage_areas": [{"codeScheme": "ISO3166-2", "areaCode": "IN-GJ", "areaLevel": "State", "areaName": "Gujarat"}], + "seasons": ["Kharif"], + "topics": ["Field drainage", "Soil moisture"], + "tags": ["synthetic-demo"], + "issued_at": "2026-07-01T08:00:00+05:30" + } +} diff --git a/tests/conftest.py b/tests/conftest.py index 4acf436..2c74027 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -20,11 +20,15 @@ os.environ["MINIO_SECRET_KEY"] = "test-secret-key" os.environ["TRANSLATION_VLLM_BASE_URL"] = "http://localhost:8000/v1" os.environ["AUTH_DISABLED"] = "true" +os.environ["KP_RUNTIME_PROFILE"] = "local" +os.environ["KEYCLOAK_PROVIDER_GROUP_ROOT"] = "states" +os.environ["PORTAL_INSTANCE"] = "bv" +os.environ["PROVIDER_DISPLAY_NAMES"] = '{"bv":"Bharat Vistaar","mh":"Maharashtra","ka":"Karnataka","up":"Uttar Pradesh","gj":"Gujarat","tn":"Tamil Nadu","wb":"West Bengal","dl":"Delhi","jk":"Jammu and Kashmir","or":"Odisha","od":"Odisha","ts":"Telangana","tg":"Telangana"}' _TRUNCATE_SQL = """ - TRUNCATE documents, users, instances, catalogs, audit_logs, email_otps, - webhook_records, search_settings CASCADE + TRUNCATE documents, users, instances, catalogs, audit_logs, + search_settings, schema_pack_versions, scheduled_source_pulls, email_sign_in_limits CASCADE """ @@ -95,27 +99,25 @@ def mock_temporal_client(): @pytest.fixture -def mock_minio_client(): +def mock_object_store(): """Mock MinIO client for testing without actual MinIO server.""" client = MagicMock() - client.bucket_exists = MagicMock(return_value=True) - client.make_bucket = MagicMock() - client.put_object = MagicMock() - client.get_object = MagicMock(return_value=MagicMock(read=lambda: b"%PDF-test")) + client.ensure_bucket = MagicMock() + client.put = MagicMock() + client.open = MagicMock(return_value=iter((b"%PDF-test",))) return client @pytest.fixture -def test_client(db_connection, mock_temporal_client, mock_minio_client): +def test_client(db_connection, mock_temporal_client, mock_object_store): """Create FastAPI TestClient with mocked dependencies. Neutralizes the app's real lifespan (which otherwise connects to a live Temporal server and MinIO) before entering TestClient — see test_scheme_catalog.py's asgi_client fixture for the same pattern. - Depends on ``db_connection`` for a file-backed SQLite: an in-memory one - lives per connection, so its tables are gone by the time a route opens - its own connection. + Uses the isolated PostgreSQL test database and resets the shared limiter + so requests in one test cannot affect a later test. """ from contextlib import asynccontextmanager @@ -125,7 +127,7 @@ def test_client(db_connection, mock_temporal_client, mock_minio_client): # Patch the global clients api.temporal_client = mock_temporal_client - api.minio_client = mock_minio_client + api.object_store = mock_object_store @asynccontextmanager async def _noop_lifespan(_app): @@ -133,11 +135,13 @@ async def _noop_lifespan(_app): original = api.app.router.lifespan_context api.app.router.lifespan_context = _noop_lifespan + api.limiter.reset() try: with TestClient(api.app) as client: yield client finally: api.app.router.lifespan_context = original + api.limiter.reset() class CapturingVectorStore: @@ -151,12 +155,12 @@ def __init__(self): self.hits: list[dict] = [] def search(self, **kwargs): - from datetime import date + from pipeline.documents.document_validity import today self.captured.update(kwargs) valid_on = kwargs.get("valid_on") if valid_on is None and kwargs.get("apply_validity", True): - valid_on = date.today().isoformat() + valid_on = today() return {"hits": list(self.hits), "valid_on": valid_on} diff --git a/tests/icar_webhook_fixture.py b/tests/icar_webhook_fixture.py new file mode 100644 index 0000000..95b9679 --- /dev/null +++ b/tests/icar_webhook_fixture.py @@ -0,0 +1,74 @@ +"""Sanitized ICAR payload exercised by canonical intake tests.""" + +import copy + +OVERVIEW = ( + "Timely harvesting of AKJ-1 helps preserve grain appearance, processing quality and " + "suitability for flaking. Proper drying and handling after harvest are essential to " + "minimize grain damage and deterioration." +) +SYMPTOMS = ( + "Panicles become mature and dry, grains become hard and attain characteristic colour, " + "and most leaves begin to dry." +) +CONDITIONS = ( + "Crop has reached physiological maturity and panicles are fully developed; grains are " + "firm and suitable for flaking quality." +) +ADVISORY = ( + "Harvest AKJ-1 at physiological maturity when the grains are fully developed and firm. " + "Timely harvesting is important for maintaining grain quality required for flaking. " + "Avoid delayed harvesting, which may increase grain weathering, shattering and mould " + "risk. After harvesting, dry the panicles properly, thresh carefully, clean the grain " + "and reduce moisture to a safe level before storage or processing." +) + +SORGHUM_PAYLOAD = { + "contentID": 987, + "title": "Sorghum- Harvesting Management", + "commodityName": "Sorghum", + "creatorName": "jak", + "domainName": "Agriculture", + "subdomainName": "Cereals", + "instituteName": "ICAR–Directorate of Knowledge Management in Agriculture (DKMA)", + "isPublished": 1, + "status": "Published", + "popList": [ + { + "popID": 819, + "season": "Rabi", + "problemType": '"Harvesting Management"', + "problemDetails": '"Method Of Harvesting"', + "startPeriod": "105", + "endPeriod": "120", + "status": "Published", + "description": f'"{OVERVIEW}"', + "applicableSymptoms": [SYMPTOMS], + "applicableCondition": [CONDITIONS], + "advisory": f'"{ADVISORY}"', + } + ], +} + + +def bulk_payload(published_count=205): + payload = copy.deepcopy(SORGHUM_PAYLOAD) + template = payload["popList"][0] + payload["popList"] = [ + { + **copy.deepcopy(template), + "popID": 10000 + index, + "problemDetails": f"Harvesting reference {index}", + "description": f"Published overview {index}", + } + for index in range(published_count) + ] + payload["popList"].insert(100, { + **copy.deepcopy(template), "popID": 90001, "status": "Draft", + "advisory": "WITHHELD-DRAFT-MARKER", + }) + payload["popList"].append({ + **copy.deepcopy(template), "popID": 90002, "status": "Withheld", + "advisory": "WITHHELD-ENTRY-MARKER", + }) + return payload diff --git a/tests/migration_kit.py b/tests/migration_kit.py new file mode 100644 index 0000000..24488e2 --- /dev/null +++ b/tests/migration_kit.py @@ -0,0 +1,23 @@ +"""Optional loader for tests of the workspace migration kit.""" + +from importlib.util import module_from_spec, spec_from_file_location +from pathlib import Path + +import pytest + + +def migration_kit_path(): + path = Path(__file__).resolve().parents[3] / "bh-prod-data" / "migrations" + if not path.is_dir(): + pytest.skip("The workspace migration kit is not present") + return path + + +def load_migration_tool(name): + path = migration_kit_path() / f"{name}.py" + if not path.is_file(): + pytest.skip(f"The workspace migration tool {name} is not present") + spec = spec_from_file_location(name, path) + module = module_from_spec(spec) + spec.loader.exec_module(module) + return module diff --git a/tests/test_activities.py b/tests/test_activities.py index 89a561c..3d7a111 100644 --- a/tests/test_activities.py +++ b/tests/test_activities.py @@ -15,6 +15,31 @@ from unittest.mock import MagicMock import pytest + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_source_first_chunking_keeps_short_advisories( + db_connection, sample_document, monkeypatch, +): + from pipeline.temporal.activities import processing as activities + + workflow_id = sample_document["workflow_id"] + body = "Check the greenhouse vents after the afternoon rain." + db_connection.save_pages(workflow_id, [{"page_number": 1, "original_markdown": body}]) + monkeypatch.setenv("CHUNKING_PROVIDER", "recursive_splitter") + monkeypatch.setenv("CHUNKING_MIN_CHUNK_TOKENS", "100") + monkeypatch.setattr( + activities, "_upload_artifact", + lambda *args: ("minio://test/chunks.json", 1, "application/json"), + ) + + result = await activities.create_chunks_from_db(workflow_id, source_first=True) + + assert result["chunk_count"] == 1 + chunk, = db_connection.get_chunks(workflow_id) + assert chunk["original_text"] == body + assert chunk["page_start"] == chunk["page_end"] == 1 from temporalio.exceptions import ApplicationError os.environ["MINIO_ACCESS_KEY"] = "test-access" @@ -36,7 +61,7 @@ class TestChunkingActivity: @pytest.mark.unit def test_create_chunks_basic(self): """Test basic chunking of pages.""" - from pipeline.activities import create_chunks + from pipeline.temporal.activities.processing import create_chunks pages = [ { @@ -65,7 +90,7 @@ def test_create_chunks_basic(self): @pytest.mark.unit def test_create_chunks_uses_edited_markdown(self): """Test that edited_markdown is preferred over original.""" - from pipeline.activities import create_chunks + from pipeline.temporal.activities.processing import create_chunks pages = [ { @@ -87,7 +112,7 @@ def test_create_chunks_uses_edited_markdown(self): @pytest.mark.unit def test_create_chunks_empty_pages(self): """Test chunking with empty pages.""" - from pipeline.activities import create_chunks + from pipeline.temporal.activities.processing import create_chunks pages = [] chunks = asyncio.run(create_chunks(pages, chunk_size=100, chunk_overlap=20, min_tokens=10)) @@ -96,7 +121,7 @@ def test_create_chunks_empty_pages(self): @pytest.mark.unit def test_create_chunks_min_tokens_filter(self): """Test that chunks below min_tokens are filtered.""" - from pipeline.activities import create_chunks + from pipeline.temporal.activities.processing import create_chunks pages = [ { @@ -120,7 +145,7 @@ class TestPrepareIngestionRecords: @pytest.mark.unit def test_prepare_records_basic(self): """Test preparing records for vector-index ingestion.""" - from pipeline.activities import prepare_ingestion_records + from pipeline.search.passage_projection import prepare_ingestion_records chunks = [ { @@ -158,7 +183,7 @@ def test_prepare_records_basic(self): @pytest.mark.unit def test_prepare_records_carries_instance_name(self): """Payload ships the readable state name beside the instance code.""" - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records chunks = [{"chunk_number": 1, "original_text": "Test chunk", "token_count": 5}] @@ -175,7 +200,7 @@ def test_prepare_records_carries_instance_name(self): @pytest.mark.unit def test_prepare_records_type_carries_kind(self): """Every record's payload type is its kind; document_kind is not written.""" - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records from pipeline.vector_store.qdrant_store import _record_payload chunks = [{"chunk_number": 1, "original_text": "Test chunk", "token_count": 5}] @@ -205,7 +230,7 @@ def test_prepare_records_type_carries_kind(self): @pytest.mark.unit def test_prepare_records_fills_scheme_aliases(self): """A scheme must never index with scheme_aliases: [].""" - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records chunks = [{"chunk_number": 1, "original_text": "Test chunk", "token_count": 5}] @@ -231,7 +256,7 @@ def test_prepare_records_fills_scheme_aliases(self): @pytest.mark.unit def test_prepare_records_non_scheme_has_no_aliases(self): """Plain documents keep the scheme fields absent, as before.""" - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records chunks = [{"chunk_number": 1, "original_text": "Test chunk", "token_count": 5}] @@ -250,7 +275,7 @@ def test_prepare_records_non_scheme_has_no_aliases(self): @pytest.mark.unit def test_prepare_records_excludes_excluded_chunks(self): """Test that excluded chunks are not included in records.""" - from pipeline.activities import prepare_ingestion_records + from pipeline.search.passage_projection import prepare_ingestion_records chunks = [ { @@ -283,7 +308,7 @@ def test_prepare_records_excludes_excluded_chunks(self): @pytest.mark.unit def test_prepare_records_uses_edited_text(self): """Test that edited_text is preferred over original.""" - from pipeline.activities import prepare_ingestion_records + from pipeline.search.passage_projection import prepare_ingestion_records chunks = [ { @@ -317,7 +342,7 @@ class TestPrepareRecordsValidity: def test_stamps_the_period_on_every_chunk(self): # Every chunk, not just the first: the filter runs per point, so a # chunk without dates would outlive its own document. - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records records = _prepare_records( document_id="test-doc", @@ -336,7 +361,7 @@ def test_stamps_the_period_on_every_chunk(self): def test_omits_the_period_when_the_document_has_none(self): # Absent, not null — the search filter's missing-field branch is what # keeps such a chunk answerable. - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records records = _prepare_records( document_id="test-doc", filename="test.pdf", chunks=self.CHUNKS @@ -351,7 +376,7 @@ def test_a_start_with_no_end_stamps_a_start_and_omits_the_end(self): # chunk is searchable from its start day and never expires. The end is # omitted rather than null — the filter's "end_date missing" branch is # what keeps it answerable. - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records records = _prepare_records( document_id="test-doc", @@ -365,7 +390,7 @@ def test_a_start_with_no_end_stamps_a_start_and_omits_the_end(self): @pytest.mark.unit def test_the_period_is_in_the_passage_schema(self): - from pipeline.activities import _passage_schema_field_names + from pipeline.search.passage_projection import _passage_schema_field_names assert {"start_date", "end_date"} <= _passage_schema_field_names() @@ -375,7 +400,7 @@ class TestValidityFieldsFromDoc: @pytest.mark.unit def test_uses_the_stored_period(self): - from pipeline.activities import _validity_fields_from_doc + from pipeline.search.passage_projection import _validity_fields_from_doc fields = _validity_fields_from_doc( {"valid_from": "2026-10-01", "valid_to": "2026-12-31"} @@ -388,7 +413,7 @@ def test_falls_back_to_the_upload_day_for_a_row_with_no_period(self): # A document uploaded before validity existed, being reingested: its # own upload day is a truer start than today, which would date it as # though it arrived this morning. - from pipeline.activities import _validity_fields_from_doc + from pipeline.search.passage_projection import _validity_fields_from_doc fields = _validity_fields_from_doc({"created_at": "2024-05-01T09:30:00"}) @@ -396,13 +421,13 @@ def test_falls_back_to_the_upload_day_for_a_row_with_no_period(self): @pytest.mark.unit def test_falls_back_to_today_when_the_row_has_no_upload_day_either(self): - from datetime import date + from datetime import datetime, timezone - from pipeline.activities import _validity_fields_from_doc + from pipeline.search.passage_projection import _validity_fields_from_doc fields = _validity_fields_from_doc({}) - assert fields["valid_from"] == date.today().isoformat() + assert fields["valid_from"] == datetime.now(timezone.utc).date().isoformat() @pytest.mark.unit @pytest.mark.parametrize( @@ -411,7 +436,7 @@ def test_falls_back_to_today_when_the_row_has_no_upload_day_either(self): def test_always_returns_a_start(self, doc): # The ingest path must never write a period with no start. The end is # optional by design — None means the document never expires. - from pipeline.activities import _validity_fields_from_doc + from pipeline.search.passage_projection import _validity_fields_from_doc fields = _validity_fields_from_doc(doc) @@ -420,7 +445,7 @@ def test_always_returns_a_start(self, doc): @pytest.mark.unit def test_a_stored_open_ended_period_survives(self): - from pipeline.activities import _validity_fields_from_doc + from pipeline.search.passage_projection import _validity_fields_from_doc fields = _validity_fields_from_doc( {"valid_from": "2026-01-01", "valid_to": None} @@ -435,7 +460,7 @@ class TestUpdateDocumentState: @pytest.mark.unit def test_update_state(self, db_connection): """Test updating document state in the database.""" - from pipeline.activities import update_document_state + from pipeline.temporal.activities.state import update_document_state workflow_id = "state-update-test" db_connection.upsert_document( @@ -470,9 +495,9 @@ class TestMinIOClient: """Tests for MinIO client creation.""" @pytest.mark.unit - def test_get_minio_client_requires_credentials(self): + def test_get_object_store_requires_credentials(self): """Test that missing credentials raise error.""" - from pipeline.activities import get_minio_client + from pipeline.temporal.activities.artifacts import get_object_store # Save original values orig_access = os.environ.get("MINIO_ACCESS_KEY") @@ -486,7 +511,7 @@ def test_get_minio_client_requires_credentials(self): del os.environ["MINIO_SECRET_KEY"] with pytest.raises(RuntimeError, match="required"): - get_minio_client() + get_object_store() finally: # Restore if orig_access: @@ -495,14 +520,14 @@ def test_get_minio_client_requires_credentials(self): os.environ["MINIO_SECRET_KEY"] = orig_secret @pytest.mark.unit - def test_get_minio_client_with_credentials(self): + def test_get_object_store_with_credentials(self): """Test MinIO client creation with credentials.""" - from pipeline.activities import get_minio_client + from pipeline.temporal.activities.artifacts import get_object_store os.environ["MINIO_ACCESS_KEY"] = "test-key" os.environ["MINIO_SECRET_KEY"] = "test-secret" - client = get_minio_client() + client = get_object_store() assert client is not None @@ -513,7 +538,7 @@ class TestOCRActivity: @pytest.mark.asyncio async def test_run_ocr_calls_provider(self, monkeypatch, tmp_path): """Test that OCR activity delegates PDF OCR to the OCR service.""" - import pipeline.activities as activities + from pipeline.temporal.activities import source as activities pdf_path = tmp_path / "test.pdf" pdf_path.write_bytes(b"%PDF-1.4 test content") @@ -547,10 +572,45 @@ class TestPublishCatalogToNetworkActivity: def _publish_enabled(self, monkeypatch): monkeypatch.setenv("ENABLE_PUBLISH_TO_NETWORK", "true") + @pytest.mark.unit + @pytest.mark.asyncio + async def test_activity_accepts_a_publisher_without_a_beckn_envelope(self, monkeypatch): + from pipeline.temporal.activities import publication as activities + import pipeline.db as db + from pipeline.network.network_publisher import PublicationReceipt + + class TestPublisher: + def publish(self, transaction_id, document_kind, workflow_id, group): + assert (transaction_id, document_kind, workflow_id, group) == ("txn", "advisory", "wf", "mh") + return PublicationReceipt( + exchange={"receipt": "local-test"}, message_id="delivery-1", result_status="ACCEPTED" + ) + + monkeypatch.setattr(activities, "get_network_publisher", TestPublisher) + monkeypatch.setattr(db, "get_document", lambda _: {"kind": "advisory", "instance": "mh"}) + monkeypatch.setattr(db, "get_latest_document_job", lambda _: None) + monkeypatch.setattr(activities, "_upload_artifact", lambda *a, **k: ("minio://documents/receipt", 10, "application/json")) + recorded = {} + monkeypatch.setattr(db, "add_document_artifact", lambda **kwargs: recorded.update(kwargs)) + + outcome = await activities.publish_catalog_to_network("wf", "txn") + + assert outcome["message_id"] == "delivery-1" + assert outcome["status"] == "published" + assert recorded["metadata"] == {"receipt": "local-test", "result_status": "ACCEPTED", "context_matches": False} + + @pytest.mark.unit + def test_factory_rejects_an_unknown_publisher(self, monkeypatch): + from pipeline.network.network_publisher import PublishConfigError, get_network_publisher + + monkeypatch.setenv("NETWORK_PUBLISHER", "unconfigured") + with pytest.raises(PublishConfigError, match="Unsupported NETWORK_PUBLISHER"): + get_network_publisher() + @pytest.mark.unit @pytest.mark.asyncio async def test_publish_records_artifact_on_success(self, monkeypatch): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities import pipeline.db as db fake_result = { @@ -583,10 +643,10 @@ def publish(self, transaction_id, kind, workflow_id=None, group=None): assert workflow_id == "wf-1" return fake_result - monkeypatch.setattr(activities, "DiscoveryPublishService", FakeService) + monkeypatch.setattr("pipeline.network.adapters.beckn.DiscoveryPublishService", lambda **_: FakeService()) monkeypatch.setattr( activities, - "_upload_file_to_minio", + "_upload_artifact", lambda *a, **k: ("minio://documents/network_publish.json", 123, "application/json"), ) monkeypatch.setattr(db, "get_document", lambda workflow_id: {"kind": "advisory"}) @@ -622,14 +682,14 @@ def fake_add_artifact(**kwargs): @pytest.mark.unit @pytest.mark.asyncio async def test_publish_failure_propagates_and_records_nothing(self, monkeypatch): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities import pipeline.db as db class FakeService: def publish(self, transaction_id, kind, workflow_id=None, group=None): raise RuntimeError("discovery service unreachable") - monkeypatch.setattr(activities, "DiscoveryPublishService", FakeService) + monkeypatch.setattr("pipeline.network.adapters.beckn.DiscoveryPublishService", lambda **_: FakeService()) monkeypatch.setattr(db, "get_document", lambda workflow_id: {"kind": "advisory"}) add_artifact_calls = [] @@ -647,7 +707,7 @@ def publish(self, transaction_id, kind, workflow_id=None, group=None): async def test_missing_publish_config_fails_without_retry(self, monkeypatch, unset): from temporalio.exceptions import ApplicationError - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities import pipeline.db as db monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") @@ -663,7 +723,7 @@ async def test_missing_publish_config_fails_without_retry(self, monkeypatch, uns @pytest.mark.unit @pytest.mark.asyncio async def test_unmapped_kind_records_a_skipped_artifact(self, monkeypatch): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities import pipeline.db as db class FakeService: @@ -677,10 +737,10 @@ def publish(self, transaction_id, kind, workflow_id=None, group=None): "errors": [], } - monkeypatch.setattr(activities, "DiscoveryPublishService", FakeService) + monkeypatch.setattr("pipeline.network.adapters.beckn.DiscoveryPublishService", lambda **_: FakeService()) monkeypatch.setattr( activities, - "_upload_file_to_minio", + "_upload_artifact", lambda *a, **k: ("minio://documents/network_publish_skipped.json", 45, "application/json"), ) monkeypatch.setattr(db, "get_document", lambda workflow_id: {"kind": "document"}) @@ -711,7 +771,7 @@ def publish(self, transaction_id, kind, workflow_id=None, group=None): @pytest.mark.asyncio @pytest.mark.parametrize("flag", [None, "false", "0", "no", ""]) async def test_disabled_publish_records_a_skip_without_calling_the_network(self, monkeypatch, flag): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities import pipeline.db as db if flag is None: @@ -720,10 +780,10 @@ async def test_disabled_publish_records_a_skip_without_calling_the_network(self, monkeypatch.setenv("ENABLE_PUBLISH_TO_NETWORK", flag) monkeypatch.delenv("DISCOVERY_SERVICE_ENDPOINT", raising=False) monkeypatch.delenv("NETWORK_SENDER_ID", raising=False) - monkeypatch.setattr(activities, "DiscoveryPublishService", lambda: pytest.fail("must not publish")) + monkeypatch.setattr("pipeline.network.adapters.beckn.DiscoveryPublishService", lambda **_: pytest.fail("must not publish")) monkeypatch.setattr( activities, - "_upload_file_to_minio", + "_upload_artifact", lambda *a, **k: ("minio://documents/network_publish_skipped.json", 45, "application/json"), ) monkeypatch.setattr(db, "get_document", lambda workflow_id: {"kind": "advisory"}) @@ -743,7 +803,7 @@ async def test_disabled_publish_records_a_skip_without_calling_the_network(self, @pytest.mark.asyncio @pytest.mark.parametrize("result_status", ["ACCEPTED", "PARTIAL", None]) async def test_non_rejected_results_complete_the_stage(self, monkeypatch, result_status): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities import pipeline.db as db class FakeService: @@ -757,10 +817,10 @@ def publish(self, transaction_id, kind, workflow_id=None, group=None): "errors": [], } - monkeypatch.setattr(activities, "DiscoveryPublishService", FakeService) + monkeypatch.setattr("pipeline.network.adapters.beckn.DiscoveryPublishService", lambda **_: FakeService()) monkeypatch.setattr( activities, - "_upload_file_to_minio", + "_upload_artifact", lambda *a, **k: ("minio://documents/network_publish.json", 12, "application/json"), ) monkeypatch.setattr(db, "get_document", lambda workflow_id: {"kind": "advisory"}) @@ -775,7 +835,7 @@ def publish(self, transaction_id, kind, workflow_id=None, group=None): @pytest.mark.unit @pytest.mark.asyncio async def test_rejected_result_records_the_errors_then_fails(self, monkeypatch): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities import pipeline.db as db errors = [{"code": "SCH_SCHEMA_VALIDATION_FAILED", "message": "topics is required"}] @@ -791,10 +851,10 @@ def publish(self, transaction_id, kind, workflow_id=None, group=None): "errors": errors, } - monkeypatch.setattr(activities, "DiscoveryPublishService", FakeService) + monkeypatch.setattr("pipeline.network.adapters.beckn.DiscoveryPublishService", lambda **_: FakeService()) monkeypatch.setattr( activities, - "_upload_file_to_minio", + "_upload_artifact", lambda *a, **k: ("minio://documents/network_publish.json", 12, "application/json"), ) monkeypatch.setattr(db, "get_document", lambda workflow_id: {"kind": "advisory"}) @@ -819,7 +879,7 @@ def _publish_enabled(self, monkeypatch): @staticmethod def _wire(monkeypatch, document, service_result, seen): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities import pipeline.db as db class FakeService: @@ -827,10 +887,10 @@ def publish(self, transaction_id, document_kind, workflow_id=None, group=None): seen["group"] = group return service_result - monkeypatch.setattr(activities, "DiscoveryPublishService", FakeService) + monkeypatch.setattr("pipeline.network.adapters.beckn.DiscoveryPublishService", lambda **_: FakeService()) monkeypatch.setattr( activities, - "_upload_file_to_minio", + "_upload_artifact", lambda *a, **k: ("minio://documents/x.json", 1, "application/json"), ) monkeypatch.setattr(db, "get_document", lambda workflow_id: document) @@ -852,9 +912,9 @@ def publish(self, transaction_id, document_kind, workflow_id=None, group=None): @pytest.mark.asyncio @pytest.mark.parametrize("instance,group", [("mh", "mh"), ("MH", "mh"), ("bv", None), ("default", None)]) async def test_passes_the_documents_group(self, monkeypatch, instance, group): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities - monkeypatch.delenv("PORTAL_INSTANCE", raising=False) + monkeypatch.setenv("PORTAL_INSTANCE", "bv") monkeypatch.delenv("DEFAULT_INSTANCE", raising=False) seen = {} self._wire( @@ -871,7 +931,7 @@ async def test_passes_the_documents_group(self, monkeypatch, instance, group): @pytest.mark.unit @pytest.mark.asyncio async def test_unmapped_kind_records_a_skip(self, monkeypatch): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities skipped = { "skipped": True, @@ -899,15 +959,15 @@ async def test_unmapped_kind_records_a_skip(self, monkeypatch): @pytest.mark.unit @pytest.mark.asyncio async def test_an_invalid_provider_id_fails_without_retry_or_artifact(self, monkeypatch): - import pipeline.activities as activities + from pipeline.temporal.activities import publication as activities import pipeline.db as db - from pipeline.network_provider import InvalidProviderIdError + from pipeline.network.network_provider import InvalidProviderIdError class FakeService: def publish(self, transaction_id, document_kind, workflow_id=None, group=None): raise InvalidProviderIdError("Provider id 'advisory.ladakh_ut.provider.oan.dev' is invalid") - monkeypatch.setattr(activities, "DiscoveryPublishService", FakeService) + monkeypatch.setattr("pipeline.network.adapters.beckn.DiscoveryPublishService", lambda **_: FakeService()) monkeypatch.setattr(db, "get_document", lambda workflow_id: {"kind": "advisory", "instance": "ladakh_ut"}) recorded = {} monkeypatch.setattr(db, "add_document_artifact", lambda **kwargs: recorded.update(kwargs)) @@ -921,7 +981,7 @@ def publish(self, transaction_id, document_kind, workflow_id=None, group=None): class TestWorkerActivityRegistration: """ - Every @activity.defn in activities.py must be registered on the worker + Every @activity.defn in the Temporal activities package must be registered on the worker (pipeline/worker.py), or Temporal fails at runtime with a NotFoundError when a workflow tries to run it - a silent gap that unit tests calling activity functions directly (like the tests above) can't catch on their @@ -935,12 +995,20 @@ def test_every_defined_activity_is_registered_on_the_worker(self): from temporalio.activity import _Definition - from pipeline import activities, worker + from importlib import import_module + from pkgutil import iter_modules + from pipeline.temporal import activities, worker + + modules = [ + import_module(f"{activities.__name__}.{entry.name}") + for entry in iter_modules(activities.__path__) + ] defined_activities = { name - for name, obj in vars(activities).items() - if inspect.isfunction(obj) and _Definition.from_callable(obj) + for module in modules + for name, obj in vars(module).items() + if inspect.isfunction(obj) and obj.__module__ == module.__name__ and _Definition.from_callable(obj) } worker_source = inspect.getsource(worker) @@ -983,7 +1051,7 @@ def _document(db): @pytest.mark.db @pytest.mark.unit async def test_old_keys_are_deleted_and_forgotten(self, db_connection): - from pipeline import activities + from pipeline.temporal.activities import indexing as activities doc_uuid = self._document(db_connection) store = self._Store() @@ -999,7 +1067,7 @@ async def test_old_keys_are_deleted_and_forgotten(self, db_connection): @pytest.mark.db @pytest.mark.unit async def test_a_failed_delete_keeps_the_key_for_the_next_run(self, db_connection): - from pipeline import activities + from pipeline.temporal.activities import indexing as activities doc_uuid = self._document(db_connection) store = self._Store(failing={"a" * 32}) @@ -1014,7 +1082,7 @@ async def test_a_failed_delete_keeps_the_key_for_the_next_run(self, db_connectio @pytest.mark.db @pytest.mark.unit async def test_a_key_another_live_document_is_indexed_under_is_kept_and_remembered(self, db_connection): - from pipeline import activities + from pipeline.temporal.activities import indexing as activities doc_uuid = self._document(db_connection) # A second document is indexed under the very key the first one replaced. diff --git a/tests/test_activity_logging.py b/tests/test_activity_logging.py index 915aa2e..5e53fdd 100644 --- a/tests/test_activity_logging.py +++ b/tests/test_activity_logging.py @@ -12,7 +12,7 @@ from pipeline.temporal.activities import logging as activity_logging -LOGGER = "pipeline.temporal.activities.logging" +LOGGER = "pipeline.activity_logging" def _interceptor(next_result=None, error=None): diff --git a/tests/test_advisory_activities.py b/tests/test_advisory_activities.py deleted file mode 100644 index 0ac0dd2..0000000 --- a/tests/test_advisory_activities.py +++ /dev/null @@ -1,126 +0,0 @@ -"""Tests for the advisory index/promote activities, the PROD target and the workflow exports.""" - -import asyncio -import copy -from pathlib import Path - -import pytest - -from pipeline import advisory_index -from tests.test_advisory_chunks import SORGHUM_PAYLOAD -from tests.test_advisory_index import FakeStore - - -def _seed(db): - db.upsert_webhook_records("webhook", [copy.deepcopy(SORGHUM_PAYLOAD)]) - with db.get_connection() as conn: - return str(conn.execute("SELECT id FROM webhook_records").fetchone()["id"]) - - -def _audit_actions(db): - with db.get_connection() as conn: - return [r["action_type"] for r in conn.execute("SELECT action_type FROM audit_logs")] - - -@pytest.fixture -def qdrant_env(monkeypatch): - monkeypatch.setenv("VECTOR_DB_COLLECTION_NAME", "document-data") - monkeypatch.delenv("PROD_VECTOR_DB_URL", raising=False) - monkeypatch.setattr("pipeline.vector_store.embeddings.count_tokens", lambda text: 250) - - -@pytest.mark.unit -@pytest.mark.db -class TestActivities: - def test_index_activity_writes_to_the_dev_collection_and_audits(self, db_connection, qdrant_env, monkeypatch): - from pipeline.activities import index_webhook_advisory - - rid = _seed(db_connection) - store = FakeStore() - monkeypatch.setattr("pipeline.vector_store.get_vector_store", lambda: store) - result = asyncio.run(index_webhook_advisory(rid)) - assert result["status"] == "indexed" - assert store.calls[0][1] == "document-data" - assert db_connection.get_unindexed_webhook_record_ids() == [] - assert "advisory_indexed" in _audit_actions(db_connection) - - def test_promote_activity_writes_to_prod_and_marks_promoted(self, db_connection, qdrant_env, monkeypatch): - from pipeline.activities import promote_webhook_advisory - - rid = _seed(db_connection) - prod = FakeStore() - monkeypatch.setattr(advisory_index, "prod_target", lambda: (prod, "prod-collection")) - result = asyncio.run(promote_webhook_advisory(rid)) - assert result["status"] == "indexed" - assert prod.calls[0][1] == "prod-collection" - assert db_connection.advisory_status_counts()["promoted"] == 1 - assert "advisory_promoted" in _audit_actions(db_connection) - - def test_promote_activity_requires_a_prod_target(self, db_connection, qdrant_env, monkeypatch): - from pipeline.activities import promote_webhook_advisory - - monkeypatch.setattr(advisory_index, "prod_target", lambda: None) - with pytest.raises(RuntimeError, match="PROD_VECTOR_DB_URL"): - asyncio.run(promote_webhook_advisory("any")) - - def test_dev_index_of_unpublished_record_also_clears_prod(self, db_connection, qdrant_env, monkeypatch): - from pipeline.activities import index_webhook_advisory - - payload = copy.deepcopy(SORGHUM_PAYLOAD) - payload["isPublished"] = 0 - db_connection.upsert_webhook_records("webhook", [payload]) - with db_connection.get_connection() as conn: - rid = str(conn.execute("SELECT id FROM webhook_records").fetchone()["id"]) - dev, prod = FakeStore(), FakeStore() - monkeypatch.setattr("pipeline.vector_store.get_vector_store", lambda: dev) - monkeypatch.setattr(advisory_index, "prod_target", lambda: (prod, "prod-collection")) - result = asyncio.run(index_webhook_advisory(rid)) - assert result["status"] == "deleted" - assert prod.calls == [("delete", "prod-collection", rid)] - - def test_dev_index_survives_a_broken_prod_config(self, db_connection, qdrant_env, monkeypatch): - from pipeline.activities import index_webhook_advisory - - rid = _seed(db_connection) - monkeypatch.setattr("pipeline.vector_store.get_vector_store", lambda: FakeStore()) - - def broken(): - raise ValueError("QDRANT_API_KEY is required for remote Qdrant") - - monkeypatch.setattr(advisory_index, "prod_target", broken) - assert asyncio.run(index_webhook_advisory(rid))["status"] == "indexed" - - -class TestProdTarget: - def test_none_without_prod_url(self, monkeypatch): - monkeypatch.delenv("PROD_VECTOR_DB_URL", raising=False) - assert advisory_index.prod_target() is None - - def test_uses_prod_env(self, monkeypatch): - monkeypatch.setenv("PROD_VECTOR_DB_URL", "http://prod:6333") - monkeypatch.setenv("PROD_VECTOR_DB_API_KEY", "k") - monkeypatch.setenv("PROD_VECTOR_DB_COLLECTION_NAME", "prod-docs") - seen = {} - - def fake_client(url=None, api_key=None): - seen.update(url=url, api_key=api_key) - return object() - - monkeypatch.setattr("pipeline.vector_store.qdrant_store.get_qdrant_client", fake_client) - target = advisory_index.prod_target() - assert target is not None and target[1] == "prod-docs" - assert seen == {"url": "http://prod:6333", "api_key": "k"} - - -def test_workflows_are_defined_and_registered(): - from pipeline.workflows import AdvisoryIndexWorkflow, AdvisoryPromoteWorkflow - - assert callable(AdvisoryIndexWorkflow.run) and callable(AdvisoryPromoteWorkflow.run) - source = (Path(__file__).resolve().parent.parent / "pipeline" / "worker.py").read_text() - for name in ( - "AdvisoryIndexWorkflow", - "AdvisoryPromoteWorkflow", - "index_webhook_advisory", - "promote_webhook_advisory", - ): - assert source.count(name) >= 2, name # imported and registered diff --git a/tests/test_advisory_api.py b/tests/test_advisory_api.py deleted file mode 100644 index f045998..0000000 --- a/tests/test_advisory_api.py +++ /dev/null @@ -1,175 +0,0 @@ -"""API tests: /webhook starts advisory workflows; admin backfill / promote / status endpoints.""" - -import os -from unittest.mock import AsyncMock, patch - -import pytest - -TEST_KEY = "0123456789abcdef0123456789abcdef" -H = {"x-api-key": TEST_KEY} - - -def _rec(content_id, published=1, status="Published"): - return { - "contentID": content_id, - "title": f"Advisory {content_id}", - "status": status, - "isPublished": published, - "popList": [{"popID": 1, "status": "Published"}], - } - - -@pytest.fixture -def client(db_connection, monkeypatch): - from fastapi.testclient import TestClient - - from pipeline.api import api - - monkeypatch.setattr(api.limiter, "enabled", False) # many posts per test process - with patch.dict(os.environ, {"WEBHOOK_API_KEY": TEST_KEY, "AUTH_DISABLED": "true"}): - yield TestClient(api.app) - - -@pytest.fixture -def temporal(monkeypatch): - from pipeline.api import api - - mock = AsyncMock() - monkeypatch.setattr(api, "temporal_client", mock) - return mock - - -def _row_ids(db, *content_ids): - with db.get_connection() as conn: - return { - c: str(conn.execute( - "SELECT id FROM webhook_records WHERE external_id = %s", (str(c),) - ).fetchone()["id"]) - for c in content_ids - } - - -def _started(mock): - return sorted(call.kwargs["id"] for call in mock.start_workflow.await_args_list) - - -@pytest.mark.api -@pytest.mark.unit -class TestWebhookStartsIndexing: - def test_one_workflow_per_record_in_the_call(self, client, db_connection, temporal): - r = client.post("/webhook", json=[_rec(1), _rec(2)], headers=H) - assert r.status_code == 200 - assert r.json() == { - "success": True, - "message": "Webhook processed successfully.", - "recordCount": 2, - } - ids = _row_ids(db_connection, 1, 2) - assert _started(temporal) == sorted(f"advisory-{i}" for i in ids.values()) - - def test_only_records_in_the_call_are_started(self, client, db_connection, temporal): - db_connection.upsert_webhook_records("webhook", [_rec(1)]) # existing, unindexed - client.post("/webhook", json=[_rec(2)], headers=H) - ids = _row_ids(db_connection, 1, 2) - assert _started(temporal) == [f"advisory-{ids[2]}"] - - def test_existing_never_indexed_row_is_started_when_resent(self, client, db_connection, temporal): - db_connection.upsert_webhook_records("webhook", [_rec(1)]) - client.post("/webhook", json=[_rec(1)], headers=H) - ids = _row_ids(db_connection, 1) - assert _started(temporal) == [f"advisory-{ids[1]}"] - - def test_nothing_started_when_already_indexed(self, client, db_connection, temporal): - client.post("/webhook", json=[_rec(1)], headers=H) - rid = _row_ids(db_connection, 1)[1] - db_connection.mark_webhook_record_indexed(rid, db_connection.get_webhook_record(rid)["payload_hash"]) - temporal.start_workflow.reset_mock() - r = client.post("/webhook", json=[_rec(1)], headers=H) - assert r.status_code == 200 - temporal.start_workflow.assert_not_called() - - def test_still_200_when_temporal_fails(self, client, db_connection, temporal): - temporal.start_workflow.side_effect = RuntimeError("temporal down") - r = client.post("/webhook", json=[_rec(1)], headers=H) - assert r.status_code == 200 - assert len(db_connection.get_unindexed_webhook_record_ids()) == 1 - - def test_still_200_when_temporal_client_is_none(self, client, db_connection, monkeypatch): - from pipeline.api import api - - monkeypatch.setattr(api, "temporal_client", None) - assert client.post("/webhook", json=[_rec(1)], headers=H).status_code == 200 - - -@pytest.mark.api -@pytest.mark.unit -class TestAdminEndpoints: - def _seed(self, client, temporal, *records): - client.post("/webhook", json=list(records), headers=H) - temporal.start_workflow.reset_mock() - - def test_reindex_queues_all_unindexed_by_default(self, client, db_connection, temporal): - self._seed(client, temporal, _rec(1), _rec(2)) - r = client.post("/webhook/advisory/reindex", json={}) - assert r.status_code == 200 and r.json() == {"success": True, "queued": 2} - assert len(_started(temporal)) == 2 - - def test_reindex_force_queues_indexed_rows_too(self, client, db_connection, temporal): - self._seed(client, temporal, _rec(1)) - rid = _row_ids(db_connection, 1)[1] - db_connection.mark_webhook_record_indexed(rid, db_connection.get_webhook_record(rid)["payload_hash"]) - assert client.post("/webhook/advisory/reindex", json={}).json()["queued"] == 0 - assert client.post("/webhook/advisory/reindex", json={"force": True}).json()["queued"] == 1 - - def test_reindex_explicit_ids(self, client, db_connection, temporal): - self._seed(client, temporal, _rec(1), _rec(2)) - rid = _row_ids(db_connection, 2)[2] - r = client.post("/webhook/advisory/reindex", json={"record_ids": [rid]}) - assert r.json()["queued"] == 1 - assert _started(temporal) == [f"advisory-{rid}"] - - def test_promote_defaults_to_published_records_indexed_in_dev(self, client, db_connection, temporal): - self._seed(client, temporal, _rec(1), _rec(2, published=0, status="Draft")) - ids = _row_ids(db_connection, 1, 2) - for rid in ids.values(): - db_connection.mark_webhook_record_indexed(rid, db_connection.get_webhook_record(rid)["payload_hash"]) - temporal.start_workflow.reset_mock() - r = client.post("/webhook/advisory/promote", json={}) - assert r.json() == {"success": True, "queued": 1} - assert _started(temporal) == [f"advisory-promote-{ids[1]}"] - - def test_default_promote_skips_records_not_indexed_in_dev(self, client, db_connection, temporal): - self._seed(client, temporal, _rec(1)) - temporal.start_workflow.reset_mock() - r = client.post("/webhook/advisory/promote", json={}) - assert r.json() == {"success": True, "queued": 0} - assert _started(temporal) == [] - - def test_promote_explicit_ids(self, client, db_connection, temporal): - self._seed(client, temporal, _rec(1), _rec(2)) - rid = _row_ids(db_connection, 2)[2] - r = client.post("/webhook/advisory/promote", json={"record_ids": [rid]}) - assert r.json()["queued"] == 1 - assert _started(temporal) == [f"advisory-promote-{rid}"] - - def test_status_returns_counts(self, client, db_connection, temporal): - self._seed(client, temporal, _rec(1), _rec(2, published=0, status="Draft")) - r = client.get("/webhook/advisory/status") - assert r.status_code == 200 - assert r.json() == {"total": 2, "published": 1, "indexed": 0, "unindexed": 2, "promoted": 0} - - def test_endpoints_require_a_signed_in_admin(self, db_connection, temporal, monkeypatch): - from fastapi.testclient import TestClient - - from pipeline.api import api - - env = { - "AUTH_DISABLED": "false", - "KEYCLOAK_ISSUER": "https://example.com/realms/test", - "KEYCLOAK_JWKS_URL": "https://example.com/realms/test/protocol/openid-connect/certs", - } - with patch.dict(os.environ, env): - c = TestClient(api.app) - assert c.post("/webhook/advisory/reindex", json={}).status_code == 401 - assert c.post("/webhook/advisory/promote", json={}).status_code == 401 - assert c.get("/webhook/advisory/status").status_code == 401 diff --git a/tests/test_advisory_chunks.py b/tests/test_advisory_chunks.py deleted file mode 100644 index 016d2c6..0000000 --- a/tests/test_advisory_chunks.py +++ /dev/null @@ -1,218 +0,0 @@ -"""Tests for the webhook advisory chunk builder.""" - -import copy -import hashlib - -import pytest - -from pipeline.advisory_chunks import build_chunks, clean_value - -HEADER = ( - "Sorghum- Harvesting Management\n" - "Crop: Sorghum | Problem type: Harvesting Management | Season: Rabi" -) -OVERVIEW = ( - "Timely harvesting of AKJ-1 helps preserve grain appearance, processing quality and " - "suitability for flaking. Proper drying and handling after harvest are essential to " - "minimize grain damage and deterioration." -) -SYMPTOMS = ( - "Panicles become mature and dry, grains become hard and attain characteristic colour, " - "and most leaves begin to dry." -) -CONDITIONS = ( - "Crop has reached physiological maturity and panicles are fully developed; grains are " - "firm and suitable for flaking quality." -) -ADVISORY = ( - "Harvest AKJ-1 at physiological maturity when the grains are fully developed and firm. " - "Timely harvesting is important for maintaining grain quality required for flaking. " - "Avoid delayed harvesting, which may increase grain weathering, shattering and mould " - "risk. After harvesting, dry the panicles properly, thresh carefully, clean the grain " - "and reduce moisture to a safe level before storage or processing." -) - -SORGHUM_PAYLOAD = { - "contentID": 987, - "title": "Sorghum- Harvesting Management", - "commodityName": "Sorghum", - "creatorName": "jak", - "domainName": "Agriculture", - "subdomainName": "Cereals", - "instituteName": "ICAR–Directorate of Knowledge Management in Agriculture (DKMA)", - "isPublished": 1, - "status": "Published", - "popList": [ - { - "popID": 819, - "season": "Rabi", - "problemType": '"Harvesting Management"', - "problemDetails": '"Method Of Harvesting"', - "startPeriod": "105", - "endPeriod": "120", - "status": "Published", - "description": f'"{OVERVIEW}"', - "applicableSymptoms": [SYMPTOMS], - "applicableCondition": [CONDITIONS], - "advisory": f'"{ADVISORY}"', - } - ], -} - - -def make_row(payload=None, **overrides): - payload = copy.deepcopy(SORGHUM_PAYLOAD if payload is None else payload) - row = { - "id": "row-1", - "external_id": "987", - "title": payload.get("title", ""), - "slug": "987/sorghum--harvesting-management", - "status": payload.get("status"), - "is_published": payload.get("isPublished"), - "payload": payload, - "payload_hash": "hash-1", - "updated_at": "2026-10-16T09:15:10.000Z", - } - row.update(overrides) - return row - - -def long_when_combined(text: str) -> int: - """Over the limit only when the entry carries more than one question-side field.""" - return 600 if "Overview: " in text and "Symptoms: " in text else 300 - - -def pop_of(row): - return row["payload"]["popList"][0] - - -class TestCleanValue: - def test_strips_wrapping_quotes(self): - assert clean_value('"Aphids"') == "Aphids" - assert clean_value('""') == "" - - def test_handles_none_numbers_and_whitespace(self): - assert clean_value(None) == "" - assert clean_value(819) == "819" - assert clean_value(" x ") == "x" - - -class TestShortEntry: - def test_short_entry_is_one_full_chunk(self): - chunks = build_chunks(make_row(), lambda t: 250) - assert len(chunks) == 1 - assert chunks[0]["section"] == "full" - assert chunks[0]["text"] == "\n".join( - [ - HEADER, - f"Overview: {OVERVIEW}", - f"Symptoms: {SYMPTOMS}", - f"Conditions: {CONDITIONS}", - f"Advisory: {ADVISORY}", - ] - ) - assert chunks[0]["text_for_embedding"] == chunks[0]["text"] - - -class TestLongEntry: - def test_long_entry_splits_into_three_chunks_each_with_advice(self): - chunks = build_chunks(make_row(), long_when_combined) - assert [c["section"] for c in chunks] == ["overview", "symptoms", "conditions"] - for chunk in chunks: - assert chunk["text"].startswith(HEADER) - assert chunk["text"].endswith(f"Advisory: {ADVISORY}") - assert "Symptoms:" not in chunks[0]["text"] - assert f"Symptoms: {SYMPTOMS}" in chunks[1]["text"] - assert [c["chunk_num"] for c in chunks] == [0, 1, 2] - - def test_empty_conditions_gives_two_chunks(self): - row = make_row() - pop_of(row)["applicableCondition"] = [] - chunks = build_chunks(row, long_when_combined) - assert [c["section"] for c in chunks] == ["overview", "symptoms"] - - def test_no_advisory_gives_one_chunk_per_field(self): - row = make_row() - pop_of(row)["advisory"] = None - chunks = build_chunks(row, lambda t: 600) - assert [c["section"] for c in chunks] == ["overview", "symptoms", "conditions"] - for chunk in chunks: - assert "Advisory:" not in chunk["text"] - - def test_only_advisory_is_one_advisory_chunk(self): - row = make_row() - pop = pop_of(row) - pop["description"] = None - pop["applicableSymptoms"] = [] - pop["applicableCondition"] = [] - chunks = build_chunks(row, lambda t: 600) - assert [c["section"] for c in chunks] == ["advisory"] - assert chunks[0]["text"] == f"{HEADER}\nAdvisory: {ADVISORY}" - - -class TestPublishingAndShape: - @pytest.mark.parametrize( - "overrides", - [{"is_published": 0}, {"status": "Draft"}], - ) - def test_unpublished_record_returns_empty(self, overrides): - assert build_chunks(make_row(**overrides), lambda t: 250) == [] - - def test_unpublished_entry_is_skipped(self): - row = make_row() - row["payload"]["popList"].append( - {**copy.deepcopy(pop_of(row)), "popID": 820, "status": "Draft"} - ) - chunks = build_chunks(row, lambda t: 250) - assert [c["pop_id"] for c in chunks] == [819] - - @pytest.mark.parametrize("pop_list", [None, [], "nope"]) - def test_missing_or_empty_poplist_returns_empty(self, pop_list): - row = make_row() - row["payload"]["popList"] = pop_list - assert build_chunks(row, lambda t: 250) == [] - - def test_ids_are_stable_and_unique(self): - first = build_chunks(make_row(), long_when_combined) - second = build_chunks(make_row(), long_when_combined) - assert [c["_id"] for c in first] == [c["_id"] for c in second] - assert len({c["_id"] for c in first}) == 3 - full = build_chunks(make_row(), lambda t: 250)[0] - expected = hashlib.md5(b"row-1:819:full").hexdigest() - assert full["_id"] == expected - assert full["chunk_id"] == expected - - def test_metadata_fields(self): - chunk = build_chunks(make_row(), lambda t: 250)[0] - assert "document_kind" not in chunk - assert chunk["type"] == "advisory" - assert chunk["source"] == "webhook" - assert chunk["workflow_id"] == "" - assert chunk["doc_id"] == "row-1" - assert chunk["external_id"] == "987" - assert chunk["pop_id"] == 819 - assert chunk["title_en"] == "Sorghum- Harvesting Management" - assert chunk["slug"] == "987/sorghum--harvesting-management" - assert chunk["crop"] == "Sorghum" - assert chunk["domain"] == "Agriculture" - assert chunk["subdomain"] == "Cereals" - assert chunk["institute"].startswith("ICAR") - assert chunk["creator"] == "jak" - assert chunk["problem_type"] == "Harvesting Management" - assert chunk["problem"] == "Method Of Harvesting" - assert chunk["season"] == "Rabi" - assert chunk["start_period"] == "105" - assert chunk["end_period"] == "120" - assert chunk["updated_at"] == "2026-10-16T09:15:10.000Z" - - def test_null_values_never_appear_as_text(self): - row = make_row() - pop = pop_of(row) - pop["description"] = None - pop["season"] = None - pop["applicableSymptoms"] = [None, "x"] - chunks = build_chunks(row, lambda t: 250) - assert len(chunks) == 1 - assert "None" not in chunks[0]["text"] - assert "Season:" not in chunks[0]["text"] - assert "Symptoms: x" in chunks[0]["text"] diff --git a/tests/test_advisory_index.py b/tests/test_advisory_index.py deleted file mode 100644 index 440ee04..0000000 --- a/tests/test_advisory_index.py +++ /dev/null @@ -1,153 +0,0 @@ -"""Tests for indexing one webhook advisory record into a vector store.""" - -import copy - -import pytest - -from pipeline.advisory_chunks import build_chunks -from pipeline.advisory_index import index_advisory_record -from pipeline.vector_store.qdrant_store import _record_payload -from tests.test_advisory_chunks import SORGHUM_PAYLOAD, make_row - - -class FakeStore: - def __init__(self, delete_result=None, fail_upsert=False): - self.calls = [] - self.records = [] - self._delete_result = delete_result or {"deleted": -1} - self._fail_upsert = fail_upsert - - def delete_by_doc_id(self, name, doc_id): - self.calls.append(("delete", name, doc_id)) - return self._delete_result - - def upsert(self, name, records, batch_size=32): - self.calls.append(("upsert", name, len(records))) - if self._fail_upsert: - raise RuntimeError("qdrant down") - self.records = list(records) - return {"records_ingested": len(records)} - - -def _store_row(db, payload): - db.upsert_webhook_records("webhook", [payload]) - with db.get_connection() as conn: - return str(conn.execute( - "SELECT id FROM webhook_records WHERE external_id = %s", (str(payload["contentID"]),) - ).fetchone()["id"]) - - -def _count(_text): - return 250 - - -@pytest.mark.unit -@pytest.mark.db -class TestIndexAdvisoryRecord: - def test_published_record_is_deleted_then_upserted_then_marked(self, db_connection): - db = db_connection - rid = _store_row(db, copy.deepcopy(SORGHUM_PAYLOAD)) - store = FakeStore() - result = index_advisory_record(rid, store, "document-data", _count) - assert [c[0] for c in store.calls] == ["delete", "upsert"] - assert store.calls[0][1:] == ("document-data", rid) - assert result == {"record_id": rid, "status": "indexed", "chunks": 1} - assert db.get_unindexed_webhook_record_ids() == [] - - def test_unpublished_record_only_deletes(self, db_connection): - db = db_connection - payload = copy.deepcopy(SORGHUM_PAYLOAD) - payload["isPublished"] = 0 - rid = _store_row(db, payload) - store = FakeStore() - result = index_advisory_record(rid, store, "document-data", _count) - assert [c[0] for c in store.calls] == ["delete"] - assert result["status"] == "deleted" and result["chunks"] == 0 - assert db.get_unindexed_webhook_record_ids() == [] - - def test_unpublished_record_also_deletes_from_the_other_collection(self, db_connection): - db = db_connection - payload = copy.deepcopy(SORGHUM_PAYLOAD) - payload["isPublished"] = 0 - rid = _store_row(db, payload) - dev, prod = FakeStore(), FakeStore() - index_advisory_record( - rid, dev, "document-data", _count, also_delete_from=(prod, "prod-collection") - ) - assert prod.calls == [("delete", "prod-collection", rid)] - - def test_published_record_does_not_touch_the_other_collection(self, db_connection): - db = db_connection - rid = _store_row(db, copy.deepcopy(SORGHUM_PAYLOAD)) - dev, prod = FakeStore(), FakeStore() - index_advisory_record( - rid, dev, "document-data", _count, also_delete_from=(prod, "prod-collection") - ) - assert prod.calls == [] - - def test_missing_record_does_nothing(self, db_connection): - store = FakeStore() - result = index_advisory_record("nope", store, "document-data", _count) - assert store.calls == [] - assert result == {"record_id": "nope", "status": "missing", "chunks": 0} - - def test_record_without_poplist_is_marked_indexed(self, db_connection): - db = db_connection - payload = copy.deepcopy(SORGHUM_PAYLOAD) - payload["popList"] = [] - rid = _store_row(db, payload) - store = FakeStore() - result = index_advisory_record(rid, store, "document-data", _count) - assert [c[0] for c in store.calls] == ["delete"] - assert result["status"] == "deleted" - assert db.get_unindexed_webhook_record_ids() == [] - - def test_upsert_failure_leaves_record_unindexed(self, db_connection): - db = db_connection - rid = _store_row(db, copy.deepcopy(SORGHUM_PAYLOAD)) - with pytest.raises(RuntimeError, match="qdrant down"): - index_advisory_record(rid, FakeStore(fail_upsert=True), "document-data", _count) - assert db.get_unindexed_webhook_record_ids() == [rid] - - def test_delete_error_other_than_missing_collection_fails(self, db_connection): - db = db_connection - rid = _store_row(db, copy.deepcopy(SORGHUM_PAYLOAD)) - store = FakeStore(delete_result={"deleted": 0, "error": "connection refused"}) - with pytest.raises(RuntimeError, match="connection refused"): - index_advisory_record(rid, store, "document-data", _count) - assert db.get_unindexed_webhook_record_ids() == [rid] - - def test_missing_collection_on_delete_is_not_an_error(self, db_connection): - db = db_connection - rid = _store_row(db, copy.deepcopy(SORGHUM_PAYLOAD)) - store = FakeStore(delete_result={"deleted": 0, "error": "Collection `x` doesn't exist!"}) - result = index_advisory_record(rid, store, "x", _count) - assert result["status"] == "indexed" - - def test_promote_marks_promoted_not_indexed(self, db_connection): - db = db_connection - rid = _store_row(db, copy.deepcopy(SORGHUM_PAYLOAD)) - index_advisory_record(rid, FakeStore(), "prod", _count, mark="promoted") - assert db.get_unindexed_webhook_record_ids() == [rid] - assert db.advisory_status_counts()["promoted"] == 1 - - def test_reposting_unchanged_record_reuses_point_ids(self, db_connection): - db = db_connection - rid = _store_row(db, copy.deepcopy(SORGHUM_PAYLOAD)) - first, second = FakeStore(), FakeStore() - index_advisory_record(rid, first, "document-data", _count) - index_advisory_record(rid, second, "document-data", _count) - assert [r["_id"] for r in first.records] == [r["_id"] for r in second.records] - - -def test_payload_whitelist_keeps_advisory_fields(): - chunk = build_chunks(make_row(), _count)[0] - payload = _record_payload(chunk) - for key in ( - "type", "external_id", "pop_id", "slug", "crop", "domain", "subdomain", - "institute", "creator", "problem_type", "problem", "season", "start_period", - "end_period", "updated_at", "section", "chunk_id", "text", "doc_id", "title_en", - ): - assert key in payload, key - assert payload["type"] == "advisory" - assert "document_kind" not in payload diff --git a/tests/test_advisory_start.py b/tests/test_advisory_start.py deleted file mode 100644 index 0c7056e..0000000 --- a/tests/test_advisory_start.py +++ /dev/null @@ -1,98 +0,0 @@ -"""Tests for starting advisory workflows (one per record).""" - -import asyncio -from unittest.mock import AsyncMock - -import pytest -from temporalio.exceptions import WorkflowAlreadyStartedError - -from pipeline.advisory_start import start_advisory_index, start_advisory_promote -from pipeline.workflows import AdvisoryIndexWorkflow, AdvisoryPromoteWorkflow - - -def _started_ids(client): - return [call.kwargs["id"] for call in client.start_workflow.await_args_list] - - -def test_index_starts_one_workflow_per_record(): - client = AsyncMock() - started = asyncio.run(start_advisory_index(client, ["a", "b"])) - assert started == 2 - assert _started_ids(client) == ["advisory-a", "advisory-b"] - first = client.start_workflow.await_args_list[0] - assert first.args[0] is AdvisoryIndexWorkflow.run - assert first.kwargs["args"] == ["a"] - assert first.kwargs["task_queue"] == "ocr-pipeline" - - -def test_promote_uses_separate_ids(): - client = AsyncMock() - assert asyncio.run(start_advisory_promote(client, ["a"])) == 1 - assert _started_ids(client) == ["advisory-promote-a"] - assert client.start_workflow.await_args_list[0].args[0] is AdvisoryPromoteWorkflow.run - - -def test_already_running_workflow_is_skipped_and_others_still_start(): - client = AsyncMock() - client.start_workflow.side_effect = [ - WorkflowAlreadyStartedError("advisory-a", "AdvisoryIndexWorkflow"), - None, - ] - assert asyncio.run(start_advisory_index(client, ["a", "b"])) == 1 - - -def test_other_errors_are_logged_and_do_not_stop_the_loop(): - client = AsyncMock() - client.start_workflow.side_effect = [RuntimeError("temporal down"), None] - assert asyncio.run(start_advisory_index(client, ["a", "b"])) == 1 - - -def test_no_ids_starts_nothing(): - client = AsyncMock() - assert asyncio.run(start_advisory_index(client, [])) == 0 - client.start_workflow.assert_not_called() - - -@pytest.mark.db -class TestBackfillOnStart: - def _seed(self, db): - db.upsert_webhook_records( - "webhook", - [{"contentID": 1, "status": "Published", "isPublished": 1, "popList": []}], - ) - - def test_off_by_default(self, db_connection, monkeypatch): - from pipeline.advisory_start import backfill_on_start - - monkeypatch.delenv("ADVISORY_BACKFILL_ON_START", raising=False) - self._seed(db_connection) - client = AsyncMock() - assert asyncio.run(backfill_on_start(client)) == 0 - client.start_workflow.assert_not_called() - - @pytest.mark.parametrize("value", ["true", "1", "yes", "on", "TRUE"]) - def test_on_queues_all_unindexed_rows(self, db_connection, monkeypatch, value): - from pipeline.advisory_start import backfill_on_start - - monkeypatch.setenv("ADVISORY_BACKFILL_ON_START", value) - self._seed(db_connection) - client = AsyncMock() - assert asyncio.run(backfill_on_start(client)) == 1 - assert _started_ids(client)[0].startswith("advisory-") - - def test_on_with_nothing_unindexed_starts_nothing(self, db_connection, monkeypatch): - from pipeline.advisory_start import backfill_on_start - - monkeypatch.setenv("ADVISORY_BACKFILL_ON_START", "true") - client = AsyncMock() - assert asyncio.run(backfill_on_start(client)) == 0 - client.start_workflow.assert_not_called() - - def test_off_values(self, db_connection, monkeypatch): - from pipeline.advisory_start import backfill_on_start - - self._seed(db_connection) - for value in ("false", "0", "", "no"): - monkeypatch.setenv("ADVISORY_BACKFILL_ON_START", value) - client = AsyncMock() - assert asyncio.run(backfill_on_start(client)) == 0 diff --git a/tests/test_api.py b/tests/test_api.py index 0cf8040..20ff966 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -19,7 +19,6 @@ # Set test environment os.environ["MINIO_ACCESS_KEY"] = "test-key" os.environ["MINIO_SECRET_KEY"] = "test-secret" -os.environ["ALLOWED_FILE_PATHS"] = "/app/books,/tmp" _code_seq = itertools.count(1) @@ -383,7 +382,7 @@ def test_upload_starts_today_and_never_expires(self, test_client, sample_pdf_con # only real if it is asserted through the endpoint that grants it. # Spelled out rather than compared against default_period(), so a # change to that default fails here instead of agreeing with itself. - from datetime import date + from datetime import datetime, timezone response = test_client.post( "/upload", @@ -393,7 +392,7 @@ def test_upload_starts_today_and_never_expires(self, test_client, sample_pdf_con assert response.status_code == 200 doc = test_client.get(f"/documents/{response.json()['workflow_id']}").json() - assert doc["valid_from"] == date.today().isoformat() + assert doc["valid_from"] == datetime.now(timezone.utc).date().isoformat() assert doc["valid_to"] is None @pytest.mark.api @@ -558,7 +557,7 @@ class TestSearchValidity: @pytest.mark.api @pytest.mark.unit def test_applies_validity_by_default(self, test_client, capturing_store): - from datetime import date + from datetime import datetime, timezone response = test_client.post("/search", json={"query": "kisan"}) @@ -566,7 +565,7 @@ def test_applies_validity_by_default(self, test_client, capturing_store): assert capturing_store.captured["apply_validity"] is True config = response.json()["effective_config"] assert config["apply_validity"] is True - assert config["valid_on"] == date.today().isoformat() + assert config["valid_on"] == datetime.now(timezone.utc).date().isoformat() @pytest.mark.api @pytest.mark.unit @@ -844,6 +843,47 @@ def test_update_page(self, test_client, db_connection): page = response.json() assert page["edited_markdown"] == "Updated content" + @pytest.mark.api + @pytest.mark.db + @pytest.mark.parametrize("stored_language", ["fr-ca", "fr-CA"]) + def test_review_and_edit_only_the_selected_translation(self, test_client, db_connection, stored_language): + import json + + workflow_id = "page-translation-locale" + db_connection.upsert_document( + kind="advisory", code=_next_code(), workflow_id=workflow_id, document_id="locale-test", + filename="notice.txt", filepath="/tmp/notice.txt", stage="completed", + ) + db_connection.save_pages(workflow_id, [{ + "page_number": 1, "original_markdown": "Source", + "translations": { + "en": {"translated_markdown": "Original English"}, + stored_language: {"translated_markdown": "Texte original"}, + }, + }]) + path = f"/documents/{workflow_id}/pages/1" + review = test_client.patch(path, json={"translation_language": "fr-CA", "translation_reviewed": True}) + assert review.status_code == 200, review.text + assert review.json()["translations"][stored_language]["is_reviewed"] is True + assert not db_connection.get_document(workflow_id)["reindex_required"] + + missing = test_client.patch(path, json={ + "translation_language": "de", "edited_translation": "Missing", "edited_markdown": "Uncommitted", + }) + assert missing.status_code == 409, missing.text + assert not db_connection.get_page(workflow_id, 1)["edited_markdown"] + + edited = test_client.patch(path, json={"translation_language": "fr-CA", "edited_translation": "Texte corrigé"}) + assert edited.status_code == 200, edited.text + translations = edited.json()["translations"] + assert translations["en"]["translated_markdown"] == "Original English" + assert not translations["en"].get("edited_translation") + assert translations[stored_language]["edited_translation"] == "Texte corrigé" + assert db_connection.get_document(workflow_id)["reindex_required"] + audit = db_connection.get_audit_logs(workflow_id, action_type="translation_edit") + assert len(audit) == 2 + assert all(json.loads(entry["metadata"])["language"] == "fr-ca" for entry in audit) + class TestChunkEndpoints: """Tests for chunk operations.""" @@ -1049,7 +1089,7 @@ def test_page_not_found(self, test_client, db_connection): class TestPdfHeaders: @pytest.mark.unit def test_inline_content_disposition_unicode_filename(self): - from pipeline.api.api import _inline_content_disposition + from pipeline.api.routers.document_content import _inline_content_disposition header = _inline_content_disposition("રબર મેટ.pdf") header.encode("latin-1") @@ -1058,7 +1098,7 @@ def test_inline_content_disposition_unicode_filename(self): @pytest.mark.unit def test_inline_content_disposition_strips_crlf(self): - from pipeline.api.api import _inline_content_disposition + from pipeline.api.routers.document_content import _inline_content_disposition header = _inline_content_disposition("evil.pdf\r\nX-Injected: yes") assert "\r" not in header @@ -1108,6 +1148,21 @@ def test_an_unknown_kind_is_rejected(self, test_client, sample_pdf_content, kind assert response.status_code == 400 assert "scheme" in response.json()["detail"] + @pytest.mark.api + def test_upload_accepts_a_configured_kind_without_a_catalog( + self, test_client, db_connection, sample_pdf_content, monkeypatch, + ): + monkeypatch.setenv("DOCUMENT_KINDS", "scheme,advisory,field-report") + response = test_client.post( + "/upload", + params={"kind": "field-report"}, + files={"file": ("report.pdf", sample_pdf_content, "application/pdf")}, + ) + assert response.status_code in (200, 201), response.text + document = test_client.get(f"/documents/{response.json()['workflow_id']}").json() + assert document["kind"] == "field-report" + assert db_connection.get_document(response.json()["workflow_id"])["catalog_id"] is None + @pytest.mark.api @pytest.mark.unit def test_an_invalid_code_is_rejected(self, test_client, sample_pdf_content): @@ -1140,8 +1195,8 @@ def test_upload_creates_the_catalog_and_derives_the_code_from_the_filename( assert doc["name"] == "National Mission on Natural Farming" with db_connection.get_connection() as conn: catalog = conn.execute( - "SELECT c.type, g.name, g.path FROM documents d JOIN catalogs c ON c.id = d.catalog_id " - "JOIN instances g ON g.id = c.instance_id WHERE d.code = %s", + "SELECT c.type, g.name, g.path FROM catalog_status c " + "JOIN instances g ON g.id = c.instance_id WHERE c.code = %s", (doc["code"],), ).fetchall() assert [(r["type"], r["name"], r["path"]) for r in catalog] == [ @@ -1375,7 +1430,7 @@ class TestVectorKeysToDelete: @pytest.mark.api @pytest.mark.unit def test_disable_never_deletes_a_key_another_live_document_is_indexed_under(self, db_connection): - from pipeline.api import api + from pipeline.api.routers.document_lifecycle import _vector_doc_ids db = db_connection uuid_a = db.upsert_document( @@ -1392,11 +1447,11 @@ def test_disable_never_deletes_a_key_another_live_document_is_indexed_under(self ) # The twin was only uploaded, never indexed, so nothing of it is at risk. - assert api._vector_doc_ids(db.get_document(uuid_a), "dev") == ["a" * 32] + assert _vector_doc_ids(db.get_document(uuid_a), "dev") == ["a" * 32] db.upsert_document_index_status("wf-k-b", "documents-index", doc_id="a" * 32, status="indexed") - assert api._vector_doc_ids(db.get_document(uuid_a), "dev") == [] - assert api._vector_doc_ids(db.get_document("wf-k-c"), "dev") == ["b" * 32] + assert _vector_doc_ids(db.get_document(uuid_a), "dev") == [] + assert _vector_doc_ids(db.get_document("wf-k-c"), "dev") == ["b" * 32] db.set_document_disabled("wf-k-b", True) - assert api._vector_doc_ids(db.get_document(uuid_a), "dev") == ["a" * 32] + assert _vector_doc_ids(db.get_document(uuid_a), "dev") == ["a" * 32] diff --git a/tests/test_api_routes.py b/tests/test_api_routes.py index e8fc2a3..1831fd5 100644 --- a/tests/test_api_routes.py +++ b/tests/test_api_routes.py @@ -11,12 +11,18 @@ EXPECTED_ROUTES: list[tuple[str, str]] = [ ("DELETE", "/documents/{workflow_id}"), + ("DELETE", "/documents/{workflow_id}/releases/{network_id}"), + ("DELETE", "/providers/{provider}/members/{user_id}"), ("GET", "/admin/access-options"), ("GET", "/admin/index/schema"), ("GET", "/admin/ingest-info"), ("GET", "/admin/users"), ("GET", "/audit"), ("GET", "/auth/me"), + ("GET", "/auth/google"), + ("GET", "/auth/google/callback"), + ("POST", "/auth/google/start"), + ("POST", "/auth/google/complete"), ("GET", "/catalog/v1/schemes"), ("GET", "/catalog/v1/snapshot"), ("GET", "/catalog/v1/tool-prompt"), @@ -43,33 +49,59 @@ ("GET", "/documents/{workflow_id}/pdf"), ("GET", "/documents/{workflow_id}/qdrant"), ("GET", "/documents/{workflow_id}/qdrant/chunks"), + ("GET", "/documents/{workflow_id}/revisions"), + ("GET", "/documents/{workflow_id}/revisions/{revision_id}"), ("GET", "/documents/{workflow_id}/runtime"), ("GET", "/documents/{workflow_id}/stage-io"), ("GET", "/health"), ("GET", "/indexes/summary"), ("GET", "/indexes/{index_name}/settings"), ("GET", "/indexes/{index_name}/stats"), + ("GET", "/networks/{network_id}/registry-proposals"), + ("GET", "/networks/{network_id}/registry-proposals/{proposal_id}"), + ("POST", "/providers/{provider}/networks/{network_id}/registry-proposals"), + ("PUT", "/networks/{network_id}/registry-proposals/{proposal_id}/decision"), ("GET", "/operations/queue"), ("GET", "/pipeline/stages"), ("GET", "/provenance/chunk"), + ("GET", "/providers"), + ("GET", "/providers/readable"), + ("GET", "/providers/{provider}/members"), + ("GET", "/providers/{provider}/networks/{network_id}/registry-context"), + ("GET", "/providers/{provider}/registry-networks"), + ("GET", "/providers/{provider}/scheduled-sources"), ("GET", "/runs"), ("GET", "/runs/{job_id}"), + ("GET", "/search/collections"), + ("GET", "/providers/{provider}/source-settings"), + ("PUT", "/providers/{provider}/source-settings"), + ("GET", "/providers/{provider}/crawl"), + ("GET", "/knowledge/intake/{provider}"), + ("GET", "/knowledge/intake/{provider}/{source}/deliveries"), + ("GET", "/knowledge/intake/{provider}/{source}/deliveries/original"), + ("POST", "/knowledge/intake/{provider}/mapping-preview"), ("GET", "/settings/search"), ("GET", "/settings/search/audit"), - ("GET", "/webhook/advisory/status"), + ("GET", "/upload/options"), + ("GET", "/upload/providers"), + ("GET", "/upload/scheme-matches"), + ("GET", "/knowledge/intake/schema"), + ("GET", "/knowledge/intake/{provider}/{source}"), + ("PUT", "/knowledge/intake/{provider}/{source}"), + ("POST", "/knowledge/intake/{provider}/{source}/submissions"), + ("GET", "/webhook/sources/{provider}/{source}"), ("PATCH", "/documents/{workflow_id}/chunks/{chunk_num}"), ("PATCH", "/documents/{workflow_id}/pages/{page_num}"), ("PATCH", "/documents/{workflow_id}/scheme-metadata"), ("POST", "/admin/index/create"), ("POST", "/admin/users"), ("POST", "/auth/otp/refresh"), - ("POST", "/auth/otp/request"), - ("POST", "/auth/otp/verify"), ("POST", "/auth/session/refresh"), - ("POST", "/auth/sso/exchange"), + ("POST", "/auth/email/send"), + ("POST", "/auth/email/verify"), + ("POST", "/auth/session/logout"), ("POST", "/catalog/v1/rebuild"), ("POST", "/documents"), - ("POST", "/documents/batch"), ("POST", "/documents/bulk/approve-chunks"), ("POST", "/documents/bulk/approve-ocr"), ("POST", "/documents/bulk/approve-translation"), @@ -92,14 +124,23 @@ ("POST", "/documents/{workflow_id}/retry-ingestion"), ("POST", "/documents/{workflow_id}/retry-ocr"), ("POST", "/documents/{workflow_id}/retry-translation"), + ("POST", "/providers/{provider}/crawl"), + ("POST", "/providers/{provider}/crawl/{source_id}"), + ("POST", "/knowledge/intake/{provider}/{source}/token"), + ("POST", "/providers/{provider}/members"), ("POST", "/search"), + ("POST", "/search/"), ("POST", "/settings/search/reset"), ("POST", "/upload"), - ("POST", "/webhook"), - ("POST", "/webhook/advisory/promote"), - ("POST", "/webhook/advisory/reindex"), + ("POST", "/upload/batch"), + ("POST", "/upload/{workflow_id}/start"), + ("POST", "/webhook/sources/{provider}/{source}/events"), ("PUT", "/admin/users/{user_id}/access"), + ("PUT", "/documents/{workflow_id}/releases/{network_id}"), + ("PUT", "/providers/{provider}"), + ("PUT", "/providers/{provider}/members/{user_id}"), ("PUT", "/settings/search"), + ("PUT", "/webhook/sources/{provider}/{source}"), ] def collect_routes(app) -> list[tuple[str, str]]: @@ -112,7 +153,7 @@ def collect_routes(app) -> list[tuple[str, str]]: def test_route_snapshot_matches_app(): from pipeline.api import app - assert collect_routes(app) == EXPECTED_ROUTES + assert collect_routes(app) == sorted(EXPECTED_ROUTES) @pytest.mark.unit diff --git a/tests/test_auth.py b/tests/test_auth.py index 80483d2..60b08ca 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -95,10 +95,9 @@ def test_local_bypass_has_all_permissions(): assert set(Permission).issubset(user.permissions) -def test_auth_disabled_by_default(): - with patch.dict(os.environ, {"AUTH_DISABLED": "true"}): - cfg = load_auth_config() - assert cfg.disabled is True +def test_auth_enabled_by_default(monkeypatch): + monkeypatch.delenv("AUTH_DISABLED") + assert not load_auth_config().disabled def test_enabled_auth_requires_keycloak_config_at_startup(): @@ -118,10 +117,26 @@ def test_disabled_auth_does_not_require_keycloak_config(): keycloak_issuer="", keycloak_audience="docs-pipeline-api", keycloak_jwks_url="", + runtime_profile="local", ) validate_auth_config(config) +def test_production_requires_api_audience_and_disallows_bypass(): + issuer = "https://example.com/realms/test" + with pytest.raises(RuntimeError, match="KEYCLOAK_AUDIENCE"): + validate_auth_config(AuthConfig(False, issuer, "", f"{issuer}/protocol/openid-connect/certs")) + with pytest.raises(RuntimeError, match="KP_RUNTIME_PROFILE"): + validate_auth_config(AuthConfig(True, "", "", "")) + + +def test_only_configured_client_roles_are_trusted(): + user = claims_to_user({"sub": "u1", "resource_access": { + "unrelated-client": {"roles": ["super_admin"]}, "knowledge-api": {"roles": ["viewer"]}, + }}, client_id="knowledge-api") + assert "super_admin" not in user.roles + + @pytest.mark.asyncio async def test_get_current_user_bypass_mode(): from starlette.requests import Request @@ -184,18 +199,20 @@ def test_every_route_is_gated_or_explicitly_classified(): right auth dependency instead of widening the allowlist. """ from pipeline.api import app + from pipeline.api.routers.webhook_sources import require_webhook_sender # The ONLY intentionally-public application routes. public = { ("GET", "/health"), - # Login. Necessarily unauthenticated — these are how a caller obtains a - # token in the first place. Guarded instead by RATE_LIMIT_OTP, an - # enumeration-safe send response, and single-use codes in Keycloak. - ("POST", "/auth/otp/request"), - ("POST", "/auth/otp/verify"), - # SSO code exchange: the browser cannot do it (confidential client), and - # the authorization code it posts here is single-use and PKCE-bound. - ("POST", "/auth/sso/exchange"), + # These console routes establish a session from an email code. + # Logout and refresh require the session refresh token. + ("GET", "/auth/google"), + ("POST", "/auth/google/start"), + ("GET", "/auth/google/callback"), + ("POST", "/auth/google/complete"), + ("POST", "/auth/email/send"), + ("POST", "/auth/email/verify"), + ("POST", "/auth/session/logout"), # Refresh is public for the same reason: the refresh token is itself the # credential, and the access token has usually expired by then. ("POST", "/auth/session/refresh"), @@ -215,13 +232,14 @@ def dependency_calls(dependant): # application_routes descends into included routers; a plain walk over # app.routes would skip every router-mounted endpoint and pass vacuously. for method, path, route in application_routes(app): - has_auth = get_current_user in dependency_calls(route.dependant) + calls = dependency_calls(route.dependant) + has_auth = get_current_user in calls or require_webhook_sender in calls if not has_auth and (method, path) not in public: ungated.append(f"{method} {path}") assert not ungated, ( "These routes are neither gated nor on the public allowlist " - f"(only /health may be public): {sorted(ungated)}" + f": {sorted(ungated)}" ) diff --git a/tests/test_backfill_provider_id.py b/tests/test_backfill_provider_id.py index af3b78b..cc1832b 100644 --- a/tests/test_backfill_provider_id.py +++ b/tests/test_backfill_provider_id.py @@ -5,7 +5,7 @@ from qdrant_client import QdrantClient from qdrant_client.http import models as qmodels -from pipeline.network_provider import ProviderNaming +from pipeline.network.network_provider import ProviderNaming SCRIPT = Path(__file__).parents[1] / "scripts" / "backfill_provider_id.py" COLLECTION = "documents-index" @@ -30,6 +30,7 @@ def target(backfill_module): ("wf-bv-scheme", {}), ("wf-mh-document", {"provider_id": "stale"}), ("wf-bad", {}), + ("wf-mh-advisory", {"provider_id": "frozen", "revision_id": "revision-1"}), ] client.upsert( COLLECTION, @@ -42,11 +43,11 @@ def target(backfill_module): DOCUMENTS = [ - {"workflow_id": "wf-mh-advisory", "document_kind": "advisory", "instance": "mh"}, - {"workflow_id": "wf-bv-scheme", "document_kind": "scheme", "instance": "bv"}, - {"workflow_id": "wf-mh-document", "document_kind": "document", "instance": "mh"}, - {"workflow_id": "wf-bad", "document_kind": "advisory", "instance": "ladakh_ut"}, - {"workflow_id": "wf-not-indexed", "document_kind": "advisory", "instance": "ka"}, + {"workflow_id": "wf-mh-advisory", "kind": "advisory", "instance": "mh"}, + {"workflow_id": "wf-bv-scheme", "kind": "scheme", "instance": "bv"}, + {"workflow_id": "wf-mh-document", "kind": "document", "instance": "mh"}, + {"workflow_id": "wf-bad", "kind": "advisory", "instance": "ladakh_ut"}, + {"workflow_id": "wf-not-indexed", "kind": "advisory", "instance": "ka"}, ] @@ -63,6 +64,21 @@ def test_plan_follows_the_publish_provider(backfill_module): assert planned["wf-bv-scheme"].provider_id == "bv.provider.oan.dev" assert planned["wf-mh-document"].provider_id is None assert planned["wf-bad"].error and planned["wf-bad"].provider_id is None + unknown, = backfill_module.plan([{"workflow_id": "no-kind"}], NAMING) + assert unknown.error + + +@pytest.mark.db +def test_backfill_reads_postgres_documents(backfill_module, db_connection): + db_connection.upsert_document( + workflow_id="wf-provider-backfill", document_id="a" * 32, filename="notice.pdf", + filepath="minio://documents/notice.pdf", kind="advisory", code="notice", instance="provider-a", + ) + + planned, = backfill_module.plan(backfill_module.all_documents(), NAMING) + + assert planned.provider_id == "provider-a.provider.oan.dev" + assert backfill_module.dev_collections() @pytest.mark.unit @@ -88,6 +104,7 @@ def test_apply_sets_and_clears_by_workflow(backfill_module, target): 3: "bv.provider.oan.dev", 4: None, 5: None, + 6: "frozen", } diff --git a/tests/test_catalog_builder.py b/tests/test_catalog_builder.py index a58acc6..0b50690 100644 --- a/tests/test_catalog_builder.py +++ b/tests/test_catalog_builder.py @@ -1,19 +1,15 @@ """Unit tests for the Network Catalog Envelope's catalog builder.""" +import json + import pytest -from pipeline.catalog_builder import ( +from pipeline.network.catalog.catalog_builder import ( build_catalog, is_publishable_kind, normalize_document_kind, ) -from pipeline.network_constants import ( - ADVISORY_CATALOG_ID, - ADVISORY_RESOURCE_ID, - SCHEMES_CATALOG_ID, - SCHEMES_RESOURCE_ID, -) -from pipeline.network_provider import Group +from pipeline.network.network_provider import Group PROVIDER = { "id": "bv.provider.oan.dev", @@ -23,9 +19,39 @@ "id": "mh.provider.oan.dev", "descriptor": {"code": "mh.provider.oan.dev", "name": "Maharashtra"}, } +ADVISORY_CATALOG_ID = "cat-oan-knowledge-provider-advisories" +ADVISORY_RESOURCE_ID = "res-oan-knowledge-provider-advisories" +SCHEMES_CATALOG_ID = "cat-oan-knowledge-provider-schemes" +SCHEMES_RESOURCE_ID = "res-oan-knowledge-provider-schemes" CATALOG_NAME = "Knowledge from documents" MH_CATALOG_NAME = "Maharashtra knowledge from documents" + +def test_profile_can_describe_a_catalog_for_another_domain(monkeypatch, tmp_path): + from pipeline.network.network_group import load_provider_naming, provider_for + + profile = tmp_path / "catalogs.json" + attributes = {"@type": "example:MaintenanceBulletin", "languages": ["fr"]} + profile.write_text(json.dumps({ + "maintenance": { + "catalog_name": "Maintenance notices", + "domain": "maintenance", + "catalog_id": "maintenance-catalog", + "resource_name": "Maintenance bulletin", + "resource_id": "maintenance-resource", + "resource_attributes": attributes, + }, + })) + monkeypatch.setenv("BECKN_CATALOG_PROFILE_PATH", str(profile)) + naming = load_provider_naming("network.example") + provider = provider_for(naming, "maintenance", None) + + catalog = build_catalog("maintenance", provider, naming.catalog_name("maintenance", None)) + + assert catalog["descriptor"]["name"] == "Maintenance notices" + assert catalog["resources"][0]["resourceAttributes"] == attributes + assert not is_publishable_kind("advisory") + # Forbidden by the KnowledgeAdvisory schema under informationMode OnDemand. # Both advisory and scheme resources use this schema. FORBIDDEN_ADVISORY_ATTRIBUTES = ( diff --git a/tests/test_crop_metadata.py b/tests/test_crop_metadata.py new file mode 100644 index 0000000..a44980a --- /dev/null +++ b/tests/test_crop_metadata.py @@ -0,0 +1,209 @@ +"""Crop context is supplied explicitly, persisted, and retained by retrieval.""" + +import json +from types import SimpleNamespace + +import pytest + +from pipeline.metadata.crop_metadata import parse_crop_subject_ids, parse_crop_subjects +from pipeline.documents.models import DocumentMetadata, DocumentSummary + +CROPS = [{ + "subjectId": "https://taxonomy.openagrinet.global/crops/cotton", + "descriptor": {"code": "COTTON", "name": "Cotton"}, +}] + + +@pytest.mark.unit +@pytest.mark.parametrize("raw", [ + "not-json", "{}", "null", "[{}]", '[{"subjectId":"cotton"}]', + '[{"descriptor":{}}]', '[{"descriptor":{"name":" "}}]', + '[{"descriptor":{"name":"Cotton"},"unknown":true}]', + json.dumps(CROPS * 2), + json.dumps(CROPS * 33), + json.dumps([{"subjectId": "https://example.org/cotton field"}]), + json.dumps([{"subjectId": "https:\\example.org\\cotton"}]), +]) +def test_rejects_invalid_crop_context(raw): + with pytest.raises(ValueError): + parse_crop_subjects(raw) + + +@pytest.mark.unit +def test_optional_crop_context_and_schema_identifiers(): + assert parse_crop_subjects(None) is None + assert parse_crop_subjects("[]") == [] + assert parse_crop_subjects(json.dumps(CROPS)) == CROPS + assert parse_crop_subjects('[{"subjectId":"urn:crop:cotton"}]') == [{"subjectId": "urn:crop:cotton"}] + assert parse_crop_subjects('[{"descriptor":{"name":" Cotton "}}]') == [{"descriptor": {"name": "Cotton"}}] + assert DocumentMetadata().crop_subjects == [] + response = DocumentSummary( + document_id="doc", workflow_id="wf", filename="cotton.pdf", + stage="registered", page_count=0, chunk_count=0, + crop_subjects=[{"descriptor": {"name": "Cotton"}}], + ).model_dump() + assert response["crop_subjects"] == [{"descriptor": {"name": "Cotton"}}] + + +@pytest.mark.unit +def test_crop_context_survives_chunk_payload_and_hit_projection(): + from pipeline.search import passage_projection + from pipeline.vector_store.qdrant_store import _hit_from_point, _record_payload + + record, = passage_projection._prepare_records( + "doc", "cotton.pdf", [{"chunk_number": 1, "original_text": "Irrigate cotton carefully."}], + workflow_id="wf", + **passage_projection._scheme_fields_from_doc({"kind": "advisory", "crop_subjects": CROPS}), + ) + payload = _record_payload(record) + assert payload["crop_subjects"] == CROPS + hit = _hit_from_point(SimpleNamespace(payload=payload, id=record["_id"], score=1.0)) + assert hit["crop_subjects"] == CROPS + + +@pytest.mark.unit +@pytest.mark.parametrize("value", ["cotton", ["cotton"], [None], [123], ["https://example.org/cotton"] * 33]) +def test_rejects_invalid_crop_filter(value): + with pytest.raises(ValueError): + parse_crop_subject_ids(value) + + +@pytest.mark.unit +@pytest.mark.parametrize("mode", ["TENSOR", "LEXICAL"]) +def test_real_local_qdrant_matches_crop_and_excludes_other_crops(monkeypatch, mode): + from qdrant_client import QdrantClient + + from pipeline.vector_store import qdrant_store + + monkeypatch.setattr(qdrant_store, "get_vector_size", lambda: 3) + monkeypatch.setattr(qdrant_store, "embed_passages", lambda texts, **kw: [[1.0, 0.0, 0.0] for _ in texts]) + monkeypatch.setattr(qdrant_store, "embed_query", lambda *a, **kw: [1.0, 0.0, 0.0]) + client = QdrantClient(location=":memory:") + try: + store = qdrant_store.QdrantVectorStore(client=client) + store.upsert("crops", [{ + "_id": "a" * 32, "text": "cotton irrigation", "crop_subjects": CROPS, + "provider_id": "provider-a", "is_reference": False, + }]) + common = {"name": "crops", "query": "irrigation", "search_mode": mode, "apply_validity": False} + hit, = store.search(**common, crop_subject_ids=[CROPS[0]["subjectId"]])["hits"] + assert hit["crop_subjects"] == CROPS + assert store.search(**common, crop_subject_ids=["https://taxonomy.openagrinet.global/crops/rice"])["hits"] == [] + assert store.search(**common, crop_subject_ids=[CROPS[0]["subjectId"]], provider_id="provider-b")["hits"] == [] + assert len(store.search(**common)["hits"]) == 1 + assert len(store.search(**common, crop_subject_ids=[])["hits"]) == 1 + finally: + client.close() + + +@pytest.mark.unit +@pytest.mark.parametrize("damaged_field", ["missing", "payload", "vector"]) +def test_release_upsert_verifies_the_readable_point(monkeypatch, damaged_field): + from qdrant_client import QdrantClient + + from pipeline.vector_store import qdrant_store + + monkeypatch.setattr(qdrant_store, "get_vector_size", lambda: 3) + monkeypatch.setattr(qdrant_store, "embed_passages", lambda texts, **kw: [[1.0, 0.0, 0.0] for _ in texts]) + client = QdrantClient(location=":memory:") + try: + store = qdrant_store.QdrantVectorStore(client=client) + record = {"_id": "a" * 32, "text": "cotton irrigation", "release_network_id": "network.example"} + assert store.upsert("release", [record])["records_ingested"] == 1 + retrieve = client.retrieve + + def read_damaged_point(**kwargs): + if damaged_field == "missing": + return [] + point, = retrieve(**kwargs) + if damaged_field == "payload": + point.payload = {"text": "unrelated content"} + else: + point.vector = [0.0, 0.0, 1.0] + return [point] + + monkeypatch.setattr(client, "retrieve", read_damaged_point) + with pytest.raises(ValueError, match="could not verify"): + store.upsert("release", [record]) + finally: + client.close() + + +@pytest.mark.db +@pytest.mark.api +def test_upload_persists_crop_context_and_search_returns_it( + test_client, db_connection, sample_pdf_content, capturing_store, monkeypatch, +): + from pipeline.search import passage_projection + from pipeline.api import api + from pipeline.vector_store.qdrant_store import _hit_from_point, _record_payload + + api.limiter.reset() + response = test_client.post( + "/upload", params={"kind": "advisory", "code": "crop-context"}, + files={"file": ("cotton.pdf", sample_pdf_content, "application/pdf")}, + data={"crop_subjects": json.dumps(CROPS)}, + ) + assert response.status_code == 200, response.text + assert response.json()["crop_subjects"] == CROPS + workflow_id = response.json()["workflow_id"] + document = db_connection.get_document(workflow_id) + assert document["crop_subjects"] == CROPS + with db_connection.get_connection() as conn: + row = conn.execute("SELECT metadata FROM documents WHERE id = %s::uuid", (document["id"],)).fetchone() + assert row["metadata"]["crop_subjects"] == CROPS + assert test_client.get(f"/documents/{workflow_id}").json()["crop_subjects"] == CROPS + assert test_client.get("/documents").json()[0]["crop_subjects"] == CROPS + + record, = passage_projection._prepare_records( + document["document_id"], document["filename"], + [{"chunk_number": 1, "original_text": "Cotton irrigation guidance."}], + workflow_id=workflow_id, **passage_projection._scheme_fields_from_doc(document), + ) + capturing_store.hits = [_hit_from_point(SimpleNamespace( + id=record["_id"], score=1.0, payload=_record_payload(record), + ))] + result = test_client.post("/search", json={ + "query": "cotton irrigation", "search_mode": "TENSOR", "crop_subject_ids": [CROPS[0]["subjectId"]], + }) + assert result.status_code == 200, result.text + assert result.json()["hits"][0]["crop_subjects"] == CROPS + assert capturing_store.captured["crop_subject_ids"] == [CROPS[0]["subjectId"]] + assert test_client.post("/search", json={"query": "cotton", "crop_subject_ids": ["cotton"]}).status_code == 400 + + +@pytest.mark.db +def test_reupload_preserves_omitted_crop_context_and_can_explicitly_clear_it(db_connection): + document_id = db_connection.upsert_document( + workflow_id="crop-reupload", document_id="a" * 32, filename="cotton.pdf", + filepath="minio://documents/cotton.pdf", kind="advisory", code="cotton", + crop_subjects=CROPS, + ) + db_connection.reset_document_for_reupload( + document_id, filename="cotton-v2.pdf", filepath="minio://documents/cotton-v2.pdf", + fingerprint="b" * 32, + ) + assert db_connection.get_document(document_id)["crop_subjects"] == CROPS + db_connection.reset_document_for_reupload( + document_id, filename="general.pdf", filepath="minio://documents/general.pdf", + fingerprint="c" * 32, crop_subjects=[], + ) + assert db_connection.get_document(document_id)["crop_subjects"] == [] + + +@pytest.mark.db +@pytest.mark.api +def test_invalid_crop_metadata_has_no_upload_or_workflow_side_effects( + test_client, sample_pdf_content, mock_object_store, mock_temporal_client, +): + from pipeline.api import api + + api.limiter.reset() + response = test_client.post( + "/upload", params={"kind": "advisory", "code": "invalid-crop"}, + files={"file": ("cotton.pdf", sample_pdf_content, "application/pdf")}, + data={"crop_subjects": "[{}]"}, + ) + assert response.status_code == 422 + mock_object_store.put.assert_not_called() + mock_temporal_client.start_workflow.assert_not_called() diff --git a/tests/test_db.py b/tests/test_db.py index 01820a6..8fa8ac1 100644 --- a/tests/test_db.py +++ b/tests/test_db.py @@ -614,33 +614,43 @@ def _new_doc(db, workflow_id, **kwargs): class TestSchemaConstraints: """Rules the database itself enforces.""" - @staticmethod - def _catalog_id(db): - with db.get_connection() as conn: - return db.get_or_create_catalog(db.get_or_create_instance("mh", conn=conn), "scheme", conn=conn)["id"] - @pytest.mark.db @pytest.mark.unit def test_document_code_must_be_a_slug(self, db_connection): - catalog_id = self._catalog_id(db_connection) - with db_connection.get_connection() as conn: + db = db_connection + with db.get_connection() as conn: + provider_id = db.get_or_create_instance("mh", conn=conn) with pytest.raises(psycopg.errors.CheckViolation): conn.execute( - "INSERT INTO documents (catalog_id, code, name, source) VALUES (%s::uuid, 'Bad Code', 'N', %s)", - (catalog_id, psycopg.types.json.Jsonb({"fingerprint": _md5("k")})), + "INSERT INTO documents (provider_id, kind, code, name, source) " + "VALUES (%s, 'scheme', 'Bad Code', 'N', %s)", + (provider_id, psycopg.types.json.Jsonb({"fingerprint": _md5("k")})), ) @pytest.mark.db @pytest.mark.unit def test_source_requires_a_hex_fingerprint(self, db_connection): - catalog_id = self._catalog_id(db_connection) - with db_connection.get_connection() as conn: + db = db_connection + with db.get_connection() as conn: + provider_id = db.get_or_create_instance("mh", conn=conn) with pytest.raises(psycopg.errors.CheckViolation): conn.execute( - "INSERT INTO documents (catalog_id, code, name, source) VALUES (%s::uuid, 'c', 'C', %s)", - (catalog_id, psycopg.types.json.Jsonb({"fingerprint": "not-hex"})), + "INSERT INTO documents (provider_id, kind, code, name, source) " + "VALUES (%s, 'scheme', 'c', 'C', %s)", + (provider_id, psycopg.types.json.Jsonb({"fingerprint": "not-hex"})), ) + @pytest.mark.db + def test_document_is_readable_without_a_catalog(self, db_connection): + db = db_connection + document_id = _new_doc(db, "wf-owner", instance="mh", kind="scheme", code="owned") + with db.get_connection() as conn: + conn.execute("DELETE FROM catalogs WHERE id = %s", (db.get_document("wf-owner")["catalog_id"],)) + document = db.find_document_by_code("mh", "scheme", "owned") + assert (document["id"], document["instance"], document["kind"]) == (document_id, "mh", "scheme") + assert document["catalog_id"] is None + assert db.get_catalog_entry("owned")["type"] == "openagrinet:AgricultureResource" + @pytest.mark.db @pytest.mark.unit def test_stage_timestamps_only_allow_known_keys(self, db_connection): @@ -1149,26 +1159,6 @@ def test_uploader_is_a_user_row(self, db_connection): assert (doc["uploaded_by_user_id"], doc["uploaded_by_username"]) == ("kc-u", "uma") assert doc["uploaded_by_roles"] == "contributor,reviewer" - @pytest.mark.db - @pytest.mark.unit - def test_otp_attempts_increment_atomically(self, db_connection): - db_connection.store_email_otp( - email="A@X.in", code_hash="h", expires_at="2099-01-01T00:00:00+00:00", - created_at="x", last_sent_at="2026-01-01T00:00:00+00:00", - ) - assert db_connection.increment_email_otp_attempts("a@x.in") == 1 - assert db_connection.increment_email_otp_attempts("a@x.in") == 2 - row = db_connection.get_email_otp("a@x.in") - assert row["attempts"] == 2 - assert row["expires_at"].endswith("+00:00") - db_connection.store_email_otp( - email="a@x.in", code_hash="h2", expires_at="2099-01-01T00:00:00+00:00", - created_at="x", last_sent_at="2026-01-02T00:00:00+00:00", - ) - assert db_connection.get_email_otp("a@x.in")["attempts"] == 0 - db_connection.delete_email_otp("a@x.in") - assert db_connection.get_email_otp("a@x.in") is None - class TestOperationsViews: @pytest.mark.db @@ -1210,18 +1200,77 @@ def test_reconcile_moves_a_stalled_document_into_review(self, db_connection): assert result["updated"] is True and result["stage"] == "ocr_review" assert db_connection.get_latest_document_job("wf-rec")["status"] == "waiting_review" +class TestMigrationRunner: @pytest.mark.db - @pytest.mark.unit - def test_webhook_upsert_skips_unchanged_payloads(self, db_connection): - db = db_connection - assert db.upsert_webhook_records("webhook", [{"contentID": "c1", "title": "T", "isPublished": True}]) == 1 - assert db.upsert_webhook_records("webhook", [{"contentID": "c1", "title": "T", "isPublished": True}]) == 1 - with db.get_connection() as conn: - rows = conn.execute("SELECT title, is_published FROM webhook_records").fetchall() - assert rows == [{"title": "T", "is_published": True}] - + def test_document_and_release_migrations_backfill_existing_rows(self, db_connection): + from pipeline import migrate + from tests.migration_kit import migration_kit_path + + migrations = dict(migrate.list_migrations(migration_kit_path() / "legacy_postgres" / "upgrade_history")) + with db_connection.get_connection() as conn, conn.transaction(force_rollback=True): + conn.execute("CREATE SCHEMA legacy_owner_test") + conn.execute("SET LOCAL search_path TO legacy_owner_test, public") + conn.execute(migrations["0001"].read_text(encoding="utf-8")) + provider_id = conn.execute( + "INSERT INTO instances (name, path) VALUES ('provider', '/providers/provider') RETURNING id" + ).fetchone()["id"] + catalog_id = conn.execute( + "INSERT INTO catalogs (instance_id, type, name) VALUES (%s, 'example:Resource', 'Catalog') RETURNING id", + (provider_id,), + ).fetchone()["id"] + before = conn.execute( + """ + INSERT INTO documents (catalog_id, code, name, source, updated_at) + VALUES (%s, 'original', 'Original', %s, '2024-01-01T00:00:00Z') + RETURNING id, updated_at + """, + (catalog_id, psycopg.types.json.Jsonb({"fingerprint": _md5("original")})), + ).fetchone() + conn.execute(migrations["0007"].read_text(encoding="utf-8")) + conn.execute(migrations["0008"].read_text(encoding="utf-8")) + after = conn.execute( + "SELECT provider_id, kind, updated_at FROM documents WHERE id = %s", (before["id"],) + ).fetchone() + assert after == {"provider_id": provider_id, "kind": "example:Resource", "updated_at": before["updated_at"]} + + for version in ("0003", "0006"): + conn.execute(migrations[version].read_text(encoding="utf-8")) + job = conn.execute( + """ + INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, config) + VALUES (%s, 'pipeline', 'legacy-release', '{"release_network_id":"test.net"}') + RETURNING id + """, + (before["id"],), + ).fetchone() + snapshot = { + "index_name": "published", + "expected_release_generation": None, + "passages": [{"text": "Original", "provider_id": "provider.test.net"}], + } + revision = conn.execute( + """ + INSERT INTO document_revisions (id, document_id, workflow_id, content_sha256, snapshot) + VALUES (gen_random_uuid(), %s, %s, repeat('a', 64), %s) RETURNING id + """, + (before["id"], job["id"], psycopg.types.json.Jsonb(snapshot)), + ).fetchone() + conn.execute(migrations["0009"].read_text(encoding="utf-8")) + staged = conn.execute( + """ + SELECT s.network_id, s.collection, s.provider_id, s.expected_generation, + s.generation_captured, r.snapshot, r.content_sha256 + FROM network_release_staging s JOIN document_revisions r ON r.id = s.revision_id + WHERE s.revision_id = %s + """, + (revision["id"],), + ).fetchone() + assert staged == { + "network_id": "test.net", "collection": "published", "provider_id": "provider.test.net", + "expected_generation": None, "generation_captured": True, "snapshot": snapshot, + "content_sha256": "a" * 64, + } -class TestMigrationRunner: @pytest.mark.db @pytest.mark.unit def test_applied_migrations_are_recorded_and_not_reapplied(self, db_connection, postgres_url): @@ -1229,22 +1278,29 @@ def test_applied_migrations_are_recorded_and_not_reapplied(self, db_connection, assert migrate.apply_migrations(postgres_url) == [] assert migrate.is_current(postgres_url) - assert migrate.current_version(postgres_url) == migrate.list_migrations()[-1][0] + migrations = migrate.list_migrations() + assert migrate.current_version(postgres_url) == (migrations[-1][0] if migrations else "0022") @pytest.mark.db @pytest.mark.unit - def test_migration_files_are_numbered_and_unversioned_by_schema_name(self): + def test_clean_schema_omits_import_only_tables(self, db_connection): from pipeline import migrate - names = [path.name for _, path in migrate.list_migrations()] - assert names[0] == "0001_initial_schema.sql" + assert all(version > "0022" for version, _ in migrate.list_migrations()) + with db_connection.get_connection() as conn: + row = conn.execute(""" + SELECT to_regclass('public.legacy_import_evidence') AS evidence, + to_regclass('public.webhook_records') AS webhook_records, + to_regclass('public.email_otps') AS email_otps, + to_regclass('public.webhook_deliveries') AS webhook_deliveries + """).fetchone() + assert row == {"evidence": None, "webhook_records": None, "email_otps": None, "webhook_deliveries": "webhook_deliveries"} @pytest.mark.db @pytest.mark.unit def test_a_new_migration_runs_once(self, db_connection, postgres_url, tmp_path): from pipeline import migrate - (tmp_path / "0001_initial_schema.sql").write_text("SELECT 1;") (tmp_path / "0099_probe.sql").write_text("CREATE TABLE mig_probe (n int);") try: assert migrate.apply_migrations(postgres_url, tmp_path) == ["0099"] @@ -1259,7 +1315,6 @@ def test_a_new_migration_runs_once(self, db_connection, postgres_url, tmp_path): def test_a_failing_migration_applies_nothing_of_itself(self, db_connection, postgres_url, tmp_path): from pipeline import migrate - (tmp_path / "0001_initial_schema.sql").write_text("SELECT 1;") (tmp_path / "0099_bad.sql").write_text("CREATE TABLE mig_half (n int); SELECT 1/0;") with pytest.raises(psycopg.errors.DivisionByZero): migrate.apply_migrations(postgres_url, tmp_path) diff --git a/tests/test_discovery_publish_service.py b/tests/test_discovery_publish_service.py index 3539abf..accc1d1 100644 --- a/tests/test_discovery_publish_service.py +++ b/tests/test_discovery_publish_service.py @@ -39,7 +39,7 @@ def _client_returning(response): def _service(monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "docs-pipeline-bv") @@ -50,7 +50,7 @@ def _service(monkeypatch): class TestDiscoveryPublishServiceConfig: @pytest.mark.unit def test_requires_endpoint_and_sender_id(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.delenv("DISCOVERY_SERVICE_ENDPOINT", raising=False) monkeypatch.delenv("NETWORK_SENDER_ID", raising=False) @@ -60,7 +60,7 @@ def test_requires_endpoint_and_sender_id(self, monkeypatch): @pytest.mark.unit def test_missing_sender_id_alone_raises(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.delenv("NETWORK_SENDER_ID", raising=False) @@ -70,7 +70,7 @@ def test_missing_sender_id_alone_raises(self, monkeypatch): @pytest.mark.unit def test_explicit_endpoint_overrides_env(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://env.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", " provider.oan.dev ") @@ -82,9 +82,27 @@ def test_explicit_endpoint_overrides_env(self, monkeypatch): class TestDiscoveryPublishServicePublish: + @pytest.mark.unit + def test_configured_network_receiver_and_audience_are_independent_of_sender(self, monkeypatch): + monkeypatch.setenv("DISCOVERY_NETWORK_ID", "oan-prod.local") + monkeypatch.setenv("DISCOVERY_RECEIVER_ID", "discovery.oan-prod.local") + monkeypatch.setenv("DISCOVERY_VISIBLE_TO", "oan-prod.local, oan-dev.local, oan-dev.local") + service = _service(monkeypatch) + response = _on_publish_response("cat-oan-knowledge-provider-advisories", "ACCEPTED") + + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=_client_returning(response)): + envelope = service.publish("local-transaction", "advisory")["envelope"] + + assert envelope["context"]["senderId"] == "docs-pipeline-bv" + assert envelope["context"]["networkId"] == "oan-prod.local" + assert envelope["context"]["receiverId"] == "discovery.oan-prod.local" + assert envelope["message"]["publishDirectives"][0]["visibleTo"] == [ + "oan-prod.local", "oan-dev.local", + ] + @pytest.mark.unit def test_publish_success_envelope_shape(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "docs-pipeline-bv") @@ -97,7 +115,7 @@ def test_publish_success_envelope_shape(self, monkeypatch): mock_client.__exit__ = MagicMock(return_value=False) mock_client.post.return_value = mock_response - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=mock_client): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=mock_client): result = service.publish(transaction_id="txn-123", document_kind="advisory") mock_client.post.assert_called_once() @@ -136,7 +154,7 @@ def test_publish_success_envelope_shape(self, monkeypatch): @pytest.mark.unit def test_publish_generates_fresh_message_id_per_call(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "docs-pipeline-bv") @@ -152,7 +170,7 @@ def test_publish_generates_fresh_message_id_per_call(self, monkeypatch): mock_client.__exit__ = MagicMock(return_value=False) mock_client.post.return_value = mock_response - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=mock_client): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=mock_client): first = service.publish(transaction_id="txn-shared", document_kind="advisory") second = service.publish(transaction_id="txn-shared", document_kind="advisory") @@ -162,7 +180,7 @@ def test_publish_generates_fresh_message_id_per_call(self, monkeypatch): @pytest.mark.unit def test_publish_raises_on_http_error(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "docs-pipeline-bv") @@ -178,13 +196,13 @@ def test_publish_raises_on_http_error(self, monkeypatch): mock_client.__exit__ = MagicMock(return_value=False) mock_client.post.return_value = mock_response - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=mock_client): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=mock_client): with pytest.raises(httpx.HTTPStatusError): service.publish(transaction_id="txn-123", document_kind="advisory") @pytest.mark.unit def test_publish_raises_on_connection_error(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "docs-pipeline-bv") @@ -195,7 +213,7 @@ def test_publish_raises_on_connection_error(self, monkeypatch): mock_client.__exit__ = MagicMock(return_value=False) mock_client.post.side_effect = httpx.ConnectError("unreachable") - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=mock_client): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=mock_client): with pytest.raises(httpx.ConnectError): service.publish(transaction_id="txn-123", document_kind="advisory") @@ -203,7 +221,7 @@ def test_publish_raises_on_connection_error(self, monkeypatch): class TestDiscoveryPublishServiceSchemeCatalog: @pytest.mark.unit def test_scheme_kind_publishes_the_schemes_catalog(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "docs-pipeline-bv") @@ -219,7 +237,7 @@ def test_scheme_kind_publishes_the_schemes_catalog(self, monkeypatch): mock_client.__exit__ = MagicMock(return_value=False) mock_client.post.return_value = mock_response - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=mock_client): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=mock_client): result = service.publish(transaction_id="txn-9", document_kind="scheme") envelope = mock_client.post.call_args.kwargs["json"] @@ -235,7 +253,7 @@ class TestDiscoveryPublishServiceSkipsUnmappedKinds: @pytest.mark.unit @pytest.mark.parametrize("kind", ["document", "video", "how_to_faq", "", None]) def test_no_http_call_is_made(self, monkeypatch, kind): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "docs-pipeline-bv") @@ -243,7 +261,7 @@ def test_no_http_call_is_made(self, monkeypatch, kind): mock_client = MagicMock() - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=mock_client): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=mock_client): result = service.publish(transaction_id="txn-skip", document_kind=kind) mock_client.post.assert_not_called() @@ -262,7 +280,7 @@ def test_reports_the_result_status_for_our_catalog(self, monkeypatch, status): response = _on_publish_response("cat-oan-knowledge-provider-advisories", status) with patch( - "pipeline.discovery_publish_service.httpx.Client", + "pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=_client_returning(response), ): result = service.publish(transaction_id="txn-v", document_kind="advisory") @@ -278,7 +296,7 @@ def test_carries_the_errors_array_through(self, monkeypatch): ) with patch( - "pipeline.discovery_publish_service.httpx.Client", + "pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=_client_returning(response), ): result = service.publish(transaction_id="txn-v", document_kind="advisory") @@ -303,7 +321,7 @@ def test_picks_our_catalog_out_of_several_results(self, monkeypatch): response.json = MagicMock(return_value=body) with patch( - "pipeline.discovery_publish_service.httpx.Client", + "pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=_client_returning(response), ): result = service.publish(transaction_id="txn-v", document_kind="advisory") @@ -330,7 +348,7 @@ def test_unreadable_result_is_unknown_not_a_failure(self, monkeypatch, body): response.json = MagicMock(return_value=body) with patch( - "pipeline.discovery_publish_service.httpx.Client", + "pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=_client_returning(response), ): result = service.publish(transaction_id="txn-v", document_kind="advisory") @@ -348,7 +366,7 @@ def test_non_json_body_is_unknown_not_a_failure(self, monkeypatch): response.json = MagicMock(side_effect=ValueError("not json")) with patch( - "pipeline.discovery_publish_service.httpx.Client", + "pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=_client_returning(response), ): result = service.publish(transaction_id="txn-v", document_kind="advisory") @@ -360,7 +378,7 @@ def test_non_json_body_is_unknown_not_a_failure(self, monkeypatch): def test_skipped_publish_has_no_result_status(self, monkeypatch): service = _service(monkeypatch) - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=MagicMock()): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=MagicMock()): result = service.publish(transaction_id="txn-v", document_kind="video") assert result["skipped"] is True @@ -368,7 +386,7 @@ def test_skipped_publish_has_no_result_status(self, monkeypatch): assert result["errors"] == [] -LOGGER_NAME = "pipeline.discovery_publish_service" +LOGGER_NAME = "pipeline.network.catalog.discovery_publish_service" class TestDiscoveryPublishServiceLogging: @@ -382,7 +400,7 @@ def test_logs_the_request_and_the_response_status(self, monkeypatch, caplog): ok = _client_returning( _on_publish_response("cat-oan-knowledge-provider-advisories", "ACCEPTED") ) - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=ok): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=ok): service.publish( transaction_id="txn-log", document_kind="advisory", workflow_id="wf-log" ) @@ -407,7 +425,7 @@ def test_logs_an_error_when_the_call_fails(self, monkeypatch, caplog): client.post.side_effect = httpx.ConnectError("connection refused") with caplog.at_level(logging.ERROR, logger=LOGGER_NAME): - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=client): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=client): with pytest.raises(httpx.ConnectError): service.publish( transaction_id="txn-log", document_kind="advisory", workflow_id="wf-log" @@ -427,7 +445,7 @@ def test_skipped_publish_logs_no_request(self, monkeypatch, caplog): with caplog.at_level(logging.INFO, logger=LOGGER_NAME): with patch( - "pipeline.discovery_publish_service.httpx.Client", return_value=MagicMock() + "pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=MagicMock() ): service.publish( transaction_id="txn-log", document_kind="video", workflow_id="wf-log" @@ -458,18 +476,39 @@ def _default_provider_naming(monkeypatch): def _publish(monkeypatch, kind, group, sender_id="provider.oan.dev"): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", sender_id) service = DiscoveryPublishService() client = _client_returning(_on_publish_response("any", "ACCEPTED")) - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=client): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=client): result = service.publish(transaction_id="txn", document_kind=kind, group=group) return client.post.call_args.kwargs["json"], result class TestDiscoveryPublishServiceGroupProvider: + @pytest.mark.unit + def test_publish_uses_the_profile_domain(self, monkeypatch, tmp_path): + profile = tmp_path / "catalogs.json" + profile.write_text(json.dumps({ + "maintenance": { + "catalog_name": "Maintenance notices", + "domain": "maintenance", + "catalog_id": "catalog-maintenance", + "resource_id": "resource-maintenance", + "resource_name": "Maintenance bulletin", + "resource_attributes": {"@type": "example:MaintenanceBulletin"}, + }, + })) + monkeypatch.setenv("BECKN_CATALOG_PROFILE_PATH", str(profile)) + + envelope, result = _publish(monkeypatch, "maintenance", None) + + assert result["skipped"] is False + assert envelope["context"]["domain"] == "maintenance" + assert envelope["message"]["catalogs"][0]["id"] == "catalog-maintenance" + @pytest.mark.unit @pytest.mark.parametrize( "kind,catalog_id,resource_id,catalog_name", @@ -510,8 +549,8 @@ def test_no_group_publishes_under_the_shared_instance(self, monkeypatch, kind, c assert catalog["id"] == catalog_id assert catalog["descriptor"] == {"name": catalog_name} assert catalog["provider"] == { - "id": "bv.provider.oan.dev", - "descriptor": {"code": "bv.provider.oan.dev", "name": "Bharat Vistaar"}, + "id": "default.provider.oan.dev", + "descriptor": {"code": "default.provider.oan.dev", "name": "Default"}, } @pytest.mark.unit @@ -544,7 +583,7 @@ def test_catalog_names_are_read_from_the_environment(self, monkeypatch): @pytest.mark.unit def test_unmapped_kind_is_skipped_for_a_group(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "provider.oan.dev") @@ -557,8 +596,8 @@ def test_unmapped_kind_is_skipped_for_a_group(self, monkeypatch): @pytest.mark.unit def test_malformed_provider_map_fails_construction(self, monkeypatch, tmp_path): - from pipeline.discovery_publish_service import DiscoveryPublishService - from pipeline.network_group import NetworkProviderConfigError + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService + from pipeline.network.network_group import NetworkProviderConfigError monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "provider.oan.dev") @@ -569,8 +608,8 @@ def test_malformed_provider_map_fails_construction(self, monkeypatch, tmp_path): @pytest.mark.unit def test_bad_provider_id_template_fails_construction(self, monkeypatch): - from pipeline.discovery_publish_service import DiscoveryPublishService - from pipeline.network_group import NetworkProviderConfigError + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService + from pipeline.network.network_group import NetworkProviderConfigError monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") monkeypatch.setenv("NETWORK_SENDER_ID", "provider.oan.dev") @@ -578,3 +617,31 @@ def test_bad_provider_id_template_fails_construction(self, monkeypatch): with pytest.raises(NetworkProviderConfigError, match="NETWORK_PROVIDER_ID_TEMPLATE"): DiscoveryPublishService() + + +def test_named_publisher_uses_the_requested_network_connection(monkeypatch): + from pipeline.network.network_connections import PublishConfigError + from pipeline.network.network_publisher import get_network_publisher + + monkeypatch.setenv("NETWORK_CONNECTIONS", "dev,prod") + monkeypatch.setenv("NETWORK_CONNECTION_DEFAULT", "dev") + for environment in ("dev", "prod"): + prefix = f"NETWORK_CONNECTION_{environment.upper()}_" + monkeypatch.setenv(prefix + "TYPE", "beckn") + monkeypatch.setenv(prefix + "ID", f"oan.{environment}") + monkeypatch.setenv(prefix + "DISCOVERY_URL", f"https://{environment}.example.com") + monkeypatch.setenv(prefix + "SENDER_ID", f"provider.oan.{environment}") + monkeypatch.setenv("NETWORK_CONNECTION_PROD_RECEIVER_ID", "receiver.oan.prod") + monkeypatch.setenv("NETWORK_CONNECTION_PROD_VISIBLE_TO", "oan.dev,oan.prod,oan.dev") + client = _client_returning(_on_publish_response("unused", "ACCEPTED")) + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=client): + get_network_publisher("oan.prod").publish("txn", "advisory", "wf", None) + args, kwargs = client.post.call_args + assert args == ("https://prod.example.com/publish",) + assert kwargs["json"]["context"]["networkId"] == "oan.prod" + assert kwargs["json"]["context"]["senderId"] == "provider.oan.prod" + assert kwargs["json"]["context"]["receiverId"] == "receiver.oan.prod" + assert kwargs["json"]["message"]["publishDirectives"][0]["visibleTo"] == ["oan.dev", "oan.prod"] + + with pytest.raises(PublishConfigError, match="no configured connection"): + get_network_publisher("oan.unknown") diff --git a/tests/test_display_name.py b/tests/test_display_name.py index a235177..72ee4cd 100644 --- a/tests/test_display_name.py +++ b/tests/test_display_name.py @@ -70,42 +70,38 @@ def test_locked_when_ever_ingested(self): class TestNameOnRecords: @pytest.mark.unit - def test_display_name_becomes_name_en(self): - from pipeline.activities import _prepare_records + def test_display_name_is_the_primary_label(self): + from pipeline.search.passage_projection import _prepare_records - records = _prepare_records("doc-1", "scan_0042.pdf", [CHUNK], display_name="PM Kisan Guidelines") + record, = _prepare_records("doc-1", "scan_0042.pdf", [CHUNK], display_name="Reviewer Title") - assert records[0]["name_en"] == "PM Kisan Guidelines" + assert record["name"] == "Reviewer Title" + assert "name_en" not in record + assert "title_gu" not in record @pytest.mark.unit - def test_display_name_beats_the_manifest_title(self, monkeypatch): - import pipeline.activities as activities - - monkeypatch.setattr(activities, "_get_doc_metadata", lambda filename: {"title_en": "Manifest Title"}) - - records = activities._prepare_records("doc-1", "scan.pdf", [CHUNK], display_name="Reviewer Name") - - assert records[0]["name_en"] == "Reviewer Name" - - @pytest.mark.unit - def test_manifest_title_used_when_no_display_name(self, monkeypatch): - import pipeline.activities as activities - - monkeypatch.setattr(activities, "_get_doc_metadata", lambda filename: {"title_en": "Manifest Title"}) - - records = activities._prepare_records("doc-1", "scan.pdf", [CHUNK], display_name=None) + def test_primary_and_localized_display_fields_are_projected(self): + from pipeline.search.passage_projection import _prepare_records + + record, = _prepare_records( + "doc-1", "scan.pdf", [CHUNK], name="Primary title", + knowledge_metadata={ + "description": "Primary description", + "localized": {"fr-CA": {"title": "Titre", "description": "Description en français"}}, + }, + ) - assert records[0]["name_en"] == "Manifest Title" + assert record["name"] == "Primary title" + assert record["description"] == "Primary description" + assert record["knowledge_metadata"]["localized"]["fr-ca"]["title"] == "Titre" @pytest.mark.unit - def test_filename_stem_used_when_nothing_else(self, monkeypatch): - import pipeline.activities as activities - - monkeypatch.setattr(activities, "_get_doc_metadata", lambda filename: {}) + def test_filename_stem_used_when_nothing_else(self): + from pipeline.search.passage_projection import _prepare_records - records = activities._prepare_records("doc-1", "scan_0042.pdf", [CHUNK]) + record, = _prepare_records("doc-1", "scan_0042.pdf", [CHUNK]) - assert records[0]["name_en"] == "scan_0042" + assert record["name"] == "scan_0042" class _RecordsBuilt(Exception): @@ -115,7 +111,7 @@ class _RecordsBuilt(Exception): class TestIngestPathsCarryTheDisplayName: @pytest.fixture def captured(self, monkeypatch): - import pipeline.activities as activities + from pipeline.temporal.activities import indexing as activities import pipeline.db as db seen = {} @@ -127,14 +123,17 @@ def capture(*args, **kwargs): monkeypatch.setattr(activities, "_prepare_records", capture) monkeypatch.setattr(db, "get_chunks", lambda *a, **k: [CHUNK]) monkeypatch.setattr( - db, "get_document", lambda workflow_id: {"name": "PM Kisan Guidelines", "instance": "mh"} + db, "get_document", lambda workflow_id: { + "id": "00000000-0000-0000-0000-000000000001", + "name": "PM Kisan Guidelines", "instance": "mh" + } ) return seen @pytest.mark.unit @pytest.mark.asyncio async def test_dev_ingest(self, captured): - from pipeline.activities import ingest_document_from_db + from pipeline.temporal.activities.indexing import ingest_document_from_db with pytest.raises(_RecordsBuilt): await ingest_document_from_db("wf-1", "doc-1", "doc.pdf") @@ -146,10 +145,10 @@ async def test_dev_ingest(self, captured): async def test_prod_promotion(self, captured, monkeypatch): from unittest.mock import MagicMock - import pipeline.activities as activities + from pipeline.temporal.activities import indexing as activities monkeypatch.setenv("PROD_VECTOR_DB_URL", "http://localhost:6333") - monkeypatch.setattr(activities, "get_qdrant_client", lambda *a, **k: MagicMock(), raising=False) + monkeypatch.setattr("pipeline.vector_store.qdrant_store.get_qdrant_client", lambda *a, **k: MagicMock()) with pytest.raises(_RecordsBuilt): await activities.promote_document_to_prod_qdrant("wf-1", "doc-1", "doc.pdf") @@ -159,7 +158,7 @@ async def test_prod_promotion(self, captured, monkeypatch): @pytest.mark.unit @pytest.mark.asyncio async def test_prepare_for_ingestion(self, captured): - from pipeline.activities import prepare_for_ingestion + from pipeline.temporal.activities.indexing import prepare_for_ingestion with pytest.raises(_RecordsBuilt): await prepare_for_ingestion("doc-1", "doc.pdf", [CHUNK], workflow_id="wf-1") diff --git a/tests/test_document_formats.py b/tests/test_document_formats.py new file mode 100644 index 0000000..579cd4a --- /dev/null +++ b/tests/test_document_formats.py @@ -0,0 +1,95 @@ +from io import BytesIO +from pathlib import Path +from unittest.mock import Mock + +import httpx +import pytest +from pypdf import PdfWriter +from temporalio.testing import ActivityEnvironment + +from pipeline.documents import document_formats +from pipeline.intake import source_fetch as html_source +from pipeline.temporal.activities import source as activities + + +@pytest.fixture +def extra_format(monkeypatch): + def normalize(input_path, output_path): + raw = Path(input_path).read_bytes() + assert raw.startswith(b"FORMAT1\n") + Path(output_path).write_bytes(raw.removeprefix(b"FORMAT1\n")) + + definition = document_formats.DocumentFormat(".fixture", "application/vnd.example.fixture", normalize) + entry_point = Mock() + entry_point.load.return_value = definition + monkeypatch.setattr(document_formats, "entry_points", lambda **_: (entry_point,)) + monkeypatch.setenv("DOCUMENT_FORMAT_EXTENSIONS", "test-fixture") + document_formats._load_formats.cache_clear() + yield definition + document_formats._load_formats.cache_clear() + + +@pytest.mark.db +def test_configured_format_is_accepted_by_upload_and_crawl(test_client, db_connection, extra_format, monkeypatch): + media_type = extra_format.media_type + assert ".fixture" in test_client.get("/upload/options").json()["file_extensions"] + uploaded = test_client.post( + "/upload?kind=advisory", files={"file": ("one.fixture", b"FORMAT1\n%PDF-1.7\n", media_type)}, + ) + assert uploaded.status_code == 200, uploaded.text + + db_connection.get_or_create_instance("bv") + monkeypatch.setenv("CRAWL_SOURCE_PROFILES", '{"bv":{"notices":{"host":"notices.example"}}}') + monkeypatch.setattr(html_source, "_resolve_public_address", lambda _: "93.184.216.34") + + def response(_): + return httpx.Response(200, headers={"Content-Type": media_type}, stream=httpx.ByteStream(b"FORMAT1\n%PDF-1.7\n")) + + monkeypatch.setattr(httpx, "AsyncHTTPTransport", lambda **_: httpx.MockTransport(response)) + crawled = test_client.post("/providers/bv/crawl/notices", json={"url": "https://notices.example/two"}) + assert crawled.status_code == 202, crawled.text + document = db_connection.get_document(crawled.json()["document_id"]) + assert document["filename"] == "two.fixture" + assert document["canonical_input_type"] == "pdf" + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_worker_normalizes_extension_before_document_parser(test_client, db_connection, extra_format, tmp_path, monkeypatch): + writer = PdfWriter() + writer.add_blank_page(width=100, height=100) + pdf = BytesIO() + writer.write(pdf) + source = tmp_path / "notice.fixture" + source.write_bytes(b"FORMAT1\n" + pdf.getvalue()) + uploaded = test_client.post( + "/upload?kind=advisory", files={"file": (source.name, source.read_bytes(), extra_format.media_type)}, + ) + assert uploaded.status_code == 200, uploaded.text + workflow_id = uploaded.json()["workflow_id"] + document = db_connection.get_document(workflow_id) + normalized_paths = [] + + def save_artifact(filename, _, kind, display_name): + if kind == "normalized_pdf": + path = Path(filename) + assert path.read_bytes() == pdf.getvalue() + normalized_paths.append(path) + return f"blob://docs/{display_name}", Path(filename).stat().st_size, "application/octet-stream" + + monkeypatch.setenv("DOCUMENT_PARSER_TYPE", "mock") + monkeypatch.setattr(activities, "_resolve_local_path", lambda _: (str(source), False)) + monkeypatch.setattr(activities, "_upload_artifact", save_artifact) + result = await ActivityEnvironment().run(activities.run_ocr_and_store, workflow_id, document["filepath"]) + assert result["page_count"] == 1 + assert len(db_connection.get_pages(workflow_id)) == 1 + assert len(normalized_paths) == 1 and not normalized_paths[0].exists() + + +def test_formats_cannot_replace_builtin_media_type(monkeypatch): + entry_point = Mock() + entry_point.load.return_value = document_formats.DocumentFormat(".other", "application/pdf", Mock()) + monkeypatch.setattr(document_formats, "entry_points", lambda **_: (entry_point,)) + monkeypatch.setenv("DOCUMENT_FORMAT_EXTENSIONS", "duplicate-type") + with pytest.raises(ValueError, match="invalid or duplicate media type"): + document_formats.allowed_file_extensions() diff --git a/tests/test_document_repository.py b/tests/test_document_repository.py index 3335332..e9d15fc 100644 --- a/tests/test_document_repository.py +++ b/tests/test_document_repository.py @@ -7,7 +7,7 @@ import pytest -from pipeline.document_repository import DocumentRepository +from pipeline.documents.document_repository import DocumentRepository class FakeDb: @@ -103,7 +103,7 @@ def test_missing_document_reads_as_none(self, db_connection): class TestGetDocumentGroup: @pytest.fixture(autouse=True) def _default_no_group_instances(self, monkeypatch): - monkeypatch.delenv("PORTAL_INSTANCE", raising=False) + monkeypatch.setenv("PORTAL_INSTANCE", "bv") monkeypatch.delenv("DEFAULT_INSTANCE", raising=False) @pytest.mark.unit diff --git a/tests/test_document_revisions.py b/tests/test_document_revisions.py new file mode 100644 index 0000000..7fbd2b4 --- /dev/null +++ b/tests/test_document_revisions.py @@ -0,0 +1,195 @@ +from uuid import uuid4 + +import pytest + + +@pytest.mark.db +def test_revision_snapshot_retry_and_network_visibility(db_connection): + db = db_connection + network = "oan-dev.local" + workflow = "doc-revision-first" + doc_id = db.upsert_document( + workflow_id=workflow, document_id="a" * 32, filename="advisory.pdf", filepath="minio://documents/advisory.pdf", + instance="bv", kind="advisory", code="revision-test", + ) + job = db.get_latest_document_job(workflow) + db.update_document_job(job["id"], config_json={"release_network_id": network}) + first_id = str(uuid4()) + first = db.save_revision_snapshot(doc_id, workflow, network, first_id, {"passages": [{"text": "first"}]}) + retry = db.save_revision_snapshot(doc_id, workflow, network, str(uuid4()), {"passages": [{"text": "changed"}]}) + assert first == retry + + legacy = {"text": "legacy"} + first_hit = {"text": "first", "revision_id": first_id, "release_network_id": network} + wrong_network = {**first_hit, "release_network_id": "oan-prod.local"} + assert db.visible_release_hits([legacy, first_hit, wrong_network]) == [legacy] + with pytest.raises(ValueError): + db.activate_network_release(doc_id, workflow, "oan-prod.local", str(uuid4())) + db.activate_network_release(doc_id, workflow, network, str(uuid4())) + assert db.visible_release_hits([legacy, first_hit, wrong_network]) == [legacy, first_hit] + + db.update_document_job(job["id"], status="completed") + replacement = "doc-revision-second" + db.create_document_job( + workflow_id=doc_id, job_type="pipeline", temporal_workflow_id=replacement, + config={"release_network_id": network}, + ) + second_id = str(uuid4()) + second_hit = {"text": "second", "revision_id": second_id, "release_network_id": network} + db.save_revision_snapshot(doc_id, replacement, network, second_id, {"passages": [{"text": "second"}]}) + assert db.visible_release_hits([first_hit, second_hit]) == [first_hit] + db.activate_network_release(doc_id, replacement, network, str(uuid4())) + with pytest.raises(ValueError): + db.activate_network_release(doc_id, workflow, network, str(uuid4())) + assert db.visible_release_hits([first_hit, second_hit]) == [second_hit] + assert {revision["id"] for revision in db.list_document_revisions(doc_id)["revisions"]} == {first_id, second_id} + generation_before_withdrawal = db.list_document_revisions(doc_id)["releases"][0]["generation"] + assert not db.withdraw_network_release(doc_id, network, first_id) + assert db.withdraw_network_release(doc_id, network, second_id) + assert db.visible_release_hits([first_hit, second_hit]) == [] + db.activate_network_release(doc_id, replacement, network, str(uuid4())) + assert db.visible_release_hits([second_hit]) == [] + with pytest.raises(ValueError): + db.select_network_revision(doc_id, network, first_id, str(uuid4()), expected_generation=generation_before_withdrawal) + generation = db.list_document_revisions(doc_id)["releases"][0]["generation"] + db.select_network_revision(doc_id, network, first_id, str(uuid4()), expected_generation=generation) + assert db.visible_release_hits([first_hit, second_hit]) == [first_hit] + + +@pytest.mark.db +@pytest.mark.parametrize("operation", ["withdraw", "rollback"]) +def test_staged_revision_cannot_override_a_later_operator_selection(db_connection, operation): + db = db_connection + network = "network.example" + workflow = "doc-original" + doc_id = db.upsert_document( + workflow_id=workflow, document_id="b" * 32, filename="notice.pdf", filepath="minio://docs/notice.pdf", + instance="provider", kind="advisory", code="release-race", + ) + job = db.get_latest_document_job(workflow) + db.update_document_job(job["id"], config_json={"release_network_id": network}) + revisions = [] + for sequence in range(3): + if sequence: + db.update_document_job(job["id"], status="completed") + workflow = f"doc-replacement-{sequence}" + db.create_document_job( + workflow_id=doc_id, job_type="pipeline", temporal_workflow_id=workflow, + config={"release_network_id": network}, + ) + job = db.get_latest_document_job(doc_id) + revision_id = str(uuid4()) + db.save_revision_snapshot(doc_id, workflow, network, revision_id, {"passages": [{"text": workflow}]}) + revisions.append(revision_id) + if sequence < 2: + db.activate_network_release(doc_id, workflow, network, str(uuid4())) + + active = db.list_document_revisions(doc_id)["releases"][0] + if operation == "withdraw": + assert db.withdraw_network_release(doc_id, network, revisions[1]) + else: + db.select_network_revision(doc_id, network, revisions[0], str(uuid4()), active["generation"]) + selected = db.list_document_revisions(doc_id)["releases"] + with pytest.raises(ValueError, match="changed after this revision was staged"): + db.activate_network_release(doc_id, workflow, network, str(uuid4())) + assert db.list_document_revisions(doc_id)["releases"] == selected + + +@pytest.mark.db +def test_inactive_release_filter_refuses_to_silently_drop_ids(db_connection, sample_document): + db = db_connection + document_id = db.get_document(sample_document["workflow_id"])["id"] + with db.get_connection() as conn: + conn.execute( + """ + WITH new_workflows AS ( + INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, config) + SELECT %s::uuid, 'pipeline', 'inactive-release-' || i, '{"release_network_id":"test.net"}'::jsonb + FROM generate_series(1, 1000) AS i + RETURNING id, document_id + ), revisions AS ( + INSERT INTO document_revisions (id, document_id, workflow_id, content_sha256, snapshot) + SELECT gen_random_uuid(), document_id, id, repeat('a', 64), '{}'::jsonb FROM new_workflows + RETURNING id + ) + INSERT INTO network_release_staging (revision_id, network_id, collection) + SELECT id, 'test.net', 'test-release' FROM revisions + """, + (document_id,), + ) + assert len(db.inactive_network_revision_ids("test.net", "test-release")) == 1000 + with db.get_connection() as conn: + conn.execute( + """ + WITH new_workflow AS ( + INSERT INTO workflows (document_id, workflow_type, temporal_workflow_id, config) + VALUES (%s::uuid, 'pipeline', 'overflow-release', '{"release_network_id":"test.net"}'::jsonb) + RETURNING id, document_id + ), revisions AS ( + INSERT INTO document_revisions (id, document_id, workflow_id, content_sha256, snapshot) + SELECT gen_random_uuid(), document_id, id, repeat('a', 64), '{}'::jsonb FROM new_workflow + RETURNING id + ) + INSERT INTO network_release_staging (revision_id, network_id, collection) + SELECT id, 'test.net', 'test-release' FROM revisions + """, + (document_id,), + ) + with pytest.raises(ValueError, match="exceeds the complete pre-ranking filter limit"): + db.inactive_network_revision_ids("test.net", "test-release") + + +@pytest.mark.db +def test_one_revision_can_be_staged_and_released_independently_on_two_networks(db_connection, sample_document): + db = db_connection + workflow = sample_document["workflow_id"] + document_id = db.get_document(workflow)["id"] + job = db.get_latest_document_job(workflow) + db.update_document_job(job["id"], config_json={"release_network_id": "test.dev"}) + revision_id = str(uuid4()) + snapshot = {"passages": [{"text": "Shared content"}]} + db.save_revision_snapshot( + document_id, workflow, "test.dev", revision_id, snapshot, collection="dev-published", + ) + db.activate_network_release(document_id, workflow, "test.dev", str(uuid4())) + + transaction = str(uuid4()) + assert db.stage_saved_network_revision( + document_id, "test.prod", revision_id, transaction, None, "prod-published", "provider.test.prod", + ) + assert db.inactive_network_revision_ids("test.dev", "dev-published") == [] + assert db.inactive_network_revision_ids("test.prod", "prod-published") == [revision_id] + db.select_network_revision(document_id, "test.prod", revision_id, transaction, None) + assert db.inactive_network_revision_ids("test.prod", "prod-published") == [] + assert not db.stage_saved_network_revision( + document_id, "test.prod", revision_id, transaction, None, "prod-published", "provider.test.prod", + ) + assert db.withdraw_network_release(document_id, "test.dev", revision_id) + assert db.inactive_network_revision_ids("test.dev", "dev-published") == [revision_id] + assert db.inactive_network_revision_ids("test.prod", "prod-published") == [] + assert db.get_revision_snapshot(document_id, revision_id)["snapshot"] == snapshot + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_ingest_freezes_a_revision_without_releasing_it(db_connection, sample_document, monkeypatch): + from pipeline.temporal.activities import indexing as activities + + db = db_connection + workflow = sample_document["workflow_id"] + doc_id = db.get_document(workflow)["id"] + db.save_chunks(workflow, [{"chunk_number": 1, "original_text": "A draft can be released later."}]) + monkeypatch.setattr(activities, "_upload_artifact", lambda *_: ("blob://docs/export.json", 1, "application/json")) + + async def ingest(records, **_): + return {"records_ingested": len(records)} + + monkeypatch.setattr(activities, "ingest_to_vector_db", ingest) + await activities.ingest_document_from_db(workflow, "source", "draft.pdf", freeze_draft_revision=True) + + history = db.list_document_revisions(doc_id) + assert len(history["revisions"]) == 1 + assert history["releases"] == [] + snapshot = db.get_revision_snapshot(doc_id, history["revisions"][0]["id"])["snapshot"] + assert snapshot["passages"][0]["text"] == "A draft can be released later." + assert snapshot["passages"][0]["doc_id"] != db.get_document_index_status(workflow, "documents-index")["doc_id"] diff --git a/tests/test_email_otp.py b/tests/test_email_otp.py deleted file mode 100644 index 09cb115..0000000 --- a/tests/test_email_otp.py +++ /dev/null @@ -1,218 +0,0 @@ -"""Unit tests for the passwordless email-OTP login helpers. - -Keycloak itself is stubbed at the HTTP boundary (``_post``) — these cover the -parts we own: config resolution, email normalisation, and the translation of -Keycloak's error vocabulary into messages a signed-out user can act on. -""" - -from __future__ import annotations - -from datetime import datetime, timedelta, timezone -from unittest.mock import patch - -import pytest -from fastapi import HTTPException - -from pipeline.auth import email_otp - -ENV = { - "KEYCLOAK_ISSUER": "https://kc.example/auth/realms/test", - "KEYCLOAK_CLIENT_ID": "test-client", - "KEYCLOAK_CLIENT_SECRET": "s3cret", -} - - -def _post_returns(status, body): - return patch.object(email_otp, "_post", return_value=(status, body)) - - -def test_config_falls_back_to_vite_client_id(): - env = { - "KEYCLOAK_ISSUER": "https://kc.example/auth/realms/test/", - "VITE_KEYCLOAK_CLIENT_ID": "bharat-vistaar", - "KEYCLOAK_CLIENT_SECRET": "s3cret", - } - with patch.dict("os.environ", env, clear=True): - cfg = email_otp.load_email_otp_config() - assert cfg.client_id == "bharat-vistaar" - # Trailing slash must not survive into the derived URL. - assert cfg.token_url == "https://kc.example/auth/realms/test/protocol/openid-connect/token" - assert cfg.configured - - -def test_missing_secret_names_the_variable(): - with patch.dict("os.environ", {"KEYCLOAK_ISSUER": "https://kc.example/auth/realms/t"}, clear=True): - with pytest.raises(HTTPException) as exc: - email_otp.require_email_otp_config() - assert exc.value.status_code == 503 - assert "KEYCLOAK_CLIENT_SECRET" in exc.value.detail - - -@pytest.mark.parametrize("raw", [" User@Example.COM ", "user@example.com"]) -def test_normalize_email_lowercases_and_trims(raw): - assert email_otp.normalize_email(raw) == "user@example.com" - - -@pytest.mark.parametrize("raw", ["", " ", "not-an-email"]) -def test_normalize_email_rejects_junk(raw): - with pytest.raises(HTTPException) as exc: - email_otp.normalize_email(raw) - assert exc.value.status_code == 400 - - -def test_send_otp_emails_a_code_to_an_existing_account(): - with ( - patch.dict("os.environ", ENV, clear=True), - patch.object(email_otp.db, "get_email_otp", return_value=None), - patch.object(email_otp.db, "store_email_otp") as store_otp, - patch.object( - email_otp, "_find_keycloak_user", return_value={"id": "u1", "email": "user@example.com"} - ), - patch.object(email_otp, "_send_email") as send_email, - ): - result = email_otp.send_otp("user@example.com") - - # Enumeration-safe: the response never varies with account existence, so - # it can't leak Keycloak/SMTP details either. - assert result == { - "status": "ok", - "message": "If the account exists, a one-time code has been emailed to it", - "resend_after_seconds": email_otp.OTP_RESEND_COOLDOWN_SECONDS, - } - store_otp.assert_called_once() - send_email.assert_called_once() - assert send_email.call_args.args[0] == "user@example.com" - - -def test_send_otp_swallows_smtp_failures_for_enumeration_safety(): - # A broken SMTP relay must not produce a different response/status than - # the "no such account" path, or the difference becomes an oracle. - with ( - patch.dict("os.environ", ENV, clear=True), - patch.object(email_otp.db, "get_email_otp", return_value=None), - patch.object(email_otp.db, "store_email_otp"), - patch.object( - email_otp, "_find_keycloak_user", return_value={"id": "u1", "email": "user@example.com"} - ), - patch.object(email_otp, "_send_email", side_effect=HTTPException(503, "smtp down")), - ): - result = email_otp.send_otp("user@example.com") - - assert result == { - "status": "ok", - "message": "If the account exists, a one-time code has been emailed to it", - "resend_after_seconds": email_otp.OTP_RESEND_COOLDOWN_SECONDS, - } - - -def test_verify_returns_the_full_token_set(): - email = "user@example.com" - code = "123456" - minted = { - "access_token": "at", - "refresh_token": "rt", - "id_token": "it", - "token_type": "Bearer", - "expires_in": 300, - "refresh_expires_in": 1800, - } - with patch.dict("os.environ", ENV, clear=True): - row = { - "expires_at": (datetime.now(timezone.utc) + timedelta(minutes=5)).isoformat(), - "attempts": 0, - "code_hash": email_otp._hash_code(email, code), - } - with ( - patch.object(email_otp.db, "get_email_otp", return_value=row), - patch.object(email_otp.db, "delete_email_otp") as delete_otp, - patch.object(email_otp, "_mint_tokens_for_verified_email", return_value=minted), - ): - tokens = email_otp.verify_otp(email, code) - - delete_otp.assert_called_once_with(email) - assert tokens["access_token"] == "at" - assert tokens["refresh_token"] == "rt" - assert tokens["id_token"] == "it" - - -def _otp_row(email, code, **overrides): - row = { - "expires_at": (datetime.now(timezone.utc) + timedelta(minutes=5)).isoformat(), - "attempts": 0, - "code_hash": email_otp._hash_code(email, code), - } - row.update(overrides) - return row - - -@pytest.mark.parametrize( - "row,expected", - [ - (None, "no longer valid"), - ("expired", "expired"), - ("too_many_attempts", "Too many"), - ("wrong_code", "incorrect"), - ], - ids=["no-live-code", "expired", "too-many-attempts", "wrong-code"], -) -def test_verify_maps_failure_states_to_actionable_text(row, expected): - email, code = "user@example.com", "123456" - with patch.dict("os.environ", ENV, clear=True): - if row == "expired": - row = _otp_row( - email, code, expires_at=(datetime.now(timezone.utc) - timedelta(seconds=1)).isoformat() - ) - elif row == "too_many_attempts": - row = _otp_row(email, code, attempts=email_otp.OTP_MAX_ATTEMPTS) - elif row == "wrong_code": - row = _otp_row(email, code, code_hash="not-the-real-hash") - - with ( - patch.object(email_otp.db, "get_email_otp", return_value=row), - patch.object(email_otp.db, "delete_email_otp"), - patch.object(email_otp.db, "increment_email_otp_attempts"), - ): - with pytest.raises(HTTPException) as exc: - email_otp.verify_otp(email, code) - assert exc.value.status_code == 401 - assert expected in exc.value.detail - - -def test_verify_flags_a_missing_flow_binding_as_misconfiguration(): - # What Keycloak says when the client has no Direct Access Grants enabled, - # surfaced during the post-verification password-grant bridge. - email, code = "user@example.com", "123456" - with patch.dict("os.environ", ENV, clear=True): - row = _otp_row(email, code) - with ( - patch.object(email_otp.db, "get_email_otp", return_value=row), - patch.object(email_otp.db, "delete_email_otp"), - patch.object( - email_otp, "_find_keycloak_user", return_value={"id": "u1", "email": email} - ), - patch.object(email_otp.keycloak_admin, "require_admin_config", return_value=object()), - patch.object(email_otp.keycloak_admin, "_admin_token", return_value="admin-tok"), - patch.object(email_otp.keycloak_admin, "_admin_root", return_value="https://kc.example/admin"), - patch.object(email_otp.keycloak_admin, "_req", return_value=(204, None)), - _post_returns(400, {"error": "unauthorized_client", "error_description": "Client not allowed"}), - ): - with pytest.raises(HTTPException) as exc: - email_otp.verify_otp(email, code) - assert exc.value.status_code == 503 - assert "Direct Access Grants" in exc.value.detail - - -def test_verify_requires_a_code(): - with patch.dict("os.environ", ENV, clear=True): - with pytest.raises(HTTPException) as exc: - email_otp.verify_otp("user@example.com", " ") - assert exc.value.status_code == 400 - - -def test_refresh_reports_a_dead_session_as_401(): - with patch.dict("os.environ", ENV, clear=True), _post_returns( - 400, {"error": "invalid_grant", "error_description": "Token is not active"} - ): - with pytest.raises(HTTPException) as exc: - email_otp.refresh_tokens("stale") - assert exc.value.status_code == 401 diff --git a/tests/test_email_sign_in.py b/tests/test_email_sign_in.py new file mode 100644 index 0000000..a89664c --- /dev/null +++ b/tests/test_email_sign_in.py @@ -0,0 +1,90 @@ +import time +from urllib.parse import parse_qs + +import httpx +import pytest +from fastapi import HTTPException + +from pipeline.auth import email_sign_in +from pipeline.auth.oidc import OidcConfig + +PRIVATE = "http://keycloak:8080/auth/realms/knowledge" +ISSUER = "https://identity.example.invalid/auth/realms/knowledge" + + +@pytest.fixture +def oidc(monkeypatch): + cfg = OidcConfig(ISSUER, "console", "test-secret", PRIVATE) + monkeypatch.setattr(email_sign_in, "require_oidc_config", lambda: cfg) + return cfg + + +def test_email_sign_in_keeps_pkce_private_and_accepts_correct_code_after_retry(monkeypatch, oidc): + state = {} + calls = [] + + def render(field, step): + return httpx.Response(200, text=( + f'
' + f'
' + )) + + def serve(request): + assert str(request.url).startswith(PRIVATE) + calls.append(request.method) + if request.method == "GET": + state.update(request.url.params) + assert state["code_challenge_method"] == "S256" + response = render("username", "email") + response.headers["Set-Cookie"] = "AUTH_SESSION_ID=fixture; Path=/auth; HttpOnly; Secure" + return response + assert request.headers.get("Cookie") == "AUTH_SESSION_ID=fixture" + if request.url.params["step"] == "email": + assert parse_qs(request.content.decode()) == {"username": ["person@example.invalid"]} + return render("emailCode", "code") + if request.url.params["step"] == "code": + assert parse_qs(request.content.decode()) == {"emailCode": ["000000"]} + return render("emailCode", "retry") + return httpx.Response(302, headers={ + "Location": f'{state["redirect_uri"]}?state={state["state"]}&code=private-code', + }) + + def make_client(cookies=None): + return httpx.Client(cookies=cookies, transport=httpx.MockTransport(serve)) + + def exchange(code, redirect, verifier): + assert code == "private-code" + assert redirect == state["redirect_uri"] + assert len(verifier) >= 43 + return {"access_token": "fixture"} + + monkeypatch.setattr(email_sign_in, "private_client", make_client) + monkeypatch.setattr(email_sign_in, "_throttle", lambda cfg, address: None) + monkeypatch.setattr(email_sign_in, "exchange_authorization_code", exchange) + challenge = email_sign_in.send_email_code("PERSON@example.invalid") + assert PRIVATE not in challenge and state["state"] not in challenge + tokens, retry = email_sign_in.verify_email_code(challenge, "000000") + assert tokens is None and challenge != retry + cipher = email_sign_in._cipher(oidc) + assert cipher.extract_timestamp(challenge.encode()) == cipher.extract_timestamp(retry.encode()) + assert email_sign_in.verify_email_code(retry, "123456")[0] == {"access_token": "fixture"} + assert calls == ["GET", "POST", "POST", "POST"] + + +def test_expired_challenge_never_reaches_keycloak(monkeypatch, oidc): + challenge = email_sign_in._sealed_challenge( + oidc, {"action": "never called"}, issued_at=int(time.time()) - email_sign_in.CODE_TTL - 1, + ) + monkeypatch.setattr(email_sign_in, "private_client", lambda *_: pytest.fail("expired challenge reached Keycloak")) + with pytest.raises(HTTPException) as error: + email_sign_in.verify_email_code(challenge, "123456") + assert error.value.status_code == 401 + + +@pytest.mark.db +def test_send_throttle_applies_across_requests(db_connection, oidc): + email = "same-address@example.invalid" + email_sign_in._throttle(oidc, email) + with pytest.raises(HTTPException) as error: + email_sign_in._throttle(oidc, email) + assert error.value.status_code == 429 diff --git a/tests/test_embedding_prompt_styles.py b/tests/test_embedding_prompt_styles.py new file mode 100644 index 0000000..10477c4 --- /dev/null +++ b/tests/test_embedding_prompt_styles.py @@ -0,0 +1,12 @@ +from pipeline.vector_store.embeddings import format_embedding_input + + +def test_embeddinggemma_has_distinct_query_and_passage_prompts(monkeypatch): + monkeypatch.setenv("EMBEDDING_PROMPT_STYLE", "embeddinggemma") + assert format_embedding_input("water", "query", use_e5_prefix=False) == "task: search result | query: water" + assert format_embedding_input("water", "passage") == "title: none | text: water" + + +def test_plain_does_not_add_e5_prompt(monkeypatch): + monkeypatch.setenv("EMBEDDING_PROMPT_STYLE", "plain") + assert format_embedding_input("water", "query") == "water" diff --git a/tests/test_embeddings_count_tokens.py b/tests/test_embeddings_count_tokens.py index c469f9e..4f30468 100644 --- a/tests/test_embeddings_count_tokens.py +++ b/tests/test_embeddings_count_tokens.py @@ -1,5 +1,8 @@ """Tests for the embedding token counter used to size advisory chunks.""" +import sys +from concurrent.futures import ThreadPoolExecutor +from threading import Barrier, Event from types import SimpleNamespace from pipeline.vector_store import embeddings @@ -22,3 +25,35 @@ def test_count_tokens_remote_estimates_from_length(monkeypatch): def test_count_tokens_remote_is_at_least_one(monkeypatch): monkeypatch.setenv("EMBEDDING_PROVIDER", "openai_compatible") assert embeddings.count_tokens("") >= 1 + + +def test_parallel_requests_share_one_model_initialization(monkeypatch): + started = Barrier(3) + initialized = Event() + proceed = Event() + duplicate = Event() + model = object() + + def load(_model_name): + if initialized.is_set(): + duplicate.set() + initialized.set() + assert proceed.wait(timeout=5) + return model + + def request_model(): + started.wait(timeout=5) + return embeddings._get_local_embedder() + + monkeypatch.setattr(embeddings, "_local_embedder", None) + monkeypatch.setitem(sys.modules, "sentence_transformers", SimpleNamespace(SentenceTransformer=load)) + + with ThreadPoolExecutor(max_workers=2) as executor: + pending = [executor.submit(request_model) for _ in range(2)] + try: + started.wait(timeout=5) + assert initialized.wait(timeout=5) + assert not duplicate.wait(timeout=0.1) + finally: + proceed.set() + assert all(result.result(timeout=5) is model for result in pending) diff --git a/tests/test_google_sign_in.py b/tests/test_google_sign_in.py new file mode 100644 index 0000000..9431914 --- /dev/null +++ b/tests/test_google_sign_in.py @@ -0,0 +1,120 @@ +from urllib.parse import urlencode + +import httpx +import pytest +from fastapi import HTTPException + +from pipeline.auth import google_sign_in as google +from pipeline.auth.oidc import OidcConfig + +ISSUER = "https://console.example.invalid/auth/realms/kp" +PRIVATE = "http://keycloak:8080/auth/realms/kp" +CONSOLE = google.GoogleConsole("https://console.example.invalid/knowledge-provider/login") + + +def test_google_exchange_stays_private_and_rejects_wrong_state(monkeypatch): + config = OidcConfig(ISSUER, "console", "fixture-secret", PRIVATE) + monkeypatch.setattr(google, "require_oidc_config", lambda: config) + calls = [] + auth = {} + + def redirect(url): + return httpx.Response(303, headers={"Location": url}) + + def broker(request): + assert str(request.url).startswith(PRIVATE) + calls.append(request.url.path.rsplit("/", 1)[-1]) + if request.url.path.endswith("/auth"): + auth.update(request.url.params) + return httpx.Response(303, headers={ + "Location": ISSUER + "/broker/google/login?client_id=console", + "Set-Cookie": "AUTH_SESSION_ID=fixture; Secure; HttpOnly; Path=/auth", + }) + assert request.headers.get("Cookie") == "AUTH_SESSION_ID=fixture" + if request.url.path.endswith("/login"): + query = urlencode({ + "state": "google-state", "redirect_uri": ISSUER + "/broker/google/endpoint", + }) + return redirect("https://accounts.google.com/o/oauth2/v2/auth?" + query) + assert request.url.params["code"] == "google-code" + assert request.url.params["state"] == "google-state" + return redirect(auth["redirect_uri"] + "?" + urlencode({ + "code": "keycloak-code", "state": auth["state"], + })) + + monkeypatch.setattr(google, "private_client", lambda cookies=None: httpx.Client( + transport=httpx.MockTransport(broker), cookies=cookies, + )) + + def exchange(code, redirect_uri, verifier): + assert code == "keycloak-code" + assert redirect_uri == auth["redirect_uri"] + assert len(verifier) >= 43 + return {"access_token": "fixture"} + + monkeypatch.setattr(google, "exchange_authorization_code", exchange) + location, cookie = google.start_google_sign_in(CONSOLE) + assert location.startswith("https://accounts.google.com/") + assert "AUTH_SESSION_ID" not in cookie + assert auth["code_challenge_method"] == "S256" + + with pytest.raises(HTTPException) as error: + google.accept_google_callback(cookie, "google-code", "wrong-state") + assert error.value.status_code == 401 + assert calls == ["auth", "login"] + + ticket = google.accept_google_callback(cookie, "google-code", "google-state") + assert calls == ["auth", "login", "endpoint"] + assert google.complete_google_sign_in(ticket) == {"access_token": "fixture"} + + +def test_google_rejects_callback_for_a_different_origin(monkeypatch): + cfg = OidcConfig(ISSUER, "console", "fixture-secret", PRIVATE) + monkeypatch.setattr(google, "require_oidc_config", lambda: cfg) + + def broker(request): + if request.url.path.endswith("/auth"): + location = ISSUER + "/broker/google/login" + else: + params = urlencode({"state": "a", "redirect_uri": "https://unrelated.invalid/return"}) + location = "https://accounts.google.com/o/oauth2/v2/auth?" + params + return httpx.Response(303, headers={"Location": location}) + + monkeypatch.setattr(google, "private_client", lambda: httpx.Client( + transport=httpx.MockTransport(broker), + )) + with pytest.raises(HTTPException) as error: + google.start_google_sign_in(CONSOLE) + assert error.value.status_code == 502 + + +def test_user_update_drops_only_the_legacy_password_action(monkeypatch): + from pipeline.auth import keycloak_admin + + realm = "http://keycloak:8080/auth/admin/realms/kp" + user = { + "id": "existing-user", "username": "operator", "email": "operator@example.invalid", + "requiredActions": ["UPDATE_PASSWORD", "VERIFY_EMAIL"], + } + updates = [] + + def request(method, url, *, token, body=None): + if method == "PUT" and url == f"{realm}/users/existing-user": + updates.append(body) + return 204, None + if method == "GET" and url.startswith(realm + "/users?"): + return 200, [user] + if method == "GET" and url == f"{realm}/users/existing-user": + return 200, user + return 200, [] + + monkeypatch.setattr(keycloak_admin, "require_admin_config", lambda: object()) + monkeypatch.setattr(keycloak_admin, "_admin_root", lambda _: realm) + monkeypatch.setattr(keycloak_admin, "_admin_token", lambda _: "fixture") + monkeypatch.setattr(keycloak_admin, "_req", request) + monkeypatch.setattr(keycloak_admin, "ensure_access_group", lambda *a, **kw: ("/global/super-admin", {"id": "group"})) + + result = keycloak_admin.provision_user(email=user["email"], access_type="super_admin") + assert result["user_id"] == user["id"] + assert updates[0]["requiredActions"] == ["VERIFY_EMAIL"] + assert "credentials" not in updates[0] diff --git a/tests/test_groups.py b/tests/test_groups.py index 96bcb23..1fa880c 100644 --- a/tests/test_groups.py +++ b/tests/test_groups.py @@ -2,7 +2,10 @@ from __future__ import annotations +import pytest + from pipeline.auth.groups import ( + ROLE_GLOBAL_VIEWER, ROLE_STATE_ADMIN, ROLE_STATE_APPROVER, ROLE_STATE_CONTRIBUTOR, @@ -12,6 +15,7 @@ role_for_instance, ) from pipeline.auth.jwt import claims_to_user +from pipeline.auth.keycloak_admin import _access_summary_from_groups, ensure_access_group from pipeline.auth.permissions import Permission, UserRole, permissions_for_roles from pipeline.auth.tenancy import allowed_instances, user_can_access_instance @@ -103,9 +107,11 @@ def test_only_state_admin_and_super_admin_hold_delete_own(): ) -def test_bh_viewer_reads_every_state_without_write(): - access = parse_group_paths(["/global/bh-viewer"]) - assert access.is_bh_viewer is True +@pytest.mark.parametrize("path", ["/global/global-viewer", "/global/bh-viewer"]) +def test_global_viewer_reads_every_provider_without_write(path): + access = parse_group_paths([path]) + assert access.is_global_viewer is True + assert access.roles == [ROLE_GLOBAL_VIEWER] assert access.is_super_admin is False assert access.state_roles == {} # No per-state group, but any state resolves to view. @@ -113,10 +119,96 @@ def test_bh_viewer_reads_every_state_without_write(): assert role_for_instance(access, "wb") == ROLE_STATE_VIEW -def test_bh_viewer_keeps_stronger_state_role(): - access = parse_group_paths(["/global/bh-viewer", "/states/MH/admin"]) - assert role_for_instance(access, "mh") == ROLE_STATE_ADMIN - assert role_for_instance(access, "up") == ROLE_STATE_VIEW +@pytest.mark.parametrize("path", ["/global/global-viewer", "/global/bh-viewer"]) +def test_global_viewer_keeps_stronger_provider_role(path): + groups = [path, "/states/MH/admin"] + user = claims_to_user({"sub": "mixed-viewer", "groups": groups}, authoritative_groups=True) + assert allowed_instances(user) is None + assert user.has_permission_for_instance(Permission.SEARCH, "up") + assert user.has_permission_for_instance(Permission.UPLOAD, "mh") + assert not user.has_permission_for_instance(Permission.UPLOAD, "up") + summary = _access_summary_from_groups(groups) + assert summary["access_type"] == ROLE_GLOBAL_VIEWER + assert summary["access_label"] == "Viewer" + assert summary["provider_roles"] == {"mh": ROLE_STATE_ADMIN} + + +@pytest.mark.parametrize("role", ["global_viewer", "bh_viewer"]) +def test_global_viewer_with_legacy_scoped_claim_keeps_provider_write_boundary(role): + user = claims_to_user({ + "sub": "mixed-viewer", + "realm_access": {"roles": [role, ROLE_STATE_ADMIN]}, + "instances": ["mh"], + }) + assert ROLE_GLOBAL_VIEWER in user.roles + assert allowed_instances(user) is None + assert user.has_permission_for_instance(Permission.SEARCH, "up") + assert user.has_permission_for_instance(Permission.UPLOAD, "mh") + assert not user.has_permission_for_instance(Permission.UPLOAD, "up") + + +@pytest.mark.parametrize("role", ["global_viewer", "bh_viewer"]) +def test_provisioning_global_viewer_always_uses_new_keycloak_group(role, monkeypatch): + from pipeline.auth import keycloak_admin + + group_calls = [] + role_calls = [] + monkeypatch.setattr(keycloak_admin, "_admin_token", lambda _cfg: "fake-token") + monkeypatch.setattr(keycloak_admin, "_admin_root", lambda _cfg: "https://id.example.invalid") + monkeypatch.setattr(keycloak_admin, "_get_group", lambda _admin, _token, group: {"id": group}) + monkeypatch.setattr(keycloak_admin, "_map_role_on_group", lambda *_args: None) + + def ensure_child_group(_admin, _token, parent, name): + group_calls.append((parent, name)) + return {"id": name} + + def ensure_realm_role(_admin, _token, name, _description): + role_calls.append(name) + return {"name": name} + + monkeypatch.setattr(keycloak_admin, "_ensure_child_group", ensure_child_group) + monkeypatch.setattr(keycloak_admin, "_ensure_realm_role", ensure_realm_role) + path, group = ensure_access_group(None, access_type=role, state=None, role=None) + assert path == "/global/global-viewer" + assert group == {"id": "global-viewer"} + assert group_calls == [(None, "global"), ("global", "global-viewer")] + assert role_calls == [ROLE_GLOBAL_VIEWER] + + +def test_reassigning_legacy_viewer_keeps_provider_group(monkeypatch): + from pipeline.auth import keycloak_admin + + groups = { + "previous": "/global/bh-viewer", + "provider": "/states/ALPHA/admin", + } + monkeypatch.setattr(keycloak_admin, "require_admin_config", lambda: None) + monkeypatch.setattr(keycloak_admin, "_admin_token", lambda _cfg: "fake-token") + monkeypatch.setattr(keycloak_admin, "_admin_root", lambda _cfg: "https://id.example.invalid") + monkeypatch.setattr( + keycloak_admin, "ensure_access_group", + lambda *_args, **_kwargs: ("/global/global-viewer", {"id": "replacement"}), + ) + monkeypatch.setattr( + keycloak_admin, "_list_keycloak", + lambda *_args: [{"id": key, "path": path} for key, path in groups.items()], + ) + + def request(method, url, **_kwargs): + if method == "PUT": + groups[url.split("/")[-1]] = "/global/global-viewer" + elif method == "DELETE": + del groups[url.split("/")[-1]] + if url.endswith("/role-mappings/realm"): + return 200, [] + return 200, {"id": "reader", "email": "reader@example.invalid"} + + monkeypatch.setattr(keycloak_admin, "_req", request) + + outcome = keycloak_admin.set_user_access(user_id="reader", access_type="global_viewer") + assert outcome["removed_groups"] == ["/global/bh-viewer"] + assert outcome["groups"] == ["/states/ALPHA/admin", "/global/global-viewer"] + assert outcome["provider_roles"] == {"alpha": ROLE_STATE_ADMIN} def test_higher_role_wins_same_state(): diff --git a/tests/test_html_source.py b/tests/test_html_source.py new file mode 100644 index 0000000..729f0a1 --- /dev/null +++ b/tests/test_html_source.py @@ -0,0 +1,302 @@ +from unittest.mock import AsyncMock + +import httpx +import pytest + +from pipeline.intake import source_fetch as html_source + + +@pytest.mark.asyncio +async def test_fetch_pins_address_but_preserves_tls_name_and_extracts_source(monkeypatch): + def respond(request): + assert request.url.host == "93.184.216.34" + assert request.extensions["sni_hostname"] == "notices.example" + assert request.headers["Host"] == "notices.example" + return httpx.Response( + 200, headers={"Content-Type": "text/html; charset=UTF-8"}, + stream=httpx.ByteStream( + 'फसल नोटिस' + '

फसल की सलाह

' + '

दूसरा पैराग्राफ

'.encode() + ), + ) + + monkeypatch.setattr(html_source, "_resolve_public_address", lambda host: "93.184.216.34") + monkeypatch.setattr(httpx, "AsyncHTTPTransport", lambda **kwargs: httpx.MockTransport(respond)) + result = await html_source.fetch_html_source("https://notices.example/notice?id=123", {"notices.example"}) + assert result.language == "hi-in" + assert result.title == "फसल नोटिस" + assert result.text == "फसल की सलाह\n\nदूसरा पैराग्राफ" + assert result.attribution_url == "https://notices.example/notice" + + +@pytest.mark.parametrize("url", ["http://notices.example", "https://other.example", "https://notices.example:8443"]) +@pytest.mark.asyncio +async def test_source_policy_is_checked_before_resolution(monkeypatch, url): + def unexpected_resolution(host): + pytest.fail("No DNS request should be sent for a URL outside the configured source policy") + + monkeypatch.setattr(html_source, "_resolve_public_address", unexpected_resolution) + with pytest.raises(html_source.SourcePageError): + await html_source.fetch_html_source(url, {"notices.example"}) + + +@pytest.mark.db +def test_crawl_creates_one_idempotent_document_job(test_client, db_connection, mock_temporal_client, monkeypatch): + from pipeline.api.routers import crawl + + db_connection.get_or_create_instance("bv") + monkeypatch.setenv("CRAWL_SOURCE_PROFILES", '{"bv":{"notices":{"host":"notices.example"}}}') + monkeypatch.setattr(crawl, "fetch_source", AsyncMock(return_value=html_source.SourcePage( + "Cotton bulletin", "This advisory explains how to check the cotton crop after rain.", "en", + "https://notices.example/bulletin", + ))) + path = "/providers/bv/crawl/notices" + request = {"url": "https://notices.example/bulletin", "external_id": "bulletin-1"} + first = test_client.post(path, json=request) + second = test_client.post(path, json=request) + assert first.status_code == second.status_code == 202 + assert first.json()["document_id"] == second.json()["document_id"] + assert first.json()["workflow_id"] == second.json()["workflow_id"] + assert first.json()["source_url"] == "https://notices.example/bulletin" + assert db_connection.get_document(first.json()["workflow_id"])["name"] == "Cotton bulletin" + assert mock_temporal_client.start_workflow.call_count == 1 + + +@pytest.mark.db +def test_crawl_pdf_uses_upload_and_keeps_source_attribution( + test_client, db_connection, mock_temporal_client, sample_pdf_content, monkeypatch, +): + from pipeline.api.routers import crawl + + db_connection.get_or_create_instance("bv") + monkeypatch.setenv("CRAWL_SOURCE_PROFILES", '{"bv":{"notices":{"host":"notices.example"}}}') + monkeypatch.setattr(crawl, "fetch_source", AsyncMock(return_value=html_source.SourceFile( + "notice.pdf", "application/pdf", sample_pdf_content, "https://notices.example/notice.pdf", + ))) + url = "/providers/bv/crawl/notices" + body = {"url": "https://notices.example/notice.pdf?version=1", "external_id": "notice"} + first = test_client.post(url, json=body) + second = test_client.post(url, json=body) + assert first.status_code == second.status_code == 202, first.text + assert first.json()["workflow_id"] == second.json()["workflow_id"] + job = db_connection.get_latest_document_job(first.json()["document_id"]) + assert job["config"]["source_url"] == "https://notices.example/notice.pdf" + assert next(iter(job["config"]["private_source_responses"].values()))["size_bytes"] == len(sample_pdf_content) + assert mock_temporal_client.start_workflow.call_count == 1 + + +@pytest.mark.db +def test_crawl_json_uses_the_named_intake_template(test_client, db_connection, monkeypatch, tmp_path): + import json + + from pipeline.api.routers import crawl + from pipeline.intake.source_intake import materialize_intake_pages + + db_connection.get_or_create_instance("bv") + templates = tmp_path / "templates.json" + templates.write_text(json.dumps({ + "notices": {"fields": {"external_id": "/id", "title": "/name", "text": "/body"}}, + })) + monkeypatch.setenv("KNOWLEDGE_INTAKE_TEMPLATES_FILE", str(templates)) + monkeypatch.setenv("CRAWL_SOURCE_PROFILES", json.dumps({ + "bv": {"notices": {"host": "notices.example", "intake_template": "notices"}}, + })) + raw = {"id": "1", "name": "Notice", "body": "The full notice text.", "upstreamVersion": 1} + url = "https://notices.example/notice.json" + + async def fetch(*_): + return html_source.SourceFile("notice.json", "application/json", json.dumps(raw).encode(), url) + + monkeypatch.setattr(crawl, "fetch_source", fetch) + endpoint = "/providers/bv/crawl/notices" + response = test_client.post(endpoint, json={"url": url}) + assert response.status_code == 202, response.text + item = response.json() + job = db_connection.get_latest_document_job(item["document_id"]) + materialize_intake_pages(item["workflow_id"], job["config"]["upload_start_args"][2]) + assert db_connection.get_pages(item["document_id"])[0]["original_markdown"] == "The full notice text." + with db_connection.get_connection() as conn: + saved = conn.execute( + "SELECT raw_payload, normalized_document FROM source_intake_receipts WHERE id = %s", (item["receipt_id"],), + ).fetchone() + assert saved["raw_payload"] == raw + assert saved["normalized_document"]["text"] == raw["body"] + assert "upstreamVersion" not in saved["normalized_document"] + + replayed = test_client.post(endpoint, json={"url": url}) + assert replayed.status_code == 202, replayed.text + assert replayed.json()["receipt_id"] == item["receipt_id"] + with monkeypatch.context() as release_policy: + release_policy.setattr(crawl, "upload_release_target", lambda _, **kwargs: ("dev", "published")) + conflict = test_client.post(endpoint, json={"url": url}) + assert conflict.status_code == 409, conflict.text + assert "different release policy" in conflict.json()["detail"] + + db_connection.update_document_job(job["id"], status="completed") + raw["upstreamVersion"] = 2 + updated = test_client.post(endpoint, json={"url": url}) + assert updated.status_code == 202, updated.text + assert updated.json()["document_id"] == item["document_id"] + assert updated.json()["receipt_id"] != item["receipt_id"] + + +@pytest.mark.db +def test_json_crawl_reuses_a_legacy_normalized_receipt(test_client, db_connection, monkeypatch, tmp_path): + import json + + from pipeline.api.routers import crawl + from pipeline.intake.source_intake import ( + NormalizedSourceDocument, + canonical_json_sha256, + register_source_document, + ) + + db_connection.get_or_create_instance("bv") + templates = tmp_path / "templates.json" + templates.write_text(json.dumps({ + "notices": {"fields": {"external_id": "/external_id", "title": "/title", "text": "/text"}}, + })) + monkeypatch.setenv("KNOWLEDGE_INTAKE_TEMPLATES_FILE", str(templates)) + monkeypatch.setenv("CRAWL_SOURCE_PROFILES", json.dumps({ + "bv": {"notices": {"host": "notices.example", "intake_template": "notices"}}, + })) + url = "https://notices.example/notice.json" + payload = {"external_id": "1", "title": "Notice", "text": "Full notice text"} + document = NormalizedSourceDocument(**payload, source_url=url) + normalized = document.model_dump(mode="json", exclude_none=True) + with db_connection.get_connection() as conn: + old = register_source_document( + conn, instance="bv", source_kind="crawl", source_id="notices", document=document, + idempotency_key=canonical_json_sha256(normalized), raw_payload=normalized, release_target=None, + ) + raw = {**payload, "upstreamVersion": 1} + + async def fetch(*_): + return html_source.SourceFile("notice.json", "application/json", json.dumps(raw).encode(), url) + + monkeypatch.setattr(crawl, "fetch_source", fetch) + endpoint = "/providers/bv/crawl/notices" + for _ in range(2): + replayed = test_client.post(endpoint, json={"url": url}) + assert replayed.status_code == 202, replayed.text + assert replayed.json()["receipt_id"] == old["id"] + with db_connection.get_connection() as conn: + assert conn.execute("SELECT count(*) AS n FROM source_intake_receipts").fetchone()["n"] == 1 + + db_connection.update_document_job(old["job_id"], status="completed") + raw["upstreamVersion"] = 2 + updated = test_client.post(endpoint, json={"url": url}) + assert updated.status_code == 202, updated.text + assert updated.json()["document_id"] == old["document_id"] + assert updated.json()["receipt_id"] != old["id"] + with db_connection.get_connection() as conn: + saved = conn.execute( + "SELECT raw_payload AS legacy_payload, replay_evidence_payload AS observed_payload " + "FROM source_intake_receipts WHERE id = %s", + (old["id"],), + ).fetchone() + assert saved["legacy_payload"] == normalized + assert saved["observed_payload"] == {**payload, "upstreamVersion": 1} + + +@pytest.mark.asyncio +@pytest.mark.parametrize("media_type,body", [ + ("text/html", b"

Notice

\r\n"), + ("text/plain", b" Notice\r\n"), + ("application/json", b'{ "external_id": "1", "title":"Notice", "text":"Advice" }\n'), + ("application/pdf", b"%PDF-1.4\nfixture"), +]) +async def test_public_url_retains_exact_bytes_without_configured_host(monkeypatch, media_type, body): + def respond(request): + assert "authorization" not in request.headers + assert request.url.host == "93.184.216.34" + return httpx.Response(200, headers={"content-type": media_type}, stream=httpx.ByteStream(body)) + + monkeypatch.setattr(html_source, "_resolve_public_address", lambda host: "93.184.216.34") + monkeypatch.setattr(httpx, "AsyncHTTPTransport", lambda **kwargs: httpx.MockTransport(respond)) + result = await html_source.fetch_source("https://notices.example/notice") + assert result.content == body + assert result.media_type == media_type + + +@pytest.mark.asyncio +async def test_authenticated_fetch_requires_saved_host_policy(monkeypatch): + monkeypatch.setattr(html_source, "_read_source", AsyncMock()) + with pytest.raises(html_source.SourcePageError, match="saved host policy"): + await html_source.fetch_source("https://notices.example/notice", authorization="fixture-only") + html_source._read_source.assert_not_called() + + +@pytest.mark.asyncio +async def test_public_source_redirect_is_not_followed(monkeypatch): + calls = [] + + def respond(request): + calls.append(request) + return httpx.Response(302, headers={"location": "https://another.example/notice"}) + + monkeypatch.setattr(html_source, "_resolve_public_address", lambda host: "93.184.216.34") + monkeypatch.setattr(httpx, "AsyncHTTPTransport", lambda **kwargs: httpx.MockTransport(respond)) + with pytest.raises(html_source.SourcePageError): + await html_source.fetch_source("https://notices.example/notice") + assert len(calls) == 1 + + +def test_raw_response_is_private_and_job_output_omits_capture_metadata(): + import hashlib + import json + from unittest.mock import Mock + + from pipeline.intake.source_intake import public_source_job, store_source_response + + raw = b"Notice\r\n" + source = html_source.extract_text_source(raw, "text/html", "Notice") + store = Mock() + capture = store_source_response(store, "documents", "example", source) + ref, data, size, media_type = store.put.call_args.args + assert data.getvalue() == raw + assert size == len(raw) + assert media_type == "application/octet-stream" + assert ref.key.startswith("private/source-responses/") + assert capture["sha256"] == hashlib.sha256(raw).hexdigest() + config = {"private_source_responses": {capture["sha256"]: capture}, "source_url": "https://notices.example/notice"} + public = public_source_job({"config": config, "config_json": json.dumps(config)}) + assert "private_source_responses" not in public["config"] + assert "private_source_responses" not in public["config_json"] + assert "private_source_responses" in config + + +@pytest.mark.db +def test_public_crawl_without_profiles_keeps_private_original( + test_client, db_connection, mock_object_store, mock_temporal_client, monkeypatch, +): + from pipeline.api.routers import crawl + + mock_temporal_client.get_workflow_handle.return_value.describe.side_effect = RuntimeError("No live Temporal in this test") + + db_connection.get_or_create_instance("bv") + monkeypatch.delenv("CRAWL_SOURCE_PROFILES", raising=False) + raw = b"

Cotton advisory after rain.

\r\n" + source = html_source.extract_text_source(raw, "text/html", "Advisory", "https://notices.example/notice") + fetch = AsyncMock(return_value=source) + monkeypatch.setattr(crawl, "fetch_source", fetch) + result = test_client.post("/providers/bv/crawl", json={"url": "https://notices.example/notice"}) + assert result.status_code == 202, result.text + fetch.assert_awaited_once_with("https://notices.example/notice", None, None) + assert mock_object_store.put.call_args.args[1].getvalue() == raw + item = result.json() + job = db_connection.get_latest_document_job(item["document_id"]) + captures = job["config"]["private_source_responses"] + assert next(iter(captures.values()))["size_bytes"] == len(raw) + assert not db_connection.list_document_artifacts(item["workflow_id"]) + jobs = test_client.get(f'/documents/{item["workflow_id"]}/jobs') + assert jobs.status_code == 200 + assert "private_source_responses" not in jobs.text + assert "private/source-responses" not in jobs.text + document_path = f'/documents/{item["workflow_id"]}' + for endpoint in ("/runs", f'/runs/{job["id"]}', document_path, f"{document_path}/graph", f"{document_path}/runtime"): + response = test_client.get(endpoint) + assert response.status_code == 200, response.text + assert "private_source_responses" not in response.text + assert "private/source-responses" not in response.text diff --git a/tests/test_import_legacy_search.py b/tests/test_import_legacy_search.py new file mode 100644 index 0000000..0402b82 --- /dev/null +++ b/tests/test_import_legacy_search.py @@ -0,0 +1,47 @@ +from types import SimpleNamespace + +import pytest + +from pipeline import db +from tests.migration_kit import load_migration_tool + + +class LegacyCollection: + def __init__(self, fingerprint): + self.fingerprint = fingerprint + + def scroll(self, _collection, **_): + return [SimpleNamespace(payload={ + "workflow_id": "legacy-public-document", "doc_id": self.fingerprint, + "chunk_num": 1, "text": "Previously searchable passage.", + "doc_language": "gu", + })], None + + +@pytest.mark.db +def test_import_uses_published_text_without_reopening_review(db_connection): + import_legacy_search = load_migration_tool("import_legacy_search").import_legacy_search + project = load_migration_tool("rebuild_qdrant_from_postgres")._projection + fingerprint = "a" * 32 + document_id = db.upsert_document( + workflow_id="legacy-public-document", document_id=fingerprint, filename="legacy.pdf", + filepath="minio://docs/legacy.pdf", instance="bv", kind="advisory", code="legacy", + ) + db.update_document_stage("legacy-public-document", "approval_for_prod") + assert import_legacy_search(LegacyCollection(fingerprint), "published")["passages"] == 1 + assert db.list_document_revisions(document_id)["revisions"] == [] + for _ in range(2): + import_legacy_search(LegacyCollection(fingerprint), "published", apply=True) + with pytest.raises(ValueError, match="source version"): + import_legacy_search(LegacyCollection("b" * 32), "published", apply=True) + + document = db.get_document(document_id) + assert document["stage"] == "approval_for_prod" + assert document["workflow_id"] == "legacy-public-document" + revisions = db.list_document_revisions(document_id) + assert len(revisions["revisions"]) == 1 + assert revisions["releases"] == [] + records = project(document, private_rehearsal=False) + assert records[0]["text"] == "Previously searchable passage." + assert records[0]["instance"] == "bv" + assert "primary_language" not in records[0] diff --git a/tests/test_index_identity.py b/tests/test_index_identity.py new file mode 100644 index 0000000..b9a9d28 --- /dev/null +++ b/tests/test_index_identity.py @@ -0,0 +1,44 @@ +"""Pure checks for document-owned projection keys; no application configuration.""" + +import unittest + +from pipeline.search.index_identity import projection_doc_id, projection_owner, recorded_doc_id + + +class TestIndexIdentity(unittest.TestCase): + def setUp(self): + self.document = { + "id": "00000000-0000-0000-0000-000000000001", + "document_id": "a" * 32, + } + + def test_identical_files_have_independent_document_keys(self): + other = {**self.document, "id": "00000000-0000-0000-0000-000000000002"} + self.assertNotEqual(projection_doc_id(self.document), projection_doc_id(other)) + + def test_retries_keep_the_key_and_replacements_change_it(self): + key = projection_doc_id(self.document) + self.assertEqual(key, projection_doc_id({**self.document, "workflow_id": "new-attempt"})) + self.assertNotEqual(key, projection_doc_id(self.document, "b" * 32)) + + def test_source_identity_remains_unchanged(self): + projection_doc_id(self.document) + self.assertEqual(self.document["document_id"], "a" * 32) + + def test_scoped_citation_resolves_to_its_owner(self): + self.assertEqual(projection_owner(projection_doc_id(self.document)), self.document["id"]) + for identifier in ("a" * 32, "document:invalid:source", "document::"): + self.assertIsNone(projection_owner(identifier)) + + def test_inspection_and_deletion_use_the_recorded_network_key(self): + document = {**self.document, "index_status": { + "dev": {"doc_id": projection_doc_id(self.document)}, + "prod": {"doc_id": "old-source-fingerprint"}, + }} + self.assertEqual(recorded_doc_id(document, "dev"), projection_doc_id(self.document)) + self.assertEqual(recorded_doc_id(document, "prod"), "old-source-fingerprint") + self.assertEqual(recorded_doc_id(self.document, "dev"), "a" * 32) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_instances.py b/tests/test_instances.py index e37c3c7..f725b56 100644 --- a/tests/test_instances.py +++ b/tests/test_instances.py @@ -22,20 +22,20 @@ class TestGroupDisplayName: ], ) def test_known_codes(self, code, expected): - from pipeline.instances import instance_display_name + from pipeline.network.release_policy import instance_display_name assert instance_display_name(code) == expected @pytest.mark.unit @pytest.mark.parametrize("code,expected", [("or", "Odisha"), ("od", "Odisha"), ("ts", "Telangana"), ("tg", "Telangana")]) def test_legacy_code_aliases(self, code, expected): - from pipeline.instances import instance_display_name + from pipeline.network.release_policy import instance_display_name assert instance_display_name(code) == expected @pytest.mark.unit def test_blank_falls_back_to_default(self): - from pipeline.instances import instance_display_name + from pipeline.network.release_policy import instance_display_name assert instance_display_name("") == "Default" assert instance_display_name(None) == "Default" @@ -43,7 +43,7 @@ def test_blank_falls_back_to_default(self): @pytest.mark.unit def test_unknown_code_is_made_presentable(self): """A new state must degrade to a readable label, never break ingestion.""" - from pipeline.instances import instance_display_name + from pipeline.network.release_policy import instance_display_name assert instance_display_name("new-state") == "New State" @@ -54,7 +54,7 @@ class TestProdStageDisabled: @pytest.mark.unit @pytest.mark.parametrize("value", ["true", "TRUE", "True", "1", "yes", "YES"]) def test_truthy_values_disable_prod(self, monkeypatch, value): - from pipeline.instances import prod_stage_disabled + from pipeline.network.release_policy import prod_stage_disabled monkeypatch.setenv("DISABLE_PROD_SETTING", value) assert prod_stage_disabled() is True @@ -62,7 +62,7 @@ def test_truthy_values_disable_prod(self, monkeypatch, value): @pytest.mark.unit @pytest.mark.parametrize("value", ["false", "FALSE", "0", "no", "", "anything-else"]) def test_other_values_keep_prod_enabled(self, monkeypatch, value): - from pipeline.instances import prod_stage_disabled + from pipeline.network.release_policy import prod_stage_disabled monkeypatch.setenv("DISABLE_PROD_SETTING", value) assert prod_stage_disabled() is False @@ -70,14 +70,14 @@ def test_other_values_keep_prod_enabled(self, monkeypatch, value): @pytest.mark.unit def test_defaults_to_enabled_when_unset(self, monkeypatch): """Absent flag must keep the full pipeline — never silently skip PROD.""" - from pipeline.instances import prod_stage_disabled + from pipeline.network.release_policy import prod_stage_disabled monkeypatch.delenv("DISABLE_PROD_SETTING", raising=False) assert prod_stage_disabled() is False @pytest.mark.unit def test_whitespace_is_tolerated(self, monkeypatch): - from pipeline.instances import prod_stage_disabled + from pipeline.network.release_policy import prod_stage_disabled monkeypatch.setenv("DISABLE_PROD_SETTING", " true ") assert prod_stage_disabled() is True @@ -86,7 +86,7 @@ def test_whitespace_is_tolerated(self, monkeypatch): class TestProdOnlyStages: @pytest.mark.unit def test_prod_only_stages_are_the_three_promotion_stages(self): - from pipeline.models import PIPELINE_STAGES, PROD_ONLY_STAGES + from pipeline.documents.models import PIPELINE_STAGES, PROD_ONLY_STAGES assert PROD_ONLY_STAGES == {"approval_for_prod", "ingesting_prod", "publishing_to_network"} # All three must still exist in the canonical order, or filtering is a no-op. @@ -95,7 +95,7 @@ def test_prod_only_stages_are_the_three_promotion_stages(self): @pytest.mark.unit def test_filtering_leaves_a_pipeline_ending_at_completed(self): - from pipeline.models import PIPELINE_STAGES, PROD_ONLY_STAGES + from pipeline.documents.models import PIPELINE_STAGES, PROD_ONLY_STAGES remaining = [s[0] for s in PIPELINE_STAGES if s[0] not in PROD_ONLY_STAGES] @@ -105,10 +105,62 @@ def test_filtering_leaves_a_pipeline_ending_at_completed(self): @pytest.mark.unit def test_publishing_to_network_is_prod_only(self): """It must never run without a successful PROD promotion (see ADR 0004).""" - from pipeline.models import PIPELINE_STAGES, PROD_ONLY_STAGES + from pipeline.documents.models import PIPELINE_STAGES, PROD_ONLY_STAGES assert "publishing_to_network" in PROD_ONLY_STAGES stage_ids = [s[0] for s in PIPELINE_STAGES] assert stage_ids.index("publishing_to_network") == stage_ids.index("ingesting_prod") + 1 assert stage_ids.index("publishing_to_network") == stage_ids.index("completed") - 1 + + +def test_managed_networks_keep_their_collections_separate(monkeypatch): + from pipeline.network.release_policy import managed_release_collections, managed_release_targets + + monkeypatch.setenv("ENABLE_PUBLISH_TO_NETWORK", "true") + monkeypatch.setenv("NETWORK_CONNECTIONS", "dev,prod,peer") + monkeypatch.setenv("NETWORK_CONNECTION_DEFAULT", "dev") + monkeypatch.delenv("AUTO_RELEASE_NETWORK_ID", raising=False) + for name in ("dev", "prod", "peer"): + prefix = f"NETWORK_CONNECTION_{name.upper()}_" + monkeypatch.setenv(prefix + "TYPE", "beckn") + monkeypatch.setenv(prefix + "ID", f"test.{name}") + monkeypatch.setenv(prefix + "SENDER_ID", f"provider.test.{name}") + monkeypatch.setenv(prefix + "DISCOVERY_URL", f"https://{name}.example.test") + monkeypatch.setenv("NETWORK_CONNECTION_DEV_RELEASE_COLLECTION", "dev-published") + monkeypatch.setenv("NETWORK_CONNECTION_PROD_RELEASE_COLLECTION", "prod-published") + + assert managed_release_targets("advisory") == {"test.dev": "dev-published", "test.prod": "prod-published"} + assert managed_release_collections() == {"dev-published": "test.dev", "prod-published": "test.prod"} + + monkeypatch.setenv("NETWORK_CONNECTION_PROD_SCHEME_RELEASE_COLLECTION", "dev-published") + with pytest.raises(ValueError, match="separate release collection"): + managed_release_targets("scheme") + + +def test_automatic_release_uses_collections_from_the_named_network(monkeypatch): + from pipeline.network.release_policy import managed_release_collections, upload_release_target + + for key, value in { + "ENABLE_PUBLISH_TO_NETWORK": "true", + "NETWORK_CONNECTIONS": "dev", + "NETWORK_CONNECTION_DEFAULT": "dev", + "NETWORK_CONNECTION_DEV_TYPE": "beckn", + "NETWORK_CONNECTION_DEV_ID": "test.dev", + "NETWORK_CONNECTION_DEV_SENDER_ID": "provider.test.dev", + "NETWORK_CONNECTION_DEV_DISCOVERY_URL": "https://dev.example.test", + "NETWORK_CONNECTION_DEV_RELEASE_COLLECTION": "dev-advisories", + "NETWORK_CONNECTION_DEV_SCHEME_RELEASE_COLLECTION": "dev-schemes", + "AUTO_RELEASE_NETWORK_ID": "test.dev", + }.items(): + monkeypatch.setenv(key, value) + monkeypatch.delenv("AUTO_RELEASE_COLLECTION", raising=False) + monkeypatch.delenv("AUTO_RELEASE_SCHEME_COLLECTION", raising=False) + + assert upload_release_target("advisory") == ("test.dev", "dev-advisories") + assert upload_release_target("scheme") == ("test.dev", "dev-schemes") + assert managed_release_collections() == {"dev-advisories": "test.dev", "dev-schemes": "test.dev"} + + monkeypatch.setenv("AUTO_RELEASE_SCHEME_COLLECTION", "mismatched") + with pytest.raises(ValueError, match="automatic and managed release collections must agree"): + managed_release_collections() diff --git a/tests/test_jsonata_intake.py b/tests/test_jsonata_intake.py new file mode 100644 index 0000000..4580853 --- /dev/null +++ b/tests/test_jsonata_intake.py @@ -0,0 +1,27 @@ +import pytest + +from pipeline.intake.webhook_mapping import JsonataMapping, map_source_submission + + +def test_jsonata_maps_and_filters_nested_advisories(): + mapping = JsonataMapping( + runtime="jsonata-v1", + expression='items[published=true].{"external_id": $string(id), "title": headline, "text": $trim(body)}', + ) + rows = map_source_submission( + {"items": [ + {"id": 7, "published": True, "headline": "Advisory", "body": " Water before noon. "}, + {"id": 8, "published": False, "headline": "Draft", "body": "Do not serve."}, + ]}, + mapping, + ) + evidence, document = rows[0] + assert len(rows) == 1 + assert document.external_id == "7" + assert document.text == "Water before noon." + assert len(evidence["source_payload_sha256"]) == 64 + + +def test_jsonata_canonical_validation_rejects_missing_text(): + with pytest.raises(ValueError): + map_source_submission({}, JsonataMapping(runtime="jsonata-v1", expression='{"external_id": "x"}')) diff --git a/tests/test_knowledge_metadata.py b/tests/test_knowledge_metadata.py new file mode 100644 index 0000000..44573b8 --- /dev/null +++ b/tests/test_knowledge_metadata.py @@ -0,0 +1,87 @@ +import json + +import pytest +from qdrant_client import QdrantClient + +from pipeline.search.passage_projection import _prepare_records +from pipeline.metadata.knowledge_metadata import KnowledgeFilters, parse_knowledge_metadata +from pipeline.vector_store import qdrant_store + + +def test_metadata_vocabulary_accepts_labels_from_another_domain(): + fields = { + "subject_categories": ["Incident response", "incident response"], + "coverage_areas": [{"codeScheme": "Facility", "areaCode": "B-17", "areaLevel": "Floor"}], + } + + assert parse_knowledge_metadata(json.dumps(fields)) == { + **fields, "subject_categories": ["Incident response"], + } + assert KnowledgeFilters.model_validate(fields).qdrant_values()["coverage_area_keys"] == ["facility:b-17"] + + +def test_metadata_retains_typed_applicability_and_bounds_free_tags(): + metadata = parse_knowledge_metadata(json.dumps({ + "subject_categories": ["Livestock"], + "agriculture_subjects": [{"subjectId": "urn:oan:animal:cattle"}], + "coverage_areas": [{"codeScheme": "ISO-3166-2", "areaCode": "IN-GJ"}], + "seasons": ["Kharif"], "tags": ["Water", "Heat", "water"], + "issued_at": "2026-01-10T08:00:00+05:30", + })) + assert metadata["tags"] == ["Water", "Heat"] + assert metadata["issued_at"] == "2026-01-10T08:00:00+05:30" + with pytest.raises(ValueError): + parse_knowledge_metadata(json.dumps({"tags": [str(i) for i in range(17)]})) + + +def test_vector_search_matches_subject_and_applicability_before_ranking(monkeypatch): + monkeypatch.setenv("EMBEDDING_VECTOR_SIZE", "2") + monkeypatch.setattr(qdrant_store, "embed_passages", lambda texts, **kwargs: [[1.0, 0.0] for _ in texts]) + monkeypatch.setattr(qdrant_store, "embed_query", lambda *args, **kwargs: [1.0, 0.0]) + store = qdrant_store.QdrantVectorStore(client=QdrantClient(":memory:")) + text = "Provide clean drinking water and monitor cattle daily." + records = [] + for code, season in [("IN-GJ", "Kharif"), ("IN-MH", "Kharif"), ("IN-GJ", "Rabi"), (None, None), (None, "Broad")]: + metadata = { + "subject_categories": ["Livestock"], "agriculture_subjects": [{"subjectId": "urn:oan:animal:cattle"}], + "coverage_areas": [{"codeScheme": "ISO-3166-2", "areaCode": code}] if code else [], + "seasons": [season] if season and season != "Broad" else [], + "broad_applicability": season == "Broad", + } + records.extend(_prepare_records( + f"{code}:{season}", "care.pdf", [{"chunk_number": 1, "original_text": text}], + kind="advisory", knowledge_metadata=metadata, + )) + store.upsert("applicability", records) + filters = KnowledgeFilters.model_validate({ + "subject_categories": ["Livestock"], "agriculture_subject_ids": ["urn:oan:animal:cattle"], + "coverage_areas": [{"codeScheme": "iso-3166-2", "areaCode": "in-gj"}], "seasons": ["KHARIF"], + }) + hits = store.search("applicability", "cattle", applicability=filters.qdrant_values())["hits"] + assert {hit["doc_id"] for hit in hits} == {"IN-GJ:Kharif", "None:Broad"} + + +def test_grouped_applicability_keeps_region_season_pairs_together(monkeypatch): + monkeypatch.setenv("EMBEDDING_VECTOR_SIZE", "2") + monkeypatch.setattr(qdrant_store, "embed_passages", lambda texts, **kwargs: [[1.0, 0.0] for _ in texts]) + monkeypatch.setattr(qdrant_store, "embed_query", lambda *args, **kwargs: [1.0, 0.0]) + store = qdrant_store.QdrantVectorStore(client=QdrantClient(":memory:")) + + def area(code): + return {"codeScheme": "district", "areaCode": code} + + metadata = { + "applicability_groups": [ + {"coverage_areas": [area("north")], "seasons": ["dry"]}, + {"coverage_areas": [area("south")], "seasons": ["wet"]}, + ], + } + records = _prepare_records( + "grouped", "advice.pdf", [{"chunk_number": 1, "original_text": "Water guidance"}], + knowledge_metadata=metadata, + ) + store.upsert("knowledge", records) + for season, expected in [("dry", {"grouped"}), ("wet", set())]: + filters = KnowledgeFilters(coverage_areas=[area("north")], seasons=[season]) + hits = store.search("knowledge", "water", applicability=filters.qdrant_values())["hits"] + assert {hit["doc_id"] for hit in hits} == expected diff --git a/tests/test_liteparse.py b/tests/test_liteparse.py new file mode 100644 index 0000000..9cf651b --- /dev/null +++ b/tests/test_liteparse.py @@ -0,0 +1,128 @@ +"""Exercise the LiteParse CLI boundary without installing the optional binary.""" + +import json +import subprocess +from pathlib import Path + +import pytest + +from pipeline.documents.models import ( + PAGE_LAYOUT_ITEM_MAX_CHARS, + PAGE_LAYOUT_MAX_ITEMS, + PageMetadata, +) +from pipeline.ocr import service +from pipeline.ocr import liteparse +from pipeline.ocr.liteparse import LiteParseProvider + + +def _page(number, text): + return { + "page": number, + "width": 595, + "height": 842, + "text": text, + "text_items": [{"text": text, "x": 65, "y": 110, "width": 155, "height": 24}], + "complexity": {"needs_ocr": True, "full_page_image": True}, + } + + +def _mock_parse(monkeypatch, page): + def run(command, **kwargs): + assert command[:4] == ["lit", "parse", "--format", "json"] + assert "--no-ocr" in command + assert command[-2] == "--" + assert Path(command[-1]).is_absolute() + assert set(kwargs["env"]) == {"PATH"} + assert kwargs["check"] and kwargs["timeout"] > 0 + output_path = Path(command[command.index("--output") + 1]) + output_path.write_text(json.dumps({"pages": [page]}), encoding="utf-8") + + monkeypatch.setattr(subprocess, "run", run) + + +@pytest.mark.unit +def test_explicit_selection_extracts_embedded_text_and_validates_layout(monkeypatch, temp_pdf_file): + text = "કપાસના ખેતરની સંભાળ" + _mock_parse(monkeypatch, _page(1, text)) + monkeypatch.setenv("OCR_PROVIDER", "pypdf") + monkeypatch.setenv("OCR_SKIP_TEXT_LAYER", "true") + monkeypatch.setattr(service, "split_pdf_range", lambda *a, **k: pytest.fail("text shortcut ran")) + + page, = service.ocr_pdf(str(temp_pdf_file), str.strip, provider="liteparse") + + assert page["original_markdown"] == text + layout = PageMetadata.model_validate(page["metadata"]).source_layout + assert layout.parser == "liteparse" + assert layout.text_items[0].text == text + assert layout.text_items[0].x == 65 + assert layout.ocr_recommended is True + + +@pytest.mark.unit +def test_source_layout_limit_preserves_untruncated_page_text(monkeypatch, temp_pdf_file): + text = "x" * (PAGE_LAYOUT_ITEM_MAX_CHARS + 1) + raw = _page(2, text) + raw["text_items"] *= PAGE_LAYOUT_MAX_ITEMS + 1 + _mock_parse(monkeypatch, raw) + parser = LiteParseProvider(service.load_ocr_config("liteparse")) + + page, = parser.process_pdf_range(str(temp_pdf_file), 1, 2) + + assert page["page_number"] == 2 + assert page["original_markdown"] == text + layout = page["metadata"]["source_layout"] + assert layout["truncated"] is True + assert len(layout["text_items"]) == PAGE_LAYOUT_MAX_ITEMS + assert len(layout["text_items"][0]["text"]) == PAGE_LAYOUT_ITEM_MAX_CHARS + + +@pytest.mark.unit +@pytest.mark.parametrize(("raw", "error"), [ + (_page(2, "unexpected page"), "invalid page data"), + (_page(1, ""), "Select an OCR provider"), +]) +def test_liteparse_rejects_missing_or_unusable_page(monkeypatch, temp_pdf_file, raw, error): + _mock_parse(monkeypatch, raw) + parser = LiteParseProvider(service.load_ocr_config("liteparse")) + + with pytest.raises(RuntimeError, match=error): + parser.process_pdf_range(str(temp_pdf_file), 0, 1) + + +@pytest.mark.unit +def test_liteparse_timeout_returns_safe_error(monkeypatch, temp_pdf_file): + def timeout(command, **kwargs): + raise subprocess.TimeoutExpired(command, timeout=kwargs["timeout"]) + + monkeypatch.setattr(subprocess, "run", timeout) + parser = LiteParseProvider(service.load_ocr_config("liteparse")) + + with pytest.raises(RuntimeError, match="configured parser timeout"): + parser.process_pdf_range(str(temp_pdf_file), 0, 1) + + +@pytest.mark.unit +def test_liteparse_bounds_decoded_output(monkeypatch, temp_pdf_file): + _mock_parse(monkeypatch, _page(1, "Source text")) + monkeypatch.setattr(liteparse, "_MAX_OUTPUT_BYTES", 10) + parser = LiteParseProvider(service.load_ocr_config("liteparse")) + + with pytest.raises(RuntimeError, match="output exceeded the size limit"): + parser.process_pdf_range(str(temp_pdf_file), 0, 1) + + +@pytest.mark.db +def test_liteparse_layout_survives_page_round_trip(monkeypatch, temp_pdf_file, db_connection, sample_document): + _mock_parse(monkeypatch, _page(1, "Source text")) + page, = service.ocr_pdf(str(temp_pdf_file), str.strip, provider="liteparse") + workflow_id = sample_document["workflow_id"] + + db_connection.save_pages(workflow_id, [page]) + persisted = db_connection.get_page(workflow_id, 1) + persisted["translations"]["en"] = {"translated_markdown": "Translated text"} + db_connection.save_pages(workflow_id, [persisted]) + + updated = db_connection.get_page(workflow_id, 1) + assert updated["metadata"]["source_layout"] == page["metadata"]["source_layout"] + assert updated["translations"]["en"]["translated_markdown"] == "Translated text" diff --git a/tests/test_localization.py b/tests/test_localization.py new file mode 100644 index 0000000..98e4a3d --- /dev/null +++ b/tests/test_localization.py @@ -0,0 +1,124 @@ +import pytest +from fastapi import HTTPException +from qdrant_client import QdrantClient + +from pipeline import db +from pipeline.api.routers.search import run_search +from pipeline.auth.models import AuthUser +from pipeline.metadata.localization import ( + localize_search_hit, + normalize_language_tag, + project_chunk_localization, +) +from pipeline.vector_store import qdrant_store + +SOURCE = "કપાસને યોગ્ય સમયે પાણી આપો." +ENGLISH = "Water cotton at the right time." +HINDI = "कपास को सही समय पर पानी दें।" + + +def _page(): + return { + "page_number": 1, "detected_language": "gu", "original_markdown": f"

{SOURCE}

", + "translations": { + "en": {"translated_markdown": ENGLISH, "is_reviewed": True}, + "hi": {"translated_markdown": HINDI, "is_reviewed": False}, + }, + } + + +def _chunk(text): + return { + "original_text": text, "chunking_provider": "recursive_splitter", + "source_spans_json": [{"page_number": 1, "start_char": 0, "end_char": len(text)}], + } + + +def test_full_page_translation_keeps_source_and_coordinate_basis(): + projected = project_chunk_localization(_chunk(SOURCE), [_page()], source_first=True) + + assert projected["primary_language"] == projected["text_language"] == "gu" + assert projected["primary_text"] == SOURCE + assert projected["translations"] == { + "en": {"text": ENGLISH, "is_reviewed": True}, + "hi": {"text": HINDI, "is_reviewed": False}, + } + assert projected["source_evidence"] == [{ + "page_number": 1, "source_language": "gu", "translation_languages": ["en", "hi"], + "span_basis": "normalized_source_text", "spans": [{"start_char": 0, "end_char": len(SOURCE)}], + }] + + +def test_partial_page_passage_does_not_claim_whole_page_translation(): + source = "કપાસને યોગ્ય સમયે" + projected = project_chunk_localization(_chunk(source), [_page()], source_first=True) + hit = {"text": source, **projected} + + assert projected["translations"] == {} + assert projected["source_evidence"][0]["translation_languages"] == ["en", "hi"] + assert localize_search_hit(hit, "en", strict=True) is None + fallback = localize_search_hit(hit, "en") + assert fallback["text"] == source + assert fallback["text_language"] == "gu" + assert fallback["language_fallback"] is True + + +def test_language_tag_handles_script_region_extensions_and_rejects_ranges(): + assert normalize_language_tag("sr-Latn-RS-u-nu-latn") == "sr-latn-rs-u-nu-latn" + assert normalize_language_tag("fr-CA,en") is None + assert normalize_language_tag("*") is None + assert normalize_language_tag("hi_IN") is None + + +@pytest.mark.asyncio +async def test_search_localizes_one_vector_and_filters_exact_language_before_ranking(monkeypatch): + indexed_inputs = [] + + def embed(passages, **kwargs): + indexed_inputs.extend(passages) + return [[1.0, 0.0] if "English only" in passage else [0.5, 0.5] for passage in passages] + + monkeypatch.setenv("EMBEDDING_VECTOR_SIZE", "2") + monkeypatch.setattr(qdrant_store, "embed_passages", embed) + monkeypatch.setattr(qdrant_store, "embed_query", lambda *_, **__: [1.0, 0.0]) + monkeypatch.setattr(db, "get_search_settings", dict) + store = qdrant_store.QdrantVectorStore(client=QdrantClient(":memory:")) + monkeypatch.setattr("pipeline.vector_store.get_vector_store", lambda: store) + source = project_chunk_localization(_chunk(SOURCE), [_page()], source_first=True) + store.upsert("language-test", [ + {"doc_id": "other", "text": "English only", "is_reference": False, "available_languages": ["en"]}, + { + "doc_id": "advisory", "text": SOURCE, "is_reference": False, **source, + "name": "મૂળ શીર્ષક", "description": "મૂળ વર્ણન", + "knowledge_metadata": {"localized": {"hi": {"title": "स्थानीय शीर्षक", "description": "स्थानीय विवरण"}}}, + }, + ]) + assert indexed_inputs == ["English only", SOURCE] + point, = store.list_by_doc_id("language-test", "advisory") + assert point["translations"]["hi"]["text"] == HINDI + assert point["primary_text"] == SOURCE + assert point["name"] == "મૂળ શીર્ષક" and "name_en" not in point + + user = AuthUser(user_id="test-user", roles=["superadmin"]) + search = {"query": "cotton", "index_name": "language-test", "top_k": 1, "candidate_cap": 1} + strict_hit, = (await run_search({**search, "language": "hi", "strict_language": True}, user))["hits"] + assert strict_hit["text"] == HINDI + assert strict_hit["text_language"] == "hi" + assert strict_hit["primary_language"] == "gu" + assert strict_hit["language_match"] == "exact" + assert strict_hit["language_fallback"] is False + assert strict_hit["text_is_reviewed"] is False + assert strict_hit["name"] == "स्थानीय शीर्षक" + assert strict_hit["description"] == "स्थानीय विवरण" + assert strict_hit["name_en"] == "મૂળ શીર્ષક" + + lookup = await run_search({**search, "candidate_cap": 2, "top_k": 2, "language": "en-IN"}, user) + english = next(hit for hit in lookup["hits"] if hit["doc_id"] == "advisory") + assert english["text"] == ENGLISH + assert english["text_language"] == "en" + assert english["language_match"] == "lookup" + assert english["language_fallback"] is True + + with pytest.raises(HTTPException) as error: + await run_search({**search, "language": "en;q=1"}, user) + assert error.value.status_code == 422 diff --git a/tests/test_migrate_sqlite_to_postgres.py b/tests/test_migrate_sqlite_to_postgres.py index 5af95ae..7d0737d 100644 --- a/tests/test_migrate_sqlite_to_postgres.py +++ b/tests/test_migrate_sqlite_to_postgres.py @@ -1,18 +1,17 @@ -"""scripts/migrate_sqlite_to_postgres.py: a legacy SQLite file lands in Postgres. +"""A legacy SQLite file lands in Postgres using the workspace migration kit. The fixture file is built with the legacy schema DDL, the way a production database looked at cutover. """ import hashlib -import importlib.util import json import sqlite3 from pathlib import Path import pytest -_SCRIPT = Path(__file__).resolve().parent.parent / "scripts" / "migrate_sqlite_to_postgres.py" +from tests.migration_kit import load_migration_tool LEGACY_DDL = """ CREATE TABLE documents ( @@ -73,7 +72,7 @@ ); CREATE TABLE document_manifest_entries ( id INTEGER PRIMARY KEY AUTOINCREMENT, manifest_filename TEXT NOT NULL UNIQUE, title_en TEXT, - imported_at TEXT NOT NULL + title_gu TEXT, description TEXT, category_tags TEXT, imported_at TEXT NOT NULL ); CREATE TABLE email_otps ( email TEXT PRIMARY KEY, code_hash TEXT NOT NULL, expires_at TEXT NOT NULL, @@ -106,10 +105,7 @@ def _load_script(): - spec = importlib.util.spec_from_file_location("migrate_sqlite_to_postgres", _SCRIPT) - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module + return load_migration_tool("migrate_sqlite_to_postgres") def _insert(conn, table, **values): @@ -243,6 +239,9 @@ def legacy_db(tmp_path): scheme_aliases_json='["registry"]', status="live", source="pipeline") _insert(conn, "document_manifest_entries", manifest_filename="a.csv", imported_at="2026-01-01T00:00:00") _insert(conn, "document_manifest_entries", manifest_filename="b.csv", imported_at="2026-01-01T00:00:00") + _insert(conn, "document_manifest_entries", manifest_filename="GUIDELINE.PDF", title_en="Imported title", + title_gu="અનુવાદ", description="Source description", category_tags="notice, public", + imported_at="2026-01-01T00:00:00") conn.commit() conn.close() return str(path) @@ -250,25 +249,56 @@ def legacy_db(tmp_path): def _counts(db): with db.get_connection() as conn: - return { + counts = { table: conn.execute(f"SELECT count(*) AS n FROM {table}").fetchone()["n"] for table in ( "documents", "workflows", "workflow_artifacts", "pages", "chunks", "audit_logs", - "users", "instances", "catalogs", "email_otps", "webhook_records", + "users", "instances", "catalogs", ) } + if conn.execute("SELECT to_regclass('public.legacy_import_evidence') IS NOT NULL AS exists").fetchone()["exists"]: + counts["legacy_import_evidence"] = conn.execute("SELECT count(*) AS n FROM legacy_import_evidence").fetchone()["n"] + return counts @pytest.mark.db @pytest.mark.unit class TestSqliteToPostgresMigration: + @pytest.fixture(autouse=True) + def _remove_test_import_archive(self, db_connection): + yield + with db_connection.get_connection() as conn: + conn.execute("DROP TABLE IF EXISTS public.legacy_import_evidence") + + def test_explicit_manifest_association_takes_precedence_over_filename(self, db_connection, postgres_url, legacy_db): + with sqlite3.connect(legacy_db) as conn: + conn.execute("UPDATE documents SET source_manifest_name = ? WHERE workflow_id = ?", ("a.csv", WF1)) + conn.execute("UPDATE document_manifest_entries SET title_en = ? WHERE manifest_filename = ?", ("Linked title", "a.csv")) + + report = _load_script().run_migration(legacy_db, postgres_url) + + assert report.tables["document_manifest_entries"] == {"read": 3, "inserted": 1, "dropped": 2} + assert db_connection.get_document(WF1)["knowledge_metadata"] == {"localized": {"en": {"title": "Linked title"}}} + + def test_ambiguous_manifest_is_not_shared_across_providers(self, db_connection, postgres_url, legacy_db): + with sqlite3.connect(legacy_db) as conn: + conn.execute("UPDATE documents SET source_filename = ? WHERE workflow_id = ?", ("guideline.pdf", WF2)) + + report = _load_script().run_migration(legacy_db, postgres_url) + + assert report.tables["document_manifest_entries"] == {"read": 3, "dropped": 2, "skipped": 1} + assert report.mismatches() == [] + assert not db_connection.get_document(WF1).get("knowledge_metadata") + assert not db_connection.get_document(WF2).get("knowledge_metadata") + def test_dry_run_writes_nothing_but_reports_everything(self, db_connection, postgres_url, legacy_db): script = _load_script() before = _counts(db_connection) + assert "legacy_import_evidence" not in before report = script.run_migration(legacy_db, postgres_url, dry_run=True) assert _counts(db_connection) == before - assert report.tables["documents"]["inserted"] == 3 - assert report.tables["documents"]["skipped"] == 2 + assert report.tables["documents"]["inserted"] == 4 + assert report.tables["documents"]["skipped"] == 1 assert report.mismatches() == [] def test_a_full_run_maps_every_table(self, db_connection, postgres_url, legacy_db): @@ -277,26 +307,29 @@ def test_a_full_run_maps_every_table(self, db_connection, postgres_url, legacy_d assert report.mismatches() == [] tables = report.as_dict()["tables"] - assert tables["documents"] == {"read": 5, "inserted": 3, "skipped": 2} + assert tables["documents"] == {"read": 5, "inserted": 4, "skipped": 1} assert tables["document_jobs"] == {"read": 3, "inserted": 2, "skipped": 1} assert tables["pages"] == {"read": 4, "inserted": 2, "skipped": 2} assert tables["chunks"]["inserted"] == 2 - assert tables["document_manifest_entries"] == {"read": 2, "dropped": 2} - assert tables["email_otps"] == {"read": 2, "inserted": 1, "dropped": 1} + assert tables["document_manifest_entries"] == {"read": 3, "dropped": 2, "inserted": 1} + assert tables["email_otps"] == {"read": 2, "dropped": 2} assert tables["settings"]["skipped"] == 2 assert tables["scheme_catalog_entries"] == {"read": 2, "inserted": 1, "dropped": 1} skipped = {i["key"]: i["reason"] for i in report.skip_reasons if i["table"] == "documents"} assert skipped == { - WF4: "document_kind 'document' has no catalog", WF5: "demo document (is_demo)", } - assert db.get_document(WF4) is None and db.get_document(WF5) is None + assert db.get_document(WF4)["kind"] == "document" and db.get_document(WF5) is None doc = db.get_document(WF1) assert doc["stage"] == "completed" assert doc["instance"] == "mh" assert doc["document_id"] == FP1 assert doc["name"] == "Registry name" # one name per document in v2 + assert doc["knowledge_metadata"] == { + "description": "Source description", "tags": ["notice", "public"], + "localized": {"en": {"title": "Imported title"}, "gu": {"title": "અનુવાદ"}}, + } assert (doc["valid_from"], doc["valid_to"]) == ("2026-03-01", "2027-03-01") assert (doc["code"], doc["kind"]) == ("pm-kmy", "scheme") assert doc["network_visible"] is False and doc["reindex_required"] is True @@ -384,12 +417,13 @@ def test_a_full_run_maps_every_table(self, db_connection, postgres_url, legacy_d assert settings["showHighlights"] is False and settings["searchMethod"] == "LEXICAL" assert settings["efSearch"] == 256 - assert db.get_email_otp("live@x.in")["attempts"] == 2 - assert db.get_email_otp("old@x.in") is None with db.get_connection() as conn: - hook = conn.execute("SELECT * FROM webhook_records").fetchone() - assert str(hook["id"]) == "3c8a2f0e-6b1d-4f6e-9d0a-5f8f0b2e7a11" - assert hook["is_published"] is True and hook["payload"] == {"contentID": "c-1"} + hook = conn.execute( + "SELECT source_key, payload FROM legacy_import_evidence WHERE source_table = 'webhook_records'" + ).fetchone() + assert hook["source_key"] == "3c8a2f0e-6b1d-4f6e-9d0a-5f8f0b2e7a11" + assert hook["payload"]["is_published"] == 1 + assert json.loads(hook["payload"]["payload"]) == {"contentID": "c-1"} def test_in_flight_documents_keep_their_stage_and_prod_request(self, db_connection, postgres_url, legacy_db): db = db_connection @@ -429,7 +463,7 @@ def test_running_it_twice_changes_nothing(self, db_connection, postgres_url, leg report = script.run_migration(legacy_db, postgres_url) assert _counts(db_connection) == first - assert report.tables["documents"] == {"read": 5, "existing": 3, "skipped": 2} + assert report.tables["documents"] == {"read": 5, "existing": 4, "skipped": 1} assert report.tables["audit_logs"]["inserted"] == 0 assert report.tables["webhook_records"]["existing"] == 1 assert report.mismatches() == [] diff --git a/tests/test_network_adapter_extensions.py b/tests/test_network_adapter_extensions.py new file mode 100644 index 0000000..5e0f71c --- /dev/null +++ b/tests/test_network_adapter_extensions.py @@ -0,0 +1,300 @@ +import importlib +import json +from uuid import uuid4 + +import pytest +from qdrant_client import QdrantClient +from temporalio.exceptions import ApplicationError + +from pipeline.temporal.activities import indexing, publication +from pipeline.intake import scheduled_intake +from pipeline.intake.source_fetch import SourceFile +from pipeline.network.release_policy import upload_release_target +from pipeline.network.network_publisher import PublishConfigError, validate_network_adapters +from pipeline.vector_store import qdrant_store + + +@pytest.fixture +def installed_adapter(tmp_path, monkeypatch): + package = tmp_path / "kp_example_adapter.py" + package.write_text(''' +from pipeline.network.adapters.contracts import PublicationReceipt + +attempts = [] +evidence = "a" * 64 +matched = True +status = "ACCEPTED" +rejection = None +has_exchange = True + +class NoticeAdapter: + def __init__(self, connection): + self.connection = connection + + def validate(self): + assert self.connection.options == {"channel": "local-notices"} + + def supports_kind(self, kind): + return kind == "notice" + + def provider_id(self, kind, provider_code): + return f"account:{provider_code}" if self.supports_kind(kind) else None + + def publication_evidence(self, conn, request): + assert request.revision["metadata"]["kind"] == request.kind == "notice" + assert request.provider_id == "account:example" + return evidence + + def publish_revision(self, request): + attempts.append(request) + return PublicationReceipt( + exchange={"protocol": "notice-test"} if has_exchange else None, message_id="notice-receipt", + result_status=status, context_matches=matched, rejection=rejection, + ) +''') + distribution = tmp_path / "kp_example_adapter-1.0.dist-info" + distribution.mkdir() + (distribution / "entry_points.txt").write_text( + "[knowledge_provider.network_adapters]\nnotice-test = kp_example_adapter:NoticeAdapter\n", + ) + monkeypatch.syspath_prepend(str(tmp_path)) + for key, value in { + "DOCUMENT_KINDS": "notice,advisory", + "ENABLE_PUBLISH_TO_NETWORK": "true", + "NETWORK_CONNECTIONS": "external", + "NETWORK_CONNECTION_DEFAULT": "external", + "NETWORK_CONNECTION_EXTERNAL_TYPE": "notice-test", + "NETWORK_CONNECTION_EXTERNAL_ID": "external.test", + "NETWORK_CONNECTION_EXTERNAL_RELEASE_COLLECTION": "published-notices", + "NETWORK_CONNECTION_EXTERNAL_OPTION_CHANNEL": "local-notices", + "AUTO_RELEASE_NETWORK_ID": "external.test", + "AUTO_RELEASE_COLLECTION": "published-notices", + "BECKN_CATALOG_PROFILE_PATH": str(tmp_path / "unavailable-beckn-profile.json"), + "PROVIDER_CONFIG_OVERRIDE_PATH": str(tmp_path / "unavailable-beckn-names.json"), + }.items(): + monkeypatch.setenv(key, value) + module = importlib.import_module("kp_example_adapter") + module.attempts.clear() + module.matched = True + module.evidence = "a" * 64 + module.status = "ACCEPTED" + module.rejection = None + module.has_exchange = True + return module + + +def test_startup_rejects_an_ambiguous_installed_adapter(installed_adapter, monkeypatch): + monkeypatch.setenv("NETWORK_CONNECTION_EXTERNAL_TYPE", "duplicate-test") + monkeypatch.setattr("pipeline.network.network_publisher.entry_points", lambda **_: (object(), object())) + with pytest.raises(PublishConfigError, match="Install exactly one matching network adapter"): + validate_network_adapters() + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_scheduled_automatic_release_requires_a_supported_destination( + db_connection, installed_adapter, monkeypatch, mock_object_store, +): + profile = scheduled_intake.PullProfile( + provider="example", source_key="notices", host="notices.example.test", + url="https://notices.example.test/feed.json", auto_release=True, + ) + db_connection.get_or_create_instance(profile.provider) + record = {"external_id": "notice-1", "title": "Notice", "text": "Inspection notice", "kind": "advisory"} + + async def fetch(*_): + return SourceFile("feed.json", "application/json", json.dumps(record).encode(), profile.url) + + monkeypatch.setattr(scheduled_intake, "fetch_source", fetch) + monkeypatch.setattr(scheduled_intake, "open_object_store", lambda _: mock_object_store) + assert not await scheduled_intake.poll_source(profile) + with db_connection.get_connection() as conn: + assert conn.execute("SELECT count(*) AS n FROM source_intake_receipts").fetchone()["n"] == 0 + assert conn.execute("SELECT failure_count FROM scheduled_source_pulls").fetchone()["failure_count"] == 1 + conn.execute("UPDATE scheduled_source_pulls SET next_run_at = now()") + + record["kind"] = "notice" + assert await scheduled_intake.poll_source(profile) + with db_connection.get_connection() as conn: + job = conn.execute( + "SELECT w.config, w.status FROM source_intake_receipts r JOIN workflows w ON w.id = r.job_id", + ).fetchone() + assert conn.execute("SELECT count(*) AS n FROM network_releases").fetchone()["n"] == 0 + assert job["status"] == "start_pending" + assert job["config"]["release_network_id"] == "external.test" + assert job["config"]["release_collection"] == "published-notices" + assert job["config"]["upload_start_args"][6] == "published-notices" + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_scheduled_source_selects_and_pins_a_nondefault_network( + db_connection, installed_adapter, monkeypatch, mock_object_store, +): + for key, value in { + "NETWORK_CONNECTIONS": "external,secondary", + "NETWORK_CONNECTION_SECONDARY_TYPE": "notice-test", + "NETWORK_CONNECTION_SECONDARY_ID": "secondary.test", + "NETWORK_CONNECTION_SECONDARY_RELEASE_COLLECTION": "second-notices", + "NETWORK_CONNECTION_SECONDARY_OPTION_CHANNEL": "local-notices", + }.items(): + monkeypatch.setenv(key, value) + profile = scheduled_intake.PullProfile( + provider="example", source_key="notices", host="notices.example.test", + url="https://notices.example.test/feed.json", auto_release=True, release_network_id="unknown.test", + ) + db_connection.get_or_create_instance(profile.provider) + record = {"external_id": "notice-1", "title": "Notice", "text": "Inspection notice", "kind": "notice"} + records = [record] + + async def fetch(*_): + return SourceFile("feed.json", "application/json", json.dumps(records).encode(), profile.url) + + monkeypatch.setattr(scheduled_intake, "fetch_source", fetch) + monkeypatch.setattr(scheduled_intake, "open_object_store", lambda _: mock_object_store) + assert not await scheduled_intake.poll_source(profile) + with db_connection.get_connection() as conn: + assert conn.execute("SELECT count(*) AS n FROM source_intake_receipts").fetchone()["n"] == 0 + + profile = profile.model_copy(update={"release_network_id": "secondary.test"}) + assert await scheduled_intake.poll_source(profile) + with db_connection.get_connection() as conn: + receipt = conn.execute("SELECT document_id::text, job_id::text FROM source_intake_receipts").fetchone() + job = db_connection.get_document_job(receipt["job_id"]) + document = db_connection.get_document(receipt["document_id"]) + workflow_id = job["temporal_workflow_id"] + assert job["config"]["explicit_release_network"] + assert job["config"]["upload_start_args"][6] == "second-notices" + assert upload_release_target("notice") == ("external.test", "published-notices") + assert indexing._release_target_for_job(document, job, workflow_id, "secondary.test") == ( + "secondary.test", "second-notices", + ) + assert indexing._release_target_for_job(document, job, workflow_id, "external.test") is None + assert indexing._release_target_for_job(document, job, "stale-workflow", "secondary.test") is None + with monkeypatch.context() as changed: + changed.setenv("NETWORK_CONNECTION_DEFAULT", "secondary") + changed.setenv("AUTO_RELEASE_NETWORK_ID", "secondary.test") + changed.setenv("AUTO_RELEASE_COLLECTION", "second-notices") + default_profile = profile.model_copy(update={"release_network_id": None}) + assert await scheduled_intake.poll_source(default_profile) + with db_connection.get_connection() as conn: + assert conn.execute("SELECT count(*) AS n FROM source_intake_receipts").fetchone()["n"] == 1 + records.append({**record, "external_id": "notice-2", "text": "New inspection notice"}) + retargeted = profile.model_copy(update={"release_network_id": "external.test"}) + assert await scheduled_intake.poll_source(retargeted) + with db_connection.get_connection() as conn: + receipts = conn.execute( + "SELECT external_id, document_id::text, job_id::text FROM source_intake_receipts ORDER BY external_id" + ).fetchall() + assert receipts[0] == {"external_id": "notice-1", **receipt} + assert len(receipts) == 2 + assert conn.execute( + "SELECT failure_count, last_page_policy_conflicts FROM scheduled_source_pulls " + "WHERE provider = %s AND source_key = %s AND profile_sha256 = %s", retargeted.identity, + ).fetchone() == {"failure_count": 0, "last_page_policy_conflicts": 1} + conn.execute("UPDATE scheduled_source_pulls SET next_run_at = now()") + assert db_connection.get_document_job(receipts[1]["job_id"])["config"]["release_network_id"] == "external.test" + assert db_connection.get_document_job(receipt["job_id"])["config"]["release_network_id"] == "secondary.test" + db_connection.update_document_job(receipt["job_id"], status="completed", current_stage="completed") + records[0] = {**record, "text": "Revised inspection notice"} + assert await scheduled_intake.poll_source(retargeted) + with db_connection.get_connection() as conn: + revisions = conn.execute( + "SELECT document_id::text, job_id::text FROM source_intake_receipts " + "WHERE external_id = 'notice-1' AND job_id != %s", (receipt["job_id"],), + ).fetchall() + assert len(revisions) == 1 and revisions[0]["document_id"] == receipt["document_id"] + assert db_connection.get_document_job(revisions[0]["job_id"])["config"]["release_network_id"] == "external.test" + monkeypatch.setenv("NETWORK_CONNECTION_SECONDARY_RELEASE_COLLECTION", "changed-notices") + assert indexing._release_target_for_job(document, job, workflow_id, "secondary.test") is None + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_installed_adapter_releases_and_searches_a_frozen_notice( + test_client, db_connection, installed_adapter, mock_temporal_client, monkeypatch, +): + validate_network_adapters() + assert upload_release_target("notice") == ("external.test", "published-notices") + assert upload_release_target("advisory") is None + db = db_connection + workflow_id = "external-notice" + document_id = db.upsert_document( + workflow_id=workflow_id, document_id="b" * 32, filename="notice.txt", filepath="blob://docs/notice.txt", + instance="example", kind="notice", code="notice-test", + ) + revision_id, transaction_id, point_id = (str(uuid4()) for _ in range(3)) + db.save_draft_revision_snapshot(document_id, workflow_id, revision_id, { + "metadata": {"kind": "notice"}, + "passages": [{"_id": point_id, "doc_id": revision_id, "text": "Inspection notice", "is_reference": False}], + }) + selected = test_client.put(f"/documents/{workflow_id}/releases/external.test", json={"revision_id": revision_id}) + assert selected.status_code == 202, selected.text + assert mock_temporal_client.start_workflow.call_args.kwargs["args"] == [ + document_id, revision_id, "external.test", None, + ] + + monkeypatch.setenv("EMBEDDING_VECTOR_SIZE", "2") + monkeypatch.setattr(qdrant_store, "embed_passages", lambda rows, **_: [[1.0, 0.0] for _ in rows]) + monkeypatch.setattr(qdrant_store, "embed_query", lambda *_, **__: [1.0, 0.0]) + store = qdrant_store.QdrantVectorStore(client=QdrantClient(":memory:")) + monkeypatch.setattr("pipeline.vector_store.get_vector_store", lambda: store) + artifacts = iter([OSError("receipt artifact unavailable"), None, None, None]) + + def save_artifact(*_): + error = next(artifacts) + if error: + raise error + return "blob://docs/receipt.json", 10, "application/json" + + monkeypatch.setattr(publication, "_upload_artifact", save_artifact) + with pytest.raises(OSError, match="receipt artifact unavailable"): + await publication.release_saved_revision(document_id, revision_id, "external.test", transaction_id, None) + point, = store.list_by_doc_id("published-notices", revision_id) + assert point["provider_id"] == "account:example" + assert db.visible_release_hits([point]) == [] + + await publication.release_saved_revision(document_id, revision_id, "external.test", transaction_id, None) + request, = installed_adapter.attempts + assert request.revision_id == revision_id + assert request.transaction_id == transaction_id + assert db.visible_release_hits([point]) == [point] + + search = test_client.post("/search", json={ + "query": "Inspection", "search_mode": "TENSOR", "index_name": "published-notices", + "provider_id": "account:example", + }) + assert search.status_code == 200, search.text + hit, = search.json()["hits"] + assert (hit["text"], hit["revision_id"], hit["release_network_id"]) == ( + "Inspection notice", revision_id, "external.test", + ) + collections = test_client.get("/search/collections") + assert collections.json() == [{"label": "Published documents", "name": "published-notices", "network_id": "external.test"}] + + installed_adapter.matched = False + generation = db.list_document_revisions(document_id)["releases"][0]["generation"] + with pytest.raises(ApplicationError, match="did not accept"): + await publication.release_saved_revision(document_id, revision_id, "external.test", str(uuid4()), generation) + assert db.list_document_revisions(document_id)["releases"][0]["generation"] == generation + detail = test_client.get(f"/documents/{workflow_id}").json() + assert detail["network_publish"]["status"] == "rejected" + + installed_adapter.matched = True + installed_adapter.status = "REJECTED" + installed_adapter.rejection = "The notice network refused publication" + with pytest.raises(ApplicationError, match="refused publication"): + await publication.release_saved_revision(document_id, revision_id, "external.test", str(uuid4()), generation) + detail = test_client.get(f"/documents/{workflow_id}").json() + assert detail["network_publish"]["reason"] == installed_adapter.rejection + + installed_adapter.status = "ACCEPTED" + installed_adapter.rejection = None + installed_adapter.has_exchange = False + invalid_transaction = str(uuid4()) + with pytest.raises(ApplicationError, match="without its publication exchange"): + await publication.release_saved_revision(document_id, revision_id, "external.test", invalid_transaction, generation) + with db.get_connection() as conn: + invalid = conn.execute("SELECT receipt FROM publication_deliveries WHERE transaction_id = %s", (invalid_transaction,)).fetchone() + assert invalid["receipt"] is None + assert db.list_document_revisions(document_id)["releases"][0]["generation"] == generation diff --git a/tests/test_network_constants.py b/tests/test_network_constants.py index d4603c0..23d715a 100644 --- a/tests/test_network_constants.py +++ b/tests/test_network_constants.py @@ -1,6 +1,6 @@ import pytest -from pipeline.network_constants import ( +from pipeline.network.network_constants import ( KIND_TO_CATALOG_TYPE, catalog_type_for_kind, kind_for_catalog_type, diff --git a/tests/test_network_group.py b/tests/test_network_group.py index 9c0d13e..55788b7 100644 --- a/tests/test_network_group.py +++ b/tests/test_network_group.py @@ -2,9 +2,7 @@ import pytest -from pipeline.instances import INSTANCE_NAMES -from pipeline.network_group import ( - DEFAULT_PROVIDER_CONFIG_OVERRIDE_PATH, +from pipeline.network.network_group import ( PROVIDER_CONFIG_OVERRIDE_PATH_ENV, NetworkProviderConfigError, load_provider_id_overrides, @@ -27,6 +25,7 @@ def _default_env(monkeypatch): PROVIDER_CONFIG_OVERRIDE_PATH_ENV, ): monkeypatch.delenv(name, raising=False) + monkeypatch.delenv("DISCOVERY_NETWORK_ID", raising=False) monkeypatch.setenv("NETWORK_SENDER_ID", "provider.oan.dev") @@ -86,10 +85,9 @@ def test_reads_the_configured_file(self, monkeypatch, tmp_path): assert load_provider_id_overrides() == {"ka": "ka.example.com"} @pytest.mark.unit - def test_unset_path_uses_the_bundled_empty_default(self, monkeypatch): + def test_unset_path_means_no_overrides(self, monkeypatch): monkeypatch.delenv(PROVIDER_CONFIG_OVERRIDE_PATH_ENV, raising=False) - assert DEFAULT_PROVIDER_CONFIG_OVERRIDE_PATH.is_file() assert load_provider_id_overrides() == {} @pytest.mark.unit @@ -119,8 +117,8 @@ class TestLoadProviderNaming: def test_defaults(self): naming = load_provider_naming() - assert naming.shared_instance == "bv" - assert naming.shared_name == "Bharat Vistaar" + assert naming.shared_instance == "default" + assert naming.shared_name == "Default" assert naming.network_id == "provider.oan.dev" assert naming.id_template == "{instance}.{network_id}" assert naming.group_catalog_name_template == "{group_name} {catalog_name}" @@ -211,8 +209,8 @@ def test_id_template_must_render_a_valid_id(self, monkeypatch, template): load_provider_naming() @pytest.mark.unit - def test_every_known_instance_is_checked_at_boot(self, monkeypatch): - monkeypatch.setitem(INSTANCE_NAMES, "ladakh_ut", "Ladakh") + def test_configured_portal_identity_is_checked_at_boot(self, monkeypatch): + monkeypatch.setenv("PORTAL_INSTANCE", "invalid_code") with pytest.raises(NetworkProviderConfigError, match="provider id"): load_provider_naming() @@ -241,7 +239,7 @@ def broken_map(self, monkeypatch, tmp_path): @pytest.mark.unit @pytest.mark.asyncio async def test_worker_does_not_start(self, broken_map, monkeypatch): - import pipeline.worker as worker + import pipeline.temporal.worker as worker async def never_connect(*args, **kwargs): raise AssertionError("worker connected to Temporal despite a bad provider map") @@ -269,7 +267,7 @@ async def never_connect(*args, **kwargs): class TestProviderFor: @pytest.fixture(autouse=True) def _naming(self): - from pipeline.network_provider import ProviderNaming + from pipeline.network.network_provider import ProviderNaming self.naming = ProviderNaming( shared_instance="bv", diff --git a/tests/test_network_provider.py b/tests/test_network_provider.py index 6dd762f..175b916 100644 --- a/tests/test_network_provider.py +++ b/tests/test_network_provider.py @@ -1,6 +1,6 @@ import pytest -from pipeline.network_provider import Group, InvalidProviderIdError, ProviderNaming +from pipeline.network.network_provider import Group, InvalidProviderIdError, ProviderNaming MAHARASHTRA = Group(code="mh", name="Maharashtra") diff --git a/tests/test_network_publish_outcome.py b/tests/test_network_publish_outcome.py index 98e6545..f340807 100644 --- a/tests/test_network_publish_outcome.py +++ b/tests/test_network_publish_outcome.py @@ -2,7 +2,7 @@ import pytest -from pipeline.api.api import _network_publish_outcome +from pipeline.api.routers.document_detail import _network_publish_outcome def artifact(artifact_type, metadata=None, created_at="2026-10-28T10:00:00"): diff --git a/tests/test_network_registry.py b/tests/test_network_registry.py new file mode 100644 index 0000000..5ca06e2 --- /dev/null +++ b/tests/test_network_registry.py @@ -0,0 +1,254 @@ +"""Verify Sunbird participant admission and route evidence against wire records.""" + +import json +from dataclasses import FrozenInstanceError, replace + +import httpx +import pytest + +from pipeline.network.network_connections import NetworkConnection +from pipeline.network.network_registry import ( + ApprovedRoute, + RegistryLookupError, + SunbirdRegistry, + registry_for_network, +) + + +@pytest.fixture +def approval(): + return ApprovedRoute( + network_id="example.net", + participant_id="provider.example.net", + capability="openagrinet:KnowledgeAdvisory", + action="select", + base_url="http://knowledge-provider-api:8001", + method="POST", + path="/search", + mapping_url="https://mapping.example.net/v1/advisory.select.yaml", + mapping_sha256="a" * 64, + mapping_runtime="jsonata@1.5.4", + auth_profile="provider-select", + retry_max=1, + ) + + +@pytest.fixture +def participant(approval): + return { + "osid": "participant-row", + "osUpdatedAt": "2026-11-12T10:00:00Z", + "participantId": approval.participant_id, + "name": "Knowledge Provider", + "type": "upstream", + "status": "active", + "baseUrl": approval.base_url, + } + + +@pytest.fixture +def binding(approval): + return { + "osid": "binding-row", + "osUpdatedAt": "2026-11-12T10:01:00Z", + "bindingKey": approval.binding_key, + "participantId": approval.participant_id, + "capabilityCode": approval.capability, + "status": "active", + "actions": [{ + "osid": "select-action-row", + "action": approval.action, + "method": approval.method, + "path": approval.path, + "mappings": approval.mapping_url, + "timeoutMs": approval.timeout_ms, + "retryMax": approval.retry_max, + "status": "active", + }], + } + + +@pytest.fixture +def registry(monkeypatch, approval, participant, binding): + replies = {"Participant": [participant], "ProviderSchema": [binding]} + calls = [] + + def search(request): + assert request.method == "POST" + assert request.url.host == "registry.example.net" + entity = request.url.path.split("/")[-2] + assert request.url.path == f"/api/v1/{entity}/search" + calls.append((entity, json.loads(request.content)["filters"])) + payload = replies[entity] + if isinstance(payload, httpx.Response): + return payload + if entity == "Participant": + return httpx.Response(200, json=payload) + return httpx.Response(200, json={"params": {"status": "SUCCESSFUL"}, "data": payload}) + + client = httpx.Client + monkeypatch.setattr(httpx, "Client", lambda **options: client(transport=httpx.MockTransport(search), **options)) + connection = NetworkConnection( + adapter="beckn", + network_id=approval.network_id, + discovery_endpoint="https://discovery.example.net", + sender_id=approval.participant_id, + receiver_id="network.example.net", + visible_to=(), + registry_url="https://registry.example.net/api/v1/", + ) + return SunbirdRegistry(connection), replies, calls + + +def test_observation_includes_inactive_records_and_uses_creation_revision(registry, approval, participant, binding): + client, _, _ = registry + participant["status"] = "inactive" + participant["osCreatedAt"] = participant.pop("osUpdatedAt") + binding["actions"][0]["status"] = "inactive" + + observed = client.observe_participant(approval.participant_id) + assert observed.status == "inactive" + assert observed.revision == participant["osCreatedAt"] + assert observed.record_id == participant["osid"] + route = client.observe_binding(approval.participant_id, approval.capability) + assert route.revision == binding["osUpdatedAt"] + assert route.actions[0].status == "inactive" + assert route.actions[0].mapping_url == approval.mapping_url + + +def test_observation_distinguishes_absence_from_ambiguous_registry_state(registry, approval, binding): + client, replies, _ = registry + replies["Participant"] = [] + assert client.observe_participant(approval.participant_id) is None + + binding["actions"].append(binding["actions"][0].copy()) + with pytest.raises(RegistryLookupError) as error: + client.observe_binding(approval.participant_id, approval.capability) + assert (error.value.entity, error.value.reason) == ("ActionBinding", "duplicate") + + +def test_approved_route_produces_immutable_network_scoped_evidence(registry, approval, binding): + client, _, calls = registry + evidence = client.ready_route(approval) + assert calls == [ + ("Participant", {"participantId": {"eq": approval.participant_id}}), + ("ProviderSchema", {"bindingKey": {"eq": approval.binding_key}}), + ] + assert evidence.admission.network_id == approval.network_id + assert evidence.admission.registry_url == "https://registry.example.net/api/v1" + assert evidence.admission.record_id == "participant-row" + assert evidence.binding_record_id == "binding-row" + assert evidence.action_record_id == "select-action-row" + assert evidence.approval.mapping_sha256 == approval.mapping_sha256 + with pytest.raises(FrozenInstanceError): + evidence.admission.base_url = "https://other.example.net" + + assert client.ready_route(approval).digest == evidence.digest + assert client.ready_route(replace(approval, mapping_sha256="b" * 64)).digest != evidence.digest + binding["osUpdatedAt"] = "2026-11-12T10:02:00Z" + assert client.ready_route(approval).digest != evidence.digest + assert evidence.binding_updated_at == "2026-11-12T10:01:00Z" + + +@pytest.mark.parametrize("entity", ["Participant", "ProviderSchema", "ActionBinding"]) +@pytest.mark.parametrize("reason", ["missing", "duplicate", "inactive"]) +def test_denials_distinguish_registry_state(registry, approval, binding, entity, reason): + client, replies, _ = registry + rows = binding["actions"] if entity == "ActionBinding" else replies[entity] + if reason == "missing": + rows.clear() + elif reason == "duplicate": + rows.append({**rows[0], "status": "inactive"}) + else: + rows[0]["status"] = "inactive" + with pytest.raises(RegistryLookupError) as error: + client.ready_route(approval) + assert (error.value.entity, error.value.reason) == (entity, reason) + + +def test_network_mismatch_stops_before_registry_search(registry, approval): + client, _, calls = registry + with pytest.raises(RegistryLookupError) as error: + client.ready_route(replace(approval, network_id="other.net")) + assert (error.value.entity, error.value.reason) == ("network", "mismatch") + assert not calls + + +def test_configured_connection_uses_the_requested_network_registry(monkeypatch, participant): + monkeypatch.setenv("NETWORK_CONNECTIONS", "alpha,beta") + monkeypatch.setenv("NETWORK_CONNECTION_DEFAULT", "alpha") + for name in ("alpha", "beta"): + prefix = f"NETWORK_CONNECTION_{name.upper()}_" + monkeypatch.setenv(prefix + "TYPE", "beckn") + monkeypatch.setenv(prefix + "ID", f"{name}.example.net") + monkeypatch.setenv(prefix + "DISCOVERY_URL", f"https://discovery.{name}.example.net") + monkeypatch.setenv(prefix + "REGISTRY_URL", f"https://registry.{name}.example.net") + monkeypatch.setenv(prefix + "SENDER_ID", f"sender.{name}.example.net") + monkeypatch.delenv(prefix + "REGISTRY_TOKEN_FILE", raising=False) + + def search(request): + assert request.url.host == "registry.beta.example.net" + return httpx.Response(200, json={"data": [participant]}) + + client = httpx.Client + monkeypatch.setattr(httpx, "Client", lambda **options: client(transport=httpx.MockTransport(search), **options)) + registry = registry_for_network("beta.example.net") + assert registry.active_provider(participant["participantId"]).network_id == "beta.example.net" + + +@pytest.mark.parametrize( + ("entity", "field", "value"), + [ + ("Participant", "type", "node"), + ("Participant", "baseUrl", "https://other.example.net"), + ("ProviderSchema", "participantId", "other.example.net"), + ("ProviderSchema", "capabilityCode", "openagrinet:Other"), + ("ActionBinding", "method", "GET"), + ("ActionBinding", "path", "/different"), + ("ActionBinding", "mappings", "https://other.example.net/advisory.select.yaml"), + ("ActionBinding", "timeoutMs", 20_000), + ("ActionBinding", "retryMax", 0), + ], +) +def test_record_must_match_approved_identity_and_route(registry, approval, binding, entity, field, value): + client, replies, _ = registry + row = binding["actions"][0] if entity == "ActionBinding" else replies[entity][0] + row[field] = value + with pytest.raises(RegistryLookupError) as error: + client.ready_route(approval) + assert (error.value.entity, error.value.reason) == (entity, "mismatch") + + +@pytest.mark.parametrize( + ("response", "reason"), + [ + (httpx.Response(503), "unavailable"), + (httpx.Response(302, headers={"Location": "https://other.example.net/Participant/search"}), "unavailable"), + (httpx.Response(200, json={"params": {"status": "UNSUCCESSFUL"}, "data": []}), "unavailable"), + (httpx.Response(200, json={"unknown": []}), "invalid"), + (httpx.Response(200, json=[{"participantId": "wrong.provider.net"}]), "invalid"), + (httpx.Response(200, content=b"x" * (128 * 1024 + 1)), "invalid"), + ], +) +def test_registry_failure_does_not_read_as_missing(registry, approval, response, reason): + client, replies, calls = registry + replies["Participant"] = response + with pytest.raises(RegistryLookupError) as error: + client.active_provider(approval.participant_id) + assert (error.value.entity, error.value.reason) == ("Participant", reason) + assert len(calls) == 1 + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("mapping_sha256", "not-a-digest"), + ("mapping_runtime", "jsonata@latest"), + ("auth_profile", ""), + ("base_url", "https://host.example.net?token=unapproved"), + ("path", "/service/../secret"), + ], +) +def test_approval_requires_a_well_formed_route_and_immutable_mapping_pin(approval, field, value): + with pytest.raises(ValueError): + replace(approval, **{field: value}) diff --git a/tests/test_observability.py b/tests/test_observability.py index b782450..034d6fe 100644 --- a/tests/test_observability.py +++ b/tests/test_observability.py @@ -50,20 +50,20 @@ def test_an_endpoint_attaches_one_handler(monkeypatch): def test_info_lines_from_root_and_module_loggers_are_exported(exported): logging.info("search request query=%r", "cotton pest") - logging.getLogger("pipeline.activities").warning("ocr retry") + logging.getLogger("pipeline.temporal.activities.processing").warning("ocr retry") assert _bodies(exported) == ["search request query='cotton pest'", "ocr retry"] def test_exported_line_keeps_its_level(exported): - logging.getLogger("pipeline.activities").error("ingest failed") + logging.getLogger("pipeline.temporal.activities.processing").error("ingest failed") [item] = exported.get_finished_logs() assert item.log_record.severity_text == "ERROR" def test_debug_lines_are_not_exported(exported): - logging.getLogger("pipeline.activities").debug("page text") + logging.getLogger("pipeline.temporal.activities.processing").debug("page text") assert _bodies(exported) == [] diff --git a/tests/test_oidc.py b/tests/test_oidc.py new file mode 100644 index 0000000..8534a5e --- /dev/null +++ b/tests/test_oidc.py @@ -0,0 +1,61 @@ +"""Regression tests for Keycloak console session exchange.""" + +from unittest.mock import patch + +import pytest +from fastapi import HTTPException + +from pipeline.auth import oidc +from pipeline.auth.keycloak_admin import load_keycloak_admin_config + +ISSUER = "https://login.example/auth/realms/knowledge" +CALLBACK = "https://console.example/auth/sso-callback" +VERIFIER = "a" * 43 + + +@pytest.fixture +def oidc_settings(monkeypatch): + monkeypatch.setenv("KEYCLOAK_ISSUER", ISSUER) + monkeypatch.setenv("KEYCLOAK_CLIENT_ID", "knowledge-console") + monkeypatch.setenv("KEYCLOAK_CLIENT_SECRET", "test-console-secret") + monkeypatch.setenv("KEYCLOAK_BACKCHANNEL_URL", "http://keycloak:8080/auth/realms/knowledge") + + +def test_exchange_uses_confidential_client_and_pkce(oidc_settings): + with patch.object(oidc, "_post", return_value=(200, {"access_token": "access"})) as post: + assert oidc.exchange_authorization_code("code", CALLBACK, VERIFIER)["access_token"] == "access" + post.assert_called_once_with( + "http://keycloak:8080/auth/realms/knowledge/protocol/openid-connect/token", + form={ + "grant_type": "authorization_code", + "client_id": "knowledge-console", + "client_secret": "test-console-secret", + "code": "code", + "code_verifier": VERIFIER, + "redirect_uri": CALLBACK, + }, + ) + + +@pytest.mark.parametrize("verifier", [None, "too-short", "!" * 43]) +def test_exchange_rejects_invalid_pkce_before_contacting_keycloak(oidc_settings, verifier): + with patch.object(oidc, "_post") as post, pytest.raises(HTTPException) as exc: + oidc.exchange_authorization_code("code", CALLBACK, verifier) + assert exc.value.status_code == 400 + post.assert_not_called() + + +def test_expired_refresh_is_an_unauthenticated_session(oidc_settings): + with patch.object(oidc, "_post", return_value=(400, {"error": "invalid_grant"})) as post: + with pytest.raises(HTTPException) as exc: + oidc.refresh_tokens("stale") + assert exc.value.status_code == 401 + assert post.call_args.kwargs["form"]["grant_type"] == "refresh_token" + + +def test_admin_requires_a_service_account(monkeypatch, oidc_settings): + monkeypatch.delenv("KEYCLOAK_ADMIN_CLIENT_ID", raising=False) + monkeypatch.delenv("KEYCLOAK_ADMIN_CLIENT_SECRET", raising=False) + monkeypatch.setenv("KEYCLOAK_ADMIN_USERNAME", "old-admin") + monkeypatch.setenv("KEYCLOAK_ADMIN_PASSWORD", "unused-legacy-config") + assert not load_keycloak_admin_config().configured diff --git a/tests/test_parser_profiles.py b/tests/test_parser_profiles.py new file mode 100644 index 0000000..49aa559 --- /dev/null +++ b/tests/test_parser_profiles.py @@ -0,0 +1,92 @@ +import pytest + +from pipeline.ocr import service + + +def test_default_and_named_parser_profiles(monkeypatch): + monkeypatch.setenv("OCR_PROVIDER", "chandra") + monkeypatch.setenv("DOCUMENT_PARSER_PROFILES", "text,layout") + monkeypatch.setenv("DOCUMENT_PARSER_DEFAULT", "layout") + monkeypatch.setenv("DOCUMENT_PARSER_TEXT_TYPE", "pypdf") + monkeypatch.setenv("DOCUMENT_PARSER_LAYOUT_TYPE", "liteparse") + monkeypatch.setenv("DOCUMENT_PARSER_LAYOUT_SEGMENT_PAGES", "5") + monkeypatch.setenv("DOCUMENT_PARSER_LAYOUT_EXECUTABLE", "/usr/bin/lit") + + assert service.allowed_ocr_providers() == ("text", "layout") + assert service.default_ocr_profile() == "layout" + assert service.load_ocr_config().segment_pages == 5 + assert service.get_ocr_provider().executable == "/usr/bin/lit" + assert service.load_ocr_config("text").provider == "pypdf" + with pytest.raises(ValueError, match="not configured"): + service.load_ocr_config("chandra") + + +def test_parser_rejects_other_provider_options_without_exposing_values(monkeypatch): + monkeypatch.setenv("DOCUMENT_PARSER_TYPE", "pypdf") + monkeypatch.setenv("DOCUMENT_PARSER_API_KEY", "fixture-secret") + + with pytest.raises(ValueError, match="extra_forbidden") as error: + service.load_ocr_config() + assert "fixture-secret" not in str(error.value) + + +def test_parser_reads_adapter_credential_from_file(monkeypatch, tmp_path): + credential = tmp_path / "test-secret" + credential.write_text("fixture-secret\n") + monkeypatch.setenv("DOCUMENT_PARSER_TYPE", "mistral") + monkeypatch.setenv("DOCUMENT_PARSER_API_KEY_FILE", str(credential)) + monkeypatch.setenv("DOCUMENT_PARSER_SKIP_TEXT_LAYER", "false") + + config = service.load_ocr_config() + + assert config.api_key == "fixture-secret" + assert config.skip_text_layer is False + assert "fixture-secret" not in repr(config) + + +def test_deployment_installed_extension_runs_as_a_named_pdf_parser(monkeypatch, tmp_path): + from pypdf import PdfWriter + + from pipeline.ocr.service import allowed_ocr_providers, ocr_pdf + + distribution = tmp_path / "example_parser-1.0.dist-info" + distribution.mkdir() + (distribution / "entry_points.txt").write_text( + "[knowledge_provider.document_parsers]\nexample-ocr = example_parser:make_parser\n" + ) + (tmp_path / "example_parser.py").write_text( + """from pipeline.ocr.base import OcrProvider + +class ExampleParser(OcrProvider): + name = "example-ocr" + requires_all_pages = True + + def __init__(self, language): + self.language = language + + def process_pdf_range(self, pdf_path, start_idx, end_idx, *, log=None): + return [ + {"page_number": index + 1, "original_markdown": f"Parsed with {self.language}"} + for index in range(start_idx, end_idx) + ] + +def make_parser(options): + return ExampleParser(options["language"]) +""" + ) + monkeypatch.syspath_prepend(str(tmp_path)) + monkeypatch.setenv("DOCUMENT_PARSER_PROFILES", "text,scan") + monkeypatch.setenv("DOCUMENT_PARSER_DEFAULT", "text") + monkeypatch.setenv("DOCUMENT_PARSER_TEXT_TYPE", "pypdf") + monkeypatch.setenv("DOCUMENT_PARSER_SCAN_TYPE", "extension") + monkeypatch.setenv("DOCUMENT_PARSER_SCAN_PLUGIN", "example-ocr") + monkeypatch.setenv("DOCUMENT_PARSER_SCAN_OPTION_LANGUAGE", "en") + path = tmp_path / "document.pdf" + writer = PdfWriter() + writer.add_blank_page(200, 100) + writer.write(path) + + assert allowed_ocr_providers() == ("text", "scan") + assert ocr_pdf(str(path), str.strip, provider="scan") == [ + {"page_number": 1, "original_markdown": "Parsed with en"}, + ] diff --git a/tests/test_provenance.py b/tests/test_provenance.py index 6e1cd66..c497f45 100644 --- a/tests/test_provenance.py +++ b/tests/test_provenance.py @@ -6,7 +6,7 @@ import pytest from fastapi.testclient import TestClient -from pipeline.activities import prepare_ingestion_records +from pipeline.search.passage_projection import prepare_ingestion_records from pipeline.api import app _code_seq = itertools.count(1) @@ -35,7 +35,7 @@ def test_prepare_records_includes_provenance_fields(): filename="paripatra.pdf", chunks=chunks, workflow_id="doc-ef226cde5062", - name_en="Sample Circular", + name="Sample Circular", ) assert len(records) == 1 @@ -47,7 +47,7 @@ def test_prepare_records_includes_provenance_fields(): assert record["page_start"] == 12 assert record["page_end"] == 13 assert record["section"] == "Deworming Schedule" - assert record["name_en"] == "Sample Circular" + assert record["name"] == "Sample Circular" assert record["source"] == "docs-pipeline" assert record["type"] == "document" assert "document_kind" not in record diff --git a/tests/test_provider_id_on_chunks.py b/tests/test_provider_id_on_chunks.py index f9e926c..10235ff 100644 --- a/tests/test_provider_id_on_chunks.py +++ b/tests/test_provider_id_on_chunks.py @@ -37,25 +37,25 @@ class TestProviderIdFromDoc: ("advisory", "mh", "mh.provider.oan.dev"), ("Scheme", "MH", "mh.provider.oan.dev"), ("advisory", "bv", "bv.provider.oan.dev"), - ("scheme", "default", "bv.provider.oan.dev"), - ("advisory", None, "bv.provider.oan.dev"), + ("scheme", "default", "default.provider.oan.dev"), + ("advisory", None, "default.provider.oan.dev"), ], ) def test_published_kinds_get_the_provider_id(self, kind, instance, expected): - from pipeline.activities import _provider_id_from_doc + from pipeline.temporal.activities.indexing import _provider_id_from_doc assert _provider_id_from_doc({"kind": kind, "instance": instance}) == expected @pytest.mark.unit @pytest.mark.parametrize("kind", ["document", "video", "how_to_faq", None]) def test_kinds_that_never_publish_get_none(self, kind): - from pipeline.activities import _provider_id_from_doc + from pipeline.temporal.activities.indexing import _provider_id_from_doc assert _provider_id_from_doc({"kind": kind, "instance": "mh"}) is None @pytest.mark.unit def test_follows_the_overrides_file(self, monkeypatch, tmp_path): - from pipeline.activities import _provider_id_from_doc + from pipeline.temporal.activities.indexing import _provider_id_from_doc _use_overrides(monkeypatch, tmp_path, '{"ka": "ka.example.com"}') @@ -65,7 +65,7 @@ def test_follows_the_overrides_file(self, monkeypatch, tmp_path): @pytest.mark.unit def test_no_network_id_means_no_provider_id(self, monkeypatch): - from pipeline.activities import _provider_id_from_doc + from pipeline.temporal.activities.indexing import _provider_id_from_doc monkeypatch.delenv("NETWORK_SENDER_ID", raising=False) @@ -73,7 +73,7 @@ def test_no_network_id_means_no_provider_id(self, monkeypatch): @pytest.mark.unit def test_an_invalid_provider_id_fails_without_retry(self): - from pipeline.activities import _provider_id_from_doc + from pipeline.temporal.activities.indexing import _provider_id_from_doc with pytest.raises(ApplicationError, match="ladakh_ut") as excinfo: _provider_id_from_doc({"kind": "advisory", "instance": "ladakh_ut"}) @@ -85,9 +85,9 @@ def test_an_invalid_provider_id_fails_without_retry(self): "kind,instance", [("advisory", "mh"), ("scheme", "bv"), ("advisory", "default"), ("scheme", "ka")] ) def test_matches_the_provider_id_sent_in_publish(self, monkeypatch, tmp_path, kind, instance): - from pipeline.activities import _provider_id_from_doc + from pipeline.temporal.activities.indexing import _provider_id_from_doc from pipeline.auth.tenancy import group_for_instance - from pipeline.discovery_publish_service import DiscoveryPublishService + from pipeline.network.catalog.discovery_publish_service import DiscoveryPublishService _use_overrides(monkeypatch, tmp_path, '{"ka": "ka.example.com"}') monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "https://discovery.example.com") @@ -97,7 +97,7 @@ def test_matches_the_provider_id_sent_in_publish(self, monkeypatch, tmp_path, ki client.__enter__.return_value = client client.post.return_value = response - with patch("pipeline.discovery_publish_service.httpx.Client", return_value=client): + with patch("pipeline.network.catalog.discovery_publish_service.httpx.Client", return_value=client): DiscoveryPublishService().publish("txn", kind, "wf", group_for_instance(instance)) published = client.post.call_args.kwargs["json"]["message"]["catalogs"][0]["provider"]["id"] @@ -107,7 +107,7 @@ def test_matches_the_provider_id_sent_in_publish(self, monkeypatch, tmp_path, ki class TestProviderIdOnRecords: @pytest.mark.unit def test_every_record_carries_the_provider_id(self): - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records records = _prepare_records("doc-1", "doc.pdf", [CHUNK, {**CHUNK, "chunk_number": 2}], provider_id="mh.provider.oan.dev") @@ -115,7 +115,7 @@ def test_every_record_carries_the_provider_id(self): @pytest.mark.unit def test_no_provider_id_is_left_off_the_point(self): - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records from pipeline.vector_store.qdrant_store import _record_payload (record,) = _prepare_records("doc-1", "doc.pdf", [CHUNK]) @@ -124,7 +124,7 @@ def test_no_provider_id_is_left_off_the_point(self): @pytest.mark.unit def test_provider_id_is_in_the_passage_schema(self): - from pipeline.activities import _passage_schema_field_names + from pipeline.search.passage_projection import _passage_schema_field_names assert "provider_id" in _passage_schema_field_names() @@ -136,7 +136,7 @@ class _RecordsBuilt(Exception): class TestIngestPathsStampTheProviderId: @pytest.fixture def captured(self, monkeypatch): - import pipeline.activities as activities + from pipeline.temporal.activities import indexing as activities import pipeline.db as db seen = {} @@ -147,13 +147,15 @@ def capture(*args, **kwargs): monkeypatch.setattr(activities, "_prepare_records", capture) monkeypatch.setattr(db, "get_chunks", lambda *a, **k: [CHUNK]) - monkeypatch.setattr(db, "get_document", lambda workflow_id: {"kind": "scheme", "instance": "mh"}) + monkeypatch.setattr(db, "get_document", lambda workflow_id: { + "id": "00000000-0000-0000-0000-000000000001", "kind": "scheme", "instance": "mh" + }) return seen @pytest.mark.unit @pytest.mark.asyncio async def test_dev_ingest(self, captured): - from pipeline.activities import ingest_document_from_db + from pipeline.temporal.activities.indexing import ingest_document_from_db with pytest.raises(_RecordsBuilt): await ingest_document_from_db("wf-1", "doc-1", "doc.pdf") @@ -163,11 +165,11 @@ async def test_dev_ingest(self, captured): @pytest.mark.unit @pytest.mark.asyncio async def test_prod_promotion(self, captured, monkeypatch): - import pipeline.activities as activities + from pipeline.temporal.activities import indexing as activities monkeypatch.setenv("PROD_VECTOR_DB_URL", "http://localhost:6333") monkeypatch.setenv("PROD_SCHEME_VECTOR_DB_COLLECTION_NAME", "schemes-index") - monkeypatch.setattr(activities, "get_qdrant_client", lambda *a, **k: MagicMock(), raising=False) + monkeypatch.setattr("pipeline.vector_store.qdrant_store.get_qdrant_client", lambda *a, **k: MagicMock()) with pytest.raises(_RecordsBuilt): await activities.promote_document_to_prod_qdrant("wf-1", "doc-1", "doc.pdf") diff --git a/tests/test_provider_memberships.py b/tests/test_provider_memberships.py new file mode 100644 index 0000000..e178b68 --- /dev/null +++ b/tests/test_provider_memberships.py @@ -0,0 +1,233 @@ +from contextlib import asynccontextmanager, nullcontext +from types import SimpleNamespace +from urllib.parse import parse_qs, urlsplit + +from fastapi import FastAPI, HTTPException +from fastapi.testclient import TestClient +from qdrant_client import QdrantClient + +from pipeline import db, vector_store +from pipeline.api.routers import providers, search +from pipeline.auth import deps, keycloak_admin +from pipeline.auth.deps import CurrentUser +from pipeline.auth.jwt import claims_to_user +from pipeline.auth.permissions import Permission +from pipeline.vector_store import qdrant_store + + +class KeycloakGroups: + """Test double at the Keycloak HTTP boundary, with independent memberships.""" + + def __init__(self): + self.groups = {} + self.users = {} + self.memberships = {} + self._add_group("/global/super-admin") + for user in ("platform", "owner", "member"): + self.users[user] = { + "id": user, "email": f"{user}@example.test", "username": user, + "emailVerified": True, "enabled": True, + } + self.memberships[user] = set() + self.memberships["platform"].add(self.groups["/global/super-admin"]["id"]) + + def _add_group(self, path): + if path in self.groups: + raise HTTPException(409) + self.groups[path] = {"id": f"group-{len(self.groups)}", "path": path, "name": path.rsplit("/", 1)[1]} + + def _by_id(self, group_id): + return next(group for group in self.groups.values() if group["id"] == group_id) + + def request(self, method, url, *, token, body=None): + assert token == "test-admin-token" + request = urlsplit(url) + path = request.path.removeprefix("/admin/realms/local/") + query = parse_qs(request.query) + if path == "users": + email = query["email"][0] + return 200, [user for user in self.users.values() if user["email"] == email] + if path.startswith("users/"): + _, user, *rest = path.split("/") + if user not in self.users: + raise HTTPException(404) + if not rest: + return 200, self.users[user] + if method == "GET": + return 200, [self._by_id(group_id) for group_id in sorted(self.memberships[user])] + if method == "PUT": + self.memberships[user].add(rest[1]) + elif method == "DELETE": + self.memberships[user].remove(rest[1]) + return 204, None + if path.startswith("group-by-path/"): + group = self.groups.get("/" + path.removeprefix("group-by-path/")) + if group is None: + raise HTTPException(404) + return 200, group + if path == "groups": + parent = "" + else: + _, group_id, *rest = path.split("/") + group = self._by_id(group_id) + parent = group["path"] + if not rest: + return 200, group + if rest == ["members"]: + return 200, [self.users[user] for user, groups in self.memberships.items() if group_id in groups] + if method == "POST": + self._add_group(f"{parent}/{body['name']}") + return 201, None + return 200, [group for key, group in self.groups.items() if key.rsplit("/", 1)[0] == parent] + + +def test_provider_lifecycle_and_revocation_with_same_user_token(monkeypatch): + keycloak = KeycloakGroups() + monkeypatch.setenv("AUTH_DISABLED", "false") + monkeypatch.setenv("KEYCLOAK_VERIFY_LIVE_GROUPS", "true") + monkeypatch.setattr(keycloak_admin, "require_admin_config", lambda: SimpleNamespace(base_url="http://keycloak", realm="local")) + monkeypatch.setattr(keycloak_admin, "_admin_token", lambda _: "test-admin-token") + monkeypatch.setattr(keycloak_admin, "_req", keycloak.request) + monkeypatch.setattr(deps, "decode_and_validate_token", lambda token, _: claims_to_user({ + "sub": token, "groups": ["/states/RICE-LAB/admin", "/states/COTTON-HUB/admin"], + "realm_access": {"roles": ["state_admin"]}, "instances": ["rice-lab", "cotton-hub"], + })) + monkeypatch.setattr(deps, "_sync_user_row", lambda _: None) + connection = SimpleNamespace(execute=lambda *args: None) + monkeypatch.setattr(db, "get_connection", lambda: nullcontext(connection)) + instances = {} + monkeypatch.setattr(db, "upsert_instance", lambda path, keycloak_group_id: instances.setdefault(path, keycloak_group_id)) + monkeypatch.setattr(db, "list_provider_instances", lambda scope: [ + {"provider": path.rsplit("/", 1)[1].lower(), "id": group_id} + for path, group_id in instances.items() if scope is None or path.rsplit("/", 1)[1].lower() in scope + ]) + monkeypatch.setattr(db, "get_search_settings", lambda: {}) + monkeypatch.setattr(qdrant_store, "get_vector_size", lambda: 2) + monkeypatch.setattr(qdrant_store, "embed_passages", lambda texts, **_: [[1.0, 1.0] for _ in texts]) + monkeypatch.setattr(qdrant_store, "embed_query", lambda *args, **_: [1.0, 1.0]) + store = qdrant_store.QdrantVectorStore(client=QdrantClient(":memory:")) + store.upsert("test-providers", [ + {"doc_id": code, "instance": code, "text": "cotton advisory", "is_reference": False} + for code in ("rice-lab", "cotton-hub", "outside") + ]) + monkeypatch.setattr(vector_store, "get_vector_store", lambda: store) + + app = FastAPI() + app.include_router(providers.router) + app.include_router(search.router) + + @app.get("/access") + def access(user: CurrentUser): + return {provider: user.has_permission_for_instance(Permission.UPLOAD, provider) for provider in user.instances} + + def headers(user): + return {"Authorization": f"Bearer {user}"} + + with TestClient(app) as client: + for code in ("rice-lab", "cotton-hub", "rice-lab"): + response = client.put(f"/providers/{code}", headers=headers("platform"), json={ + "admin_email": "owner@example.test" if code == "rice-lab" else "platform@example.test", + }) + assert response.status_code == 200, response.text + assert len(instances) == 2 + assert [row["provider"] for row in client.get("/providers", headers=headers("owner")).json()["providers"]] == ["rice-lab"] + contributor = {"email": "member@example.test", "role": "state_contributor"} + assert client.post("/providers/cotton-hub/members", headers=headers("owner"), json=contributor).status_code == 403 + assert client.post("/providers/rice-lab/members", headers=headers("owner"), json=contributor).status_code == 200 + assert client.post("/providers/cotton-hub/members", headers=headers("platform"), json={ + **contributor, "role": "state_view", + }).status_code == 200 + assert client.get("/access", headers=headers("member")).json() == {"cotton-hub": False, "rice-lab": True} + members = client.get("/providers/rice-lab/members", headers=headers("owner")).json()["members"] + assert {row["user_id"]: row["role"] for row in members} == {"member": "state_contributor", "owner": "state_admin"} + + query = {"index_name": "test-providers", "query": "cotton"} + hits = client.post("/search", headers=headers("member"), json=query).json()["hits"] + assert {row["instance"] for row in hits} == {"cotton-hub", "rice-lab"} + assert client.delete("/providers/rice-lab/members/owner", headers=headers("owner")).status_code == 403 + assert client.delete("/providers/rice-lab/members/member", headers=headers("owner")).status_code == 200 + assert client.get("/access", headers=headers("member")).json() == {"cotton-hub": False} + assert client.post("/search", headers=headers("member"), json={**query, "group": "rice-lab"}).status_code == 403 + hits = client.post("/search", headers=headers("member"), json=query).json()["hits"] + assert {row["instance"] for row in hits} == {"cotton-hub"} + + +def test_intake_lists_all_registered_providers_only_for_platform_administrators(monkeypatch): + from pipeline.api import api + + monkeypatch.setenv("KEYCLOAK_PROVIDER_GROUP_ROOT", "providers") + monkeypatch.delenv("PORTAL_INSTANCE", raising=False) + reads = [] + + def registered(scope): + reads.append(scope) + return [{"provider": "rice-lab"}, {"provider": "cotton-hub"}] + + monkeypatch.setattr(db, "list_provider_instances", registered) + client = TestClient(api.app) + + def list_as(groups): + user = claims_to_user({"sub": "member", "groups": groups}) + monkeypatch.setitem(api.app.dependency_overrides, deps.get_current_user, lambda: user) + return client.get("/upload/providers") + + admin = list_as(["/global/super-admin"]) + assert admin.status_code == 200 + assert admin.json() == {"providers": ["cotton-hub", "rice-lab"], "default_provider": None} + + contributor = list_as([ + "/global/global-viewer", "/providers/RICE-LAB/contributor", "/providers/COTTON-HUB/view", + ]) + assert contributor.status_code == 200 + assert contributor.json() == {"providers": ["rice-lab"], "default_provider": None} + assert list_as(["/global/global-viewer"]).status_code == 403 + assert reads == [None] + + +def test_readable_provider_choices_match_document_access(monkeypatch): + from pipeline.api import api + + monkeypatch.setenv("KEYCLOAK_PROVIDER_GROUP_ROOT", "providers") + monkeypatch.setattr(db, "list_provider_instances", lambda _: [{"provider": "registered"}]) + client = TestClient(api.app) + + def choices_for(groups): + user = claims_to_user({"sub": "member", "groups": groups}) + monkeypatch.setitem(api.app.dependency_overrides, deps.get_current_user, lambda: user) + return client.get("/providers/readable").json()["providers"] + + assert choices_for(["/global/super-admin"]) == ["registered"] + assert choices_for(["/global/global-viewer"]) == ["registered"] + assert choices_for(["/providers/MEMBER/view"]) == ["member"] + assert choices_for([]) == [] + + +def test_document_actions_use_permission_in_the_document_provider(monkeypatch): + from pipeline.api import api + + actor = claims_to_user({"sub": "member", "groups": ["/states/RICE/admin", "/states/COTTON/view"]}) + monkeypatch.setitem(api.app.dependency_overrides, deps.get_current_user, lambda: actor) + docs = { + "rice-test": {"workflow_id": "rice-test", "instance": "rice", "stage": "ocr_review"}, + "cotton-test": {"workflow_id": "cotton-test", "instance": "cotton", "stage": "ocr_review"}, + } + monkeypatch.setattr(db, "get_document", docs.get) + + @asynccontextmanager + async def no_services(_app): + yield + + monkeypatch.setattr(api.app.router, "lifespan_context", no_services) + with TestClient(api.app, raise_server_exceptions=True) as client: + denied = [ + client.patch("/documents/cotton-test/pages/1", json={"edited_markdown": "Changed"}), + client.patch("/documents/cotton-test/chunks/1", json={"edited_text": "Changed"}), + client.post("/documents/cotton-test/retry-ocr"), + client.post("/documents/cotton-test/approve-ocr"), + client.post("/documents/cotton-test/approve-ingestion"), + ] + assert [response.status_code for response in denied] == [403] * len(denied) + bulk = client.post("/documents/bulk/approve-ocr", json={ + "workflow_ids": ["rice-test", "cotton-test"], "dry_run": True, + }).json() + assert {item["workflow_id"]: item["ok"] for item in bulk["results"]} == {"rice-test": True, "cotton-test": False} diff --git a/tests/test_provider_sources.py b/tests/test_provider_sources.py new file mode 100644 index 0000000..1c44c5c --- /dev/null +++ b/tests/test_provider_sources.py @@ -0,0 +1,63 @@ +import json + +import pytest + +from pipeline.intake import provider_sources + + +def test_settings_validate_templates_and_schedule_without_network(): + update = provider_sources.SourceSettingsUpdate(version=0, templates={ + "notices": {"fields": {"external_id": "/id", "title": "/title", "text": "/text"}}, + }, crawl={"notices": {"host": "notices.example", "intake_template": "notices"}}, schedules=[{ + "source_key": "daily", "host": "notices.example", "url": "https://notices.example/feed", + "interval_seconds": 900, "intake_template": "notices", + }]) + validated = provider_sources.validate_settings("example", update, {}) + assert validated["schedules"][0]["intake_template"] == "notices" + assert "mapping" not in validated["schedules"][0] + assert validated["templates"]["notices"]["runtime"] == "json-pointer-v1" + + +def test_provider_settings_cannot_select_or_retarget_credentials(): + prior = {"crawl": {"feed": {"host": "notices.example", "authorization_env": "CRAWL_AUTH_FIXTURE"}}} + same = provider_sources.SourceSettingsUpdate(version=1, crawl={"feed": {"host": "notices.example", "authenticated": True}}) + validated = provider_sources.validate_settings("example", same, prior) + assert validated["crawl"]["feed"]["authorization_env"] == "CRAWL_AUTH_FIXTURE" + public = provider_sources.public_settings({"version": 2, "settings": validated}) + assert public["crawl"]["feed"]["authenticated"] is True + assert "CRAWL_AUTH" not in json.dumps(public) + with pytest.raises(ValueError, match="another host"): + provider_sources.validate_settings("example", same.model_copy(update={"crawl": {"feed": {"host": "other.example"}}}), prior) + with pytest.raises(ValueError, match="deployment operator"): + provider_sources.validate_settings("example", same.model_copy(update={"crawl": {"new": {"host": "notices.example", "authorization_env": "CRAWL_AUTH_FIXTURE"}}}), prior) + + +def test_scheduled_settings_require_the_saved_https_host(): + update = provider_sources.SourceSettingsUpdate(version=0, schedules=[{ + "source_key": "daily", "host": "notices.example", "url": "https://other.example/feed", + }]) + with pytest.raises(ValueError, match="saved source host"): + provider_sources.validate_settings("example", update, {}) + + +@pytest.mark.db +def test_provider_settings_versioning_and_template_isolation(db_connection, monkeypatch): + from pipeline.intake.scheduled_intake import configured_profiles + from pipeline.intake.webhook_mapping import load_intake_template + + monkeypatch.delenv("SCHEDULED_PULL_PROFILES_FILE", raising=False) + monkeypatch.delenv("CRAWL_SOURCE_PROFILES", raising=False) + monkeypatch.delenv("KNOWLEDGE_INTAKE_TEMPLATES_FILE", raising=False) + db_connection.get_or_create_instance("example") + db_connection.get_or_create_instance("other") + update = provider_sources.SourceSettingsUpdate(version=0, templates={"notices": {"runtime": "canonical-v1"}}, schedules=[{ + "source_key": "daily", "host": "notices.example", "url": "https://notices.example/feed", "intake_template": "notices", + }]) + saved = provider_sources.save_settings("example", update, "fixture-admin") + assert saved["version"] == 1 + assert configured_profiles()[0].provider == "example" + assert load_intake_template("notices", provider="example").runtime == "canonical-v1" + with pytest.raises(ValueError, match="No intake templates"): + load_intake_template("notices", provider="other") + with pytest.raises(FileExistsError): + provider_sources.save_settings("example", update, "fixture-admin") diff --git a/tests/test_provider_storage_boundaries.py b/tests/test_provider_storage_boundaries.py new file mode 100644 index 0000000..17479df --- /dev/null +++ b/tests/test_provider_storage_boundaries.py @@ -0,0 +1,94 @@ +from contextlib import asynccontextmanager +from types import SimpleNamespace +from unittest.mock import AsyncMock, Mock + +import pytest +from fastapi.testclient import TestClient + +from pipeline import db, vector_store +from pipeline.api import api +from pipeline.auth.deps import get_current_user +from pipeline.auth.jwt import claims_to_user + + +@pytest.fixture +def provider_client(monkeypatch): + actor = claims_to_user({"sub": "rice-member", "groups": ["/states/RICE/contributor"]}) + monkeypatch.setitem(api.app.dependency_overrides, get_current_user, lambda: actor) + + @asynccontextmanager + async def no_services(_app): + yield + + monkeypatch.setattr(api.app.router, "lifespan_context", no_services) + with TestClient(api.app) as client: + yield client + + +def test_blob_registration_resolves_the_source_only_within_the_provider(provider_client, monkeypatch): + uri = "minio://documents/registered-advisory.pdf" + unknown_uri = "minio://documents/unassigned-advisory.pdf" + lookup = Mock(side_effect=lambda source, provider: "a" * 32 if (source, provider) == (uri, "rice") else None) + monkeypatch.setattr(db, "owned_source_fingerprint", lookup) + monkeypatch.setattr(db, "list_used_scheme_codes", lambda **_: set()) + monkeypatch.setattr(db, "get_document", lambda _: None) + monkeypatch.setattr(db, "upsert_document", Mock(return_value="document-id")) + monkeypatch.setattr(db, "create_document_job", Mock()) + monkeypatch.setattr(api, "_log_audit", Mock()) + start = AsyncMock() + monkeypatch.setattr(api, "temporal_client", SimpleNamespace(start_workflow=start)) + + unassigned = provider_client.post("/documents?kind=advisory", json={"filepath": unknown_uri}) + local_file = provider_client.post("/documents?kind=advisory", json={"filepath": "/data/documents/advisory.pdf"}) + local_batch = provider_client.post("/documents/batch?kind=advisory", json={"directory": "/data/documents"}) + assert [unassigned.status_code, local_file.status_code, local_batch.status_code] == [404, 403, 405] + start.assert_not_called() + + assigned = provider_client.post("/documents?kind=advisory", json={"filepath": uri}) + assert assigned.status_code == 200, assigned.text + assert assigned.json()["source_file_fingerprint"] == "a" * 32 + assert start.call_args.kwargs["args"][2] == uri + assert [call.args for call in lookup.call_args_list] == [(unknown_uri, "rice"), (uri, "rice")] + + +def test_vector_diagnostics_return_only_attributed_points_without_writing(provider_client, monkeypatch): + legacy_key = "a" * 32 + doc = { + "id": "6138e9a2-d87d-4ffb-9fa6-135d710b5c85", "workflow_id": "rice-document", "instance": "rice", + "index_status": {"prod": {"index_name": "published-advisories", "doc_id": legacy_key}}, + } + monkeypatch.setattr(db, "get_document", lambda _: doc) + monkeypatch.setattr(db, "get_chunks", lambda *args, **kwargs: [{}]) + write = Mock() + monkeypatch.setattr(db, "upsert_document_index_status", write) + hits = [ + {"instance": "rice", "workflow_id": "rice-document", "text": "owned passage"}, + {"instance": "rice", "workflow_id": "other-rice-document", "text": "other document"}, + {"instance": "cotton", "workflow_id": "other-document", "text": "other provider"}, + {"text": "unattributed passage"}, + ] + list_points = Mock(return_value=[{"doc_id": legacy_key, **hit} for hit in hits]) + monkeypatch.setattr(vector_store, "get_vector_store", lambda: SimpleNamespace(backend="qdrant", list_by_doc_id=list_points)) + + response = provider_client.get("/documents/rice-document/qdrant") + assert response.status_code == 200, response.text + assert [hit["text"] for hit in response.json()["hits"]] == ["owned passage"] + assert provider_client.get("/documents/rice-document/qdrant?index_name=another-index").status_code == 404 + list_points.assert_called_once_with("published-advisories", legacy_key, limit=1000) + write.assert_not_called() + + +def test_provider_index_summary_uses_scoped_counts(provider_client, monkeypatch): + counts = Mock(return_value=[{"index_name": "published-advisories", "documents": 1}]) + monkeypatch.setattr(db, "list_index_summaries", counts) + global_counts = Mock() + monkeypatch.setattr(vector_store, "get_vector_store", lambda: SimpleNamespace(backend="qdrant", get_stats=global_counts)) + + response = provider_client.get("/indexes/summary") + assert response.status_code == 200, response.text + assert response.json()[0]["documents"] == 1 + assert response.json()[0]["live_stats"] is None + assert provider_client.get("/indexes/published-advisories/stats").status_code == 403 + assert provider_client.get("/indexes/published-advisories/settings").status_code == 403 + counts.assert_called_once_with(include_disabled=None, instances=["rice"]) + global_counts.assert_not_called() diff --git a/tests/test_publication_delivery.py b/tests/test_publication_delivery.py new file mode 100644 index 0000000..84b7df7 --- /dev/null +++ b/tests/test_publication_delivery.py @@ -0,0 +1,216 @@ +from unittest.mock import Mock +from uuid import uuid4 + +import pytest +from temporalio.exceptions import ApplicationError + +from pipeline.network import publication_delivery +from pipeline.temporal.activities import publication as activities +from pipeline.network.adapters.beckn import BecknNetworkPublisher +from pipeline.network.network_publisher import ( + PublicationReceipt, + PublishConfigError, + get_network_adapter, +) + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_recorded_acceptance_survives_artifact_failure_and_retries( + db_connection, sample_document, monkeypatch, +): + db = db_connection + workflow_id = sample_document["workflow_id"] + network_id = "test.local" + transaction_id = str(uuid4()) + publisher = Mock() + publisher.publish.return_value = PublicationReceipt( + exchange={"result_status": "ACCEPTED"}, message_id="accepted-message", + result_status="ACCEPTED", context_matches=True, + ) + monkeypatch.setenv("ENABLE_PUBLISH_TO_NETWORK", "true") + monkeypatch.setenv("DISCOVERY_NETWORK_ID", network_id) + monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "http://discovery.test") + monkeypatch.setenv("NETWORK_SENDER_ID", "provider.test.local") + monkeypatch.setattr(BecknNetworkPublisher, "publish", publisher.publish) + upload = Mock(side_effect=[ + OSError("artifact unavailable"), + ("blob://docs/exchange.json", 10, "application/json"), + ("blob://docs/exchange.json", 10, "application/json"), + ]) + monkeypatch.setattr(activities, "_upload_artifact", upload) + args = { + "require_matched_receipt": True, "publication_network_id": network_id, + } + + with pytest.raises(OSError, match="artifact unavailable"): + await activities.publish_catalog_to_network(workflow_id, transaction_id, **args) + with monkeypatch.context() as changed: + changed.setenv("DISCOVERY_VISIBLE_TO", "different.test") + with pytest.raises(ApplicationError, match="publication policy changed"): + await activities.publish_catalog_to_network(workflow_id, transaction_id, **args) + publisher.publish.assert_called_once() + result = await activities.publish_catalog_to_network(workflow_id, transaction_id, **args) + assert result["message_id"] == "accepted-message" + await activities.publish_catalog_to_network(workflow_id, transaction_id, **args) + publisher.publish.assert_called_once() + artifacts = db.list_document_artifacts(workflow_id) + assert len([a for a in artifacts if a["artifact_type"] == "network_publish_payload"]) == 1 + + with pytest.raises(PublishConfigError, match="different release"): + publication_delivery.deliver_publication( + network_id, transaction_id, db.get_document(workflow_id)["id"], "another-workflow", "advisory", + None, None, + ) + with db.get_connection() as conn: + conn.execute( + "UPDATE publication_deliveries SET request_identity = request_identity - 'publication_policy_sha256' " + "WHERE network_id = %s AND transaction_id = %s", (network_id, transaction_id), + ) + with pytest.raises(ApplicationError, match="policy changed or was not recorded"): + await activities.publish_catalog_to_network(workflow_id, transaction_id, **args) + publisher.publish.assert_called_once() + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_automatic_release_activates_from_recorded_receipt_after_database_error( + db_connection, sample_document, monkeypatch, +): + from unittest.mock import AsyncMock + + db = db_connection + workflow_id = sample_document["workflow_id"] + doc = db.get_document(workflow_id) + network_id = "test.local" + transaction_id, revision_id = str(uuid4()), str(uuid4()) + collection = "test-published" + for key, value in { + "ENABLE_PUBLISH_TO_NETWORK": "true", + "DISCOVERY_NETWORK_ID": network_id, + "DISCOVERY_SERVICE_ENDPOINT": "http://discovery.test", + "NETWORK_SENDER_ID": "provider.test.local", + "AUTO_RELEASE_NETWORK_ID": network_id, + "AUTO_RELEASE_COLLECTION": collection, + }.items(): + monkeypatch.setenv(key, value) + db.update_document_job(doc["latest_job_id"], config_json={"release_network_id": network_id}) + provider_id = get_network_adapter(network_id).provider_id(doc["kind"], doc["instance"]) + db.save_revision_snapshot( + doc["id"], workflow_id, network_id, revision_id, {"metadata": {"kind": doc["kind"]}, "passages": []}, + collection=collection, provider_id=provider_id, + ) + monkeypatch.setattr(db, "get_document_index_status", lambda *_: { + "status": "indexed", "chunk_count_indexed": 1, "doc_id": revision_id, + }) + monkeypatch.setattr(activities, "_drop_superseded_points", AsyncMock()) + monkeypatch.setattr(activities, "_upload_artifact", lambda *_: ( + "blob://docs/exchange.json", 10, "application/json", + )) + publisher = Mock() + publisher.publish.return_value = PublicationReceipt( + exchange={"result_status": "ACCEPTED"}, result_status="ACCEPTED", context_matches=True, + ) + monkeypatch.setattr(BecknNetworkPublisher, "publish", publisher.publish) + + with monkeypatch.context() as changed: + changed.setenv("NETWORK_PROVIDER_ID_TEMPLATE", "new-{instance}.{network_id}") + with pytest.raises(ApplicationError, match="no longer matches the staged release"): + await activities.publish_catalog_to_network(workflow_id, transaction_id, network_id) + publisher.publish.assert_not_called() + + activate = db.activate_network_release + with monkeypatch.context() as attempt: + attempt.setattr(db, "activate_network_release", Mock(side_effect=OSError("database disconnected"))) + with pytest.raises(OSError, match="database disconnected"): + await activities.publish_catalog_to_network(workflow_id, transaction_id, network_id) + hit = {"revision_id": revision_id, "release_network_id": network_id} + assert db.visible_release_hits([hit]) == [] + + assert db.activate_network_release is activate + await activities.publish_catalog_to_network(workflow_id, transaction_id, network_id) + assert db.visible_release_hits([hit]) == [hit] + publisher.publish.assert_called_once() + + +@pytest.mark.db +def test_managed_publication_validates_wire_attributes_against_network_pack( + db_connection, sample_document, monkeypatch, tmp_path, +): + import json + + from pipeline.network.catalog.catalog_profile import load_catalog_profile + from pipeline.metadata.schema_packs import SchemaPack, install_schema_pack + + db = db_connection + workflow_id = sample_document["workflow_id"] + document_id = db.get_document(workflow_id)["id"] + entrypoint = "https://schemas.example.test/advisory.yaml" + pack = SchemaPack( + capability="openagrinet:KnowledgeAdvisory", version="v1.0", entrypoint=entrypoint, + component="Advisory", schemas={entrypoint: {"components": {"schemas": { + "Advisory": { + "type": "object", "required": ["informationMode"], + "properties": {"informationMode": {"const": "OnDemand"}}, + }, + }}}}, + ) + with db.get_connection() as conn: + install_schema_pack(conn, "test.local", pack) + + profile = load_catalog_profile()["advisory"].model_dump() + profile["schema_pack"] = {"capability": pack.capability, "digest": pack.digest} + profile_path = tmp_path / "profile.json" + profile_path.write_text(json.dumps({"advisory": profile})) + monkeypatch.setenv("BECKN_CATALOG_PROFILE_PATH", str(profile_path)) + publisher = Mock() + publisher.publish.return_value = PublicationReceipt( + exchange={"result_status": "ACCEPTED"}, result_status="ACCEPTED", context_matches=True, + ) + monkeypatch.setattr(BecknNetworkPublisher, "publish", publisher.publish) + + monkeypatch.setenv("DISCOVERY_NETWORK_ID", "other.local") + monkeypatch.setenv("DISCOVERY_SERVICE_ENDPOINT", "http://discovery.test") + monkeypatch.setenv("NETWORK_SENDER_ID", "provider.test.local") + with pytest.raises(PublishConfigError, match="pinned schema pack"): + publication_delivery.deliver_publication( + "other.local", str(uuid4()), document_id, workflow_id, "advisory", None, None, + ) + publisher.publish.assert_not_called() + + from types import SimpleNamespace + + from pipeline.network import publication_admission + + approvals = tmp_path / "approvals.json" + approvals.write_text(json.dumps([{ + "network_id": "test.local", "participant_id": "bv.test.local", + "capability": pack.capability, "action": "select", "base_url": "http://provider:8000", + "method": "POST", "path": "/search", "mapping_url": "https://example.test/advisory.select.yaml", + "mapping_sha256": "a" * 64, "mapping_runtime": "jsonata@2.1.0", "auth_profile": "provider-search", + }])) + for key, value in { + "NETWORK_CONNECTIONS": "test", "NETWORK_CONNECTION_DEFAULT": "test", + "NETWORK_CONNECTION_TEST_TYPE": "beckn", "NETWORK_CONNECTION_TEST_ID": "test.local", + "NETWORK_CONNECTION_TEST_SENDER_ID": "bv.test.local", + "NETWORK_CONNECTION_TEST_DISCOVERY_URL": "http://discovery:8080", + "NETWORK_CONNECTION_TEST_APPROVED_ROUTES_FILE": str(approvals), + }.items(): + monkeypatch.setenv(key, value) + registry = Mock() + registry.ready_route.return_value = SimpleNamespace(digest="b" * 64) + monkeypatch.setattr(publication_admission, "registry_for_network", lambda _: registry) + transaction_id = str(uuid4()) + receipt = publication_delivery.deliver_publication( + "test.local", transaction_id, document_id, workflow_id, "advisory", None, None, + ) + assert receipt.result_status == "ACCEPTED" + assert receipt.exchange["binding_evidence_sha256"] == "b" * 64 + registry.ready_route.assert_called_once() + publisher.publish.assert_called_once() + registry.ready_route.return_value = SimpleNamespace(digest="c" * 64) + with pytest.raises(PublishConfigError, match="changed after the publication receipt"): + publication_delivery.deliver_publication( + "test.local", transaction_id, document_id, workflow_id, "advisory", None, None, + ) + publisher.publish.assert_called_once() diff --git a/tests/test_qdrant_validity.py b/tests/test_qdrant_validity.py index ce5c3e1..70968d7 100644 --- a/tests/test_qdrant_validity.py +++ b/tests/test_qdrant_validity.py @@ -5,7 +5,7 @@ actually applies them and honours the injected clock. """ -from datetime import datetime +from datetime import datetime, timezone import pytest from pydantic import ValidationError @@ -193,7 +193,7 @@ def test_validity_is_on_by_default(self): def test_defaults_to_the_real_clock(self): store = QdrantVectorStore(client=FakeClient()) - assert store.today() == datetime.now().date().isoformat() + assert store.today() == datetime.now(timezone.utc).date().isoformat() class TestPayload: @@ -296,3 +296,21 @@ def test_search_builds_a_usable_filter_from_an_unpadded_override(self): assert result["valid_on"] == "2026-01-02" assert ("start_date", "lte", "2026-01-02") in dates_in(client.scroll_filter) + + +@pytest.mark.unit +def test_region_exclusion_applies_before_qdrant_returns_points(): + from qdrant_client import QdrantClient + + client = QdrantClient(":memory:") + client.create_collection("geography", vectors_config=qmodels.VectorParams(size=2, distance=qmodels.Distance.COSINE)) + client.upsert("geography", [ + qmodels.PointStruct(id=1, vector=[1.0, 0.0], payload={"coverage_area_keys": ["fixture:north"]}), + qmodels.PointStruct(id=2, vector=[1.0, 0.0], payload={"excluded_coverage_area_keys": ["fixture:north"]}), + qmodels.PointStruct(id=3, vector=[1.0, 0.0], payload={"coverage_area_keys": ["fixture:south"]}), + qmodels.PointStruct(id=4, vector=[1.0, 0.0], payload={}), + ]) + points, _ = client.scroll("geography", scroll_filter=_build_filter( + applicability={"coverage_area_keys": ["fixture:north"]}, + )) + assert {point.id for point in points} == {1, 4} diff --git a/tests/test_rebuild_qdrant.py b/tests/test_rebuild_qdrant.py new file mode 100644 index 0000000..d5f40ec --- /dev/null +++ b/tests/test_rebuild_qdrant.py @@ -0,0 +1,71 @@ +from uuid import uuid4 + +import pytest +from qdrant_client import QdrantClient + +from pipeline.vector_store.qdrant_store import QdrantVectorStore +from tests.migration_kit import load_migration_tool + + +@pytest.mark.db +def test_rebuild_uses_completed_snapshot_and_never_reuses_collection(db_connection, monkeypatch): + from pipeline.vector_store import qdrant_store + rebuild = load_migration_tool("rebuild_qdrant_from_postgres") + + db = db_connection + workflow = "frozen-projection" + doc_id = db.upsert_document( + workflow_id=workflow, document_id="a" * 32, filename="source.pdf", + filepath="minio://docs/source.pdf", instance="bv", kind="advisory", code="frozen-projection", + ) + accepted_id = str(uuid4()) + db.save_draft_revision_snapshot(doc_id, workflow, accepted_id, {"passages": [{ + "_id": "a" * 32, "doc_id": doc_id, "chunk_num": 0, "text": "accepted passage", "workflow_id": workflow, + }]}) + job = db.get_latest_document_job(workflow) + db.update_document_job(job["id"], status="completed") + db.create_document_job(doc_id, "pipeline", temporal_workflow_id="pending-projection") + db.save_draft_revision_snapshot(doc_id, "pending-projection", str(uuid4()), {"passages": [{ + "_id": "b" * 32, "doc_id": doc_id, "text": "unfinished changes", "workflow_id": workflow, + }]}) + client = QdrantClient(":memory:") + store = QdrantVectorStore(client=client) + monkeypatch.setenv("VECTOR_DB_COLLECTION_NAME", "serving") + monkeypatch.setattr(rebuild, "QdrantVectorStore", lambda: store) + monkeypatch.setattr(qdrant_store, "get_vector_size", lambda: 2) + def embed(texts, *, batch_size, **_): + assert batch_size == 4 + return [[1.0, 0.0] for _ in texts] + + monkeypatch.setattr(qdrant_store, "embed_passages", embed) + assert rebuild.rebuild("replacement", embedding_batch_size=4)["indexed_passages"] == 1 + points, _ = client.scroll("replacement") + assert [point.payload["text"] for point in points] == ["accepted passage"] + assert db.visible_release_hits([point.payload for point in points]) == [point.payload for point in points] + from pipeline.search.index_identity import projection_doc_id + + assert points[0].payload["doc_id"] == projection_doc_id(db.get_document(doc_id)) + with pytest.raises(ValueError, match="already exists"): + rebuild.rebuild("replacement") + with pytest.raises(ValueError, match="serving collection"): + rebuild.rebuild("serving") + assert client.count("replacement", exact=True).count == 1 + + +def test_removes_only_stale_passages_for_the_reindexed_document(monkeypatch): + from pipeline.vector_store import qdrant_store + + store = QdrantVectorStore(client=QdrantClient(":memory:")) + monkeypatch.setattr(qdrant_store, "get_vector_size", lambda: 2) + monkeypatch.setattr(qdrant_store, "embed_passages", lambda texts, **_: [[1.0, 0.0] for _ in texts]) + original = [ + {"_id": "a" * 32, "doc_id": "first", "text": "obsolete"}, + {"_id": "b" * 32, "doc_id": "second", "text": "unrelated"}, + ] + updated = [{"_id": "c" * 32, "doc_id": "first", "text": "edited"}] + store.upsert("index", original) + store.upsert("index", updated) + store.remove_superseded_passages("index", "first", updated) + store.remove_superseded_passages("index", "first", updated) + points, _ = store.client.scroll("index") + assert {point.payload["text"] for point in points} == {"edited", "unrelated"} diff --git a/tests/test_registry_operator.py b/tests/test_registry_operator.py new file mode 100644 index 0000000..317e99b --- /dev/null +++ b/tests/test_registry_operator.py @@ -0,0 +1,233 @@ +"""Database tests for scoped, append-only Registry operator decisions.""" + +from concurrent.futures import ThreadPoolExecutor +from dataclasses import replace +from threading import Barrier +from uuid import UUID, uuid4 + +import httpx +import psycopg +import pytest +from psycopg.types.json import Jsonb + +from pipeline.network.network_registry import ApprovedRoute +from pipeline.network.registry_operator import ( + AdmissionChange, + RegistryActor, + RegistryProposalConflict, + RegistryRecordVersion, + RouteChange, + decide_registry_proposal, + read_registry_proposal, + submit_registry_proposal, +) + +pytestmark = pytest.mark.db + +NETWORK = "dev.example.net" +OTHER_NETWORK = "prod.example.net" +CAPABILITY = "openagrinet:KnowledgeAdvisory" +DIGEST = "a" * 64 + + +@pytest.fixture +def operator_ledger(db_connection, monkeypatch, tmp_path): + monkeypatch.setenv("NETWORK_CONNECTIONS", "dev,prod") + monkeypatch.setenv("NETWORK_CONNECTION_DEFAULT", "dev") + overrides = tmp_path / "provider-ids.json" + overrides.write_text("{}", encoding="utf-8") + monkeypatch.setenv("PROVIDER_CONFIG_OVERRIDE_PATH", str(overrides)) + monkeypatch.setenv("NETWORK_PROVIDER_ID_TEMPLATE", "{instance}.{network_id}") + for name, network in (("DEV", NETWORK), ("PROD", OTHER_NETWORK)): + prefix = f"NETWORK_CONNECTION_{name}_" + monkeypatch.setenv(prefix + "TYPE", "beckn") + monkeypatch.setenv(prefix + "ID", network) + monkeypatch.setenv(prefix + "DISCOVERY_URL", f"https://discovery.{network}") + monkeypatch.setenv(prefix + "REGISTRY_URL", f"https://registry.{network}") + monkeypatch.setenv(prefix + "SENDER_ID", f"operator.{network}") + + def unexpected_request(*args, **kwargs): + pytest.fail("Operator proposals must not call a remote Registry") + + monkeypatch.setattr(httpx, "Client", unexpected_request) + provider = UUID(db_connection.get_or_create_instance("knowledge-provider")) + manager = RegistryActor("provider-manager", managed_provider_ids={provider}) + operator = RegistryActor("network-operator", operator_network_ids={NETWORK}) + return db_connection, provider, manager, operator + + +def admission_args(provider, manager, **overrides): + return { + "actor": manager, + "network_id": NETWORK, + "provider_id": provider, + "request_id": uuid4(), + "expected_version": 0, + "change": AdmissionChange("Knowledge provider", "https://provider.example.net", "active"), + "observed_record": None, + "reason": "Request admission", + **overrides, + } + + +def decision_args(proposal, operator, **overrides): + return { + "actor": operator, + "network_id": proposal.network_id, + "proposal_id": proposal.id, + "expected_version": proposal.version, + "outcome": "approved", + "reason": "Verified provider ownership", + **overrides, + } + + +def test_idempotent_submission_and_decision_preserve_audit_history(operator_ledger): + _, provider, manager, operator = operator_ledger + submitted = admission_args(provider, manager) + proposal = submit_registry_proposal(**submitted) + assert proposal.version == 1 + assert proposal.participant_id == f"knowledge-provider.{NETWORK}" + assert proposal.decision is None + assert proposal.submitted_by == manager.principal_id + assert submit_registry_proposal(**submitted) == proposal + + decision = decision_args(proposal, operator) + approved = decide_registry_proposal(**decision) + assert approved.decision.outcome == "approved" + assert approved.decision.decided_by == operator.principal_id + assert approved.decision.reason == decision["reason"] + assert approved.decision.decided_at >= approved.submitted_at + assert decide_registry_proposal(**decision) == approved + assert submit_registry_proposal(**submitted) == approved + assert read_registry_proposal(actor=manager, network_id=NETWORK, proposal_id=proposal.id) == approved + + with pytest.raises(RegistryProposalConflict, match="different proposal"): + submit_registry_proposal(**{**submitted, "reason": "Different request"}) + with pytest.raises(RegistryProposalConflict, match="different decision"): + decide_registry_proposal(**{**decision, "outcome": "rejected"}) + + +def test_providers_submit_own_records_and_operators_decide_only_their_network(operator_ledger): + db, provider, manager, operator = operator_ledger + other = UUID(db.get_or_create_instance("unrelated-provider")) + with pytest.raises(PermissionError, match="Provider manager"): + submit_registry_proposal(**admission_args(other, manager)) + proposal = submit_registry_proposal(**admission_args(provider, manager, network_id=OTHER_NETWORK)) + + with pytest.raises(PermissionError, match="Operator access"): + decide_registry_proposal(**decision_args(proposal, operator)) + with pytest.raises(PermissionError, match="Operator access"): + decide_registry_proposal(**decision_args(proposal, manager)) + with pytest.raises(LookupError, match="not found"): + read_registry_proposal(actor=operator, network_id=OTHER_NETWORK, proposal_id=proposal.id) + with pytest.raises(LookupError, match="not found"): + read_registry_proposal(actor=manager, network_id=NETWORK, proposal_id=proposal.id) + + +def test_stale_versions_are_not_decided_and_suspension_keeps_observed_revision(operator_ledger): + _, provider, manager, operator = operator_ledger + first = submit_registry_proposal(**admission_args(provider, manager)) + with pytest.raises(RegistryProposalConflict, match="newer version"): + submit_registry_proposal(**admission_args(provider, manager)) + + inactive = replace(first.change, status="inactive") + with pytest.raises(ValueError, match="observed Registry record"): + submit_registry_proposal(**admission_args(provider, manager, expected_version=1, change=inactive)) + revision = RegistryRecordVersion("participant-row", "2026-11-12T10:00:00Z") + second = submit_registry_proposal(**admission_args( + provider, manager, expected_version=1, change=inactive, observed_record=revision, + )) + assert second.version == 2 + assert second.observed_record == revision + with pytest.raises(RegistryProposalConflict, match="superseded"): + decide_registry_proposal(**decision_args(first, operator)) + rejected = decide_registry_proposal(**decision_args(second, operator, outcome="rejected")) + assert rejected.decision.outcome == "rejected" + assert rejected.change.status == "inactive" + + +def test_route_versions_share_the_capability_binding_and_pin_an_installed_schema(operator_ledger): + db, provider, manager, operator = operator_ledger + submit_registry_proposal(**admission_args(provider, manager)) + route = ApprovedRoute( + network_id=NETWORK, + participant_id=f"knowledge-provider.{NETWORK}", + capability=CAPABILITY, + action="select", + base_url="https://provider.example.net", + method="POST", + path="/search", + mapping_url="https://mapping.example.net/v1/advisory.select.yaml", + mapping_sha256="b" * 64, + mapping_runtime="jsonata@1.5.4", + auth_profile="provider-select", + ) + change = RouteChange(route, DIGEST, "active") + with db.get_connection() as conn: + conn.execute( + """ + INSERT INTO schema_pack_versions (network_id, capability, digest, version, bundle) + VALUES (%s, %s, %s, 'v1.0', %s) + """, + (OTHER_NETWORK, CAPABILITY, DIGEST, Jsonb({})), + ) + with pytest.raises(ValueError, match="not installed for this network"): + submit_registry_proposal(**admission_args(provider, manager, change=change)) + + with db.get_connection() as conn: + conn.execute( + """ + INSERT INTO schema_pack_versions (network_id, capability, digest, version, bundle) + VALUES (%s, %s, %s, 'v1.0', %s) + """, + (NETWORK, CAPABILITY, DIGEST, Jsonb({})), + ) + with pytest.raises(ValueError, match="configured provider identity"): + wrong = replace(change, route=replace(route, participant_id=f"unrelated-provider.{NETWORK}")) + submit_registry_proposal(**admission_args(provider, manager, change=wrong)) + + select = submit_registry_proposal(**admission_args(provider, manager, change=change)) + assert select.version == 1 + assert select.change == change + + discover = replace(change, route=replace( + route, action="discover", mapping_url="https://mapping.example.net/v1/advisory.discover.yaml", + )) + update = submit_registry_proposal(**admission_args(provider, manager, expected_version=1, change=discover)) + assert update.version == 2 + assert update.change.route.action == "discover" + with pytest.raises(RegistryProposalConflict, match="superseded"): + decide_registry_proposal(**decision_args(select, operator)) + + +def test_concurrent_proposals_cannot_claim_the_same_stream_version(operator_ledger): + _, provider, manager, _ = operator_ledger + start = Barrier(2) + + def submit(): + start.wait(timeout=5) + try: + return submit_registry_proposal(**admission_args(provider, manager)) + except RegistryProposalConflict: + return None + + with ThreadPoolExecutor(max_workers=2) as pool: + futures = [pool.submit(submit) for _ in range(2)] + proposals = [future.result(timeout=10) for future in futures] + assert sorted(proposal.version for proposal in proposals if proposal is not None) == [1] + + +def test_database_rejects_rewriting_decided_history(operator_ledger): + db, provider, manager, operator = operator_ledger + proposal = submit_registry_proposal(**admission_args(provider, manager)) + decide_registry_proposal(**decision_args(proposal, operator)) + + with pytest.raises(psycopg.errors.ObjectNotInPrerequisiteState, match="append-only"): + with db.get_connection() as conn: + conn.execute("UPDATE registry_operator_proposals SET submission_reason = 'rewritten' WHERE id = %s", ( + proposal.id, + )) + with pytest.raises(psycopg.errors.ObjectNotInPrerequisiteState, match="append-only"): + with db.get_connection() as conn: + conn.execute("DELETE FROM registry_operator_decisions WHERE proposal_id = %s", (proposal.id,)) diff --git a/tests/test_registry_proposal_api.py b/tests/test_registry_proposal_api.py new file mode 100644 index 0000000..7baf56b --- /dev/null +++ b/tests/test_registry_proposal_api.py @@ -0,0 +1,115 @@ +from types import SimpleNamespace +from uuid import uuid4 + +import pytest + +from pipeline.api import app +from pipeline.auth.deps import get_current_user +from pipeline.auth.models import AuthUser +from pipeline.network.network_registry import ( + ActionObservation, + BindingObservation, + ParticipantObservation, + RegistryLookupError, +) + + +@pytest.mark.db +def test_local_proposal_api_enforces_live_provider_and_network_scopes(test_client, db_connection, monkeypatch): + monkeypatch.setenv("PORTAL_INSTANCE", "mh") + monkeypatch.setenv("KEYCLOAK_PROVIDER_GROUP_ROOT", "states") + db_connection.upsert_instance("/global") + db_connection.upsert_instance("/providers/MH") + db_connection.upsert_instance("/states/MH") + for key, value in { + "KEYCLOAK_VERIFY_LIVE_GROUPS": "true", + "NETWORK_CONNECTIONS": "dev", + "NETWORK_CONNECTION_DEFAULT": "dev", + "NETWORK_CONNECTION_DEV_TYPE": "beckn", + "NETWORK_CONNECTION_DEV_ID": "test.dev", + "NETWORK_CONNECTION_DEV_DISCOVERY_URL": "https://discovery.example.test", + "NETWORK_CONNECTION_DEV_REGISTRY_URL": "https://registry.example.test", + "NETWORK_CONNECTION_DEV_SENDER_ID": "provider.test.dev", + }.items(): + monkeypatch.setenv(key, value) + + def act_as(user): + monkeypatch.setitem(app.dependency_overrides, get_current_user, lambda: user) + + manager = AuthUser(user_id="manager", roles=["state_admin"], instances=["mh"], state_roles={"mh": "state_admin"}) + act_as(manager) + network_list = test_client.get("/providers/mh/registry-networks") + assert network_list.status_code == 200 + [network] = network_list.json()["networks"] + assert network["network_id"] == "test.dev" + assert test_client.get("/providers/other/registry-networks").status_code == 403 + + def observe_participant(participant_id): + assert participant_id == network["participant_id"] + return ParticipantObservation("participant-1", "revision-1", "Existing", "https://provider.example.test", "inactive") + + def observe_binding(participant_id, capability): + assert participant_id == network["participant_id"] + assert capability == "openagrinet:KnowledgeAdvisory" + return BindingObservation("binding-1", "revision-2", "active", ( + ActionObservation("select", "POST", "/search", "https://example.test/one.select.yaml", 15000, 0, "inactive"), + )) + + registry = SimpleNamespace(observe_participant=observe_participant, observe_binding=observe_binding) + monkeypatch.setattr("pipeline.network.registry_operator.registry_for_network", lambda _: registry) + context_url = "/providers/mh/networks/test.dev/registry-context" + context = test_client.get(context_url).json() + assert context["admission"]["expected_version"] == 0 + assert context["admission"]["current"]["status"] == "inactive" + assert context["route"] is None + context = test_client.get(context_url, params={"capability": "openagrinet:KnowledgeAdvisory"}).json() + assert context["route"]["current"]["actions"][0]["status"] == "inactive" + assert context["route"]["latest_proposal"] is None + + submission = { + "request_id": str(uuid4()), "expected_version": 0, "reason": "Onboard provider", + "change": {"name": "Maharashtra Provider", "base_url": "https://provider.example.test", "status": "active"}, + } + path = "/providers/mh/networks/test.dev/registry-proposals" + proposed = test_client.post(path, json=submission) + assert proposed.status_code == 201, proposed.text + record = proposed.json() + assert record["version"] == 1 and record["decision"] is None + assert test_client.post(path, json=submission).json()["id"] == record["id"] + context = test_client.get(context_url).json() + assert context["admission"]["expected_version"] == 1 + assert context["admission"]["latest_proposal"]["id"] == record["id"] + assert context["admission"]["current"]["status"] == "inactive" + collection = "/networks/test.dev/registry-proposals" + decision_url = f"{collection}/{record['id']}/decision" + decision = {"expected_version": 1, "outcome": "approved", "reason": "Checked desired state"} + assert test_client.put(decision_url, json=decision).status_code == 403 + + act_as(AuthUser(user_id="other-operator", groups=["/network-operators/test.prod"])) + assert test_client.get(collection).status_code == 403 + assert test_client.get(f"{collection}/{record['id']}").status_code == 404 + assert test_client.put(decision_url, json=decision).status_code == 403 + + act_as(manager) + amended = test_client.post(path, json={**submission, "request_id": str(uuid4()), "expected_version": 1}) + assert amended.status_code == 201, amended.text + latest = amended.json() + act_as(AuthUser(user_id="dev-operator", groups=["/network-operators/test.dev"])) + assert [item["id"] for item in test_client.get(collection).json()] == [latest["id"]] + assert test_client.put(decision_url, json=decision).status_code == 409 + accepted = test_client.put( + f"{collection}/{latest['id']}/decision", json={**decision, "expected_version": 2}, + ) + assert accepted.status_code == 200, accepted.text + assert accepted.json()["decision"]["outcome"] == "approved" + assert test_client.get(collection).json() == [] + assert test_client.get(context_url).status_code == 403 + + def unavailable(_participant_id): + raise RegistryLookupError("Participant", "unavailable") + + act_as(manager) + registry.observe_participant = unavailable + failure = test_client.get(context_url) + assert failure.status_code == 503 + assert failure.json()["detail"] == "The current Registry record could not be verified" diff --git a/tests/test_scheduled_intake.py b/tests/test_scheduled_intake.py new file mode 100644 index 0000000..83f905a --- /dev/null +++ b/tests/test_scheduled_intake.py @@ -0,0 +1,193 @@ +import json + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from pipeline.intake import scheduled_intake +from pipeline.api.routers import providers +from pipeline.auth.deps import get_current_user +from pipeline.auth.jwt import claims_to_user +from pipeline.intake.source_fetch import SourceFile, SourcePageError + + +@pytest.mark.db +def test_provider_scheduled_status_filters_scope_and_current_profile(db_connection, tmp_path, monkeypatch): + monkeypatch.setenv("KEYCLOAK_PROVIDER_GROUP_ROOT", "providers") + monkeypatch.setenv("KEYCLOAK_VERIFY_LIVE_GROUPS", "true") + config = tmp_path / "sources.json" + monkeypatch.setenv("SCHEDULED_PULL_PROFILES_FILE", str(config)) + base = { + "provider": "example", "host": "reports.example.test", + "url": "https://reports.example.test/feed?key=fixture-only", + "authorization_env": "CRAWL_AUTH_FIXTURE", + } + entries = [ + dict( + base, source_key=key, auto_release=(key == "retry"), + release_network_id="notices.test" if key == "retry" else None, + ) + for key in ("awaiting", "active", "retry", "done") + ] + config.write_text(json.dumps([*entries, dict(base, provider="other", source_key="other-only")])) + checkpoints = {entry["source_key"]: scheduled_intake.PullProfile.model_validate(entry).identity for entry in entries} + with db_connection.get_connection() as conn: + conn.execute( + "INSERT INTO scheduled_source_pulls (provider, source_key, profile_sha256, failure_count, cursor) " + "VALUES (%s, %s, %s, 9, 'stale-cursor')", ("example", "awaiting", "f" * 64), + ) + conn.execute( + "INSERT INTO scheduled_source_pulls (provider, source_key, profile_sha256, lease_expires_at) " + "VALUES (%s, %s, %s, now() + interval '1 minute')", checkpoints["active"], + ) + conn.execute( + "INSERT INTO scheduled_source_pulls (provider, source_key, profile_sha256, failure_count, last_fetch_at, cursor) " + "VALUES (%s, %s, %s, 2, now(), 'private-cursor')", checkpoints["retry"], + ) + conn.execute( + "INSERT INTO scheduled_source_pulls (provider, source_key, profile_sha256, last_fetch_at, last_page_policy_conflicts) " + "VALUES (%s, %s, %s, now(), 3)", checkpoints["done"], + ) + app = FastAPI() + app.include_router(providers.router) + app.dependency_overrides[get_current_user] = lambda: claims_to_user({ + "sub": "manager", "groups": ["/providers/example/admin"], + }) + client = TestClient(app) + assert client.get("/providers/other/scheduled-sources").status_code == 403 + response = client.get("/providers/example/scheduled-sources") + assert response.status_code == 200, response.text + assert response.json()["checked_at"] + assert all(value not in response.text for value in ("fixture-only", "private-cursor", "CRAWL_AUTH", "other-only")) + sources = {source["source_key"]: source for source in response.json()["sources"]} + assert {key: state["status"] for key, state in sources.items()} == { + "active": "polling", "awaiting": "awaiting_worker", "done": "scheduled", "retry": "retrying", + } + assert sources["active"]["next_check_at"] is None + assert sources["awaiting"]["last_page_received_at"] is None + assert sources["awaiting"]["last_page_policy_conflicts"] == 0 + assert sources["done"]["last_page_policy_conflicts"] == 3 + assert sources["retry"]["consecutive_failures"] == 2 + assert sources["retry"]["last_page_received_at"] and sources["retry"]["next_check_at"] + assert sources["retry"]["auto_release"] + assert sources["retry"]["release_network_id"] == "notices.test" + + config.write_text("{malformed") + invalid = client.get("/providers/example/scheduled-sources") + assert invalid.status_code == 503 + assert "malformed" not in invalid.text + + +def test_scheduled_profile_checkpoint_depends_on_resolved_template(tmp_path, monkeypatch): + profile = { + "provider": "example", "source_key": "reports", "host": "reports.example.test", + "url": "https://reports.example.test/feed", + } + mapping = {"fields": {"external_id": "/id", "title": "/name", "text": "/body"}} + template_file = tmp_path / "templates.json" + monkeypatch.setenv("KNOWLEDGE_INTAKE_TEMPLATES_FILE", str(template_file)) + template_file.write_text(json.dumps({"reports": mapping})) + + inline = scheduled_intake.PullProfile.model_validate({**profile, "mapping": mapping}) + old_profile = inline.model_dump(mode="json", exclude={"intake_template", "release_network_id"}) + assert inline.identity[2] == scheduled_intake.canonical_json_sha256(old_profile) + named = scheduled_intake.PullProfile.model_validate({**profile, "intake_template": "reports"}) + assert named.identity == inline.identity + with pytest.raises(ValueError, match="Choose an intake template or supply a mapping"): + scheduled_intake.PullProfile.model_validate({**profile, "intake_template": "reports", "mapping": mapping}) + + mapping["fields"]["text"] = "/fullText" + template_file.write_text(json.dumps({"reports": mapping})) + changed = scheduled_intake.PullProfile.model_validate({**profile, "intake_template": "reports"}) + assert changed.identity != named.identity + with pytest.raises(ValueError, match="requires automatic release"): + scheduled_intake.PullProfile.model_validate({**profile, "release_network_id": "notices.test"}) + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_scheduled_pull_resumes_cursor_after_failure_without_duplicate_receipts(db_connection, monkeypatch, mock_object_store): + db = db_connection + monkeypatch.setattr(scheduled_intake, "open_object_store", lambda _: mock_object_store) + db.get_or_create_instance("bv") + profile = scheduled_intake.PullProfile.model_validate({ + "provider": "bv", "source_key": "reports", "host": "reports.example.test", + "url": "https://reports.example.test/feed", "next_cursor_pointer": "/next", + "mapping": {"records": "/items", "fields": {"external_id": "/id", "title": "/name", "text": "/body"}}, + }) + calls = [] + + async def fetch(url, hosts, authorization): + assert hosts == {"reports.example.test"} + calls.append(url) + if len(calls) == 2: + raise SourcePageError("The source timed out") + second_page = "cursor=page-2" in url + payload = { + "next": None if second_page else "page-2", + "items": [{"id": "2" if second_page else "1", "name": "Report", "body": "Full report text"}], + } + return SourceFile("feed.json", "application/json", json.dumps(payload).encode(), profile.url) + + monkeypatch.setattr(scheduled_intake, "fetch_source", fetch) + + def state(): + with db.get_connection() as conn: + return conn.execute("SELECT cursor, failure_count FROM scheduled_source_pulls").fetchone() + + assert await scheduled_intake.poll_source(profile) + assert state() == {"cursor": "page-2", "failure_count": 0} + assert not await scheduled_intake.poll_source(profile) + assert state() == {"cursor": "page-2", "failure_count": 1} + assert not await scheduled_intake.poll_source(profile) + with db.get_connection() as conn: + conn.execute("UPDATE scheduled_source_pulls SET next_run_at = now()") + assert await scheduled_intake.poll_source(profile) + assert state() == {"cursor": None, "failure_count": 0} + assert calls == [profile.url, profile.url + "?cursor=page-2", profile.url + "?cursor=page-2"] + with db.get_connection() as conn: + receipts = conn.execute( + "SELECT external_id FROM source_intake_receipts WHERE source_id = 'reports' ORDER BY external_id" + ).fetchall() + conn.execute("UPDATE scheduled_source_pulls SET next_run_at = now()") + assert receipts == [{"external_id": "1"}, {"external_id": "2"}] + + assert await scheduled_intake.poll_source(profile) + with db.get_connection() as conn: + assert conn.execute("SELECT count(*) AS n FROM source_intake_receipts").fetchone()["n"] == 2 + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_scheduled_pdf_registers_real_file_and_deduplicates_unchanged_content( + db_connection, monkeypatch, mock_object_store, sample_pdf_content, +): + db = db_connection + monkeypatch.setenv("VECTOR_DB_COLLECTION_NAME", "scheduled-review") + db.get_or_create_instance("bv") + profile = scheduled_intake.PullProfile( + provider="bv", source_key="bulletin", host="reports.example.test", + url="https://reports.example.test/bulletin.pdf", + ) + + async def fetch(*_): + return SourceFile("bulletin.pdf", "application/pdf", sample_pdf_content, profile.url) + + monkeypatch.setattr(scheduled_intake, "fetch_source", fetch) + monkeypatch.setattr(scheduled_intake, "open_object_store", lambda _: mock_object_store) + assert await scheduled_intake.poll_source(profile) + with db.get_connection() as conn: + receipt = conn.execute("SELECT document_id FROM source_intake_receipts").fetchone() + conn.execute("UPDATE scheduled_source_pulls SET next_run_at = now()") + document = db.get_document(str(receipt["document_id"])) + job = db.get_latest_document_job(document["id"]) + assert document["filename"] == "bulletin.pdf" + assert document["filepath"].startswith("blob://documents/scheduled/bv/bulletin/") + assert document["canonical_input_type"] == "pdf" + assert job["config"]["upload_start_args"][2] == document["filepath"] + assert job["config"]["upload_start_args"][6] == "scheduled-review" + assert job["config"]["source_url"] == profile.url + assert db.list_document_artifacts(document["id"])[0]["artifact_type"] == "original_upload" + assert await scheduled_intake.poll_source(profile) + with db.get_connection() as conn: + assert conn.execute("SELECT count(*) AS n FROM source_intake_receipts").fetchone()["n"] == 1 diff --git a/tests/test_schema_packs.py b/tests/test_schema_packs.py new file mode 100644 index 0000000..3909fbc --- /dev/null +++ b/tests/test_schema_packs.py @@ -0,0 +1,114 @@ +import json + +import pytest + +from pipeline.metadata import schema_packs + + +def _manifest(tmp_path): + entry = "https://schemas.example.test/current/attributes.yaml" + base = "https://schemas.example.test/base/attributes.yaml" + manifest = { + "capability": "example:Knowledge", + "version": "v1.0", + "entrypoint": entry, + "component": "Knowledge", + "files": {entry: "entry.json", base: "base.json"}, + } + (tmp_path / "manifest.json").write_text(json.dumps(manifest)) + (tmp_path / "entry.json").write_text(json.dumps({ + "components": {"schemas": {"Knowledge": { + "$ref": "../base/attributes.yaml#/components/schemas/Code", + }}}, + })) + return tmp_path / "manifest.json" + + +def _write_dependency(path, code): + path.write_text(json.dumps({ + "components": {"schemas": {"Code": { + "type": "object", "required": ["code"], + "properties": {"code": {"const": code}}, + }}}, + })) + + +def test_schema_bundle_requires_all_references_to_be_approved(tmp_path): + manifest = _manifest(tmp_path) + (tmp_path / "base.json").write_text(json.dumps({ + "components": {"schemas": {"Code": {"$ref": "https://other.example.test/schema"}}}, + })) + with pytest.raises(ValueError, match="outside its approved bundle"): + schema_packs.load_schema_pack(manifest) + + +@pytest.mark.db +def test_intake_keeps_the_original_pack_after_installing_a_changed_bundle(tmp_path, db_connection): + manifest = _manifest(tmp_path) + dependency = tmp_path / "base.json" + db = db_connection + + _write_dependency(dependency, "original") + original = schema_packs.load_schema_pack(manifest) + _write_dependency(dependency, "replacement") + replacement = schema_packs.load_schema_pack(manifest) + assert original.digest != replacement.digest + + with db.get_connection() as conn: + for pack in (original, replacement): + schema_packs.install_schema_pack(conn, "schema-test.net", pack) + + metadata = {"schema_pack": { + "network_id": "schema-test.net", + "capability": original.capability, + "digest": original.digest, + "attributes": {"code": "original"}, + }} + args = { + "workflow_id": "wf-pinned-pack", + "document_id": "pack-fixture", + "filename": "pack.pdf", + "filepath": "blob://documents/pack.pdf", + "code": "pack", + "kind": "advisory", + } + db.upsert_document(**args, knowledge_metadata=metadata) + assert db.get_document("wf-pinned-pack")["knowledge_metadata"] == metadata + + invalid = {"schema_pack": {**metadata["schema_pack"], "digest": replacement.digest}} + with pytest.raises(ValueError, match="rejected code"): + db.upsert_document(**{**args, "workflow_id": "wf-invalid", "code": "invalid"}, knowledge_metadata=invalid) + assert db.get_document("wf-invalid") is None + + +def test_registry_lookup_ignores_inactive_schemas_and_requires_a_unique_match(monkeypatch): + import httpx + + from pipeline.network import network_registry + from pipeline.network.network_connections import NetworkConnection + + active = { + "capabilityCode": "example:Knowledge", "status": "active", "version": "v1.0", + "schemaUrl": "https://schemas.example.test/current/attributes.yaml", + } + rows = [{**active, "status": "inactive", "version": "v0.9"}, active] + + def lookup(request): + assert request.url.path == "/api/v1/SchemaRegistry/search" + assert json.loads(request.content)["filters"]["capabilityCode"] == {"eq": "example:Knowledge"} + return httpx.Response(200, json={"data": rows}) + + client_type = httpx.Client + monkeypatch.setattr(network_registry.httpx, "Client", lambda **options: client_type( + transport=httpx.MockTransport(lookup), **options, + )) + connection = NetworkConnection( + "beckn", "test.net", "http://discovery", "provider", "", (), registry_url="http://registry", + ) + registry = network_registry.SunbirdRegistry(connection) + assert registry.active_schema("example:Knowledge") == network_registry.RegisteredSchema( + "example:Knowledge", "v1.0", active["schemaUrl"], + ) + rows.append(active) + with pytest.raises(ValueError, match="exactly one active registration"): + registry.active_schema("example:Knowledge") diff --git a/tests/test_schema_v2.py b/tests/test_schema_v2.py index bfb8c77..81196b6 100644 --- a/tests/test_schema_v2.py +++ b/tests/test_schema_v2.py @@ -77,7 +77,7 @@ def test_a_instance_with_catalogs_cannot_be_deleted(self, db_connection): instance_id = _instance(db) db.get_or_create_catalog(instance_id, "scheme") with db.get_connection() as conn: - with pytest.raises(psycopg.errors.ForeignKeyViolation): + with pytest.raises((psycopg.errors.ForeignKeyViolation, psycopg.errors.RestrictViolation)): conn.execute("DELETE FROM instances WHERE id = %s::uuid", (instance_id,)) @pytest.mark.db @@ -101,7 +101,7 @@ class TestCatalogs: @pytest.mark.db @pytest.mark.unit def test_a_new_instance_gets_one_catalog_per_supported_type(self, db_connection): - from pipeline.network_constants import KIND_TO_CATALOG_TYPE + from pipeline.network.network_constants import KIND_TO_CATALOG_TYPE db = db_connection mh = _instance(db, "mh") @@ -222,15 +222,6 @@ def test_catalog_metadata_must_be_an_object(self, db_connection): (instance_id,), ) - @pytest.mark.db - @pytest.mark.unit - def test_a_catalog_with_documents_cannot_be_deleted(self, db_connection): - db = db_connection - _doc(db, "wf-held") - with db.get_connection() as conn: - with pytest.raises(psycopg.errors.ForeignKeyViolation): - conn.execute("DELETE FROM catalogs") - class TestDocumentIdentity: @pytest.mark.db @@ -598,7 +589,7 @@ def test_the_source_has_no_page_count(self, db_connection): @pytest.mark.db @pytest.mark.unit - def test_there_are_no_instance_kind_or_demo_columns(self, db_connection): + def test_document_ownership_and_kind_are_stored_directly(self, db_connection): with db_connection.get_connection() as conn: cols = { r["column_name"] @@ -606,9 +597,9 @@ def test_there_are_no_instance_kind_or_demo_columns(self, db_connection): "SELECT column_name FROM information_schema.columns WHERE table_name = 'documents'" ).fetchall() } - assert {"instance", "document_kind", "is_demo", "catalog_entry_id", "valid_from", "valid_to", + assert {"instance", "document_kind", "is_demo", "catalog_entry_id", "catalog_id", "valid_from", "valid_to", "index_status", "uploaded_by_user_id"}.isdisjoint(cols) - assert {"catalog_id", "code", "name", "aliases", "source", "is_disabled", "metadata"} <= cols + assert {"provider_id", "kind", "code", "name", "aliases", "source", "is_disabled", "metadata"} <= cols @pytest.mark.db @pytest.mark.unit diff --git a/tests/test_scheme_catalog.py b/tests/test_scheme_catalog.py index 4e4cab7..5966374 100644 --- a/tests/test_scheme_catalog.py +++ b/tests/test_scheme_catalog.py @@ -7,7 +7,7 @@ @pytest.fixture def catalog_db(db_connection): - from pipeline import scheme_catalog + from pipeline.network.catalog import scheme_catalog yield db_connection, scheme_catalog @@ -29,20 +29,33 @@ def _doc(db, workflow_id, *, code, kind="scheme", name=None, stage="completed", ) -def test_catalog_has_no_bootstrap_entries(catalog_db): - """The catalog is documents only; the code-defined lists stay as hints.""" - db, scheme_catalog = catalog_db +def test_catalog_has_no_bootstrap_entries(catalog_db, monkeypatch): + monkeypatch.delenv("LEGACY_SCHEME_PROFILE_PATH", raising=False) + _, scheme_catalog = catalog_db assert not hasattr(scheme_catalog, "bootstrap_catalog_if_empty") snap = scheme_catalog.build_snapshot() assert snap["vector_schemes"] == [] assert snap["catalog_version"] == 0 - # Static lists are untouched: curated alias source and legacy tool list. - assert len(scheme_catalog.BOOTSTRAP_VECTOR_SCHEMES) == 13 - assert "nbm" not in scheme_catalog.BOOTSTRAP_VECTOR_SCHEMES # legacy-only - assert any(s["scheme_code"] == "nbm" for s in snap["legacy_schemes"]) + assert snap["legacy_schemes"] == [] assert "search_schemes_doc" in snap["tool_prompts"] - assert snap["routing_exceptions"]["pkvy"] == "search_schemes" - assert snap["routing_exceptions"]["nbm"] == "get_scheme_info" + assert snap["routing_exceptions"] == {} + + +def test_deployment_profile_preserves_legacy_tool_routing(catalog_db, monkeypatch, tmp_path): + _, scheme_catalog = catalog_db + before = scheme_catalog.build_snapshot() + profile = tmp_path / "scheme-profile.json" + profile.write_text( + '{"legacy_schemes":[{"scheme_code":"example","scheme_name":"Example scheme",' + '"tool":"legacy_lookup","tool_routing":"legacy"}],"routing_exceptions":{"example":"legacy_lookup"}}' + ) + monkeypatch.setenv("LEGACY_SCHEME_PROFILE_PATH", str(profile)) + after = scheme_catalog.build_snapshot() + + assert after["legacy_schemes"][0]["scheme_code"] == "example" + assert after["routing_exceptions"] == {"example": "legacy_lookup"} + assert after["tool_prompts"]["legacy_scheme_count"] == "1" + assert before["etag"] != after["etag"] def test_snapshot_has_one_entry_per_live_scheme_document(catalog_db): @@ -162,7 +175,7 @@ def test_kind_is_fixed_after_upload(catalog_db): def test_prepare_records_scheme_payload(): - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records chunks = [ { @@ -201,7 +214,7 @@ def test_prepare_records_scheme_payload(): def test_prepare_records_document_unchanged(): - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records chunks = [ { @@ -227,7 +240,7 @@ def test_prepare_records_document_unchanged(): [("advisory", "advisory"), ("video", "video"), (" Advisory ", "advisory")], ) def test_prepare_records_type_equals_kind(kind, expected): - from pipeline.activities import _prepare_records + from pipeline.search.passage_projection import _prepare_records chunks = [{"chunk_number": 0, "original_text": "Some text.", "token_count": 3, "is_excluded": False}] records = _prepare_records("doc3", "x.pdf", chunks, kind=kind) @@ -236,7 +249,7 @@ def test_prepare_records_type_equals_kind(kind, expected): @pytest.fixture -def asgi_client(db_connection, mock_temporal_client, mock_minio_client): +def asgi_client(db_connection, mock_temporal_client, mock_object_store): """HTTP client that does not run real Temporal/MinIO lifespan.""" from contextlib import asynccontextmanager @@ -245,7 +258,7 @@ def asgi_client(db_connection, mock_temporal_client, mock_minio_client): from pipeline.api import api api.temporal_client = mock_temporal_client - api.minio_client = mock_minio_client + api.object_store = mock_object_store @asynccontextmanager async def _noop_lifespan(_app): @@ -261,7 +274,8 @@ async def _noop_lifespan(_app): def test_catalog_api_snapshot(asgi_client): - from pipeline import db, scheme_catalog + from pipeline import db + from pipeline.network.catalog import scheme_catalog _doc(db, "wf-snap-1", code="pkvy", name="Paramparagat Krishi Vikas Yojana") _doc(db, "wf-snap-2", code="pm-ddky", name="Deen Dayal Upadhyaya Krishi Kalyan Yojana") @@ -294,6 +308,24 @@ def test_catalog_bootstrap_endpoint_is_gone(asgi_client): assert asgi_client.post("/catalog/v1/bootstrap").status_code in (404, 405) +def test_catalog_reads_follow_provider_access(asgi_client, monkeypatch): + from pipeline import db + from pipeline.api import api + from pipeline.auth.deps import get_current_user + from pipeline.auth.jwt import claims_to_user + + _doc(db, "wf-provider-a", code="scheme-a", instance="provider-a") + _doc(db, "wf-provider-b", code="scheme-b", instance="provider-b") + reader = claims_to_user({"sub": "reader", "groups": ["/states/PROVIDER-A/view"]}) + monkeypatch.setitem(api.app.dependency_overrides, get_current_user, lambda: reader) + for endpoint in ("schemes", "snapshot"): + result = asgi_client.get(f"/catalog/v1/{endpoint}").raise_for_status().json() + assert [entry["scheme_code"] for entry in result["vector_schemes"]] == ["scheme-a"] + assert asgi_client.get(f"/catalog/v1/{endpoint}", params={"instance": "provider-b"}).status_code == 403 + prompt = asgi_client.get("/catalog/v1/tool-prompt").raise_for_status().json() + assert prompt["vector_scheme_codes"] == ["scheme-a"] + + def test_catalog_service_key_auth(asgi_client, monkeypatch): monkeypatch.setenv("CATALOG_SERVICE_API_KEYS", "test-secret-key") @@ -401,7 +433,7 @@ class TestResolveSchemeAliases: @pytest.mark.unit def test_nmeoop_gets_aliases_without_any_curation(self): """The real case: a scheme indexed with scheme_aliases: [] matched nothing.""" - from pipeline.scheme_catalog import resolve_scheme_aliases + from pipeline.network.catalog.scheme_catalog import resolve_scheme_aliases aliases = resolve_scheme_aliases( "nmeoop", "National Mission on Edible Oils - Oil Palm", None @@ -419,7 +451,7 @@ def test_nmeoop_gets_aliases_without_any_curation(self): @pytest.mark.unit def test_acronym_is_derived_from_significant_words(self): - from pipeline.scheme_catalog import resolve_scheme_aliases + from pipeline.network.catalog.scheme_catalog import resolve_scheme_aliases aliases = resolve_scheme_aliases("cdp-2", "Crop Diversification Programme", None) @@ -427,7 +459,7 @@ def test_acronym_is_derived_from_significant_words(self): @pytest.mark.unit def test_stopwords_are_excluded_from_the_acronym(self): - from pipeline.scheme_catalog import resolve_scheme_aliases + from pipeline.network.catalog.scheme_catalog import resolve_scheme_aliases aliases = resolve_scheme_aliases("x", "National Mission on Edible Oils", None) @@ -435,7 +467,7 @@ def test_stopwords_are_excluded_from_the_acronym(self): @pytest.mark.unit def test_curator_aliases_are_kept_and_come_first(self): - from pipeline.scheme_catalog import resolve_scheme_aliases + from pipeline.network.catalog.scheme_catalog import resolve_scheme_aliases aliases = resolve_scheme_aliases( "nmeoop", @@ -449,17 +481,23 @@ def test_curator_aliases_are_kept_and_come_first(self): assert "nmeoop" in {a.lower() for a in aliases} @pytest.mark.unit - def test_bootstrap_aliases_are_merged_for_known_codes(self): - from pipeline.scheme_catalog import resolve_scheme_aliases + def test_deployment_aliases_are_merged_for_known_codes(self, monkeypatch, tmp_path): + from pipeline.network.catalog.scheme_catalog import resolve_scheme_aliases - aliases = {a.lower() for a in resolve_scheme_aliases("nmeo", None, None)} + profile = tmp_path / "scheme-profile.json" + profile.write_text( + '{"aliases_by_code":{"example":{"scheme_name":"Example Scheme",' + '"scheme_aliases":["configured alias"]}}}' + ) + monkeypatch.setenv("LEGACY_SCHEME_PROFILE_PATH", str(profile)) + aliases = {a.lower() for a in resolve_scheme_aliases("example", None, None)} - assert "nmeo-os" in aliases - assert "oilseeds mission" in aliases + assert "configured alias" in aliases + assert "example scheme" in aliases @pytest.mark.unit def test_aliases_are_deduped_case_insensitively(self): - from pipeline.scheme_catalog import resolve_scheme_aliases + from pipeline.network.catalog.scheme_catalog import resolve_scheme_aliases aliases = resolve_scheme_aliases("pkvy", "Paramparagat Krishi Vikas Yojana", ["PKVY", "pkvy"]) @@ -469,7 +507,7 @@ def test_aliases_are_deduped_case_insensitively(self): @pytest.mark.unit def test_json_string_aliases_are_accepted(self): """Aliases may arrive as a JSON string.""" - from pipeline.scheme_catalog import resolve_scheme_aliases + from pipeline.network.catalog.scheme_catalog import resolve_scheme_aliases aliases = resolve_scheme_aliases("makhana", "Makhana Scheme", '["fox nut", "lotus seed"]') @@ -478,6 +516,6 @@ def test_json_string_aliases_are_accepted(self): @pytest.mark.unit def test_no_code_and_no_name_yields_nothing(self): - from pipeline.scheme_catalog import resolve_scheme_aliases + from pipeline.network.catalog.scheme_catalog import resolve_scheme_aliases assert resolve_scheme_aliases(None, None, None) == [] diff --git a/tests/test_scoped_index_keys.py b/tests/test_scoped_index_keys.py new file mode 100644 index 0000000..ffdbe69 --- /dev/null +++ b/tests/test_scoped_index_keys.py @@ -0,0 +1,99 @@ +"""Regression checks for duplicate sources and legacy projection cleanup.""" + +import pytest + +from pipeline.search.index_identity import projection_doc_id + + +@pytest.mark.unit +def test_same_source_crops_remain_independent_and_deleting_one_preserves_the_other(monkeypatch): + from qdrant_client import QdrantClient + from pipeline.search import passage_projection + from pipeline.vector_store import qdrant_store + + monkeypatch.setattr(qdrant_store, "get_vector_size", lambda: 3) + monkeypatch.setattr(qdrant_store, "embed_passages", lambda texts, **kw: [[1.0, 0.0, 0.0] for _ in texts]) + monkeypatch.setattr(qdrant_store, "embed_query", lambda *a, **kw: [1.0, 0.0, 0.0]) + documents = [ + {"id": f"00000000-0000-0000-0000-00000000000{n}", "document_id": "a" * 32} + for n in (1, 2) + ] + records = [passage_projection._prepare_records( + projection_doc_id(document), "same.pdf", + [{"chunk_number": 1, "original_text": "Garden preparation guidance"}], + workflow_id=f"workflow-{crop}", kind="advisory", + crop_subjects=[{"subjectId": f"urn:crop:{crop}"}], + )[0] for document, crop in zip(documents, ("cotton", "rice"))] + assert records[0]["_id"] != records[1]["_id"] + client = QdrantClient(location=":memory:") + try: + store = qdrant_store.QdrantVectorStore(client=client) + store.upsert("duplicates", records) + def search(crop): + return store.search("duplicates", "garden", apply_validity=False, + crop_subject_ids=[f"urn:crop:{crop}"])["hits"] + assert len(search("cotton")) == len(search("rice")) == 1 + store.delete_by_doc_id("duplicates", projection_doc_id(documents[0])) + assert search("cotton") == [] + assert len(search("rice")) == 1 + finally: + client.close() + + +@pytest.mark.db +def test_scoped_index_migration_retains_legacy_key_and_reupload_tracks_exact_key(db_connection): + db = db_connection + owner = db.upsert_document( + workflow_id="scoped-document", document_id="a" * 32, filename="same.pdf", + filepath="minio://documents/same.pdf", kind="advisory", code="scoped-document", + ) + db.upsert_document_index_status("scoped-document", "documents-index", doc_id="a" * 32) + document = db.get_document(owner) + key = projection_doc_id(document) + db.upsert_document_index_status("scoped-document", "documents-index", doc_id=key) + assert db.get_document(owner)["superseded_doc_ids"] == {"dev": ["a" * 32]} + assert db.get_document(key)["id"] == owner + db.clear_superseded_doc_ids(owner, "dev", ["a" * 32]) + db.reset_document_for_reupload(owner, filename="same.pdf", filepath="minio://documents/same.pdf", + fingerprint="a" * 32) + assert db.get_document(owner)["superseded_doc_ids"] == {} + db.reset_document_for_reupload(owner, filename="changed.pdf", filepath="minio://documents/changed.pdf", + fingerprint="b" * 32) + assert db.get_document(owner)["superseded_doc_ids"] == {"dev": [key]} + + +@pytest.mark.db +def test_api_cleanup_uses_recorded_key_after_source_replacement(db_connection): + from pipeline.api.routers.document_lifecycle import _vector_doc_ids + + db = db_connection + owner = db.upsert_document( + workflow_id="scoped-cleanup", document_id="a" * 32, filename="same.pdf", + filepath="minio://documents/same.pdf", kind="advisory", code="scoped-cleanup", + ) + key = projection_doc_id(db.get_document(owner)) + db.upsert_document_index_status(owner, "documents-index", doc_id=key) + db.reset_document_for_reupload(owner, filename="changed.pdf", filepath="minio://documents/changed.pdf", + fingerprint="b" * 32) + assert _vector_doc_ids(db.get_document(owner), "dev") == [key] + + +def test_index_only_cleanup_targets_the_serving_collection(monkeypatch): + from pipeline.api import index_helpers + + calls = [] + + class Store: + def delete_by_doc_id(self, collection, document): + calls.append((collection, document)) + return {"deleted": 1} + + def delete_chunk(self, collection, document, chunk): + calls.append((collection, document, chunk)) + return {"deleted": 1} + + monkeypatch.setenv("VECTOR_DB_COLLECTION_NAME", "new-serving") + monkeypatch.setattr("pipeline.vector_store.get_vector_store", lambda: Store()) + index_helpers.delete_chunks_from_index("doc-1") + index_helpers.delete_single_chunk_from_index("doc-1", 2) + assert calls == [("new-serving", "doc-1"), ("new-serving", "doc-1", 2)] diff --git a/tests/test_search_group_integration.py b/tests/test_search_group_integration.py index 505de70..dfb5980 100644 --- a/tests/test_search_group_integration.py +++ b/tests/test_search_group_integration.py @@ -101,3 +101,24 @@ def test_group_and_validity_compose_in_the_engine(local_store, test_client): assert {hit["doc_id"] for hit in current_only["hits"]} == {"mh-current"} assert {hit["doc_id"] for hit in with_expired["hits"]} == {"mh-current", "mh-expired"} + + +@pytest.mark.integration +@pytest.mark.api +def test_search_offers_ambiguous_schemes_and_filters_the_selected_workflow(local_store, test_client): + local_store.upsert(INDEX, [ + { + "doc_id": f"scheme-document-{code}", "workflow_id": f"scheme-workflow-{code}", + "instance": "bv", "text": "Water support eligibility information.", + "scheme_code": code, "scheme_name": name, "scheme_aliases": ["water support"], + "chunk_num": 1, "is_reference": False, + } + for code, name in [("first", "Water Support North"), ("second", "Water Support South")] + ]) + response = search(test_client, query="Who can apply for water support?", group="bv") + assert {choice["code"] for choice in response["scheme_choices"]} == {"first", "second"} + response = search(test_client, query="Who can apply for water support?", group="bv", document_workflow_id="scheme-workflow-second") + assert response["scheme_choices"] == [] + assert [hit["workflow_id"] for hit in response["hits"]] == ["scheme-workflow-second"] + response = search(test_client, query="Who can apply for water support?", group="mh", document_workflow_id="scheme-workflow-second") + assert response["hits"] == [] diff --git a/tests/test_tenancy.py b/tests/test_tenancy.py index bf78d09..6672814 100644 --- a/tests/test_tenancy.py +++ b/tests/test_tenancy.py @@ -11,7 +11,6 @@ from pipeline.auth.jwt import claims_to_user from pipeline.auth.models import local_bypass_user from pipeline.auth.tenancy import ( - PORTAL_INSTANCE, allowed_instances, assert_document_instance_access, assert_instance_access, @@ -20,6 +19,8 @@ user_can_access_instance, ) +PORTAL_INSTANCE = "bv" # compatibility fixture configured in conftest + def test_bypass_user_is_unrestricted(): user = local_bypass_user() @@ -393,8 +394,8 @@ def _default_instances(self, monkeypatch): ("mh", "mh"), (" MH ", "mh"), ("ka", "ka"), - ("bv", None), - ("BV", None), + ("bv", "bv"), + ("BV", "bv"), ("default", None), ("", None), (None, None), @@ -438,7 +439,7 @@ def test_instance_is_reached_through_the_documents_catalog(db_connection): PORTAL_INSTANCE: "/global" if PORTAL_INSTANCE == "bv" else f"/states/{PORTAL_INSTANCE.upper()}", } # Every group is seeded with one catalog per supported type. - from pipeline.network_constants import KIND_TO_CATALOG_TYPE + from pipeline.network.network_constants import KIND_TO_CATALOG_TYPE assert sorted((r["instance_name"], r["type"]) for r in catalogs) == sorted( (group, catalog_type) diff --git a/tests/test_translation.py b/tests/test_translation.py index 472fc25..1bc8dd5 100644 --- a/tests/test_translation.py +++ b/tests/test_translation.py @@ -118,6 +118,7 @@ async def test_translate_pages_skips_english(self, monkeypatch): assert result[0].get("translated_markdown") is None assert result[0]["detected_language"] == "en" assert result[1]["translated_markdown"] == "Gujarati content translated." + assert result[1]["translations"]["en"]["translated_markdown"] == "Gujarati content translated." assert result[1]["detected_language"] == "gu" mock_provider.translate.assert_called_once() @@ -209,11 +210,8 @@ def test_empty_page_is_english(self): @pytest.mark.unit @pytest.mark.asyncio - async def test_gate_skips_pyfranc_for_english_pages(self, monkeypatch): - """No disambiguation call at all when every page is Latin script. - - pyfranc lives in script_detect now, not service — patch it there. - """ + async def test_gate_leaves_short_latin_text_unclassified(self, monkeypatch): + """Short Latin-script headings carry too little evidence for a language label.""" from pipeline.translation import script_detect, service from pipeline.translation.base import TranslationConfig @@ -223,14 +221,14 @@ async def test_gate_skips_pyfranc_for_english_pages(self, monkeypatch): ] def explode(*args, **kwargs): - raise AssertionError("pyfranc must not be called for Latin-script pages") + raise AssertionError("pyfranc must not be called for short headings") monkeypatch.setattr(script_detect.franc, "lang_detect", explode) config = TranslationConfig(provider="gemma_vllm", model="gemma-4") detected = await service.detect_page_languages(pages, config=config) - assert detected == {0: "en", 1: "en"} + assert detected == {0: "und", 1: "und"} @pytest.mark.unit @pytest.mark.asyncio @@ -250,11 +248,11 @@ def log(msg, *args): config = TranslationConfig(provider="gemma_vllm", model="gemma-4") detected = await service.detect_page_languages(pages, log=log, config=config) - assert detected == {0: "en", 1: "gu"} + assert detected == {0: "und", 1: "gu"} joined = "\n".join(messages) - assert "Page 1: regex" in joined and "SKIP translation" in joined + assert "Page 1: language=und" in joined assert "Page 2: regex" in joined and "TRANSLATE" in joined - assert "1/2 page(s) need translation" in joined + assert "1/2 page(s) identified by script" in joined class TestScriptDetectDisambiguation: diff --git a/tests/test_upload_batch.py b/tests/test_upload_batch.py new file mode 100644 index 0000000..2bf4b0e --- /dev/null +++ b/tests/test_upload_batch.py @@ -0,0 +1,252 @@ +import pytest + + +@pytest.mark.api +@pytest.mark.db +def test_upload_pins_only_an_enabled_parser( + test_client, db_connection, mock_temporal_client, sample_pdf_content, monkeypatch, +): + monkeypatch.setenv("OCR_ALLOWED_PROVIDERS", "mock,liteparse") + for parser, expected_status in (("mistral", 422), ("liteparse", 200)): + response = test_client.post( + "/upload", params={"kind": "advisory"}, data={"parser_provider": parser}, + files={"file": ("parser-choice.pdf", sample_pdf_content, "application/pdf")}, + ) + assert response.status_code == expected_status + + config = db_connection.get_latest_document_job(response.json()["workflow_id"])["config"] + assert config["parser_provider"] == "liteparse" + assert config["upload_start_args"][-2:] == [None, "liteparse"] + assert mock_temporal_client.start_workflow.call_count == 1 + + +@pytest.mark.api +@pytest.mark.db +def test_batch_starts_independent_workflows_and_reports_bad_pdf( + test_client, db_connection, mock_temporal_client, sample_pdf_content, +): + observed = [] + + async def start_workflow(*args, **kwargs): + workflow_id = kwargs["id"] + assert db_connection.get_document(workflow_id) is not None + assert db_connection.get_latest_document_job(workflow_id) is not None + observed.append(workflow_id) + + mock_temporal_client.start_workflow.side_effect = start_workflow + response = test_client.post( + "/upload/batch", + params={"kind": "advisory", "instance": "bv"}, + files=[ + ("files", ("rice.pdf", sample_pdf_content, "application/pdf")), + ("files", ("broken.pdf", b"not a PDF", "application/pdf")), + ("files", ("cotton.pdf", sample_pdf_content, "application/pdf")), + ], + ) + assert response.status_code == 207 + items = response.json()["items"] + assert [item["status"] for item in items] == [202, 400, 202] + ids = [item["document"]["workflow_id"] for item in items if item["document"]] + assert ids == observed + assert len(set(ids)) == 2 + assert all(db_connection.get_document(id_)["instance"] == "bv" for id_ in ids) + + +@pytest.mark.api +@pytest.mark.db +def test_index_only_upload_uses_deployment_collection_without_review_or_release( + test_client, db_connection, sample_pdf_content, monkeypatch, +): + monkeypatch.setenv("KP_INDEX_ONLY", "true") + monkeypatch.setenv("VECTOR_DB_COLLECTION_NAME", "all-documents") + monkeypatch.setenv("ENABLE_PUBLISH_TO_NETWORK", "true") + monkeypatch.setenv("AUTO_RELEASE_NETWORK_ID", "unconfigured-network") + + response = test_client.post( + "/upload", params={"kind": "advisory", "index_name": "ignored"}, + files={"file": ("index-only.pdf", sample_pdf_content, "application/pdf")}, + ) + assert response.status_code == 200, response.text + config = db_connection.get_latest_document_job(response.json()["workflow_id"])["config"] + assert config["upload_start_args"][6:] == ["all-documents", True, False, True, None, None, True] + assert not config.get("release_network_id") + + +@pytest.mark.api +@pytest.mark.db +def test_automatic_release_uses_server_destination( + test_client, db_connection, mock_temporal_client, sample_pdf_content, monkeypatch, +): + monkeypatch.setenv("ENABLE_PUBLISH_TO_NETWORK", "true") + monkeypatch.setenv("DISCOVERY_NETWORK_ID", "oan-dev.local") + monkeypatch.setenv("AUTO_RELEASE_NETWORK_ID", "oan-dev.local") + monkeypatch.setenv("AUTO_RELEASE_COLLECTION", "published-advisories") + + response = test_client.post( + "/upload", params={"kind": "advisory", "index_name": "caller-selected", "auto_approve": "false"}, + files={"file": ("policy.pdf", sample_pdf_content, "application/pdf")}, + ) + + assert response.status_code == 200 + workflow_id = response.json()["workflow_id"] + config = db_connection.get_latest_document_job(workflow_id)["config"] + args = mock_temporal_client.start_workflow.call_args.kwargs["args"] + assert args == config["upload_start_args"] + assert args[6:8] == ["published-advisories", True] + assert args[-1] == config["release_network_id"] == "oan-dev.local" + db_connection.update_document_stage(workflow_id, "completed") + assert test_client.post(f"/documents/{workflow_id}/approve-prod").status_code == 409 + assert mock_temporal_client.start_workflow.call_count == 1 + + +@pytest.mark.api +@pytest.mark.db +def test_oversized_file_count_does_not_start_partial_batch( + test_client, mock_temporal_client, sample_pdf_content, +): + response = test_client.post( + "/upload/batch", + params={"kind": "advisory"}, + files=[ + ("files", (f"part-{i}.pdf", sample_pdf_content, "application/pdf")) + for i in range(11) + ], + ) + assert response.status_code == 413 + mock_temporal_client.start_workflow.assert_not_called() + + +@pytest.mark.api +@pytest.mark.db +def test_retry_uncertain_start_uses_the_original_document_and_workflow( + test_client, db_connection, mock_temporal_client, sample_pdf_content, +): + from temporalio.exceptions import WorkflowAlreadyStartedError + + mock_temporal_client.start_workflow.side_effect = TimeoutError("start reply lost") + upload = test_client.post( + "/upload/batch", + params={"kind": "advisory", "instance": "bv"}, + files=[("files", ("test.pdf", sample_pdf_content, "application/pdf"))], + ) + item, = upload.json()["items"] + assert item["status"] == 503 + workflow_id = item["document"]["workflow_id"] + assert db_connection.get_latest_document_job(workflow_id)["status"] == "start_pending" + + mock_temporal_client.start_workflow.side_effect = WorkflowAlreadyStartedError( + workflow_id, "DocumentPipelineWorkflow", + ) + assert test_client.post(f"/upload/{workflow_id}/start").status_code == 202 + assert db_connection.get_latest_document_job(workflow_id)["status"] == "running" + calls = mock_temporal_client.start_workflow.call_args_list + assert [call.kwargs["id"] for call in calls] == [workflow_id, workflow_id] + assert calls[0].kwargs["args"] == calls[1].kwargs["args"] + assert test_client.post(f"/upload/{workflow_id}/start").status_code == 202 + assert mock_temporal_client.start_workflow.call_count == 2 + + +@pytest.mark.api +@pytest.mark.db +def test_failed_registration_rolls_back_database_rows( + test_client, db_connection, mock_temporal_client, sample_pdf_content, monkeypatch, +): + def failed_artifact(*args, **kwargs): + raise RuntimeError("artifact registration failed") + + monkeypatch.setattr(db_connection, "add_document_artifact", failed_artifact) + response = test_client.post( + "/upload/batch", + params={"kind": "advisory", "instance": "bv"}, + files=[("files", ("rolled-back.pdf", sample_pdf_content, "application/pdf"))], + ) + assert response.json()["items"][0]["status"] == 500 + with db_connection.get_connection() as conn: + assert conn.execute("SELECT count(*) AS n FROM documents").fetchone()["n"] == 0 + assert conn.execute("SELECT count(*) AS n FROM workflows").fetchone()["n"] == 0 + mock_temporal_client.start_workflow.assert_not_called() + + +@pytest.mark.db +def test_batch_accepts_supported_mixed_file_formats(test_client, sample_pdf_content): + response = test_client.post( + "/upload/batch?kind=advisory", + files=[ + ("files", ("notice.pdf", sample_pdf_content, "application/pdf")), + ("files", ("measurements.csv", b"label,value\nreading,20\n", "text/csv")), + ], + ) + assert response.status_code == 207, response.text + assert [item["status"] for item in response.json()["items"]] == [202, 202] + + +@pytest.mark.db +@pytest.mark.asyncio +async def test_canonical_json_upload_preserves_submitted_text_and_translation( + test_client, db_connection, tmp_path, monkeypatch, +): + import json + + from pipeline.temporal.activities import source as activities + + body = { + "external_id": "notice-8", "title": "Field notice", "kind": "advisory", + "text": "Keep the entire notice.", "language": "en", + "translations": { + "hi": {"text": "पूरी सूचना रखें।", "title": "सूचना", "description": "पूरा विवरण"}, + "fr": "Conservez l’avis en entier.", + }, + "knowledge_metadata": {"subject_categories": ["Crop"]}, + } + path = tmp_path / "notice.json" + path.write_text(json.dumps(body)) + response = test_client.post( + "/upload?kind=advisory", files={"file": (path.name, path.read_bytes(), "application/json")}, + ) + assert response.status_code == 200, response.text + workflow_id = response.json()["workflow_id"] + monkeypatch.setattr(activities, "_resolve_local_path", lambda _: (str(path), False)) + monkeypatch.setattr(activities, "_upload_artifact", lambda *args: ( + "blob://docs/normalized.txt", 1, "text/plain", + )) + await activities.run_ocr_and_store(workflow_id, db_connection.get_document(workflow_id)["filepath"]) + page, = db_connection.get_pages(workflow_id) + assert page["original_markdown"] == body["text"] + assert page["translations"]["hi"]["translated_markdown"] == body["translations"]["hi"]["text"] + assert page["translations"]["fr"]["translated_markdown"] == body["translations"]["fr"] + doc = db_connection.get_document(workflow_id) + assert doc["name"] == body["title"] + assert doc["knowledge_metadata"] == { + **body["knowledge_metadata"], "localized": {"hi": {"title": "सूचना", "description": "पूरा विवरण"}}, + } + assert doc["canonical_input_type"] == "structured_text" + + +@pytest.mark.db +def test_uploaded_html_download_is_not_rendered_inline(test_client, db_connection): + response = test_client.post( + "/upload?kind=advisory", + files={"file": ("notice.html", b"

Notice

", "text/html")}, + ) + assert response.status_code == 200, response.text + workflow_id = response.json()["workflow_id"] + artifact = next(a for a in db_connection.list_document_artifacts(workflow_id) if a["artifact_type"] == "original_upload") + download = test_client.get(f"/documents/{workflow_id}/artifacts/{artifact['id']}/content") + assert download.status_code == 200 + assert download.headers["content-disposition"].startswith("attachment") + assert download.headers["content-type"] == "application/octet-stream" + assert download.headers["x-content-type-options"] == "nosniff" + + +@pytest.mark.api +@pytest.mark.db +def test_scheme_matches_accepts_existing_code_and_alias(test_client, db_connection): + db_connection.upsert_document( + workflow_id="scheme-match", document_id="c" * 32, filename="guide.pdf", filepath="minio://documents/guide.pdf", + instance="bv", kind="scheme", code="example-code", name="Assistance guide", + ) + db_connection.update_document_fields("scheme-match", aliases=["Different label"]) + for name in ("example-code", "Different label"): + response = test_client.get("/upload/scheme-matches", params={"instance": "bv", "name": name}) + assert response.status_code == 200, response.text + assert [row["code"] for row in response.json()["matches"]] == ["example-code"] diff --git a/tests/test_webhook.py b/tests/test_webhook.py deleted file mode 100644 index 920e9f3..0000000 --- a/tests/test_webhook.py +++ /dev/null @@ -1,129 +0,0 @@ -"""Tests for POST /webhook (x-api-key protected).""" - -import os -from unittest.mock import patch - -import pytest - -TEST_KEY = "0123456789abcdef0123456789abcdef" - -SAMPLE = [ - { - "contentID": 101, - "title": "Bacterial Leaf Blight in Rice", - "domainName": "Agriculture", - "status": "Published", - "isPublished": 1, - "slug": "101/bacterial-leaf-blight-in-rice", - "createdAt": "2026-09-09T07:00:00.000Z", - "updatedAt": "2026-09-09T07:00:00.000Z", - "popCount": 1, - "popList": [{"popID": 1001, "contentID": 101, "keywords": ["rice"]}], - } -] - -OK_BODY = {"success": True, "message": "Webhook processed successfully.", "recordCount": 1} - - -@pytest.fixture -def test_client(db_connection): - """TestClient without lifespan (no Temporal/MinIO needed for /webhook).""" - from fastapi.testclient import TestClient - - from pipeline.api import api - - return TestClient(api.app) - - -@pytest.fixture -def key_env(): - with patch.dict(os.environ, {"WEBHOOK_API_KEY": TEST_KEY}): - yield - - -def _rows(db_connection): - from pipeline import db - - with db.get_connection() as conn: - return conn.execute("SELECT * FROM webhook_records").fetchall() - - -@pytest.mark.api -@pytest.mark.unit -class TestWebhookAuth: - def test_missing_header_401(self, test_client, db_connection, key_env): - assert test_client.post("/webhook", json=SAMPLE).status_code == 401 - - def test_empty_header_401(self, test_client, db_connection, key_env): - r = test_client.post("/webhook", json=SAMPLE, headers={"x-api-key": ""}) - assert r.status_code == 401 - - def test_wrong_key_401(self, test_client, db_connection, key_env): - r = test_client.post("/webhook", json=SAMPLE, headers={"x-api-key": "nope"}) - assert r.status_code == 401 - - @pytest.mark.parametrize("value", [None, ""]) - def test_unconfigured_key_503_even_if_auth_disabled(self, test_client, db_connection, value): - env = {"AUTH_DISABLED": "true"} - if value is not None: - env["WEBHOOK_API_KEY"] = value - with patch.dict(os.environ, env): - if value is None: - os.environ.pop("WEBHOOK_API_KEY", None) - r = test_client.post("/webhook", json=SAMPLE, headers={"x-api-key": "anything"}) - assert r.status_code == 503 - assert "WEBHOOK_API_KEY" in r.json()["detail"] - - -@pytest.mark.api -@pytest.mark.unit -class TestWebhookIngest: - H = {"x-api-key": TEST_KEY} - - def test_valid_payload(self, test_client, db_connection, key_env): - r = test_client.post("/webhook", json=SAMPLE, headers=self.H) - assert r.status_code == 200 - assert r.json() == OK_BODY - rows = _rows(db_connection) - assert len(rows) == 1 - assert rows[0]["external_id"] == "101" - assert rows[0]["id"].version == 4 - assert rows[0]["category"] == "Agriculture" - assert "popList" in rows[0]["payload"] - - def test_repost_is_idempotent(self, test_client, db_connection, key_env): - test_client.post("/webhook", json=SAMPLE, headers=self.H) - first_id = _rows(db_connection)[0]["id"] - changed = [{**SAMPLE[0], "title": "New title"}] - r = test_client.post("/webhook", json=changed, headers=self.H) - assert r.json()["recordCount"] == 1 - rows = _rows(db_connection) - assert len(rows) == 1 and rows[0]["title"] == "New title" - assert rows[0]["id"] == first_id - - def test_two_records(self, test_client, db_connection, key_env): - body = SAMPLE + [{**SAMPLE[0], "contentID": 102}] - r = test_client.post("/webhook", json=body, headers=self.H) - assert r.json()["recordCount"] == 2 - assert len(_rows(db_connection)) == 2 - - def test_empty_array(self, test_client, db_connection, key_env): - r = test_client.post("/webhook", json=[], headers=self.H) - assert r.status_code == 200 and r.json()["recordCount"] == 0 - - def test_non_array_422(self, test_client, db_connection, key_env): - r = test_client.post("/webhook", json={"contentID": 1}, headers=self.H) - assert r.status_code == 422 - - def test_non_object_item_422(self, test_client, db_connection, key_env): - r = test_client.post("/webhook", json=[1, 2], headers=self.H) - assert r.status_code == 422 - - def test_too_many_records_413(self, test_client, db_connection, key_env): - from pipeline.api import api - - with patch.object(api, "WEBHOOK_MAX_RECORDS", 1): - body = SAMPLE + [{**SAMPLE[0], "contentID": 102}] - r = test_client.post("/webhook", json=body, headers=self.H) - assert r.status_code == 413 - assert _rows(db_connection) == [] diff --git a/tests/test_webhook_index_db.py b/tests/test_webhook_index_db.py deleted file mode 100644 index 0c09b1a..0000000 --- a/tests/test_webhook_index_db.py +++ /dev/null @@ -1,107 +0,0 @@ -"""Tests for the webhook_records indexing state (indexed_hash / promoted_hash) helpers.""" - - -import pytest - - -def _rec(content_id, title="Wheat Insect Management", published=1, status="Published"): - return { - "contentID": content_id, - "title": title, - "status": status, - "isPublished": published, - "domainName": "Agriculture", - "slug": f"{content_id}/x", - "popList": [{"popID": 1, "status": "Published"}], - } - - -def _row_id(db, content_id): - with db.get_connection() as conn: - return str(conn.execute( - "SELECT id FROM webhook_records WHERE external_id = %s", (str(content_id),) - ).fetchone()["id"]) - - -@pytest.mark.unit -@pytest.mark.db -class TestIndexState: - def test_new_rows_are_unindexed_until_marked(self, db_connection): - db = db_connection - db.upsert_webhook_records("webhook", [_rec(1)]) - rid = _row_id(db, 1) - assert db.get_unindexed_webhook_record_ids() == [rid] - row = db.get_webhook_record(rid) - db.mark_webhook_record_indexed(rid, row["payload_hash"]) - assert db.get_unindexed_webhook_record_ids() == [] - - def test_changed_payload_becomes_unindexed_again(self, db_connection): - db = db_connection - db.upsert_webhook_records("webhook", [_rec(1)]) - rid = _row_id(db, 1) - db.mark_webhook_record_indexed(rid, db.get_webhook_record(rid)["payload_hash"]) - - db.upsert_webhook_records("webhook", [_rec(1)]) # identical - assert db.get_unindexed_webhook_record_ids() == [] - - db.upsert_webhook_records("webhook", [_rec(1, title="Changed")]) - assert db.get_unindexed_webhook_record_ids() == [rid] - - def test_marking_a_stale_hash_leaves_the_row_unindexed(self, db_connection): - db = db_connection - db.upsert_webhook_records("webhook", [_rec(1)]) - rid = _row_id(db, 1) - stale = db.get_webhook_record(rid)["payload_hash"] - db.upsert_webhook_records("webhook", [_rec(1, title="Changed")]) - db.mark_webhook_record_indexed(rid, stale) - assert db.get_unindexed_webhook_record_ids() == [rid] - - def test_unindexed_ids_can_be_scoped_to_external_ids(self, db_connection): - db = db_connection - db.upsert_webhook_records("webhook", [_rec(1), _rec(2), _rec(3)]) - ids = db.get_unindexed_webhook_record_ids(external_ids=["1", "3"]) - assert sorted(ids) == sorted([_row_id(db, 1), _row_id(db, 3)]) - assert db.get_unindexed_webhook_record_ids(external_ids=[]) == [] - - def test_get_webhook_record_parses_payload(self, db_connection): - db = db_connection - db.upsert_webhook_records("webhook", [_rec(101)]) - row = db.get_webhook_record(_row_id(db, 101)) - assert row["payload"]["contentID"] == 101 - assert row["external_id"] == "101" - assert row["is_published"] == 1 - assert row["status"] == "Published" - assert db.get_webhook_record("missing") is None - - def test_webhook_external_id_matches_upsert_rule(self, db_connection): - db = db_connection - assert db.webhook_external_id({"contentID": 7}) == "7" - assert db.webhook_external_id({"id": "abc"}) == "abc" - assert db.webhook_external_id({"slug": "s"}) == "s" - assert len(db.webhook_external_id({"title": "x"})) == 64 # payload hash - - def test_record_id_lists(self, db_connection): - db = db_connection - db.upsert_webhook_records( - "webhook", [_rec(1), _rec(2, published=0, status="Draft")] - ) - assert len(db.get_webhook_record_ids()) == 2 - assert db.get_webhook_record_ids(published_only=True) == [_row_id(db, 1)] - - def test_status_counts(self, db_connection): - db = db_connection - db.upsert_webhook_records( - "webhook", [_rec(1), _rec(2), _rec(3, published=0, status="Draft")] - ) - a, b = _row_id(db, 1), _row_id(db, 2) - db.mark_webhook_record_indexed(a, db.get_webhook_record(a)["payload_hash"]) - db.mark_webhook_record_promoted(a, db.get_webhook_record(a)["payload_hash"]) - counts = db.advisory_status_counts() - assert counts == { - "total": 3, - "published": 2, - "indexed": 1, - "unindexed": 2, - "promoted": 1, - } - assert b # silence unused diff --git a/tests/test_webhook_sources.py b/tests/test_webhook_sources.py new file mode 100644 index 0000000..f387efa --- /dev/null +++ b/tests/test_webhook_sources.py @@ -0,0 +1,285 @@ +import asyncio +import copy +import json + +import pytest + +from pipeline.api import app +from pipeline.api.routers import webhook_sources +from pipeline.auth.deps import get_current_user +from pipeline.auth.models import AuthUser +from pipeline.intake.source_intake import ( + NormalizedSourceDocument, + materialize_intake_pages, + reconcile_pending_sources, +) +from pipeline.intake.webhook_mapping import WebhookMapping, map_webhook +from tests.icar_webhook_fixture import bulk_payload + +MAPPING = { + "records": "/popList", + "each_root": True, + "fields": { + "external_id": "/popID", "title": {"scope": "root", "path": "/title"}, + "text": "/advisory", "language": "/language", "translations": "/translations", + "crop_subjects": {"scope": "root", "path": "/crops"}, + "knowledge_metadata": {"scope": "root", "path": "/metadata"}, + }, + "filters": [ + {"scope": "root", "path": "/isPublished", "equals": 1}, + {"path": "/status", "equals": "Published"}, + ], +} + +PAYLOAD = { + "title": "Sorghum field note", "newTitle": "Revised sorghum field note", "isPublished": 1, + "crops": [{"subjectId": "https://example.test/taxonomy/sorghum"}], + "metadata": {"subject_categories": ["Crop"], "seasons": ["Rabi"]}, + "popList": [ + {"popID": 987, "status": "Published", "advisory": "જુવારને પાણી આપો.", + "language": "gu", "translations": {"en": "Water sorghum."}}, + {"popID": 900, "status": "Draft", "advisory": "Private draft."}, + ], +} + + +def test_mapping_normalizes_one_published_nested_entry(): + _, advisory = map_webhook(PAYLOAD, WebhookMapping.model_validate(MAPPING))[0] + + assert advisory.external_id == "987" + assert advisory.title == PAYLOAD["title"] + assert advisory.language == "gu" + assert advisory.translations == {"en": "Water sorghum."} + assert advisory.model_dump(mode="json")["translations"] == {"en": "Water sorghum."} + assert advisory.crop_subjects[0].subjectId == PAYLOAD["crops"][0]["subjectId"] + assert advisory.knowledge_metadata.storage_value() == PAYLOAD["metadata"] + assert map_webhook({**PAYLOAD, "isPublished": 0}, WebhookMapping.model_validate(MAPPING)) == [] + assert map_webhook([PAYLOAD], WebhookMapping.model_validate(MAPPING))[0][1] == advisory + + +@pytest.mark.parametrize("translations", [ + {"fr": "Bonjour", "FR": {"text": "Bonjour"}}, + {"en": {"text": "The source language"}}, + {"fr": {"text": " ", "title": "Titre"}}, + {"fr": {"text": "Bonjour", "unsupported": "field"}}, + {"fr": {"text": "Bonjour", "title": "Conflicting title"}}, +]) +def test_structured_translations_reject_invalid_or_conflicting_values(translations): + with pytest.raises(ValueError): + NormalizedSourceDocument.model_validate({ + "external_id": "item", "title": "Title", "text": "Original", "language": "en", + "translations": translations, + "knowledge_metadata": {"localized": {"fr": {"title": "Titre"}}}, + }) + + +def test_outer_array_maps_all_published_legacy_fixture_entries(): + payload = bulk_payload() + withheld_parent = {**payload, "isPublished": 0, "popList": payload["popList"][:1]} + documents = map_webhook([payload, withheld_parent], WebhookMapping.model_validate(MAPPING)) + + assert len(documents) == 205 + assert {document.external_id for _, document in documents} == {str(10_000 + i) for i in range(205)} + assert all(document.title == payload["title"] for _, document in documents) + + +@pytest.mark.api +@pytest.mark.db +def test_webhook_policy_receipts_replay_revision_and_revoke( + test_client, db_connection, mock_temporal_client, mock_object_store, monkeypatch, +): + db_connection.upsert_instance("/states/MH") + monkeypatch.setenv("KEYCLOAK_VERIFY_LIVE_GROUPS", "true") + monkeypatch.setenv("KEYCLOAK_ISSUER", "https://issuer.example.test/realms/demo") + monkeypatch.setenv("KEYCLOAK_AUDIENCE", "knowledge-intake") + monkeypatch.setenv("ENABLE_PUBLISH_TO_NETWORK", "true") + monkeypatch.setenv("DISCOVERY_NETWORK_ID", "oan-dev.local") + monkeypatch.setenv("AUTO_RELEASE_NETWORK_ID", "oan-dev.local") + monkeypatch.setenv("AUTO_RELEASE_COLLECTION", "published-advisories") + for key, value in { + "NETWORK_CONNECTIONS": "dev,prod", + "NETWORK_CONNECTION_DEFAULT": "dev", + "NETWORK_CONNECTION_DEV_ID": "oan-dev.local", + "NETWORK_CONNECTION_DEV_TYPE": "beckn", + "NETWORK_CONNECTION_DEV_SENDER_ID": "publisher.oan-dev.local", + "NETWORK_CONNECTION_DEV_DISCOVERY_URL": "https://discovery-dev.example.test", + "NETWORK_CONNECTION_DEV_RELEASE_COLLECTION": "published-advisories", + "NETWORK_CONNECTION_PROD_ID": "oan-prod.local", + "NETWORK_CONNECTION_PROD_TYPE": "beckn", + "NETWORK_CONNECTION_PROD_SENDER_ID": "publisher.oan-prod.local", + "NETWORK_CONNECTION_PROD_DISCOVERY_URL": "https://discovery-prod.example.test", + "NETWORK_CONNECTION_PROD_RELEASE_COLLECTION": "prod-advisories", + }.items(): + monkeypatch.setenv(key, value) + monkeypatch.setattr(webhook_sources._api.limiter, "enabled", False) + monkeypatch.setattr(webhook_sources.keycloak_admin, "find_webhook_service_subject", lambda _: "fixture-sender") + monkeypatch.setattr(webhook_sources, "decode_verified_claims", lambda token, config: { + "typ": "Bearer", "azp": "fixture-client", "sub": token, + }) + monkeypatch.setitem(app.dependency_overrides, get_current_user, lambda: AuthUser( + user_id="fixture-admin", roles=["state_admin"], instances=["mh"], state_roles={"mh": "state_admin"}, + )) + base = "/webhook/sources/mh/field-feed" + + def put(mapping, *, version=0, enabled=True, network=None, auto_release=True): + return test_client.put(base, json={ + "expected_version": version, "mapping": mapping, "fixture": PAYLOAD, + "service_client": "fixture-client", "service_subject": "fixture-sender", + "auto_release": auto_release, "enabled": enabled, "release_network_id": network, + }) + + def send(key, body=PAYLOAD, sender="fixture-sender"): + return test_client.post(f"{base}/events", json=body, headers={ + "Authorization": f"Bearer {sender}", "Idempotency-Key": key, + }) + + assert put(MAPPING, network="unknown.test").status_code == 422 + disabled_release = put(MAPPING, network="oan-prod.local", auto_release=False) + assert disabled_release.status_code == 422 + assert disabled_release.json() == {"detail": "A release network requires automatic release"} + policy = put(MAPPING) + assert policy.status_code == 200, policy.text + assert policy.json()["version"] == 1 + assert send("no-access", sender="wrong-subject").status_code == 403 + assert test_client.post(f"{base}/events", json=PAYLOAD, headers={"Idempotency-Key": "no-token"}).status_code == 401 + + preview = test_client.post("/knowledge/intake/mh/mapping-preview", json={"mapping": MAPPING, "fixture": PAYLOAD}) + assert preview.status_code == 200, preview.text + assert [row["external_id"] for row in preview.json()["documents"]] == ["987"] + mock_object_store.put.assert_not_called() + + mock_temporal_client.start_workflow.side_effect = TimeoutError("start reply lost") + original = (json.dumps(PAYLOAD, ensure_ascii=False, indent=2) + "\n").encode() + headers = {"Content-Type": "application/json", "Authorization": "Bearer fixture-sender"} + mock_object_store.put.side_effect = OSError("temporary storage failure") + assert test_client.post(f"{base}/events", content=original, headers={ + **headers, "Idempotency-Key": "first-event", + }).status_code == 503 + mock_temporal_client.start_workflow.assert_not_awaited() + mock_object_store.put.reset_mock(side_effect=True) + first = test_client.post(f"{base}/events", content=original, headers={ + **headers, "Idempotency-Key": "first-event", + }) + assert first.status_code == 202, first.text + mock_object_store.put.assert_called_once() + ref, body, size, mime_type = mock_object_store.put.call_args.args + assert (body.getvalue(), size, mime_type) == (original, len(original), "application/json") + assert ref.key.startswith("webhook-deliveries/") + deliveries = "/knowledge/intake/mh/field-feed/deliveries" + assert test_client.get(deliveries).json()["deliveries"][0]["has_original"] is True + mock_object_store.open.return_value = iter((original,)) + downloaded = test_client.get(f"{deliveries}/original", params={"delivery_key": "first-event"}) + assert downloaded.content == original + assert downloaded.headers["cache-control"] == "no-store" + assert test_client.get( + "/knowledge/intake/ka/field-feed/deliveries/original", params={"delivery_key": "first-event"}, + ).status_code == 403 + item, = first.json()["items"] + assert item["status"] == "start_pending" + assert db_connection.get_document(item["document_id"])["name"] == PAYLOAD["title"] + assert db_connection.get_document(item["document_id"])["knowledge_metadata"] == PAYLOAD["metadata"] + job = db_connection.get_latest_document_job(item["document_id"]) + assert job["config"]["release_network_id"] == "oan-dev.local" + args = job["config"]["upload_start_args"] + assert args[6:] == ["published-advisories", True, False, True, "oan-dev.local"] + materialize_intake_pages(item["workflow_id"], args[2]) + page, = db_connection.get_pages(item["document_id"]) + assert page["original_markdown"] == PAYLOAD["popList"][0]["advisory"] + assert page["translations"]["en"]["translated_markdown"] == "Water sorghum." + + revised_mapping = copy.deepcopy(MAPPING) + revised_mapping["fields"]["title"]["path"] = "/newTitle" + second_policy = put(revised_mapping, version=1, network="oan-prod.local") + assert second_policy.status_code == 200, second_policy.text + assert second_policy.json()["release_network_id"] == "oan-prod.local" + assert second_policy.json()["mapping_id"] != first.json()["mapping_version_id"] + + with db_connection.get_connection() as conn: + conn.execute("UPDATE source_intake_receipts SET created_at = now() - interval '6 seconds' WHERE id = %s::uuid", (item["receipt_id"],)) + queue = job["config"]["source_task_queue"] + assert asyncio.run(reconcile_pending_sources(mock_temporal_client, queue)) == 0 + assert db_connection.get_document_job(job["id"])["config"]["source_dispatch_attempts"] == 1 + mock_temporal_client.start_workflow.side_effect = None + assert asyncio.run(reconcile_pending_sources(mock_temporal_client, queue)) == 0 + with db_connection.get_connection() as conn: + conn.execute( + "UPDATE workflows SET config = config - 'source_next_dispatch_at' WHERE id = %s::uuid", (job["id"],), + ) + assert asyncio.run(reconcile_pending_sources(mock_temporal_client, queue)) == 1 + replay = send("first-event") + assert replay.status_code == 202 + assert replay.json()["mapping_version_id"] == first.json()["mapping_version_id"] + mock_object_store.put.assert_called_once() + assert replay.json()["items"] == [{**item, "status": "accepted"}] + assert db_connection.get_document_job(job["id"])["config"]["release_network_id"] == "oan-dev.local" + assert mock_temporal_client.start_workflow.call_args.kwargs["id"] == item["workflow_id"] + assert send("first-event", {**PAYLOAD, "title": "conflict"}).status_code == 409 + assert send("running-revision").status_code == 409 + + db_connection.update_document_job(job["id"], status="completed", current_stage="completed") + with monkeypatch.context() as unavailable: + unavailable.setenv("NETWORK_CONNECTIONS", "dev") + assert send("no-destination").status_code == 503 + new_event = send("new-event", {**PAYLOAD, "release_network_id": "oan-dev.local"}) + assert new_event.status_code == 202, new_event.text + revision, = new_event.json()["items"] + assert revision["document_id"] == item["document_id"] + assert revision["workflow_id"] != item["workflow_id"] + assert db_connection.get_document(item["document_id"])["name"] == PAYLOAD["newTitle"] + assert new_event.json()["mapping_version_id"] == second_policy.json()["mapping_id"] + latest = db_connection.get_latest_document_job(item["document_id"])["config"] + assert latest["release_network_id"] == "oan-prod.local" + assert latest["release_collection"] == "prod-advisories" + assert latest["explicit_release_network"] + + assert put(revised_mapping, version=2, enabled=False).status_code == 200 + assert send("first-event").status_code == 403 + + +@pytest.mark.db +def test_named_intake_template_is_pinned_and_canonical_submission_preserves_text( + test_client, db_connection, monkeypatch, tmp_path, +): + import json + + db_connection.upsert_instance("/states/MH") + monkeypatch.setattr(webhook_sources.keycloak_admin, "find_webhook_service_subject", lambda _: "fixture-sender") + monkeypatch.setitem(app.dependency_overrides, webhook_sources.require_webhook_sender, lambda: ( + "fixture-client", "fixture-sender", + )) + monkeypatch.setitem(app.dependency_overrides, get_current_user, lambda: AuthUser( + user_id="fixture-admin", roles=["state_admin"], instances=["mh"], state_roles={"mh": "state_admin"}, + )) + monkeypatch.setenv("KEYCLOAK_VERIFY_LIVE_GROUPS", "true") + template = tmp_path / "intake.json" + template.write_text(json.dumps({"canonical": {"runtime": "canonical-v1"}})) + monkeypatch.setenv("KNOWLEDGE_INTAKE_TEMPLATES_FILE", str(template)) + submission = { + "external_id": "note-1", "title": "Field note", "text": "First paragraph.\n\nSecond paragraph.", + "kind": "advisory", "language": "en", + "translations": {"fr-CA": {"text": "Premier paragraphe.", "title": "Note", "description": "Résumé"}}, + "knowledge_metadata": {"localized": {"fr-ca": {"title": "Note"}}, "tags": ["public"]}, + } + base = "/knowledge/intake/mh/field-feed" + policy = test_client.put(base, json={ + "template": "canonical", "fixture": submission, "service_client": "fixture-client", + }) + assert policy.status_code == 200, policy.text + template.write_text("{}") + response = test_client.post(f"{base}/submissions", json=submission, headers={"Idempotency-Key": "note"}) + assert response.status_code == 202, response.text + item, = response.json()["items"] + job = db_connection.get_latest_document_job(item["document_id"]) + materialize_intake_pages(item["workflow_id"], job["config"]["upload_start_args"][2]) + page, = db_connection.get_pages(item["document_id"]) + assert page["original_markdown"] == submission["text"] + assert page["translations"]["fr-ca"]["translated_markdown"] == "Premier paragraphe." + doc = db_connection.get_document(item["document_id"]) + assert doc["knowledge_metadata"] == { + "localized": {"fr-ca": {"title": "Note", "description": "Résumé"}}, "tags": ["public"], + } + with db_connection.get_connection() as conn: + receipt = conn.execute("SELECT raw_payload, normalized_document FROM source_intake_receipts").fetchone() + assert receipt["raw_payload"] == submission + assert receipt["normalized_document"]["translations"] == {"fr-ca": submission["translations"]["fr-CA"]} + assert test_client.get("/knowledge/intake/schema").json()["anyOf"] diff --git a/tests/test_workflows.py b/tests/test_workflows.py index d56c608..5d3fd17 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -1,19 +1,19 @@ -"""Tests for pipeline/workflows.py module-level configuration.""" +"""Temporal workflow execution and replay checks.""" import uuid import pytest from temporalio import activity from temporalio.testing import WorkflowEnvironment -from temporalio.worker import UnsandboxedWorkflowRunner, Worker +from temporalio.worker import Replayer, UnsandboxedWorkflowRunner, Worker -from pipeline import workflows as workflows_module -from pipeline.workflows import ( +from pipeline.temporal.workflows import document, policies, releases +from pipeline.temporal.workflows import ( DocumentPipelineWorkflow, PromoteToProdWorkflow, ReingestionWorkflow, - _network_publish_retry_policy, ) +from pipeline.temporal.workflows.policies import _network_publish_retry_policy class TestNetworkPublishRetryPolicy: @@ -49,9 +49,9 @@ def test_reads_overrides_from_env(self, monkeypatch): assert policy.maximum_attempts == 3 -def _stub_activities(call_log: list): +def _stub_activities(call_log: list, source_first_flags: dict | None = None): """Fake activities recording call order; registered on the test Worker and - monkeypatched over the real ones referenced by pipeline.workflows.""" + monkeypatched over the real activities referenced by the workflow.""" @activity.defn(name="run_ocr_and_store") async def run_ocr_and_store(workflow_id: str, filepath: str) -> dict: @@ -65,9 +65,11 @@ async def detect_and_translate_pages_from_db(workflow_id: str) -> dict: @activity.defn(name="create_chunks_from_db") async def create_chunks_from_db( - workflow_id: str, chunk_size: int, chunk_overlap: int, min_tokens: int + workflow_id: str, chunk_size: int, chunk_overlap: int, min_tokens: int, source_first: bool = False, ) -> dict: call_log.append("create_chunks_from_db") + if source_first_flags is not None: + source_first_flags["chunking"] = source_first return {"chunk_count": 1} @activity.defn(name="auto_tag_chunks_from_db") @@ -77,9 +79,13 @@ async def auto_tag_chunks_from_db(workflow_id: str, filename: str = "") -> dict: @activity.defn(name="ingest_document_from_db") async def ingest_document_from_db( - workflow_id: str, document_id: str, filename: str, index_name: str = "documents-index" + workflow_id: str, document_id: str, filename: str, index_name: str = "documents-index", + batch_size: int = 10, release_network_id: str | None = None, source_first: bool = False, + freeze_draft_revision: bool = False, revision_workflow_id: str | None = None, ) -> dict: call_log.append("ingest_document_from_db") + if source_first_flags is not None: + source_first_flags["indexing"] = source_first return {"records_ingested": 1} @activity.defn(name="promote_document_to_prod_qdrant") @@ -115,6 +121,13 @@ async def update_document_state( } +def _install_stubs(monkeypatch, workflow_module, stubs): + monkeypatch.setattr(policies, "update_document_state", stubs["update_document_state"]) + for name, fn in stubs.items(): + if hasattr(workflow_module, name): + monkeypatch.setattr(workflow_module, name, fn) + + def _activity_calls(call_log: list) -> list: """call_log entries that aren't state-mirroring noise.""" return [c for c in call_log if not c.startswith("state:")] @@ -129,8 +142,7 @@ class TestPublishAfterProdOrdering: async def test_document_pipeline_publishes_after_prod_promotion(self, monkeypatch): call_log: list = [] stubs = _stub_activities(call_log) - for name, fn in stubs.items(): - monkeypatch.setattr(workflows_module, name, fn) + _install_stubs(monkeypatch, document, stubs) task_queue = f"test-queue-{uuid.uuid4()}" async with await WorkflowEnvironment.start_time_skipping() as env: @@ -157,39 +169,58 @@ async def test_document_pipeline_publishes_after_prod_promotion(self, monkeypatc ) @pytest.mark.asyncio - async def test_document_pipeline_skip_prod_never_publishes(self, monkeypatch): + @pytest.mark.parametrize("filename,filepath,source_first,legacy", [ + ("test.pdf", "/tmp/test.pdf", False, False), + ("notice.json", "/tmp/notice.json", True, False), + ("source.txt", "intake://test", True, False), + ("notice.json", "/tmp/notice.json", False, True), + ]) + async def test_document_pipeline_skip_prod_never_publishes(self, monkeypatch, filename, filepath, source_first, legacy): call_log: list = [] - stubs = _stub_activities(call_log) - for name, fn in stubs.items(): - monkeypatch.setattr(workflows_module, name, fn) + source_first_flags = {} + stubs = _stub_activities(call_log, source_first_flags) + _install_stubs(monkeypatch, document, stubs) task_queue = f"test-queue-{uuid.uuid4()}" - async with await WorkflowEnvironment.start_time_skipping() as env: - async with Worker( - env.client, - task_queue=task_queue, - workflows=[DocumentPipelineWorkflow], - activities=list(stubs.values()), - workflow_runner=UnsandboxedWorkflowRunner(), - ): - handle = await env.client.start_workflow( - DocumentPipelineWorkflow.run, - args=["doc-1", "test.pdf", "/tmp/test.pdf", 450, 128, 100, "documents-index", True, False, True], - id=f"test-wf-{uuid.uuid4()}", - task_queue=task_queue, + with monkeypatch.context() as execution: + if legacy: + patched = document.workflow.patched + execution.setattr( + document.workflow, "patched", + lambda name: False if name == "source-first-structured-draft-v1" else patched(name), ) - await handle.result() + async with await WorkflowEnvironment.start_time_skipping() as env: + async with Worker( + env.client, + task_queue=task_queue, + workflows=[DocumentPipelineWorkflow], + activities=list(stubs.values()), + workflow_runner=UnsandboxedWorkflowRunner(), + ): + handle = await env.client.start_workflow( + DocumentPipelineWorkflow.run, + args=["doc-1", filename, filepath, 450, 128, 100, "documents-index", True, False, True], + id=f"test-wf-{uuid.uuid4()}", + task_queue=task_queue, + ) + await handle.result() + history = await handle.fetch_history() + + await Replayer( + workflows=[DocumentPipelineWorkflow], workflow_runner=UnsandboxedWorkflowRunner(), + ).replay_workflow(history) activities_called = _activity_calls(call_log) assert "publish_catalog_to_network" not in activities_called assert "promote_document_to_prod_qdrant" not in activities_called + assert source_first_flags == {"chunking": source_first, "indexing": source_first} @pytest.mark.asyncio - async def test_reingestion_workflow_does_not_publish(self, monkeypatch): + @pytest.mark.parametrize("index_only", [False, True]) + async def test_reingestion_workflow_does_not_publish(self, monkeypatch, index_only): call_log: list = [] stubs = _stub_activities(call_log) - for name, fn in stubs.items(): - monkeypatch.setattr(workflows_module, name, fn) + _install_stubs(monkeypatch, releases, stubs) task_queue = f"test-queue-{uuid.uuid4()}" async with await WorkflowEnvironment.start_time_skipping() as env: @@ -202,7 +233,7 @@ async def test_reingestion_workflow_does_not_publish(self, monkeypatch): ): handle = await env.client.start_workflow( ReingestionWorkflow.run, - args=["doc-1", "test.pdf", "orig-wf-1", 1, 1, "documents-index"], + args=["doc-1", "test.pdf", "orig-wf-1", 1, 1, "documents-index", *([True] if index_only else [])], id=f"test-wf-{uuid.uuid4()}", task_queue=task_queue, ) @@ -211,14 +242,13 @@ async def test_reingestion_workflow_does_not_publish(self, monkeypatch): activities_called = _activity_calls(call_log) assert "ingest_document_from_db" in activities_called assert "publish_catalog_to_network" not in activities_called - assert result["stage"] == "approval_for_prod" + assert result["stage"] == ("completed" if index_only else "approval_for_prod") @pytest.mark.asyncio async def test_promote_to_prod_workflow_publishes_after_promotion(self, monkeypatch): call_log: list = [] stubs = _stub_activities(call_log) - for name, fn in stubs.items(): - monkeypatch.setattr(workflows_module, name, fn) + _install_stubs(monkeypatch, releases, stubs) task_queue = f"test-queue-{uuid.uuid4()}" async with await WorkflowEnvironment.start_time_skipping() as env: diff --git a/ui/package-lock.json b/ui/package-lock.json index 10c4aac..028ee8a 100644 --- a/ui/package-lock.json +++ b/ui/package-lock.json @@ -24,8 +24,8 @@ "react": "^18.2.0", "react-dom": "^18.2.0", "react-markdown": "^9.0.1", - "react-pdf": "^7.7.0", - "react-router-dom": "^6.20.0", + "react-pdf": "10.5.0", + "react-router-dom": "7.18.5", "remark-gfm": "^4.0.0", "tailwind-merge": "^3.5.0" }, @@ -851,24 +851,6 @@ "node": ">=12" } }, - "node_modules/@esbuild/netbsd-arm64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.28.2.tgz", - "integrity": "sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "netbsd" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, "node_modules/@esbuild/netbsd-x64": { "version": "0.21.5", "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.21.5.tgz", @@ -886,24 +868,6 @@ "node": ">=12" } }, - "node_modules/@esbuild/openbsd-arm64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.28.2.tgz", - "integrity": "sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "openbsd" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, "node_modules/@esbuild/openbsd-x64": { "version": "0.21.5", "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.21.5.tgz", @@ -921,24 +885,6 @@ "node": ">=12" } }, - "node_modules/@esbuild/openharmony-arm64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.28.2.tgz", - "integrity": "sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "openharmony" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, "node_modules/@esbuild/sunos-x64": { "version": "0.21.5", "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.21.5.tgz", @@ -1307,64 +1253,269 @@ "@jridgewell/sourcemap-codec": "^1.4.14" } }, - "node_modules/@mapbox/node-pre-gyp": { - "version": "1.0.11", - "resolved": "https://registry.npmjs.org/@mapbox/node-pre-gyp/-/node-pre-gyp-1.0.11.tgz", - "integrity": "sha512-Yhlar6v9WQgUp/He7BdgzOz8lqMQ8sU+jkCq7Wx8Myc5YFJLbEe7lgui/V7G1qB1DJykHSGwreceSaD60Y0PUQ==", - "license": "BSD-3-Clause", + "node_modules/@napi-rs/canvas": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas/-/canvas-0.1.100.tgz", + "integrity": "sha512-xglYA6q3XO5P3BNJYxVZ1IV7DLVjp1Py6nwag88YntrS+3vKHyYcMqXVS4ZztJmwz2uGvz1FWhI/4LgbR5uQDA==", + "license": "MIT", "optional": true, - "dependencies": { - "detect-libc": "^2.0.0", - "https-proxy-agent": "^5.0.0", - "make-dir": "^3.1.0", - "node-fetch": "^2.6.7", - "nopt": "^5.0.0", - "npmlog": "^5.0.1", - "rimraf": "^3.0.2", - "semver": "^7.3.5", - "tar": "^6.1.11" + "workspaces": [ + "e2e/*" + ], + "engines": { + "node": ">= 10" }, - "bin": { - "node-pre-gyp": "bin/node-pre-gyp" + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + }, + "optionalDependencies": { + "@napi-rs/canvas-android-arm64": "0.1.100", + "@napi-rs/canvas-darwin-arm64": "0.1.100", + "@napi-rs/canvas-darwin-x64": "0.1.100", + "@napi-rs/canvas-linux-arm-gnueabihf": "0.1.100", + "@napi-rs/canvas-linux-arm64-gnu": "0.1.100", + "@napi-rs/canvas-linux-arm64-musl": "0.1.100", + "@napi-rs/canvas-linux-riscv64-gnu": "0.1.100", + "@napi-rs/canvas-linux-x64-gnu": "0.1.100", + "@napi-rs/canvas-linux-x64-musl": "0.1.100", + "@napi-rs/canvas-win32-arm64-msvc": "0.1.100", + "@napi-rs/canvas-win32-x64-msvc": "0.1.100" + } + }, + "node_modules/@napi-rs/canvas-android-arm64": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-android-arm64/-/canvas-android-arm64-0.1.100.tgz", + "integrity": "sha512-hjhCKhntPv9+t4ckHymdx0phYNcVW+GKQR6Lzw2zE+pOVjOplSmtx9nNNknTjbEDLcuLZqA1y8ufKg1XfgftzQ==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" } }, - "node_modules/@mapbox/node-pre-gyp/node_modules/make-dir": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/make-dir/-/make-dir-3.1.0.tgz", - "integrity": "sha512-g3FeP20LNwhALb/6Cz6Dd4F2ngze0jz7tbzrD2wAV+o9FeNHe4rL+yK2md0J/fiSf1sa1ADhXqi5+oVwOM/eGw==", + "node_modules/@napi-rs/canvas-darwin-arm64": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-darwin-arm64/-/canvas-darwin-arm64-0.1.100.tgz", + "integrity": "sha512-2PcswRaC7Ly645DGt88///zuFDhJxJYdKAs1uU3mfk1atYkXufgcgLfBpk6Tm12nCQBaNt1wpybuPZ4qOhTo8A==", + "cpu": [ + "arm64" + ], "license": "MIT", "optional": true, - "dependencies": { - "semver": "^6.0.0" + "os": [ + "darwin" + ], + "engines": { + "node": ">= 10" }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-darwin-x64": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-darwin-x64/-/canvas-darwin-x64-0.1.100.tgz", + "integrity": "sha512-ePNZtj7pNIva/siZMg+HmbeozkIjqUIYdoymH8HaA3qK7LfzFN4WMBM8G6HQ9ZC+H3+Dnn5pqtiXpgLykaPOhw==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], "engines": { - "node": ">=8" + "node": ">= 10" }, "funding": { - "url": "https://github.com/sponsors/sindresorhus" + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" } }, - "node_modules/@mapbox/node-pre-gyp/node_modules/make-dir/node_modules/semver": { - "version": "6.3.1", - "resolved": "https://registry.npmjs.org/semver/-/semver-6.3.1.tgz", - "integrity": "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA==", - "license": "ISC", + "node_modules/@napi-rs/canvas-linux-arm-gnueabihf": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-arm-gnueabihf/-/canvas-linux-arm-gnueabihf-0.1.100.tgz", + "integrity": "sha512-d5cDB48oWFGU8/XPhUOFAlySgb/VAu7D+s8fi55K1Pcfg8aPplHWqMgibhVLU8ky7Pyg/fuiVLz4Nf3JrSTuUA==", + "cpu": [ + "arm" + ], + "license": "MIT", "optional": true, - "bin": { - "semver": "bin/semver.js" + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" } }, - "node_modules/@mapbox/node-pre-gyp/node_modules/semver": { - "version": "7.8.5", - "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", - "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==", - "license": "ISC", + "node_modules/@napi-rs/canvas-linux-arm64-gnu": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-arm64-gnu/-/canvas-linux-arm64-gnu-0.1.100.tgz", + "integrity": "sha512-rDxgxRu69RvDlX/bh9o22DxLsGr8EqsNgotL9+RwQE1S0b0cqeatqsw6aW45mukm0B42DIAaAacKaYQ8cqS1nw==", + "cpu": [ + "arm64" + ], + "libc": [ + "glibc" + ], + "license": "MIT", "optional": true, - "bin": { - "semver": "bin/semver.js" + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-arm64-musl": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-arm64-musl/-/canvas-linux-arm64-musl-0.1.100.tgz", + "integrity": "sha512-K3mDW66N+xT2/V439u1alFANiBUjdEx2gLiNYnCmUsva5jZMxWTjafBYwTzYK+EMFMHrUoabuU+T1BIP5CgbYQ==", + "cpu": [ + "arm64" + ], + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-riscv64-gnu": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-riscv64-gnu/-/canvas-linux-riscv64-gnu-0.1.100.tgz", + "integrity": "sha512-mooqUBTIsccZpnoQC4NgrC1v6C1vof39etLNMnBwCY+p0gajWJvAHLGQ6g/gGyS5YrpDW+GefSN4+Cvcr08UWw==", + "cpu": [ + "riscv64" + ], + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], "engines": { - "node": ">=10" + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-x64-gnu": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-x64-gnu/-/canvas-linux-x64-gnu-0.1.100.tgz", + "integrity": "sha512-1eCvkDCazm7FFhsT7DfGOdSaHgZVK3bt/dSBl5EWHOWmnz+I7j8tPseJqqD81NF+MH21jKUK4wQSDjN0mdhnTg==", + "cpu": [ + "x64" + ], + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-x64-musl": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-linux-x64-musl/-/canvas-linux-x64-musl-0.1.100.tgz", + "integrity": "sha512-20arT6lnI19S68qNlii73TSEDbECNgzMz2EpldC1V3mZFuRkeujXkcebRk0LRJe9SEUAooYiLokfMViY8IX7yA==", + "cpu": [ + "x64" + ], + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-win32-arm64-msvc": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-win32-arm64-msvc/-/canvas-win32-arm64-msvc-0.1.100.tgz", + "integrity": "sha512-DZFFT1wIAg37LJw37yhMRFfjATd3vTQzjZ1Yki8u2vhO6Hi5VE6BVaGQ1aaDu7xb4iMErz+9EOwjpS7xcxFeBw==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-win32-x64-msvc": { + "version": "0.1.100", + "resolved": "https://registry.npmjs.org/@napi-rs/canvas-win32-x64-msvc/-/canvas-win32-x64-msvc-0.1.100.tgz", + "integrity": "sha512-MyT1j3mHC2+Lu4pBi9mKyMJhtP6U7k7EldY7sj/uS5gJA65gTXt8MefJQXLJo5d/vZbuWmfxzkEUNc/urV3pHA==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" } }, "node_modules/@napi-rs/lzma-linux-x64-gnu": { @@ -2223,15 +2374,6 @@ "integrity": "sha512-JtyZR+mqgBibTo8xea3B6ZRmzZiM/YeVBtUkas6zMuXjAlfIFIW2FgqeM9eLyvEaYX66vr6DJMK+4U6LV0KhNw==", "license": "MIT" }, - "node_modules/@remix-run/router": { - "version": "1.23.4", - "resolved": "https://registry.npmjs.org/@remix-run/router/-/router-1.23.4.tgz", - "integrity": "sha512-q7j5geK7xs3UJSdm9/iytUNclBnLmYx1EnSeCFXHPeutdqgIMeFeHtUZgS3EhlKxdBEAu8OwtJCwmLrEzpSs7Q==", - "license": "MIT", - "engines": { - "node": ">=14.0.0" - } - }, "node_modules/@rolldown/binding-android-arm-eabi": { "version": "1.2.7", "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm-eabi/-/binding-android-arm-eabi-1.2.7.tgz", @@ -3522,13 +3664,6 @@ "url": "https://opencollective.com/vitest" } }, - "node_modules/abbrev": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/abbrev/-/abbrev-1.1.1.tgz", - "integrity": "sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q==", - "license": "ISC", - "optional": true - }, "node_modules/acorn": { "version": "8.18.0", "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.18.0.tgz", @@ -3552,19 +3687,6 @@ "acorn": "^6.0.0 || ^7.0.0 || ^8.0.0" } }, - "node_modules/agent-base": { - "version": "6.0.2", - "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-6.0.2.tgz", - "integrity": "sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==", - "license": "MIT", - "optional": true, - "dependencies": { - "debug": "4" - }, - "engines": { - "node": ">= 6.0.0" - } - }, "node_modules/ajv": { "version": "6.15.0", "resolved": "https://registry.npmjs.org/ajv/-/ajv-6.15.0.tgz", @@ -3586,8 +3708,9 @@ "version": "5.0.1", "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "devOptional": true, + "dev": true, "license": "MIT", + "peer": true, "engines": { "node": ">=8" } @@ -3606,28 +3729,6 @@ "url": "https://github.com/chalk/ansi-styles?sponsor=1" } }, - "node_modules/aproba": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/aproba/-/aproba-2.1.0.tgz", - "integrity": "sha512-tLIEcj5GuR2RSTnxNKdkK0dJ/GrC7P38sUkiDmDuHfsHmbagTFAxDVIBltoklXEVIQ/f14IL8IMJ5pn9Hez1Ew==", - "license": "ISC", - "optional": true - }, - "node_modules/are-we-there-yet": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/are-we-there-yet/-/are-we-there-yet-2.0.0.tgz", - "integrity": "sha512-Ci/qENmwHnsYo9xKIcUJN5LeDKdJ6R1Z1j9V/J5wyq8nh/mYPEpIKJbBZXtZjG04HiK7zV/p6Vs9952MrMeUIw==", - "deprecated": "This package is no longer supported.", - "license": "ISC", - "optional": true, - "dependencies": { - "delegates": "^1.0.0", - "readable-stream": "^3.6.0" - }, - "engines": { - "node": ">=10" - } - }, "node_modules/aria-hidden": { "version": "1.2.6", "resolved": "https://registry.npmjs.org/aria-hidden/-/aria-hidden-1.2.6.tgz", @@ -3699,33 +3800,10 @@ "node": "18 || 20 || >=22" } }, - "node_modules/base64-js": { - "version": "1.5.1", - "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", - "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==", - "dev": true, - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "optional": true, - "peer": true - }, - "node_modules/baseline-browser-mapping": { - "version": "2.11.20", - "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.20.tgz", - "integrity": "sha512-H0ulySigv6icDJ1F7SjtdCD6PrhTpdYCmP0CactWy1+ekh0AFd0o1Wn5T8b+hnTmdBx19u9yhL6wvCylXMY7zw==", + "node_modules/baseline-browser-mapping": { + "version": "2.11.20", + "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.20.tgz", + "integrity": "sha512-H0ulySigv6icDJ1F7SjtdCD6PrhTpdYCmP0CactWy1+ekh0AFd0o1Wn5T8b+hnTmdBx19u9yhL6wvCylXMY7zw==", "dev": true, "license": "Apache-2.0", "bin": { @@ -3745,20 +3823,6 @@ "require-from-string": "^2.0.2" } }, - "node_modules/bl": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz", - "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "buffer": "^5.5.0", - "inherits": "^2.0.4", - "readable-stream": "^3.4.0" - } - }, "node_modules/brace-expansion": { "version": "5.0.9", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz", @@ -3806,33 +3870,6 @@ "node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7" } }, - "node_modules/buffer": { - "version": "5.7.1", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz", - "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==", - "dev": true, - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.1.13" - } - }, "node_modules/caniuse-lite": { "version": "1.0.30001810", "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001810.tgz", @@ -3854,23 +3891,6 @@ ], "license": "CC-BY-4.0" }, - "node_modules/canvas": { - "version": "3.2.3", - "resolved": "https://registry.npmjs.org/canvas/-/canvas-3.2.3.tgz", - "integrity": "sha512-PzE5nJZPz72YUAfo8oTp0u3fqqY7IzlTubneAihqDYAUcBk7ryeCmBbdJBEdaH0bptSOe2VT2Zwcb3UaFyaSWw==", - "dev": true, - "hasInstallScript": true, - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "node-addon-api": "^7.0.0", - "prebuild-install": "^7.1.3" - }, - "engines": { - "node": "^18.12.0 || >= 20.9.0" - } - }, "node_modules/ccount": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/ccount/-/ccount-2.0.1.tgz", @@ -3931,15 +3951,6 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/chownr": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", - "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", - "dev": true, - "license": "ISC", - "optional": true, - "peer": true - }, "node_modules/class-variance-authority": { "version": "0.7.1", "resolved": "https://registry.npmjs.org/class-variance-authority/-/class-variance-authority-0.7.1.tgz", @@ -3961,16 +3972,6 @@ "node": ">=6" } }, - "node_modules/color-support": { - "version": "1.1.3", - "resolved": "https://registry.npmjs.org/color-support/-/color-support-1.1.3.tgz", - "integrity": "sha512-qiBjkpbMLO/HL68y+lh4q0/O1MZFj2RX6X/KmMa3+gJD3z+WwI1ZzDHysvqHGS3mP6mznPckpXmw1nI9cJjyRg==", - "license": "ISC", - "optional": true, - "bin": { - "color-support": "bin.js" - } - }, "node_modules/comma-separated-tokens": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.3.tgz", @@ -3981,20 +3982,6 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/concat-map": { - "version": "0.0.1", - "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", - "integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==", - "license": "MIT", - "optional": true - }, - "node_modules/console-control-strings": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/console-control-strings/-/console-control-strings-1.1.0.tgz", - "integrity": "sha512-ty/fTekppD2fIwRvnZAVdeOiGd1c7YXEixbgJTNzqcxJWKQnjJ/V1bNEEE6hygpM3WjwHFUVK6HTjWSzV4a8sQ==", - "license": "ISC", - "optional": true - }, "node_modules/convert-source-map": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", @@ -4002,6 +3989,19 @@ "dev": true, "license": "MIT" }, + "node_modules/cookie": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/cookie/-/cookie-1.1.1.tgz", + "integrity": "sha512-ei8Aos7ja0weRpFzJnEA9UHJ/7XQmqglbRwnf2ATjcB9Wq874VKH9kfjjirM6UhU2/E5fFYadylyhFldcqSidQ==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, "node_modules/cross-spawn": { "version": "7.0.6", "resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz", @@ -4095,36 +4095,6 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/decompress-response": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz", - "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "mimic-response": "^3.1.0" - }, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/deep-extend": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz", - "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "engines": { - "node": ">=4.0.0" - } - }, "node_modules/deep-is": { "version": "0.1.4", "resolved": "https://registry.npmjs.org/deep-is/-/deep-is-0.1.4.tgz", @@ -4132,13 +4102,6 @@ "dev": true, "license": "MIT" }, - "node_modules/delegates": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/delegates/-/delegates-1.0.0.tgz", - "integrity": "sha512-bd2L678uiWATM6m5Z1VzNCErI3jiGzt6HGY8OVICs40JQq/HALfbyNJmp0UDakEY4pMMaN0Ly5om/B1VI/+xfQ==", - "license": "MIT", - "optional": true - }, "node_modules/dequal": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz", @@ -4152,7 +4115,7 @@ "version": "2.1.2", "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", - "devOptional": true, + "dev": true, "license": "Apache-2.0", "engines": { "node": ">=8" @@ -4192,25 +4155,6 @@ "dev": true, "license": "ISC" }, - "node_modules/emoji-regex": { - "version": "8.0.0", - "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", - "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", - "license": "MIT", - "optional": true - }, - "node_modules/end-of-stream": { - "version": "1.4.5", - "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz", - "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "once": "^1.4.0" - } - }, "node_modules/enhanced-resolve": { "version": "5.24.5", "resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-5.24.5.tgz", @@ -4512,18 +4456,6 @@ "node": ">=0.10.0" } }, - "node_modules/expand-template": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz", - "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==", - "dev": true, - "license": "(MIT OR WTFPL)", - "optional": true, - "peer": true, - "engines": { - "node": ">=6" - } - }, "node_modules/expect-type": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz", @@ -4630,55 +4562,6 @@ "dev": true, "license": "ISC" }, - "node_modules/fs-constants": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz", - "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true - }, - "node_modules/fs-minipass": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/fs-minipass/-/fs-minipass-2.1.0.tgz", - "integrity": "sha512-V/JgOLFCS+R6Vcq0slCuaeWEdNC3ouDlJMNIsacH2VtALiu9mV4LPrHc5cDl8k5aw6J8jwgWWpiTo5RYhmIzvg==", - "license": "ISC", - "optional": true, - "dependencies": { - "minipass": "^3.0.0" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/fs-minipass/node_modules/minipass": { - "version": "3.3.6", - "resolved": "https://registry.npmjs.org/minipass/-/minipass-3.3.6.tgz", - "integrity": "sha512-DxiNidxSEK+tHG6zOIklvNOwm3hvCrbUrdtzY74U6HKTJxvIDfOUL5W5P2Ghd3DTkhhKPYGqeNUIh5qcM4YBfw==", - "license": "ISC", - "optional": true, - "dependencies": { - "yallist": "^4.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/fs-minipass/node_modules/yallist": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz", - "integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==", - "license": "ISC", - "optional": true - }, - "node_modules/fs.realpath": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz", - "integrity": "sha512-OO0pH2lK6a0hZnAdau5ItzHPI6pUlvI7jMVnxUQRtw4owF2wk8lOSabtGDCTP4Ggrg2MbGnWO9X8K1t4+fGMDw==", - "license": "ISC", - "optional": true - }, "node_modules/fsevents": { "version": "2.3.3", "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", @@ -4694,28 +4577,6 @@ "node": "^8.16.0 || ^10.6.0 || >=11.0.0" } }, - "node_modules/gauge": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/gauge/-/gauge-3.0.2.tgz", - "integrity": "sha512-+5J6MS/5XksCuXq++uFRsnUd7Ovu1XenbeuIuNRJxYWjgQbPuFhT14lAvsWfqfAmnwluf1OwMjz39HjfLPci0Q==", - "deprecated": "This package is no longer supported.", - "license": "ISC", - "optional": true, - "dependencies": { - "aproba": "^1.0.3 || ^2.0.0", - "color-support": "^1.1.2", - "console-control-strings": "^1.0.0", - "has-unicode": "^2.0.1", - "object-assign": "^4.1.1", - "signal-exit": "^3.0.0", - "string-width": "^4.2.3", - "strip-ansi": "^6.0.1", - "wide-align": "^1.1.2" - }, - "engines": { - "node": ">=10" - } - }, "node_modules/gensync": { "version": "1.0.0-beta.2", "resolved": "https://registry.npmjs.org/gensync/-/gensync-1.0.0-beta.2.tgz", @@ -4735,37 +4596,6 @@ "node": ">=6" } }, - "node_modules/github-from-package": { - "version": "0.0.0", - "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz", - "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true - }, - "node_modules/glob": { - "version": "7.2.3", - "resolved": "https://registry.npmjs.org/glob/-/glob-7.2.3.tgz", - "integrity": "sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q==", - "deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me", - "license": "ISC", - "optional": true, - "dependencies": { - "fs.realpath": "^1.0.0", - "inflight": "^1.0.4", - "inherits": "2", - "minimatch": "^3.1.1", - "once": "^1.3.0", - "path-is-absolute": "^1.0.0" - }, - "engines": { - "node": "*" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/glob-parent": { "version": "6.0.2", "resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-6.0.2.tgz", @@ -4779,37 +4609,6 @@ "node": ">=10.13.0" } }, - "node_modules/glob/node_modules/balanced-match": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", - "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", - "license": "MIT", - "optional": true - }, - "node_modules/glob/node_modules/brace-expansion": { - "version": "1.1.18", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz", - "integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==", - "license": "MIT", - "optional": true, - "dependencies": { - "balanced-match": "^1.0.0", - "concat-map": "0.0.1" - } - }, - "node_modules/glob/node_modules/minimatch": { - "version": "3.1.5", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.5.tgz", - "integrity": "sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==", - "license": "ISC", - "optional": true, - "dependencies": { - "brace-expansion": "^1.1.7" - }, - "engines": { - "node": "*" - } - }, "node_modules/globals": { "version": "17.12.0", "resolved": "https://registry.npmjs.org/globals/-/globals-17.12.0.tgz", @@ -4840,13 +4639,6 @@ "node": ">=8" } }, - "node_modules/has-unicode": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/has-unicode/-/has-unicode-2.0.1.tgz", - "integrity": "sha512-8Rf9Y83NBReMnx0gFzA8JImQACstCYWUplepDa9xprwwtmgEZUF0h/i5xSA625zB/I37EtrswSST6OXxwaaIJQ==", - "license": "ISC", - "optional": true - }, "node_modules/hast-util-to-jsx-runtime": { "version": "2.3.6", "resolved": "https://registry.npmjs.org/hast-util-to-jsx-runtime/-/hast-util-to-jsx-runtime-2.3.6.tgz", @@ -4934,43 +4726,6 @@ "url": "https://opencollective.com/unified" } }, - "node_modules/https-proxy-agent": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz", - "integrity": "sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA==", - "license": "MIT", - "optional": true, - "dependencies": { - "agent-base": "6", - "debug": "4" - }, - "engines": { - "node": ">= 6" - } - }, - "node_modules/ieee754": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz", - "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==", - "dev": true, - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "BSD-3-Clause", - "optional": true, - "peer": true - }, "node_modules/ignore": { "version": "5.3.2", "resolved": "https://registry.npmjs.org/ignore/-/ignore-5.3.2.tgz", @@ -5001,34 +4756,6 @@ "node": ">=8" } }, - "node_modules/inflight": { - "version": "1.0.6", - "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz", - "integrity": "sha512-k92I/b08q4wvFscXCLvqfsHCrjrF7yiXsQuIVvVE7N82W3+aqpzuUdBbfhWcy/FZR3/4IgflMgKLOsvPDrGCJA==", - "deprecated": "This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.", - "license": "ISC", - "optional": true, - "dependencies": { - "once": "^1.3.0", - "wrappy": "1" - } - }, - "node_modules/inherits": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", - "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", - "license": "ISC", - "optional": true - }, - "node_modules/ini": { - "version": "1.3.8", - "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz", - "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", - "dev": true, - "license": "ISC", - "optional": true, - "peer": true - }, "node_modules/inline-style-parser": { "version": "0.2.7", "resolved": "https://registry.npmjs.org/inline-style-parser/-/inline-style-parser-0.2.7.tgz", @@ -5079,16 +4806,6 @@ "node": ">=0.10.0" } }, - "node_modules/is-fullwidth-code-point": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", - "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">=8" - } - }, "node_modules/is-glob": { "version": "4.0.3", "resolved": "https://registry.npmjs.org/is-glob/-/is-glob-4.0.3.tgz", @@ -5667,9 +5384,9 @@ } }, "node_modules/make-cancellable-promise": { - "version": "1.3.2", - "resolved": "https://registry.npmjs.org/make-cancellable-promise/-/make-cancellable-promise-1.3.2.tgz", - "integrity": "sha512-GCXh3bq/WuMbS+Ky4JBPW1hYTOU+znU+Q5m9Pu+pI8EoUqIHk9+tviOKC6/qhHh8C4/As3tzJ69IF32kdz85ww==", + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/make-cancellable-promise/-/make-cancellable-promise-2.0.0.tgz", + "integrity": "sha512-3SEQqTpV9oqVsIWqAcmDuaNeo7yBO3tqPtqGRcKkEo0lrzD3wqbKG9mkxO65KoOgXqj+zH2phJ2LiAsdzlogSw==", "license": "MIT", "funding": { "url": "https://github.com/wojtekmaj/make-cancellable-promise?sponsor=1" @@ -5705,9 +5422,9 @@ } }, "node_modules/make-event-props": { - "version": "1.6.2", - "resolved": "https://registry.npmjs.org/make-event-props/-/make-event-props-1.6.2.tgz", - "integrity": "sha512-iDwf7mA03WPiR8QxvcVHmVWEPfMY1RZXerDVNCRYW7dUr2ppH3J58Rwb39/WG39yTZdRSxr3x+2v22tvI0VEvA==", + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/make-event-props/-/make-event-props-2.0.0.tgz", + "integrity": "sha512-G/hncXrl4Qt7mauJEXSg3AcdYzmpkIITTNl5I+rH9sog5Yw0kK6vseJjCaPfOXqOqQuPUP89Rkhfz5kPS8ijtw==", "license": "MIT", "funding": { "url": "https://github.com/wojtekmaj/make-event-props?sponsor=1" @@ -6013,9 +5730,9 @@ "license": "CC0-1.0" }, "node_modules/merge-refs": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/merge-refs/-/merge-refs-1.3.0.tgz", - "integrity": "sha512-nqXPXbso+1dcKDpPCXvwZyJILz+vSLqGGOnDrYHQYE+B8n9JTCekVLC65AfCpR4ggVyA/45Y0iR9LDyS2iI+zA==", + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/merge-refs/-/merge-refs-2.0.0.tgz", + "integrity": "sha512-3+B21mYK2IqUWnd2EivABLT7ueDhb0b8/dGK8LoFQPrU61YITeCMn14F7y7qZafWNZhUEKb24cJdiT5Wxs3prg==", "license": "MIT", "funding": { "url": "https://github.com/wojtekmaj/merge-refs?sponsor=1" @@ -6592,21 +6309,6 @@ ], "license": "MIT" }, - "node_modules/mimic-response": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz", - "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "node_modules/min-indent": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/min-indent/-/min-indent-1.0.1.tgz", @@ -6633,96 +6335,11 @@ "url": "https://github.com/sponsors/isaacs" } }, - "node_modules/minimist": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz", - "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/minipass": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/minipass/-/minipass-5.0.0.tgz", - "integrity": "sha512-3FnjYuehv9k6ovOEbyOswadCDPX1piCfhV8ncmYtHOjuPwylVWsghTLo7rabjC3Rx5xD4HDx8Wm1xnMF7S5qFQ==", - "license": "ISC", - "optional": true, - "engines": { - "node": ">=8" - } - }, - "node_modules/minizlib": { - "version": "2.1.2", - "resolved": "https://registry.npmjs.org/minizlib/-/minizlib-2.1.2.tgz", - "integrity": "sha512-bAxsR8BVfj60DWXHE3u30oHzfl4G7khkSuPW+qvpd7jFRHm7dLxOjUk1EHACJ/hxLY8phGJ0YhYHZo7jil7Qdg==", - "license": "MIT", - "optional": true, - "dependencies": { - "minipass": "^3.0.0", - "yallist": "^4.0.0" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/minizlib/node_modules/minipass": { - "version": "3.3.6", - "resolved": "https://registry.npmjs.org/minipass/-/minipass-3.3.6.tgz", - "integrity": "sha512-DxiNidxSEK+tHG6zOIklvNOwm3hvCrbUrdtzY74U6HKTJxvIDfOUL5W5P2Ghd3DTkhhKPYGqeNUIh5qcM4YBfw==", - "license": "ISC", - "optional": true, - "dependencies": { - "yallist": "^4.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/minizlib/node_modules/yallist": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz", - "integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==", - "license": "ISC", - "optional": true - }, - "node_modules/mkdirp": { - "version": "1.0.4", - "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-1.0.4.tgz", - "integrity": "sha512-vVqVZQyf3WLx2Shd0qJ9xuvqgAyKPLAiqITEtqW0oIUjzo3PePDd6fW9iFz30ef7Ysp/oiWqbhszeGWW2T6Gzw==", - "license": "MIT", - "optional": true, - "bin": { - "mkdirp": "bin/cmd.js" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/mkdirp-classic": { - "version": "0.5.3", - "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz", - "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true - }, - "node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, - "node_modules/nan": { - "version": "2.28.0", - "resolved": "https://registry.npmjs.org/nan/-/nan-2.28.0.tgz", - "integrity": "sha512-fTsDz99OTq2sVePhGdp4qQhggZFtKr64ZNVyVajRKtMOkJxYekplBh577PiJB12v/D3s2E5cGtOI45LWp6rnLQ==", - "license": "MIT", - "optional": true + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" }, "node_modules/nanoid": { "version": "3.3.18", @@ -6743,15 +6360,6 @@ "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" } }, - "node_modules/napi-build-utils": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz", - "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true - }, "node_modules/natural-compare": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/natural-compare/-/natural-compare-1.4.0.tgz", @@ -6759,91 +6367,6 @@ "dev": true, "license": "MIT" }, - "node_modules/node-abi": { - "version": "3.96.0", - "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.96.0.tgz", - "integrity": "sha512-rebQ/lz7i0EkoLzUVSrKRzA69zMkwLp95kKMWoMDkkM00Suxz0D7zEQPwRml5fQum24mj7bPvmlgLAmu2JCiYg==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "semver": "^7.3.5" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/node-abi/node_modules/semver": { - "version": "7.8.5", - "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", - "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==", - "dev": true, - "license": "ISC", - "optional": true, - "peer": true, - "bin": { - "semver": "bin/semver.js" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/node-addon-api": { - "version": "7.1.1", - "resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-7.1.1.tgz", - "integrity": "sha512-5m3bsyrjFWE1xf7nz7YXdN4udnVtXK6/Yfgn5qnahL6bCkf2yKt4k3nuTKAtT4r3IG8JNR2ncsIMdZuAzJjHQQ==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true - }, - "node_modules/node-fetch": { - "version": "2.7.0", - "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz", - "integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==", - "license": "MIT", - "optional": true, - "dependencies": { - "whatwg-url": "^5.0.0" - }, - "engines": { - "node": "4.x || >=6.0.0" - }, - "peerDependencies": { - "encoding": "^0.1.0" - }, - "peerDependenciesMeta": { - "encoding": { - "optional": true - } - } - }, - "node_modules/node-fetch/node_modules/tr46": { - "version": "0.0.3", - "resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz", - "integrity": "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==", - "license": "MIT", - "optional": true - }, - "node_modules/node-fetch/node_modules/webidl-conversions": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz", - "integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==", - "license": "BSD-2-Clause", - "optional": true - }, - "node_modules/node-fetch/node_modules/whatwg-url": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz", - "integrity": "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==", - "license": "MIT", - "optional": true, - "dependencies": { - "tr46": "~0.0.3", - "webidl-conversions": "^3.0.0" - } - }, "node_modules/node-releases": { "version": "2.0.54", "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.54.tgz", @@ -6854,45 +6377,6 @@ "node": ">=18" } }, - "node_modules/nopt": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/nopt/-/nopt-5.0.0.tgz", - "integrity": "sha512-Tbj67rffqceeLpcRXrT7vKAN8CwfPeIBgM7E6iBkmKLV7bEMwpGgYLGv0jACUsECaa/vuxP0IjEont6umdMgtQ==", - "license": "ISC", - "optional": true, - "dependencies": { - "abbrev": "1" - }, - "bin": { - "nopt": "bin/nopt.js" - }, - "engines": { - "node": ">=6" - } - }, - "node_modules/npmlog": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/npmlog/-/npmlog-5.0.1.tgz", - "integrity": "sha512-AqZtDUWOMKs1G/8lwylVjrdYgqA4d9nu8hc+0gzRxlDb1I10+FHBGMXs6aiQHFdCUUlqH99MUMuLfzWDNDtfxw==", - "deprecated": "This package is no longer supported.", - "license": "ISC", - "optional": true, - "dependencies": { - "are-we-there-yet": "^2.0.0", - "console-control-strings": "^1.1.0", - "gauge": "^3.0.0", - "set-blocking": "^2.0.0" - } - }, - "node_modules/object-assign": { - "version": "4.1.1", - "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", - "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, "node_modules/obug": { "version": "2.1.4", "resolved": "https://registry.npmjs.org/obug/-/obug-2.1.4.tgz", @@ -6907,16 +6391,6 @@ "node": ">=12.20.0" } }, - "node_modules/once": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", - "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", - "license": "ISC", - "optional": true, - "dependencies": { - "wrappy": "1" - } - }, "node_modules/optionator": { "version": "0.9.4", "resolved": "https://registry.npmjs.org/optionator/-/optionator-0.9.4.tgz", @@ -7015,16 +6489,6 @@ "node": ">=8" } }, - "node_modules/path-is-absolute": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz", - "integrity": "sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">=0.10.0" - } - }, "node_modules/path-key": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", @@ -7035,16 +6499,6 @@ "node": ">=8" } }, - "node_modules/path2d-polyfill": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/path2d-polyfill/-/path2d-polyfill-2.0.1.tgz", - "integrity": "sha512-ad/3bsalbbWhmBo0D6FZ4RNMwsLsPpL6gnvhuSaU5Vm7b06Kr5ubSltQQ0T7YKsiJQO+g22zJ4dJKNTXIyOXtA==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">=8" - } - }, "node_modules/pathe": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", @@ -7053,70 +6507,15 @@ "license": "MIT" }, "node_modules/pdfjs-dist": { - "version": "3.11.174", - "resolved": "https://registry.npmjs.org/pdfjs-dist/-/pdfjs-dist-3.11.174.tgz", - "integrity": "sha512-TdTZPf1trZ8/UFu5Cx/GXB7GZM30LT+wWUNfsi6Bq8ePLnb+woNKtDymI2mxZYBpMbonNFqKmiz684DIfnd8dA==", + "version": "5.4.296", + "resolved": "https://registry.npmjs.org/pdfjs-dist/-/pdfjs-dist-5.4.296.tgz", + "integrity": "sha512-DlOzet0HO7OEnmUmB6wWGJrrdvbyJKftI1bhMitK7O2N8W2gc757yyYBbINy9IDafXAV9wmKr9t7xsTaNKRG5Q==", "license": "Apache-2.0", "engines": { - "node": ">=18" + "node": ">=20.16.0 || >=22.3.0" }, "optionalDependencies": { - "canvas": "^2.11.2", - "path2d-polyfill": "^2.0.1" - } - }, - "node_modules/pdfjs-dist/node_modules/canvas": { - "version": "2.11.2", - "resolved": "https://registry.npmjs.org/canvas/-/canvas-2.11.2.tgz", - "integrity": "sha512-ItanGBMrmRV7Py2Z+Xhs7cT+FNt5K0vPL4p9EZ/UX/Mu7hFbkxSjKF2KVtPwX7UYWp7dRKnrTvReflgrItJbdw==", - "hasInstallScript": true, - "license": "MIT", - "optional": true, - "dependencies": { - "@mapbox/node-pre-gyp": "^1.0.0", - "nan": "^2.17.0", - "simple-get": "^3.0.3" - }, - "engines": { - "node": ">=6" - } - }, - "node_modules/pdfjs-dist/node_modules/decompress-response": { - "version": "4.2.1", - "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-4.2.1.tgz", - "integrity": "sha512-jOSne2qbyE+/r8G1VU+G/82LBs2Fs4LAsTiLSHOCOMZQl2OKZ6i8i4IyHemTe+/yIXOtTcRQMzPcgyhoFlqPkw==", - "license": "MIT", - "optional": true, - "dependencies": { - "mimic-response": "^2.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/pdfjs-dist/node_modules/mimic-response": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-2.1.0.tgz", - "integrity": "sha512-wXqjST+SLt7R009ySCglWBCFpjUygmCIfD790/kVbiGmUgfYGuB14PiTd5DwVxSV4NcYHjzMkoj5LjQZwTQLEA==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">=8" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/pdfjs-dist/node_modules/simple-get": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-3.1.1.tgz", - "integrity": "sha512-CQ5LTKGfCpvE1K0n2us+kuMPbk/q0EKl82s4aheV9oXjFEz6W/Y7oQFVJuU6QG77hRT4Ghb5RURteF5vnWjupA==", - "license": "MIT", - "optional": true, - "dependencies": { - "decompress-response": "^4.2.0", - "once": "^1.3.1", - "simple-concat": "^1.0.0" + "@napi-rs/canvas": "^0.1.80" } }, "node_modules/picocolors": { @@ -7168,36 +6567,6 @@ "node": "^10 || ^12 || >=14" } }, - "node_modules/prebuild-install": { - "version": "7.1.3", - "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz", - "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==", - "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "detect-libc": "^2.0.0", - "expand-template": "^2.0.3", - "github-from-package": "0.0.0", - "minimist": "^1.2.3", - "mkdirp-classic": "^0.5.3", - "napi-build-utils": "^2.0.0", - "node-abi": "^3.3.0", - "pump": "^3.0.0", - "rc": "^1.2.7", - "simple-get": "^4.0.0", - "tar-fs": "^2.0.0", - "tunnel-agent": "^0.6.0" - }, - "bin": { - "prebuild-install": "bin.js" - }, - "engines": { - "node": ">=10" - } - }, "node_modules/prelude-ls": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/prelude-ls/-/prelude-ls-1.2.1.tgz", @@ -7232,17 +6601,6 @@ "license": "MIT", "peer": true }, - "node_modules/prop-types": { - "version": "15.8.1", - "resolved": "https://registry.npmjs.org/prop-types/-/prop-types-15.8.1.tgz", - "integrity": "sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg==", - "license": "MIT", - "dependencies": { - "loose-envify": "^1.4.0", - "object-assign": "^4.1.1", - "react-is": "^16.13.1" - } - }, "node_modules/property-information": { "version": "7.2.0", "resolved": "https://registry.npmjs.org/property-information/-/property-information-7.2.0.tgz", @@ -7253,19 +6611,6 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/pump": { - "version": "3.0.4", - "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz", - "integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "end-of-stream": "^1.1.0", - "once": "^1.3.1" - } - }, "node_modules/punycode": { "version": "2.3.1", "resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz", @@ -7276,24 +6621,6 @@ "node": ">=6" } }, - "node_modules/rc": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", - "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==", - "dev": true, - "license": "(BSD-2-Clause OR MIT OR Apache-2.0)", - "optional": true, - "peer": true, - "dependencies": { - "deep-extend": "^0.6.0", - "ini": "~1.3.0", - "minimist": "^1.2.0", - "strip-json-comments": "~2.0.1" - }, - "bin": { - "rc": "cli.js" - } - }, "node_modules/react": { "version": "18.3.1", "resolved": "https://registry.npmjs.org/react/-/react-18.3.1.tgz", @@ -7319,12 +6646,6 @@ "react": "^18.3.1" } }, - "node_modules/react-is": { - "version": "16.13.1", - "resolved": "https://registry.npmjs.org/react-is/-/react-is-16.13.1.tgz", - "integrity": "sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==", - "license": "MIT" - }, "node_modules/react-markdown": { "version": "9.1.0", "resolved": "https://registry.npmjs.org/react-markdown/-/react-markdown-9.1.0.tgz", @@ -7353,18 +6674,17 @@ } }, "node_modules/react-pdf": { - "version": "7.7.3", - "resolved": "https://registry.npmjs.org/react-pdf/-/react-pdf-7.7.3.tgz", - "integrity": "sha512-a2VfDl8hiGjugpqezBTUzJHYLNB7IS7a2t7GD52xMI9xHg8LdVaTMsnM9ZlNmKadnStT/tvX5IfV0yLn+JvYmw==", + "version": "10.5.0", + "resolved": "https://registry.npmjs.org/react-pdf/-/react-pdf-10.5.0.tgz", + "integrity": "sha512-zGKb6Ph5AYn2OfO4TS8Gg0Go3eW5K/TKy0+0FTm8EwFNbpgbuaWBZoJdYkJt+cRBCkizE9dYvqtMtHRj522Q4w==", "license": "MIT", "dependencies": { "clsx": "^2.0.0", "dequal": "^2.0.3", - "make-cancellable-promise": "^1.3.1", - "make-event-props": "^1.6.0", - "merge-refs": "^1.2.1", - "pdfjs-dist": "3.11.174", - "prop-types": "^15.6.2", + "make-cancellable-promise": "^2.0.0", + "make-event-props": "^2.0.0", + "merge-refs": "^2.0.0", + "pdfjs-dist": "5.4.296", "tiny-invariant": "^1.0.0", "warning": "^4.0.0" }, @@ -7372,9 +6692,9 @@ "url": "https://github.com/wojtekmaj/react-pdf?sponsor=1" }, "peerDependencies": { - "@types/react": "^16.8.0 || ^17.0.0 || ^18.0.0", - "react": "^16.8.0 || ^17.0.0 || ^18.0.0", - "react-dom": "^16.8.0 || ^17.0.0 || ^18.0.0" + "@types/react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", + "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", + "react-dom": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0" }, "peerDependenciesMeta": { "@types/react": { @@ -7440,35 +6760,41 @@ } }, "node_modules/react-router": { - "version": "6.30.6", - "resolved": "https://registry.npmjs.org/react-router/-/react-router-6.30.6.tgz", - "integrity": "sha512-5HfK7k5im7LTOB0EqCQmfvy4C13G92Ssj1VTmouTK3AJvyjKTnFuCV0vcMAD/JS+JC4DvDIBRrlAeJIFjh5VWg==", + "version": "7.18.5", + "resolved": "https://registry.npmjs.org/react-router/-/react-router-7.18.5.tgz", + "integrity": "sha512-dTgvcXDOrLPrlLHrcZAqWveiK1hMSqN+rGAJzegP8HtmNkfRxOoshGWBNX3ZJ0cZjOHkRdlUJtaIhZMLLyGyPw==", "license": "MIT", "dependencies": { - "@remix-run/router": "1.23.4" + "cookie": "^1.0.1", + "set-cookie-parser": "^2.6.0" }, "engines": { - "node": ">=14.0.0" + "node": ">=20.0.0" }, "peerDependencies": { - "react": ">=16.8" + "react": ">=18", + "react-dom": ">=18" + }, + "peerDependenciesMeta": { + "react-dom": { + "optional": true + } } }, "node_modules/react-router-dom": { - "version": "6.30.6", - "resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-6.30.6.tgz", - "integrity": "sha512-0RHKZz7wwffvkU+2MFVT2NnjK44ssLEV+m0CAJaS2Ksmorrwj7WxH00jO0SOCW26/tINUnJHToXblDs33I38YQ==", + "version": "7.18.5", + "resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-7.18.5.tgz", + "integrity": "sha512-9HcfDMU6JJ/fp6gJZYGwNE6RknVGS4kBjH1HkNw959mCJi70TVwdipQWv2FEol9uGpi+SZTFtVpJPtEQfayeOw==", "license": "MIT", "dependencies": { - "@remix-run/router": "1.23.4", - "react-router": "6.30.6" + "react-router": "7.18.5" }, "engines": { - "node": ">=14.0.0" + "node": ">=20.0.0" }, "peerDependencies": { - "react": ">=16.8", - "react-dom": ">=16.8" + "react": ">=18", + "react-dom": ">=18" } }, "node_modules/react-style-singleton": { @@ -7493,21 +6819,6 @@ } } }, - "node_modules/readable-stream": { - "version": "3.6.2", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", - "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", - "license": "MIT", - "optional": true, - "dependencies": { - "inherits": "^2.0.3", - "string_decoder": "^1.1.1", - "util-deprecate": "^1.0.1" - }, - "engines": { - "node": ">= 6" - } - }, "node_modules/redent": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/redent/-/redent-3.0.0.tgz", @@ -7598,23 +6909,6 @@ "node": ">=0.10.0" } }, - "node_modules/rimraf": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz", - "integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==", - "deprecated": "Rimraf versions prior to v4 are no longer supported", - "license": "ISC", - "optional": true, - "dependencies": { - "glob": "^7.1.3" - }, - "bin": { - "rimraf": "bin.js" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/rolldown": { "version": "1.2.7", "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.2.7.tgz", @@ -7702,27 +6996,6 @@ "fsevents": "~2.3.2" } }, - "node_modules/safe-buffer": { - "version": "5.2.1", - "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", - "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "optional": true - }, "node_modules/saxes": { "version": "6.0.0", "resolved": "https://registry.npmjs.org/saxes/-/saxes-6.0.0.tgz", @@ -7755,12 +7028,11 @@ "semver": "bin/semver.js" } }, - "node_modules/set-blocking": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/set-blocking/-/set-blocking-2.0.0.tgz", - "integrity": "sha512-KiKBS8AnWGEyLzofFfmvKwpdPzqiy16LvQfK3yv/fVH7Bj13/wl3JSR1J+rfgRE9q7xUJK4qvgS8raSOeLUehw==", - "license": "ISC", - "optional": true + "node_modules/set-cookie-parser": { + "version": "2.7.3", + "resolved": "https://registry.npmjs.org/set-cookie-parser/-/set-cookie-parser-2.7.3.tgz", + "integrity": "sha512-irZHtTqurLna57KQG/s+fzOwv/J6XSH4cs8M/vsrY3uCaQ3S6+QJguAH0WdN47SXlk82voMZ4naWMQf8fdEoIg==", + "license": "MIT" }, "node_modules/shebang-command": { "version": "2.0.0", @@ -7792,62 +7064,6 @@ "dev": true, "license": "ISC" }, - "node_modules/signal-exit": { - "version": "3.0.7", - "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.7.tgz", - "integrity": "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ==", - "license": "ISC", - "optional": true - }, - "node_modules/simple-concat": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz", - "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "optional": true - }, - "node_modules/simple-get": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz", - "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==", - "dev": true, - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "decompress-response": "^6.0.0", - "once": "^1.3.1", - "simple-concat": "^1.0.0" - } - }, "node_modules/source-map-js": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", @@ -7882,31 +7098,6 @@ "dev": true, "license": "MIT" }, - "node_modules/string_decoder": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", - "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==", - "license": "MIT", - "optional": true, - "dependencies": { - "safe-buffer": "~5.2.0" - } - }, - "node_modules/string-width": { - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", - "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", - "license": "MIT", - "optional": true, - "dependencies": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.1" - }, - "engines": { - "node": ">=8" - } - }, "node_modules/stringify-entities": { "version": "4.0.4", "resolved": "https://registry.npmjs.org/stringify-entities/-/stringify-entities-4.0.4.tgz", @@ -7921,19 +7112,6 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/strip-ansi": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", - "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", - "license": "MIT", - "optional": true, - "dependencies": { - "ansi-regex": "^5.0.1" - }, - "engines": { - "node": ">=8" - } - }, "node_modules/strip-indent": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/strip-indent/-/strip-indent-3.0.0.tgz", @@ -7947,18 +7125,6 @@ "node": ">=8" } }, - "node_modules/strip-json-comments": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", - "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "engines": { - "node": ">=0.10.0" - } - }, "node_modules/style-to-js": { "version": "1.1.21", "resolved": "https://registry.npmjs.org/style-to-js/-/style-to-js-1.1.21.tgz", @@ -8028,76 +7194,6 @@ "url": "https://opencollective.com/webpack" } }, - "node_modules/tar": { - "version": "6.2.1", - "resolved": "https://registry.npmjs.org/tar/-/tar-6.2.1.tgz", - "integrity": "sha512-DZ4yORTwrbTj/7MZYq2w+/ZFdI6OZ/f9SFHR+71gIVUZhOQPHzVCLpvRnPgyaMpfWxxk/4ONva3GQSyNIKRv6A==", - "deprecated": "Old versions of tar are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me", - "license": "ISC", - "optional": true, - "dependencies": { - "chownr": "^2.0.0", - "fs-minipass": "^2.0.0", - "minipass": "^5.0.0", - "minizlib": "^2.1.1", - "mkdirp": "^1.0.3", - "yallist": "^4.0.0" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/tar-fs": { - "version": "2.1.5", - "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz", - "integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "chownr": "^1.1.1", - "mkdirp-classic": "^0.5.2", - "pump": "^3.0.0", - "tar-stream": "^2.1.4" - } - }, - "node_modules/tar-stream": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz", - "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "dependencies": { - "bl": "^4.0.3", - "end-of-stream": "^1.4.1", - "fs-constants": "^1.0.0", - "inherits": "^2.0.3", - "readable-stream": "^3.1.1" - }, - "engines": { - "node": ">=6" - } - }, - "node_modules/tar/node_modules/chownr": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/chownr/-/chownr-2.0.0.tgz", - "integrity": "sha512-bIomtDF5KGpdogkLd9VspvFzk9KfpyyGlS8YFVZl7TGPBHL5snIOnxeshwVgPteQ9b4Eydl+pVbIyE1DcvCWgQ==", - "license": "ISC", - "optional": true, - "engines": { - "node": ">=10" - } - }, - "node_modules/tar/node_modules/yallist": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz", - "integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==", - "license": "ISC", - "optional": true - }, "node_modules/tiny-invariant": { "version": "1.3.3", "resolved": "https://registry.npmjs.org/tiny-invariant/-/tiny-invariant-1.3.3.tgz", @@ -8220,21 +7316,6 @@ "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", "license": "0BSD" }, - "node_modules/tunnel-agent": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz", - "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==", - "dev": true, - "license": "Apache-2.0", - "optional": true, - "peer": true, - "dependencies": { - "safe-buffer": "^5.0.1" - }, - "engines": { - "node": "*" - } - }, "node_modules/type-check": { "version": "0.4.0", "resolved": "https://registry.npmjs.org/type-check/-/type-check-0.4.0.tgz", @@ -8429,13 +7510,6 @@ } } }, - "node_modules/util-deprecate": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", - "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", - "license": "MIT", - "optional": true - }, "node_modules/vfile": { "version": "6.0.3", "resolved": "https://registry.npmjs.org/vfile/-/vfile-6.0.3.tgz", @@ -8614,420 +7688,6 @@ } } }, - "node_modules/vitest/node_modules/@esbuild/aix-ppc64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.28.2.tgz", - "integrity": "sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==", - "cpu": [ - "ppc64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "aix" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/android-arm": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.28.2.tgz", - "integrity": "sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==", - "cpu": [ - "arm" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "android" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/android-arm64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.28.2.tgz", - "integrity": "sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "android" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/android-x64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.28.2.tgz", - "integrity": "sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "android" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/darwin-arm64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.28.2.tgz", - "integrity": "sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/darwin-x64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.28.2.tgz", - "integrity": "sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/freebsd-arm64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.28.2.tgz", - "integrity": "sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "freebsd" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/freebsd-x64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.28.2.tgz", - "integrity": "sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "freebsd" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/linux-arm": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.28.2.tgz", - "integrity": "sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==", - "cpu": [ - "arm" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/linux-arm64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.28.2.tgz", - "integrity": "sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/linux-ia32": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.28.2.tgz", - "integrity": "sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==", - "cpu": [ - "ia32" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/linux-loong64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.28.2.tgz", - "integrity": "sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==", - "cpu": [ - "loong64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/linux-mips64el": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.28.2.tgz", - "integrity": "sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==", - "cpu": [ - "mips64el" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/linux-ppc64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.28.2.tgz", - "integrity": "sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==", - "cpu": [ - "ppc64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/linux-riscv64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.28.2.tgz", - "integrity": "sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==", - "cpu": [ - "riscv64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/linux-s390x": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.28.2.tgz", - "integrity": "sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==", - "cpu": [ - "s390x" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/linux-x64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.28.2.tgz", - "integrity": "sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/netbsd-x64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.28.2.tgz", - "integrity": "sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "netbsd" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/openbsd-x64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.28.2.tgz", - "integrity": "sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "openbsd" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/sunos-x64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.28.2.tgz", - "integrity": "sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "sunos" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/win32-arm64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.28.2.tgz", - "integrity": "sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "win32" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/win32-ia32": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.28.2.tgz", - "integrity": "sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==", - "cpu": [ - "ia32" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "win32" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, - "node_modules/vitest/node_modules/@esbuild/win32-x64": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.28.2.tgz", - "integrity": "sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "win32" - ], - "peer": true, - "engines": { - "node": ">=18" - } - }, "node_modules/vitest/node_modules/@vitest/mocker": { "version": "4.1.11", "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.11.tgz", @@ -9055,50 +7715,6 @@ } } }, - "node_modules/vitest/node_modules/esbuild": { - "version": "0.28.2", - "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.28.2.tgz", - "integrity": "sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==", - "dev": true, - "hasInstallScript": true, - "license": "MIT", - "optional": true, - "peer": true, - "bin": { - "esbuild": "bin/esbuild" - }, - "engines": { - "node": ">=18" - }, - "optionalDependencies": { - "@esbuild/aix-ppc64": "0.28.2", - "@esbuild/android-arm": "0.28.2", - "@esbuild/android-arm64": "0.28.2", - "@esbuild/android-x64": "0.28.2", - "@esbuild/darwin-arm64": "0.28.2", - "@esbuild/darwin-x64": "0.28.2", - "@esbuild/freebsd-arm64": "0.28.2", - "@esbuild/freebsd-x64": "0.28.2", - "@esbuild/linux-arm": "0.28.2", - "@esbuild/linux-arm64": "0.28.2", - "@esbuild/linux-ia32": "0.28.2", - "@esbuild/linux-loong64": "0.28.2", - "@esbuild/linux-mips64el": "0.28.2", - "@esbuild/linux-ppc64": "0.28.2", - "@esbuild/linux-riscv64": "0.28.2", - "@esbuild/linux-s390x": "0.28.2", - "@esbuild/linux-x64": "0.28.2", - "@esbuild/netbsd-arm64": "0.28.2", - "@esbuild/netbsd-x64": "0.28.2", - "@esbuild/openbsd-arm64": "0.28.2", - "@esbuild/openbsd-x64": "0.28.2", - "@esbuild/openharmony-arm64": "0.28.2", - "@esbuild/sunos-x64": "0.28.2", - "@esbuild/win32-arm64": "0.28.2", - "@esbuild/win32-ia32": "0.28.2", - "@esbuild/win32-x64": "0.28.2" - } - }, "node_modules/vitest/node_modules/lightningcss": { "version": "1.33.0", "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.33.0.tgz", @@ -9528,16 +8144,6 @@ "node": ">=8" } }, - "node_modules/wide-align": { - "version": "1.1.5", - "resolved": "https://registry.npmjs.org/wide-align/-/wide-align-1.1.5.tgz", - "integrity": "sha512-eDMORYaPNZ4sQIuuYPDHdQvf4gyCF9rEEV/yPxGfwPkRodwEgiMUUXTx/dex+Me0wxx53S+NgUHaP7y3MGlDmg==", - "license": "ISC", - "optional": true, - "dependencies": { - "string-width": "^1.0.2 || 2 || 3 || 4" - } - }, "node_modules/word-wrap": { "version": "1.2.5", "resolved": "https://registry.npmjs.org/word-wrap/-/word-wrap-1.2.5.tgz", @@ -9548,13 +8154,6 @@ "node": ">=0.10.0" } }, - "node_modules/wrappy": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", - "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", - "license": "ISC", - "optional": true - }, "node_modules/xml-name-validator": { "version": "5.0.0", "resolved": "https://registry.npmjs.org/xml-name-validator/-/xml-name-validator-5.0.0.tgz", diff --git a/ui/package.json b/ui/package.json index 502f156..e65e029 100644 --- a/ui/package.json +++ b/ui/package.json @@ -27,8 +27,8 @@ "react": "^18.2.0", "react-dom": "^18.2.0", "react-markdown": "^9.0.1", - "react-pdf": "^7.7.0", - "react-router-dom": "^6.20.0", + "react-pdf": "10.5.0", + "react-router-dom": "7.18.5", "remark-gfm": "^4.0.0", "tailwind-merge": "^3.5.0" }, diff --git a/ui/src/App.jsx b/ui/src/App.jsx index ded4634..8f0a75b 100644 --- a/ui/src/App.jsx +++ b/ui/src/App.jsx @@ -16,7 +16,6 @@ import ChunkExplorerView from './views/ChunkExplorerView' import GlobalAuditView from './views/GlobalAuditView' import UsersAdminView from './views/UsersAdminView' import LoginView from './views/LoginView' -import SsoCallbackView from './views/SsoCallbackView' function ProtectedLayout() { return ( @@ -32,7 +31,6 @@ export default function App() { return ( } /> - } /> }> } /> diff --git a/ui/src/auth/AuthProvider.jsx b/ui/src/auth/AuthProvider.jsx index fdf8e27..76d94be 100644 --- a/ui/src/auth/AuthProvider.jsx +++ b/ui/src/auth/AuthProvider.jsx @@ -3,20 +3,15 @@ import { Navigate, useLocation } from 'react-router-dom' import { API_BASE } from '../config' import { appPath } from '../basePath' import { AuthLoadingScreen } from '../components/AuthLoadingScreen' -import { normalizeProductRole, roleGrants } from '../lib/roleCapabilities' +import { normalizeProductRole, roleGrants, UserRole } from '../lib/roleCapabilities' import { AUTH_ENABLED, applyBackendSession, clearPersistedSession, - clearStoredAuthError, ensureFreshToken, - getAuthErrorMessage, getCurrentToken, - getStoredAuthError, initKeycloak, - isKeycloakConfigured, loadPersistedSession, - loginWithKeycloakRedirect, logoutFromKeycloak, mergeUserWithJwtProfile, ROUTES, @@ -67,10 +62,6 @@ async function loadBackendUser(token) { return res.json() } -function isSsoCallbackPath() { - return typeof window !== 'undefined' && window.location.pathname === appPath(ROUTES.AUTH_SSO_CALLBACK) -} - function isLoginPath() { return typeof window !== 'undefined' && window.location.pathname === appPath(ROUTES.LOGIN) } @@ -88,23 +79,12 @@ function hasSessionHint() { export function AuthProvider({ children }) { const sessionHint = hasSessionHint() - const [isInitializing, setIsInitializing] = useState( - AUTH_ENABLED && isKeycloakConfigured && !isSsoCallbackPath(), - ) + const [isInitializing, setIsInitializing] = useState(AUTH_ENABLED) // If we already have tokens in storage, treat as "pending restore" not logged-out. const [isAuthenticated, setIsAuthenticated] = useState(!AUTH_ENABLED) - const [isSsoLoading, setIsSsoLoading] = useState(false) - const [authError, setAuthError] = useState(() => getStoredAuthError()) const [user, setUser] = useState(null) // Keep true when a session hint exists so UI never assumes "logged out" mid-boot. - const [bootstrapped, setBootstrapped] = useState( - !(AUTH_ENABLED && isKeycloakConfigured) || isSsoCallbackPath(), - ) - - const clearAuthError = useCallback(() => { - setAuthError(null) - clearStoredAuthError() - }, []) + const [bootstrapped, setBootstrapped] = useState(!AUTH_ENABLED) const syncUnauthenticated = useCallback((options = {}) => { const { clearStorage = true } = options @@ -139,40 +119,9 @@ export function AuthProvider({ children }) { return } - if (!isKeycloakConfigured) { - setIsInitializing(false) - setIsAuthenticated(false) - setBootstrapped(true) - setAuthError( - 'Auth is enabled but Keycloak is not configured. Set VITE_KEYCLOAK_URL, VITE_KEYCLOAK_REALM, and VITE_KEYCLOAK_CLIENT_ID.', - ) - return - } - - // Callback page owns the OAuth code exchange — do not init Keycloak here. - if (isSsoCallbackPath()) { - setIsInitializing(false) - setIsAuthenticated(false) - setBootstrapped(true) - return - } - - // Login page with no stored session: skip Keycloak init so the first - // adapter init happens in loginWithKeycloakRedirect with the callback - // redirectUri (avoids PKCE / redirect_uri mismatches). - if (isLoginPath() && !loadPersistedSession()?.token) { - setIsInitializing(false) - setIsAuthenticated(false) - setBootstrapped(true) - return - } - setUnauthorizedHandler(() => { syncUnauthenticated({ clearStorage: true }) - if ( - window.location.pathname !== appPath(ROUTES.LOGIN) && - window.location.pathname !== appPath(ROUTES.AUTH_SSO_CALLBACK) - ) { + if (!isLoginPath()) { window.location.replace(appPath(ROUTES.LOGIN)) } }) @@ -181,8 +130,7 @@ export function AuthProvider({ children }) { ;(async () => { try { - // Rehydrates (and renews if stale) the session the SSO callback or the - // OTP verify wrote to localStorage. No adapter is involved any more. + // Restore the console session and renew it through the API if needed. const restored = await initKeycloak() if (cancelled) return @@ -209,27 +157,12 @@ export function AuthProvider({ children }) { console.warn('Token-only restore failed:', err2) } } - if (error && typeof error === 'object' && 'error' in error && typeof error.error === 'string') { - setAuthError( - getAuthErrorMessage( - error.error, - 'error_description' in error && typeof error.error_description === 'string' - ? error.error_description - : null, - ), - ) - } // Do not wipe storage on cancelled StrictMode races; only clear on hard failure. if (!cancelled) { syncUnauthenticated({ clearStorage: true }) } } finally { if (!cancelled) { - const storedErr = getStoredAuthError() - if (storedErr) { - setAuthError(storedErr) - clearStoredAuthError() - } setIsInitializing(false) setBootstrapped(true) } @@ -252,62 +185,15 @@ export function AuthProvider({ children }) { return () => clearInterval(id) }, [isAuthenticated]) - const loginWithSso = useCallback(async ({ idpHint, loginHint } = {}) => { - clearAuthError() - setIsSsoLoading(true) - + const signInWithSession = useCallback(async (tokens) => { + if (!applyBackendSession(tokens)) throw new Error('Sign-in did not return a session.') try { - const result = await loginWithKeycloakRedirect({ idpHint, loginHint }) - - if (result.status === 'success' && result.tokens?.token) { - try { - applyBackendSession(result.tokens) - await applyAuthenticatedUser(result.tokens.token) - setIsSsoLoading(false) - return true - } catch (err) { - console.error('Failed to apply existing SSO session:', err) - setAuthError( - 'Could not establish the session from Keycloak. Check AUTH_DISABLED / KEYCLOAK_* on the backend.', - ) - syncUnauthenticated({ clearStorage: true }) - setIsSsoLoading(false) - return false - } - } - - // Redirecting to Keycloak — keep button loading until the page unloads. - return false - } catch (error) { - console.error('Keycloak SSO login failed:', error) - const msg = - error instanceof Error ? error.message : typeof error === 'string' ? error : '' - setAuthError( - msg && msg !== 'undefined' - ? `Sign-in failed: ${msg}` - : 'Sign-in could not be completed. Please try again.', - ) - syncUnauthenticated({ clearStorage: false }) - setIsSsoLoading(false) - return false - } - }, [applyAuthenticatedUser, clearAuthError, syncUnauthenticated]) - - /** - * Establish a session from email-OTP tokens. Identical in every respect to an - * SSO session — same client, same claims, same renewal path — because both - * are minted by the backend against the one confidential client. - */ - const loginWithOtpTokens = useCallback( - async (tokens) => { - if (!tokens?.token) throw new Error('No access token returned') - clearAuthError() - applyBackendSession(tokens) await applyAuthenticatedUser(tokens.token) - return true - }, - [applyAuthenticatedUser, clearAuthError], - ) + } catch { + syncUnauthenticated({ clearStorage: true }) + throw new Error('Could not load your account. Please sign in again.') + } + }, [applyAuthenticatedUser, syncUnauthenticated]) const logout = useCallback(async () => { try { @@ -334,16 +220,9 @@ export function AuthProvider({ children }) { return k === 'super_admin' || k === 'superadmin' }), ) - // Bharat Vistaar viewer: every state, read-only. Deliberately separate from - // isSuperAdmin — folding the two together would grant write access. - const isBhViewer = Boolean( - user?.is_bh_viewer || - roles.some((r) => { - const k = String(r || '') - .toLowerCase() - .replace(/-/g, '_') - return k === 'bh_viewer' || k === 'bh_view' - }), + const isGlobalViewer = Boolean( + user?.is_global_viewer ?? user?.is_bh_viewer + ?? roles.some((role) => normalizeProductRole(role) === UserRole.GLOBAL_VIEWER), ) const displayName = user?.name || user?.username || user?.user_id || null const email = user?.email || null @@ -371,22 +250,22 @@ export function AuthProvider({ children }) { if (!AUTH_ENABLED || isSuperAdmin) return 'super_admin' if (!instance) return null const key = String(instance).trim().toLowerCase() - // A BH viewer holds no per-state group but reads every state. - return stateRoles[key] || (isBhViewer ? 'state_view' : null) + // A global viewer can read every provider and can hold a separate scoped role. + return stateRoles[key] || (isGlobalViewer ? 'state_view' : null) }, - [isSuperAdmin, isBhViewer, stateRoles], + [isSuperAdmin, isGlobalViewer, stateRoles], ) /** Whether the user may see a tenant (state). Super-admin → all. */ const canAccessInstance = useCallback( (instance) => { - if (!AUTH_ENABLED || isSuperAdmin || isBhViewer) return true + if (!AUTH_ENABLED || isSuperAdmin || isGlobalViewer) return true if (!instance) return false const key = String(instance).trim().toLowerCase() if (instances.length === 0) return false return instances.map((i) => String(i).toLowerCase()).includes(key) }, - [isSuperAdmin, isBhViewer, instances], + [isSuperAdmin, isGlobalViewer, instances], ) /** @@ -402,12 +281,16 @@ export function AuthProvider({ children }) { if (!AUTH_ENABLED) return true if (isSuperAdmin) return hasPermission(perm) const role = normalizeProductRole(roleForInstance(instance)) - if (!role) return false + if (!role) { + const hasScopedRoles = Object.keys(user?.state_roles || {}).length > 0 + const isMember = instances.some((id) => String(id).toLowerCase() === String(instance).toLowerCase()) + return !hasScopedRoles && isMember && hasPermission(perm) + } // Platform-level capabilities are never granted by a state role. if (perm === 'admin' || perm === 'manage_users') return false return roleGrants(role, perm) }, - [hasPermission, isSuperAdmin, roleForInstance], + [hasPermission, instances, isSuperAdmin, roleForInstance, user?.state_roles], ) const value = useMemo( @@ -417,8 +300,6 @@ export function AuthProvider({ children }) { isInitializing, bootstrapped, sessionHint, - isSsoLoading, - authError, user, username: displayName, displayName, @@ -430,24 +311,20 @@ export function AuthProvider({ children }) { groups, stateRoles, isSuperAdmin, - isBhViewer, + isGlobalViewer, hasPermission, hasPermissionForInstance, hasRole, roleForInstance, canAccessInstance, - loginWithSso, - loginWithOtpTokens, + signInWithSession, logout, - clearAuthError, }), [ isAuthenticated, isInitializing, bootstrapped, sessionHint, - isSsoLoading, - authError, user, displayName, email, @@ -458,22 +335,19 @@ export function AuthProvider({ children }) { groups, stateRoles, isSuperAdmin, - isBhViewer, + isGlobalViewer, hasPermission, hasPermissionForInstance, hasRole, roleForInstance, canAccessInstance, - loginWithSso, - loginWithOtpTokens, + signInWithSession, logout, - clearAuthError, ], ) // Global bootstrap gate: never paint login/app routes until session restore finishes. - // SSO callback is excluded so the OAuth return page can run. - if (AUTH_ENABLED && isKeycloakConfigured && !bootstrapped && !isSsoCallbackPath()) { + if (AUTH_ENABLED && !bootstrapped) { return ( 120_000) { - sessionStorage.removeItem(SSO_RESULT_STORAGE_KEY) - return null - } - return parsed - } catch { - return null - } -} - -function stripOAuthCallbackParams(url) { - for (const param of OAUTH_CALLBACK_PARAMS) { - url.searchParams.delete(param) - } - return `${url.pathname}${url.search}${url.hash}` -} - -/** - * Detect OAuth callback params in query or hash (Keycloak response modes). - */ -export function readOAuthCallbackParams(href = window.location.href) { - const url = new URL(href) - const hash = url.hash.startsWith('#') ? url.hash.slice(1) : url.hash - const fromHash = new URLSearchParams(hash) - const get = (key) => url.searchParams.get(key) || fromHash.get(key) || null - return { - error: get('error'), - errorDescription: get('error_description'), - code: get('code'), - state: get('state'), - // Prefer query when code is in search — more reliable with SPA routers. - responseMode: url.searchParams.has('code') || url.searchParams.has('error') ? 'query' : 'fragment', - } -} - -/** - * Runs before React mounts. - * - Forwards authorization codes that landed on the wrong path to /auth/sso-callback - * - On OAuth error params outside the callback route, sends the user to /login - */ -export function handleOAuthCallbackRedirect() { - if (typeof window === 'undefined') return - - const callbackPath = appPath(ROUTES.AUTH_SSO_CALLBACK) - const url = new URL(window.location.href) - const oauth = readOAuthCallbackParams(url.href) - - // Keycloak sometimes returns to a broader Valid Redirect URI (e.g. /login or /*). - // Always complete the code exchange on the dedicated callback route. - if (oauth.code && url.pathname !== callbackPath) { - const target = new URL(callbackPath, window.location.origin) - // Preserve whichever form Keycloak used (query or fragment). - if (url.search && url.search.length > 1) { - target.search = url.search - } - if (url.hash && url.hash.length > 1) { - target.hash = url.hash - } - // If params were only in hash, keep hash; if only query, keep query. - console.info('[auth] Forwarding OAuth code to SSO callback', { - from: url.pathname, - to: target.pathname, - }) - window.location.replace(target.pathname + target.search + target.hash) - return - } - - if (url.pathname === callbackPath) return - - if (!oauth.error) return - - const description = oauth.errorDescription - storeAuthError(getAuthErrorMessage(oauth.error, description)) - - if (window.location.pathname !== appPath(ROUTES.LOGIN)) { - window.location.replace(appPath(ROUTES.LOGIN)) - return - } - - window.history.replaceState(window.history.state, '', stripOAuthCallbackParams(url)) -} - -/** Lazily construct the singleton Keycloak instance (null when auth is off). */ export function setCurrentToken(token) { currentToken = token || null } @@ -357,38 +100,7 @@ export function parseJwtPayload(token) { } } -/** Canonical product roles (must match Keycloak group leaves / realm roles). */ -export const UserRole = { - SUPER_ADMIN: 'super_admin', - BH_VIEWER: 'bh_viewer', - STATE_ADMIN: 'state_admin', - STATE_APPROVER: 'state_approver', - STATE_CONTRIBUTOR: 'state_contributor', - STATE_VIEW: 'state_view', -} - -// Mirrors _ROLE_ALIASES in pipeline/auth/groups.py. -// NOTE: `contributor` is the WEAKEST upload role, not state_admin. -const ROLE_ALIASES = { - super_admin: UserRole.SUPER_ADMIN, - 'super-admin': UserRole.SUPER_ADMIN, - superadmin: UserRole.SUPER_ADMIN, - bh_viewer: UserRole.BH_VIEWER, - 'bh-viewer': UserRole.BH_VIEWER, - bh_view: UserRole.BH_VIEWER, - state_admin: UserRole.STATE_ADMIN, - 'state-admin': UserRole.STATE_ADMIN, - admin: UserRole.STATE_ADMIN, - state_approver: UserRole.STATE_APPROVER, - 'state-approver': UserRole.STATE_APPROVER, - approver: UserRole.STATE_APPROVER, - state_contributor: UserRole.STATE_CONTRIBUTOR, - 'state-contributor': UserRole.STATE_CONTRIBUTOR, - contributor: UserRole.STATE_CONTRIBUTOR, - state_view: UserRole.STATE_VIEW, - 'state-view': UserRole.STATE_VIEW, - view: UserRole.STATE_VIEW, -} +export { UserRole } const ROLE_RANK = { [UserRole.SUPER_ADMIN]: 100, @@ -396,15 +108,7 @@ const ROLE_RANK = { [UserRole.STATE_APPROVER]: 40, [UserRole.STATE_CONTRIBUTOR]: 20, [UserRole.STATE_VIEW]: 10, - [UserRole.BH_VIEWER]: 5, -} - -function normalizeRoleName(value) { - if (!value || typeof value !== 'string') return null - const key = value.trim().toLowerCase().replace(/\s+/g, '_').replace(/-/g, '_') - if (ROLE_ALIASES[key]) return ROLE_ALIASES[key] - if (ROLE_ALIASES[value.trim().toLowerCase()]) return ROLE_ALIASES[value.trim().toLowerCase()] - return null + [UserRole.GLOBAL_VIEWER]: 5, } /** @@ -424,7 +128,7 @@ export function parseGroupsClaim(claims) { } let isSuperAdmin = false - let isBhViewer = false + let isGlobalViewer = false const stateRoles = {} for (let path of paths) { @@ -435,24 +139,16 @@ export function parseGroupsClaim(claims) { isSuperAdmin = true continue } - if (/^\/global\/(bh[_-]?viewer|bh[_-]?view)$/i.test(path)) { - isBhViewer = true + if (/^\/global\/(global[_-]viewer|bh[_-]?viewer|bh[_-]?view)$/i.test(path)) { + isGlobalViewer = true continue } const stateMatch = path.match(/^\/states\/([A-Za-z0-9_-]+)\/([A-Za-z0-9_-]+)$/i) if (!stateMatch) continue const state = stateMatch[1].toLowerCase() - const role = normalizeRoleName(stateMatch[2]) - if (!state || !role) continue - if (role === UserRole.SUPER_ADMIN) { - isSuperAdmin = true - continue - } - if (role === UserRole.BH_VIEWER) { - isBhViewer = true - continue - } + const role = normalizeProductRole(stateMatch[2]) + if (!state || !role || role === UserRole.SUPER_ADMIN || role === UserRole.GLOBAL_VIEWER) continue const current = stateRoles[state] if (!current || (ROLE_RANK[role] || 0) > (ROLE_RANK[current] || 0)) { stateRoles[state] = role @@ -461,17 +157,17 @@ export function parseGroupsClaim(claims) { const roles = new Set() if (isSuperAdmin) roles.add(UserRole.SUPER_ADMIN) - if (isBhViewer) roles.add(UserRole.BH_VIEWER) + if (isGlobalViewer) roles.add(UserRole.GLOBAL_VIEWER) Object.values(stateRoles).forEach((r) => roles.add(r)) return { groups: [...new Set(paths)].sort(), isSuperAdmin, - isBhViewer, + isGlobalViewer, stateRoles, - // super_admin and bh_viewer both span every state, so neither is scoped + // super_admin and global_viewer both span every state, so neither is scoped // by an instance list. - instances: isSuperAdmin || isBhViewer ? [] : Object.keys(stateRoles).sort(), + instances: isSuperAdmin || isGlobalViewer ? [] : Object.keys(stateRoles).sort(), roles: [...roles].sort((a, b) => (ROLE_RANK[b] || 0) - (ROLE_RANK[a] || 0)), } } @@ -487,7 +183,7 @@ function collectRolesFromClaims(claims) { const realmRoles = claims.realm_access?.roles if (Array.isArray(realmRoles)) { for (const role of realmRoles) { - const canon = normalizeRoleName(role) + const canon = normalizeProductRole(role) if (canon) roles.add(canon) else if (typeof role === 'string' && role.trim()) roles.add(role.trim()) } @@ -499,7 +195,7 @@ function collectRolesFromClaims(claims) { if (!clientData || typeof clientData !== 'object') continue if (Array.isArray(clientData.roles)) { for (const role of clientData.roles) { - const canon = normalizeRoleName(role) + const canon = normalizeProductRole(role) if (canon) roles.add(canon) else if (typeof role === 'string' && role.trim()) roles.add(role.trim()) } @@ -509,7 +205,7 @@ function collectRolesFromClaims(claims) { if (Array.isArray(claims.roles)) { for (const role of claims.roles) { - const canon = normalizeRoleName(role) + const canon = normalizeProductRole(role) if (canon) roles.add(canon) else if (typeof role === 'string' && role.trim()) roles.add(role.trim()) } @@ -546,6 +242,7 @@ export function profileFromAccessToken(token) { const displayName = fullName || composed || preferred || email || '' const username = preferred || email || displayName || String(claims.sub || '') const groupAccess = parseGroupsClaim(claims) + const roles = collectRolesFromClaims(claims) // Legacy multivalued instances claim when groups are absent let instances = groupAccess.instances @@ -566,12 +263,12 @@ export function profileFromAccessToken(token) { username, name: displayName || username, email, - roles: collectRolesFromClaims(claims), + roles, groups: groupAccess.groups, state_roles: groupAccess.stateRoles, instances, is_super_admin: groupAccess.isSuperAdmin, - is_bh_viewer: groupAccess.isBhViewer, + is_global_viewer: roles.includes(UserRole.GLOBAL_VIEWER), claims, } } @@ -599,11 +296,15 @@ export function mergeUserWithJwtProfile(backendUser, token) { groups: profile.groups || [], state_roles: profile.state_roles || {}, is_super_admin: Boolean(profile.is_super_admin), - is_bh_viewer: Boolean(profile.is_bh_viewer), + is_global_viewer: Boolean(profile.is_global_viewer), auth_disabled: false, } } + if (backendUser.live_provider_access) { + return { ...backendUser, name: profile.name || backendUser.username || backendUser.user_id || '' } + } + // Prefer JWT identity labels; keep backend permissions (incl. bypass mode). const jwtRoles = profile.roles || [] const backendRoles = Array.isArray(backendUser.roles) ? backendUser.roles : [] @@ -632,7 +333,7 @@ export function mergeUserWithJwtProfile(backendUser, token) { is_super_admin: Boolean( backendUser.is_super_admin ?? profile.is_super_admin, ), - is_bh_viewer: Boolean(backendUser.is_bh_viewer ?? profile.is_bh_viewer), + is_global_viewer: Boolean(backendUser.is_global_viewer ?? backendUser.is_bh_viewer ?? profile.is_global_viewer), } } @@ -727,38 +428,6 @@ export async function apiFetch(url, options = {}) { return response } -export function getKeycloakRedirectUri() { - return `${window.location.origin}${appPath(ROUTES.LOGIN)}` -} - -export function getKeycloakSsoCallbackUri() { - // Exact match required in Keycloak Valid Redirect URIs (no trailing slash on callback). - return `${window.location.origin}${appPath(ROUTES.AUTH_SSO_CALLBACK)}` -} - -/** Human-readable Keycloak admin checklist for local / prod. */ -export function getKeycloakSetupHints() { - const origin = typeof window !== 'undefined' ? window.location.origin : 'http://localhost:3001' - return { - clientId: keycloakClientId, - realm: keycloakRealm, - validRedirectUris: [ - `${origin}${appPath(ROUTES.LOGIN)}`, - `${origin}${appPath(ROUTES.AUTH_SSO_CALLBACK)}`, - `${origin}${appPath('/').replace(/\/$/, '') || ''}/*`, - ], - webOrigins: [origin, '+'], - notes: [ - 'Client authentication: ON (confidential) — required by the email-otp extension', - 'Standard flow: enabled (browser gets only the code; backend exchanges it)', - 'Direct access grants: enabled, with Direct grant flow = email-otp-direct-grant', - 'PKCE: S256', - 'Backend needs KEYCLOAK_CLIENT_ID + KEYCLOAK_CLIENT_SECRET for the same client', - 'Production UI is under /docs-pipeline/*', - ], - } -} - /** * Restore a session on page load. * @@ -768,7 +437,7 @@ export function getKeycloakSetupHints() { * backend, so all this has to do is rehydrate localStorage and renew if stale. */ export async function initKeycloak() { - if (!isKeycloakConfigured) return false + if (!AUTH_ENABLED) return false const persisted = loadPersistedSession() if (!persisted?.token) return false @@ -813,179 +482,54 @@ export function applyBackendSession(tokens) { return true } - - -/** RFC 7636 verifier: 43-128 chars from the unreserved set. */ -function createCodeVerifier() { - const bytes = new Uint8Array(32) - window.crypto.getRandomValues(bytes) - return base64UrlEncode(bytes.buffer) -} - -function base64UrlEncode(buffer) { - const bytes = new Uint8Array(buffer) - let binary = '' - for (let i = 0; i < bytes.length; i += 1) binary += String.fromCharCode(bytes[i]) - return window.btoa(binary).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '') -} - -async function createCodeChallenge(verifier) { - const digest = await window.crypto.subtle.digest('SHA-256', new TextEncoder().encode(verifier)) - return base64UrlEncode(digest) -} - -/** - * Stash the PKCE verifier for the round trip to Keycloak. - * - * sessionStorage, not localStorage: it must survive the redirect but die with - * the tab, and it must not leak into other tabs mid-login. - */ -function stashPkce(verifier, state) { - try { - window.sessionStorage.setItem(PKCE_STORAGE_KEY, JSON.stringify({ verifier, state })) - } catch (err) { - console.warn('[auth] Could not stash the PKCE verifier:', err) - } -} - -function takePkce() { - try { - const raw = window.sessionStorage.getItem(PKCE_STORAGE_KEY) - window.sessionStorage.removeItem(PKCE_STORAGE_KEY) - return raw ? JSON.parse(raw) : null - } catch { - return null - } -} - -/** - * Complete SSO by having the backend exchange the authorization code. - * - * The browser cannot do this itself — the token endpoint requires the - * confidential client's secret. Sharing one client id between SSO and email-OTP - * is exactly what that buys us. - * - * @returns the token set on success. - */ -export async function exchangeSsoCode(code, { state } = {}) { - const stashed = takePkce() - if (stashed?.state && state && stashed.state !== state) { - throw new Error('Sign-in state did not match. Please start again.') - } - - const response = await fetch(`${API_BASE}/auth/sso/exchange`, { +async function postAuth(path, body) { + const response = await fetch(`${API_BASE}/auth/${path}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - code, - redirect_uri: getKeycloakSsoCallbackUri(), - code_verifier: stashed?.verifier || null, - }), + body: JSON.stringify(body), }) - + if (response.status === 204) return null const data = await response.json().catch(() => null) if (!response.ok) { - throw new Error(data?.detail || data?.message || 'Could not complete sign-in.') + const message = typeof data?.detail === 'string' ? data.detail : 'Sign-in could not be completed.' + const error = new Error(message) + error.challenge = data?.challenge + throw error } - if (!data?.access_token) throw new Error('Keycloak returned no access token.') + return data +} - return { - token: data.access_token, - refreshToken: data.refresh_token || null, - idToken: data.id_token || null, - } +export async function requestEmailCode(email) { + return postAuth('email/send', { email }) } -/** - * Full-page Keycloak SSO. - * - * The authorize URL is built by hand rather than through keycloak-js: the - * adapter insists on completing the token exchange in the browser, which a - * confidential client rejects. Only the authorize leg happens here; the code - * that comes back goes to the backend (see exchangeSsoCode). - * - * Navigates away on success; does not resolve. - */ -export async function loginWithKeycloakRedirect({ idpHint, loginHint } = {}) { - if (!isKeycloakConfigured) { - throw new Error( - 'Keycloak is not configured. Set VITE_AUTH_ENABLED=true plus VITE_KEYCLOAK_URL, VITE_KEYCLOAK_REALM, and VITE_KEYCLOAK_CLIENT_ID.', - ) - } +export async function verifyEmailCode(challenge, code) { + const data = await postAuth('email/verify', { challenge, code }) + return { token: data.access_token, refreshToken: data.refresh_token, idToken: data.id_token } +} - clearSsoResult() +export async function googleSignInEnabled() { + const response = await fetch(`${API_BASE}/auth/google`) + return response.ok && (await response.json()).enabled === true +} - // Reuse a live local session instead of bouncing through Keycloak. - const existing = loadPersistedSession() - if (existing?.token && existing.via === BACKEND_SESSION && !isJwtExpired(existing.token, 10)) { - setCurrentToken(existing.token) - return { status: 'success', tokens: existing } - } +export async function startGoogleSignIn() { + const { location } = await postAuth('google/start', {}) + window.location.assign(location) +} - const redirectUri = getKeycloakSsoCallbackUri() - const verifier = createCodeVerifier() - const state = createCodeVerifier() - stashPkce(verifier, state) - - const params = new URLSearchParams({ - client_id: keycloakClientId, - redirect_uri: redirectUri, - response_type: 'code', - scope: 'openid profile email', - state, - code_challenge: await createCodeChallenge(verifier), - code_challenge_method: 'S256', - prompt: 'select_account', - }) - // Empty hint = show the Keycloak page (Google button + password/email-OTP - // form). idpHint passed in overrides the env default (undefined = env - // default, e.g. the Google button; '' = explicitly no hint, e.g. the email - // login path, which needs Keycloak's own hosted email-otp-browser flow — - // the mesutpiskin email-authenticator plugin only supports OTP delivery - // through a real browser session, not a headless/direct-grant REST call. - const hint = idpHint === undefined ? keycloakIdpHint : idpHint - if (hint) params.set('kc_idp_hint', hint) - // Pre-fills Keycloak's username field so the user doesn't retype the email - // they already entered on our page. - if (loginHint) params.set('login_hint', loginHint) - - const authorizeUrl = `${keycloakUrl}/realms/${encodeURIComponent( - keycloakRealm, - )}/protocol/openid-connect/auth?${params.toString()}` - - console.info('[auth] Starting Keycloak SSO redirect', { - redirectUri, - realm: keycloakRealm, - clientId: keycloakClientId, - idpHint: hint || '(none — Keycloak login page)', - loginHint: loginHint || undefined, - }) +let googleSessionRequest - window.location.assign(authorizeUrl) - return { status: 'redirecting' } +export function finishGoogleSignIn() { + googleSessionRequest ??= postAuth('google/complete', {}).then(data => ({ + token: data.access_token, refreshToken: data.refresh_token, idToken: data.id_token, + })).finally(() => { googleSessionRequest = undefined }) + return googleSessionRequest } - export async function logoutFromKeycloak() { - const stored = loadPersistedSession() - setCurrentToken(null) - clearSsoResult() - clearPersistedSession() - if (!isKeycloakConfigured) return - - // Built by hand for the same reason login is: the adapter is never initialized - // against this confidential client, so kc.logout() has nothing to work with. - const params = new URLSearchParams({ - post_logout_redirect_uri: `${window.location.origin}${appPath(ROUTES.LOGIN)}`, - client_id: keycloakClientId, - }) - // Keycloak requires either an id_token_hint or client_id to skip the - // "do you want to log out?" confirmation page. - if (stored?.idToken) params.set('id_token_hint', stored.idToken) - - window.location.assign( - `${keycloakUrl}/realms/${encodeURIComponent( - keycloakRealm, - )}/protocol/openid-connect/logout?${params.toString()}`, - ) + const session = loadPersistedSession() + if (session?.refreshToken) { + await postAuth('session/logout', { refresh_token: session.refreshToken }) + } } diff --git a/ui/src/auth/keycloak.test.js b/ui/src/auth/keycloak.test.js new file mode 100644 index 0000000..42ce4a2 --- /dev/null +++ b/ui/src/auth/keycloak.test.js @@ -0,0 +1,40 @@ +import { afterEach, expect, it, vi } from 'vitest' +import { mergeUserWithJwtProfile, parseGroupsClaim, requestEmailCode, verifyEmailCode } from './keycloak' + +afterEach(() => vi.unstubAllGlobals()) + +it('sends email and code only to the application API and keeps the retry challenge', async () => { + const fetch = vi.fn() + .mockResolvedValueOnce({ ok: true, status: 202, json: async () => ({ challenge: 'first' }) }) + .mockResolvedValueOnce({ + ok: false, + status: 401, + json: async () => ({ detail: 'That code is invalid or expired.', challenge: 'retry' }), + }) + .mockResolvedValueOnce({ ok: true, status: 200, json: async () => ({ access_token: 'access', refresh_token: 'refresh' }) }) + vi.stubGlobal('fetch', fetch) + + const { challenge } = await requestEmailCode('reader@example.invalid') + await expect(verifyEmailCode(challenge, '123456')).rejects.toMatchObject({ challenge: 'retry' }) + await expect(verifyEmailCode('retry', '654321')).resolves.toMatchObject({ token: 'access', refreshToken: 'refresh' }) + expect(fetch.mock.calls.map(([url, options]) => [url, options.method, JSON.parse(options.body)])).toEqual([ + ['/api/auth/email/send', 'POST', { email: 'reader@example.invalid' }], + ['/api/auth/email/verify', 'POST', { challenge: 'first', code: '123456' }], + ['/api/auth/email/verify', 'POST', { challenge: 'retry', code: '654321' }], + ]) +}) + +it.each(['/global/global-viewer', '/global/bh-viewer'])('maps %s to the global viewer role', (group) => { + expect(parseGroupsClaim({ groups: [group, '/states/ALPHA/admin'] })).toMatchObject({ + isGlobalViewer: true, + stateRoles: { alpha: 'state_admin' }, + roles: ['state_admin', 'global_viewer'], + }) +}) + +it('uses the canonical viewer flag first and accepts the legacy profile field', () => { + const token = `ignored.${btoa(JSON.stringify({ sub: 'reader', groups: ['/global/global-viewer'] }))}.signature` + + expect(mergeUserWithJwtProfile({ is_bh_viewer: true }, token).is_global_viewer).toBe(true) + expect(mergeUserWithJwtProfile({ is_global_viewer: false, is_bh_viewer: true }, token).is_global_viewer).toBe(false) +}) diff --git a/ui/src/main.jsx b/ui/src/main.jsx index 5ddefbe..f89a7e2 100644 --- a/ui/src/main.jsx +++ b/ui/src/main.jsx @@ -5,11 +5,8 @@ import App from './App' import './index.css' import { ThemeProvider } from './styles/theme' import { AuthProvider } from './auth/AuthProvider' -import { handleOAuthCallbackRedirect } from './auth/keycloak' import { APP_BASENAME } from './basePath' -handleOAuthCallbackRedirect() - ReactDOM.createRoot(document.getElementById('root')).render( diff --git a/ui/src/views/LoginView.jsx b/ui/src/views/LoginView.jsx index 8ff34f2..540e937 100644 --- a/ui/src/views/LoginView.jsx +++ b/ui/src/views/LoginView.jsx @@ -1,188 +1,121 @@ -import { useEffect, useRef, useState } from 'react' -import { Navigate, useNavigate } from 'react-router-dom' +import { useEffect, useState } from 'react' +import { Navigate } from 'react-router-dom' -import { AuthLoadingScreen, InlineSpinner } from '../components/AuthLoadingScreen' +import { AuthLoadingScreen } from '../components/AuthLoadingScreen' import { HeroPanel } from '../components/HeroPanel' import { PlatformLogoIcon } from '../components/PlatformLogoIcon' import { Button } from '../components/ui/button' import { useAuth } from '../auth/AuthProvider' -import { AUTH_ENABLED, ROUTES } from '../auth/keycloak' -import { API_BASE } from '../config' +import { + AUTH_ENABLED, + ROUTES, + finishGoogleSignIn, + googleSignInEnabled, + requestEmailCode, + startGoogleSignIn, + verifyEmailCode, +} from '../auth/keycloak' import { APP_DESCRIPTION, APP_NAME } from '../lib/app-brand' -const OTP_LENGTH = 6 -/** Fallback only — the send response carries the real cooldown. */ -const RESEND_SECONDS = 30 - -/** Google's brand mark — kept as the official four-colour glyph. */ -function GoogleIcon({ className }) { - return ( - - ) -} - -function MailIcon({ className }) { - return ( - - ) -} - -function BackArrow({ className }) { - return ( - - ) -} - -function isValidEmail(value) { - return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(value.trim()) -} - -/** Unauthenticated POST — the session does not exist yet, so no bearer token. */ -async function postJson(path, body) { - const response = await fetch(`${API_BASE}${path}`, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(body), - }) - const isJson = response.headers.get('content-type')?.includes('application/json') - const data = isJson ? await response.json() : null - if (!response.ok) { - const detail = data?.detail - throw new Error( - typeof detail === 'string' ? detail : `Request failed (${response.status})`, - ) - } - return data -} - export default function LoginView() { - const navigate = useNavigate() - const { - isAuthenticated, - isInitializing, - bootstrapped, - isSsoLoading, - authError, - loginWithSso, - loginWithOtpTokens, - } = useAuth() - - const [step, setStep] = useState('choose') // choose | code + const { isAuthenticated, isInitializing, bootstrapped, signInWithSession } = useAuth() const [email, setEmail] = useState('') + const [challenge, setChallenge] = useState('') const [code, setCode] = useState('') - const [otpBusy, setOtpBusy] = useState(false) - const [otpError, setOtpError] = useState('') - const [secondsLeft, setSecondsLeft] = useState(0) - const codeInputRef = useRef(null) + const [error, setError] = useState('') + const [busy, setBusy] = useState(false) + const [showGoogle, setShowGoogle] = useState(false) - // Resend cooldown, so a stuck email can't be hammered. useEffect(() => { - if (secondsLeft <= 0) return undefined - const timer = setTimeout(() => setSecondsLeft(s => s - 1), 1000) - return () => clearTimeout(timer) - }, [secondsLeft]) + if (!AUTH_ENABLED || isAuthenticated) return + let active = true + googleSignInEnabled().then(enabled => { + if (active) setShowGoogle(enabled) + }).catch(() => {}) + return () => { active = false } + }, [isAuthenticated]) useEffect(() => { - if (step === 'code') codeInputRef.current?.focus() - }, [step]) + if (!bootstrapped || isAuthenticated) return + const result = window.location.hash + if (result !== '#google-complete' && result !== '#google-error') return + const clearResult = () => window.history.replaceState( + window.history.state, '', window.location.pathname + window.location.search, + ) + if (result === '#google-error') { + setError('Google sign-in could not be completed. Try again or use email.') + clearResult() + return + } + let active = true + setBusy(true) + finishGoogleSignIn().then(async tokens => { + if (!active) return + clearResult() + await signInWithSession(tokens) + }).catch(failure => { + if (!active) return + clearResult() + setError(failure.message) + }).finally(() => { + if (active) setBusy(false) + }) + return () => { active = false } + }, [bootstrapped, isAuthenticated, signInWithSession]) - // Still restoring session from storage — never flash the login form. if (AUTH_ENABLED && (!bootstrapped || isInitializing)) { return } + if (isAuthenticated) return - if (AUTH_ENABLED && isAuthenticated) { - return - } - - if (isSsoLoading) { - return ( - - ) - } - - const handleSsoClick = async () => { - const ok = await loginWithSso() - if (ok) navigate(ROUTES.HOME, { replace: true }) - } - - async function sendCode(event) { - event?.preventDefault() - if (!isValidEmail(email)) { - setOtpError('Enter a valid email address.') - return - } + const send = async () => { + setBusy(true) + setError('') try { - setOtpBusy(true) - setOtpError('') - const sent = await postJson('/auth/otp/request', { email: email.trim().toLowerCase() }) - setStep('code') + const result = await requestEmailCode(email.trim()) + setChallenge(result.challenge) setCode('') - // Backend owns the cooldown; resending earlier is a silent no-op there. - setSecondsLeft(Number(sent?.resend_after_seconds) || RESEND_SECONDS) - } catch (err) { - setOtpError(err.message) + } catch (failure) { + setError(failure.message) } finally { - setOtpBusy(false) + setBusy(false) } } - async function verifyCode(event) { - event?.preventDefault() - if (code.length !== OTP_LENGTH) { - setOtpError(`Enter the ${OTP_LENGTH}-digit code.`) + const submit = async (event) => { + event.preventDefault() + if (!challenge) { + await send() return } + setBusy(true) + setError('') try { - setOtpBusy(true) - setOtpError('') - const tokens = await postJson('/auth/otp/verify', { - email: email.trim().toLowerCase(), - code, - }) - // Same token set SSO produces, so the session behaves identically from here. - await loginWithOtpTokens({ - token: tokens.access_token, - refreshToken: tokens.refresh_token, - idToken: tokens.id_token, - }) - navigate(ROUTES.HOME, { replace: true }) - } catch (err) { - setOtpError(err.message) - setCode('') - codeInputRef.current?.focus() + await signInWithSession(await verifyEmailCode(challenge, code)) + } catch (failure) { + if (failure.challenge) setChallenge(failure.challenge) + setError(failure.message) } finally { - setOtpBusy(false) + setBusy(false) + } + } + + const signInWithGoogle = async () => { + setBusy(true) + setError('') + try { + await startGoogleSignIn() + } catch (failure) { + setError(failure.message) + setBusy(false) } } - const errorText = otpError || authError - const emailLabel = email.trim().toLowerCase() + const changeEmail = () => { + setChallenge('') + setCode('') + setError('') + } return (
@@ -200,139 +133,81 @@ export default function LoginView() {
- {step === 'code' ? ( - - ) : null} -

- {step === 'code' ? 'Check your email' : 'Sign in'} + {challenge ? 'Check your email' : 'Sign in'}

- {step === 'code' ? ( - <> - We sent a {OTP_LENGTH}-digit code to{' '} - {emailLabel}. It - expires in a few minutes. - - ) : ( - APP_DESCRIPTION - )} + {challenge ? `Enter the six-digit code sent to ${email.trim()}.` : APP_DESCRIPTION}

- {errorText ? ( + {error ? (
- {errorText} + {error}
) : null} - {step === 'choose' ? ( -
- - - - ) : ( -
- - { - setCode(e.target.value.replace(/\D/g, '').slice(0, OTP_LENGTH)) - setOtpError('') - }} - disabled={otpBusy} - className="h-16 w-full rounded-lg border border-canopy-form-border bg-canopy-form-card text-center font-mono text-[1.75rem] tracking-[0.45em] text-canopy-form-text outline-none transition-colors placeholder:text-canopy-form-muted/30 focus:border-canopy-form-accent focus:ring-2 focus:ring-canopy-form-accent/20 disabled:opacity-70" - /> - - - -

- Didn’t get it?{' '} - -

-
- )} + ) : ( +
+ + setEmail(event.target.value)} + autoComplete="email" + maxLength={254} + required + autoFocus + className="h-[3.25rem] w-full rounded-lg border border-canopy-form-muted/50 bg-white px-3 text-slate-900" + /> +
+ )} + + {challenge ? ( +
+ + +
+ ) : null} + + {showGoogle && !challenge ? ( + + ) : null}