diff --git a/README.md b/README.md index cb103c6..0c54907 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,7 @@ [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) -**SQLGuard**（仓库 [`tangyf07/SQLGuard`](https://github.com/tangyf07/SQLGuard)；PyPI/CLI 包名仍为 `sql-write-gate`）= 面向 AI Agent / Text2SQL 出站路径的 **确定性 SQL 安全执行网关**。 - -Suite 第三环：**[RetailDW](https://github.com/tangyf07/RetailDW) → [GameStream](https://github.com/tangyf07/GameStream) → SQLGuard**。数据消费安全向的工程探索，服务数据开发主线 — **不是**个人作品集的一号旗舰项目。 +**SQLGuard**（仓库 [`tangyf07/SQLGuard`](https://github.com/tangyf07/SQLGuard)；PyPI/CLI 包名仍为 `sql-write-gate`）= 面向 AI Agent / Text2SQL 出站路径的 **确定性 SQL 安全执行网关**。Complements [RetailDW](https://github.com/tangyf07/RetailDW) / [GameStream](https://github.com/tangyf07/GameStream) as the data-consumption safety layer. ## Why @@ -39,13 +37,14 @@ Deterministic policy engine (sqlglot AST + catalog + policy.yaml). **No LLM. No ## Guarantees -在已声明支持矩阵上（**pilot-ready**：DuckDB / PostgreSQL / MySQL / SQLite + 已列 SQL / 入口）： +在已声明 **support matrix** 上（**pilot-ready**：DuckDB / PostgreSQL / MySQL / SQLite + 已列 SQL / 入口）： - 确定性 `ALLOW` / `BLOCK` / `REQUIRE_APPROVAL`，带 `rule_id` + evidence - 未列 / 歧义 SQL → fail-closed `unsupported_sql`（从不静默当只读 ALLOW） - Seal 四案稳定：legal→ALLOW/ok · PII→BLOCK/pii_column · schema→BLOCK/schema_hallucination · expired→BLOCK/expired_partition -- **HTTP `serve` server-locked**：`--policy` / `--catalog` / `--database` / environment 在进程启动时绑定；请求体不得覆盖（覆盖 → `400 trust_boundary_violation`） +- **HTTP `serve` server-locked（P0）**：`--policy` / `--catalog` / `--database` / environment 在进程启动时绑定；请求体不得覆盖（覆盖 → `400 trust_boundary_violation`） - **Audit SQL 默认脱敏**：`SQL_WRITE_GATE_AUDIT_SQL_MODE=redact`（可 `hash` / `plain`） +- **Trusted executor**：`approve` / `resolve` / `reject` 需 `SQL_WRITE_GATE_APPROVAL_TOKEN`（密钥文件默认 `.logs/approval.key` / `SQL_WRITE_GATE_APPROVAL_KEY_FILE`） **非生产唯一边界 / 非唯一边界** — **not** the sole production DB security boundary. Combine with least-privilege DB roles, network isolation, and human workflows. @@ -61,7 +60,7 @@ sql-write-gate check "DELETE FROM orders" # → BLOCKED rule=delete_without_where ``` -更多安装与入口见下方 [Install](#install) / [Entrypoints](#entrypoints-stable)。 +Stable CLI entrypoints: `check` · `hook` · `mcp` · `proxy` · `approve` · `audit`（详情见 [docs/api.md](docs/api.md)；安装见 [docs/install.md](docs/install.md)）。 ## Evidence @@ -83,356 +82,15 @@ sql-write-gate check "DELETE FROM orders" - **审计默认 redact** — JSONL 审计对 SQL 字面量默认脱敏，降低日志侧泄露面（`redact|hash|plain`）。 - **Trusted executor** — DB 凭证、`approve`/`resolve`/`reject` 与 policy 改写权留在受信执行侧；Agent 面只评估 / 入队。 - **Fail-closed** — 未列语法、歧义写形态、估计失败、缺 flock 等拒绝执行，不静默降级为 ALLOW。 -- **Suite 角色** — RetailDW（离线仓）→ GameStream（实时 ADS）→ SQLGuard（出站门禁）；本仓服务主线消费安全，不做作品集旗舰叙事。 ## Limitations - **非生产唯一边界 / 非唯一边界** — 须与最小权限 DB 角色、网络隔离、人工流程并用 -- 仅声明矩阵：DuckDB / PostgreSQL / MySQL / SQLite + 已列 SQL；未列语法拒绝 +- 仅声明 support matrix：DuckDB / PostgreSQL / MySQL / SQLite + 已列 SQL；未列语法 → `unsupported_sql` - 不是分布式审批锁、MySQL wire 代理、Web UI、企业 DQ/血缘/多租户平台 -- 非 loopback / `0.0.0.0` 的 HTTP `serve` 须显式 token（详见下方 Trust boundary） +- 非 loopback / `0.0.0.0` 的 HTTP `serve` 须显式 token（见 [docs/api.md](docs/api.md) Trust boundary） - GitHub Latest Release 可能滞后 `main`；以包版本 / commit 为准 -- 单机审批依赖 Unix `fcntl.flock`；Windows 上审批变更 fail-closed（不静默解锁） - -## Docs - -| Doc | Purpose | -|-----|---------| -| [docs/compatibility.md](docs/compatibility.md) | SemVer / breaking-change policy | -| [docs/upgrade-0.23-to-1.0.md](docs/upgrade-0.23-to-1.0.md) | Upgrade path from 0.23 | -| [docs/pilot-checklist.md](docs/pilot-checklist.md) | Pilot evidence pack | -| [docs/v1-acceptance.md](docs/v1-acceptance.md) | System acceptance scenarios + proof | -| [docs/troubleshooting.md](docs/troubleshooting.md) | Common failures, `unknown`, token, CI | - -See [CHANGELOG.md](CHANGELOG.md) for version history. - ---- - -## Install - -```bash -pip install sql-write-gate -pip install 'sql-write-gate[postgres]' # optional: psycopg -pip install 'sql-write-gate[mysql]' # optional: pymysql -pip install 'sql-write-gate[mcp]' # optional: MCP server -``` - -From a clone: - -```bash -pip install -e ".[dev]" # or: make install -pip install -e ".[postgres,mysql]" -make test # PG/MySQL live tests skip if no service -``` - -```bash -sql-write-gate check "DELETE FROM users" -# → BLOCKED rule=delete_without_where -``` - -## Try it - -```bash -make install && make test && make seal # or: make demo -sql-write-gate check "DELETE FROM orders" -# → BLOCKED delete_without_where -sql-write-gate datapilot --json "SELECT o.order_id FROM orders o CROSS JOIN orders p" -# → {"datapilot": "BLOCK", "rule_id": "cartesian_join", ...} -sql-write-gate serve --port 8787 -# POST /v1/check {"sql": "..."} → action + risk_score + datapilot -``` - -## Entrypoints (stable) - -| Entrypoint | Role | -|------------|------| -| CLI `check` | Evaluate SQL; no execute | -| CLI `hook` | PreToolUse: block raw `psql` / `mysql` / … | -| CLI `mcp` | MCP stdio (`query_sql` / `write_sql`) | -| CLI `proxy` | Gate then execute if ALLOW | -| CLI `approve` / `resolve` / `reject` | Human approve / recover (trusted executor + token) | -| CLI `audit` / `pending` / `init` / `exec` | Ops helpers | -| CLI `serve` | SQLGuard DataPilot HTTP (`/v1/check`, `/v1/block`, `/v1/execute`, `/v1/datapilot`) | - -```bash -sql-write-gate check "SQL" # evaluate SQL; no execute -sql-write-gate hook # PreToolUse: block raw psql/mysql/… -sql-write-gate mcp # MCP stdio (query_sql / write_sql) -sql-write-gate proxy --sql "..." # gate then execute if ALLOW -sql-write-gate approve # human approve then write (once) -sql-write-gate resolve --as succeeded|failed|rejected -sql-write-gate audit # TIME / SOURCE / OP / TABLE / VERDICT -sql-write-gate init # scaffold policy.yaml + catalog.json -``` - -## DataPilot API contract (SQLGuard) - -**契约稳定（v1.1+）**：DataPilot 出站只依赖 `BLOCK` / `EXECUTE`（HTTP `/v1/check`·`/v1/block` / `/v1/execute`·`/v1/datapilot`，以及等价 MCP/CLI）。已合入的 1.1 路径勿破坏字段语义；扩展只加字段、不改既有含义。 - -DataPilot calls this gate **outbound**. Prefer MCP `query_sql` / `write_sql`, CLI `check` / `exec` / `proxy`, or HTTP: - -```bash -sql-write-gate serve --host 127.0.0.1 --port 8787 \ - --policy policy.yaml --catalog catalog.json --database seed/warehouse.duckdb -# Non-loopback / 0.0.0.0 requires: --auth-token SECRET (or SQL_WRITE_GATE_HTTP_TOKEN) -``` - -| Method | Path | Behavior | -|--------|------|----------| -| `GET` | `/healthz` | `{ok, product: SQLGuard, version}` | -| `POST` | `/v1/check` | Evaluate only → `action` + `risk_score` (`executed: false`) | -| `POST` | `/v1/execute` | Gate then execute **only on ALLOW** | -| `POST` | `/v1/block` | Alias of `/v1/check` | -| `POST` | `/v1/datapilot` | Alias of `/v1/execute` (1.1 semantics unchanged) | - -Request JSON: `{ "sql": "...", "actor"?, "model_id"?, "prompt_summary"? }` only. `serve` locks `--policy` / `--catalog` / `--database` / environment at startup; body overrides of those fields are **rejected**. - -Response always includes `action` (`ALLOW` \| `BLOCK` \| `REQUIRE_APPROVAL`), `rule_id`, `reason`, `risk_score`, `risk_factors`, `executed`. Treat anything other than `ALLOW` as non-executing. - -### Trust boundary (HTTP `serve`) - -- **Server-locked at startup.** `sql-write-gate serve --policy/--catalog/--database` (plus environment from the locked policy) is bound for the process lifetime. Request bodies may only supply `sql` / `actor` / `model_id` / `prompt_summary`; overrides of `policy` / `catalog` / `database` / `db_path` / `environment` return `400 trust_boundary_violation`. -- **Auth for non-loopback.** Binding `127.0.0.1` / `::1` may omit auth. Non-loopback hosts (including `0.0.0.0` / `::`) **require** `--auth-token` or `SQL_WRITE_GATE_HTTP_TOKEN`; all-interfaces without auth is refused at startup. Present `Authorization: Bearer ` or `X-SQLGuard-Token`. -- **GitHub Release lags main.** Package / `main` tracks the latest commit; GitHub **Latest Release** may lag. Prefer install-from-main / pin the tip SHA for suite acceptance until a matching Release is cut. - -### GameStream-style permissions - -```yaml -permissions: - enforce: true - tables: - orders: [select, insert, update] -hallucination: - allow_unknown_tables: false - allow_unknown_columns: false -``` - -Python alias: `from write_gate import sqlguard` (product helpers); package/CLI names unchanged. - -## Declared databases (supported) - -| Backend | How to connect | Notes | -|---------|----------------|-------| -| **DuckDB** | file path / default `seed/warehouse.duckdb` | Default local warehouse | -| **PostgreSQL** | `POSTGRES_URL` or `postgresql://…` / `postgres://…` | Extra: `sql-write-gate[postgres]` | -| **MySQL** | `MYSQL_URL` or `mysql://…` / `mysql+pymysql://…` | Extra: `sql-write-gate[mysql]` | -| **SQLite** | `sqlite:///` / `sqlite+aiosqlite://` (incl. `C:/…`) | stdlib | - -Priority: `database=` → `database_url=` → `db_path=` → `POSTGRES_URL` → `MYSQL_URL` → `DATABASE_URL` → DuckDB default. - -Anything **not** in this matrix (other warehouses, wire-protocol proxies, distributed locks, Web UI) is **out of scope** for v1.0. - -## SQL support matrix - -| Supported (gated) | Explicitly rejected (`unsupported_sql` BLOCK) | -|-------------------|-----------------------------------------------| -| Single-statement `SELECT` / `INSERT` / `UPDATE` / `DELETE` | Multi-statement scripts (`stmt1; stmt2`) | -| DuckDB / PostgreSQL / MySQL / SQLite dialects via adapters | `MERGE` / `COPY` / `REPLACE` / raw `Command` | -| Simple CTEs over read-only SELECT | Data-modifying CTE / nested DML under any root | -| UPSERT `ON CONFLICT DO UPDATE` (PII/restricted on SET cols) | PostgreSQL `SELECT … INTO` | -| Catalog-backed schema / PII / freshness / blast-radius | Ambiguous or unlisted write-shaped SQL | - -**Anything not listed on the supported side → `unsupported_sql`** (BLOCK/REJECT, fail closed). Never silent ALLOW. - -## What it does (on the matrix) - -- `DROP` / `TRUNCATE` / `ALTER` → BLOCK -- `DELETE` / `UPDATE` without `WHERE` (incl. tautology `WHERE 1=1` / `TRUE`) → BLOCK -- Cartesian / missing-predicate JOINs → BLOCK (`cartesian_join`) -- Unknown tables/columns → BLOCK (`schema_hallucination`) with evidence -- GameStream-style `permissions.tables` allowlists in `policy.yaml` -- Numeric `risk_score` (0–100) + `risk_factors` on every Decision -- Blast-radius COUNT vs `update_rows` / `delete_rows` (dialect quoting; fail-closed on estimate error) -- Schema / PII / restricted columns; PII `SELECT` → REQUIRE_APPROVAL (approve executes once) -- Freshness partitions (`dt`); range / NOT / OR / UPSERT SET expired → BLOCK -- Nested / data-modifying CTE / `SELECT INTO` → REJECT (`unsupported_sql`) -- Approval state machine (SQLite source of truth + JSONL mirror): `pending`→`executing`→`succeeded`|`failed`|`unknown` (+ `rejected`) -- Atomic claim under `fcntl.flock` + SQLite `BEGIN IMMEDIATE` (single-host; fail closed without flock) -- Three-state execute outcomes; **`unknown`/`executing` never auto-retried** — use `resolve` or `approve --allow-unknown-retry` after manual DB verify -- JSONL audit (redacts URL passwords; **SQL literals redacted by default** via `SQL_WRITE_GATE_AUDIT_SQL_MODE=redact|hash|plain`; records execute failures / unknown; `request_id` + `approval_id` + `execution_outcome` correlation; rotatable) - -## Platform support matrix - -| Surface | Linux / macOS | Windows | -|---------|---------------|---------| -| `pip install` / CLI `check` / `init` / `audit` | ✅ | ✅ | -| DuckDB file backend | ✅ | ✅ | -| SQLite `sqlite:///` paths (incl. `C:/…`) | ✅ | ✅ | -| Postgres / MySQL URL adapters | ✅ | ✅ (drivers via extras) | -| PreToolUse hook / MCP stdio | ✅ | ✅ (same Python entrypoints) | -| Concurrent `approve` (flock + SQLite claim) | ✅ | ❌ **fail closed** — `ApprovalError` if `fcntl.flock` unavailable (no silent unlock) | - -Windows: install, CLI evaluate/execute on DuckDB/SQLite/URL backends work. Approval mutations require Unix `fcntl` flock (plus SQLite transactions); without flock they **refuse** rather than silently degrading. - -### Approval outcomes & crash recovery - -| Status | Meaning | Default `approve` | -|--------|---------|-------------------| -| `pending` | Queued; not executed | Claims → executes | -| `executing` | Claim held (in flight) | **Refuse** (no steal) | -| `succeeded` | DB write/query completed | Idempotent; **no re-write** | -| `failed` | Known not committed / never sent | May reclaim & retry | -| `unknown` | Timeout/disconnect/crash/indeterminate | **Refuse** — never auto-retry | -| `rejected` | Human rejected | Refuse | - -Recovery rules: - -1. Process crash while `executing`: after TTL (`SQL_WRITE_GATE_EXECUTING_TTL_SEC`, default 120s) → `unknown` (via `approve --force-unknown-check` / next store access). **Never** silent re-claim that re-runs SQL. -2. Operator path for `unknown`: verify target DB manually, then either - - `sql-write-gate resolve --as succeeded|failed|rejected` (no SQL), or - - `sql-write-gate approve --allow-unknown-retry` (explicit re-exec; double-write risk). -3. Default second `approve` on `succeeded` / `unknown` does **not** write again. - -## Database URLs - -| Env / kwarg | Backend | -|-------------|---------| -| `POSTGRES_URL` or `postgresql://…` / `postgres://…` | PostgreSQL | -| `MYSQL_URL` or `mysql://…` / `mysql+pymysql://…` | MySQL | -| `DATABASE_URL` (scheme-detected) | Postgres / MySQL / SQLite | -| `sqlite:///` / `sqlite+aiosqlite://` | SQLite (stdlib) | -| file path / default `seed/warehouse.duckdb` | DuckDB | - -### Live integration tests (optional locally) - -```bash -export POSTGRES_URL=postgresql://gate:gate@localhost:5432/writegate -export MYSQL_URL=mysql://gate:gate@127.0.0.1:3306/writegate -pip install -e ".[dev,postgres,mysql]" -make test -``` - -Without those services, live tests **skip**; CI runs Postgres + MySQL service containers. - -## Policy (default production) - -| operation | rule | -|-----------|------| -| select | allow | -| insert | approval | -| update | approval | -| delete | block | -| ddl | block | - -Limits: `update_rows: 100`, `delete_rows: 50`. Demo policy (`examples/policy.demo.yaml`) allows insert/update for walkthroughs. - -Guards (any **BLOCK** wins, else any **APPROVAL**, else **ALLOW**): - -`destructive` → `schema` → `pii` → `freshness` → `blast_radius` → `environment` - -## Stable interfaces - -Public surfaces for SemVer (see [docs/compatibility.md](docs/compatibility.md)): - -### CLI (public commands) - -`check` · `exec` · `hook` · `mcp` · `proxy` · `approve` · `resolve` · `reject` · `pending` · `audit` · `init` - -### Python API - -```python -from write_gate import WriteGate, Decision, Evidence # Evidence is Decision alias - -with WriteGate(database="postgresql://…") as gate: - decision = gate.check("DELETE FROM orders WHERE order_id = 1") - decision, result = gate.execute("SELECT 1") - decision, result = gate.approve(approval_id) # trusted executor + token env - gate.reject(approval_id) -``` - -Key methods: `check`, `execute`, `approve`, `reject`, `close` / context manager. - -### Decision JSON fields (`Decision.to_dict()` / `--json`) - -| Field | Type | Notes | -|-------|------|-------| -| `allowed` | bool | True only for `ALLOW` | -| `action` | str | `ALLOW` \| `BLOCK` \| `REQUIRE_APPROVAL` | -| `risk` | str | `low` \| `medium` \| `critical` | -| `rule_id` | str | e.g. `ok`, `delete_without_where`, `unsupported_sql` | -| `reason` / `message` | str | Human-readable (same text) | -| `evidence` | object | Guard evidence map | -| `sql` | str | Evaluated statement | -| `operation` | str \| null | `select` / `insert` / `update` / `delete` / `ddl` | -| `table` | str \| null | Primary table when known | -| `estimated_rows` | int \| null | Blast-radius estimate | -| `approval_id` | str \| null | When queued for approval | -| `rows` / `rowcount` / `truncated` | optional | Present on execute/approve `--json` when materializing | - -## Decision model - -`ALLOW` | `BLOCK` | `REQUIRE_APPROVAL` with `risk`, `rule_id`, `reason`, `evidence`. - -## Deployment model (trusted executor) - -**非生产唯一边界 / 非唯一边界** — this gate is not the sole production security control. - -| Concern | Where it lives | -|---------|----------------| -| DB credentials (`DATABASE_URL` / …) | **Trusted executor** only | -| `policy.yaml` / catalog | **Trusted executor** (agents have no rewrite API) | -| `approve` / `resolve` / `reject` | **Trusted executor** with approval token | -| `check` / `hook` / MCP `query_sql`/`write_sql` | Agent-facing: evaluate / enqueue only | - -### Approval privilege (`SQL_WRITE_GATE_APPROVAL_TOKEN`) - -1. On the trusted executor, create a secret file (default `.logs/approval.key`, or set `SQL_WRITE_GATE_APPROVAL_KEY_FILE`). -2. When calling `approve` / `resolve` / `reject`, set env `SQL_WRITE_GATE_APPROVAL_TOKEN` to that file's contents. -3. Missing key file, missing token, or wrong token → **refuse** (CLI exit non-zero). Correct token → approve/resolve/reject proceeds. -4. Agents must **not** receive the key file or token. They may still enqueue `REQUIRE_APPROVAL` via normal write paths. - -### Target binding - -Approval records store `database_config_id` (fingerprint). Approve reconnect binds trusted credentials only for the **same** target. Queue against DB A then change env to DB B → approve **fail closed** (will not write to B). - -## Boundaries (non-goals) - -- **非生产唯一边界 / 非唯一边界** — combine with least-privilege DB roles, network isolation, and human workflows -- Not a distributed approval lock, MySQL wire-protocol proxy, or Web UI -- Not an enterprise DQ / lineage / ChatBI / multi-tenant platform -- Not new cloud warehouses beyond the declared DuckDB / PostgreSQL / MySQL / SQLite matrix - -## Ops knobs - -| Env | Default | Purpose | -|-----|---------|---------| -| `SQL_WRITE_GATE_STATEMENT_TIMEOUT_SEC` | `0` (off) | Wall-clock statement timeout for check/execute/approve (caller returns at deadline; abandoned worker may still run → indeterminate/`unknown`; no blind retry) | -| `SQL_WRITE_GATE_RESULT_ROW_LIMIT` | `1000` | Cap SELECT/approve rows (truncate + `truncated=true`) | -| `SQL_WRITE_GATE_RESULT_BYTE_LIMIT` | `0` (off) | Hard byte cap on materialized rows (payload ≤ limit, or `ResultOversizeError` when `RESULT_OVERSIZE=block`; oversized single row never returned intact) | -| `SQL_WRITE_GATE_RESULT_OVERSIZE` | `truncate` | `truncate` (shrink/omit to keep ≤ byte/row caps) or `block` (`ResultOversizeError`) | -| `SQL_WRITE_GATE_AUDIT_SQL_MODE` | `redact` | Audit SQL storage: `redact` (literal scrub, default), `hash` (`sha256:…`), or `plain` (verbatim) | -| `SQL_WRITE_GATE_HTTP_TOKEN` | (optional on loopback) | Bearer token for `serve`; **required** for non-loopback / `0.0.0.0` binds | -| `SQL_WRITE_GATE_AUDIT_MAX_BYTES` | `10 MiB` | Rotate audit / approvals JSONL by size | -| `SQL_WRITE_GATE_AUDIT_ROTATE_DAILY` | `false` | Also rotate JSONL per UTC day | -| `SQL_WRITE_GATE_REQUEST_ID` | auto uuid4 | Audit correlation id | -| `SQL_WRITE_GATE_EXECUTING_TTL_SEC` | `120` | Stuck `executing` → `unknown` | -| `SQL_WRITE_GATE_APPROVAL_TOKEN` | (required for approve) | Presenter token | -| `SQL_WRITE_GATE_APPROVAL_KEY_FILE` | `.logs/approval.key` | Trusted-executor key path | - -## Backlog (post-1.0) - -- [x] Statement timeout + failed/unknown mapping (0.23) -- [x] Result row/byte caps with truncate flag (0.23) -- [x] Audit `request_id` / correlation fields (0.23) -- [x] JSONL audit / approvals mirror rotation (0.23) -- [x] Troubleshooting guide (0.23) -- [x] Trusted-executor approval token + key file privilege separation (0.22) -- [x] Approve target fingerprint fail-closed on DATABASE_URL swap (0.22) -- [x] SQL support matrix + unsupported variant regressions (0.22) -- [x] Three-state approve outcomes + unknown ≠ auto-retry (0.21) -- [x] SQLite durable approval store + crash TTL → unknown (0.21) -- [x] Multi-process single-write approve regressions (0.21) -- [x] Real Postgres / MySQL CI services + persist/recheck integration tests (0.20) -- [x] R1–R6 permanent regression (dangerous + safe paths) (0.20) -- [x] Windows support matrix + flock fail-closed (0.20) -- [x] Release gate: wheel install smoke; publish needs test+build on same tag (0.20) -- [x] **v1.0.0** limited support-matrix pilot-ready packaging + upgrade/acceptance docs -- [x] **v1.0.1** prompt timeout return + hard result byte limit -- [ ] Deferred: distributed / multi-host approval lock -- [ ] Deferred: MySQL wire-protocol proxy -- [ ] Deferred: Web UI -- [ ] Deferred: additional cloud warehouses - -## 许可 +- 单机审批依赖 Unix `fcntl.flock`；**Windows** 上审批变更 **fail closed**（`ApprovalError`，不静默解锁） +- 连接可用 `POSTGRES_URL` / `MYSQL_URL` / `DATABASE_URL`（见 [docs/support-matrix.md](docs/support-matrix.md)） -MIT。见 [LICENSE](LICENSE). +See also: [docs/install.md](docs/install.md) · [docs/api.md](docs/api.md) · [docs/support-matrix.md](docs/support-matrix.md) · [docs/troubleshooting.md](docs/troubleshooting.md) · [docs/compatibility.md](docs/compatibility.md) · [docs/pilot-checklist.md](docs/pilot-checklist.md) · [CHANGELOG.md](CHANGELOG.md) · MIT [LICENSE](LICENSE) diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..e9df9ba --- /dev/null +++ b/docs/api.md @@ -0,0 +1,175 @@ +# Entrypoints & API + +**非生产唯一边界 / 非唯一边界** — this gate is not the sole production security control. Prefer least-privilege DB roles, network isolation, and human workflows alongside SQLGuard. + +## Entrypoints (stable) + +| Entrypoint | Role | +|------------|------| +| CLI `check` | Evaluate SQL; no execute | +| CLI `hook` | PreToolUse: block raw `psql` / `mysql` / … | +| CLI `mcp` | MCP stdio (`query_sql` / `write_sql`) | +| CLI `proxy` | Gate then execute if ALLOW | +| CLI `approve` / `resolve` / `reject` | Human approve / recover (trusted executor + token) | +| CLI `audit` / `pending` / `init` / `exec` | Ops helpers | +| CLI `serve` | SQLGuard DataPilot HTTP (`/v1/check`, `/v1/block`, `/v1/execute`, `/v1/datapilot`) | + +```bash +sql-write-gate check "SQL" # evaluate SQL; no execute +sql-write-gate hook # PreToolUse: block raw psql/mysql/… +sql-write-gate mcp # MCP stdio (query_sql / write_sql) +sql-write-gate proxy --sql "..." # gate then execute if ALLOW +sql-write-gate approve # human approve then write (once) +sql-write-gate resolve --as succeeded|failed|rejected +sql-write-gate audit # TIME / SOURCE / OP / TABLE / VERDICT +sql-write-gate init # scaffold policy.yaml + catalog.json +``` + +## DataPilot API contract (SQLGuard) + +**契约稳定（v1.1+）**：DataPilot 出站只依赖 `BLOCK` / `EXECUTE`（HTTP `/v1/check`·`/v1/block` / `/v1/execute`·`/v1/datapilot`，以及等价 MCP/CLI）。已合入的 1.1 路径勿破坏字段语义；扩展只加字段、不改既有含义。 + +DataPilot calls this gate **outbound**. Prefer MCP `query_sql` / `write_sql`, CLI `check` / `exec` / `proxy`, or HTTP: + +```bash +sql-write-gate serve --host 127.0.0.1 --port 8787 \ + --policy policy.yaml --catalog catalog.json --database seed/warehouse.duckdb +# Non-loopback / 0.0.0.0 requires: --auth-token SECRET (or SQL_WRITE_GATE_HTTP_TOKEN) +``` + +| Method | Path | Behavior | +|--------|------|----------| +| `GET` | `/healthz` | `{ok, product: SQLGuard, version}` | +| `POST` | `/v1/check` | Evaluate only → `action` + `risk_score` (`executed: false`) | +| `POST` | `/v1/execute` | Gate then execute **only on ALLOW** | +| `POST` | `/v1/block` | Alias of `/v1/check` | +| `POST` | `/v1/datapilot` | Alias of `/v1/execute` (1.1 semantics unchanged) | + +Request JSON: `{ "sql": "...", "actor"?, "model_id"?, "prompt_summary"? }` only. `serve` locks `--policy` / `--catalog` / `--database` / environment at startup; body overrides of those fields are **rejected**. + +Response always includes `action` (`ALLOW` \| `BLOCK` \| `REQUIRE_APPROVAL`), `rule_id`, `reason`, `risk_score`, `risk_factors`, `executed`. Treat anything other than `ALLOW` as non-executing. + +### Trust boundary (HTTP `serve`) — P0 + +- **Server-locked at startup.** `sql-write-gate serve --policy/--catalog/--database` (plus environment from the locked policy) is bound for the process lifetime. Request bodies may only supply `sql` / `actor` / `model_id` / `prompt_summary`; overrides of `policy` / `catalog` / `database` / `db_path` / `environment` return `400 trust_boundary_violation`. +- **Auth for non-loopback.** Binding `127.0.0.1` / `::1` may omit auth. Non-loopback hosts (including `0.0.0.0` / `::`) **require** `--auth-token` or `SQL_WRITE_GATE_HTTP_TOKEN`; all-interfaces without auth is refused at startup. Present `Authorization: Bearer ` or `X-SQLGuard-Token`. +- **GitHub Release lags main.** Package / `main` tracks the latest commit; GitHub **Latest Release** may lag. Prefer install-from-main / pin the tip SHA for suite acceptance until a matching Release is cut. + +### GameStream-style permissions + +```yaml +permissions: + enforce: true + tables: + orders: [select, insert, update] +hallucination: + allow_unknown_tables: false + allow_unknown_columns: false +``` + +Python alias: `from write_gate import sqlguard` (product helpers); package/CLI names unchanged. + +## Policy (default production) + +| operation | rule | +|-----------|------| +| select | allow | +| insert | approval | +| update | approval | +| delete | block | +| ddl | block | + +Limits: `update_rows: 100`, `delete_rows: 50`. Demo policy (`examples/policy.demo.yaml`) allows insert/update for walkthroughs. + +Guards (any **BLOCK** wins, else any **APPROVAL**, else **ALLOW**): + +`destructive` → `schema` → `pii` → `freshness` → `blast_radius` → `environment` + +## Stable interfaces + +Public surfaces for SemVer (see [compatibility.md](compatibility.md)): + +### CLI (public commands) + +`check` · `exec` · `hook` · `mcp` · `proxy` · `approve` · `resolve` · `reject` · `pending` · `audit` · `init` + +### Python API + +```python +from write_gate import WriteGate, Decision, Evidence # Evidence is Decision alias + +with WriteGate(database="postgresql://…") as gate: + decision = gate.check("DELETE FROM orders WHERE order_id = 1") + decision, result = gate.execute("SELECT 1") + decision, result = gate.approve(approval_id) # trusted executor + token env + gate.reject(approval_id) +``` + +Key methods: `check`, `execute`, `approve`, `reject`, `close` / context manager. + +### Decision JSON fields (`Decision.to_dict()` / `--json`) + +| Field | Type | Notes | +|-------|------|-------| +| `allowed` | bool | True only for `ALLOW` | +| `action` | str | `ALLOW` \| `BLOCK` \| `REQUIRE_APPROVAL` | +| `risk` | str | `low` \| `medium` \| `critical` | +| `rule_id` | str | e.g. `ok`, `delete_without_where`, `unsupported_sql` | +| `reason` / `message` | str | Human-readable (same text) | +| `evidence` | object | Guard evidence map | +| `sql` | str | Evaluated statement | +| `operation` | str \| null | `select` / `insert` / `update` / `delete` / `ddl` | +| `table` | str \| null | Primary table when known | +| `estimated_rows` | int \| null | Blast-radius estimate | +| `approval_id` | str \| null | When queued for approval | +| `rows` / `rowcount` / `truncated` | optional | Present on execute/approve `--json` when materializing | + +## Decision model + +`ALLOW` | `BLOCK` | `REQUIRE_APPROVAL` with `risk`, `rule_id`, `reason`, `evidence`. + +## Deployment model (trusted executor) + +| Concern | Where it lives | +|---------|----------------| +| DB credentials (`DATABASE_URL` / …) | **Trusted executor** only | +| `policy.yaml` / catalog | **Trusted executor** (agents have no rewrite API) | +| `approve` / `resolve` / `reject` | **Trusted executor** with approval token | +| `check` / `hook` / MCP `query_sql`/`write_sql` | Agent-facing: evaluate / enqueue only | + +### Approval privilege (`SQL_WRITE_GATE_APPROVAL_TOKEN`) + +1. On the trusted executor, create a secret file (default `.logs/approval.key`, or set `SQL_WRITE_GATE_APPROVAL_KEY_FILE`). +2. When calling `approve` / `resolve` / `reject`, set env `SQL_WRITE_GATE_APPROVAL_TOKEN` to that file's contents. +3. Missing key file, missing token, or wrong token → **refuse** (CLI exit non-zero). Correct token → approve/resolve/reject proceeds. +4. Agents must **not** receive the key file or token. They may still enqueue `REQUIRE_APPROVAL` via normal write paths. + +### Target binding + +Approval records store `database_config_id` (fingerprint). Approve reconnect binds trusted credentials only for the **same** target. Queue against DB A then change env to DB B → approve **fail closed** (will not write to B). + +## Boundaries (non-goals) + +- **非生产唯一边界 / 非唯一边界** — combine with least-privilege DB roles, network isolation, and human workflows +- Not a distributed approval lock, MySQL wire-protocol proxy, or Web UI +- Not an enterprise DQ / lineage / ChatBI / multi-tenant platform +- Not new cloud warehouses beyond the declared DuckDB / PostgreSQL / MySQL / SQLite matrix + +## Ops knobs + +| Env | Default | Purpose | +|-----|---------|---------| +| `SQL_WRITE_GATE_STATEMENT_TIMEOUT_SEC` | `0` (off) | Wall-clock statement timeout for check/execute/approve (caller returns at deadline; abandoned worker may still run → indeterminate/`unknown`; no blind retry) | +| `SQL_WRITE_GATE_RESULT_ROW_LIMIT` | `1000` | Cap SELECT/approve rows (truncate + `truncated=true`) | +| `SQL_WRITE_GATE_RESULT_BYTE_LIMIT` | `0` (off) | Hard byte cap on materialized rows (payload ≤ limit, or `ResultOversizeError` when `RESULT_OVERSIZE=block`; oversized single row never returned intact) | +| `SQL_WRITE_GATE_RESULT_OVERSIZE` | `truncate` | `truncate` (shrink/omit to keep ≤ byte/row caps) or `block` (`ResultOversizeError`) | +| `SQL_WRITE_GATE_AUDIT_SQL_MODE` | `redact` | Audit SQL storage: `redact` (literal scrub, default), `hash` (`sha256:…`), or `plain` (verbatim) | +| `SQL_WRITE_GATE_HTTP_TOKEN` | (optional on loopback) | Bearer token for `serve`; **required** for non-loopback / `0.0.0.0` binds | +| `SQL_WRITE_GATE_AUDIT_MAX_BYTES` | `10 MiB` | Rotate audit / approvals JSONL by size | +| `SQL_WRITE_GATE_AUDIT_ROTATE_DAILY` | `false` | Also rotate JSONL per UTC day | +| `SQL_WRITE_GATE_REQUEST_ID` | auto uuid4 | Audit correlation id | +| `SQL_WRITE_GATE_EXECUTING_TTL_SEC` | `120` | Stuck `executing` → `unknown` | +| `SQL_WRITE_GATE_APPROVAL_TOKEN` | (required for approve) | Presenter token | +| `SQL_WRITE_GATE_APPROVAL_KEY_FILE` | `.logs/approval.key` | Trusted-executor key path | + +See also: [install.md](install.md), [support-matrix.md](support-matrix.md), [troubleshooting.md](troubleshooting.md), [CHANGELOG.md](../CHANGELOG.md). diff --git a/docs/install.md b/docs/install.md new file mode 100644 index 0000000..05789f6 --- /dev/null +++ b/docs/install.md @@ -0,0 +1,38 @@ +# Install + +**非生产唯一边界 / 非唯一边界** — pilot-ready on the declared support matrix; **not** the sole production DB security boundary. + +```bash +pip install sql-write-gate +pip install 'sql-write-gate[postgres]' # optional: psycopg +pip install 'sql-write-gate[mysql]' # optional: pymysql +pip install 'sql-write-gate[mcp]' # optional: MCP server +``` + +From a clone: + +```bash +pip install -e ".[dev]" # or: make install +pip install -e ".[postgres,mysql]" +make test # PG/MySQL live tests skip if no service +make seal # 4 core ALLOW/BLOCK cases +``` + +```bash +sql-write-gate check "DELETE FROM users" +# → BLOCKED rule=delete_without_where +``` + +## Try it + +```bash +make install && make test && make seal # or: make demo +sql-write-gate check "DELETE FROM orders" +# → BLOCKED delete_without_where +sql-write-gate datapilot --json "SELECT o.order_id FROM orders o CROSS JOIN orders p" +# → {"datapilot": "BLOCK", "rule_id": "cartesian_join", ...} +sql-write-gate serve --port 8787 +# POST /v1/check {"sql": "..."} → action + risk_score + datapilot +``` + +See also: [support-matrix.md](support-matrix.md), [api.md](api.md), [troubleshooting.md](troubleshooting.md). diff --git a/docs/support-matrix.md b/docs/support-matrix.md new file mode 100644 index 0000000..1658abf --- /dev/null +++ b/docs/support-matrix.md @@ -0,0 +1,98 @@ +# Support matrix + +**非生产唯一边界 / 非唯一边界** — pilot-ready on DuckDB / PostgreSQL / MySQL / SQLite + listed SQL / entrypoints. Anything not listed is **out of scope** for v1.0 (`unsupported_sql`, fail-closed). + +## Declared databases + +| Backend | How to connect | Notes | +|---------|----------------|-------| +| **DuckDB** | file path / default `seed/warehouse.duckdb` | Default local warehouse | +| **PostgreSQL** | `POSTGRES_URL` or `postgresql://…` / `postgres://…` | Extra: `sql-write-gate[postgres]` | +| **MySQL** | `MYSQL_URL` or `mysql://…` / `mysql+pymysql://…` | Extra: `sql-write-gate[mysql]` | +| **SQLite** | `sqlite:///` / `sqlite+aiosqlite://` (incl. `C:/…`) | stdlib | + +Priority: `database=` → `database_url=` → `db_path=` → `POSTGRES_URL` → `MYSQL_URL` → `DATABASE_URL` → DuckDB default. + +## SQL support matrix + +| Supported (gated) | Explicitly rejected (`unsupported_sql` BLOCK) | +|-------------------|-----------------------------------------------| +| Single-statement `SELECT` / `INSERT` / `UPDATE` / `DELETE` | Multi-statement scripts (`stmt1; stmt2`) | +| DuckDB / PostgreSQL / MySQL / SQLite dialects via adapters | `MERGE` / `COPY` / `REPLACE` / raw `Command` | +| Simple CTEs over read-only SELECT | Data-modifying CTE / nested DML under any root | +| UPSERT `ON CONFLICT DO UPDATE` (PII/restricted on SET cols) | PostgreSQL `SELECT … INTO` | +| Catalog-backed schema / PII / freshness / blast-radius | Ambiguous or unlisted write-shaped SQL | + +**Anything not listed on the supported side → `unsupported_sql`** (BLOCK/REJECT, fail closed). Never silent ALLOW. + +## What it does (on the matrix) + +- `DROP` / `TRUNCATE` / `ALTER` → BLOCK +- `DELETE` / `UPDATE` without `WHERE` (incl. tautology `WHERE 1=1` / `TRUE`) → BLOCK +- Cartesian / missing-predicate JOINs → BLOCK (`cartesian_join`) +- Unknown tables/columns → BLOCK (`schema_hallucination`) with evidence +- GameStream-style `permissions.tables` allowlists in `policy.yaml` +- Numeric `risk_score` (0–100) + `risk_factors` on every Decision +- Blast-radius COUNT vs `update_rows` / `delete_rows` (dialect quoting; fail-closed on estimate error) +- Schema / PII / restricted columns; PII `SELECT` → REQUIRE_APPROVAL (approve executes once) +- Freshness partitions (`dt`); range / NOT / OR / UPSERT SET expired → BLOCK +- Nested / data-modifying CTE / `SELECT INTO` → REJECT (`unsupported_sql`) +- Approval state machine (SQLite source of truth + JSONL mirror): `pending`→`executing`→`succeeded`|`failed`|`unknown` (+ `rejected`) +- Atomic claim under `fcntl.flock` + SQLite `BEGIN IMMEDIATE` (single-host; fail closed without flock) +- Three-state execute outcomes; **`unknown`/`executing` never auto-retried** — use `resolve` or `approve --allow-unknown-retry` after manual DB verify +- JSONL audit (redacts URL passwords; **SQL literals redacted by default** via `SQL_WRITE_GATE_AUDIT_SQL_MODE=redact|hash|plain`; records execute failures / unknown; `request_id` + `approval_id` + `execution_outcome` correlation; rotatable) + +## Platform support matrix + +| Surface | Linux / macOS | Windows | +|---------|---------------|---------| +| `pip install` / CLI `check` / `init` / `audit` | ✅ | ✅ | +| DuckDB file backend | ✅ | ✅ | +| SQLite `sqlite:///` paths (incl. `C:/…`) | ✅ | ✅ | +| Postgres / MySQL URL adapters | ✅ | ✅ (drivers via extras) | +| PreToolUse hook / MCP stdio | ✅ | ✅ (same Python entrypoints) | +| Concurrent `approve` (flock + SQLite claim) | ✅ | ❌ **fail closed** — `ApprovalError` if `fcntl.flock` unavailable (no silent unlock) | + +Windows: install, CLI evaluate/execute on DuckDB/SQLite/URL backends work. Approval mutations require Unix `fcntl` flock (plus SQLite transactions); without flock they **refuse** rather than silently degrading. + +## Approval outcomes & crash recovery + +| Status | Meaning | Default `approve` | +|--------|---------|-------------------| +| `pending` | Queued; not executed | Claims → executes | +| `executing` | Claim held (in flight) | **Refuse** (no steal) | +| `succeeded` | DB write/query completed | Idempotent; **no re-write** | +| `failed` | Known not committed / never sent | May reclaim & retry | +| `unknown` | Timeout/disconnect/crash/indeterminate | **Refuse** — never auto-retry | +| `rejected` | Human rejected | Refuse | + +Recovery rules: + +1. Process crash while `executing`: after TTL (`SQL_WRITE_GATE_EXECUTING_TTL_SEC`, default 120s) → `unknown` (via `approve --force-unknown-check` / next store access). **Never** silent re-claim that re-runs SQL. +2. Operator path for `unknown`: verify target DB manually, then either + - `sql-write-gate resolve --as succeeded|failed|rejected` (no SQL), or + - `sql-write-gate approve --allow-unknown-retry` (explicit re-exec; double-write risk). +3. Default second `approve` on `succeeded` / `unknown` does **not** write again. + +## Database URLs + +| Env / kwarg | Backend | +|-------------|---------| +| `POSTGRES_URL` or `postgresql://…` / `postgres://…` | PostgreSQL | +| `MYSQL_URL` or `mysql://…` / `mysql+pymysql://…` | MySQL | +| `DATABASE_URL` (scheme-detected) | Postgres / MySQL / SQLite | +| `sqlite:///` / `sqlite+aiosqlite://` | SQLite (stdlib) | +| file path / default `seed/warehouse.duckdb` | DuckDB | + +### Live integration tests (optional locally) + +```bash +export POSTGRES_URL=postgresql://gate:gate@localhost:5432/writegate +export MYSQL_URL=mysql://gate:gate@127.0.0.1:3306/writegate +pip install -e ".[dev,postgres,mysql]" +make test +``` + +Without those services, live tests **skip**; CI runs Postgres + MySQL service containers. + +See also: [install.md](install.md), [api.md](api.md), [troubleshooting.md](troubleshooting.md). diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 6bafd8c..fe8c71e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -10,7 +10,7 @@ | `REQUIRE_APPROVAL` then nothing writes | INSERT/UPDATE / PII SELECT needs human approve | On the **trusted executor**, run `sql-write-gate approve ` with the approval token | | `TrustError` / missing approval key | Approve/resolve/reject without trust | Create `.logs/approval.key` (or set `SQL_WRITE_GATE_APPROVAL_KEY_FILE`) and export matching `SQL_WRITE_GATE_APPROVAL_TOKEN` | | `approval target mismatch` | Queue on DB A, approve with env pointing at DB B | Restore the original `DATABASE_URL` / `POSTGRES_URL` / `MYSQL_URL` for that fingerprint; never swap targets | -| `unsupported_sql` | Multi-statement, MERGE/COPY, nested DML, SELECT INTO, etc. | Rewrite to a single supported statement (see README SQL support matrix) | +| `unsupported_sql` | Multi-statement, MERGE/COPY, nested DML, SELECT INTO, etc. | Rewrite to a single supported statement (see [support-matrix.md](support-matrix.md)) | | `blast_radius_exceeded` / `blast_radius_unknown` | UPDATE/DELETE touches too many rows, or COUNT failed | Narrow the WHERE; fix connectivity before retrying writes | | `StatementTimeoutError` / `statement_timeout` | Statement exceeded `SQL_WRITE_GATE_STATEMENT_TIMEOUT_SEC` | Raise the timeout, optimize SQL, or split work; see **Timeouts** below | | Result `truncated=true` / `ResultOversizeError` | SELECT / approve hit row/byte cap (hard byte limit; single oversized row not returned intact) | Raise limits, page the query, or set `RESULT_OVERSIZE=block` for hard reject |