diff --git a/.claude/rules/cross-language-porting.md b/.claude/rules/cross-language-porting.md index fb479ac9e1..630ae79ac2 100644 --- a/.claude/rules/cross-language-porting.md +++ b/.claude/rules/cross-language-porting.md @@ -48,7 +48,7 @@ Preserve the following contracts exactly across all language ports: - Overlay/merge: same `package` + same object `name` across multiple files → merged. Last-writer-wins on attr conflicts; structural children accumulate. - Default scan path: `metaobjects/**/*.json` (recursive). -**D1 is TS-only.** Cloudflare D1 is a peer of `sqlite`/`postgres` in TS's `dialect` vocabulary. It is SQLite at the SQL level — Java/Python/C# don't have an analogue (Cloudflare Workers run JS). When adding cross-language vocabulary, D1 doesn't constrain anything: its uniqueness is wrangler-CLI transport + Wrangler-native file layout (`migrations/_.sql`), both of which are TS-only concerns. +**D1 is TS-only.** Cloudflare D1 is a peer of `sqlite`/`postgres` in `meta migrate`'s `dialect` vocabulary (`migrate.dialect`, `--dialect d1`); codegen's `dialect` is `sqlite`/`postgres`/`mysql` and D1 uses `sqlite` (`"d1"` there is an error). It is SQLite at the SQL level — Java/Python/C# don't have an analogue (Cloudflare Workers run JS). When adding cross-language vocabulary, D1 doesn't constrain anything: its uniqueness is wrangler-CLI transport + Wrangler-native file layout (`migrations/_.sql`), both of which are TS-only concerns. **Constants discipline:** - TS: named constants, imported from `@metaobjectsdev/metadata/constants` (defined in 16 per-concern `*-constants.ts` modules under `packages/metadata/src/`, e.g. `core/field/field-constants.ts`, `persistence/db/db-constants.ts`). Never inline metamodel strings as literals in code. diff --git a/CHANGELOG.md b/CHANGELOG.md index 68b5a5ae8f..e75772e92f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -235,6 +235,29 @@ here.** the command and rolls the sync back. A stale `docs/CONFORMANCE.md` cell that claimed Python passed only 2 of the dependency corpus's cases is corrected. +- **Generated routes no longer accept or return fields they must not (#484).** A field marked + `@formExclude`, or a credential (a field that declares a `view.password`), is now **ignored** by + the generated create, PATCH and PUT in every port: the value is dropped, not rejected, so + an edit form that submits every field still saves, and a present `null` no longer clears it. A credential is + also never returned: not by list, get, or the echo of a write, or by a served report that groups by + it or takes its `min` or `max` (a report column is derived from its `@of` field, and it keeps the + credential mark; a report export names no column for it, and that column is not filterable or sortable). Before this, `@formExclude` kept + a field out of the generated form while the route still wrote it, and a generated read returned password + hashes. Generated forms (React, Angular) and the api docs omit credentials too. Your own code is unaffected: + the entity, DTO and generated queries still carry the field, so you keep writing a hash or token yourself. + A project with neither mark generates byte-identical output. If you relied on a route writing a + `@formExclude` field, write it from your own code. A field that is `@required` and also + `@formExclude` or a credential now makes the generated create route fail, because the route + can no longer supply its value: make the field optional, or create that entity from your own + code (see "Credentials and excluded fields" in `docs/features/api-contract.md`). Gated by the new `secret-fields/` sub-corpus + of the api-contract conformance corpus, in the generated lane of all five ports. + +- **`meta gen` and `meta verify --codegen` refuse an unknown codegen `dialect`** instead of silently + generating Postgres code. `dialect: "d1"` gets a message that says to use `"sqlite"` (D1 is + SQLite; `"d1"` names only the `meta migrate` transport). `meta init --d1` now scaffolds + `dialect: "sqlite"` in `metaobjects.config.ts`; it used to write `"d1"`. If your config has `dialect: "d1"`, + change it to `"sqlite"`. + - **The "Model / metadata" link in the Python, C#, Java and Kotlin API docs no longer points at a page that does not exist.** Those ports write no model pages, so the relative link was dead. It is now written only when `modelBaseUrl` is set. @@ -280,6 +303,12 @@ here.** ### Documentation +- **New recipes and a real D1 example.** `docs/recipes/` gains keeping drizzle-kit migrations + (`drizzle-kit-migrations.md`), guarding generated routes with your own auth + (`guarding-generated-routes.md`) and proving your `meta verify` gate can fail + (`prove-your-gate-fails.md`). The Cloudflare Workers recipe now has a per-request `drizzle(env.DB)` + example, and the `drizzle({} as never)` placeholder and the wrong `"d1"` codegen-dialect statements are corrected. + - **Two doc errors corrected against the code.** `docs/recipes/mysql.md` said Python ships a Postgres driver only; it ships SQLite and MySQL drivers beside it. `docs/features/api-contract.md` said only TypeScript enforces the `in`-list cap of 100; every port does. diff --git a/agent-context/skills/metaobjects-audit/references/capability-checklist.md b/agent-context/skills/metaobjects-audit/references/capability-checklist.md index 39d9a51cdd..e145d1ccf6 100644 --- a/agent-context/skills/metaobjects-audit/references/capability-checklist.md +++ b/agent-context/skills/metaobjects-audit/references/capability-checklist.md @@ -198,8 +198,10 @@ soft-delete / status / type view it models. and are NOT in the cross-port registry — `view.text`, `view.textarea`, `view.date`, `view.month`, `view.hotlink`, `view.image`, `view.dropdown`, `view.radio`, `view.checkbox`, `view.number`, `view.password`, `view.hidden`, `view.web`. Every port LOADS them, so shared metadata that - carries them is valid in a backend port; nothing outside TS/web reads them. **Audit these - only for TS adopters.** Only + carries them is valid in a backend port. The one exception is `view.password`: every port's generated + routes read it as a credential, so it is never returned and never written (see + "Credentials and excluded fields" in `docs/features/api-contract.md`). + Audit the other widgets only for TS adopters. Only `view.base` / `view.currency` are cross-port-gated. ## Template — `template.*` (prompt pillar) diff --git a/agent-context/skills/metaobjects-audit/references/typescript.md b/agent-context/skills/metaobjects-audit/references/typescript.md index d4b42126a0..3fc99245c4 100644 --- a/agent-context/skills/metaobjects-audit/references/typescript.md +++ b/agent-context/skills/metaobjects-audit/references/typescript.md @@ -199,4 +199,4 @@ TypeScript is the reference implementation — it ships the full feature set: - Output-parser codegen: **complete** (`outputParser()` generator; `parse*`/`safeParse*`/`extract*` emitted as `.response.ts` per **responding `template.prompt`** — one carrying `@responseRef`; ADR-0052). - ObjectManager: **complete** (`@metaobjectsdev/runtime-ts`; `kyselyDriver` / `inMemoryDriver`). - Schema migrate / `meta verify --db`: **complete** (TS owns the shared migrate engine — ADR-0015). -- D1 dialect: **TS-only by design** — `dialect: "d1"` in `metaobjects.config.ts` targets Cloudflare D1 via the Wrangler CLI layout. Do NOT recommend porting D1 support to other languages. +- D1 dialect: **TS-only by design** — `migrate.dialect: "d1"` (and `meta migrate --dialect d1`) targets Cloudflare D1 via the Wrangler CLI layout; codegen's `dialect` in `metaobjects.config.ts` is `"sqlite"` for D1 and `"d1"` there is refused. Do NOT recommend porting D1 support to other languages. diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index 20babfaa1b..bb2eb0f07a 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -36,7 +36,7 @@ regenerate with `ls -d fixtures//*/ | wc -l` for directory-shaped corpor | [`fixtures/extract-conformance/`](../fixtures/extract-conformance/) | 48 | ✓ | ✓ | inherits the shared JVM engine | ✓ | ✓ | | [`fixtures/output-prompt-conformance/`](../fixtures/output-prompt-conformance/) | 17 | ✓ | ✓ | ✓ | ✓ | ✓ | | [`fixtures/persistence-conformance/`](../fixtures/persistence-conformance/) | 59 (53 query + 6 migration) | all 59; the 24 `report-*` and the `-15` decimal scenarios also on SQLite, MySQL and D1, `decimal-exact` also on MySQL (a scenario's `engines:` key is exact) | 53 query (migrations TS-only, ADR-0015); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 53 query (via Exposed); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 53 query; the `report-*` and `engines:` scenarios also on SQLite and MySQL | 53 query (pg8000); the `report-*` and `engines:` scenarios also on SQLite (`sqlite3`) and MySQL (PyMySQL) | -| [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 117 (31 core + 10 tph + 9 m2m + 5 jsonb + 2 write-through + 12 projection + 37 report + 11 decimal) | ✓ (Fastify reference + generated lane); `report/` and `decimal/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC); `report/` and `decimal/` generated controllers over OMDB on Postgres, SQLite and MySQL | ✓ (embedded HTTP + Exposed); `report/` and `decimal/` also on SQLite and MySQL | ✓ (Kestrel + Npgsql); `report/` and `decimal/` also on SQLite and MySQL | ✓ (FastAPI + pg8000); `report/` and `decimal/` also on SQLite and MySQL | +| [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 119 (31 core + 10 tph + 9 m2m + 5 jsonb + 2 write-through + 2 secret-fields + 12 projection + 37 report + 11 decimal) | ✓ (Fastify reference + generated lane); `report/` and `decimal/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC); `report/` and `decimal/` generated controllers over OMDB on Postgres, SQLite and MySQL | ✓ (embedded HTTP + Exposed); `report/` and `decimal/` also on SQLite and MySQL | ✓ (Kestrel + Npgsql); `report/` and `decimal/` also on SQLite and MySQL | ✓ (FastAPI + pg8000); `report/` and `decimal/` also on SQLite and MySQL | | [`fixtures/decimal-wire/`](../fixtures/decimal-wire/) | 1 vector file (format, parse, filter operand, 15-digit read) | ✓ (`runtime-ts` and the generated helper) | ✓ (`CanonicalDecimal`) | ✓ (the generated wire file) | ✓ (`DecimalWire`) | ✓ (`decimal_wire`) | | [`fixtures/validation-conformance/`](../fixtures/validation-conformance/) | 42 cases | ✓ (generated Zod + run-time `runValidators`) | ✓ | ✓ | ✓ | ✓ (generated Pydantic + run-time `run_validators`) | | [`fixtures/registry-conformance/`](../fixtures/registry-conformance/) | 1 canonical manifest | ✓ (reference emitter) | ✓ | ✓ | ✓ | ✓ | @@ -365,9 +365,9 @@ REAL, a MySQL `DECIMAL` keeps its scale, a MySQL `DATETIME` is the UTC wall cloc `report-relative-date` carries a per-engine seed (`seed-data-engine`). `op: relate` and the write ops stay Postgres-only. See [features/reporting.md](features/reporting.md#what-the-corpus-gates). -### `fixtures/api-contract-conformance/` (117) +### `fixtures/api-contract-conformance/` (119) -All 117 scenarios → [features/api-contract.md](features/api-contract.md) (cross-port +All 119 scenarios → [features/api-contract.md](features/api-contract.md) (cross-port REST API URL grammar + JSON wire format). Verifies every backend's emitted CRUD routes answer identically over HTTP — list / get / create / patch+put / delete, plus pagination (`limit`/`offset`), sort (`sort=field:dir`), the `withCount=1` @@ -378,11 +378,11 @@ The corpus also covers the 9 cross-port filter operators (`eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `like`, `isNull`) plus the implicit-AND combinator and 2 error shapes (`invalid_filter_field` / `invalid_filter_op`) under the URL grammar `?filter[][]=` (FR-009). On top of the 31 core -scenarios the corpus carries seven sub-corpora — `tph/` (10, single-table +scenarios the corpus carries eight sub-corpora — `tph/` (10, single-table inheritance; a subtype decimal filters and refuses extra digits), `m2m/` (9 — 3 plain, 5 gating TPH x M:N together, the combination each corpus alone could not reach, and 1 pinning the collection-URL spelling), `jsonb/` (5, typed value-object columns; numbers inside them: a double stays a JSON number, a decimal is the canonical string), -`write-through/` (2, table-write + view-read entities), `projection/` (12, a +`write-through/` (2, table-write + view-read entities), `secret-fields/` (2, #484: a `@formExclude` or credential `view.password` field is ignored on create, PATCH and PUT, and a credential is never returned), `projection/` (12, a read-only view answers reads and refuses writes with 405; a projection with no declared identity has no item route; a projection keyed on a field not named `id` is addressed by it, whether its identity names `@fields` or derives them; decimal and float fields filter), `decimal/` (11, ADR-0019 Amendment 1: a decimal, array element and undeclared-precision column included, is one @@ -392,7 +392,7 @@ its derived fields, answers `POST` with 405 and mounts no `/{id}`; a `@spine` re a row with no facts, and a measure with `@default` filters and sorts as its default). All 5 ports — TS, Java, Kotlin, C#, Python — run it in BOTH lanes: a hand-rolled reference server and the port's own GENERATED API artifact booted over HTTP. `write-through/`, -`projection/`, `decimal/` and `report/` run the generated lane only, on all five ports: what +`secret-fields/`, `projection/`, `decimal/` and `report/` run the generated lane only, on all five ports: what they test is whether a port's generator emits the routes. The `report/` sub-corpus is the one that reads real views, so it runs full stack on Postgres, SQLite and MySQL (`report/schema.sqlite.sql`, `report/schema.mysql.sql`, both TypeScript-produced) in every @@ -606,7 +606,7 @@ files into a real Cube. ## Orphaned fixtures (tested but not yet documented) The fixtures in the nine corpora mapped above (metamodel 413 + yaml 16 + verify 37 -+ render 16 + persistence 59 + api-contract 117 + source-resolution 25 + scope 10 + ++ render 16 + persistence 59 + api-contract 119 + source-resolution 25 + scope 10 + dependency 37) each map to a feature doc, and so do the two requirement corpora, `shared-model-conformance` and the `cube-model` corpus, whose case lists live in their own READMEs. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider diff --git a/docs/README.md b/docs/README.md index 48dea43324..1f3a9dc2c5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -84,6 +84,9 @@ this tree is documentation, not the source of truth. | See a worked, non-toy model end to end (projections, value objects, TPH, prompt payloads on one runnable spine) | [`../examples/advanced-modeling/`](../examples/advanced-modeling/) | | Read the canonical spec (target-agnostic) | [`../spec/`](../spec/) | | Find a release recipe (Cloudflare D1, etc.) | [`recipes/`](recipes/) | +| Keep drizzle-kit migrations while MetaObjects owns the model | [`recipes/drizzle-kit-migrations.md`](recipes/drizzle-kit-migrations.md) | +| Put generated routes behind your own auth (Fastify, Hono, `expose`) | [`recipes/guarding-generated-routes.md`](recipes/guarding-generated-routes.md) | +| Show that your `meta verify` CI gate actually fails on drift | [`recipes/prove-your-gate-fails.md`](recipes/prove-your-gate-fails.md) | | See the deeper design rationale for a feature | [`superpowers/specs/`](superpowers/specs/) | ## Conventions used in these docs diff --git a/docs/features/api-contract.md b/docs/features/api-contract.md index 91cbcf3fe5..4f3f91f9f3 100644 --- a/docs/features/api-contract.md +++ b/docs/features/api-contract.md @@ -529,6 +529,34 @@ log-only. | Kotlin | shipped — `metaobjects-codegen-kotlin` `KotlinSpringControllerGenerator` → Spring `@RestController` | One controller per writable entity (`source.rdb @kind="table"`) plus a READ-ONLY one per view-kind `object.projection` (F22, see "Read-only projections" below); 5 CRUD endpoints (GET list / GET by id / POST / PATCH+PUT / DELETE); `?sort`, `?limit/?offset`, `?withCount=1` envelope, 404 + 400 envelopes per the contract. Filter operators ship via the generated `FilterAllowlist` (`KotlinFilterAllowlistGenerator`) + an inline `parseFilter` helper emitted in the controller. | | Python | shipped — `metaobjects.codegen.generators.router_generator` → FastAPI `APIRouter` | One router per writable entity (`source.rdb @kind="table"`) plus a READ-ONLY one per view-kind `object.projection` (F22, see "Read-only projections" below); 5 CRUD endpoints (GET list / GET by id / POST / PATCH+PUT / DELETE); `?sort`, `?limit/?offset`, `?withCount=1` envelope, 404 + 400 envelopes per the contract. Consumer wires the repository via FastAPI `app.dependency_overrides`; the generator emits a `Protocol` interface so the persistence layer (SQLAlchemy / asyncpg / etc.) is the consumer's choice. Filter operators ship via the generated `_filter_allowlist.py` (`filter_allowlist_generator.py`) + the shared `filter_parser` helper, wired into the list handler. | +## Credentials and excluded fields + +Two marks narrow what the generated routes accept and send, in every port (#484): + +| Mark | Create, PATCH, PUT | List, get, and the echo of a write | +|---|---|---| +| `@formExclude: true` | the field is **ignored** | returned | +| a child `view.password` (a credential) | the field is **ignored** | **never returned** (the key is absent, not `null`) | + +A value for an ignored field is dropped, never rejected: the request answers `200` or `201` as if the key +were absent, and a present `null` does not clear it. That is the rule for every key outside the settable +set (`@mutability: writeOnce` on PATCH, the TPH discriminator, `@autoSet: onCreate`), and it is load-bearing: +a generated edit form submits every field it was given, so a `400` would fail every save. A required field +that is also ignored can never be created through the API, so mark such a field optional. + +Only the generated HTTP boundary changes. The entity, its DTO and the generated queries still carry the +field, so your own code writes a hash or token and reads it back. Allowlists are unchanged: a field is +filterable or sortable only if you wrote `@filterable` or `@sortable` on it, so do not put those on a credential +(a filter on a secret answers whether a guess is right). To return a credential from a route, hand-write that +route; the generated one will not. Generated forms and the api docs omit both kinds of field as well. + +A served report is covered too. A report column is derived from its `@of` field, and a derived column keeps +that field's credential mark, so a report that groups by a credential (or takes its `min` or `max`) never +returns the column, its `report-export` output has no column for it, and it is neither filterable nor sortable +(a report's other columns all are, so this one is taken out). `count` over a credential is a number and is returned. + +`fixtures/api-contract-conformance/secret-fields/` pins this in the generated lane of every port. + ## Read-only projections **All five ports serve an `object.projection` whose only source is a read-only diff --git a/docs/features/reporting.md b/docs/features/reporting.md index fa235d796c..ab2140965b 100644 --- a/docs/features/reporting.md +++ b/docs/features/reporting.md @@ -299,7 +299,9 @@ A derived column carries the type-shaping attributes of its `@of` field where th (`@currency`, `@values`, `@intValueMap`, `@maxLength`, `@precision`, `@scale`, `@localTime`, `@objectRef`, `@storage`, `@dbColumnType`, `isArray`) and nothing else: no `@column`, no `@required` beyond the rules above, no `@default` (a measure's `@default` is applied in the -view, not carried), no validators. +view, not carried), no validators. The one view it does carry is a credential: a column whose +`@of` field declares or inherits `view.password` is itself a credential, so the served list route +omits it, an export has no column for it, and it is not filterable or sortable (#484). ### Measures diff --git a/docs/ports/typescript.md b/docs/ports/typescript.md index c251073a4a..e89dd15c8f 100644 --- a/docs/ports/typescript.md +++ b/docs/ports/typescript.md @@ -50,6 +50,9 @@ export default defineConfig({ }); ``` +Already on drizzle-kit and want to keep its migrations? Re-export the generated tables from your own +schema file: [drizzle-kit migrations](../recipes/drizzle-kit-migrations.md). + A model of only value objects and sourceless projections needs no `dialect`. A sourceless object generates its type and nothing else: no table, no migration, no routes. See [Own your codegen](../features/own-your-codegen.md) for ejecting a generator, and the @@ -149,7 +152,7 @@ import { entityFile, queriesFile, routesFile, barrel, docsFile, apiDocsFile } fr export default defineConfig({ outDir: "src/generated", - dialect: "postgres", // "postgres" | "sqlite" | "d1" + dialect: "postgres", // "postgres" | "sqlite" | "mysql" (Cloudflare D1: "sqlite") extStyle: "js", // ".js"-extensioned imports — nodenext- AND bundler-safe ("none" to opt out) apiPrefix: "/api", dbImport: "../db", // the module routesFile() imports `db` from — yours to write @@ -743,7 +746,7 @@ import { entityFile, queriesFile, routesFileHono, barrel } from "@metaobjectsdev export default defineConfig({ outDir: "src/generated", - dialect: "sqlite", // or "postgres" / "d1" + dialect: "sqlite", // or "postgres" / "mysql". Cloudflare D1 is "sqlite"; "d1" is refused extStyle: "js", apiPrefix: "/api", generators: [entityFile(), queriesFile(), routesFileHono(), barrel()], @@ -783,7 +786,8 @@ export function registerAuthorRoutes( } ``` -Consumer wiring (Workers example): +Consumer wiring (Workers example). The routes take `db` when they are mounted, and a D1 +binding exists only on `env`, so build the client and mount inside `fetch`: ```ts import { Hono } from "hono"; @@ -792,11 +796,20 @@ import { registerAuthorRoutes } from "./generated/Author.routes.hono.js"; interface Env { DB: D1Database } -const app = new Hono<{ Bindings: Env }>(); -registerAuthorRoutes(app, { db: drizzle({} as never) }); // replace with c.env.DB-derived db at request time -export default app; +export default { + async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise { + const app = new Hono<{ Bindings: Env }>(); + registerAuthorRoutes(app, { db: drizzle(env.DB) }); // this request's binding + return app.fetch(request, env, ctx); + }, +}; ``` +The full Workers recipe (wrangler binding, queries, why `dialect` is `"sqlite"` for D1) is in +[Cloudflare Workers](../recipes/cloudflare-workers.md#generated-routes-and-queries-on-a-worker-per-request). +To put these routes behind your own authentication, see +[Guarding generated routes](../recipes/guarding-generated-routes.md). + The Hono flavor differs from Fastify in two intentional ways, both reflecting Hono idioms rather than contract drift: diff --git a/docs/recipes/cloudflare-workers.md b/docs/recipes/cloudflare-workers.md index e8264caf52..7243d1a51c 100644 --- a/docs/recipes/cloudflare-workers.md +++ b/docs/recipes/cloudflare-workers.md @@ -17,13 +17,13 @@ is V8-isolate-safe. The table below is the authoritative split: | `@metaobjectsdev/codegen-ts` | Build-time only | Same. | | `@metaobjectsdev/migrate-ts` | Build-time only | Migrations emit SQL, then `wrangler d1 migrations apply` runs them. | | `@metaobjectsdev/render` | ✅ Runtime-safe | Pure JS, no Node-only deps. Used at request time. | -| `@metaobjectsdev/runtime-ts` | ❌ Node-only | Fastify + Kysely drivers. Not for Workers. | +| `@metaobjectsdev/runtime-ts` | ⚠️ Only the `/hono` entry | The Fastify and Kysely entry points are Node-only. The `/hono` entry is what generated Hono routes import. | | `@metaobjectsdev/runtime-web` | ✅ Runtime-safe | Pure browser/edge JS. | | `@metaobjectsdev/react` | ✅ Runtime-safe | Browser; works in Workers if you render React on the edge. | | `@metaobjectsdev/tanstack` | ✅ Runtime-safe | Browser-side. | | Generated `.entity.ts` / `.queries.ts` | ✅ Runtime-safe | The generated code is the runtime; uses `drizzle-orm/d1`. | | Generated `.routes.ts` (Fastify, default) | ❌ Node-only | The handlers are Workers-safe but the Fastify adapter from `runtime-ts` is not. Use the Hono generator instead. | -| Generated Hono routes (`routesFileHono()`) | ✅ Runtime-safe | The opt-in Workers routes generator — emits Hono handlers with no `runtime-ts` dependency. Wire it in `metaobjects.config.ts` in place of `routesFile()`. | +| Generated Hono routes (`routesFileHono()`) | ✅ Runtime-safe | The opt-in Workers routes generator. It imports only the `@metaobjectsdev/runtime-ts/hono` entry point (Hono, Drizzle, Zod; no Fastify, Kysely or `node:` import in that module graph). Wire it in `metaobjects.config.ts` in place of `routesFile()`. | ## Project layout @@ -123,42 +123,101 @@ agree (default `migrations/` matches Wrangler's expectation). For the full D1 dialect documentation, see the [`@metaobjectsdev/migrate-ts` README](../../server/typescript/packages/migrate-ts/README.md). -## Skipping the routes generator +## Codegen dialect is `sqlite`, not `d1` -Workers consumers typically skip the routes generator entirely — Fastify-based -routes don't run in V8 isolates. Hand-write your `fetch` handler against the -generated queries: +D1 is SQLite. In `metaobjects.config.ts` set `dialect: "sqlite"`: the generated tables are +`sqliteTable`, timestamps are ISO strings, and the routes use SQLite `RETURNING`. `"d1"` names +only the **`meta migrate` transport** (`migrate.dialect` in `.metaobjects/config.json`, +`--dialect d1`), which shells out to Wrangler. `meta gen` and `meta verify --codegen` refuse any +codegen dialect other than `sqlite`, `postgres` or `mysql`, and say so for `"d1"`, rather than +quietly generating Postgres. + +## Generated routes and queries on a Worker, per request + +Use `routesFileHono()` (not the Fastify `routesFile()`) and `queriesFile()`: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; +import { entityFile, queriesFile, routesFileHono, barrel } from "@metaobjectsdev/codegen-ts/generators"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "sqlite", // D1 is SQLite + extStyle: "js", + apiPrefix: "/api", + generators: [entityFile(), queriesFile(), routesFileHono(), barrel()], +}); +``` + +Neither generated module imports a module-level `db`. Both take it as an argument: + +- `Note.queries.ts` exports `listNotes(db, opts?)`, `findNoteById(db, id)`, `createNote(db, ...)` + and so on, each with `db` as the first parameter. +- `Note.routes.hono.ts` exports `registerNoteRoutes(app, { db })`. The `db` is captured when the + routes are **mounted**, not looked up per call. + +A D1 binding only exists on `env`, inside `fetch`, so build the Drizzle client there and mount +the generated routes on a Hono app for that request: ```ts // src/worker.ts +import { Hono } from "hono"; import { drizzle } from "drizzle-orm/d1"; -import { findUserById, listUsers } from "./generated/User.queries"; -import * as schema from "./generated/schema"; +import { registerNoteRoutes } from "./generated/Note.routes.hono.js"; +import { listNotes } from "./generated/Note.queries.js"; + +interface Env { + DB: D1Database; // from @cloudflare/workers-types +} export default { - async fetch(request: Request, env: Env): Promise { - const db = drizzle(env.DB, { schema }); - const url = new URL(request.url); + async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise { + const db = drizzle(env.DB); // one client per request - if (url.pathname.startsWith("/api/users/")) { - const id = url.pathname.slice("/api/users/".length); - const user = await findUserById(db, id); - return Response.json(user); - } - if (url.pathname === "/api/users") { - return Response.json(await listUsers(db, {})); + const url = new URL(request.url); + if (url.pathname === "/health") { + return Response.json({ notes: (await listNotes(db, {})).length }); // queries: plain calls } - // Static assets handled by Workers Static Assets binding. - return env.ASSETS.fetch(request); + + const app = new Hono<{ Bindings: Env }>(); + registerNoteRoutes(app, { db }); // routes: mounted with this request's db + return app.fetch(request, env, ctx); }, }; +``` -interface Env { - DB: D1Database; - ASSETS: Fetcher; -} +Registering the five routes is cheap, so doing it per request is fine. If you would rather mount +once, build the app lazily on the first request (`let app: Hono | undefined`) and keep it for the +isolate: the binding on `env` does not change for the life of the deployed version. Do not +mount at module scope, where `env` does not exist yet, and do not pass a placeholder such as +`drizzle({} as never)`: it type-checks and then fails on the first query. + +Add your guard before the mount: [Guarding generated routes](guarding-generated-routes.md). + +Wrangler needs only the binding the code reads: + +```toml +# wrangler.toml +name = "myapp" +main = "./src/worker.ts" +compatibility_date = "2026-01-01" +compatibility_flags = ["nodejs_compat"] + +[[d1_databases]] +binding = "DB" # env.DB above +database_name = "myapp-prod" +database_id = "" +migrations_dir = "migrations" # where `meta migrate --dialect d1` writes ``` -> Note: `findUserById(db, id)` parameter-passing landing in 0.7.0 (FR2). -> If you're on 0.6.x today, the generated queries import a module-level `db` -> from `../db` instead — see the [0.6.0 release notes](../../CHANGELOG.md) for migration guidance. +### What was checked + +Generated from a one-entity model with `dialect: "sqlite"`, then compiled with `tsc --strict` +(`module: nodenext`) together with the `worker.ts` above against the real `drizzle-orm/d1`, +`hono` and `@metaobjectsdev/runtime-ts` types. `D1Database` and `ExecutionContext` came from a +local stub in place of `@cloudflare/workers-types`, which that environment does not install. +The same per-request mount (a new Hono app and `registerNoteRoutes` per request) was run against +an in-memory libSQL database, which shares D1's SQLite dialect: `POST /api/notes` answered 201 +and `GET /api/notes` answered 200 with the row. D1 itself, `wrangler dev` and a deployed Worker +were not exercised. diff --git a/docs/recipes/drizzle-kit-migrations.md b/docs/recipes/drizzle-kit-migrations.md new file mode 100644 index 0000000000..b13bb998f5 --- /dev/null +++ b/docs/recipes/drizzle-kit-migrations.md @@ -0,0 +1,91 @@ +# Recipe: keep drizzle-kit as your migration tool + +You already run `drizzle-kit generate` / `drizzle-kit migrate`, and you want MetaObjects to own +the **model** (entities, Zod schemas, queries, routes) without taking over your **migrations**. +That is a supported split: the TypeScript `entity` generator emits ordinary Drizzle tables +(`pgTable`, `sqliteTable` or `mysqlTable`, matching `dialect`), and drizzle-kit only needs a +schema file that exports them. `meta migrate` never has to run. + +## Wire it + +1. Generate as usual. `entityFile()` writes the table beside its Zod schemas: + + ```ts + // metaobjects.config.ts + import { defineConfig } from "@metaobjectsdev/cli"; + import { entityFile, queriesFile, barrel } from "@metaobjectsdev/codegen-ts/generators"; + + export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", // must match the database drizzle-kit migrates + generators: [entityFile(), queriesFile(), barrel()], + }); + ``` + +2. Write **your own** schema file and re-export the generated tables from it. This file is + yours: `meta gen` never touches it, so it is where you also add hand-written tables that are + not in the model. + + ```ts + // src/db/schema.ts + export { customers } from "../generated/Customer.js"; // name the tables you want migrated + export { orders } from "../generated/Order.js"; + // or, to follow the model as it grows: + // export * from "../generated/index.js"; + export * from "./hand-written-tables.js"; + ``` + + `export *` also re-exports Zod schemas and types. drizzle-kit ignores everything that is not + a table, view or enum, so that is harmless; name the tables if you would rather keep the + file explicit. + +3. Point drizzle-kit at that file, not at `src/generated`: + + ```ts + // drizzle.config.ts + import { defineConfig } from "drizzle-kit"; + + export default defineConfig({ + dialect: "postgresql", + schema: "./src/db/schema.ts", + out: "./drizzle", + }); + ``` + +4. The loop is now: edit `metaobjects/*.json`, run `meta gen`, run `drizzle-kit generate`, + review the SQL, run `drizzle-kit migrate`. + +Keep one tool per database. Do not also run `meta migrate` against the same database: it +diffs the model against the live schema and would write a second, competing migration chain. +`meta verify --db ` is read-only and still useful as a drift check beside drizzle-kit: +it reports where the live database no longer matches the model, whoever applied the DDL. + +## What was checked + +The re-export was run against the real `entityFile()` output (SQLite dialect): both the named +re-export and `export *` from the barrel expose the generated table to Drizzle's own table +detection (`is(value, SQLiteTable)`), with the model's column names. drizzle-kit itself was not +installed in the checking environment, so the `drizzle-kit generate` step is the documented +drizzle-kit contract (it scans the schema file's exports), not something this repository gates. + +## When to prefer `meta migrate` instead + +Choose `meta migrate` (see [migrations and drift](../features/migrations-and-drift.md)) when +you want the schema derived from the model rather than from the generated Drizzle code: + +- **No second source of truth.** With drizzle-kit, the chain of record is the diff of the + generated TypeScript; with `meta migrate` it is the model itself, and `meta verify --db` + and `meta verify --replay` gate the chain against the model. +- **Things the model owns that a table export does not carry.** SQL views for projections and + `object.report` ([reporting](../features/reporting.md)) are created and kept in step by + `meta migrate`, including recreating a view when a table it reads changes. +- **Renames and populated tables.** `meta migrate` resolves renames (`--on-ambiguous`) and + treats destructive changes on populated tables explicitly + ([Renames and populated tables](../features/migrations-and-drift.md#renames-and-populated-tables)). +- **Cloudflare D1.** `meta migrate --dialect d1` writes Wrangler's migration layout and + handles SQLite's table-rebuild rules ([Cloudflare Workers recipe](cloudflare-workers.md)). +- **MySQL is the exception:** MetaObjects does not own a MySQL schema, so drizzle-kit (or + hand-written DDL) is the only option ([MySQL recipe](mysql.md)). + +You can adopt `meta migrate` later without changing the generated code. Baseline from the live +database with `meta migrate baseline --from-db --db ` and stop running drizzle-kit. diff --git a/docs/recipes/guarding-generated-routes.md b/docs/recipes/guarding-generated-routes.md new file mode 100644 index 0000000000..2eeed7bcee --- /dev/null +++ b/docs/recipes/guarding-generated-routes.md @@ -0,0 +1,87 @@ +# Recipe: mount generated routes inside your own auth scope + +Generated CRUD routes are **unauthenticated**. Nothing in the metamodel says otherwise and +nothing should: whether a caller may act is not derivable from a model, so there is no `@auth` +attribute. You guard the routes with your framework's own composition. No edit to the generated +file is needed. + +This is the same recipe the generated routes file prints in its handler's JSDoc. Both +snippets below are exercised against the real mount helpers by +`server/typescript/packages/runtime-ts/test/route-auth-seam.test.ts`; the text is produced by +`authSeamJsDoc` in `server/typescript/packages/codegen-ts/src/routes-expose.ts`. If you change a recipe's shape, +change that test first. + +## Fastify + +Register the generated handler inside a scope that carries your hook. Fastify hooks are +encapsulated per plugin and inherited by child scopes, so the hook reaches through the generated +handler's own `register(..., { prefix })`: + +```ts +import { noteRoutes } from "./generated/Note.routes.js"; + +app.register(async (s) => { + s.addHook("preHandler", requireAuth); // your hook: 401 without credentials + await noteRoutes(s); // Fastify spells it `Routes` +}); + +app.get("/health", async () => ({ ok: true })); // outside the scope: stays open +``` + +Every verb (list, get, create, update, delete) is guarded. Routes registered outside the scope +are not. Use `preHandler`, not `onRequest`, if your check needs the parsed body. + +## Hono + +Put middleware on the mount path **with a trailing `/*`**, before calling the generated +register function: + +```ts +import { Note } from "./generated/Note.js"; +import { registerNoteRoutes } from "./generated/Note.routes.hono.js"; + +app.use(`/api${Note.$path}/*`, requireAuth); // `${apiPrefix}${Entity.$path}/*` +registerNoteRoutes(app, { db }); +``` + +The wildcard is load-bearing: in Hono `"/api/notes/*"` also matches the collection path +`/api/notes` itself, so one `app.use` covers list and create as well as every `/:id` route. +Without it, `GET /api/notes` and `POST /api/notes` stay open while the recipe looks right. +Order matters: `app.use` must be registered **before** the routes it should guard. + +## Narrow what is mounted with `expose` + +A guard answers "is this caller authenticated?". It cannot answer "may this caller see **this +row**". For an entity holding credentials, per-user rows, or anything a list endpoint would hand +to the wrong person, stock CRUD is not safe at any verb. Hand-write those verbs and narrow the +generated file to what remains: + +```ts +import { routesFile, routesFileHono } from "@metaobjectsdev/codegen-ts/generators"; + +generators: [ + routesFile({ expose: ["list", "get"] }), // Fastify + routesFileHono({ expose: ["list", "get"] }), // Hono +] +``` + +`expose` takes any subset of `"list" | "get" | "create" | "update" | "delete"`, or a function +`(entity) => verbs | undefined` to choose per entity (`undefined` means all five). An unlisted verb +is not mounted at all: with `expose: ["list", "get"]` a `POST /api/notes` answers 404, whatever +the caller's credentials. A discriminated-union base and a read-only projection are already +narrower than the five verbs, and `expose` can only narrow them further, never widen. `expose` +is a generator option, not metadata, because which verbs a deployment serves is a property of +the app and not of the shared model. + +## What was checked + +- The runtime-ts test above asserts, for both frameworks, `401` on all five verbs without the + header, `200` with it, and that a sibling route outside the guarded scope stays open. +- A model generated with `routesFile({ expose: ["list", "get"] })` and + `routesFileHono({ expose: ["list", "get"] })` emits `expose: ["list", "get"]` in the mount call. + Run against in-memory SQLite behind the Hono recipe: `GET /api/notes` answered 401 without + credentials and 200 with them, and `POST /api/notes` and `DELETE /api/notes/1` answered 404 + with them. + +See also: [TypeScript port, Hono variant](../ports/typescript.md#hono-variant-workers--bun--edge), +[Cloudflare Workers](cloudflare-workers.md). diff --git a/docs/recipes/precommit-meta-verify.md b/docs/recipes/precommit-meta-verify.md index ae96613bcf..92d1e9ed9b 100644 --- a/docs/recipes/precommit-meta-verify.md +++ b/docs/recipes/precommit-meta-verify.md @@ -40,6 +40,8 @@ won't block a commit. Silence it with `--no-antipatterns` or `META_NO_ANTIPATTER Text output caps it at 20 lines (raise with `--limit ` / `--limit all`); a `--format json`/`toon` run carries every finding, uncapped. +Before you trust the hook, prove it can fail: [Prove your gate fails](prove-your-gate-fails.md). + ## Choose your hook manager The MetaObjects framework is hook-manager-agnostic. Three common setups: diff --git a/docs/recipes/prove-your-gate-fails.md b/docs/recipes/prove-your-gate-fails.md new file mode 100644 index 0000000000..68b1706bf9 --- /dev/null +++ b/docs/recipes/prove-your-gate-fails.md @@ -0,0 +1,61 @@ +# Recipe: prove your `meta verify` gate can fail + +A drift gate that has never gone red is not yet a gate. `meta verify` exits 0 on a tree it +checked, and also on a tree it checked nothing of: a config with `generators: []`, a wrong +working directory, a gate that was never wired into CI. So seed a deliberate break once, watch +the exit code flip to 1, and keep that test. + +The runnable version, which this repository runs on every change, is +[`verify-gate-not-vacuous.test.ts`](../../server/typescript/packages/cli/test/integration/verify-gate-not-vacuous.test.ts). +It works in a throwaway project and asserts, in this order: + +| Step | State | `meta verify` exit | +|---|---|---| +| 0 | `meta gen` ran, so the gate has generated files to compare | n/a | +| 1 | Clean tree | **0** | +| 2 | A field added to the metadata, `meta gen` not re-run | **1** (codegen gate) | +| 3 | Metadata restored | **0** (the failure was the break, not the harness) | +| 4 | A hand edit inside a generated file | **0** by default, **1** with `meta verify --codegen --forbid-hand-edits` | +| 5 | A generated file deleted | **1** | +| 6 | `meta gen` re-run | **0** | + +Step 4 is deliberate: `meta gen` three-way-merges hand edits, so a hand-edited generated file +is a notice, not drift. If your policy is "generated code is never touched", gate on +`--forbid-hand-edits`. A second test in the same file pins why step 0 exists: a project with no +generators passes `meta verify` with exit 0 having checked nothing. + +## The same check in a shell, for your own CI + +```bash +set -u +meta gen || exit 1 +meta verify || { echo "clean tree must pass"; exit 1; } + +# Seed a break: change the model without regenerating. +cp metaobjects/meta.shop.json /tmp/meta.shop.json.bak +sed -i 's/"name": "email"/"name": "emailAddress"/' metaobjects/meta.shop.json + +if meta verify; then echo "GATE IS VACUOUS: drifted tree passed"; status=1; else status=0; fi +cp /tmp/meta.shop.json.bak metaobjects/meta.shop.json +meta verify || status=1 # restored tree must pass again +exit $status +``` + +Adjust the file and the string to a field in your own model. Run it once when you wire the gate +(and after any change to `generators`, `sources` or the CI job), not on every build. Never leave +the break in the working tree: the restore is part of the check. + +## As a bun:test in your repo + +Copy the integration test above. It needs only the built `meta` binary on `PATH` (or +`node node_modules/@metaobjectsdev/cli/dist/bin/meta.js`), a project with at least one generator +declared, and a way to run `meta` with `cwd` set to a temp copy of the project. Replace its +`newShopProject` helper with a copy of your own fixture directory. + +## What was checked + +`bun test test/integration/verify-gate-not-vacuous.test.ts` in `server/typescript/packages/cli` +against the built CLI: both tests pass (steps 1-6 above, plus the empty-generators case). The +shell snippet is the same sequence with `sed` as the break and was not run separately. + +See also: [pre-commit `meta verify`](precommit-meta-verify.md) for wiring the gate itself. diff --git a/docs/recipes/wiring-generated-queries.md b/docs/recipes/wiring-generated-queries.md index ad15832d74..80262a3a0d 100644 --- a/docs/recipes/wiring-generated-queries.md +++ b/docs/recipes/wiring-generated-queries.md @@ -280,10 +280,10 @@ export default defineConfig({ }); ``` -> **Note for D1 consumers:** codegen-ts only knows `"sqlite" | "postgres"`. D1 is -> SQLite at the SQL level, so codegen uses `dialect: "sqlite"`. The D1-specific -> `--dialect d1` is a `meta migrate` CLI flag (it controls migration transport + -> file layout), not a codegen config value. +> **Note for D1 consumers:** codegen's `dialect` is `"sqlite" | "postgres" | "mysql"`. D1 is +> SQLite at the SQL level, so codegen uses `dialect: "sqlite"`. `--dialect d1` is a +> `meta migrate` CLI flag (migration transport + file layout), not a codegen config value, and +> `dialect: "d1"` in `metaobjects.config.ts` is refused with an error. ## Migration from 0.6.0 diff --git a/examples/advanced-modeling/codegen/generators/routes.ts b/examples/advanced-modeling/codegen/generators/routes.ts index 067e9460ee..56a79d85b6 100644 --- a/examples/advanced-modeling/codegen/generators/routes.ts +++ b/examples/advanced-modeling/codegen/generators/routes.ts @@ -69,6 +69,7 @@ import { isReport, itemRouteField, decimalColumnsLine, + writeDeniedColumnsLine, servesReadApi, formatTs, entityOutputPath, @@ -308,7 +309,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ", ctx.loadedRoot)}${exposeLinePrefixed} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ", ctx.loadedRoot)}${writeDeniedColumnsLine(entity, " ")}${exposeLinePrefixed} }); ${m2mMountsPrefixed} }, { prefix: ${JSON.stringify(ctx.apiPrefix)} }); } @@ -333,7 +334,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ", ctx.loadedRoot)}${exposeLineFlat} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ", ctx.loadedRoot)}${writeDeniedColumnsLine(entity, " ")}${exposeLineFlat} }); ${m2mMountsFlat}} `; @@ -402,6 +403,15 @@ interface TphM2mSource { value: string; } +/** + * The object a M:N mount reads rows of, plus the subtypes of a TPH base target: its rows are + * shared-table rows of every subtype, so a credential declared by any of them is hidden too. + */ +function m2mTargetObjects(target: MetaObject, ctx: RenderContext): MetaObject[] { + const plan = ctx.loadedRoot === undefined ? null : tphPlan(target, ctx.loadedRoot); + return plan === null ? [target] : [target, ...plan.subtypes.map((st) => st.entity)]; +} + /** * Render one M:N traversal mount. The junction + target Drizzle table consts are * imported from their sibling entity files (imp() lets ts-poet track + emit the @@ -489,7 +499,7 @@ function renderM2mMount( sourceColumn: ${sourceColumn}, targetColumn: ${targetColumn}, targetPkColumn: ${targetPkColumn}, - symmetric: ${entry.symmetric ? "true" : "false"},${decimalColumnsLine(entry.target, " ", ctx.loadedRoot)}${discriminatorLine}${sourceDiscriminatorLine} + symmetric: ${entry.symmetric ? "true" : "false"},${decimalColumnsLine(m2mTargetObjects(entry.target, ctx), " ", ctx.loadedRoot)}${discriminatorLine}${sourceDiscriminatorLine} });`; } @@ -580,6 +590,7 @@ function renderTphRoutes( // single writable shape — so an author-supplied `expose` intersects with that fixed set // rather than replacing it: it may narrow to just `list`, never widen to `create`. const polymorphicExposeLine = exposeLine(intersectExpose([...TPH_POLYMORPHIC_VERBS], expose), " "); + const hierarchy = [base, ...plan.subtypes.map((st) => st.entity)]; const polymorphic = code` ${mountCrudRoutesSym}({ fastify: ${fastifyRef}, @@ -590,7 +601,7 @@ function renderTphRoutes( updateSchema: ${baseUpdateSym}, filterAllowlist: ${baseFilterSym}, sortAllowlist: ${baseSortSym}, - dialect: ${dialectLit},${decimalColumnsLine([base, ...plan.subtypes.map((st) => st.entity)], " ", ctx.loadedRoot)}${polymorphicExposeLine} + dialect: ${dialectLit},${decimalColumnsLine(hierarchy, " ", ctx.loadedRoot)}${polymorphicExposeLine} });`; // FR-018 x FR-017 — M:N traversal inside a TPH hierarchy. This file never consulted @@ -633,7 +644,7 @@ function renderTphRoutes( updateSchema: ${subUpdateSym}, filterAllowlist: ${subFilterSym}, sortAllowlist: ${subSortSym}, - dialect: ${dialectLit},${decimalColumnsLine(sub, " ", ctx.loadedRoot)} + dialect: ${dialectLit},${decimalColumnsLine(sub, " ", ctx.loadedRoot, hierarchy)}${writeDeniedColumnsLine(sub, " ", hierarchy)} discriminator: { column: ${JSON.stringify(discField)}, value: ${JSON.stringify(value)} },${exposeLine(expose, " ")} });`; // This subtype's own M:N navigations, mounted beneath its segment and gated on the diff --git a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/index.ts b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/index.ts index 930c44d394..78baf3f97a 100644 --- a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/index.ts +++ b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/index.ts @@ -29,7 +29,7 @@ export type { FilterAllowlist, SortAllowlist } from "./filter-allowlist.js"; import { parseFilterParams, parsePageBound, FilterParseError } from "./filter-parser.js"; import { isTruthyFlag, contractErrorCode, coerceIdForColumn, firstRow, extractRowCount } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; -import { withWire, type WireOptions } from "../wire-format.js"; +import { stripDeniedColumns, withWire, type WireOptions } from "../wire-format.js"; import { supportsReturning, type SqlDialect } from "../sql-dialect.js"; import { withContractErrorHandler } from "./route-error-handler.js"; import { validationErrorBody } from "../route-errors.js"; @@ -72,6 +72,13 @@ export interface CrudRoutesOptions extends WireOptions { insertSchema: ZodTypeAny; /** Zod schema for update payloads (typically `UpdateSchema`). */ updateSchema: ZodTypeAny; + /** + * Fields the API never writes (`@formExclude`, or a credential `view.password`). A key + * named here is stripped from a create or update body before validation, so the value never + * reaches the table; it is ignored, not rejected. The generated route passes them; the + * schemas stay whole for the application's own writes. + */ + writeDeniedColumns?: readonly string[]; /** Limit which verbs are mounted. Defaults to all five. */ expose?: readonly CrudVerb[]; /** @@ -274,7 +281,7 @@ export function mountGetRoute(opts: VerbOptions): void { export function mountCreateRoute(opts: VerbOptions): void { const toWire = echoWire(opts); opts.fastify.post(opts.path, routeOpts(opts), async (req, reply) => { - const parsed = opts.insertSchema.safeParse(req.body); + const parsed = opts.insertSchema.safeParse(stripDeniedColumns(req.body, opts.writeDeniedColumns)); if (!parsed.success) { return reply.code(400).send(validationErrorBody(parsed.error.issues)); } @@ -315,7 +322,7 @@ export function mountUpdateRoute(opts: VerbOptions): void { reply: { code: (n: number) => { send: (b: unknown) => unknown } }, ) => { const { id } = req.params as { id: string }; - const parsed = opts.updateSchema.safeParse(req.body); + const parsed = opts.updateSchema.safeParse(stripDeniedColumns(req.body, opts.writeDeniedColumns)); if (!parsed.success) { return reply.code(400).send(validationErrorBody(parsed.error.issues)); } diff --git a/examples/advanced-modeling/codegen/runtime/wire-format.ts b/examples/advanced-modeling/codegen/runtime/wire-format.ts index 086147c555..b21817d409 100644 --- a/examples/advanced-modeling/codegen/runtime/wire-format.ts +++ b/examples/advanced-modeling/codegen/runtime/wire-format.ts @@ -43,12 +43,43 @@ export interface WireOptions { readonly nullMembers?: NullMembersMode; /** The jsonb columns that hold a value object (or an array / map of them), by column name. */ readonly valueObjects?: Readonly>; + /** + * Credential fields (a `view.password` field): removed from every row sent, on list, get, + * the echo of a create or update, and an M:N traversal. The generated route passes them; + * absent, rows are sent as read. + */ + readonly hiddenColumns?: readonly string[]; +} + +/** Remove `hidden` keys from a row. `undefined` when there is nothing to hide. */ +export function hiddenColumnsWire(hidden: readonly string[] | undefined): ((row: unknown) => unknown) | undefined { + if (hidden === undefined || hidden.length === 0) return undefined; + return (row) => (isPlainObject(row) ? omitKeys(row, hidden) : row); +} + +/** + * Remove `denied` keys from a request body, before it is validated. The route's schema is the + * entity's shared `Insert`/`Update` schema, which app code writes through too, so the API + * boundary strips here and the schema stays whole. A key stripped is IGNORED, never rejected + * (the uniform rule for every key outside the settable set, `@mutability: writeOnce` on PATCH + * among them): a generated edit form submits every field it was given. + */ +export function stripDeniedColumns(body: unknown, denied: readonly string[] | undefined): unknown { + if (denied === undefined || denied.length === 0 || !isPlainObject(body)) return body; + return omitKeys(body, denied); } function isPlainObject(v: unknown): v is Record { return v !== null && typeof v === "object" && !Array.isArray(v); } +/** A shallow copy of `obj` without the named keys. */ +function omitKeys(obj: Record, keys: readonly string[]): Record { + const out = { ...obj }; + for (const key of keys) delete out[key]; + return out; +} + function shapeMember(value: Record, shape: ValueObjectShape, mode: NullMembersMode): Record { const out: Record = {}; if (mode === "emit") { @@ -95,9 +126,11 @@ export function nullMembersWire( }; } -/** A wire mapper: `base` (timestamps), then the decimal rewrite, then the null-member rewrite. */ +/** A wire mapper: `base` (timestamps), then the decimal rewrite, the null-member rewrite, then the credential strip. */ export function withWire(base: (row: unknown) => unknown, opts: WireOptions): (row: unknown) => unknown { const decimals = withDecimals(base, opts.decimalColumns, opts.decimalWire); const members = nullMembersWire(opts.valueObjects, opts.nullMembers); - return members === undefined ? decimals : (row) => members(decimals(row)); + const hidden = hiddenColumnsWire(opts.hiddenColumns); + const shaped = members === undefined ? decimals : (row: unknown) => members(decimals(row)); + return hidden === undefined ? shaped : (row) => hidden(shaped(row)); } diff --git a/examples/showcase/codegen/generators/routes.ts b/examples/showcase/codegen/generators/routes.ts index 067e9460ee..56a79d85b6 100644 --- a/examples/showcase/codegen/generators/routes.ts +++ b/examples/showcase/codegen/generators/routes.ts @@ -69,6 +69,7 @@ import { isReport, itemRouteField, decimalColumnsLine, + writeDeniedColumnsLine, servesReadApi, formatTs, entityOutputPath, @@ -308,7 +309,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ", ctx.loadedRoot)}${exposeLinePrefixed} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ", ctx.loadedRoot)}${writeDeniedColumnsLine(entity, " ")}${exposeLinePrefixed} }); ${m2mMountsPrefixed} }, { prefix: ${JSON.stringify(ctx.apiPrefix)} }); } @@ -333,7 +334,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ", ctx.loadedRoot)}${exposeLineFlat} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ", ctx.loadedRoot)}${writeDeniedColumnsLine(entity, " ")}${exposeLineFlat} }); ${m2mMountsFlat}} `; @@ -402,6 +403,15 @@ interface TphM2mSource { value: string; } +/** + * The object a M:N mount reads rows of, plus the subtypes of a TPH base target: its rows are + * shared-table rows of every subtype, so a credential declared by any of them is hidden too. + */ +function m2mTargetObjects(target: MetaObject, ctx: RenderContext): MetaObject[] { + const plan = ctx.loadedRoot === undefined ? null : tphPlan(target, ctx.loadedRoot); + return plan === null ? [target] : [target, ...plan.subtypes.map((st) => st.entity)]; +} + /** * Render one M:N traversal mount. The junction + target Drizzle table consts are * imported from their sibling entity files (imp() lets ts-poet track + emit the @@ -489,7 +499,7 @@ function renderM2mMount( sourceColumn: ${sourceColumn}, targetColumn: ${targetColumn}, targetPkColumn: ${targetPkColumn}, - symmetric: ${entry.symmetric ? "true" : "false"},${decimalColumnsLine(entry.target, " ", ctx.loadedRoot)}${discriminatorLine}${sourceDiscriminatorLine} + symmetric: ${entry.symmetric ? "true" : "false"},${decimalColumnsLine(m2mTargetObjects(entry.target, ctx), " ", ctx.loadedRoot)}${discriminatorLine}${sourceDiscriminatorLine} });`; } @@ -580,6 +590,7 @@ function renderTphRoutes( // single writable shape — so an author-supplied `expose` intersects with that fixed set // rather than replacing it: it may narrow to just `list`, never widen to `create`. const polymorphicExposeLine = exposeLine(intersectExpose([...TPH_POLYMORPHIC_VERBS], expose), " "); + const hierarchy = [base, ...plan.subtypes.map((st) => st.entity)]; const polymorphic = code` ${mountCrudRoutesSym}({ fastify: ${fastifyRef}, @@ -590,7 +601,7 @@ function renderTphRoutes( updateSchema: ${baseUpdateSym}, filterAllowlist: ${baseFilterSym}, sortAllowlist: ${baseSortSym}, - dialect: ${dialectLit},${decimalColumnsLine([base, ...plan.subtypes.map((st) => st.entity)], " ", ctx.loadedRoot)}${polymorphicExposeLine} + dialect: ${dialectLit},${decimalColumnsLine(hierarchy, " ", ctx.loadedRoot)}${polymorphicExposeLine} });`; // FR-018 x FR-017 — M:N traversal inside a TPH hierarchy. This file never consulted @@ -633,7 +644,7 @@ function renderTphRoutes( updateSchema: ${subUpdateSym}, filterAllowlist: ${subFilterSym}, sortAllowlist: ${subSortSym}, - dialect: ${dialectLit},${decimalColumnsLine(sub, " ", ctx.loadedRoot)} + dialect: ${dialectLit},${decimalColumnsLine(sub, " ", ctx.loadedRoot, hierarchy)}${writeDeniedColumnsLine(sub, " ", hierarchy)} discriminator: { column: ${JSON.stringify(discField)}, value: ${JSON.stringify(value)} },${exposeLine(expose, " ")} });`; // This subtype's own M:N navigations, mounted beneath its segment and gated on the diff --git a/examples/showcase/codegen/runtime/drizzle-fastify/index.ts b/examples/showcase/codegen/runtime/drizzle-fastify/index.ts index 930c44d394..78baf3f97a 100644 --- a/examples/showcase/codegen/runtime/drizzle-fastify/index.ts +++ b/examples/showcase/codegen/runtime/drizzle-fastify/index.ts @@ -29,7 +29,7 @@ export type { FilterAllowlist, SortAllowlist } from "./filter-allowlist.js"; import { parseFilterParams, parsePageBound, FilterParseError } from "./filter-parser.js"; import { isTruthyFlag, contractErrorCode, coerceIdForColumn, firstRow, extractRowCount } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; -import { withWire, type WireOptions } from "../wire-format.js"; +import { stripDeniedColumns, withWire, type WireOptions } from "../wire-format.js"; import { supportsReturning, type SqlDialect } from "../sql-dialect.js"; import { withContractErrorHandler } from "./route-error-handler.js"; import { validationErrorBody } from "../route-errors.js"; @@ -72,6 +72,13 @@ export interface CrudRoutesOptions extends WireOptions { insertSchema: ZodTypeAny; /** Zod schema for update payloads (typically `UpdateSchema`). */ updateSchema: ZodTypeAny; + /** + * Fields the API never writes (`@formExclude`, or a credential `view.password`). A key + * named here is stripped from a create or update body before validation, so the value never + * reaches the table; it is ignored, not rejected. The generated route passes them; the + * schemas stay whole for the application's own writes. + */ + writeDeniedColumns?: readonly string[]; /** Limit which verbs are mounted. Defaults to all five. */ expose?: readonly CrudVerb[]; /** @@ -274,7 +281,7 @@ export function mountGetRoute(opts: VerbOptions): void { export function mountCreateRoute(opts: VerbOptions): void { const toWire = echoWire(opts); opts.fastify.post(opts.path, routeOpts(opts), async (req, reply) => { - const parsed = opts.insertSchema.safeParse(req.body); + const parsed = opts.insertSchema.safeParse(stripDeniedColumns(req.body, opts.writeDeniedColumns)); if (!parsed.success) { return reply.code(400).send(validationErrorBody(parsed.error.issues)); } @@ -315,7 +322,7 @@ export function mountUpdateRoute(opts: VerbOptions): void { reply: { code: (n: number) => { send: (b: unknown) => unknown } }, ) => { const { id } = req.params as { id: string }; - const parsed = opts.updateSchema.safeParse(req.body); + const parsed = opts.updateSchema.safeParse(stripDeniedColumns(req.body, opts.writeDeniedColumns)); if (!parsed.success) { return reply.code(400).send(validationErrorBody(parsed.error.issues)); } diff --git a/examples/showcase/codegen/runtime/wire-format.ts b/examples/showcase/codegen/runtime/wire-format.ts index 086147c555..b21817d409 100644 --- a/examples/showcase/codegen/runtime/wire-format.ts +++ b/examples/showcase/codegen/runtime/wire-format.ts @@ -43,12 +43,43 @@ export interface WireOptions { readonly nullMembers?: NullMembersMode; /** The jsonb columns that hold a value object (or an array / map of them), by column name. */ readonly valueObjects?: Readonly>; + /** + * Credential fields (a `view.password` field): removed from every row sent, on list, get, + * the echo of a create or update, and an M:N traversal. The generated route passes them; + * absent, rows are sent as read. + */ + readonly hiddenColumns?: readonly string[]; +} + +/** Remove `hidden` keys from a row. `undefined` when there is nothing to hide. */ +export function hiddenColumnsWire(hidden: readonly string[] | undefined): ((row: unknown) => unknown) | undefined { + if (hidden === undefined || hidden.length === 0) return undefined; + return (row) => (isPlainObject(row) ? omitKeys(row, hidden) : row); +} + +/** + * Remove `denied` keys from a request body, before it is validated. The route's schema is the + * entity's shared `Insert`/`Update` schema, which app code writes through too, so the API + * boundary strips here and the schema stays whole. A key stripped is IGNORED, never rejected + * (the uniform rule for every key outside the settable set, `@mutability: writeOnce` on PATCH + * among them): a generated edit form submits every field it was given. + */ +export function stripDeniedColumns(body: unknown, denied: readonly string[] | undefined): unknown { + if (denied === undefined || denied.length === 0 || !isPlainObject(body)) return body; + return omitKeys(body, denied); } function isPlainObject(v: unknown): v is Record { return v !== null && typeof v === "object" && !Array.isArray(v); } +/** A shallow copy of `obj` without the named keys. */ +function omitKeys(obj: Record, keys: readonly string[]): Record { + const out = { ...obj }; + for (const key of keys) delete out[key]; + return out; +} + function shapeMember(value: Record, shape: ValueObjectShape, mode: NullMembersMode): Record { const out: Record = {}; if (mode === "emit") { @@ -95,9 +126,11 @@ export function nullMembersWire( }; } -/** A wire mapper: `base` (timestamps), then the decimal rewrite, then the null-member rewrite. */ +/** A wire mapper: `base` (timestamps), then the decimal rewrite, the null-member rewrite, then the credential strip. */ export function withWire(base: (row: unknown) => unknown, opts: WireOptions): (row: unknown) => unknown { const decimals = withDecimals(base, opts.decimalColumns, opts.decimalWire); const members = nullMembersWire(opts.valueObjects, opts.nullMembers); - return members === undefined ? decimals : (row) => members(decimals(row)); + const hidden = hiddenColumnsWire(opts.hiddenColumns); + const shaped = members === undefined ? decimals : (row: unknown) => members(decimals(row)); + return hidden === undefined ? shaped : (row) => hidden(shaped(row)); } diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md index 39d9a51cdd..e145d1ccf6 100644 --- a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md @@ -198,8 +198,10 @@ soft-delete / status / type view it models. and are NOT in the cross-port registry — `view.text`, `view.textarea`, `view.date`, `view.month`, `view.hotlink`, `view.image`, `view.dropdown`, `view.radio`, `view.checkbox`, `view.number`, `view.password`, `view.hidden`, `view.web`. Every port LOADS them, so shared metadata that - carries them is valid in a backend port; nothing outside TS/web reads them. **Audit these - only for TS adopters.** Only + carries them is valid in a backend port. The one exception is `view.password`: every port's generated + routes read it as a credential, so it is never returned and never written (see + "Credentials and excluded fields" in `docs/features/api-contract.md`). + Audit the other widgets only for TS adopters. Only `view.base` / `view.currency` are cross-port-gated. ## Template — `template.*` (prompt pillar) diff --git a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md index 39d9a51cdd..e145d1ccf6 100644 --- a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md +++ b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md @@ -198,8 +198,10 @@ soft-delete / status / type view it models. and are NOT in the cross-port registry — `view.text`, `view.textarea`, `view.date`, `view.month`, `view.hotlink`, `view.image`, `view.dropdown`, `view.radio`, `view.checkbox`, `view.number`, `view.password`, `view.hidden`, `view.web`. Every port LOADS them, so shared metadata that - carries them is valid in a backend port; nothing outside TS/web reads them. **Audit these - only for TS adopters.** Only + carries them is valid in a backend port. The one exception is `view.password`: every port's generated + routes read it as a credential, so it is never returned and never written (see + "Credentials and excluded fields" in `docs/features/api-contract.md`). + Audit the other widgets only for TS adopters. Only `view.base` / `view.currency` are cross-port-gated. ## Template — `template.*` (prompt pillar) diff --git a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md index 39d9a51cdd..e145d1ccf6 100644 --- a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md +++ b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md @@ -198,8 +198,10 @@ soft-delete / status / type view it models. and are NOT in the cross-port registry — `view.text`, `view.textarea`, `view.date`, `view.month`, `view.hotlink`, `view.image`, `view.dropdown`, `view.radio`, `view.checkbox`, `view.number`, `view.password`, `view.hidden`, `view.web`. Every port LOADS them, so shared metadata that - carries them is valid in a backend port; nothing outside TS/web reads them. **Audit these - only for TS adopters.** Only + carries them is valid in a backend port. The one exception is `view.password`: every port's generated + routes read it as a credential, so it is never returned and never written (see + "Credentials and excluded fields" in `docs/features/api-contract.md`). + Audit the other widgets only for TS adopters. Only `view.base` / `view.currency` are cross-port-gated. ## Template — `template.*` (prompt pillar) diff --git a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md index 39d9a51cdd..e145d1ccf6 100644 --- a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md +++ b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md @@ -198,8 +198,10 @@ soft-delete / status / type view it models. and are NOT in the cross-port registry — `view.text`, `view.textarea`, `view.date`, `view.month`, `view.hotlink`, `view.image`, `view.dropdown`, `view.radio`, `view.checkbox`, `view.number`, `view.password`, `view.hidden`, `view.web`. Every port LOADS them, so shared metadata that - carries them is valid in a backend port; nothing outside TS/web reads them. **Audit these - only for TS adopters.** Only + carries them is valid in a backend port. The one exception is `view.password`: every port's generated + routes read it as a credential, so it is never returned and never written (see + "Credentials and excluded fields" in `docs/features/api-contract.md`). + Audit the other widgets only for TS adopters. Only `view.base` / `view.currency` are cross-port-gated. ## Template — `template.*` (prompt pillar) diff --git a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/typescript.md b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/typescript.md index d4b42126a0..3fc99245c4 100644 --- a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/typescript.md +++ b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/typescript.md @@ -199,4 +199,4 @@ TypeScript is the reference implementation — it ships the full feature set: - Output-parser codegen: **complete** (`outputParser()` generator; `parse*`/`safeParse*`/`extract*` emitted as `.response.ts` per **responding `template.prompt`** — one carrying `@responseRef`; ADR-0052). - ObjectManager: **complete** (`@metaobjectsdev/runtime-ts`; `kyselyDriver` / `inMemoryDriver`). - Schema migrate / `meta verify --db`: **complete** (TS owns the shared migrate engine — ADR-0015). -- D1 dialect: **TS-only by design** — `dialect: "d1"` in `metaobjects.config.ts` targets Cloudflare D1 via the Wrangler CLI layout. Do NOT recommend porting D1 support to other languages. +- D1 dialect: **TS-only by design** — `migrate.dialect: "d1"` (and `meta migrate --dialect d1`) targets Cloudflare D1 via the Wrangler CLI layout; codegen's `dialect` in `metaobjects.config.ts` is `"sqlite"` for D1 and `"d1"` there is refused. Do NOT recommend porting D1 support to other languages. diff --git a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md index 39d9a51cdd..e145d1ccf6 100644 --- a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md +++ b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/capability-checklist.md @@ -198,8 +198,10 @@ soft-delete / status / type view it models. and are NOT in the cross-port registry — `view.text`, `view.textarea`, `view.date`, `view.month`, `view.hotlink`, `view.image`, `view.dropdown`, `view.radio`, `view.checkbox`, `view.number`, `view.password`, `view.hidden`, `view.web`. Every port LOADS them, so shared metadata that - carries them is valid in a backend port; nothing outside TS/web reads them. **Audit these - only for TS adopters.** Only + carries them is valid in a backend port. The one exception is `view.password`: every port's generated + routes read it as a credential, so it is never returned and never written (see + "Credentials and excluded fields" in `docs/features/api-contract.md`). + Audit the other widgets only for TS adopters. Only `view.base` / `view.currency` are cross-port-gated. ## Template — `template.*` (prompt pillar) diff --git a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/typescript.md b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/typescript.md index d4b42126a0..3fc99245c4 100644 --- a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/typescript.md +++ b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/typescript.md @@ -199,4 +199,4 @@ TypeScript is the reference implementation — it ships the full feature set: - Output-parser codegen: **complete** (`outputParser()` generator; `parse*`/`safeParse*`/`extract*` emitted as `.response.ts` per **responding `template.prompt`** — one carrying `@responseRef`; ADR-0052). - ObjectManager: **complete** (`@metaobjectsdev/runtime-ts`; `kyselyDriver` / `inMemoryDriver`). - Schema migrate / `meta verify --db`: **complete** (TS owns the shared migrate engine — ADR-0015). -- D1 dialect: **TS-only by design** — `dialect: "d1"` in `metaobjects.config.ts` targets Cloudflare D1 via the Wrangler CLI layout. Do NOT recommend porting D1 support to other languages. +- D1 dialect: **TS-only by design** — `migrate.dialect: "d1"` (and `meta migrate --dialect d1`) targets Cloudflare D1 via the Wrangler CLI layout; codegen's `dialect` in `metaobjects.config.ts` is `"sqlite"` for D1 and `"d1"` there is refused. Do NOT recommend porting D1 support to other languages. diff --git a/fixtures/api-contract-conformance/README.md b/fixtures/api-contract-conformance/README.md index cb5eee0703..7574beeab5 100644 --- a/fixtures/api-contract-conformance/README.md +++ b/fixtures/api-contract-conformance/README.md @@ -52,18 +52,19 @@ fixtures/api-contract-conformance/ └── filter-invalid-value.yaml # uncoercible value → 400; the fourth envelope, previously ungated ``` -Beside the 31 core scenarios above sit seven **sub-corpora**, each a directory with its -own `meta.json`, `seed.json`, `scenarios/` and `README.md` (75 scenarios in all, 106 with the +Beside the 31 core scenarios above sit eight **sub-corpora**, each a directory with its +own `meta.json`, `seed.json`, `scenarios/` and `README.md` (88 scenarios in all, 119 with the core): | Directory | Scenarios | Gates | Lanes | |---|---|---|---| | `tph/` | 10 | single-table inheritance | both | | `m2m/` | 9 | M:N traversal, and TPH x M:N together | both | -| `jsonb/` | 4 | typed value-object columns, and the numbers inside them (a double stays a JSON number; a decimal is a canonical string, option A of the 1.1.0 review) | both | +| `jsonb/` | 5 | typed value-object columns, and the numbers inside them (a double stays a JSON number; a decimal is a canonical string, option A of the 1.1.0 review) | both | | `write-through/` | 2 | an entity that writes a table and reads a view | generated only | +| `secret-fields/` | 2 | #484: a `@formExclude` or credential (`view.password`) field is ignored on create, PATCH and PUT, and a credential is never returned | generated only | | `projection/` | 12 | a read-only `object.projection`: reads served, every write `405`; no declared identity means no `/{id}` route; a key not named `id` addresses the row; decimal and float filters. Its `docs-routes.json` pins the routes each port documents | generated only | -| `report/` | 27 | a view-backed `object.report` (FR-044): list, filter, sort and paging on derived fields, `POST` is `405`, no `/{id}` route; a `@spine` report keeps a row with no facts, and a measure with `@default` filters and sorts as its default; a dimension over an array or a map is neither filterable nor sortable. Also carries `schema.postgres.sql`, the TypeScript-produced table and views the C# lane executes | generated only | +| `report/` | 37 | a view-backed `object.report` (FR-044): list, filter, sort and paging on derived fields, `POST` is `405`, no `/{id}` route; a `@spine` report keeps a row with no facts, and a measure with `@default` filters and sorts as its default; a dimension over an array or a map is neither filterable nor sortable. Also carries `schema.postgres.sql`, the TypeScript-produced table and views the C# lane executes | generated only | | `decimal/` | 11 | `field.decimal` is an exact canonical string on the wire (ADR-0019 Amendment 1): read, filter, sort, write and a report over decimals. A scenario's `engines:` key is exact; the `-15` twins run on SQLite and D1 as well. Also carries `schema.postgres.sql` / `schema.sqlite.sql` / `schema.mysql.sql` | generated only | `meta.json` declares a single canonical `Author` entity in the `acme::blog` diff --git a/fixtures/api-contract-conformance/secret-fields/README.md b/fixtures/api-contract-conformance/secret-fields/README.md new file mode 100644 index 0000000000..29bc632038 --- /dev/null +++ b/fixtures/api-contract-conformance/secret-fields/README.md @@ -0,0 +1,50 @@ +# `api-contract-conformance/secret-fields/` — #484 credentials and excluded fields + +Cross-port REST contract for the fields a generated API must not expose or accept: a +field excluded from forms (`@formExclude`) and a credential (a field that declares a +`view.password`). + +The model is one `Account`: + +| Field | Marked | Generated write routes | Generated read routes | +|-------------|-----------------------------|------------------------|-----------------------| +| `name` | (nothing) | accepted | returned | +| `tokenHash` | `@formExclude: true` | **ignored** | returned | +| `pinHash` | child `view.password` | **ignored** | **never returned** | + +## The contract + +- **A `@formExclude` or credential field is not writable through create, PATCH or PUT.** The + value is ignored, not rejected: the request answers as if the key were absent. That is the rule + for every key outside the settable set (`@mutability: writeOnce` on PATCH, the TPH discriminator, + `@autoSet: onCreate`), and it is load-bearing here too: a generated edit form submits every + field it was given, so a `400` on a present key would fail every save. +- **A present `null` is a write**, so it does not reach a denied column either. +- **A credential is never returned**: not by list, not by get, not in the echo of a create or + an update. The key is absent from the object, not `null`. +- **`@formExclude` alone does not hide a field.** It says how the field is written, and the + field is still part of the resource. +- The application still writes these fields through its own code (the generated queries, the + repository, the ORM). Only the generated HTTP boundary refuses them. + +## How `pinHash`'s write is asserted + +A credential is never read, so no scenario can observe over HTTP whether a write reached it. +Instead, each port's runner reads the stored rows back after `write-ignores-excluded` (from the +database, or from the in-memory repository behind the generated controller): the seeded rows +still hold their seeded `tokenHash` and `pinHash`, and the created row holds `null` in both. + +## Files + +``` +secret-fields/ +├── README.md +├── meta.json # Account (accounts table) +├── seed.json # two rows, both secrets set +└── scenarios/ + ├── read-omits-credential.yaml # GET by id and list: pinHash absent, tokenHash present + └── write-ignores-excluded.yaml # POST / PATCH / PUT / present-null leave tokenHash alone +``` + +Generated lane only: what is under test is whether each port's GENERATOR emits routes that +honour the marks, which a hand-rolled reference server would satisfy by construction. diff --git a/fixtures/api-contract-conformance/secret-fields/meta.json b/fixtures/api-contract-conformance/secret-fields/meta.json new file mode 100644 index 0000000000..d47906d8ad --- /dev/null +++ b/fixtures/api-contract-conformance/secret-fields/meta.json @@ -0,0 +1,18 @@ +{ + "metadata.root": { + "package": "acme::secrets", + "children": [ + { "object.entity": { + "name": "Account", + "children": [ + { "source.rdb": { "@table": "accounts" } }, + { "field.long": { "name": "id", "@filterable": true, "@sortable": true } }, + { "field.string": { "name": "name", "@required": true, "@maxLength": 100, "@filterable": true, "@sortable": true } }, + { "field.string": { "name": "tokenHash", "@maxLength": 100, "@formExclude": true } }, + { "field.string": { "name": "pinHash", "@maxLength": 100, "children": [ { "view.password": {} } ] } }, + { "identity.primary": { "name": "id", "@fields": "id", "@generation": "increment" } } + ] + }} + ] + } +} diff --git a/fixtures/api-contract-conformance/secret-fields/scenarios/read-omits-credential.yaml b/fixtures/api-contract-conformance/secret-fields/scenarios/read-omits-credential.yaml new file mode 100644 index 0000000000..92f48eb50f --- /dev/null +++ b/fixtures/api-contract-conformance/secret-fields/scenarios/read-omits-credential.yaml @@ -0,0 +1,33 @@ +name: read-omits-credential +description: > + A credential field (`view.password`, "masked input for sensitive values") is never + returned by a generated read route. `pinHash` is stored on every seeded row and is + absent from GET by id and from the list; `equals` is a deep-equal, so it asserts the + ABSENCE of the key and not merely that the value differs. + + `tokenHash` is `@formExclude` only. Excluding a field from a form is about WRITING it; + the field is still part of the resource and is returned. +requests: + - id: r1 + method: GET + path: /api/accounts/1 + expect: + status: 200 + body: + equals: + id: 1 + name: "Ada Lovelace" + tokenHash: "token-hash-1" + - id: r2 + method: GET + path: /api/accounts?sort=id + expect: + status: 200 + body: + equals: + - id: 1 + name: "Ada Lovelace" + tokenHash: "token-hash-1" + - id: 2 + name: "Alan Turing" + tokenHash: "token-hash-2" diff --git a/fixtures/api-contract-conformance/secret-fields/scenarios/write-ignores-excluded.yaml b/fixtures/api-contract-conformance/secret-fields/scenarios/write-ignores-excluded.yaml new file mode 100644 index 0000000000..05f5e36306 --- /dev/null +++ b/fixtures/api-contract-conformance/secret-fields/scenarios/write-ignores-excluded.yaml @@ -0,0 +1,72 @@ +name: write-ignores-excluded +description: > + A field excluded from forms (`@formExclude`) or marked as a credential (`view.password`) + is not writable through the generated routes: create, PATCH and PUT all leave it as it + was. The value is IGNORED, never rejected (HTTP 200/201, not 400), the same rule as + `@mutability: writeOnce` on PATCH (`writeonce-patch-stripped`), because a generated + edit form submits every field it was given. + + The ordinary `name` rides in each body DELIBERATELY: it is applied, and that pairing is + what makes the assertion about the field and not about the request being ignored whole. + `tokenHash` is the observable one (it is still returned); `pinHash` is hidden on read, + so each port's runner pins its write by reading the stored rows back after this + scenario (seeded values unchanged, null on the created row). +requests: + - id: r1 + method: PATCH + path: /api/accounts/1 + body: + name: "Ada L. (probe)" + tokenHash: "forged" + pinHash: "forged" + expect: + status: 200 + body: + equals: + id: 1 + name: "Ada L. (probe)" + tokenHash: "token-hash-1" + - id: r2 + method: PUT + path: /api/accounts/2 + body: + name: "Alan T. (probe)" + tokenHash: "forged" + pinHash: "forged" + expect: + status: 200 + body: + equals: + id: 2 + name: "Alan T. (probe)" + tokenHash: "token-hash-2" + - id: r3 + method: POST + path: /api/accounts + body: + name: "Grace Hopper" + tokenHash: "forged" + pinHash: "forged" + expect: + status: 201 + body: + row: + name: "Grace Hopper" + # Never written: the column stays null, and `forged` never landed. + tokenHash: null + hasId: true + - id: r4 + method: PATCH + path: /api/accounts/1 + body: + # Clearing is a write too: a present null must not reach a denied column. + name: "Ada L. (probe 2)" + tokenHash: null + pinHash: null + expect: + status: 200 + body: + equals: + id: 1 + name: "Ada L. (probe 2)" + tokenHash: "token-hash-1" diff --git a/fixtures/api-contract-conformance/secret-fields/seed.json b/fixtures/api-contract-conformance/secret-fields/seed.json new file mode 100644 index 0000000000..6697ba1fcb --- /dev/null +++ b/fixtures/api-contract-conformance/secret-fields/seed.json @@ -0,0 +1,6 @@ +{ + "rows": [ + { "id": 1, "name": "Ada Lovelace", "tokenHash": "token-hash-1", "pinHash": "pin-hash-1" }, + { "id": 2, "name": "Alan Turing", "tokenHash": "token-hash-2", "pinHash": "pin-hash-2" } + ] +} diff --git a/server/csharp/MetaObjects.Codegen.Tests/ReportCredentialCodegenTests.cs b/server/csharp/MetaObjects.Codegen.Tests/ReportCredentialCodegenTests.cs new file mode 100644 index 0000000000..6f23085912 --- /dev/null +++ b/server/csharp/MetaObjects.Codegen.Tests/ReportCredentialCodegenTests.cs @@ -0,0 +1,165 @@ +// ReportCredentialCodegenTests — #484 follow-up: a served report hides a credential. +// +// A report column is DERIVED from its `@of` field and a derived field keeps only type-shaping +// attrs, so a dimension (or a min/max measure) over a field that declares a view.password used to +// serve the credential. The row model now marks such a derived field as a credential, and the +// routes and the export treat it exactly as they treat an entity's. + +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using MetaObjects.Codegen.Generators; +using MetaObjects.Loader; +using MetaObjects.Meta; +using Xunit; + +namespace MetaObjects.Codegen.Tests; + +public class ReportCredentialCodegenTests +{ + private const string Model = """ + { "metadata.root": { "package": "acme::secrets", "children": [ + { "object.entity": { "name": "Account", "children": [ + { "source.rdb": { "@table": "accounts" } }, + { "field.long": { "name": "id", "@required": true } }, + { "field.string": { "name": "name", "@maxLength": 100 } }, + { "field.string": { "name": "pinHash", "@maxLength": 100, "children": [ { "view.password": {} } ] } }, + { "identity.primary": { "name": "id", "@fields": ["id"] } }, + { "dimension.attribute": { "name": "name", "@of": "Account.name" } }, + { "dimension.attribute": { "name": "pinHash", "@of": "Account.pinHash" } }, + { "measure.aggregate": { "name": "accounts", "@agg": "count", "@of": "Account.id" } }, + { "measure.aggregate": { "name": "firstPin", "@agg": "min", "@of": "Account.pinHash" } } + ] } }, + { "object.report": { "name": "AccountsByPin", "@from": "Account", "@dimensions": ["pinHash"], "@measures": ["accounts"], + "children": [ { "source.rdb": { "@kind": "view", "@view": "v_accounts_by_pin" } } ] } }, + { "object.report": { "name": "AccountPinRange", "@from": "Account", "@dimensions": ["name"], "@measures": ["firstPin"], + "children": [ { "source.rdb": { "@kind": "view", "@view": "v_account_pin_range" } } ] } }, + { "object.report": { "name": "AccountsByName", "@from": "Account", "@dimensions": ["name"], "@measures": ["accounts"], + "children": [ { "source.rdb": { "@kind": "view", "@view": "v_accounts_by_name" } } ] } } + ] } } + """; + + private static MetaRoot Load() + { + var r = new MetaDataLoader().Load([new InMemoryStringSource(Model, id: "meta.secrets.json")]); + Assert.True(r.Errors.Count == 0, string.Join("\n", r.Errors.Select(e => e.Message))); + return r.Root; + } + + private static GenContext Ctx(MetaRoot root) => new() + { + Entities = root.Objects(), + Root = root, + Config = new GenConfig { OutDir = "/unused", Namespace = "Acme.Gen", ColumnNamingStrategy = ColumnNamingStrategy.Literal, IncludeNames = true }, + Generators = [new RoutesGenerator(), new ReportExportGenerator { MaxRows = 60 }], + }; + + private static string Routes(GenContext ctx, string report) => + new RoutesGenerator().Generate(ctx).Single(f => f.Path == report + "Routes.g.cs").Content; + + private static string Export(GenContext ctx, string report) => + new ReportExportGenerator { MaxRows = 60 }.Generate(ctx).Single(f => f.Path == report + "Export.g.cs").Content; + + [Fact] + public void The_row_model_marks_a_derived_credential() + { + var root = Load(); + var pin = ReportRows.RowModel(root.FindObject("AccountsByPin")!, root).FindField("pinHash")!; + Assert.True(RoutesGenerator.IsCredentialField(pin)); + var range = ReportRows.RowModel(root.FindObject("AccountPinRange")!, root).FindField("firstPin")!; + Assert.True(RoutesGenerator.IsCredentialField(range)); + var plain = ReportRows.RowModel(root.FindObject("AccountsByName")!, root); + Assert.All(plain.Fields(), f => Assert.False(RoutesGenerator.IsCredentialField(f))); + } + + [Theory] + [InlineData("AccountsByPin", "pinHash")] + [InlineData("AccountPinRange", "firstPin")] + public void A_report_over_a_credential_omits_it_from_rows_and_export(string report, string column) + { + var ctx = Ctx(Load()); + var routes = Routes(ctx, report); + Assert.Contains($"new string[] {{ \"{column}\" }}", routes); + Assert.DoesNotContain("Results.Ok(rows)", routes); + // The JSON export strips it too; the CSV / XML column list names no credential. + Assert.Contains($"rows, new string[] {{ \"{column}\" }}", routes); + var export = Export(ctx, report); + Assert.DoesNotContain($"\"{column}\"", export); + Assert.DoesNotContain($"row.{char.ToUpperInvariant(column[0])}{column[1..]}", export); + } + + [Fact] + public void A_plain_report_is_unchanged() + { + var ctx = Ctx(Load()); + var routes = Routes(ctx, "AccountsByName"); + Assert.DoesNotContain("CredentialOmittingResult", routes); + Assert.Contains("Results.Ok(rows)", routes); + Assert.Contains("ReportExportHttp.Json(http, ", routes); + Assert.Contains("\", rows);", routes); + Assert.Contains("\"name\"", Export(ctx, "AccountsByName")); + } + + // A hidden column that can still narrow or order the rows is an oracle: `?filter[pinHash][like]=ab%` + // or `?sort=pinHash` would recover the secret a prefix at a time. + [Theory] + [InlineData("AccountsByPin", "pinHash", "accounts")] + [InlineData("AccountPinRange", "firstPin", "name")] + public void A_credential_column_is_neither_filterable_nor_sortable(string report, string column, string kept) + { + var ctx = Ctx(Load()); + var allowlist = new FilterAllowlistGenerator().Generate(ctx).Single(f => f.Path == report + "FilterAllowlist.g.cs").Content; + Assert.DoesNotContain($"\"{column}\"", allowlist); + Assert.Contains($"\"{kept}\"", allowlist); + + var routes = Routes(ctx, report); + var pascal = char.ToUpperInvariant(column[0]) + column[1..]; + Assert.DoesNotContain($"\"{pascal}\"", routes); + Assert.Contains($"\"{char.ToUpperInvariant(kept[0])}{kept[1..]}\"", routes); + } + + [Fact] + public void A_plain_reports_allowlists_keep_every_column() + { + var ctx = Ctx(Load()); + var allowlist = new FilterAllowlistGenerator().Generate(ctx).Single(f => f.Path == "AccountsByNameFilterAllowlist.g.cs").Content; + Assert.Contains("\"name\"", allowlist); + Assert.Contains("\"accounts\"", allowlist); + var routes = Routes(ctx, "AccountsByName"); + Assert.Contains("\"Name\"", routes); + Assert.Contains("\"Accounts\"", routes); + } + + [Fact] + public void The_credential_report_routes_and_export_compile() + { + var ctx = Ctx(Load()); + var files = new IGenerator[] + { + new EntityGenerator(), new DbContextGenerator(), new NamesGenerator(), + new FilterAllowlistGenerator(), new RoutesGenerator(), new ReportExportGenerator { MaxRows = 60 }, + } + .SelectMany(g => g.Generate(ctx)).ToList(); + + var paths = DbContextCompileTests.BuildReferences() + .OfType().Select(r => r.FilePath!) + .ToHashSet(StringComparer.OrdinalIgnoreCase); + string aspNetDir = Path.GetDirectoryName(typeof(Microsoft.AspNetCore.Http.IQueryCollection).Assembly.Location)!; + foreach (var dll in Directory.GetFiles(aspNetDir, "*.dll")) paths.Add(dll); + paths.Add(typeof(Microsoft.AspNetCore.Builder.WebApplication).Assembly.Location); + paths.Add(typeof(Microsoft.AspNetCore.Http.Results).Assembly.Location); + paths.Add(typeof(RoutesGenerator).Assembly.Location); + + var trees = files + .Select(f => CSharpSyntaxTree.ParseText(f.Content, new CSharpParseOptions(LanguageVersion.CSharp12), path: f.Path)) + .ToList(); + var comp = CSharpCompilation.Create( + "report_credential_" + Guid.NewGuid().ToString("N"), trees, + paths.Where(File.Exists).Select(p => (MetadataReference)MetadataReference.CreateFromFile(p)).ToList(), + new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary)); + var errors = comp.GetDiagnostics() + .Where(d => d.Severity == DiagnosticSeverity.Error) + .Select(d => $"{d.Location.GetLineSpan().Path}: {d.Id}: {d.GetMessage()}") + .ToList(); + Assert.True(errors.Count == 0, string.Join("\n", errors)); + } +} diff --git a/server/csharp/MetaObjects.Codegen.Tests/ReportExportGeneratorTests.cs b/server/csharp/MetaObjects.Codegen.Tests/ReportExportGeneratorTests.cs index 774d3a79e1..2c3fe6fe11 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/ReportExportGeneratorTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/ReportExportGeneratorTests.cs @@ -254,4 +254,48 @@ public void No_format_is_never_refused_and_withCount_is_refused_for_csv_and_xml_ Assert.Null(ReportExportHttp.Refuse("csv", Q(("format", "csv"), ("withCount", "0")))); Assert.Equal(["csv", "json", "xml"], ReportExportHttp.Formats); } + + // docs/features/api-contract.md, "Report export": a credential column (a grouped or min/max + // view.password field) is never returned, so its key is absent from a JSON export's rows, + // with or without the { rows, total } envelope, and the report's other columns stay. + public sealed record PinRow(string Name, string PinHash, long Accounts); + + private static async Task ExportJson(object body, bool envelope, bool camelCase) + { + var services = new Microsoft.Extensions.DependencyInjection.ServiceCollection(); + Microsoft.Extensions.DependencyInjection.LoggingServiceCollectionExtensions.AddLogging(services); + Microsoft.Extensions.DependencyInjection.OptionsServiceCollectionExtensions.Configure( + services, o => + { + o.SerializerOptions.PropertyNamingPolicy = camelCase ? System.Text.Json.JsonNamingPolicy.CamelCase : null; + o.SerializerOptions.PropertyNameCaseInsensitive = false; + }); + var http = new DefaultHttpContext + { + RequestServices = Microsoft.Extensions.DependencyInjection.ServiceCollectionContainerBuilderExtensions.BuildServiceProvider(services), + }; + http.Response.Body = new MemoryStream(); + await ReportExportHttp.Json(http, "accounts-by-pin", body, ["pinHash"], envelope).ExecuteAsync(http); + Assert.Equal("attachment; filename=\"accounts-by-pin.json\"", http.Response.Headers.ContentDisposition.ToString()); + http.Response.Body.Position = 0; + return System.Text.Json.Nodes.JsonNode.Parse(http.Response.Body)!; + } + + [Theory] + [InlineData(false, true), InlineData(true, true), InlineData(false, false), InlineData(true, false)] + public async Task Json_export_omits_the_credential_key_from_every_row(bool envelope, bool camelCase) + { + PinRow[] rows = [new("Ann", "hash-a", 2), new("Bob", "hash-b", 3)]; + var json = await ExportJson(envelope ? new { rows, total = 2 } : rows, envelope, camelCase); + var served = (envelope ? json["rows"] : json)!.AsArray(); + Assert.Equal(2, served.Count); + foreach (var row in served.Select(r => r!.AsObject())) + { + Assert.DoesNotContain(row, p => string.Equals(p.Key, "pinHash", StringComparison.OrdinalIgnoreCase)); + Assert.Contains(row, p => string.Equals(p.Key, "accounts", StringComparison.OrdinalIgnoreCase)); + Assert.Contains(row, p => string.Equals(p.Key, "name", StringComparison.OrdinalIgnoreCase)); + } + Assert.DoesNotContain("hash-", json.ToJsonString()); + if (envelope) Assert.Equal(2, json["total"]!.GetValue()); + } } diff --git a/server/csharp/MetaObjects.Codegen.Tests/SecretFieldsCodegenTests.cs b/server/csharp/MetaObjects.Codegen.Tests/SecretFieldsCodegenTests.cs new file mode 100644 index 0000000000..e84bc9e601 --- /dev/null +++ b/server/csharp/MetaObjects.Codegen.Tests/SecretFieldsCodegenTests.cs @@ -0,0 +1,140 @@ +// SecretFieldsCodegenTests — #484: the generated routes ignore a write-denied key (@formExclude +// or a credential) and never send a credential. The HTTP behaviour is gated by the api-contract +// secret-fields corpus; this pins the emission, including the one thing HTTP cannot observe: a +// credential's write path names it exactly as an @formExclude field's. + +using MetaObjects.Codegen; +using MetaObjects.Codegen.Generators; +using MetaObjects.Loader; +using MetaObjects.Meta; +using Xunit; + +namespace MetaObjects.Codegen.Tests; + +public class SecretFieldsCodegenTests +{ + private const string Model = """ + { "metadata.root": { "package": "acme::secrets", "children": [ + { "object.entity": { "name": "Account", "children": [ + { "source.rdb": { "@table": "accounts" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "name", "@maxLength": 100 } }, + { "field.string": { "name": "tokenHash", "@maxLength": 100, "@formExclude": true } }, + { "field.string": { "name": "pinHash", "@maxLength": 100, "children": [ { "view.password": {} } ] } }, + { "identity.primary": { "@fields": "id" } } + ]}} + ]}} + """; + + // The same entity with an INHERITED credential and an inherited @formExclude (ADR-0039). + private const string InheritedModel = """ + { "metadata.root": { "package": "acme::secrets", "children": [ + { "object.entity": { "name": "Base", "abstract": true, "children": [ + { "field.string": { "name": "tokenHash", "@maxLength": 100, "@formExclude": true } }, + { "field.string": { "name": "pinHash", "@maxLength": 100, "children": [ { "view.password": {} } ] } } + ]}}, + { "object.entity": { "name": "Account", "children": [ + { "source.rdb": { "@table": "accounts" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "tokenHash", "extends": "Base.tokenHash" } }, + { "field.string": { "name": "pinHash", "extends": "Base.pinHash" } }, + { "identity.primary": { "@fields": "id" } } + ]}} + ]}} + """; + + private const string PlainModel = """ + { "metadata.root": { "package": "acme::secrets", "children": [ + { "object.entity": { "name": "Account", "children": [ + { "source.rdb": { "@table": "accounts" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "name", "@maxLength": 100 } }, + { "identity.primary": { "@fields": "id" } } + ]}} + ]}} + """; + + // A TPH base whose subtypes share the token and memo columns: plain on AThing, marked on BThing. + private const string TphSharedColumnModel = """ + { "metadata.root": { "package": "acme::secrets", "children": [ + { "object.entity": { "name": "Thing", "@discriminator": "kind", "children": [ + { "source.rdb": { "@table": "things" } }, + { "field.long": { "name": "id" } }, + { "field.enum": { "name": "kind", "@values": ["A", "B"] } }, + { "identity.primary": { "@fields": "id", "@generation": "increment" } } + ]}}, + { "object.entity": { "name": "AThing", "extends": "Thing", "@discriminatorValue": "A", "children": [ + { "field.string": { "name": "token", "@maxLength": 100 } }, + { "field.string": { "name": "memo", "@maxLength": 100 } } + ]}}, + { "object.entity": { "name": "BThing", "extends": "Thing", "@discriminatorValue": "B", "children": [ + { "field.string": { "name": "token", "@maxLength": 100, "children": [ { "view.password": {} } ] } }, + { "field.string": { "name": "memo", "@maxLength": 100, "@formExclude": true } } + ]}} + ]}} + """; + + private static string Routes(string model, string file = "AccountRoutes.g.cs") + { + var r = new MetaDataLoader().Load([new InMemoryStringSource(model, id: "secrets.json")]); + Assert.Empty(r.Errors); + var ctx = new GenContext + { + Entities = r.Root.Objects(), + Root = r.Root, + Config = new GenConfig { OutDir = "/tmp", Namespace = "Acme.Generated", ColumnNamingStrategy = ColumnNamingStrategy.Literal }, + }; + return new RoutesGenerator().Generate(ctx).Single(f => f.Path == file).Content; + } + + [Fact] + public void A_tph_column_marked_on_another_subtype_is_hidden_and_ignored_on_this_one() + { + var src = Routes(TphSharedColumnModel, "ThingRoutes.g.cs"); + var start = src.IndexOf("\"/things/a\"", StringComparison.Ordinal); + var end = src.IndexOf("\"/things/b\"", StringComparison.Ordinal); + Assert.True(start >= 0 && end > start, src); + var aRoutes = src[start..end]; + + Assert.Contains("new string[] { \"token\" }", aRoutes); + Assert.Contains("input.Token = default!;", aRoutes); + Assert.Contains("input.Memo = default!;", aRoutes); + Assert.Contains("string.Equals(prop.Name, \"token\", System.StringComparison.OrdinalIgnoreCase)) continue;", aRoutes); + Assert.Contains("string.Equals(prop.Name, \"memo\", System.StringComparison.OrdinalIgnoreCase)) continue;", aRoutes); + Assert.DoesNotContain("\"memo\" }", aRoutes); + } + + [Theory] + [InlineData(Model)] + [InlineData(InheritedModel)] + public void Write_paths_name_both_denied_fields_and_read_hides_only_the_credential(string model) + { + var src = Routes(model); + + // Create resets both; PATCH / PUT skip both (a present null is a write too). + Assert.Contains("input.TokenHash = default!;", src); + Assert.Contains("input.PinHash = default!;", src); + Assert.Contains("string.Equals(prop.Name, \"tokenHash\", System.StringComparison.OrdinalIgnoreCase)) continue;", src); + Assert.Contains("string.Equals(prop.Name, \"pinHash\", System.StringComparison.OrdinalIgnoreCase)) continue;", src); + + // Only the credential is omitted on read (list, get, create echo, update echo). + Assert.Contains("new string[] { \"pinHash\" }", src); + Assert.DoesNotContain("\"tokenHash\" }", src); + Assert.DoesNotContain("new string[] { \"tokenHash\"", src); + Assert.DoesNotContain("Results.Ok(rows)", src); + Assert.DoesNotContain("Results.Ok(found)", src); + Assert.DoesNotContain("Results.Ok(existing)", src); + Assert.DoesNotContain("Results.Created(", src); + } + + [Fact] + public void A_model_with_neither_mark_is_unchanged() + { + var src = Routes(PlainModel); + Assert.DoesNotContain("CredentialOmittingResult", src); + Assert.DoesNotContain("default!;", src); + Assert.DoesNotContain("formExclude", src, StringComparison.OrdinalIgnoreCase); + Assert.Contains("Results.Ok(rows)", src); + Assert.Contains("Results.Ok(existing)", src); + } +} diff --git a/server/csharp/MetaObjects.Codegen/Generators/ReportExportGenerator.cs b/server/csharp/MetaObjects.Codegen/Generators/ReportExportGenerator.cs index f9b770af29..60fdca3d03 100644 --- a/server/csharp/MetaObjects.Codegen/Generators/ReportExportGenerator.cs +++ b/server/csharp/MetaObjects.Codegen/Generators/ReportExportGenerator.cs @@ -90,7 +90,8 @@ public IEnumerable Generate(GenContext ctx) private string ReportModule(MetaObject report, GenContext ctx) { var cls = CSharpNaming.Pascal(report.Name); - var fields = report.Fields().ToList(); + // A credential column (a derived field over a view.password field) is never exported. + var fields = report.Fields().Where(f => !f.Views().Any(v => v.SubType == MetaObjects.Presentation.View.ViewConstants.VIEW_SUBTYPE_PASSWORD)).ToList(); var sb = new StringBuilder(); sb.AppendLine("// "); sb.AppendLine("// Generated by MetaObjects report-export. Do not edit by hand."); diff --git a/server/csharp/MetaObjects.Codegen/Generators/RoutesGenerator.cs b/server/csharp/MetaObjects.Codegen/Generators/RoutesGenerator.cs index f939ec99ee..12f9b08bc5 100644 --- a/server/csharp/MetaObjects.Codegen/Generators/RoutesGenerator.cs +++ b/server/csharp/MetaObjects.Codegen/Generators/RoutesGenerator.cs @@ -229,6 +229,9 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte var autoSetNavs = autoSetFields.Select(a => a.Nav).ToList(); // FR-037 R1 — readOnly / writeOnce are excluded from the PATCH merge. var frozenNavs = FrozenNavs(entity); + // #484 — write-denied (@formExclude or credential) keys are ignored on every write; credentials never leave. + var deniedFields = WriteDeniedFields(entity); + var credentials = CredentialNames(entity); // Physical table + PK column for the post-save array-null clear (EF OwnsMany ToJson // writes [] for a null nav, so a nullable array column is NULLed via raw SQL). Both // are C# EXPRESSIONS — the Names constant when the names artifact is in this @@ -258,7 +261,8 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte var sortFields = entity.Fields() .Where(f => (CSharpNaming.ScalarFor(f.SubType) is not null || (isReport && f.SubType == FIELD_SUBTYPE_ENUM)) - && !f.ResolvedIsArray()) + && !f.ResolvedIsArray() + && !(isReport && IsCredentialField(f))) // a credential column would be a sort oracle .Select(f => CSharpNaming.Pascal(f.Name)) .ToList(); @@ -359,15 +363,15 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte sb.AppendLine(" // withCount=1 → { rows, total } envelope; otherwise bare row array."); sb.AppendLine(" var withCount = qs.TryGetValue(\"withCount\", out var wc) && (wc == \"1\" || string.Equals(wc, \"true\", System.StringComparison.OrdinalIgnoreCase));"); if (export is null) - sb.AppendLine(" if (!withCount) return Results.Ok(rows);"); + sb.AppendLine($" if (!withCount) return {OkResult("rows", credentials)};"); else - sb.AppendLine($" if (!withCount) return format is null ? Results.Ok(rows) : ReportExportHttp.Json(http, \"{route}\", rows);"); + sb.AppendLine($" if (!withCount) return format is null ? {OkResult("rows", credentials)} : ReportExportHttp.Json(http, \"{route}\", rows{ExportOmit(credentials, false)});"); sb.AppendLine($" var totalQ = EfCoreFilterDispatch.ApplyFilter(db.{readDbSet}.AsNoTracking(), filter.Predicates);"); sb.AppendLine(" var total = await totalQ.CountAsync();"); if (export is null) - sb.AppendLine(" return Results.Ok(new { rows, total });"); + sb.AppendLine($" return {OkResult("new { rows, total }", credentials, envelope: true)};"); else - sb.AppendLine($" return format is null ? Results.Ok(new {{ rows, total }}) : ReportExportHttp.Json(http, \"{route}\", new {{ rows, total }});"); + sb.AppendLine($" return format is null ? {OkResult("new { rows, total }", credentials, envelope: true)} : ReportExportHttp.Json(http, \"{route}\", new {{ rows, total }}{ExportOmit(credentials, true)});"); sb.AppendLine(" });"); if (hasItem) @@ -375,7 +379,7 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte sb.AppendLine(); sb.AppendLine(" app.MapGet(prefix + \"/" + route + "/{id}\", async (" + pkType + " id, AppDbContext db) =>"); sb.AppendLine(" await db." + readDbSet + ".FindAsync(id) is { } found"); - sb.AppendLine(" ? Results.Ok(found)"); + sb.AppendLine(" ? " + OkResult("found", credentials)); sb.AppendLine(" : Results.NotFound(new { error = \"not_found\" }));"); } @@ -391,8 +395,9 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte // the table, then RE-READS the row through the replica view by PK so the returned // body carries the derived (view-computed) fields (read-your-writes). AppendCreateHandler(sb, "/" + route, cls, dbSet, requiredKeys, - "Results.Created(prefix + \"/" + route + "\", input)", voFields, mapVoFields, autoSetFields, - writeThrough ? (readDbSet, pkProp!, route) : null, parseBody: HasDecimalField(entity)); + CreatedResult("prefix + \"/" + route + "\"", "input", credentials), voFields, mapVoFields, autoSetFields, + writeThrough ? (readDbSet, pkProp!, route) : null, parseBody: HasDecimalField(entity), + deniedFields.Select(f => CSharpNaming.Pascal(f.Name)).ToList(), credentials); // PATCH + PUT share the same handler — TS reference exposes both verbs, and both // return the updated row (HTTP 200), matching the cross-port api-contract. @@ -427,7 +432,8 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte sb.AppendLine(" .Value.SerializerOptions;"); sb.AppendLine(" var entry = db.Entry(existing);"); AppendUpdateAutoSet(sb, autoSetFields); - AppendPartialMergeLoop(sb, null, voFields, mapVoFields, autoSetNavs, frozenNavs, HasDecimalField(entity)); + AppendPartialMergeLoop(sb, null, voFields, mapVoFields, autoSetNavs, frozenNavs, HasDecimalField(entity), + deniedFields.Select(f => f.Name).ToList()); AppendSaveChanges(sb); AppendArrayNullClears(sb, voFields, tableRef, pkColumnRef); // #214 — writes target the table; for a write-through entity re-read the row @@ -438,11 +444,11 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte // #214 [2] — a @kind:materializedView (unrefreshed) or filtered replica view may // not surface the just-written row; fall back to the write entity so the body is // never null (degraded — missing only the derived fields; matches the Python port). - sb.AppendLine(" return Results.Ok((object?)__view ?? existing);"); + sb.AppendLine($" return {OkResult("(object?)__view ?? existing", credentials)};"); } else { - sb.AppendLine(" return Results.Ok(existing);"); + sb.AppendLine($" return {OkResult("existing", credentials)};"); } sb.AppendLine(" }"); sb.AppendLine(" app.MapPatch(prefix + \"/" + route + "/{id}\", Update" + cls + ");"); @@ -495,6 +501,7 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte // fields (writable entities only; a read-only projection has no insert path). if (writable && autoSetFields.Count > 0) AppendInsertPreserving(sb, cls, dbSet); + AppendOmitResultClass(sb); sb.AppendLine("}"); return new EmittedFile($"{cls}Routes.g.cs", sb.ToString()); @@ -567,6 +574,11 @@ protected virtual EmittedFile GenerateTphRoutes(MetaObject baseEntity, TphPlan t bool hasItem = pkType is not null; var discProp = CSharpNaming.Pascal(tph.DiscriminatorField); + // #484 — the polymorphic base rows carry every subtype's columns, so a credential declared + // on the base OR on any subtype is omitted from them. + var baseCredentials = CredentialNames(baseEntity) + .Concat(tph.Subtypes.SelectMany(st => CredentialNames(st.Entity))) + .Distinct(StringComparer.Ordinal).ToList(); // Sortable base scalar fields (the polymorphic list sorts on base columns). var sortFields = baseEntity.Fields() @@ -646,9 +658,9 @@ protected virtual EmittedFile GenerateTphRoutes(MetaObject baseEntity, TphPlan t sb.AppendLine(" if (qs.TryGetValue(\"limit\", out var limRaw) && int.TryParse(limRaw, System.Globalization.NumberStyles.Integer, System.Globalization.CultureInfo.InvariantCulture, out var lim) && lim > 0) q = q.Take(lim);"); sb.AppendLine(" var rows = await q.ToListAsync();"); sb.AppendLine(" var withCount = qs.TryGetValue(\"withCount\", out var wc) && (wc == \"1\" || string.Equals(wc, \"true\", System.StringComparison.OrdinalIgnoreCase));"); - sb.AppendLine(" if (!withCount) return Results.Ok(rows);"); + sb.AppendLine($" if (!withCount) return {OkResult("rows", baseCredentials)};"); sb.AppendLine($" var total = await EfCoreFilterDispatch.ApplyFilter(db.{dbSet}.AsNoTracking(), filter.Predicates).CountAsync();"); - sb.AppendLine(" return Results.Ok(new { rows, total });"); + sb.AppendLine($" return {OkResult("new { rows, total }", baseCredentials, envelope: true)};"); sb.AppendLine(" });"); if (hasItem) @@ -657,7 +669,7 @@ protected virtual EmittedFile GenerateTphRoutes(MetaObject baseEntity, TphPlan t sb.AppendLine(); sb.AppendLine(" app.MapGet(prefix + \"/" + baseRoute + "/{id}\", async (" + pkType + " id, AppDbContext db) =>"); sb.AppendLine($" await db.{dbSet}.AsNoTracking().FirstOrDefaultAsync(x => x.{pkProp} == id) is {{ }} found"); - sb.AppendLine(" ? Results.Ok(found)"); + sb.AppendLine(" ? " + OkResult("found", baseCredentials)); sb.AppendLine(" : Results.NotFound(new { error = \"not_found\" }));"); // FW-8 — M:N traversal declared on the BASE: every row of the shared table @@ -670,8 +682,9 @@ protected virtual EmittedFile GenerateTphRoutes(MetaObject baseEntity, TphPlan t AppendM2mRoute(sb, nav, baseRoute, pkType!, ctx.Root); // --- Per-subtype CRUD sets --- + var hierarchy = tph.Subtypes.Select(s => s.Entity).Prepend(baseEntity).ToList(); foreach (var st in tph.Subtypes) - AppendTphSubtypeRoutes(sb, st, ctx.Root, baseRoute, baseCls, dbSet, pkType!, pkProp!, discProp, ctx.Warn); + AppendTphSubtypeRoutes(sb, st, hierarchy, ctx.Root, baseRoute, baseCls, dbSet, pkType!, pkProp!, discProp, ctx.Warn); } sb.AppendLine(); @@ -696,6 +709,7 @@ protected virtual EmittedFile GenerateTphRoutes(MetaObject baseEntity, TphPlan t foreach (var st in tph.Subtypes) if (AutoSetFields(st.Entity).Count > 0) AppendInsertPreserving(sb, CSharpNaming.Pascal(st.Entity.Name), dbSet); + AppendOmitResultClass(sb); sb.AppendLine("}"); return new EmittedFile($"{baseCls}Routes.g.cs", sb.ToString()); @@ -751,6 +765,103 @@ private static List FrozenNavs(MetaObject entity) => .Select(f => CSharpNaming.Pascal(f.Name)) .ToList(); + // #484 — a credential is a field that declares (or inherits, ADR-0039) a view.password child. + // A credential is never returned by a generated route; it is also never written by one. + public static bool IsCredentialField(MetaField f) => + f.Views().Any(v => v.SubType == MetaObjects.Presentation.View.ViewConstants.VIEW_SUBTYPE_PASSWORD); + + // #484 — write-denied: the generated create / PATCH / PUT IGNORE the key (never a 400): the + // effective @formExclude (resolving) or a credential. + private static bool IsApiWriteDenied(MetaField f) => + f.Attr(FIELD_ATTR_FORM_EXCLUDE) is true || IsCredentialField(f); + + private static List WriteDeniedFields(MetaObject entity) => + entity.Fields().Where(IsApiWriteDenied).ToList(); + + /// + /// An entity's credential names plus, for a TPH base, those of every subtype: its rows are + /// shared-table rows of every subtype (#484). + /// + private static List TphUnionCredentialNames(MetaObject entity, MetaRoot root) => + CredentialNames(entity) + .Concat(TphPlanBuilder.For(entity, root) is { } tph + ? tph.Subtypes.SelectMany(st => CredentialNames(st.Entity)) + : Enumerable.Empty()) + .Distinct(StringComparer.Ordinal).ToList(); + + /// + /// The fields of a TPH subtype that holds for on ANY declaration in + /// its (the base and every subtype): a single-table column declared + /// by several subtypes is one column (#484). + /// + private static List HierarchyMarkedFields( + MetaObject subtype, IEnumerable hierarchy, Func isMarked) + { + var marked = hierarchy.SelectMany(o => o.Fields()).Where(isMarked) + .Select(f => f.Name).ToHashSet(StringComparer.Ordinal); + return subtype.Fields().Where(f => marked.Contains(f.Name)).ToList(); + } + + private static List CredentialNames(MetaObject entity) => + entity.Fields().Where(IsCredentialField).Select(f => f.Name).ToList(); + + // A C# string[] expression of the names, spliced into the credential-omitting result. + private static string NamesLiteral(IEnumerable names) => + "new string[] { " + string.Join(", ", names.Select(n => "\"" + n + "\"")) + " }"; + + // The ok / created result for a body that may carry credentials: byte-identical `Results.*` + // when there are none, else the nested CredentialOmittingResult (emitted by AppendOmitResultClass). + private static string OkResult(string value, IReadOnlyList credentials, bool envelope = false) => + credentials.Count == 0 + ? $"Results.Ok({value})" + : $"(IResult)new CredentialOmittingResult({value}, {NamesLiteral(credentials)}{(envelope ? ", envelope: true" : "")})"; + + // The credential arguments of a JSON report export (ReportExportHttp.Json), "" without a credential. + private static string ExportOmit(IReadOnlyList credentials, bool envelope) => + credentials.Count == 0 ? "" : $", {NamesLiteral(credentials)}{(envelope ? ", envelope: true" : "")}"; + + private static string CreatedResult(string location, string value, IReadOnlyList credentials) => + credentials.Count == 0 + ? $"Results.Created({location}, {value})" + : $"(IResult)new CredentialOmittingResult({value}, {NamesLiteral(credentials)}, status: 201, location: {location})"; + + // Emitted only into a routes class that uses it (never for a model without a credential). + private static void AppendOmitResultClass(StringBuilder sb) + { + if (!sb.ToString().Contains("new CredentialOmittingResult(")) return; + sb.AppendLine(); + sb.AppendLine(" // #484 — a credential (view.password) is never sent: the key is OMITTED from the body, not nulled."); + sb.AppendLine(" // Serializes with the app's configured JSON options, then removes the named keys from each row."); + sb.AppendLine(" private sealed class CredentialOmittingResult : IResult"); + sb.AppendLine(" {"); + sb.AppendLine(" private readonly object _value;"); + sb.AppendLine(" private readonly string[] _names;"); + sb.AppendLine(" private readonly bool _envelope;"); + sb.AppendLine(" private readonly int _status;"); + sb.AppendLine(" private readonly string? _location;"); + sb.AppendLine(" public CredentialOmittingResult(object value, string[] names, bool envelope = false, int status = 200, string? location = null)"); + sb.AppendLine(" { _value = value; _names = names; _envelope = envelope; _status = status; _location = location; }"); + sb.AppendLine(" public async System.Threading.Tasks.Task ExecuteAsync(HttpContext http)"); + sb.AppendLine(" {"); + sb.AppendLine(" var opts = ((Microsoft.Extensions.Options.IOptions)"); + sb.AppendLine(" http.RequestServices.GetService(typeof(Microsoft.Extensions.Options.IOptions))!)"); + sb.AppendLine(" .Value.SerializerOptions;"); + sb.AppendLine(" var node = System.Text.Json.JsonSerializer.SerializeToNode(_value, _value.GetType(), opts);"); + sb.AppendLine(" Strip(_envelope && node is System.Text.Json.Nodes.JsonObject env ? env[\"rows\"] : node);"); + sb.AppendLine(" http.Response.StatusCode = _status;"); + sb.AppendLine(" if (_location is not null) http.Response.Headers.Location = _location;"); + sb.AppendLine(" await http.Response.WriteAsJsonAsync(node, opts);"); + sb.AppendLine(" }"); + sb.AppendLine(" private void Strip(System.Text.Json.Nodes.JsonNode? n)"); + sb.AppendLine(" {"); + sb.AppendLine(" if (n is System.Text.Json.Nodes.JsonArray a) { foreach (var e in a) Strip(e); return; }"); + sb.AppendLine(" if (n is not System.Text.Json.Nodes.JsonObject o) return;"); + sb.AppendLine(" foreach (var key in o.Select(p => p.Key).Where(k => _names.Any(x => string.Equals(x, k, System.StringComparison.OrdinalIgnoreCase))).ToList())"); + sb.AppendLine(" o.Remove(key);"); + sb.AppendLine(" }"); + sb.AppendLine(" }"); + } + private static List AutoSetFields(MetaObject entity) { var list = new List(); @@ -853,7 +964,9 @@ private static void AppendCreateHandler( IReadOnlyList mapVoFields, IReadOnlyList autoSetFields, (string ReadDbSet, string PkProp, string Route)? reRead = null, - bool parseBody = false) + bool parseBody = false, + IReadOnlyList? deniedNavs = null, + IReadOnlyList? credentials = null) { // A decimal field can refuse its value while the body is bound (out of declared range, not a // number). The framework's own binding answers that with a bare 400 and no body, so an entity @@ -862,10 +975,11 @@ private static void AppendCreateHandler( { sb.AppendLine($" app.MapPost(prefix + \"{mapPath}\", async ({cls} input, AppDbContext db) =>"); sb.AppendLine(" {"); + AppendCreateIgnoreDenied(sb, deniedNavs); AppendCreateValidation(sb); AppendCreateVoValidation(sb, voFields, mapVoFields); AppendCreateAutoSet(sb, autoSetFields); - AppendCreatePersist(sb, dbSet, createdExpr, reRead); + AppendCreatePersist(sb, dbSet, createdExpr, reRead, credentials ?? []); sb.AppendLine(" });"); return; } @@ -897,10 +1011,11 @@ private static void AppendCreateHandler( sb.AppendLine(" try { input = System.Text.Json.JsonSerializer.Deserialize<" + cls + ">(__body.RootElement.GetRawText(), jsonOpts); }"); sb.AppendLine(" catch (System.Text.Json.JsonException) { return Results.BadRequest(new { error = \"validation\" }); }"); sb.AppendLine(" if (input is null) return Results.BadRequest(new { error = \"validation\" });"); + AppendCreateIgnoreDenied(sb, deniedNavs); AppendCreateValidation(sb); AppendCreateVoValidation(sb, voFields, mapVoFields); AppendCreateAutoSet(sb, autoSetFields); - AppendCreatePersist(sb, dbSet, createdExpr, reRead); + AppendCreatePersist(sb, dbSet, createdExpr, reRead, credentials ?? []); sb.AppendLine(" });"); } @@ -911,7 +1026,8 @@ private static void AppendCreateHandler( // carries the derived (view-computed) fields. Assumes `input` + `db` are in scope. private static void AppendCreatePersist( StringBuilder sb, string dbSet, string createdExpr, - (string ReadDbSet, string PkProp, string Route)? reRead) + (string ReadDbSet, string PkProp, string Route)? reRead, + IReadOnlyList credentials) { sb.AppendLine($" db.{dbSet}.Add(input);"); AppendSaveChanges(sb); @@ -921,7 +1037,7 @@ private static void AppendCreatePersist( // #214 [2] — a @kind:materializedView (unrefreshed) or filtered replica view may not // surface the just-inserted row; fall back to the bound write `input` so the 201 body // is never null (degraded — missing only the derived fields; matches the Python port). - sb.AppendLine($" return Results.Created(prefix + \"/{rr.Route}/\" + input.{rr.PkProp}, (object?)__created ?? input);"); + sb.AppendLine($" return {CreatedResult($"prefix + \"/{rr.Route}/\" + input.{rr.PkProp}", "(object?)__created ?? input", credentials)};"); } else { @@ -929,6 +1045,16 @@ private static void AppendCreatePersist( } } + // #484 — a write-denied key (@formExclude or credential) on a create body is IGNORED, never + // rejected: the bound value is reset to the CLR default (unset / null) before validation. + private static void AppendCreateIgnoreDenied(StringBuilder sb, IReadOnlyList? deniedNavs) + { + if (deniedNavs is null || deniedNavs.Count == 0) return; + sb.AppendLine(" // @formExclude / credential: not settable through the generated create; the value is ignored."); + foreach (var nav in deniedNavs) + sb.AppendLine($" input.{nav} = default!;"); + } + // Issue #203 — stamp EVERY @autoSet column (onCreate AND onUpdate) with now() on // insert, overriding whatever the client sent (the model's value is ignored, so a // fresh row's updated-at equals its created-at). Emits nothing when the entity has no @@ -1084,7 +1210,8 @@ private static void AppendCreateValidation(StringBuilder sb) private static void AppendPartialMergeLoop( StringBuilder sb, string? discProp, IReadOnlyList voFields, IReadOnlyList mapVoFields, - IReadOnlyList autoSetNavs, IReadOnlyList frozenNavs, bool hasDecimal) + IReadOnlyList autoSetNavs, IReadOnlyList frozenNavs, bool hasDecimal, + IReadOnlyList deniedKeys) { // A nullable array-of-VO column that a request clears to null (present-null) can NOT be // NULLed by an owned-nav assignment — EF OwnsMany(...).ToJson writes `[]` for a null @@ -1096,6 +1223,10 @@ private static void AppendPartialMergeLoop( sb.AppendLine(" foreach (var prop in body.RootElement.EnumerateObject())"); sb.AppendLine(" {"); + // #484 — @formExclude / credential keys are IGNORED (never a 400), ahead of every arm below + // (a present null is a write too). + foreach (var key in deniedKeys) + sb.AppendLine($" if (string.Equals(prop.Name, \"{key}\", System.StringComparison.OrdinalIgnoreCase)) continue; // @formExclude / credential: not writable"); // Program D — typed value-object arms FIRST (owned-nav columns are invisible to // EF's FindProperty below, so the generic scalar path would silently drop them). // Present value → deserialize + recursive VO validation + assign the CLR nav @@ -1222,7 +1353,7 @@ private static void AppendPartialMergeLoop( // subtype CLR type (discriminator injected by EF on Add); update is a partial merge // that never touches the PK or the discriminator. private static void AppendTphSubtypeRoutes( - StringBuilder sb, TphSubtypePlan st, MetaRoot root, string baseRoute, string baseCls, string dbSet, + StringBuilder sb, TphSubtypePlan st, IReadOnlyList hierarchy, MetaRoot root, string baseRoute, string baseCls, string dbSet, string pkType, string pkProp, string discProp, Action onWarn) { // #362 — field.object columns on a TPH subtype stay out of scope (Program D §6), but a @@ -1241,6 +1372,10 @@ private static void AppendTphSubtypeRoutes( // FR-037 R1 — readOnly / writeOnce on the SUBTYPE's effective fields (a shared // base's mode is inherited by every subtype), excluded from the PATCH merge. var frozenNavs = FrozenNavs(st.Entity); + // #484 — as on a plain entity: write-denied keys ignored, credentials omitted. One shared + // column may be declared by several subtypes, so a mark on any declaration counts. + var deniedFields = HierarchyMarkedFields(st.Entity, hierarchy, IsApiWriteDenied); + var credentials = HierarchyMarkedFields(st.Entity, hierarchy, IsCredentialField).Select(f => f.Name).ToList(); // Per-subtype list: GET // sb.AppendLine(); @@ -1264,16 +1399,16 @@ private static void AppendTphSubtypeRoutes( sb.AppendLine(" if (qs.TryGetValue(\"limit\", out var limRaw) && int.TryParse(limRaw, System.Globalization.NumberStyles.Integer, System.Globalization.CultureInfo.InvariantCulture, out var lim) && lim > 0) q = q.Take(lim);"); sb.AppendLine(" var rows = await q.ToListAsync();"); sb.AppendLine(" var withCount = qs.TryGetValue(\"withCount\", out var wc) && (wc == \"1\" || string.Equals(wc, \"true\", System.StringComparison.OrdinalIgnoreCase));"); - sb.AppendLine(" if (!withCount) return Results.Ok(rows);"); + sb.AppendLine($" if (!withCount) return {OkResult("rows", credentials)};"); sb.AppendLine($" var total = await EfCoreFilterDispatch.ApplyFilter(db.{dbSet}.OfType<{subCls}>().AsNoTracking(), filter.Predicates).CountAsync();"); - sb.AppendLine(" return Results.Ok(new { rows, total });"); + sb.AppendLine($" return {OkResult("new { rows, total }", credentials, envelope: true)};"); sb.AppendLine(" });"); // Per-subtype get: GET ///{id} — scoped (cross-subtype → 404). sb.AppendLine(); sb.AppendLine(" app.MapGet(prefix + \"/" + subRoute + "/{id}\", async (" + pkType + " id, AppDbContext db) =>"); sb.AppendLine($" await db.{dbSet}.OfType<{subCls}>().AsNoTracking().FirstOrDefaultAsync(x => x.{pkProp} == id) is {{ }} found"); - sb.AppendLine(" ? Results.Ok(found)"); + sb.AppendLine(" ? " + OkResult("found", credentials)); sb.AppendLine(" : Results.NotFound(new { error = \"not_found\" }));"); // Per-subtype create: POST // — discriminator injected by EF via the @@ -1289,8 +1424,9 @@ private static void AppendTphSubtypeRoutes( sb.AppendLine(); // VO columns on a TPH subtype are out of scope (Program D §6) — no typed arms. AppendCreateHandler(sb, "/" + subRoute, subCls, dbSet, requiredKeys, - "Results.Created(prefix + \"/" + subRoute + "/\" + input." + pkProp + ", input)", [], mapVoFields, autoSetFields, - parseBody: HasDecimalField(st.Entity)); + CreatedResult("prefix + \"/" + subRoute + "/\" + input." + pkProp, "input", credentials), [], mapVoFields, autoSetFields, + parseBody: HasDecimalField(st.Entity), + deniedNavs: deniedFields.Select(f => CSharpNaming.Pascal(f.Name)).ToList(), credentials: credentials); // Per-subtype update (PATCH + PUT): partial merge of the JSON body onto the scoped // entity, NEVER touching the PK or the discriminator. Cross-subtype id → 404. @@ -1311,9 +1447,10 @@ private static void AppendTphSubtypeRoutes( AppendUpdateAutoSet(sb, autoSetFields); // ADR-0045 + the 0.19.4 lesson: TPH is a SEPARATE code path per port, so the // per-subtype merge states the same exclusion rather than inheriting it. - AppendPartialMergeLoop(sb, discProp, [], mapVoFields, autoSetNavs, frozenNavs, HasDecimalField(st.Entity)); + AppendPartialMergeLoop(sb, discProp, [], mapVoFields, autoSetNavs, frozenNavs, HasDecimalField(st.Entity), + deniedFields.Select(f => f.Name).ToList()); AppendSaveChanges(sb); - sb.AppendLine(" return Results.Ok(existing);"); + sb.AppendLine($" return {OkResult("existing", credentials)};"); sb.AppendLine(" }"); sb.AppendLine(" app.MapPatch(prefix + \"/" + subRoute + "/{id}\", Update" + subCls + ");"); sb.AppendLine(" app.MapPut(prefix + \"/" + subRoute + "/{id}\", Update" + subCls + ");"); @@ -1449,7 +1586,7 @@ private static void AppendM2mRoute( sb.AppendLine($" var rows = await {targetFrom}.AsNoTracking()"); sb.AppendLine($" .Where(t => relatedIds.Contains(EF.Property<{fkType}>(t, \"{targetPk}\")))"); sb.AppendLine(" .ToListAsync();"); - sb.AppendLine(" return Results.Ok(rows);"); + sb.AppendLine($" return {OkResult("rows", TphUnionCredentialNames(nav.Target, root))};"); sb.AppendLine(" });"); } diff --git a/server/csharp/MetaObjects.Codegen/ReportRows.cs b/server/csharp/MetaObjects.Codegen/ReportRows.cs index f2ee3b67ed..afe8752bdc 100644 --- a/server/csharp/MetaObjects.Codegen/ReportRows.cs +++ b/server/csharp/MetaObjects.Codegen/ReportRows.cs @@ -173,13 +173,23 @@ private static MetaField DerivedField(ReportField f) // `isArray` is a native flag, not an attr; ResolvedIsArray() is its resolving read. if (src.ResolvedIsArray()) field.SetIsArray(true); } + // A credential stays a credential when read through a report: a derived field whose type + // source declares or inherits a view.password carries one, so every generator and runtime + // that reads the row model (routes, export) hides it exactly as it hides an entity's. + // ADR-0039: resolving (Views()), an inherited view.password counts. + bool credential = f.TypeSource is { } cred + && cred.Views().Any(v => v.SubType == MetaObjects.Presentation.View.ViewConstants.VIEW_SUBTYPE_PASSWORD); + if (credential) + field.AddChild(new MetaView(new TypeId(TYPE_VIEW, MetaObjects.Presentation.View.ViewConstants.VIEW_SUBTYPE_PASSWORD), "password")); // Table C: every derived field whose type has a filter band is filterable. Asked // last, of the finished field, because an int-backed enum's band depends on a // carried attr. A report author cannot narrow this set: there is no node to put // `@filterable` on. // An array or a map has no scalar operator and no ordering, so a dimension over one is // neither filterable nor (see RoutesGenerator's sort list) sortable. - if (!field.ResolvedIsArray() && field.SubType != FIELD_SUBTYPE_MAP + // A credential is neither filterable nor sortable: a hidden column that still orders or + // narrows the rows is an oracle (`?filter[pin][like]=ab%` recovers it a prefix at a time). + if (!credential && !field.ResolvedIsArray() && field.SubType != FIELD_SUBTYPE_MAP && QueryConstants.OpsForField(field).Length > 0) field.SetAttr(FIELD_ATTR_FILTERABLE, true); return field; diff --git a/server/csharp/MetaObjects.Codegen/Runtime/ReportExportHttp.cs b/server/csharp/MetaObjects.Codegen/Runtime/ReportExportHttp.cs index 82bd486aa2..4c655bce8d 100644 --- a/server/csharp/MetaObjects.Codegen/Runtime/ReportExportHttp.cs +++ b/server/csharp/MetaObjects.Codegen/Runtime/ReportExportHttp.cs @@ -14,6 +14,7 @@ using System; using System.Collections.Generic; using System.IO; +using System.Linq; using System.Text; using System.Threading.Tasks; using Microsoft.AspNetCore.Http; @@ -61,7 +62,13 @@ public static IResult TooLarge(int maxRows) => /// configured JSON options (so a wire setting applies exactly as on the list route), as /// application/json, with the attachment header. /// - public static IResult Json(HttpContext http, string segment, T body) + public static IResult Json(HttpContext http, string segment, T body) => Json(http, segment, body, [], false); + + /// + /// The JSON export with credential columns (a derived field over a view.password field) OMITTED from + /// every row; says the body is { rows, total } rather than the rows. + /// + public static IResult Json(HttpContext http, string segment, T body, string[] omit, bool envelope = false) { Attach(http, segment, "json"); var options = http.RequestServices @@ -69,7 +76,18 @@ public static IResult Json(HttpContext http, string segment, T body) is Microsoft.Extensions.Options.IOptions configured ? configured.Value.SerializerOptions : null; - return Results.Json(body, options, "application/json"); + if (omit.Length == 0) return Results.Json(body, options, "application/json"); + var node = System.Text.Json.JsonSerializer.SerializeToNode(body, body!.GetType(), options); + Strip(envelope && node is System.Text.Json.Nodes.JsonObject env ? env["rows"] : node, omit); + return Results.Json(node, options, "application/json"); + } + + private static void Strip(System.Text.Json.Nodes.JsonNode? n, string[] omit) + { + if (n is System.Text.Json.Nodes.JsonArray a) { foreach (var e in a) Strip(e, omit); return; } + if (n is not System.Text.Json.Nodes.JsonObject o) return; + foreach (var key in o.Select(p => p.Key).Where(k => omit.Any(x => string.Equals(x, k, StringComparison.OrdinalIgnoreCase))).ToList()) + o.Remove(key); } /// The CSV export: fills a UTF-8 (no BOM of the encoder's own) body. diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractCorpusPaths.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractCorpusPaths.cs index 3108d9e6f5..d4ff3e9a5f 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractCorpusPaths.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractCorpusPaths.cs @@ -98,4 +98,10 @@ internal static class ApiContractCorpusPaths public static readonly string WriteThroughScenariosDir = Path.Combine(WriteThroughDir, "scenarios"); public static readonly string WriteThroughSeedFile = Path.Combine(WriteThroughDir, "seed.json"); public static readonly string WriteThroughMetaJson = Path.Combine(WriteThroughDir, "meta.json"); + + // #484 secret-fields subcorpus (credentials and @formExclude fields), generated lane only. + public static readonly string SecretFieldsDir = Path.Combine(Corpus, "secret-fields"); + public static readonly string SecretFieldsScenariosDir = Path.Combine(SecretFieldsDir, "scenarios"); + public static readonly string SecretFieldsSeedFile = Path.Combine(SecretFieldsDir, "seed.json"); + public static readonly string SecretFieldsMetaJson = Path.Combine(SecretFieldsDir, "meta.json"); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractSecretFieldsConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractSecretFieldsConformanceTest.cs new file mode 100644 index 0000000000..9314afaa29 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractSecretFieldsConformanceTest.cs @@ -0,0 +1,116 @@ +// ApiContractSecretFieldsConformanceTest — the C# #484 secret-fields api-contract lane. +// +// Drives the fixtures/api-contract-conformance/secret-fields/ scenarios over HTTP against the +// GENERATED Account routes booted unmodified on Kestrel against a Postgres testcontainer. +// Proves the deployed REST surface ignores writes to @formExclude / credential fields and +// never returns a credential. Generated lane only. + +using System.Text.Json; +using System.Text.Json.Nodes; +using MetaObjects.IntegrationTests.Runner; +using Xunit; + +namespace MetaObjects.IntegrationTests.Api; + +public sealed class ApiContractSecretFieldsConformanceTest +{ + [Theory] + [MemberData(nameof(Scenarios))] + public async Task Api_contract_secret_fields_generated(string scenarioPath) + { + var scenario = ApiContractScenarioLoader.LoadScenario(scenarioPath); + await using var pg = await PostgresContainer.StartAsync(); + await using var server = await SecretFieldsGeneratedServerFactory.StartAsync(pg); + await server.ApplySeedAsync(); + await RunAsync(scenario, server.BaseUrl); + await AssertStoredSecretsUntouchedAsync(pg.ConnectionString); + } + + // Behavioural proof of write-denial, read straight from the database (not through the API): + // every seeded row keeps its seeded tokenHash and pinHash, and a row a scenario created holds + // null in both, so neither a forged value nor a present null ever reached a denied column. + private static async Task AssertStoredSecretsUntouchedAsync(string connString) + { + var seed = JsonNode.Parse(File.ReadAllText(ApiContractCorpusPaths.SecretFieldsSeedFile))!["rows"]!.AsArray() + .ToDictionary(r => r!["id"]!.GetValue(), r => r!.AsObject()); + + await using var c = new Npgsql.NpgsqlConnection(connString); + await c.OpenAsync(); + await using var cmd = c.CreateCommand(); + cmd.CommandText = "SELECT \"id\", \"tokenHash\", \"pinHash\" FROM \"accounts\" ORDER BY \"id\""; + await using var reader = await cmd.ExecuteReaderAsync(); + var seen = new HashSet(); + while (await reader.ReadAsync()) + { + long id = reader.GetInt64(0); + string? token = reader.IsDBNull(1) ? null : reader.GetString(1); + string? pin = reader.IsDBNull(2) ? null : reader.GetString(2); + seen.Add(id); + if (seed.TryGetValue(id, out var row)) + { + Assert.Equal(row["tokenHash"]!.GetValue(), token); + Assert.Equal(row["pinHash"]!.GetValue(), pin); + } + else + { + Assert.Null(token); + Assert.Null(pin); + } + } + Assert.True(seed.Keys.All(seen.Contains), "a seeded row went missing"); + } + + private static async Task RunAsync(ApiScenario scenario, string baseUrl) + { + using var client = new HttpClient { BaseAddress = new Uri(baseUrl) }; + foreach (var req in scenario.Requests) + { + var request = new HttpRequestMessage(new HttpMethod(req.Method), ApiContractWire.VerbatimUri(client, req.Path)); + if (req.Body is not null) + { + string json = JsonSerializer.Serialize(req.Body, JsonOpts); + request.Content = new StringContent(json, System.Text.Encoding.UTF8, "application/json"); + } + var response = await client.SendAsync(request); + string bodyText = await response.Content.ReadAsStringAsync(); + object? parsed = string.IsNullOrEmpty(bodyText) ? null : ToObject(JsonNode.Parse(bodyText)); + ApiContractAssertions.AssertResponse(scenario.Name, req, (int)response.StatusCode, parsed); + } + } + + public static IEnumerable Scenarios() => + Directory.EnumerateFiles(ApiContractCorpusPaths.SecretFieldsScenariosDir, "*.yaml", SearchOption.TopDirectoryOnly) + .OrderBy(p => p, StringComparer.Ordinal) + .Select(p => new object[] { p }); + + private static readonly JsonSerializerOptions JsonOpts = new() + { + DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.Never, + }; + + private static object? ToObject(JsonNode? node) + { + if (node is null) return null; + if (node is JsonObject obj) + { + var d = new Dictionary(StringComparer.Ordinal); + foreach (var kvp in obj) d[kvp.Key] = ToObject(kvp.Value); + return d; + } + if (node is JsonArray arr) + { + var l = new List(arr.Count); + foreach (var item in arr) l.Add(ToObject(item)); + return l; + } + if (node is JsonValue jv) + { + if (jv.TryGetValue(out var b)) return b; + if (jv.TryGetValue(out var lv)) return lv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) + if (jv.TryGetValue(out var sv)) return sv; + return jv.ToString(); + } + return null; + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/SecretFieldsFixture.cs b/server/csharp/MetaObjects.IntegrationTests/Api/SecretFieldsFixture.cs new file mode 100644 index 0000000000..2aaf7d81ca --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Api/SecretFieldsFixture.cs @@ -0,0 +1,53 @@ +// SecretFieldsFixture — schema + seed for the #484 secret-fields api-contract GENERATED lane. +using System.Text.Json.Nodes; +using Npgsql; + +namespace MetaObjects.IntegrationTests.Api; + +internal static class SecretFieldsFixture +{ + public const string SchemaDdl = """ + CREATE TABLE "accounts" ( + "id" BIGINT GENERATED BY DEFAULT AS IDENTITY NOT NULL, + "name" VARCHAR(100) NOT NULL, + "tokenHash" VARCHAR(100) NULL, + "pinHash" VARCHAR(100) NULL, + CONSTRAINT "accounts_pkey" PRIMARY KEY ("id") + ); + """; + + public static async Task ProvisionSchemaAsync(string connString) + { + await using var c = new NpgsqlConnection(connString); + await c.OpenAsync(); + await using var cmd = c.CreateCommand(); + cmd.CommandText = SchemaDdl; + await cmd.ExecuteNonQueryAsync(); + } + + public static async Task ApplySeedAsync(string connString, string seedPath) + { + var root = JsonNode.Parse(File.ReadAllText(seedPath)) as JsonObject + ?? throw new InvalidOperationException($"{seedPath}: top-level must be an object"); + var cols = new[] { "id", "name", "tokenHash", "pinHash" }; + await using var c = new NpgsqlConnection(connString); + await c.OpenAsync(); + foreach (var rowNode in root["rows"]!.AsArray()) + { + var row = rowNode!.AsObject(); + await using var ins = c.CreateCommand(); + ins.CommandText = "INSERT INTO \"accounts\" (" + string.Join(", ", cols.Select(x => "\"" + x + "\"")) + + ") VALUES (" + string.Join(", ", cols.Select((_, i) => "@p" + i)) + ")"; + for (int i = 0; i < cols.Length; i++) + { + var v = row[cols[i]]; + object val = v is null ? DBNull.Value : cols[i] == "id" ? v.GetValue() : v.GetValue(); + ins.Parameters.AddWithValue("@p" + i, val); + } + await ins.ExecuteNonQueryAsync(); + } + await using var bump = c.CreateCommand(); + bump.CommandText = "SELECT setval(pg_get_serial_sequence('accounts', 'id'), COALESCE((SELECT MAX(id) FROM \"accounts\"), 1))"; + await bump.ExecuteScalarAsync(); + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/SecretFieldsGeneratedServerFactory.cs b/server/csharp/MetaObjects.IntegrationTests/Api/SecretFieldsGeneratedServerFactory.cs new file mode 100644 index 0000000000..c2ec132342 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Api/SecretFieldsGeneratedServerFactory.cs @@ -0,0 +1,181 @@ +// SecretFieldsGeneratedServerFactory — the C# GENERATED-server lane for the #484 secret-fields +// corpus. Runs the real Codegen generators (Entity + DbContext + FilterAllowlist + Routes) on the +// Account model, Roslyn-compiles the output in memory and hosts it on Kestrel against Postgres. + +using MetaObjects.Meta; +using System.Reflection; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using MetaObjects.Codegen; +using MetaObjects.Codegen.Generators; +using MetaObjects.IntegrationTests.Runner; +using MetaObjects.Loader; + +namespace MetaObjects.IntegrationTests.Api; + +internal sealed class SecretFieldsGeneratedServerFactory : IAsyncDisposable +{ + private const string GeneratedNamespace = "MetaObjects.ApiContract.SecretFieldsGenerated"; + + private readonly PostgresContainer _pg; + private readonly WebApplication _app; + + public string BaseUrl { get; } + + private SecretFieldsGeneratedServerFactory(PostgresContainer pg, WebApplication app, string baseUrl) + { + _pg = pg; + _app = app; + BaseUrl = baseUrl; + } + + public static async Task StartAsync(PostgresContainer pg) + { + await SecretFieldsFixture.ProvisionSchemaAsync(pg.ConnectionString); + + var (assembly, entityNames) = CompileGeneratedServer(); + var dbContextType = assembly.GetType($"{GeneratedNamespace}.AppDbContext") + ?? throw new InvalidOperationException("generated AppDbContext type not found"); + + var builder = WebApplication.CreateBuilder(); + LoopbackKestrel.BindAnyPort(builder); + builder.Logging.ClearProviders(); + RegisterGeneratedDbContext(builder.Services, dbContextType, pg.ConnectionString); + + var app = builder.Build(); + + // Mount each generated MapRoutes(app, "/api"). Both Customer and Order + // emit a routes class; only Order's secret-fields routes are exercised. + foreach (var entity in entityNames) + { + var routesType = assembly.GetType($"{GeneratedNamespace}.{entity}Routes"); + if (routesType is null) continue; + var mapMethod = routesType.GetMethod($"Map{entity}Routes", BindingFlags.Public | BindingFlags.Static); + mapMethod?.Invoke(null, new object[] { app, "/api" }); + } + + await app.StartAsync(); + return new SecretFieldsGeneratedServerFactory(pg, app, LoopbackKestrel.BoundUrl(app)); + } + + public async Task ApplySeedAsync() => + await SecretFieldsFixture.ApplySeedAsync(_pg.ConnectionString, ApiContractCorpusPaths.SecretFieldsSeedFile); + + public async ValueTask DisposeAsync() + { + try { await _app.StopAsync(); } catch { /* ignored */ } + try { await _app.DisposeAsync(); } catch { /* ignored */ } + } + + private static (Assembly Assembly, IReadOnlyList EntityNames) CompileGeneratedServer() + { + var loadResult = new MetaDataLoader().Load([new FileSource(ApiContractCorpusPaths.SecretFieldsMetaJson)]); + // ADR-0058 — the corpus's committed naming rule, not a hardcoded default. + loadResult.Root.PhysicalNaming = ApiContractCorpusPaths.Naming; + loadResult.Root.WireFormat = ApiContractCorpusPaths.Wire; + if (loadResult.Errors.Count != 0) + throw new InvalidOperationException( + "secret-fields corpus metadata failed to load: " + + string.Join("; ", loadResult.Errors.Select(e => e.ToString()))); + + var entityNames = loadResult.Root.Objects() + .Where(o => o.IsEntity()) + .Select(o => CSharpNaming.Pascal(o.Name)) + .ToList(); + + var ctx = new GenContext + { + Entities = loadResult.Root.Objects(), + Root = loadResult.Root, + Config = new GenConfig + { + OutDir = "/unused", + Namespace = GeneratedNamespace, + ColumnNamingStrategy = ApiContractCorpusPaths.Naming.Columns, + EmitAbstractShapes = false, + // GenConfig.IncludeNames defaults to false (§A6 task 4); without it + // here the entity/DbContext output below never references the + // Names constants, and the .Concat(new NamesGenerator()...) + // a few lines down is dead input the compile never depends on. This + // is the only lane that proves an Names.NAME reference + // actually RESOLVES against a real EF Core compile. + IncludeNames = true, + }, + }; + + // §A6 (task 4) -- the entity/DbContext output now references the names artifact. + var files = new EntityGenerator().Generate(ctx) + .Concat(new DbContextGenerator().Generate(ctx)) + .Concat(new FilterAllowlistGenerator().Generate(ctx)) + .Concat(new RoutesGenerator().Generate(ctx)) + .Concat(new NamesGenerator().Generate(ctx)) + .ToList(); + + var trees = files + .Select(f => CSharpSyntaxTree.ParseText(f.Content, new CSharpParseOptions(LanguageVersion.CSharp12))) + .ToArray(); + + var refs = BuildReferenceSet(); + var comp = CSharpCompilation.Create( + "apicontract_secretfields_generated_" + Guid.NewGuid().ToString("N"), + trees, refs, + new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary)); + + using var ms = new MemoryStream(); + var emit = comp.Emit(ms); + if (!emit.Success) + { + var errors = emit.Diagnostics + .Where(d => d.Severity == DiagnosticSeverity.Error) + .Select(d => $"{d.Id}: {d.GetMessage()}") + .ToList(); + throw new InvalidOperationException( + "generated secret-fields server failed to compile:\n " + string.Join("\n ", errors)); + } + + ms.Seek(0, SeekOrigin.Begin); + return (Assembly.Load(ms.ToArray()), entityNames); + } + + private static List BuildReferenceSet() + { + var byFileName = new Dictionary(StringComparer.OrdinalIgnoreCase); + var tpa = (string?)AppContext.GetData("TRUSTED_PLATFORM_ASSEMBLIES") ?? ""; + foreach (var path in tpa.Split(Path.PathSeparator)) + if (path.Length > 0 && path.EndsWith(".dll", StringComparison.OrdinalIgnoreCase)) + byFileName[Path.GetFileName(path)] = path; + + var aspNetDir = Path.GetDirectoryName(typeof(WebApplication).Assembly.Location); + if (aspNetDir is not null && Directory.Exists(aspNetDir)) + foreach (var dll in Directory.EnumerateFiles(aspNetDir, "*.dll")) + byFileName[Path.GetFileName(dll)] = dll; + + return byFileName.Values + .Select(loc => (MetadataReference)MetadataReference.CreateFromFile(loc)) + .ToList(); + } + + private static void RegisterGeneratedDbContext( + IServiceCollection services, Type dbContextType, string connString) + { + var addDbContext = typeof(EntityFrameworkServiceCollectionExtensions) + .GetMethods(BindingFlags.Public | BindingFlags.Static) + .First(m => m.Name == "AddDbContext" + && m.IsGenericMethodDefinition + && m.GetGenericArguments().Length == 1 + && m.GetParameters().Length == 4) + .MakeGenericMethod(dbContextType); + + Action configure = opts => opts.UseNpgsql(connString); + addDbContext.Invoke(null, new object?[] + { + services, configure, ServiceLifetime.Scoped, ServiceLifetime.Scoped, + }); + } +} diff --git a/server/csharp/MetaObjects/Core/Field/FieldConstants.cs b/server/csharp/MetaObjects/Core/Field/FieldConstants.cs index f1d65897dd..5daaedb59a 100644 --- a/server/csharp/MetaObjects/Core/Field/FieldConstants.cs +++ b/server/csharp/MetaObjects/Core/Field/FieldConstants.cs @@ -121,6 +121,12 @@ public static class FieldConstants /// public const string FIELD_ATTR_MUTABILITY = "mutability"; + /// + /// @formExclude — the field is out of the generated form; the generated HTTP write + /// routes ignore it too (#484). + /// + public const string FIELD_ATTR_FORM_EXCLUDE = "formExclude"; + /// readWrite — settable on create, changeable on update (the default). public const string MUTABILITY_READ_WRITE = "readWrite"; diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinFilterAllowlistGenerator.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinFilterAllowlistGenerator.kt index 98d4d6bfb6..baddea1d3e 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinFilterAllowlistGenerator.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinFilterAllowlistGenerator.kt @@ -175,8 +175,14 @@ open class KotlinFilterAllowlistGenerator : MultiFileDirectGeneratorBase> = - computeFilterableOps(entity.metaFields) + fun computeFilterableOps(entity: MetaObject): Map> { + val ops = computeFilterableOps(entity.metaFields) + if (!com.metaobjects.generator.util.GeneratorUtil.isReport(entity)) return ops + // #484: a report's credential column is hidden from the response, so it must not be filterable + // either: a filter would let a caller recover the hidden value a prefix at a time. + val credentials = KotlinGenUtil.reportCredentialNames(entity).toSet() + return ops.filterKeys { it !in credentials } + } /** * [computeFilterableOps] over an explicit field list — used for a TPH base, whose allowlist diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinGenUtil.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinGenUtil.kt index 32a905c2a0..62b74b74fc 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinGenUtil.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinGenUtil.kt @@ -759,9 +759,36 @@ public object KotlinGenUtil { fun isNullableShapeProperty(field: MetaField<*>): Boolean = !isRequiredField(field) && !originGuaranteedNonNull(field) && !isAssignedPrimaryKeyField(field) - fun isRequiredField(field: MetaField<*>): Boolean { - if (!field.hasMetaAttr(MetaField.ATTR_REQUIRED, true)) return false - val raw = runCatching { field.getMetaAttr(MetaField.ATTR_REQUIRED, true).value }.getOrNull() + /** + * #484 — a credential: the field declares (or inherits, ADR-0039: resolving read) a child + * `view.password`. A credential is never returned by a generated read route and never + * written by a generated write route. + */ + fun isCredentialField(field: MetaField<*>): Boolean = + field.getChildren(com.metaobjects.view.MetaView::class.java, true) + .any { it.subType == com.metaobjects.view.PresentationView.SUBTYPE_PASSWORD } + + /** + * #484 — the field names of a served report that are credentials. A report column is DERIVED from its + * `@of` field (a dimension attribute, or a min/max measure over it), and the derived field carries + * only type-shaping attrs, so it loses the source's `view.password`; the credential is read off the + * type source instead ([com.metaobjects.reporting.ReportShape.Field.typeSource], resolving accessors). + * Takes the declared `object.report` or its read model. + */ + fun reportCredentialNames(report: MetaObject): List { + val declared = (report as? com.metaobjects.reporting.ReportReadModel)?.report() ?: report + return com.metaobjects.reporting.ReportShape.of(declared).fields() + .filter { it.typeSource() != null && isCredentialField(it.typeSource()) } + .map { it.name() } + } + + /** #484 — `@formExclude: true` (resolving read): the field is not part of the write shape. */ + fun isFormExcluded(field: MetaField<*>): Boolean = resolvedBooleanAttr(field, ATTR_FORM_EXCLUDE) + + /** Resolving boolean read: a Boolean value, or the string "true" in any case. */ + private fun resolvedBooleanAttr(field: MetaField<*>, name: String): Boolean { + if (!field.hasMetaAttr(name, true)) return false + val raw = runCatching { field.getMetaAttr(name, true).value }.getOrNull() return when (raw) { is Boolean -> raw is String -> raw.equals("true", ignoreCase = true) @@ -769,6 +796,16 @@ public object KotlinGenUtil { } } + /** + * #484 — true when a generated create / PATCH / PUT route must IGNORE the field (strip it, never + * reject it): `@formExclude` or a credential. The application's own repository still writes it. + */ + fun isApiWriteDenied(field: MetaField<*>): Boolean = isFormExcluded(field) || isCredentialField(field) + + private const val ATTR_FORM_EXCLUDE = "formExclude" + + fun isRequiredField(field: MetaField<*>): Boolean = resolvedBooleanAttr(field, MetaField.ATTR_REQUIRED) + /** * The `@autoSet` policy of [field] — [AUTO_SET_ON_CREATE] / [AUTO_SET_ON_UPDATE] — or null * when the field is not auto-set (issue #203). Read RESOLVING (ADR-0039, `includeParentData diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinM2mSupport.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinM2mSupport.kt index 5f15e74ba0..d45460a02f 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinM2mSupport.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinM2mSupport.kt @@ -86,6 +86,8 @@ object KotlinM2mSupport { val targetDiscriminator: TargetDiscriminator?, val symmetric: Boolean, val junctionPackage: String = "", + /** #484: target fields that are credentials, never returned (their keys are omitted from the serialized row). */ + val targetCredentialFields: Set = emptySet(), ) /** @@ -199,6 +201,8 @@ object KotlinM2mSupport { targetDiscriminator = targetDiscriminator, symmetric = rel.isSymmetric, junctionPackage = junctionPkg, + targetCredentialFields = (target.metaFields + KotlinTphPlan.everySubtypeField(target, loader)) + .filter { KotlinGenUtil.isCredentialField(it) }.map { it.name }.toSet(), ) ) } diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinReportExportGenerator.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinReportExportGenerator.kt index 0686e582ab..88361a7b38 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinReportExportGenerator.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinReportExportGenerator.kt @@ -125,7 +125,9 @@ open class KotlinReportExportGenerator : MultiFileDirectGeneratorBase, arg: String): String = + if (creds.isEmpty()) "rowTo$shortName($arg)" else "redact$shortName(objectMapper, rowTo$shortName($arg))" + + /** + * #484: a credential is never returned. The shared data class keeps its declared types, so the key is + * dropped from the serialized tree at the HTTP boundary (absent, not null), through the same mapper + * (and so the same wire annotations) the response would use. + */ + private fun appendRedactor(sb: StringBuilder, shortName: String, creds: Collection) { + if (creds.isEmpty()) return + sb.append("/** GENERATED — the ${shortName} response value with its credential field(s) removed. */\n") + sb.append("private fun redact${shortName}(mapper: ObjectMapper, row: ${shortName}): Any =\n") + sb.append(" mapper.valueToTree(row).apply { remove(listOf(${creds.joinToString(", ") { "\"$it\"" }})) }\n\n") + } + + private fun credentialNames(fields: Collection>): List = + fields.filter { KotlinGenUtil.isCredentialField(it) }.map { it.name }.distinct() + protected open fun emit(entity: MetaObject, outRoot: Path, loader: MetaDataLoader) { val api = exposedApi() val (pkg, shortName) = PackageMapping.splitFqn(entity.name) @@ -274,9 +293,15 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase navs.any { it.targetCredentialFields.isNotEmpty() } } val src = buildString { if (pkg.isNotEmpty()) append("package $pkg\n\n") @@ -828,6 +864,8 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase = transaction {\n") append(" val row = $table.selectAll().where { $table.$pkFieldName eq id }.singleOrNull()\n") append(" ?: return@transaction ResponseEntity.status(HttpStatus.NOT_FOUND).body(mapOf(\"error\" to \"not_found\") as Any)\n") - append(" ResponseEntity.ok(rowTo${shortName}(row) as Any)\n") + append(" ResponseEntity.ok(${wireRow(shortName, creds, "row")} as Any)\n") append(" }\n\n") // Base-declared M:N — unscoped, same shape emitM2mEndpoint gives a vanilla entity. @@ -1024,9 +1065,9 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase = transaction {\n") append(" val row = $table.selectAll().where { ($table.$pkFieldName eq id) and ($table.${plan.discriminatorField} eq $disc) }.singleOrNull()\n") append(" ?: return@transaction ResponseEntity.status(HttpStatus.NOT_FOUND).body(mapOf(\"error\" to \"not_found\") as Any)\n") - append(" ResponseEntity.ok(rowTo${shortName}(row) as Any)\n") + append(" ResponseEntity.ok(${wireRow(shortName, creds, "row")} as Any)\n") append(" }\n\n") // per-subtype create (discriminator injected from URL). FR-036: validate each body @@ -1076,7 +1117,7 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase> = transaction {\n") - out.append(" val rows = $sourceTable.$queryFnName(id).map { row ->\n") + out.append(" fun ${nav.relationName}(@PathVariable id: $pkParamType): ResponseEntity> = transaction {\n") + appendM2mRows(out, nav, targetType, "$sourceTable.$queryFnName(id)") + out.append(" }\n") + } + + /** #484: the element type an M:N traversal answers — `Any` (a redacted tree) when the target has a credential. */ + private fun m2mReturnElement(nav: KotlinM2mSupport.M2mNav, targetType: String): String = + if (nav.targetCredentialFields.isEmpty()) targetType else "Any" + + /** + * The shared body of an M:N traversal: map each joined row to the target data class and, when the target + * declares credentials (#484), drop those keys from the serialized value (the source controller holds an + * `objectMapper` whenever one of its targets does). + */ + private fun appendM2mRows(out: StringBuilder, nav: KotlinM2mSupport.M2mNav, targetType: String, query: String) { + val creds = nav.targetCredentialFields + out.append(" val rows${if (creds.isNotEmpty()) ": List" else ""} = $query.map { row ->\n") + if (creds.isNotEmpty()) out.append(" objectMapper.valueToTree(\n") out.append(" $targetType(\n") for (fname in nav.targetScalarFields) { out.append(" $fname = row[${nav.targetTableObj}.$fname],\n") } out.append(" )\n") + if (creds.isNotEmpty()) { + out.append(" ).apply { remove(listOf(${creds.joinToString(", ") { "\"$it\"" }})) }\n") + } out.append(" }\n") out.append(" ResponseEntity.ok(rows)\n") - out.append(" }\n") } /** @@ -1876,18 +1935,12 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase> = transaction {\n") + val elem = m2mReturnElement(nav, targetType) + out.append(" fun ${nav.relationName}$sfx(@PathVariable id: $pkParamType): ResponseEntity> = transaction {\n") out.append(" if ($table.selectAll().where { ($table.$pkFieldName eq id) and ($table.$discField eq $discExpr) }.singleOrNull() == null) {\n") - out.append(" return@transaction ResponseEntity.ok(emptyList<$targetType>())\n") - out.append(" }\n") - out.append(" val rows = $table.$queryFnName(id).map { row ->\n") - out.append(" $targetType(\n") - for (fname in nav.targetScalarFields) { - out.append(" $fname = row[${nav.targetTableObj}.$fname],\n") - } - out.append(" )\n") + out.append(" return@transaction ResponseEntity.ok(emptyList<$elem>())\n") out.append(" }\n") - out.append(" ResponseEntity.ok(rows)\n") + appendM2mRows(out, nav, targetType, "$table.$queryFnName(id)") out.append(" }\n") } @@ -2027,9 +2080,11 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase