Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ Remotes are managed via utilities in `modules/db/utils/remotes.ts`:
| `postkit db plan` | Generate schema diff with pgschema |
| `postkit db apply` | Apply migration to local DB (creates dbmate migration) |
| `postkit db commit` | Commit session migrations for deployment |
| `postkit db deploy [--remote <name>]` | Deploy committed migrations (with dry-run verification) |
| `postkit db deploy [--remote <name>] [--with-data]` | Deploy committed migrations (with schema-only dry-run verification; `--with-data` clones target rows too) |
| `postkit db status` | Show session state |
| `postkit db abort` | Cancel session, cleanup local resources |
| `postkit db migration [<name>]` | Create a manual SQL migration |
Expand Down
9 changes: 6 additions & 3 deletions cli/docs/db.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ postkit db start --remote staging # Use specific remote

**Why infra is applied before cloning:** the remote's dump includes `CREATE POLICY ... TO <role>`, `GRANT ... TO <role>`, and `ALTER DEFAULT PRIVILEGES ... TO <role>` statements for every custom role referenced by your RLS policies and grants. Those statements fail with `role "<role>" does not exist` if replayed before the role exists — which is exactly what happens on a brand-new local DB/container. Applying `db/infra/` first (same roles/schemas your project already declares) means those statements succeed, so RLS policies and grants clone correctly. The clone's `psql` runs with `-v ON_ERROR_STOP=1`, so if a referenced role genuinely doesn't exist anywhere in `db/infra/`, the clone now fails loudly with a clear error instead of silently dropping that policy/grant.

**Why the clone is schema-only:** copying full production row data (customer records, emails, tokens, etc.) into every disposable local dev container on every `db start` is unnecessary and a privacy/compliance risk. `db start` only needs to reproduce *structure* — tables, RLS policies, grants, indexes, triggers, functions — so it doesn't copy any rows. For synthetic local test data, use `db/schema/<name>/seeds/`, applied separately during `db plan`/`db apply`. (`postkit db deploy`'s own dry-run clone is unaffected by this — it still clones full data, since it's specifically verifying real migrations against realistic data before they touch production.)
**Why the clone is schema-only:** copying full production row data (customer records, emails, tokens, etc.) into every disposable local dev container on every `db start` is unnecessary and a privacy/compliance risk. `db start` only needs to reproduce *structure* — tables, RLS policies, grants, indexes, triggers, functions — so it doesn't copy any rows. For synthetic local test data, use `db/schema/<name>/seeds/`, applied separately during `db plan`/`db apply`. (`postkit db deploy`'s dry-run clone is schema-only for the same reasons; pass `--with-data` if a migration's risk lies in the data itself.)

---

Expand Down Expand Up @@ -320,7 +320,7 @@ postkit db commit -f # Skip confirmation

---

### `postkit db deploy [--remote <name>] [--url <url>]`
### `postkit db deploy [--remote <name>] [--url <url>] [--with-data]`

Deploy committed migrations to a remote database. Performs a full dry-run verification on a local clone before touching the target.

Expand All @@ -330,6 +330,7 @@ postkit db deploy --remote staging # Use specific remote
postkit db deploy --url=postgres://... # Direct URL override
postkit db deploy --remote production -f # Skip confirmations
postkit db deploy --dry-run # Verify only, don't touch target
postkit db deploy --with-data # Clone target row data into the dry-run clone
```

**What it does:**
Expand All @@ -339,7 +340,7 @@ postkit db deploy --dry-run # Verify only, don't touch target
4. Tests the target database connection and detects its PostgreSQL major version
5. **If `localDbUrl` is empty**: Starts a temporary `postgres:{version}-alpine` container (version-matched to the target) for the dry-run
6. **Applies `db/infra/` (roles, schemas, extensions) to the local/temp database** — before cloning, for the same reason `db start` does (RLS policies and grants in the target's dump reference roles that must already exist)
7. Clones the target database (full data — this dry-run intentionally tests against realistic data) to the local URL. When using a temp container, cloning runs via `docker exec` inside the container
7. Clones the target database's **structure only** (`pg_dump --schema-only`, no row data) to the local URL. When using a temp container, cloning runs via `docker exec` inside the container. Pass `--with-data` to clone row data as well
8. Runs a full dry-run on the local clone: infra (reapplied, idempotently), dbmate migrate, seeds
9. If `--dry-run` is set, stops here and reports results without touching the target
10. Reports dry-run results and confirms deployment (unless `-f`)
Expand All @@ -348,6 +349,8 @@ postkit db deploy --dry-run # Verify only, don't touch target

If the dry run fails, deployment is aborted and no changes are made to the target database.

**`--with-data`:** the dry-run verifies that committed DDL applies cleanly, which only needs the target's structure. Cloning row data as well means every deploy pulls the entire target dataset across the network into a clone that is dropped minutes later — slow, costly in egress, and it puts production rows on developer machines. Reach for `--with-data` when the migration's risk lives in the data rather than the schema: adding `NOT NULL` or a `UNIQUE` constraint to a populated column, a type narrowing that existing rows might not satisfy, or a data backfill. Note that seeds run as part of the dry-run, so against a clone that already holds the target's rows seed inserts may conflict on existing keys.

---

### `postkit db remote`
Expand Down
41 changes: 36 additions & 5 deletions cli/src/modules/db/commands/deploy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import {PostkitError} from "../../../common/errors";
interface DeployOptions extends CommandOptions {
remote?: string;
url?: string;
withData?: boolean;
}

function resolveTargetUrl(options: DeployOptions): {url: string; label: string} {
Expand Down Expand Up @@ -224,15 +225,45 @@ export async function deployCommand(options: DeployOptions): Promise<void> {
logger.step(3, totalSteps, "Applying infrastructure to local database...");
await applyInfraStep(spinner, localDbUrl, "local clone");

logger.step(4, totalSteps, "Cloning target database to local...");
spinner.start("Cloning target database to local for dry-run verification...");
// The dry-run verifies that committed DDL applies cleanly, which only needs
// the target's *structure*. Cloning row data too means every deploy drags the
// entire production dataset across the network into a container we throw away
// minutes later. --with-data opts back in for migrations whose risk is in the
// data itself (adding NOT NULL or a UNIQUE constraint to a populated column).
const schemaOnly = !options.withData;
logger.step(
4,
totalSteps,
schemaOnly
? "Cloning target database structure to local..."
: "Cloning target database (with data) to local...",
);
spinner.start(
schemaOnly
? "Cloning database structure for dry-run verification..."
: "Cloning database with row data for dry-run verification (this may take a while)...",
);
if (tempContainerID) {
await cloneDatabaseViaContainer(tempContainerID, targetUrl, localDbUrl);
await cloneDatabaseViaContainer(tempContainerID, targetUrl, localDbUrl, schemaOnly);
} else {
await cloneDatabase(targetUrl, localDbUrl);
await cloneDatabase(targetUrl, localDbUrl, schemaOnly);
}
const localTableCount = await getTableCount(localDbUrl);
spinner.succeed(`Target cloned to local (${localTableCount} tables)`);
spinner.succeed(
schemaOnly
? `Target structure cloned to local (${localTableCount} tables, no row data)`
: `Target cloned to local (${localTableCount} tables, with row data)`,
);

if (options.withData) {
// Seeds are written to populate an empty database. Run against a clone that
// already holds the target's rows, they can collide on primary/unique keys
// and fail the dry-run for reasons unrelated to the migration being deployed.
logger.warn(
"--with-data: seeds run against a clone that already contains the target's rows; " +
"seed inserts may conflict on existing keys.",
);
}

// Steps 5-7: Dry run on local clone
logger.blank();
Expand Down
1 change: 1 addition & 0 deletions cli/src/modules/db/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ export function registerDbModule(program: Command): void {
.description("Deploy committed migrations (defaults to remote DB)")
.option("--remote <name>", "Target remote name")
.option("--url <url>", "Direct database URL to deploy to")
.option("--with-data", "Clone the target's row data into the dry-run clone (slow; off by default)")
.option("-f, --force", "Skip confirmation prompts")
.action(async (cmdOptions) => {
await withInitCheck(async () => {
Expand Down
7 changes: 6 additions & 1 deletion cli/src/modules/db/services/container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -119,11 +119,16 @@ function toDockerHost(host: string): string {
return host;
}

/**
* `schemaOnly` is required here for the same reason as in `cloneDatabase` — the
* cost gap between structure and full data is large enough that no caller
* should get one by accident.
*/
export async function cloneDatabaseViaContainer(
containerID: string,
sourceUrl: string,
targetUrl: string,
schemaOnly = false,
schemaOnly: boolean,
): Promise<void> {
const src = parseConnectionUrl(sourceUrl);
const dst = parseConnectionUrl(targetUrl);
Expand Down
8 changes: 7 additions & 1 deletion cli/src/modules/db/services/database.ts
Original file line number Diff line number Diff line change
Expand Up @@ -113,10 +113,16 @@ export function sanitizeCloneLine(line: string): string {
return makeSchemaCreationIdempotent(neutralizeUnsupportedPreambleSettings(line));
}

/**
* `schemaOnly` is intentionally required, with no default. Cloning row data is
* enormously more expensive than cloning structure — a silent default here is
* how `db deploy` ended up pulling entire production datasets across the
* network for a DDL dry-run. Every caller must state which one it wants.
*/
export async function cloneDatabase(
sourceUrl: string,
targetUrl: string,
schemaOnly = false,
schemaOnly: boolean,
): Promise<void> {
const src = parseConnectionUrl(sourceUrl);
const dst = parseConnectionUrl(targetUrl);
Expand Down
8 changes: 8 additions & 0 deletions cli/test/e2e/smoke/basic-commands.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@ describe("Smoke tests — basic CLI commands (no Docker)", () => {
expect(result.stdout).toContain("abort");
});

it("db deploy help documents --with-data as opt-in", async () => {
const result = await runCli(["db", "deploy", "--help"]);
expect(result.exitCode).toBe(0);
expect(result.stdout).toContain("--with-data");
// Row data is opt-in: the dry-run clone is schema-only unless asked otherwise.
expect(result.stdout).toContain("off by default");
});

it("db status fails without config file", async () => {
const tmpDir = await createEmptyDir();
try {
Expand Down
12 changes: 6 additions & 6 deletions cli/test/modules/db/services/container.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ describe("container", () => {
it("runs pg_dump and psql inside the container via docker exec", async () => {
vi.mocked(runPipedCommands).mockResolvedValue({stdout: "", stderr: "", exitCode: 0});

await cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl);
await cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl, false);

expect(runPipedCommands).toHaveBeenCalledTimes(1);
const [producer, consumer] = vi.mocked(runPipedCommands).mock.calls[0]!;
Expand All @@ -210,7 +210,7 @@ describe("container", () => {
it("psql connects to container-internal localhost:5432, not the mapped port", async () => {
vi.mocked(runPipedCommands).mockResolvedValue({stdout: "", stderr: "", exitCode: 0});

await cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl);
await cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl, false);

const [, consumer] = vi.mocked(runPipedCommands).mock.calls[0]!;
expect(consumer.args).toContain("localhost");
Expand All @@ -219,9 +219,9 @@ describe("container", () => {
expect(consumer.args).not.toContain("15432");
});

it("does not pass --schema-only by default", async () => {
it("omits --schema-only when schemaOnly is false", async () => {
vi.mocked(runPipedCommands).mockResolvedValue({stdout: "", stderr: "", exitCode: 0});
await cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl);
await cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl, false);
const [producer] = vi.mocked(runPipedCommands).mock.calls[0]!;
expect(producer.args).not.toContain("--schema-only");
});
Expand All @@ -236,7 +236,7 @@ describe("container", () => {
it("passes PGPASSWORD for source via -e flag in docker exec args", async () => {
vi.mocked(runPipedCommands).mockResolvedValue({stdout: "", stderr: "", exitCode: 0});

await cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl);
await cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl, false);

const [producer] = vi.mocked(runPipedCommands).mock.calls[0]!;
const envFlag = producer.args.findIndex((a) => a === "-e");
Expand All @@ -252,7 +252,7 @@ describe("container", () => {
});

await expect(
cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl),
cloneDatabaseViaContainer(containerID, sourceUrl, targetUrl, false),
).rejects.toThrow("Failed to clone database via container");
});
});
Expand Down
12 changes: 6 additions & 6 deletions cli/test/modules/db/services/database.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ describe("database", () => {
describe("cloneDatabase()", () => {
it("calls runPipedCommands with pg_dump and psql", async () => {
vi.mocked(runPipedCommands).mockResolvedValue({stdout: "", stderr: "", exitCode: 0});
await cloneDatabase("postgres://src:pass@src-host:5432/srcdb", "postgres://dst:pass@dst-host:5432/dstdb");
await cloneDatabase("postgres://src:pass@src-host:5432/srcdb", "postgres://dst:pass@dst-host:5432/dstdb", false);
expect(runPipedCommands).toHaveBeenCalledTimes(1);
const [producer, consumer] = vi.mocked(runPipedCommands).mock.calls[0]!;
expect(producer.args[0]).toBe("pg_dump");
Expand All @@ -144,28 +144,28 @@ describe("database", () => {
it("throws on failure", async () => {
vi.mocked(runPipedCommands).mockResolvedValue({stdout: "", stderr: "error", exitCode: 1});
await expect(
cloneDatabase("postgres://src:pass@host:5432/src", "postgres://dst:pass@host:5432/dst"),
cloneDatabase("postgres://src:pass@host:5432/src", "postgres://dst:pass@host:5432/dst", false),
).rejects.toThrow("Failed to clone");
});

it("runs psql with -v ON_ERROR_STOP=1 so restore failures surface instead of being silently swallowed", async () => {
vi.mocked(runPipedCommands).mockResolvedValue({stdout: "", stderr: "", exitCode: 0});
await cloneDatabase("postgres://src:pass@src-host:5432/srcdb", "postgres://dst:pass@dst-host:5432/dstdb");
await cloneDatabase("postgres://src:pass@src-host:5432/srcdb", "postgres://dst:pass@dst-host:5432/dstdb", false);
const [, consumer] = vi.mocked(runPipedCommands).mock.calls[0]!;
expect(consumer.args).toContain("-v");
expect(consumer.args).toContain("ON_ERROR_STOP=1");
});

it("passes sanitizeCloneLine as the transformLine argument", async () => {
vi.mocked(runPipedCommands).mockResolvedValue({stdout: "", stderr: "", exitCode: 0});
await cloneDatabase("postgres://src:pass@src-host:5432/srcdb", "postgres://dst:pass@dst-host:5432/dstdb");
await cloneDatabase("postgres://src:pass@src-host:5432/srcdb", "postgres://dst:pass@dst-host:5432/dstdb", false);
const call = vi.mocked(runPipedCommands).mock.calls[0]!;
expect(call[2]).toBe(sanitizeCloneLine);
});

it("does not pass --schema-only by default", async () => {
it("omits --schema-only when schemaOnly is false", async () => {
vi.mocked(runPipedCommands).mockResolvedValue({stdout: "", stderr: "", exitCode: 0});
await cloneDatabase("postgres://src:pass@src-host:5432/srcdb", "postgres://dst:pass@dst-host:5432/dstdb");
await cloneDatabase("postgres://src:pass@src-host:5432/srcdb", "postgres://dst:pass@dst-host:5432/dstdb", false);
const [producer] = vi.mocked(runPipedCommands).mock.calls[0]!;
expect(producer.args).not.toContain("--schema-only");
});
Expand Down
Loading