Skip to content

graphql: add discovery APIs - #3545

Open
jshearer wants to merge 4 commits into
masterfrom
jshearer/discovers-graphql-codex
Open

jshearer wants to merge 4 commits into
masterfrom
jshearer/discovers-graphql-codex

Conversation

@jshearer

@jshearer jshearer commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Description:

Add createDiscover and discover(id) to GraphQL. Discovery uses the capture staged in an owned draft, or copies a readable live capture while retaining its publication precondition. Submission atomically copies the definition and queues the existing discovery executor.

Workflow steps:

  1. Create a draft. Stage an initial capture with bindings: [], or use an existing live capture.
  2. Call createDiscover(draftId, captureName, dataPlane); the optional data plane must match an existing capture's plane or be permitted by the new capture's storage mapping.
  3. Poll discover(id) until status leaves QUEUED. Read logs through discover(id).logs, and inspect the resulting definitions through draft(id).specs.
  4. Review and publish the draft separately.

@jshearer jshearer changed the title graphql: add draft-backed capture discovery graphql: add discovery APIs Sep 29, 2026
@jshearer jshearer self-assigned this Sep 29, 2026
@jshearer
jshearer force-pushed the jshearer/discovers-graphql-codex branch 2 times, most recently from 5dfbeaf to 2ccffef Compare September 29, 2026 16:18
@jshearer
jshearer force-pushed the jshearer/discovers-graphql-codex branch from 057ddd7 to 7c0739a Compare September 29, 2026 19:09
@jshearer
jshearer added this pull request to stack #3555 September 29, 2026 19:28
@jshearer
jshearer force-pushed the jshearer/discovers-graphql-codex branch 6 times, most recently from 5ae1132 to bf90ad6 Compare October 1, 2026 17:48
pub enum JobStatus {
/// The discover is queued or in progress.
Queued,
Success,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inner fields that Success used to have were vestigial. I found no reads of either field in the UI, and no consumers remain in the backend. #1764 moved auto-discovers into capture controllers and stopped producing meaningful values for these fields. Since then, the discover handler/executor has written discovers.job_status from Success { publication_id: None, specs_unchanged: false }. Serde's skip_serializing_if = "Option::is_none" and skip_serializing_if = "std::ops::Not::not" already omitted both fields, so successful jobs were already stored as {"type":"success"}.

@jshearer
jshearer force-pushed the jshearer/discovers-graphql-codex branch from bf90ad6 to 5045442 Compare October 1, 2026 19:36
@jshearer
jshearer marked this pull request as ready for review October 1, 2026 19:51
@jshearer
jshearer requested a review from GregorShear October 1, 2026 19:51
@strix-security

strix-security Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 6 commits after the last Strix review (5045442). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

No security issues found.

Review summary

Reviewed all 35 changed files for PR #3545, focusing on the new createDiscover/discover GraphQL API in crates/control-plane-api/src/server/public/graphql/discovers.rs, the moved storage-mapping helpers in crates/control-plane-api/src/storage_mappings.rs, the refactored errors_for_draft in drafts/mod.rs, and the JobStatus consolidation in models/src/discovers.rs. The implementation consistently enforces draft ownership (rechecked inside the transaction), hard-gates capture edits via verify_authorization (SpecEdit), gates data-plane access via may_access (Read), scopes storage-mapping lookups to the caller's tenant while excluding recovery/ prefixes, keeps all SQL parameterized, validates log/cursor pagination bounds, and preserves SOPS-encrypted endpoint config without decrypting or leaking it. Concurrency is handled with explicit row-level locks and ON CONFLICT DO NOTHING. Deleted code is a no-behavior-change move/refactor. The accompanying tests cover cross-tenant visibility, token capability enforcement, plane gating, race conditions, and rollback atomicity. Semgrep (p/rust) on the changed production files reported no findings. No security vulnerabilities were identified in the changed code.

Updated for 5045442.


Reviewed by Strix
Re-run review 路 Configure security review settings

Move StorageRow, resolve_storage_mappings, join_storage_mappings, and their
regression test into the existing storage_mappings module. Preserve the
function bodies and test expectations, adjusting visibility and module paths.

Publications still fetch and join mappings separately. Leave publication
placement unchanged so discovery can reuse resolution without rewriting it.
Keep the persisted discovery status model independent of its executor, retaining
the executor's existing import path through a re-export. Remove success payload
fields that discovery no longer produces.

Pin every persisted status tag and retain deserialization of historical success
records containing publication results.
Allow draft owners to query existing discovery jobs, current draft errors, and
paginated logs. Reuse the persisted status model and the draft error reader,
and document historical statuses and the limits of log pagination.

Seed existing jobs directly in read-side tests so ownership, historical status,
diagnostics, and pagination are covered independently of GraphQL submission.
Add createDiscover using an owned draft's capture or a readable live capture.
Validate authorization, connector readiness, and placement before atomically
copying any required capture definition and scheduling discovery.

Preserve serialized definitions and publication preconditions. Cover submission
policy, capability checks, encrypted configuration, conflicting staging,
rollback after writes, and the discovery-to-publication workflow.
@jshearer
jshearer force-pushed the jshearer/discovers-graphql-codex branch from 5045442 to 4c5cf32 Compare October 1, 2026 22:51

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant