[Concept – not for review] Data flow graph on details pages

[SeanWhelan]

Warning

Concept only.

This is an exploratory prototype to show an idea: a lineage graph on the details pages. It's up as a draft so people can look at it and react to the direction. It hasn't been scoped, designed or agreed as something we're building, and the code hasn't been through review. If the idea has legs, it will come back as properly scoped PRs.

The idea

A Data flow tab on capture, collection (including derivation) and materialization details. It shows the spec's lineage as a left-to-right graph: captures → collections → derivations → materializations.

Derivations are the main motivation. Today the details pages show direct neighbours in RelatedEntities, but nothing about transforms, chains of derivations, or what sits more than one hop upstream or downstream.

What the prototype does

• Walks the lineage: 3 hops upstream and downstream, with Show more adding 2. Each hop is one batched liveSpecs(by: { names }) request, plus one request for derivation models (model includes the schema, so only derivations fetch it).
• Shows each transform: one edge per derivation transform, so two transforms on the same source are two edges. Disabled transforms are dashed, and edge tooltips show the transform name, shuffle and readDelay.
• Handles cycles: self-sourcing derivations render as a loop, and other cycles as edges routed over the top.
• Shows inaccessible specs: sources the viewer can't read render as locked placeholders that keep their name.
• Selection: clicking a spec highlights its full upstream and downstream. A panel below the graph shows its details: connector or runtime, bindings, key, alerts, connector status, last publication, and a table of related specs (transforms for derivations, targets and sources for tasks, writers and readers for collections), with links to each one's Data flow tab.
• Rendering: React Flow (@xyflow/react, MIT) handles pan, zoom, fit, controls, the minimap and dragging. Node positions come from a small in-house layered layout, so there's no elkjs. The React Flow attribution is hidden via proOptions; whether to support React Flow Pro would be a call for us to make if this went further.

Findings worth keeping even if the UI isn't

Checked against flow master @ 93f99e1 on a local stack, as three users with different grants:

• liveSpecs(by: { names }) fails the whole request (plain-text PermissionDenied, not GraphQL JSON) if any single name is unreadable, contrary to the schema comment on LiveSpecRef.userCapability. Batch lookups must only send names whose ref came back with a user capability.
• readBy / writtenBy omit unreadable specs entirely (require_min_capability: Read), while readsFrom / writesTo return them with userCapability: null. So "no access" placeholders can only ever appear upstream.
• Disabled transforms and bindings are left out of readsFrom / writesTo / readBy; they're only visible in model.
• readsFrom is deduped per source: two transforms on one source give one entry, so per-transform edges need model.derive.transforms.
• Self-sourcing derivations list themselves in their own readsFrom and readBy.
• Edge lists default to 50 items and are silently cut off beyond that without first:.
• Dark theme: the dark palette never sets background.paper, so MUI's neutral #121212 shows up next to our blue-grey surfaces. The prototype works around it locally; worth fixing in the theme separately.

Try it

Storybook: Details/DataFlow has seven stories built from real responses captured from the local stack (DataFlow/fixtures/).

Not done

No screenshots here, no product or design sign-off, and no thought yet about very large tenants beyond a 200-node cap. Also not started: a prefix-wide graph, throughput on edges, impact analysis, and a warning on transform rename.

Screenshots