Skip to content

fix(core): read a scalar subquery in a select list - #12

Merged
IL-William merged 1 commit into
masterfrom
fix/scalar-subquery-master
Oct 30, 2026
Merged

IL-William merged 1 commit into
masterfrom
fix/scalar-subquery-master

Conversation

@IL-William

@IL-William IL-William commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Patch 12 of the engine fork that collin pins (dbt-collin decision 0028). A scalar subquery in a select list is now analyzed, and its column feeds the item that reads it.

Until now it never was: only the item's column references were read, and those skip a subquery. With no source, the output column got a relation-level edge from the FROM table, carrying the subquery's text:

SELECT b.book_id,
       (SELECT max(r.stars) FROM reviews AS r WHERE r.book_id = b.book_id) AS best_stars,
       (SELECT max(r.written_at) FROM reviews AS r) AS last_review_at
FROM books AS b

came out with best_stars and last_review_at fed by the books table, and reviews nowhere. A reader of that edge's text could only guess, and collin's credited b.book_id to best_stars.

The change

  • Each Expr::Subquery an item reads as a value, alone or inside a function, a CASE, a cast or an operator, is analyzed into a (subquery) node of its own, through the helper analyze_predicate_subquery now shares, and its columns are added to the item's sources.
  • Its columns are registered in the enclosing select's scope under the subquery's own key (register_scalar_subquery_in_scope), until that select ends, so a later item reading the item's alias reaches them too. The key is a subquery alias, never a relation of the select: unqualified names, wildcards, items with no source (count(*), a literal) and the implied schema are as before.
  • The enclosing select's grouping (grouping_columns, has_group_by) is saved and restored around any subquery it analyzes; the subquery's own select used to clear it, so later items lost their grouping keys.
  • The walk follows the same value positions as collect_column_refs, lists every variant as it does, and says they must stay in step. Row-testing forms are not walked into: EXISTS, IN (subquery), and the subquery of = ANY (...) or = ALL (...) in a select list are left as they were.

Tests

  • correlated and uncorrelated subqueries feed their columns from reviews, with no edge from books;
  • inside COALESCE, a CASE, a cast and a concatenation, beside the outer columns read;
  • inside TRIM, SUBSTRING ... FROM ... FOR and AT TIME ZONE;
  • in a grouped select, the later items keep their grouping keys (fails without the restore);
  • in a CTE body;
  • the correlated subquery's own filter stays on reviews, and b resolves;
  • EXISTS in a select list unchanged.

Three of the new tests fail without the patch. cargo test -p flowscope-core passes, cargo fmt --all -- --check and CI's cargo clippy --workspace --locked -- -D warnings are clean. The same commit applies unchanged to collin and master.

Measured through collin

With a local [patch] on this branch:

Project Effect
lineage-library the four book_ratings value lines collin marks now hold: best_stars <- raw.reviews.stars aggregate and last_review_at <- raw.reviews.written_at aggregate given, best_stars <- books.book_id and the lost last_review_at gone; nothing else moves
Jaffle Shop cache identical, --indirect too
Shopify cache identical, --indirect too

🤖 Generated with Claude Code

https://claude.ai/code/session_018LDZwk297SKdJXXLpTFZEh

A subquery in a select list was never analyzed: only the item's column
references were read, and those skip a subquery. With no source, the
output column got a relation-level edge from the FROM table, carrying
the subquery's text.

    SELECT b.book_id,
           (SELECT max(r.stars) FROM reviews AS r
            WHERE r.book_id = b.book_id) AS best_stars
    FROM books AS b

best_stars came out fed by the books table, and reviews appeared
nowhere.

Each scalar subquery an item reads as a value, alone or inside a
function, a CASE, a cast or an operator, is now analyzed into a
(subquery) node of its own, as a predicate subquery already is, and its
column feeds the item. Its columns are registered in the enclosing
select's scope under the subquery's own key, so a later item reading
the alias reaches them, but never as a relation of that select:
unqualified names, wildcards, items with no source and the implied
schema are as before. The enclosing select's grouping is saved and
restored around any subquery it analyzes. EXISTS, IN and the subquery
of ANY or ALL in a select list are left as they were.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018LDZwk297SKdJXXLpTFZEh
@IL-William
IL-William force-pushed the fix/scalar-subquery-master branch from 49f39e9 to 36d3597 Compare October 30, 2026 20:22
@IL-William
IL-William merged commit 9b29bdf into master Oct 30, 2026
8 checks passed
@IL-William
IL-William deleted the fix/scalar-subquery-master branch October 30, 2026 20:47
IL-William added a commit that referenced this pull request Oct 30, 2026
Patch 13 of the engine fork that collin pins (dbt-collin decision 0028).
A table or CTE read without an alias is now known by its bare name in
its own scope.

Until now it registered nothing there, so a qualifier naming it was
looked up in the enclosing scopes first:

```sql
SELECT reviews.id FROM events AS reviews
WHERE EXISTS (SELECT 1 FROM raw.reviews WHERE reviews.book_id = 1)
```

put the subquery's filter on `events`, quoted table or not; with `FROM
raw.reviews AS reviews` it landed right. Since patch 12 (#12) a
select-list subquery is analysed too, and collin's tests for correlated
subqueries caught it there. Where no alias competed, the qualifier fell
through to the bare name taken as a table, without its schema, so
`SELECT orders.id FROM db.sch.orders` read no column at all.

## The change

- A `TableFactor::Table` without an alias, table or CTE, registers its
  bare name in its scope (`Scope::bare_tables`), normalized as
  qualifiers are (`normalize_identifier`), mapping to its canonical.
- `resolve_table_alias` and `resolve_alias_instance` check, innermost
  scope first: the aliases, the bare names, then the names with no table
  behind them; the first scope that knows the qualifier answers.
- A bare name two different relations of one scope share resolves to
  neither and stops the search; the same relation twice keeps it.
- A `LATERAL VIEW`'s name, and the aliases of table functions, `PIVOT`,
  `UNPIVOT` and `UNNEST`, are registered normalized in the scope
  (`register_subquery_alias`), so they shadow an outer alias of their
  name too. The global alias set keeps them as written, as before.

Left as it is:
- a quoted table's bare key is folded as if unquoted, which matters only
  for SQL that would not compile on Postgres or Snowflake;
- the qualifier reaches the bare-name lookup unquoted, so a quoted outer
  alias is folded: `FROM events AS "Reviews" WHERE EXISTS (SELECT 1 FROM
  raw.reviews WHERE "Reviews".kind = 'x')` puts the filter on `reviews`,
  with a warning that it has no `kind`, where `master` placed none;
  `master` already fails a quoted alias in the outer select;
- an ordinary alias still matches exactly as written (DuckDB's
  `UNNEST(...) AS U(id)` under an outer `u`), as before this patch.

## Tests

- an unaliased table shadows an outer alias of its name, in an `EXISTS`
  and in a select-list subquery, quoted and not, on Snowflake and
  DuckDB;
- an unaliased CTE shadows one too;
- an outer unaliased table is still read through its bare name;
- columns qualified with the bare names of unaliased tables are read;
- a bare name two tables share resolves to neither;
- a `LATERAL VIEW` name (`e` and `E`), a `TABLE(GENERATOR(...))` alias
  and an upper-case `UNNEST` alias shadow an outer alias.

Each fails without its part of the patch. `cargo test -p
flowscope-core`, `cargo fmt --all -- --check` and CI's `cargo clippy
--workspace --locked -- -D warnings` pass. The same commit applies
unchanged to `collin` and `master`.

## Measured through collin

With a local `[patch]` on this branch, collin's `main`:

| Project | Effect |
| --- | --- |
| lineage-library | `book_ratings` (with patch 12), `loan_titles` and `shelf_tags` give their lineage as written: nineteen marks go, nothing else moves |
| collin's tests | the two correlated-subquery tests that failed on patch 12 alone pass; one test that relied on the engine's false alarm for a CTE named like the alias inside it now finds none |
| Jaffle Shop | cache identical, `--indirect` too |
| Shopify | cache identical, `--indirect` too |

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
IL-William added a commit that referenced this pull request Oct 30, 2026
Patch 13 of the engine fork that collin pins (dbt-collin decision 0028).
A table or CTE read without an alias is now known by its bare name in
its own scope.

Until now it registered nothing there, so a qualifier naming it was
looked up in the enclosing scopes first:

```sql
SELECT reviews.id FROM events AS reviews
WHERE EXISTS (SELECT 1 FROM raw.reviews WHERE reviews.book_id = 1)
```

put the subquery's filter on `events`, quoted table or not; with `FROM
raw.reviews AS reviews` it landed right. Since patch 12 (#12) a
select-list subquery is analysed too, and collin's tests for correlated
subqueries caught it there. Where no alias competed, the qualifier fell
through to the bare name taken as a table, without its schema, so
`SELECT orders.id FROM db.sch.orders` read no column at all.

## The change

- A `TableFactor::Table` without an alias, table or CTE, registers its
  bare name in its scope (`Scope::bare_tables`), normalized as
  qualifiers are (`normalize_identifier`), mapping to its canonical.
- `resolve_table_alias` and `resolve_alias_instance` check, innermost
  scope first: the aliases, the bare names, then the names with no table
  behind them; the first scope that knows the qualifier answers.
- A bare name two different relations of one scope share resolves to
  neither and stops the search; the same relation twice keeps it.
- A `LATERAL VIEW`'s name, and the aliases of table functions, `PIVOT`,
  `UNPIVOT` and `UNNEST`, are registered normalized in the scope
  (`register_subquery_alias`), so they shadow an outer alias of their
  name too. The global alias set keeps them as written, as before.

Left as it is:
- a quoted table's bare key is folded as if unquoted, which matters only
  for SQL that would not compile on Postgres or Snowflake;
- the qualifier reaches the bare-name lookup unquoted, so a quoted outer
  alias is folded: `FROM events AS "Reviews" WHERE EXISTS (SELECT 1 FROM
  raw.reviews WHERE "Reviews".kind = 'x')` puts the filter on `reviews`,
  with a warning that it has no `kind`, where `master` placed none;
  `master` already fails a quoted alias in the outer select;
- an ordinary alias still matches exactly as written (DuckDB's
  `UNNEST(...) AS U(id)` under an outer `u`), as before this patch.

## Tests

- an unaliased table shadows an outer alias of its name, in an `EXISTS`
  and in a select-list subquery, quoted and not, on Snowflake and
  DuckDB;
- an unaliased CTE shadows one too;
- an outer unaliased table is still read through its bare name;
- columns qualified with the bare names of unaliased tables are read;
- a bare name two tables share resolves to neither;
- a `LATERAL VIEW` name (`e` and `E`), a `TABLE(GENERATOR(...))` alias
  and an upper-case `UNNEST` alias shadow an outer alias.

Each fails without its part of the patch. `cargo test -p
flowscope-core`, `cargo fmt --all -- --check` and CI's `cargo clippy
--workspace --locked -- -D warnings` pass. The same commit applies
unchanged to `collin` and `master`.

## Measured through collin

With a local `[patch]` on this branch, collin's `main`:

| Project | Effect |
| --- | --- |
| lineage-library | `book_ratings` (with patch 12), `loan_titles` and `shelf_tags` give their lineage as written: nineteen marks go, nothing else moves |
| collin's tests | the two correlated-subquery tests that failed on patch 12 alone pass; one test that relied on the engine's false alarm for a CTE named like the alias inside it now finds none |
| Jaffle Shop | cache identical, `--indirect` too |
| Shopify | cache identical, `--indirect` too |

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
IL-William added a commit to IL-William/dbt-collin that referenced this pull request Oct 30, 2026
It drops the nineteen marks the engine side of two Snowflake shapes left
after #48: `book_ratings` (a scalar subquery in a select list),
`loan_titles` and `shelf_tags` (a table without an alias whose columns
are qualified with its own name).

**The pin.** `rev` moves to `b904397`, tagged `collin-engine-8`: the
fork's `master` with

- **patch 12** (IL-William/flowscope#12, #13): a scalar subquery in a
  select list is analysed against a node of its own, and its column
  feeds the item. It was never analysed, so its column was drawn from
  the `FROM` table with the subquery's text, and `engine.rs` read the
  outer side of a correlation out of that text as the value;
- **patch 13** (IL-William/flowscope#14, #15): a table or CTE read
  without an alias is known by its bare name in its scope. With patch 12
  alone (`collin-engine-7`, never pinned), two tests of 0049 failed: the
  engine now placed a subquery's filter, and an unaliased inner table
  did not shadow an outer alias of its name. It also reads a column
  qualified with an unaliased table's bare name, which fell to a name
  without its schema.

**What moves.**

| | Before | After |
| --- | --- | --- |
| lineage-library facts given | 192 of 221 | 203 of 221 |
| Snowflake shapes marked | 5 | 3: `TABLE(FLATTEN(...))`, a star inside a call, a lambda's argument |
| wrong lines given | 6 | 5 |
| Jaffle Shop cache and totals, with and without `--indirect` | | identical |
| Shopify cache and totals, with and without `--indirect` | | identical |

One test,
`a_column_a_child_reads_and_its_parent_s_list_lacks_is_the_parent_s_finding`,
held a CTE named like the alias inside it, where the engine raised a
false alarm that stayed on the child; the engine now reads it right and
the child has no entry, so the case asserts that. The rule it covered, a
column the parent lacks with no edge behind the child's read, which then
stays on the child, has a new case: `quiet_kid` reads `amount` only in a
`WHERE`; it fails with the rule removed.

**Docs.** 0028 lists the eleventh to thirteenth patches and records the
move; `docs/next-steps.md` item 2 and the README's known limits name the
three shapes left, and item 3 counts thirteen patches. No version bump.

246 tests pass with `cargo test --release`.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant