Skip to content

[HSTACK] fix: share filtered-read predicates across fragments - #3

Merged
flobinsa merged 1 commit into
main-v14.0.0-beta.6from
fix_large_filters
Oct 21, 2026
Merged

flobinsa merged 1 commit into
main-v14.0.0-beta.6from
fix_large_filters

Conversation

@flobinsa

@flobinsa flobinsa commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

fix: share filtered-read predicates across fragments

Before

Filtered-read planning deep-cloned logical predicates for each fragment.
Converting the plan into fragment read tasks cloned them again.

Large predicates, such as long IN lists, multiplied memory usage by the
number of fragments and could cause OOMs.

After

Fragments share immutable logical predicates through Arc<Expr>, including
when creating read tasks. Filter-selection and evaluation behavior remain
unchanged.

Changes

  • Allocate shared full and refine predicates once per scan plan.
  • Clone Arc references instead of expression trees for each fragment.
  • Preserve shared ownership in ScopedFragmentRead.
  • Extend the existing plan round-trip test to assert that fragment-plan
    filters share one allocation.

Validation

  • Filtered-read suite: 86 passed.
  • Full Lance library suite: 4,922 passed, 3 ignored.
  • Local planner probe with 37,574 IDs across 64 fragments: peak RSS decreased
    from approximately 409 MB to 66 MB.
  • Issued a COUNT(*) on local and finished without OOM

@github-actions github-actions Bot added the bug Something isn't working label Oct 21, 2026
@flobinsa flobinsa changed the title fix: share filtered-read predicates across fragments [HSTACK] fix: share filtered-read predicates across fragments Oct 21, 2026
@github-actions

Copy link
Copy Markdown

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@flobinsa
flobinsa merged commit d049150 into main-v14.0.0-beta.6 Oct 21, 2026
1 of 2 checks passed
@flobinsa
flobinsa deleted the fix_large_filters branch October 21, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants