Selectivity-adaptive filtered search: sparse filters get exact results - #5
Merged
Conversation
A sparse allow-bitmap makes graph traversal the wrong algorithm: too few eligible nodes enter the beam, pruning never engages, and the walk decays toward an expensive, approximate scan of the reachable graph. The provably right strategy at high selectivity is a direct scan of the allowed ids - cheaper AND exact. All three search modes now route their candidate pool through collect_pool(): when popcount(allow) <= 16*ef (the popcount is a microsecond pass, bits beyond capacity masked), a zero-allocation exact scan of the allowed set fills the pool - recall 1.0 over the filter by construction, with ascending-id iteration reproducing the brute-force (distance, id) tie order exactly. Larger filters keep the existing traversal with filtered-out nodes routing. Tests: sparse-filter results gated EQUAL to brute force (ids and distances), re-ranked modes stay within the filter, empty bitmap returns 0. Benchmark gains a filtered-search table across 0.1%/1%/10% selectivities measured against exact filtered ground truth. Windows: 6/6 suites green both configs; cross-platform validation via CI (local WSL is congested by unrelated workload load-avg ~30 today - measured numbers for the README follow from a detached run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ULiWeodALX2ZQiLKvTw25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sparse allow-bitmaps now take a zero-allocation exact scan of the allowed set (when
popcount(allow) <= 16*ef) instead of degraded graph traversal — cheaper and recall-1.0 by construction, with tie order matching brute force exactly. Larger filters keep the routed traversal. Applies to all three search modes.Tests gate sparse-filter results equal to brute force (ids + distances); benchmark gains a filtered table across 0.1%/1%/10% selectivities vs exact filtered ground truth.
🤖 Generated with Claude Code
https://claude.ai/code/session_017ULiWeodALX2ZQiLKvTw25