[joein]

Cross-encoders now batch (query, document) pairs with similar token counts together, using the length sorting from #831. Each batch is padded to its longest pair, so short documents batched with a long one were processed at the long one's length. Scores still come back in the original order.

rerank and rerank_pairs sort pairs by token count within windows of 8 batches, and rerank_pairs does the same on the parallel path. iter_length_sorted_batches is now generic, so it accepts pairs as well as texts.

Case	main	This PR	Speedup
rerank, 100 documents	2.45 s	1.69 s	1.4x
rerank, 1,000 documents	27.05 s	5.14 s	5.3x
rerank_pairs, 4,096 pairs	110.79 s	19.39 s	5.7x
rerank_pairs, 4,096 pairs, parallel=2	307.30 s	54.67 s	5.6x
rerank_pairs, single pair	0.02 s	0.02 s	1.0x

• Apple M2 Max, 12 cores, 32 GB RAM, CPU only.
• Python 3.10.10, onnxruntime 1.23.2, Xenova/ms-marco-MiniLM-L-6-v2, batch size 64. Times are the best of two runs after a warmup.
• Documents are 90% short texts of 3 to 25 words and 10% texts of 250 to 350 words. Queries are 3 to 10 words. rerank uses one query, and rerank_pairs uses a different query for each pair.
• Scores match main within 5.2e-6. The difference comes only from padding length: pairs padded to the same length on both branches score identically, and the rest differ in the last few bits of the float32 score.
• Reranking batch_size documents or fewer is one batch either way, so it doesn't get faster.
• parallel=2 is slower than serial on both branches on this machine, because each worker runs ONNX Runtime with one thread.

The reranker tests used two documents with the same token count, so their order never changed and they passed even with order restoration removed. test_batch_rerank and test_rerank_pairs_parallel now add a longer third document and compare every score with that document scored on its own.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 6899c09f-f428-41cb-9ad9-e1ae94622344

Commits

Reviewing files that changed from the base of the PR and between b25a95e and e4c03c3.

Files selected for processing: 3

• fastembed/common/utils.py
• fastembed/rerank/cross_encoder/onnx_text_model.py
• tests/test_text_cross_encoder.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Walkthrough

Reranking now processes documents and query-document pairs in token-count-sorted batches, then restores scores to input order. The parallel pair-processing path tracks original indices alongside worker outputs. The batching helper now accepts generic item types. Tests check batched scores against individually computed scores and verify result ordering.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Refactor

Possibly related PRs

• qdrant/fastembed#831: Introduced the length-sorted batching helper that this PR generalizes and uses for reranking pairs.

Suggested reviewers: i8dnlo, tbung

Merge Risk: Minimal · up to e4c03

No actionable merge-blocking risk is established; the change is ready for normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surv…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly and concisely describes the main change: sorting cross-encoder batches by token count for performance.
Description check		The description directly explains the batching changes, order restoration, parallel processing, benchmarks, and test updates.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 6 functions across 3 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}