[ypriverol]

This makes the Olink CSV reader faster and lighter. It's the last commit of #45 (merged before it was pushed), now reworked.

Result on PAD000061 (1.7 GB Explore HT CSV, 6.1 M rows, 1,118 samples × 5,425 assays)

	read time	peak memory
main	294 s	4.2 GB
this PR	~6.7 s	~2.5 GB

Peak is the process peak over three runs (2.5–2.7 GB). The previous version of this PR took 43 s and 4.7 GB.

How

1. Parsing: the file is parsed with pyarrow, which is already a dependency.
   • Only the columns the reader uses are read: 14 of 30 in this file.
   • Identifiers and flags are stored as categories.
   • Files of 256 MB or more are streamed in blocks to bound memory; smaller files use one multithreaded read, because streaming adds about 1 s.
   • Anything pyarrow can't parse (ragged rows, text in NPX) falls back to pandas.
2. Matrices: NPX and LOD are placed into the sample × assay matrix by position instead of with pivot_table, with the same behaviour: first non-missing value per cell, assays sorted, assays with no values dropped. QC flags stay as integer codes until the final table.

Output

• Identical to main's reader on all 15 local Olink files (CSV and parquet): expression, samples, features, LOD and QC-flag tables.
• On PAD000061: samples, features and both QC-flag tables match main exactly. NPX and LOD differ in the last binary digit (|diff| ≤ 9×10⁻¹⁶ NPX, 2×10⁻¹⁵ LOD). pandas' default fast float parser misrounds 87,407 of the 5.9 M NPX values, while pyarrow matches Python's correctly rounded float() for all of them. The new values are the exact ones.
• One intentional change: sample IDs are always read as text, so numeric-looking IDs keep their leading zeros (0670628553 stayed 670628553 before).

Tests

• New tests cover:
  • the streamed and single reads agreeing;
  • leading zeros kept, and unused columns skipped;
  • MissingFreq values like 30%;
  • falling back when NPX contains text;
  • delimiter detection;
  • first-value-wins and missing QC flags.
• With CI's versions: 194 passed; ruff and mypy are clean.

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of Olink CSV files with different delimiters, duplicate measurements, missing QC flags, and nonnumeric expression values.
  • Preserved leading zeros in sample IDs and text values in feature data.
• Performance
  • Large files can be processed in streaming mode, and unused columns are skipped when supported.
• Tests
  • Added coverage for delimiter detection, streamed parsing, value preservation, and QC flag handling.

[qodo-code-review]

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: dc1fd243-9409-4ac7-a178-0735dad48221

📥 Commits

Reviewing files that changed from the base of the PR and between 05a9c9c and 4d2b622.

📒 Files selected for processing (2)

• src/pyprideap/io/readers/olink_csv.py
• tests/test_qc_pipeline.py

 ________________________________
< I bench-press race conditions. >
 --------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}