Skip to content

Fix zero protein counts on PRIDE (proteins-above-lod) - #43

Merged
ypriverol merged 13 commits into
devfrom
fix/protein-counts
Oct 7, 2026
Merged

ypriverol merged 13 commits into
devfrom
fix/protein-counts

Conversation

@ypriverol

@ypriverol ypriverol commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Five datasets show 0 proteins on the PRIDE affinity page. The PRIDE API's noOfProteins comes from this library's proteins-above-lod, which returned an empty list for them. I downloaded the deposited files and found these causes:

Dataset File(s) Cause Fix Count after fix
PAD000037 Olink Explore media .npx.csv File has both LODNPX and a count-based LOD (median 1,002); NPX was compared with LOD, so 0 of 1,319 assays passed Reader prefers LODNPX 868
PAD000024 MDD_rearranged_NPX.npx.csv Hand-rearranged file with shifted columns (PlateID.1–3, QC.1–3); text in NPX/LOD made the comparison raise NPX/LOD/LODNPX coerced to numbers; broken rows become NaN 333
PAD000045 v4.0 ADAT Table-only ADAT with no ^HEADER/^COL_DATA markers; the reader rejected it Parsed as a ^TABLE_BEGIN body 4,078 (both files combined; v4.0 ADAT via eLOD)
PAD000003, PAD000011, PAD000045 (1.3k ADAT) SomaScan ADATs Study samples only (no buffers), so there is no LOD source Return all measured protein reagents (Type == "Protein") with a warning 6,404 / 6,405 / 1,274

Why fall back to measured proteins for SomaScan: in datasets that do have buffers, nearly every SomaScan assay is above eLOD: 94.9–99.7% per dilution on PAD000021. Olink datasets without any LOD still return none, because many Olink assays really are below LOD.

Also in this PR

  • Same LOD as the QC report. get_proteins_above_lod now uses resolve_lod_with_source, moved to processing/lod.py and re-exported from viz.qc.compute. SomaScan therefore uses eLOD rather than an NCLOD computed on buffer samples, matching Use one LOD resolver across the QC report (SomaScan eLOD, FixedLOD shown as active) #30.
  • Study samples only. Controls, which mostly sit below LOD, no longer lower the per-assay share. PAD000002 goes from 1,858 to 1,943, and PAD000004 from 149 to 186.

Other PRIDE values will change on regeneration

Several current counts were produced by an older pyprideap build and are stale; for example, main already gives 2,193 for PAD000022 against 1,882 on PRIDE. On local data, the rest move by −8 to +36: PAD000001 1,347→1,361, PAD000006 2,447→2,458, PAD000013 6,384→6,415, PAD000021 6,364→6,400.

Tests

  • Updated: SomaScan without buffers now returns measured proteins.
  • New tests cover:
    • Olink without any LOD still returns an empty list;
    • Type filter for SomaScan control reagents;
    • controls don't dilute the above-LOD share;
    • LODNPX preference and text coercion;
    • reading a table-only ADAT.
  • With CI's versions: ruff clean, mypy clean (45 files), pytest 171 passed.

Deployment

This targets main so it can be part of 1.2.0, which isn't tagged yet. After release, rebuilding pride-affinity-ws regenerates the cached counts (#2 there). PRIDE's noOfProteins is filled from that endpoint, so whatever job writes it needs to run again.

Separately, PAD000024's deposited file has shifted columns, which the curators may want to check with the submitter.

Summary by CodeRabbit

  • Bug Fixes
    • Olink data imports now convert measurement and detection-limit values to numbers, preserving invalid entries as missing values. Detection-limit metadata is read from the available source.
    • SomaScan table-only ADAT exports can now be read without section markers.
    • Protein counts above the detection limit now follow platform-specific detection-limit availability. SomaScan counts exclude control reagents, and control samples no longer affect Olink percentages.
    • Blank or placeholder UniProt values are excluded from protein results.

claude and others added 13 commits March 24, 2026 21:25
Adds a new `get_unique_samples()` function and `pyprideap unique-samples`
CLI command that extracts unique sample identifiers from a dataset. Supports
local file input or PRIDE accession download, with output to stdout or file.
Control/QC samples are excluded by default (toggle with --include-controls).

Closes #23

https://claude.ai/code/session_01FRn28g9LvUfggGWSzm7hud
Changed default behavior: all samples (including controls) are returned
by default. Use --exclude-controls flag or exclude_controls=True to
filter out control/QC samples.

https://claude.ai/code/session_01FRn28g9LvUfggGWSzm7hud
Import specific functions directly from their submodules at the top of
cli.py instead of lazily importing the full package inside each command.

https://claude.ai/code/session_01FRn28g9LvUfggGWSzm7hud
Skip SampleID when its cardinality matches the feature count (indicating
it is assay-indexed rather than sample-level). Also recognize SampleId
(lowercase 'd') used by SomaScan datasets.

https://claude.ai/code/session_01FRn28g9LvUfggGWSzm7hud
Add unique-samples command to list unique sample identifiers
CI typecheck has failed on every dev push since 2026-10-06 (the April runs
passed). The dev extras are unpinned, so CI picks up pandas 3 /
pandas-stubs 3.0.5 / numpy 2.5 / mypy 2.4, which report 25 errors in code
that had not changed; later merges added 8 more.

- returns of pandas expressions that the stubs type as Any are cast to
  the declared return type (no behaviour change)
- dict-or-Series parameters converted with pd.Series(...) are cast so the
  union narrows (normalization)
- Series.replace("", pd.NA) -> replace({"": pd.NA}); Hashable-keyed status
  lookup; reused variable names in the summary table and dilution QC
  renamed; manifest tabs typed as list[dict[str, Any]]
Fix mypy errors from current pandas-stubs (unblocks #39)
- unique-samples unpacked _download_pad_files() as a list, but it returns
  (files, sdrf_path) since #38, so `unique-samples -a PADxxxxxx` crashed
  (caught by mypy on main)
- get_unique_samples(exclude_controls=True) matched SampleType without
  normalising separators, so NEGATIVE_CONTROL / PLATE_CONTROL /
  SAMPLE_CONTROL were kept; it now uses control_sample_mask like
  filter_controls
- CLI tests for unique-samples with a mocked PRIDE download
Five PAD datasets show 0 proteins on PRIDE because proteins-above-lod
returned an empty list. Causes, from the deposited files:

- Olink exports with both LODNPX and LOD (count-based) were compared with
  LOD: no NPX ever exceeded it (PAD000037). The reader now uses LODNPX.
- hand-rearranged NPX files with shifted columns put text in NPX/LOD and
  the LOD comparison raised (PAD000024). NPX/LOD/LODNPX are coerced to
  numbers; broken rows become NaN.
- table-only ADATs without ^HEADER/^COL_DATA markers were rejected
  (PAD000045 v4.0 file). They are parsed as a TABLE_BEGIN body.
- SomaScan ADATs with study samples only (no buffers) have no LOD source
  (PAD000003, PAD000011, PAD000045 1.3k file). All measured protein
  reagents (Type == "Protein") are returned with a warning; Olink without
  any LOD still returns none.

Counting also now uses the report's LOD resolver (moved to
processing.lod.resolve_lod_with_source; SomaScan uses eLOD, not NCLOD on
buffers) and only study samples, so controls below LOD no longer lower
the per-assay share (PAD000002 1858 -> 1943, PAD000004 149 -> 186).
@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@coderabbitai

coderabbitai Bot commented Oct 7, 2026

Copy link
Copy Markdown

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a35b8836-bdd8-41e9-9775-a5eed090d18f
📥 Commits

Reviewing files that changed from the base of the PR and between 107fc83 and 41aed2d.

📒 Files selected for processing (5)
  • src/pyprideap/io/readers/olink_csv.py
  • src/pyprideap/io/readers/somascan_adat.py
  • src/pyprideap/processing/lod.py
  • src/pyprideap/viz/qc/compute.py
  • tests/test_qc_pipeline.py
 ________________________________________________
< I raised Series-B funding to review your code. >
 ------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ypriverol
ypriverol changed the base branch from main to dev October 7, 2026 10:55
@ypriverol
ypriverol merged commit 06f5f81 into dev Oct 7, 2026
6 of 7 checks passed
@ypriverol
ypriverol deleted the fix/protein-counts branch October 7, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants