From 6a5abef794a02d298fd904ba4e8b27d222a87ea0 Mon Sep 17 00:00:00 2001 From: Clay Richard Date: Tue, 10 Nov 2026 15:32:55 -0600 Subject: [PATCH] updated readme --- README.md | 72 ++++++++++---------------------------------- licenses/README.md | 30 +++++------------- plug/external/SOURCE | 63 ++++++++------------------------------ 3 files changed, 36 insertions(+), 129 deletions(-) diff --git a/README.md b/README.md index 0b26306..8f962dd 100644 --- a/README.md +++ b/README.md @@ -4,26 +4,15 @@ ## Motivation -Data leakage is ubiquitous in protein function prediction. A model tested on homologs of -what it trained on can score well by recalling a label rather than generalising, and each -benchmark defines its own split, so scores cannot be compared across functions. PLUG applies -one definition of leakage to eleven functional datasets at once: two proteins are too alike -to separate at mmseqs identity > 0.3 over 80% coverage or foldseek TM > 0.7, and two ligands -at ecfp4 tanimoto > 0.5. It offers two kinds of split and a screen for pretraining data: - -- **lohi** solves [Lo-Hi](https://github.com/SteshinSS/lohi_neurips2023)'s balanced vertex - minimum k-cut over the pooled graph. Nothing similar crosses a split in any dataset, so it - supports transfer between benchmarks, and it pays for that in dropped data. -- **datasail** runs [DataSAIL](https://github.com/kalininalab/DataSAIL) (leakage minimization) once per benchmark. - It keeps more data and minimises sequence leakage inside a benchmark rather than forbidding - it, so it suits one benchmark at a time and shouldn't be used for transfer. +Data leakage is ubiquitous in protein function prediction. A model tested on homologs of what it trained on can score well by recalling a label rather than generalising, and each benchmark defines its own split, so scores cannot be compared across functions. PLUG applies one definition of leakage to eleven functional datasets at once: two proteins are too alike to separate at mmseqs identity > 0.3 over 80% coverage or foldseek TM > 0.7, and two ligands at ecfp4 tanimoto > 0.5. It offers two kinds of split and a screen for pretraining data: + +- **lohi** solves [Lo-Hi](https://github.com/SteshinSS/lohi_neurips2023)'s balanced vertex minimum k-cut over the pooled graph. Nothing similar crosses a split in any dataset, so it supports transfer between benchmarks, and it pays for that in dropped data. +- **datasail** runs [DataSAIL](https://github.com/kalininalab/DataSAIL) (leakage minimization) once per benchmark. It keeps more data and minimises sequence leakage inside a benchmark rather than forbidding it, so it suits one benchmark at a time and shouldn't be used for transfer. - **sample_unlabeled_trainset** holds a training set from a reservoir clear of every benchmark. ## Usage -Needs [uv](https://docs.astral.sh/uv/), [mmseqs2](https://github.com/soedinglab/MMseqs2) and -[foldseek](https://github.com/steineggerlab/foldseek) on your `PATH`. Every setting is a -constant in `plug/config.py` or at the top of the script that uses it. +Needs [uv](https://docs.astral.sh/uv/), [mmseqs2](https://github.com/soedinglab/MMseqs2) and [foldseek](https://github.com/steineggerlab/foldseek) on your `PATH`. Every setting is a constant in `plug/config.py` or at the top of the script that uses it. ```bash uv sync # the pinned solver is part of what makes a split reproducible @@ -43,19 +32,11 @@ uv run python3 -m plug.sample_unlabeled_trainset --struct # the pdb, screened uv run python3 -m plug.sample_unlabeled_trainset --both # the pdb, screened on either ``` -A split is written to `results///`, and `splits/.csv.gz` has one line -per datapoint in the order of that dataset's `rows()`, so labelling a copy is a zip. Check the two lengths match first: a dataset whose rows have -changed would otherwise be mislabelled without any error. A training -set is written to `out/trainset_.fasta`. +A split is written to `results///`, and `splits/.csv.gz` has one line per datapoint in the order of that dataset's `rows()`, so labelling a copy is a zip. Check the two lengths match first: a dataset whose rows have changed would otherwise be mislabelled without any error. A training set is written to `out/trainset_.fasta`. -The downloads fetch whatever each source currently publishes, so a split you build covers the -data as it stands today. The splits in `results/` are the paper's, built on the data of that -time; their labels only line up with a download whose rows have not changed, and BindingDB's -already have. plug-analysis is the archive of the paper's results. +The downloads fetch whatever each source currently publishes, so a split you build covers the data as it stands today. The splits in `results/` are the paper's, built on the data of that time; their labels only line up with a download whose rows have not changed, and BindingDB's already have. -The first build runs mmseqs over the pool (an hour or more) and foldseek over 22k chains (a -day), then caches both in `work/`. The sequence and ligand caches are named for the thresholds -and the dataset list, not their contents, so delete `work/` after downloading newer data. +The first build runs mmseqs over the pool (an hour or more) and foldseek over 22k chains (a day), then caches both in `work/`. The sequence and ligand caches are named for the thresholds and the dataset list, not their contents, so delete `work/` after downloading newer data. ## Adding a dataset @@ -80,11 +61,7 @@ class Mydata(Benchmark): yield from csv.DictReader(f) ``` -Import it in `plug/benchmarks/__init__.py` and add its name to `DATASETS`, the tuple of datasets -every split and screen runs over. With the `ligand` view, set `ligand` to the field holding -each row's smiles (default `"smiles"`). With `structure`, give each row a `pdb` field and -return `rcsb.structures(cls)` from `structures()`, as `allobench.py` does; every protein needs -a structure or the build stops. Add a `data/download_mydata.sh` so others can fetch it. +Import it in `plug/benchmarks/__init__.py` and add its name to `DATASETS`, the tuple of datasets every split and screen runs over. With the `ligand` view, set `ligand` to the field holding each row's smiles (default `"smiles"`). With `structure`, give each row a `pdb` field and return `rcsb.structures(cls)` from `structures()`, as `allobench.py` does; every protein needs a structure or the build stops. Add a `data/download_mydata.sh` so others can fetch it. Then rerun whichever you need, exactly as above: @@ -93,36 +70,19 @@ uv run python3 -m plug.split --lohi # or --datasail uv run python3 -m plug.sample_unlabeled_trainset --seq # or --struct, --both ``` -Adding a dataset changes the dataset list, so the build gets a new tag in `results/` and the -searches run again over the larger pool. lohi is solved over that pool, so every dataset's -lohi split can change, not only the new one's; datasail solves each dataset on its own. +Adding a dataset changes the dataset list, so the build gets a new tag in `results/` and the searches run again over the larger pool. lohi is solved over that pool, so every dataset's lohi split can change, not only the new one's; datasail solves each dataset on its own. ## Main results -- **Leakage inflates scores.** Against random controls dealt to the same sizes and matched on - zero-shot difficulty, lohi lowers frozen ESM3 probe performance by 25.9% (linear) and 26.3% - (MLP) on average over the eleven datasets, with AUROC measured above chance. The drop is - largest for ligand binding (LP-PDBBind 82%, BindingDB 66%, linear) and smallest for - residue-level readouts such as ATLAS, where the linear probe does not drop at all. -- **Overlap with pretraining data is large.** Of one million UniRef90 sequences, 38.5% have a - homolog in a PLUG dataset at 20% identity, 21.5% at 30% and 8.0% at 50%. -- **Datasets overlap each other.** A protein appearing in more than one dataset (2,051 of them) - carries one split label under lohi; 1,097 carry two under datasail, which is why only lohi may be - trained across benchmarks (transfer learning) +- **Leakage inflates scores.** Against random controls dealt to the same sizes and matched on zero-shot difficulty, lohi lowers frozen ESM3 probe performance by 25.9% (linear) and 26.3% (MLP) on average over the eleven datasets, with AUROC measured above chance. The drop is largest for ligand binding (LP-PDBBind 82%, BindingDB 66%, linear) and smallest for residue-level readouts such as ATLAS, where the linear probe does not drop at all. +- **Overlap with pretraining data is large.** Of one million UniRef90 sequences, 38.5% have a homolog in a PLUG dataset at 20% identity, 21.5% at 30% and 8.0% at 50%. +- **Datasets overlap each other.** A protein appearing in more than one dataset (2,051 of them) carries one split label under lohi; 1,097 carry two under datasail, which is why only lohi may be trained across benchmarks (transfer learning) ## Licenses -PLUG does not redistribute data. Each source keeps its own terms, and two of them restrict -reuse: ATLAS is CC BY-NC 4.0, and LP-PDBBind is licensed for educational, research and -not-for-profit use only, with affinities that come from PDBbind, whose license forbids -redistribution. A bundle containing either would carry those limits over all of it. So this -repo ships code, download scripts that fetch each dataset from its source, and split labels, -and every user accepts each source's terms directly. Terms, full texts and citations for every -dataset are in [`licenses/README.md`](licenses/README.md). - -PLUG's own code is MIT ([`licenses/PLUG-MIT-LICENSE`](licenses/PLUG-MIT-LICENSE)). -`plug/external/` vendors Lo-Hi and DataSAIL, both MIT, with every edit recorded in -[`plug/external/SOURCE`](plug/external/SOURCE). +PLUG does not redistribute data. Each source keeps its own terms, and two of them restrict reuse: ATLAS is CC BY-NC 4.0, and LP-PDBBind is licensed for educational, research and not-for-profit use only, with affinities that come from PDBbind, whose license forbids redistribution. This repo ships code, download scripts that fetch each dataset from its source, and split labels, and every user accepts each source's terms directly. Terms, full texts and citations for every dataset are in [`licenses/README.md`](licenses/README.md). + +PLUG's own code is MIT ([`licenses/PLUG-MIT-LICENSE`](licenses/PLUG-MIT-LICENSE)). `plug/external/` vendors Lo-Hi and DataSAIL, both MIT, with every edit recorded in [`plug/external/SOURCE`](plug/external/SOURCE). diff --git a/licenses/README.md b/licenses/README.md index 4aa3ba5..ad91cd2 100644 --- a/licenses/README.md +++ b/licenses/README.md @@ -1,12 +1,8 @@ # dataset licenses -plug's git repo ships **download scripts, not data** — `data/*` is gitignored except the -`download_*.sh` scripts. so publishing this repo redistributes code and split labels (row indices), -not the datasets, which is unrestricted. +plug's git repo ships **download scripts, not data** — `data/*` is gitignored except the `download_*.sh` scripts. so publishing this repo redistributes code and split labels (row indices), not the datasets, which is unrestricted. -these licenses matter the moment you redistribute the *fetched data itself* — a cached csv/fasta under -`data/`, the `data/formatted/*.fasta` bundle, or a released benchmark tarball. each source keeps its own -terms; the full texts are in this directory. +these licenses matter the moment you redistribute the *fetched data itself* — a cached csv/fasta under `data/`, the `data/formatted/*.fasta` bundle, or a released benchmark tarball. each source keeps its own terms; the full texts are in this directory. ## plug's code @@ -37,29 +33,17 @@ MIT: [PLUG-MIT-LICENSE](PLUG-MIT-LICENSE). | **lp-pdbbind** | `download_lp_pdbbind.sh` | github THGLab/LP-PDBBind | **UC Regents — research / non-profit only** | [LP-PDBBind-LICENSE.txt](LP-PDBBind-LICENSE.txt) | no | | **atlas** | `download_atlas.sh` | dsimb.inserm.fr | **CC BY-NC 4.0** | [CC-BY-NC-4.0.txt](CC-BY-NC-4.0.txt) | no | -## watch-outs **two sources are non-commercial** — everything else is freely redistributable with attribution: - **atlas** — CC BY-NC 4.0. commercial use needs the authors' permission. -- **lp-pdbbind** — UC Regents license grants use/copy/modify/**distribute** for *educational, research, and - not-for-profit* purposes only. the affinity values ultimately come from **PDBbind**, whose own license - forbids redistribution; lp-pdbbind is the republication vehicle, so stay within its non-commercial bound - and cite it. this is the softest spot in the set. +- **lp-pdbbind** — UC Regents license grants use/copy/modify/**distribute** for *educational, research, and not-for-profit* purposes only. the affinity values ultimately come from **PDBbind**, whose own license forbids redistribution; lp-pdbbind is the republication vehicle, so stay within its non-commercial bound and cite it. this is the softest spot in the set. -if you ever bundle *all* the formatted test fastas into one release, these two NC terms cover the whole -bundle. keeping them download-on-demand (as the scripts already do) avoids that. +**the generated trainset is clean.** it derives only from uniref90 (CC BY 4.0) + pdb (CC0), so it's freely redistributable — commercial included — with a UniProt attribution line. -**the generated trainset is clean.** it derives only from uniref90 (CC BY 4.0) + pdb (CC0), so it's freely -redistributable — commercial included — with a UniProt attribution line. - -**minor:** -- **passerrank** — the github repo has no LICENSE file, but the script only pulls the ASD table (ASD v3+ is - CC BY) and fetches sequences straight from UniProt (CC BY). the *content* is CC BY on both sides. -- **shs27k** — redistributed via the GNN-PPI repo, but the *content* is a subset of STRING (interactions + - sequences), which is CC BY 4.0. cite GNN-PPI and PIPR for the subset, STRING for the underlying data. -- **bindingdb** — the *Articles* subset used here is bindingdb's own literature curation (CC BY 4.0). the - more restrictive ChEMBL terms apply only to the separate ChEMBL download, which this repo doesn't touch. +- **passerrank** — the github repo has no LICENSE file, but the script only pulls the ASD table (ASD v3+ is CC BY) and fetches sequences straight from UniProt (CC BY). the *content* is CC BY on both sides. +- **shs27k** — redistributed via the GNN-PPI repo, but the *content* is a subset of STRING (interactions + sequences), which is CC BY 4.0. cite GNN-PPI and PIPR for the subset, STRING for the underlying data. +- **bindingdb** — the *Articles* subset used here is bindingdb's own literature curation (CC BY 4.0). the more restrictive ChEMBL terms apply only to the separate ChEMBL download, which this repo doesn't touch. ## cite the sources diff --git a/plug/external/SOURCE b/plug/external/SOURCE index 1a8a7b0..4faa957 100644 --- a/plug/external/SOURCE +++ b/plug/external/SOURCE @@ -6,12 +6,9 @@ min_vertex_k_cut.py is taken unmodified from Lo-Hi: sha256 c9410bd8f02de2a558e918036e28fa6add9ca0b6087438f1c207cd5237cda9fc licence MIT, see ../../licenses/LOHI-MIT-LICENSE -Simon Steshin, "Lo-Hi: Practical ML Drug Discovery Benchmark", NeurIPS 2023. -https://arxiv.org/abs/2310.06399 +Simon Steshin, "Lo-Hi: Practical ML Drug Discovery Benchmark", NeurIPS 2023. https://arxiv.org/abs/2310.06399 -It is vendored rather than cloned at run time because that repo ships no -setup.py and is not installable, and because a split that is reproducible has -to depend on a fixed file rather than on whatever a remote branch holds today. +It is vendored rather than cloned at run time because that repo ships no setup.py and is not installable, and because a split that is reproducible has to depend on a fixed file rather than on whatever a remote branch holds today. ---------------------------------------------------------------------------- @@ -23,42 +20,19 @@ datasail_cluster_1d.py is DataSAIL's cluster-based cold split (their "C1"): sha256 0802b3774fbebdfb70fd26cafff7470c7a512876ff3bc9bb4e1be97d270cfb07 licence MIT, see ../../licenses/DATASAIL-MIT-LICENSE -Roman Joeres, David B. Blumenthal, Olga V. Kalinina, "DataSAIL: Data Splitting -Against Information Leakage". https://doi.org/10.1101/2023.11.15.566305 +Roman Joeres, David B. Blumenthal, Olga V. Kalinina, "DataSAIL: Data Splitting Against Information Leakage". https://doi.org/10.1101/2023.11.15.566305 Four edits, all recorded here and nowhere else: - 1. the import of solve / compute_limits / stratification_constraints now - points at datasail_support.py beside it. Importing the installed datasail - would pull in its whole clustering stack (grakel, h5py, sklearn, cvxopt, - matplotlib) to reach three functions, and its solve() sets a time limit - and fourteen threads, which rules.md's "one shot" forbids. - - 2. the objective is summed over a `pairs` argument -- the cluster pairs that - have a similarity -- instead of over all of them. Every term left out is - multiplied by a similarity of zero, so the objective is the same number; - built over all pairs it would be 1.18 million cvxpy atoms where 1,931 - matter. - - 3. the loss is built as one cvxpy atom over all the pairs at once rather than - one atom per pair. cvxpy warns about exactly this ("Objective contains too - many subexpressions. Consider vectorizing"), and with 5,239 pairs its - canonicalisation ran twelve minutes without finishing, for a model SCIP - then solves in seconds. max over the splits of x[s,e1] - x[s,e2] is 1 when - a pair is separated and 0 when it is not, term for term the same as the - loop. - - 4. an optional `floors` argument: (weights, one floor per split) pairs, each - added as constraint (17) again over a second weight vector. C1 takes ONE - weight vector, so a benchmark weighted by its proteins had no way to keep - rows in test as well -- fireprot's test set came out at 9 proteins and 72 - rows, too few to score. split.py passes the rows as the second vector with - a floor on every split (train 60%, val and test 5%). Left out, as it is for - every benchmark weighted by rows, the model is exactly theirs. - -The model itself -- one binary per (split, cluster), every cluster in exactly -one split, a weight floor per split, and the weighted cross-split similarity as -the loss -- is otherwise untouched. That is the method. + 1. the import of solve / compute_limits / stratification_constraints now points at datasail_support.py beside it. Importing the installed datasail would pull in its whole clustering stack (grakel, h5py, sklearn, cvxopt, matplotlib) to reach three functions, and its solve() sets a time limit and fourteen threads, which rules.md's "one shot" forbids. + + 2. the objective is summed over a `pairs` argument -- the cluster pairs that have a similarity -- instead of over all of them. Every term left out is multiplied by a similarity of zero, so the objective is the same number; built over all pairs it would be 1.18 million cvxpy atoms where 1,931 matter. + + 3. the loss is built as one cvxpy atom over all the pairs at once rather than one atom per pair. cvxpy warns about exactly this ("Objective contains too many subexpressions. Consider vectorizing"), and with 5,239 pairs its canonicalisation ran twelve minutes without finishing, for a model SCIP then solves in seconds. max over the splits of x[s,e1] - x[s,e2] is 1 when a pair is separated and 0 when it is not, term for term the same as the loop. + + 4. an optional `floors` argument: (weights, one floor per split) pairs, each added as constraint (17) again over a second weight vector. C1 takes ONE weight vector, so a benchmark weighted by its proteins had no way to keep rows in test as well -- fireprot's test set came out at 9 proteins and 72 rows, too few to score. split.py passes the rows as the second vector with a floor on every split (train 60%, val and test 5%). Left out, as it is for every benchmark weighted by rows, the model is exactly theirs. + +The model itself -- one binary per (split, cluster), every cluster in exactly one split, a weight floor per split, and the weighted cross-split similarity as the loss -- is otherwise untouched. That is the method. ---------------------------------------------------------------------------- @@ -69,15 +43,4 @@ DataSAIL's double-cold split (their "C2") is NOT used and not vendored: path datasail/solver/cluster_2d.py sha256 506a6316390d7a3b97dddd0d8f1a0db732d1256fa99122a5e157a0f1536b2950 -It was run on bindingdb and lp-pdbbind so their molecules would be split by the -same model as their proteins, and it cannot hold the band there: it assigns the -two axes independently at 58.6/20.7/20.7, and ONE molecule cluster carries 83.8% -of bindingdb's row weight (lp-pdbbind 68.5%, and its protein axis 59.4%, already -past the 58.6% train target). A cluster that big lands whole in one part, so -validation and test cannot both reach their floors at any epsilon or cluster -count -- bindingdb came out 99.4/0.2/0.4 and lp-pdbbind 93.2/3.6/3.1. DataSAIL's -own PDBBind example runs on the core set, 182 entities, where no such blob -exists, and their paper says only that assigning every interaction "cannot be -guaranteed", not that the survivors land on the asked-for ratios. Those molecules -follow their rows instead and the chemistry rule is paid for in dropped rows, in -split.py. +It was run on bindingdb and lp-pdbbind so their molecules would be split by the same model as their proteins, and it cannot hold the band there: it assigns the two axes independently at 58.6/20.7/20.7, and ONE molecule cluster carries 83.8% of bindingdb's row weight (lp-pdbbind 68.5%, and its protein axis 59.4%, already past the 58.6% train target). A cluster that big lands whole in one part, so validation and test cannot both reach their floors at any epsilon or cluster count -- bindingdb came out 99.4/0.2/0.4 and lp-pdbbind 93.2/3.6/3.1. DataSAIL's own PDBBind example runs on the core set, 182 entities, where no such blob exists, and their paper says only that assigning every interaction "cannot be guaranteed", not that the survivors land on the asked-for ratios. Those molecules follow their rows instead and the chemistry rule is paid for in dropped rows, in split.py.