Repository navigation
feat: save outputs using the appropriate CIF categories and numbering from the input/PDB deposition - #613
feat: save outputs using the appropriate CIF categories and numbering from the input/PDB deposition#613k-chrispens wants to merge 8 commits into
Conversation
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 29 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
486c702 to
2633494
Compare
carry_polymer_entity_categories read residue identity back from an output CIF after set_structure had written chain_id and res_id into both the label and the author columns, so a residue's identity had to be guessed again from the written values. The deposit code moves out of cif_utils into a new deposit_ids module, and deposited label ids are now resolved once on the AtomArray: - Deposit parses the deposited CIF once. - resolve_deposit_labels does the author lookup and the per-entity alignment fallback on residues. - annotate_deposit_ids stores the result as deposit_label_* annotations, named apart from the label_* fields readers can load. - set_deposited_structure writes the structure with those labels, its struct_conn partners, and the deposit's categories. save_everything resolves once per run, and every saved frame reuses the annotations. The save path now catches only the ValueError and KeyError that a deposit the structure does not fit raises, instead of every exception. Written output is unchanged: on the test inputs it matches the previous carry column for column. Entity ids are written explicitly, because atomworks' patch of set_structure takes them from the chain_entity annotation. BREAKING CHANGE: carry_polymer_entity_categories is replaced by deposit_ids.annotate_deposit_ids followed by set_deposited_structure.
…bering Output CIFs carried the deposit's author numbering only when the author-number lookup succeeded. Label-numbered inputs (RCSB CIFs) and model-added residues numbered past the deposit's numbering fell back to sequence alignment, which kept the output chain as label_asym_id, carried no pdbx_poly_seq_scheme, and left the input's label numbers in the auth columns (1VME author -3 written as 9). resolve_deposit_positions now places each residue at a deposited PolySeqSchemePosition, trying in turn: - its ids as the deposit's author ids; - its chain and number as the deposit's label chain and position; - per-chain sequence alignment, mapping each chain to the deposited chain whose asym id or author strand id it carries. A chain matching neither fails instead of being carried under its own name. annotate_deposit_ids writes the position's author ids into chain_id, res_id and ins_code, so set_structure writes the deposit's author numbers, including for residues the model builds from the deposited sequence. pdbx_poly_seq_scheme is carried on every path. Deposit parses the scheme once, and residues are keyed by number and insertion code.
Ligands and waters kept the input's chain as both their label and author chain, so 1VME's chloride was written with author chain C instead of A. A non-polymer residue the deposit's pdbx_nonpoly_scheme lists, under the same number, insertion code and monomer and under its label or author chain, now takes that row's label chain and entity and its author ids, and the matching struct_asym, entity and pdbx_nonpoly_scheme rows are carried. annotate_deposit_ids records the matched row as a deposit_nonpoly_row annotation, so writing does not look ligands up again. A ligand the deposit lacks keeps its ids but gets a new entity id, one per residue name after the deposit's highest, instead of the per-chain number set_structure gives it, which could collide with a deposited entity.
A match can give two residues the same deposited position, e.g. a residue the model built numbered like a resolved one. Both then took one label, monomer and model, so the CIF named two atoms identically. Such a match now fails and resolution falls through to the next one. Microheterogeneity (different monomers at one position) and altlocs are still allowed.
relabel_model_residues numbered a residue the input lacks consecutively from its nearest matched neighbour. In author numbering that can skip or insert (thrombin's 149A-149E loop), so a built residue could take a resolved residue's number and the saved CIF could not place it. Given the deposit, relabeling now places every model residue in it through place_in_deposit. Matched residues take their input residue's deposited position. Each built residue is placed as many positions along the deposited chain from its neighbour's as it is in the model. A built residue whose name the deposit does not list there fails the placement, and the consecutive labels are kept with a warning. _run_guidance loads the deposit once (_load_deposit, which warns and returns None when it cannot be fetched or read) for relabeling and saving. save_everything takes it as a required deposit keyword instead of fetching again when it is None, and only places the input array itself; a model was placed when it was relabeled.
_load_deposit caught every exception and wrote outputs without the deposit's categories, so a programming error looked like an unreachable RCSB. It now catches only what fetching and reading a deposit can raise: OSError (which covers requests' errors), biotite's RequestError and file errors, and the ValueError or KeyError of a file without the expected block, categories or columns. Deposit.from_cif reads the block inside the loader, so a corrupt download fails there instead of later, while saving.
2633494 to
2148698
Compare
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
DorisMai
left a comment
There was a problem hiding this comment.
I was only able to do a coarse pass today, but I already have some preliminary questions/concerns. I will do another pass tomorrow more closely on deposit_ids.py and the tests, ideally with pod access. If the comments don't make sense / I misunderstood things we can chat more at Wednesday's PR review meeting?
| raise TypeError("Unknown guidance type!") | ||
|
|
||
| metadata = result.metadata or {} | ||
| deposit = _load_deposit(args.protein) |
There was a problem hiding this comment.
_load_deposit currently always look through RCSB for the deposited file. What about unpublished structure, or other custom files (e.g. I canonicalize the residues of an rcsb cif file when making synthetic data)? We probably should allow passing a --deposit-cif file path to construct Deposit?
| model_atom_array: AtomArray, | ||
| struct_atom_array: AtomArray, | ||
| reconciler: AtomReconciler, | ||
| deposit: Deposit | None = None, |
There was a problem hiding this comment.
It bothers me that we need two structures/files for guidance, one for guidance input and one for guidance output. Is there an intentional distinction between them, like guidance input is minimal just for observed coordinates and b-factors, to support synthetic cases? Or is it just somehow the current situation we end up in? Should we aim for eventually only have 1 reference structure, both for guidance input and for output labeling?
| _Structure = TypeVar("_Structure", AtomArray, AtomArrayStack) | ||
|
|
||
|
|
||
| class _Residues(NamedTuple): |
There was a problem hiding this comment.
seems unnecessary? most fields are just copy of atomarray attributes, the only thing new is polymer, which you could have just make a _polymer_mask helper function?
| ) | ||
|
|
||
|
|
||
| _Structure = TypeVar("_Structure", AtomArray, AtomArrayStack) |
There was a problem hiding this comment.
nitpick: kinda strange to have this line here? maybe either at top or closer to the only place that uses this (annotate_deposit_ids) much further down
|
|
||
| The key holds the author ids or the label ids of the residue | ||
| (``_resolve_deposit_positions``). A polymer ``HETATM`` residue with no position is treated | ||
| as a ligand, since a free amino acid (a GLU ligand beside a chain containing GLU) looks like |
There was a problem hiding this comment.
this specific situation (an unplaced free GLU ligand) seems still problematic. _positions_by_lookup treats an unplaced residue that is polymer and hetero as a ligand, but _walk_deposited_positions doesn't, and depending on whether this GLU is its own chain or not would raise at line 637/651/653.
| residue_ids.append( | ||
| ( | ||
| (str(chain), int(number), str(code)), | ||
| (str(chain), _INAPPLICABLE, new_entity_ids[str(name)]), |
There was a problem hiding this comment.
In this case, the label_asym_id is the raw chain letter, and it could collide with any letter already labels another entity: for example, the renumber=True case of test_carry_keeps_polymer_beside_non_polymer and test_carry_relabels_struct_conn_partners. one asym shouldn't hold two entities, so these leftover residues should probably get a fresh label_asym_id?
Why
Outputs from
save_everythingcould not be lined up residue by residue with the PDB entry they came from. A PDB entry numbers every residue twice: label ids (label_asym_id,label_seq_id), assigned by the PDB and counting along the deposited sequence, and author ids (auth_asym_id,auth_seq_id, insertion code), chosen by the depositors. Model wrappers may renumber residues and build residues that are unmodeled in the input, so neither numbering was being accurately written to the output from a sampleworks run.The old code for this,
carry_polymer_entity_categories, ran afterset_structure, which writeschain_idandres_idinto both the label and the author categories. It then had to guess where each residue sits in the deposited mmCIF from those label and author categories, after the original numbering was already lost. In practice:carry_polymer_entity_categorieschange kept the output chain name aslabel_asym_id, droppedpdbx_poly_seq_scheme, and wrote label numbers into the author columns: residue −3 of 1VME (author numbering) was written as residue 9.Changes
A new module,
utils/deposit_ids.py, works out where each residue of a structure sits in the deposited mmCIF, once, on theAtomArray, before anything is written. Several functions fromutils/cif_utils.pyare now refactored here:Deposit.from_cifreads the deposited mmCIF once, including every polymer position inpdbx_poly_seq_schemeand the ligands and waters inpdbx_nonpoly_scheme.Each residue is matched to a deposited polymer position by the first of three methods that matches every polymer residue:
A match that gives two residues the same position and monomer is rejected. Two monomers at one position (microheterogeneity, such as a CYS/CSO site) and altlocs are allowed.
place_in_depositlabels a model output using the input structure it was matched to (theAtomReconcilerindices). A model residue with a matched atom takes the position of its input residue. A residue the model built is placed by counting along the deposited sequence from its nearest placed neighbor, which follows gaps and insertion codes in the author numbering.annotate_deposit_idsstores the result on a copy of theAtomArray: the deposited author ids go intochain_id,res_idandins_code, and the label ids intodeposit_label_*annotations. Ligands and waters listed inpdbx_nonpoly_schemetake the ids of their row. Ligands missing from the PDB entry get a new entity id per residue name, numbered after the highest deposited one.set_deposited_structurewrites the CIF with those ids and copies, for the chains, entities and ligands present, the deposited rows ofentity,entity_poly,entity_poly_seq,struct_asym,pdbx_poly_seq_schemeandpdbx_nonpoly_scheme. It also copieschem_comprows (from the CCD when the entry lacks one) and pointsstruct_connbond partners at the deposited ids. Where the model writes the unmodified parent of a modified residue (MET where the entry has MSE), MET is added toentity_poly_seqas a second monomer at that position, so every residue inatom_sitestill has a matching sequence row.In
guidance_script_utils.py,_run_guidancefetches the deposited mmCIF once with_load_depositand passes it torelabel_model_residuesandsave_everything. Fetching now catches only network and file-reading errors. If a structure cannot be matched to the PDB entry, it is saved without deposited ids and categories, with a warning, so a finished run never loses its coordinates.Breaking changes
cif_utils.carry_polymer_entity_categoriesis removed. Usedeposit_ids.annotate_deposit_ids, thendeposit_ids.set_deposited_structure.save_everythingtakes a required keyword argumentdeposit: Deposit | Noneinstead ofreference_cif, and no longer fetches from RCSB itself, which allows getting deposit .save_trajectorytakesdepositinstead ofreference_cif.Reviewing
Most of the
cif_utils.pydiff is the removal of the old code, whichdeposit_ids.pyreplaces. About 30k lines of the diff are three unmodified RCSB mmCIFs added totests/resources/(1PPB, 2YL0, 6NI6).Suggested order:
deposit_ids.py:Deposit,place_in_deposit,annotate_deposit_ids,set_deposited_structure.guidance_script_utils.pycalls them:_load_deposit,relabel_model_residues,save_everything._resolve_deposit_positions,_positions_by_lookup,_positions_by_alignment,_one_residue_per_position,_walk_deposited_positions._match_deposited_ligands,_new_entity_ids,_carried_categories,_add_modeled_monomers.tests/utils/test_cif_entity_categories.py.Testing
Tests on real PDB entries cover:
Where it applies, each structure is read both by label ids and by author ids.
tests/utilsandtests/integration/test_mismatch_integration.pyinboltz-dev(fast tests): 784 passed, 2 skipped