Skip to content

feat: save outputs using the appropriate CIF categories and numbering from the input/PDB deposition - #613

Open
k-chrispens wants to merge 8 commits into
kc/residue-placementfrom
kc/deposit-numbering-v2
Open

k-chrispens wants to merge 8 commits into
kc/residue-placementfrom
kc/deposit-numbering-v2

Conversation

@k-chrispens

@k-chrispens k-chrispens commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Outputs from save_everything could not be lined up residue by residue with the PDB entry they came from. A PDB entry numbers every residue twice: label ids (label_asym_id, label_seq_id), assigned by the PDB and counting along the deposited sequence, and author ids (auth_asym_id, auth_seq_id, insertion code), chosen by the depositors. Model wrappers may renumber residues and build residues that are unmodeled in the input, so neither numbering was being accurately written to the output from a sampleworks run.

The old code for this, carry_polymer_entity_categories, ran after set_structure, which writes chain_id and res_id into both the label and the author categories. It then had to guess where each residue sits in the deposited mmCIF from those label and author categories, after the original numbering was already lost. In practice:

  • Inputs read from an RCSB mmCIF (which has both label and auth numbering) and outputs a model had renumbered fell back to sequence alignment. Both the patch script and the carry_polymer_entity_categories change kept the output chain name as label_asym_id, dropped pdbx_poly_seq_scheme, and wrote label numbers into the author columns: residue −3 of 1VME (author numbering) was written as residue 9.
  • Residues the model built were numbered consecutively from their neighbor. Where the author numbering has gaps or insertion codes (the 149A–149E loop of thrombin), a built residue could take the number of an observed residue.
  • Ligands and waters kept the input chain name (the 1VME chloride was written under author chain C instead of A), and an entity id numbered per chain, which could collide with a deposited entity id.
  • Fetching the deposited mmCIF caught every exception, so a bug in our code was reported the same way as an unreachable RCSB: a warning, and outputs without metadata.

Changes

A new module, utils/deposit_ids.py, works out where each residue of a structure sits in the deposited mmCIF, once, on the AtomArray, before anything is written. Several functions from utils/cif_utils.py are now refactored here:

  • Deposit.from_cif reads the deposited mmCIF once, including every polymer position in pdbx_poly_seq_scheme and the ligands and waters in pdbx_nonpoly_scheme.

  • Each residue is matched to a deposited polymer position by the first of three methods that matches every polymer residue:

    1. its ids already match deposited author ids
    2. its chain and sequence index match deposited label ids
    3. sequence alignment of each chain to the deposited chain with the same name

    A match that gives two residues the same position and monomer is rejected. Two monomers at one position (microheterogeneity, such as a CYS/CSO site) and altlocs are allowed.

  • place_in_deposit labels a model output using the input structure it was matched to (the AtomReconciler indices). A model residue with a matched atom takes the position of its input residue. A residue the model built is placed by counting along the deposited sequence from its nearest placed neighbor, which follows gaps and insertion codes in the author numbering.

  • annotate_deposit_ids stores the result on a copy of the AtomArray: the deposited author ids go into chain_id, res_id and ins_code, and the label ids into deposit_label_* annotations. Ligands and waters listed in pdbx_nonpoly_scheme take the ids of their row. Ligands missing from the PDB entry get a new entity id per residue name, numbered after the highest deposited one.

  • set_deposited_structure writes the CIF with those ids and copies, for the chains, entities and ligands present, the deposited rows of entity, entity_poly, entity_poly_seq, struct_asym, pdbx_poly_seq_scheme and pdbx_nonpoly_scheme. It also copies chem_comp rows (from the CCD when the entry lacks one) and points struct_conn bond partners at the deposited ids. Where the model writes the unmodified parent of a modified residue (MET where the entry has MSE), MET is added to entity_poly_seq as a second monomer at that position, so every residue in atom_site still has a matching sequence row.

In guidance_script_utils.py, _run_guidance fetches the deposited mmCIF once with _load_deposit and passes it to relabel_model_residues and save_everything. Fetching now catches only network and file-reading errors. If a structure cannot be matched to the PDB entry, it is saved without deposited ids and categories, with a warning, so a finished run never loses its coordinates.

Breaking changes

  • cif_utils.carry_polymer_entity_categories is removed. Use deposit_ids.annotate_deposit_ids, then deposit_ids.set_deposited_structure.
  • save_everything takes a required keyword argument deposit: Deposit | None instead of reference_cif, and no longer fetches from RCSB itself, which allows getting deposit . save_trajectory takes deposit instead of reference_cif.
  • Saved CIFs change: the author columns hold the deposited author numbering, and ligands carry their deposited chains and entities.

Reviewing

Most of the cif_utils.py diff is the removal of the old code, which deposit_ids.py replaces. About 30k lines of the diff are three unmodified RCSB mmCIFs added to tests/resources/ (1PPB, 2YL0, 6NI6).

Suggested order:

  1. The public functions in deposit_ids.py: Deposit, place_in_deposit, annotate_deposit_ids, set_deposited_structure.
  2. How guidance_script_utils.py calls them: _load_deposit, relabel_model_residues, save_everything.
  3. Matching residues to positions: _resolve_deposit_positions, _positions_by_lookup, _positions_by_alignment, _one_residue_per_position, _walk_deposited_positions.
  4. Ligands and copied categories: _match_deposited_ligands, _new_entity_ids, _carried_categories, _add_modeled_monomers.
  5. tests/utils/test_cif_entity_categories.py.

Testing

Tests on real PDB entries cover:

  • 1VME: two chains of one entity, label numbers equal to author numbers + 12, and a gap inside the His-tag.
  • 1PPB (thrombin): author chains L and H, which have label chains A and B, insertion codes running from 1H down to 1 in chain L and in the 149A–149E loop of chain H.
  • 6NI6: a gap inside a run of identical residues (LEU 8–9).
  • 2YL0: a density input read by author ids.

Where it applies, each structure is read both by label ids and by author ids.

  • tests/utils and tests/integration/test_mismatch_integration.py in boltz-dev (fast tests): 784 passed, 2 skipped
  • Slow and GPU tests
  • Other model environments (protenix, rf3, protpardelle)
  • An end-to-end guidance run whose saved CIF is checked against its PDB entry

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 29 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 37bc0c5c-251e-4618-901a-1e4aa971ab56
Commits

Reviewing files that changed from the base of the PR and between 37885a2 and 2148698.

Files selected for processing: 10

  • src/sampleworks/utils/cif_utils.py
  • src/sampleworks/utils/deposit_ids.py
  • src/sampleworks/utils/guidance_script_utils.py
  • tests/integration/test_mismatch_integration.py
  • tests/resources/1PPB/1ppb_rcsb.cif
  • tests/resources/2YL0/2yl0_rcsb.cif
  • tests/resources/6NI6/6ni6_rcsb.cif
  • tests/utils/test_cif_entity_categories.py
  • tests/utils/test_guidance_script_utils.py
  • tests/utils/test_save_reconciliation.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

carry_polymer_entity_categories read residue identity back from an
output CIF after set_structure had written chain_id and res_id into both
the label and the author columns, so a residue's identity had to be
guessed again from the written values. The deposit code moves out of
cif_utils into a new deposit_ids module, and deposited label ids are now
resolved once on the AtomArray:

- Deposit parses the deposited CIF once.
- resolve_deposit_labels does the author lookup and the per-entity
  alignment fallback on residues.
- annotate_deposit_ids stores the result as deposit_label_* annotations,
  named apart from the label_* fields readers can load.
- set_deposited_structure writes the structure with those labels, its
  struct_conn partners, and the deposit's categories.

save_everything resolves once per run, and every saved frame reuses the
annotations. The save path now catches only the ValueError and KeyError
that a deposit the structure does not fit raises, instead of every
exception.

Written output is unchanged: on the test inputs it matches the
previous carry column for column. Entity ids are written explicitly,
because atomworks' patch of set_structure takes them from the
chain_entity annotation.

BREAKING CHANGE: carry_polymer_entity_categories is replaced by
deposit_ids.annotate_deposit_ids followed by set_deposited_structure.
…bering

Output CIFs carried the deposit's author numbering only when the
author-number lookup succeeded. Label-numbered inputs (RCSB CIFs) and
model-added residues numbered past the deposit's numbering fell back to
sequence alignment, which kept the output chain as label_asym_id,
carried no pdbx_poly_seq_scheme, and left the input's label numbers in
the auth columns (1VME author -3 written as 9).

resolve_deposit_positions now places each residue at a deposited
PolySeqSchemePosition, trying in turn:

- its ids as the deposit's author ids;
- its chain and number as the deposit's label chain and position;
- per-chain sequence alignment, mapping each chain to the deposited
  chain whose asym id or author strand id it carries. A chain matching
  neither fails instead of being carried under its own name.

annotate_deposit_ids writes the position's author ids into chain_id,
res_id and ins_code, so set_structure writes the deposit's author
numbers, including for residues the model builds from the deposited
sequence. pdbx_poly_seq_scheme is carried on every path. Deposit parses
the scheme once, and residues are keyed by number and insertion code.
Ligands and waters kept the input's chain as both their label and
author chain, so 1VME's chloride was written with author chain C
instead of A. A non-polymer residue the deposit's pdbx_nonpoly_scheme
lists, under the same number, insertion code and monomer and under its
label or author chain, now takes that row's label chain and entity and
its author ids, and the matching struct_asym, entity and
pdbx_nonpoly_scheme rows are carried. annotate_deposit_ids records the
matched row as a deposit_nonpoly_row annotation, so writing does not
look ligands up again.

A ligand the deposit lacks keeps its ids but gets a new entity id, one
per residue name after the deposit's highest, instead of the per-chain
number set_structure gives it, which could collide with a deposited
entity.
A match can give two residues the same deposited position, e.g. a
residue the model built numbered like a resolved one. Both then took
one label, monomer and model, so the CIF named two atoms identically.
Such a match now fails and resolution falls through to the next one.
Microheterogeneity (different monomers at one position) and altlocs
are still allowed.
relabel_model_residues numbered a residue the input lacks consecutively
from its nearest matched neighbour. In author numbering that can skip
or insert (thrombin's 149A-149E loop), so a built residue could take a
resolved residue's number and the saved CIF could not place it.

Given the deposit, relabeling now places every model residue in it
through place_in_deposit. Matched residues take their input residue's
deposited position. Each built residue is placed as many positions
along the deposited chain from its neighbour's as it is in the model. A
built residue whose name the deposit does not list there fails the
placement, and the consecutive labels are kept with a warning.

_run_guidance loads the deposit once (_load_deposit, which warns and
returns None when it cannot be fetched or read) for relabeling and
saving. save_everything takes it as a required deposit keyword instead
of fetching again when it is None, and only places the input array
itself; a model was placed when it was relabeled.
_load_deposit caught every exception and wrote outputs without the
deposit's categories, so a programming error looked like an unreachable
RCSB. It now catches only what fetching and reading a deposit can
raise: OSError (which covers requests' errors), biotite's RequestError
and file errors, and the ValueError or KeyError of a file without the
expected block, categories or columns. Deposit.from_cif reads the block
inside the loader, so a corrupt download fails there instead of later,
while saving.
@k-chrispens
k-chrispens force-pushed the kc/deposit-numbering-v2 branch from 2633494 to 2148698 Compare November 16, 2026 00:59
@k-chrispens k-chrispens changed the title Kc/deposit numbering v2 feat: save outputs using the appropriate CIF categories and numbering from the input/PDB deposition Nov 16, 2026
@k-chrispens
k-chrispens marked this pull request as ready for review November 16, 2026 01:35
Copilot AI balanced review requested due to automatic review settings November 16, 2026 01:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@DorisMai DorisMai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was only able to do a coarse pass today, but I already have some preliminary questions/concerns. I will do another pass tomorrow more closely on deposit_ids.py and the tests, ideally with pod access. If the comments don't make sense / I misunderstood things we can chat more at Wednesday's PR review meeting?

raise TypeError("Unknown guidance type!")

metadata = result.metadata or {}
deposit = _load_deposit(args.protein)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_load_deposit currently always look through RCSB for the deposited file. What about unpublished structure, or other custom files (e.g. I canonicalize the residues of an rcsb cif file when making synthetic data)? We probably should allow passing a --deposit-cif file path to construct Deposit?

model_atom_array: AtomArray,
struct_atom_array: AtomArray,
reconciler: AtomReconciler,
deposit: Deposit | None = None,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It bothers me that we need two structures/files for guidance, one for guidance input and one for guidance output. Is there an intentional distinction between them, like guidance input is minimal just for observed coordinates and b-factors, to support synthetic cases? Or is it just somehow the current situation we end up in? Should we aim for eventually only have 1 reference structure, both for guidance input and for output labeling?

_Structure = TypeVar("_Structure", AtomArray, AtomArrayStack)


class _Residues(NamedTuple):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems unnecessary? most fields are just copy of atomarray attributes, the only thing new is polymer, which you could have just make a _polymer_mask helper function?

)


_Structure = TypeVar("_Structure", AtomArray, AtomArrayStack)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: kinda strange to have this line here? maybe either at top or closer to the only place that uses this (annotate_deposit_ids) much further down


The key holds the author ids or the label ids of the residue
(``_resolve_deposit_positions``). A polymer ``HETATM`` residue with no position is treated
as a ligand, since a free amino acid (a GLU ligand beside a chain containing GLU) looks like

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this specific situation (an unplaced free GLU ligand) seems still problematic. _positions_by_lookup treats an unplaced residue that is polymer and hetero as a ligand, but _walk_deposited_positions doesn't, and depending on whether this GLU is its own chain or not would raise at line 637/651/653.

residue_ids.append(
(
(str(chain), int(number), str(code)),
(str(chain), _INAPPLICABLE, new_entity_ids[str(name)]),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, the label_asym_id is the raw chain letter, and it could collide with any letter already labels another entity: for example, the renumber=True case of test_carry_keeps_polymer_beside_non_polymer and test_carry_relabels_struct_conn_partners. one asym shouldn't hold two entities, so these leftover residues should probably get a fresh label_asym_id?

This branch is waiting to be deployed

1 waiting deployment
gpu-testing — 21486980 Waiting Nov 16, 2026 by k-chrispens via rf3-dev #1209
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants