FastRNA-Analysis

Notebooks, figures, and the full data-generation pipeline for the PARSE manuscript. Everything here is reproducible from the public PARSE-RNA engine and the archived data - from viewing a finished figure down to regenerating every feature from raw structures.

Structure

notebooks/
  figures/               main-text figure notebooks
  supplemental_figures/  supplemental figure notebooks
figures/
  main/                  rendered main-text PNGs
  supplemental/          rendered supplemental PNGs
scripts/
  data_gen/              feature + scoring-table regeneration
  validation/            PDB-REDO validation + tool comparison
  benchmark/             runtime benchmark of PARSE and six other annotators
  fetch_data.sh          download the data archive into data/
  fetch_structures.py    download the mmCIF structures into data/structures/
data/                    fetched from the archive (not in git)
  structures/            mmCIF files, downloaded separately (see Data)
  pymol/                 PyMOL sessions and the images the figure notebooks read
  reference/, validation/, comparison/, benchmark/   analysis inputs and outputs

Setup

Requires Python 3.12.

git clone https://github.com/YesselmanLab/FastRNA-Analysis.git
cd FastRNA-Analysis
pip install -r requirements.txt

Reproducing the analysis

Level 0: View the figures

The rendered PNGs live in figures/, and each executed notebook embeds its output. Nothing to install or fetch.

Level 1: Re-run the figures from the archived data

The data/ directory is not tracked in git (see Data). Fetch it:

bash scripts/fetch_data.sh

Then open the notebooks under notebooks/figures/ and notebooks/supplemental_figures/ and run each one; every figure is written into figures/main/ or figures/supplemental/.

Level 2: Rebuild the scoring tables from the feature tables

The scoring tables in data/reference/scoring_tables/ are derived from the per-pair / per-residue feature CSVs in data/reference/high_quality_features/ (and low_quality_features/ for the penalty weights). Regenerate them with the self-contained data_gen scripts (no engine needed):

cd scripts/data_gen
python build_distributions.py               # -> parameter_distributions.json
python build_prosco_distributions.py       # -> prosco_distributions.json
python build_z_tables.py                    # -> z_tables.json
python build_backbone_distributions.py      # -> backbone_prosco_distributions.json
python calculate_penalty_weights.py         # -> penalty_weights.json

Run them in this order: calculate_penalty_weights.py reads the tables the others write.

Level 3: Regenerate the feature tables from raw structures

The feature CSVs come straight from the PARSE C++ engine. Build the engine, then re-extract:

Run everything below from the repo root (FastRNA-Analysis/); the engine is cloned as a sibling of scripts/ and data/.

# 1. Build the engine (C++20, CMake >= 3.28): from the repo root
git clone https://github.com/KamshinenD/PARSE-RNA
cmake -S PARSE-RNA -B PARSE-RNA/build -DCMAKE_BUILD_TYPE=Release
cmake --build PARSE-RNA/build -j
export PARSE_ENGINE="$PWD/PARSE-RNA/build/parse"     # or put `parse` on PATH

# 2. Re-extract every feature table (the engine reads data/structures/<ID>.cif;
#    an entry that is not there is downloaded into data/structures/engine_cache/)
cd scripts/data_gen
python extract_pair_features.py        # -> pairs.csv, hbonds.csv, torsions.csv
python extract_backbone_torsions.py    # -> torsions_all.csv

The commands above regenerate the curated reference set. The penalty weights also need the non-curated set's pair/hbond features:

python extract_pair_features.py \
    --reference-set ../../data/reference/reference_sets/reference_set_poor.json \
    --output-dir    ../../data/reference/low_quality_features

Then rebuild the tables (Level 2) and the figures (Level 1). Both extractors take --limit N for a quick smoke test and --engine /path/to/parse to override PARSE_ENGINE.

Validation data (PDB-REDO)

The PDB-REDO comparison behind the validation figure is regenerated the same way - fetch the re-refined coordinates, then rescore original vs REDO with the engine:

cd scripts/validation             # from the repo root
python fetch_pdb_redo.py          # uniqueRNAS.csv -> data/validation/redo_files/
python compare_pdb_redo_cpp.py    # score original (RCSB) vs REDO -> the two JSONs

fetch_pdb_redo.py downloads <id>_final.pdb from pdb-redo.eu for every candidate in reference_sets/uniqueRNAS.csv that has a PDB-REDO entry (the rest are skipped - the ones that succeed are the validation set). compare_pdb_redo_cpp.py then writes pdb_redo_comparison_all.json and pdb_redo_pairs_comparison.json. PDB-REDO and RCSB are living databases, so a re-fetch may add or refresh a few entries versus the archived comparison.

Other analysis scripts

Run from the repo root. Scripts marked engine need PARSE_ENGINE (Level 3); those marked structures read data/structures/ (see Structures).

script	writes	needs
scripts/reference/score_pairs_uniques.py	data/reference/scores/pair_scores_uniques_parse.jsonl	engine, structures
scripts/reference/count_nucleotides.py	nucleotide counts in data/reference/metadata/metadata_cache.json	structures
scripts/validation/build_resolved_cohort.py	data/reference/resolved_again/resolve_strict.csv
scripts/validation/time_reversal_resolved.py	resolve_pairs_matched.csv, data/validation/time_reversal_resolved.json	engine
scripts/validation/time_reversal_controls.py	data/validation/time_reversal_controls.json
scripts/validation/extract_parse_candidates.py	data/comparison/parse_candidates.json	engine, structures
scripts/validation/extract_residue_names.py	data/comparison/residue_names.json	structures
scripts/validation/adjudicate_consensus.py	data/validation/annotator_consensus.json
scripts/validation/consensus_reported_conflicts.py	data/validation/annotator_consensus_reported.json
scripts/validation/consensus_shared_conflicts.py	data/validation/annotator_consensus_shared.json
scripts/validation/annotator_agreement.py	data/validation/annotator_agreement.json
scripts/validation/referee_coverage.py	data/validation/referee_coverage_current.json
scripts/validation/rejection_referee_support.py	data/validation/rejection_referee_support.json

The tool-comparison scripts run in the order listed: adjudicate_consensus.py first, then the scripts that read annotator_consensus.json.

Runtime benchmark

The timing of FastRNA and the six other annotators is documented in scripts/benchmark/README.md; it reads data/structures/ and writes data/benchmark/runtime_direct_all/.

Data

data/ is not tracked in git (see .gitignore), so a fresh clone can view the figures (Level 0) but needs the archive fetched for anything below that (Levels 1-3). The fetch is a single command - see Level 1:

bash scripts/fetch_data.sh

It downloads data.zip (~300 MB) from Google Drive and unzips it into data/. Extraction is an update rather than a plain overwrite: a file is replaced only when the archived copy is newer than the one on disk, anything missing is added, and tables regenerated locally (Levels 2-3) are newer and so survive. Re-running after the archive is refreshed therefore picks up the new data without clobbering your own. The archive will later be moved to Zenodo.

Structures (data/structures/)

The mmCIF files are not in the archive (12 GB); they are downloaded straight from RCSB. Every script reads structures from here; when the engine is given an entry that is not here it downloads it into data/structures/engine_cache/, so nothing is read from or written to anywhere outside the repository. Fetch them with:

python scripts/fetch_structures.py            # the 2,697 non-redundant RNAs (~12 GB)
python scripts/fetch_structures.py --all      # every RNA entry in the PDB (~10k)

The non-redundant set is the unique_pdb_id column of data/reference/reference_sets/uniqueRNAS.csv; --all uses all_pdb_rnas.csv, which scripts/fetch_all_pdb_rnas.py regenerates from the RCSB search API. Existing files are skipped, so an interrupted download can be repeated. RCSB occasionally re-issues an entry, so a new download can differ slightly from the copy used in the paper; the runtime benchmark records each file's checksum.

What's in data/reference/

subdir	contents
reference_sets/	the curated reference set, the non-curated set, the non-redundant list (uniqueRNAS.csv) and the RCSB RNA universe
high_quality_features/	per-pair / per-hbond / per-residue features (curated set)
low_quality_features/	same, for the non-curated set (penalty-weight derivation)
scoring_tables/	ProSco / Z′ / backbone / penalty-weight tables
scores/, metadata/	per-structure PARSE scores and PDB metadata; per-pair scores (pair_scores_uniques_parse.jsonl, one row per selected pair, each non-redundant entry scored once) are regenerated by scripts/reference/score_pairs_uniques.py; nucleotide counts in metadata_cache.json are recomputed from the local structure files by scripts/reference/count_nucleotides.py (counts every residue of an RNA/DNA polymer entity, so modified, locked and L-nucleotides are included)
resolved_again/	the re-solved cohort (resolve_strict.csv, built by scripts/validation/build_resolved_cohort.py) and the per-molecule / per-structure tables behind Figure 5; see its PROVENANCE.txt
engine_archive/	engine output the re-solved cohort builder reads: per-residue and per-pair rows for the non-redundant set and for the extra members of each BGSU equivalence class (read through scripts/validation/_archive.py)

PyMOL sessions and images (data/pymol/)

subdir	contents
sessions/	every PyMOL session (.pse) behind the structure panels, each saved in its final camera view; README.txt maps sessions to panels
renders/	the ray-traced images the figure notebooks read (Figures 3 and 5, Supplementary Figures S2, S3 and S10), plus 7D_labels.json, the label anchors for Figure S10D

The notebooks read only renders/, never the sessions. The images are rendered from the sessions by a local script that is not part of the repository.

The engine is public at https://github.com/KamshinenD/PARSE-RNA.