[hyphaltip]

What

Per-species clone groups for the reads survey are now computed with the error-tolerant solid containment distance. This is the distance from sketch_query.py, validated against SNPs on 136 A. fumigatus strains. The worker sketches from the 2026-11-08 sketch pass provide the input. The site uses these tables in place of the older plain-containment pair tables.

• datasets/reads/sra_survey/pilot/clone_solid.py (new), which:
  • drops k-mers seen once, as the sketch store does;
  • downsamples to scaled 1000;
  • computes solid containment ANI;
  • clusters by average linkage (scipy);
  • for each run, records clone ids at 0.9999, 0.9995, 0.999 and 0.998, median k-mer abundance, and its nearest neighbour;
  • adds a same-strain-name check for calibration.
• clone_solid/ (new, 2.4 MB) contains:
  • <slug>.clones.tsv for 202 species groups, covering 21,618 of 22,138 runs;
  • summary.tsv;
  • same_strain_pairs.tsv.gz;
  • README.md with the method, calibration, limits and the reproduce command.
• scripts/build_reads_site.py: the site prefers clone_solid/<slug>.clones.tsv and falls back to the old pair tables. It groups at 0.9995, with the plateau checked against 0.999. Counts are withheld when:
  • median k-mer depth is below 8 (21 species);
  • the species is uniform at sketch resolution, meaning median between-run ANI is 0.999 or more (13 species, for example E. dermatitidis, T. indotineae, L. parapsilosis).
• Pages: species.html and index.html read the thresholds and method from the data, and say why counts were withheld.

Calibration

There are 30,045 pairs of runs filed under the same strain name. The ANI distribution has two modes, with a valley between 0.999 and 0.9995 (881 pairs). About 6,300 pairs sit at or above 0.9995: resequencing of one strain. Most of the rest sit at or below 0.999: one name reused for different lines. This matches the sketch store's 30x validation, where the same-strain median was 0.99996 and the between-strain p99 was 0.999. The table is in clone_solid/README.md.

A first run kept the k-mers seen once, and its median abundance came out at 1–5 for 167 species. It is not used; it is described in the README and kept on the HPCC for comparison.

Effect on the site

• 202 species have clone groups.
• 83 get counts per clone.
• 103 pass the plateau rule.
• 13 are withheld as uniform, and 21 for low coverage.

Checks

• New tests:
  • tests/test_clone_solid.py, 8 tests on synthetic sketches, no sourmash needed;
  • 3 builder tests: the solid table is preferred, low coverage withholds counts, uniform species withhold counts.
• Full suite on the HPCC at 8aff1a9: 1962 passed, 6 skipped. The last commit only edits the README.
• Site build from the branch ran without error, and produced the tallies above.

Note

This branches from main without #317 (BioProject links). Both PRs edit build_reads_site.py and species.html in separate places. Whichever merges second may need a small rebase, and I can do it.

🤖 Generated with Claude Code