ZoomLDM: Latent Diffusion Model for multi-scale image generation

CVPR 2025

         

Setup

git clone https://github.com/cvlab-stonybrook/ZoomLDM/
conda activate zoomldm
pip install -r requirements.txt

The model weights are hosted on huggingface. The inference scripts we provide below download the model weights using huggingface hub.

Inference

Patch level generation

We demonstrate patch-level generation at any scale in sample_patches_brca.ipynb and sample_patches_naip.ipynb.

Large image generation

For large image generation, we use the proposed joint multi-scale sampling algorithm. For a more in-depth analysis of the inference algorithm refer to Fast constrained sampling in pre-trained diffusion models (NeurIPS 2025)

An implementation of the algorithm for generating very large images using ZoomLDM is given in joint_multiscale.ipynb.

For more examples of large images, see our website.

Super-resolution

Super-resolution uses the condition inversion algorithm proposed in the paper and with the joint multi-scale sampling to enforce the low-resolution constraint.

We prove an implementation in superres.ipynb.

Training

To train the model, you need to prepare a multi-scale dataset of {images, conditioning}.

Patch extraction

We use the codebase of DS-MIL to extract regions from the WSIs, first at the base 20x magnification. The patches range from 256x256 to 32768x32768 pixels. You might want to use a lower tissue threshold for larger images.

The following command will extract 1024x1024 patches at 20x:

python deepzoom_tiler.py -m 0 -b 20 -s 1024

Refer to this issue for satellite image patch extraction.

Feature extraction

Histopathology

We pre-extract UNI embeddings (conditioning) from the full resolution images in a patch-based manner. A 2048x2048 image would result in 64x256x256 patches -> 64x1024 UNI embedding.

We then resize images to 256x256, extract VAE features, and save them together with the UNI embeddings.

For NAIP, we use the pre-trained DINO-v2 ViT-Large (dinov2_vitl14_reg) checkpoint to extract embeddings.

Please take a look at the demo datasets: brca/naip or our dataloader scripts: brca/naip for more details.

Training command

Create a config file similar to this, which specifies the dataset, model, and training parameters.

Then, run the training script:

python main.py -t --gpus 0,1,2 --base configs/zoomldm_brca.yaml

Conditioning diffusion model (CDM)

Our image synthesis pipeline requires a set of SSL embeddings and a magnification level, which involves extracting embeddings from a reference image. This becomes impractical when access to the reference image is unavailable. To address this, we introduce the Conditioning Diffusion Model (CDM) that generates SSL embeddings conditioned on the specified magnification level.

We provide the training and inference scripts for CDM in cdm_dit/.

Pretrained model weights for CDM are available here.

Healthy-tissue fine-tune

Results from the Rendeiro group. The published ZoomLDM weights, trained on breast-cancer slides, were further trained on healthy tissue from the Genotype-Tissue Expression (GTEx) project, across organs. Conditioning is the UNI embedding of each patch, as in the original model. The validation loss reached 0.128 at training step 56,000.

The table is the patch Fréchet Inception Distance used in the ZoomLDM paper: 10,000 generated patches at each magnification, features from an ImageNet Inception-v3. A lower value is closer to real patches. The two columns are not the same test. The published numbers are breast-cancer patches of 256 pixels. The new numbers are healthy-tissue patches of 224 pixels.

Magnification	Published (breast cancer, 256 px)	Step 6,000	Step 21,000	Step 36,000
20×	6.77	17.88	10.49	9.18
10×	7.60	17.44	11.45	10.40
5×	7.98	15.54	11.34	10.55
2.5×	10.73	16.88	12.09	13.49
1.25×	8.74	17.39	8.70	9.99
0.625×	7.99	16.98	7.92	8.77
0.3125×	8.34	18.30	8.61	8.93
0.15625×	13.42	18.58	9.67	9.70

Bibtex

@InProceedings{Yellapragada_2025_CVPR,
  author = {Yellapragada, Srikar and Graikos, Alexandros and Triaridis, Kostas and Prasanna, Prateek and Gupta, Rajarsi and Saltz, Joel and Samaras, Dimitris},
  title = {ZoomLDM: Latent Diffusion Model for Multi-scale Image Generation},
  booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
  month = {June},
  year = {2025},
  pages = {23453-23463}
}