Integrated Forward-Inverse Network for Lensless Image Reconstruction

Accepted to ECCV 2026

Donggeon Bae¹, Jaewoo Jung^2,3, Yong Guk Kang², Kyung Chul Lee⁴, Taeyoung Kim³, Jongho Kim¹, Sangjun Byun¹, Joonsik Park³, and Seung Ah Lee^1,2

¹Seoul National University, Department of Mechanical Engineering
²Seoul National University, School of Mechanical and Aerospace Engineering/SNU-IAMD
³Yonsei University, Department of Electrical and Electronic Engineering
⁴University of Michigan, Department of Biomedical Engineering

Abstract

Lensless imaging enables compact and versatile computational cameras by replacing bulky optics with thin coded elements. However, reconstruction from the resulting measurements is challenging: large-footprint PSFs produce highly multiplexed observations, making inversion severely ill-conditioned and sensitive to calibration errors and model mismatch. While deep learning approaches, including hybrid models that incorporate physics priors, have shown promise, explicitly maintaining data fidelity throughout the network hierarchy remains difficult. Here, we propose the Integrated Forward-Inverse Network (IFIN), a physics-guided architecture that interleaves differentiable forward projections with learnable inverse updates at every stage, enabling complementary cues to be exploited jointly in the measurement and image domains. This bidirectional coupling supports progressive, physics-consistent refinement and permits system-constrained PSF kernel adaptation under model uncertainty. On challenging lensless benchmarks, including a newly introduced dataset, IFIN achieves state-of-the-art reconstruction quality. We further observe competitive performance on Gaussian deblurring and simulated inline holography reconstruction, suggesting that the same interleaving principle can extend beyond lensless cameras.

Method

IFIN uses an encoder-decoder backbone with Integrated Forward-Inverse Blocks (IFIBs) at every scale. Each IFIB exchanges information between an image-domain stream and a measurement-domain stream using:

• Forward System Operator (FSO): projects the current image-domain representation through the optical forward model.
• Inverse System Operator (ISO): applies a learnable Wiener-like inverse update from the measurement domain.
• Learnable PSF field: refines system kernels end-to-end for calibration mismatch and shift-variant degradation.

Rather than applying a single inversion and then relying only on learned refinement, IFIN repeatedly exchanges forward projections and inverse updates throughout the feature hierarchy. This keeps measurement-domain evidence available during reconstruction.

_{Overall architecture of IFIN. IFIBs are inserted at each encoder-decoder scale to jointly apply FSO and ISO with a shared learnable PSF field.}

_{Integrated Forward-Inverse Block. IFIN couples forward projection and inverse restoration in either a single-PSF or PSF-field setting.}

Results

IFIN improves reconstruction quality across three lensless benchmarks: DiffuserCam, WiderCam, and the MultiWienerNet dataset.

DiffuserCam

Method	PSNR	LPIPS	SSIM
ADMM	12.252	0.607	0.346
Wiener Deconv.	12.552	0.591	0.384
ISO (Ours)	16.528	0.544	0.404
UNet	21.230	0.394	0.656
NAFNet	24.830	0.239	0.810
Le-ADMM-U	23.261	0.312	0.765
DeepLIR	25.958	0.260	0.829
MWNet	24.832	0.247	0.810
UPDN	28.228	0.194	0.877
MWDNs	27.298	0.217	0.845
LensNet	27.650	0.201	0.868
MoDL	27.958	0.183	0.878
IFIN (Ours)	29.862	0.174	0.893

WiderCam

Method	PSNR	LPIPS	SSIM
ADMM	11.843	0.643	0.323
Wiener Deconv.	12.405	0.607	0.369
ISO (Ours)	17.240	0.462	0.444
UNet	21.890	0.474	0.646
NAFNet	23.857	0.245	0.769
Le-ADMM-U	21.956	0.278	0.748
DeepLIR	20.523	0.339	0.642
MWNet	23.001	0.255	0.766
UPDN	23.920	0.229	0.801
MWDNs	24.525	0.224	0.801
LensNet	24.615	0.219	0.806
MoDL	24.791	0.202	0.810
IFIN (Ours)	25.444	0.201	0.824

MultiWienerNet

Method	PSNR	LPIPS	SSIM
ADMM	19.189	0.557	0.420
Wiener Deconv.	18.658	0.640	0.302
ISO (Ours)	20.202	0.623	0.380
UNet	23.859	0.389	0.589
NAFNet	24.657	0.282	0.712
Le-ADMM-U	23.732	0.335	0.702
DeepLIR	22.556	0.379	0.642
MWNet	25.660	0.260	0.728
UPDN	24.364	0.287	0.707
MWDNs	27.436	0.236	0.780
LensNet	27.546	0.221	0.809
MoDL	28.504	0.202	0.831
IFIN (Ours)	31.083	0.175	0.866

_{Visual comparison on DiffuserCam display-capture data. IFIN preserves color fidelity and high-frequency textures while suppressing artifacts.}

_{Comparison on WiderCam. IFIN mitigates field-dependent peripheral blur and geometric distortion while preserving fine textures and edges.}

_{Comparison on MWNet. Results include simulated spatially variant measurements and experimental miniscope captures.}

More qualitative results and system details are available on the project page.

Inline Holography Reconstruction

IFIN also transfers to simulated inline holography reconstruction by replacing the system operators with angular-spectrum forward propagation and corresponding back-propagation.

Method	PSNR	LPIPS	SSIM
RCAN	23.764	0.354	0.702
NAFNet	23.224	0.389	0.634
NAFNet+	25.751	0.242	0.817
IFIN (Ours)	28.302	0.166	0.890

_{Inline holography reconstruction. IFIN adapts to the holographic forward model and recovers cleaner object structures from a single intensity hologram.}

Installation

Python 3.10 or newer is required. Install a CUDA-compatible PyTorch build first when running on a GPU. Perceptual losses and metrics may download VGG weights on first use; the smoke configuration does not require them.

git clone https://github.com/IIL-SNU/IFIN.git
cd IFIN
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Quick Start

IFIN has one implementation, src/models/ifin.py, with paper names IFINNet, IFIB, FSO, ISO, and RB. Dataset and historical architecture differences are configured in YAML, not separate model files. Dataset configurations default to k=1; the pretrained DiffuserCam preset uses k=16. Select the number of PSF-field kernels with model.k in YAML or --k 1, --k 4, --k 9, or --k 16. The paper uses k=1/4/9/16 for the DiffuserCam ablation and k=9 for WiderCam and MultiWienerNet. This IFIN setting does not change baseline architectures.

Dataset	Configuration	Evaluation notebook
WiderCam	configs/widercam.yaml	WiderCam
DiffuserCam	configs/diffusercam.yaml	DiffuserCam
MultiWienerNet	configs/multiwienernet.yaml	MultiWienerNet

Set DATA_ROOT to the selected dataset's root or pass --data-root. PSF paths are relative to that root unless an absolute --psf-path is supplied.

python train.py --config configs/widercam.yaml --data-root /path/to/WiderCam --k 9
python eval.py --config configs/widercam.yaml --data-root /path/to/WiderCam --k 9 --checkpoint /path/to/ifin_checkpoint.pth
python infer.py --config configs/widercam.yaml --data-root /path/to/WiderCam --k 9 --checkpoint /path/to/ifin_checkpoint.pth --output-dir outputs/reconstruction

Inference accepts --input measurement.png or --input measurement.npy; without it, the first test sample is used. NumPy DiffuserCam measurements use the dataset's BGR-to-RGB conversion. Checkpoints are loaded strictly, including legacy CAW state-key renaming. New training saves best/last checkpoints, both optimizer states when applicable, scheduler states, and the configuration. When supplying a checkpoint, select its original k with --k; learned PSF tensors are never resized to fit a different field size.

Pretrained Checkpoints

Download weights (Google Drive) and follow the restoration and evaluation guide.

Dataset	k	Configuration
DiffuserCam	16	configs/diffusercam_k16.yaml
DiffuserCam	1	configs/diffusercam.yaml
WiderCam	9	configs/widercam_k9.yaml
WiderCam	4	configs/widercam_k4.yaml
WiderCam	1	configs/widercam_k1.yaml
MultiWienerNet	9	configs/multiwienernet_k9.yaml
MultiWienerNet	4	configs/multiwienernet_k4.yaml
MultiWienerNet	1	configs/multiwienernet.yaml

Download the selected checkpoint's folder (parts are at most 96 MiB each), together with the root IFIN_checkpoints.json. Keep the folder name unchanged. Select its ID when restoring; the .pth is written inside that folder:

python scripts/restore_checkpoint.py --manifest /path/to/downloads/IFIN_checkpoints.json --checkpoint widercam_k9

Learned PSFs are loaded without resizing; a separate calibration PSF is not required. The guide lists filenames, checkpoint IDs, and dataset conventions. Checkpoints were trained in separate runs; see the guide for their provenance. The WiderCam k=1/4 presets select the historical refinement-block variant; changing only --k on the k=9 preset is insufficient. The supplementary material reports WiderCam k=1/4/9, not k=16. The released WiderCam k=1/4/9 and MultiWienerNet k=1/4/9 weights were each tested on the full available evaluation split with the public CLI; see release validation for results.

python eval.py --config configs/diffusercam_k16.yaml --data-root /path/to/DiffuserCam --checkpoint /path/to/downloads/DiffuserCam_k16/IFIN_DiffuserCam_k16_20251101_best.pth --output outputs/diffusercam_metrics.json
python eval.py --config configs/diffusercam.yaml --data-root /path/to/DiffuserCam --checkpoint /path/to/downloads/DiffuserCam_k1/IFIN_DiffuserCam_k1_20260507_best.pth --output outputs/diffusercam_k1_metrics.json
python eval.py --config configs/widercam_k9.yaml --data-root /path/to/WiderCam --checkpoint /path/to/downloads/WiderCam_k9/IFIN_WiderCam_k9_20250923_best.pth --output outputs/widercam_metrics.json
python eval.py --config configs/widercam_k4.yaml --data-root /path/to/WiderCam --checkpoint /path/to/downloads/WiderCam_k4/IFIN_WiderCam_k4_20251008_best.pth --output outputs/widercam_k4_metrics.json
python eval.py --config configs/widercam_k1.yaml --data-root /path/to/WiderCam --checkpoint /path/to/downloads/WiderCam_k1/IFIN_WiderCam_k1_20251004_best.pth --output outputs/widercam_k1_metrics.json
python eval.py --config configs/multiwienernet_k4.yaml --data-root /path/to/MultiWienerNet --checkpoint /path/to/downloads/MultiWienerNet_k4/IFIN_MultiWienerNet_k4_20251030_best.pth --output outputs/multiwienernet_k4_metrics.json
python eval.py --config configs/multiwienernet_k9.yaml --data-root /path/to/MultiWienerNet --checkpoint /path/to/downloads/MultiWienerNet_k9/IFIN_MultiWienerNet_k9_20250910_best.pth --output outputs/multiwienernet_k9_metrics.json
python eval.py --config configs/multiwienernet.yaml --data-root /path/to/MultiWienerNet --checkpoint /path/to/downloads/MultiWienerNet_k1/IFIN_MultiWienerNet_k1_20251030_best.pth --output outputs/multiwienernet_k1_metrics.json
python infer.py --config configs/diffusercam_k16.yaml --checkpoint /path/to/downloads/DiffuserCam_k16/IFIN_DiffuserCam_k16_20251101_best.pth --input /path/to/measurement.npy --output-dir outputs/diffusercam

For a short test without external data or perceptual weights:

python train.py --config configs/smoke.yaml
python eval.py --config configs/smoke.yaml --checkpoint outputs/checkpoints/smoke/ifin_last.pth
python -m pytest -q tests

Comparison Models

The shared train/eval/infer entry points also support wiener, admm, unet, nafnet, leadmmu, deeplir, multiwienernet, updn, mwdns, lensnet, and modl. --model selects the dataset-specific constructor and PSF recipe automatically.

python eval.py --config configs/diffusercam.yaml --model wiener --data-root /path/to/DiffuserCam
python eval.py --config configs/widercam.yaml --model mwdns --data-root /path/to/WiderCam --checkpoint /path/to/mwdns_checkpoint.pth
python train.py --config configs/widercam.yaml --model nafnet --data-root /path/to/WiderCam

DeepLIR and the research MoDL implementation are external-source integrations, not redistributed copies. WiderCam/MultiWiener ADMM and Le-ADMM-U also use the ADMM class from that external MoDL source. Their adapters retain the research operations and checkpoint structure. Setup, source mappings, model-specific settings, and known notebook inconsistencies are documented in Baseline Implementations and Research Code Provenance. See Third-Party Notices for upstream licenses.

Checkpoint files are distributed separately from this Git repository. Without trained weights, evaluation is a pipeline check, not a paper result.

Dataset Layout

DiffuserCam/
  dataset_train.csv
  dataset_test.csv
  diffuser_images/*.npy
  ground_truth_lensed/*.npy
  psf.tiff

WiderCam/
  dataset_train.csv
  dataset_test.csv
  images/*_rgb8.png
  labels/*.jpg
  sv_psfs/point_4_4_rgb8.png

MultiWienerNet/
  2D/Ground_truth_downsampled/*.png
  2D/Simulated_Miniscope_2D_Training_data/*.png
  PSF/multiWienerPSFStack_40z_aligned.mat

WiderCam target alignment and the MultiWienerNet crop/downsampling are applied by the dataset loaders. Datasets and trained checkpoints remain external.

Evaluation follows each notebook's prediction normalization without rescaling ground truth. DiffuserCam additionally reports _crop metrics; WiderCam reports _dewarped metrics after undoing target alignment. The unsuffixed metrics use the full frame. See Evaluation Convention before comparing results across datasets.

WiderCam Dataset

We introduce WiderCam, a wide-field lensless reconstruction benchmark captured with a compact phase-mask camera. The dataset contains 25,000 paired measurements, split into 24,000 training and 1,000 test images, with strong field-dependent degradation over a wide field of view. Raw measurements are captured at 4608 x 2592 using a Sony IMX708 sensor and resized to 480 x 270; supervision is affine-aligned offline from the display-capture pair.

Dataset: Dataset page

Citation

@inproceedings{bae2026ifin,
  title     = {Integrated Forward-Inverse Network for Lensless Image Reconstruction},
  author    = {Bae, Donggeon and Jung, Jaewoo and Kang, Yong Guk and Lee, Kyung Chul and Kim, Taeyoung and Kim, Jongho and Byun, Sangjun and Park, Joonsik and Lee, Seung Ah},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Contact

For access to additional datasets, checkpoints, or code, please contact donggeonbae@snu.ac.kr.