Light Field Imaging with Multi-Camera Array

A complete pipeline for light field photography using a 3×3 multi-camera array, including calibration, preprocessing, and interactive refocusing visualization.

---

Table of Contents

• Overview
• Features
• Pipeline Architecture
• Repository Structure
• Requirements
• Installation
• Usage
  • Step 1: Camera Calibration
  • Step 2: Image Preprocessing
  • Step 3: Light Field Rendering (MATLAB GUI)
• Algorithm Details
  • Homography-Based Calibration
  • Shift-and-Sum Refocusing
• Sample Data
• Configuration
• Troubleshooting
• License

---

Overview

This project implements an end-to-end light field imaging system using a 3×3 planar camera array (9 cameras). Light field photography captures not only the intensity but also the direction of light rays, enabling post-capture refocusing, depth estimation, and novel view synthesis.

The pipeline consists of three main stages:

1. Calibration — Align all 9 cameras to a common reference using a chessboard pattern and homography estimation
2. Preprocessing — Apply calibration matrices to raw images for geometric rectification
3. Rendering — Interactive MATLAB GUI for shift-and-sum refocusing at arbitrary depths

---

Features

Feature	Description
Multi-camera calibration	Chessboard-based homography estimation with sub-pixel corner detection
Geometric rectification	Perspective warp alignment of all sub-aperture images
Batch preprocessing	Recursive processing of nested sample directories
Interactive refocusing	GUI with live slider control over focal depth
Click-to-focus	Automatic depth selection via Sobel edge sharpness maximization
Focal stack generation	Export a full sequence of refocused images across a depth range
Chinese path support	OpenCV read/write wrappers for non-ASCII file paths (Windows)

---

Pipeline Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     RAW CAMERA IMAGES                           │
│                  (9 cameras × N sample sets)                    │
└──────────────────────────┬──────────────────────────────────────┘
                           │
           ┌───────────────▼───────────────┐
           │   findHomography.py           │
           │   CALIBRATION STAGE           │
           │                               │
           │  • Chessboard corner detection │
           │  • Sub-pixel refinement        │
           │  • Homography estimation       │
           │  • Save: 0.npy ~ 8.npy        │
           └───────────────┬───────────────┘
                           │ Homography matrices
           ┌───────────────▼───────────────┐
           │   do_change.py                │
           │   PREPROCESSING STAGE          │
           │                               │
           │  • Load .npy matrices          │
           │  • warpPerspective()           │
           │  • Batch apply to raw images   │
           │  • Save aligned output         │
           └───────────────┬───────────────┘
                           │ Aligned light field
           ┌───────────────▼───────────────┐
           │   Main4.m (MATLAB GUI)         │
           │   RENDERING STAGE              │
           │                               │
           │  • Build LF tensor (M×N×H×W×C) │
           │  • Shift-and-sum refocusing    │
           │  • Interactive slider          │
           │  • Click-to-focus              │
           │  • Focal stack export          │
           └───────────────────────────────┘
                           │
                           ▼
                  REFOCUSED IMAGES
              (arbitrary focal planes)

---

Repository Structure

Light-field/
├── Main4.m                  # MATLAB GUI — Light field refocusing application
├── findHomography.py        # Stage 1: Chessboard calibration → homography matrices
├── do_change.py             # Stage 2: Apply calibration matrices to raw images
│
├── CalibrationPic/          # Calibration images (chessboard, 9 views, 0.bmp – 8.bmp)
├── calibration/             # Output of findHomography.py
│   ├── 0.npy … 8.npy        #   9 homography matrices (3×3 camera array)
│   └── pic/                 #   Calibrated chessboard images (verification)
│
├── pics/                    # Input: Raw light field images (organized by sample)
│   ├── polyethylene+kieselguhr_1/
│   │   ├── 1-1.bmp, 1-2.bmp, 1-3.bmp
│   │   ├── 2-1.bmp, 2-2.bmp, 2-3.bmp
│   │   └── 3-1.bmp, 3-2.bmp, 3-3.bmp
│   └── polyethylene+kieselguhr_2/
│
└── result/                  # Output: Calibrated & aligned images (mirrors pics/ tree)
    ├── polyethylene+kieselguhr_1/
    └── polyethylene+kieselguhr_2/

Naming Convention

Images follow a row-column naming scheme: {row}-{col}.bmp (1-indexed), matching the physical camera layout:

Camera Array (M=3 rows × N=3 cols):

  (1,1)  (1,2)  (1,3)       ┌───────┬───────┬───────┐
  (2,1)  (2,2)  (2,3)   →   │ 1-1   │ 1-2   │ 1-3   │   ← Center camera (2,2) is the reference
  (3,1)  (3,2)  (3,3)       ├───────┼───────┼───────┤
                             │ 2-1   │ 2-2 ★ │ 2-3   │
                             ├───────┼───────┼───────┤
                             │ 3-1   │ 3-2   │ 3-3   │
                             └───────┴───────┴───────┘

---

Requirements

Python (Calibration + Preprocessing)

Dependency	Version	Purpose
Python	≥ 3.6	Runtime
OpenCV (opencv-python)	≥ 4.0	Chessboard detection, homography, warpPerspective
NumPy	≥ 1.18	Matrix operations, .npy file I/O

MATLAB (Rendering)

Component	Purpose
MATLAB	Core environment
Image Processing Toolbox	imshow, imread, imwrite, interpn
No extra toolboxes required

Hardware

• Camera array: 3×3 planar arrangement (9 cameras)
• Calibration target: Chessboard with 5×7 inner corners
• Storage: ~9.4 MB per BMP image (9 calibration + 9 per sample)

---

Installation

Python Setup

# Clone the repository
git clone https://github.com/ofen1996/Light-field.git
cd Light-field

# (Optional) Create a virtual environment
python -m venv venv
# Windows:
venv\Scripts\activate
# Linux/macOS:
source venv/bin/activate

# Install dependencies
pip install opencv-python numpy

MATLAB Setup

No installation needed — simply open MATLAB and run Main4.m (ensure the file is on the MATLAB path).

---

Usage

Step 1: Camera Calibration

Capture 9 images of a chessboard from your 3×3 camera array, all focusing on the same planar target. Place them in CalibrationPic/ as 0.bmp through 8.bmp (indexed left-to-right, top-to-bottom across the array).

Then run:

python findHomography.py

What it does:

1. Loads all 9 calibration images from CalibrationPic/
2. Uses the center image (index 4 = middle camera) as the reference frame
3. Detects 5×7 chessboard corners in each image via cv2.findChessboardCorners
4. Refines corner locations to sub-pixel accuracy with cv2.cornerSubPix
5. Computes a 3×3 homography matrix H for each camera mapping its corners to the reference
6. Saves each matrix as calibration/{index}.npy
7. Saves the rectified chessboard images to calibration/pic/ for visual verification

Parameter to adjust — cornersSize in findHomography.py (line ~36):

cornersSize = (5, 7)  # inner corner count of your chessboard

Note: The index ordering (0–8 vs row-col) must be consistent with how your hardware triggers cameras. Verify that the center image is indeed the reference before proceeding.

---

Step 2: Image Preprocessing

Place raw light field captures into pics/, organized into subdirectories by sample (each containing exactly 9 images). Then run:

python do_change.py

What it does:

1. Loads all homography matrices from calibration/*.npy
2. Walks recursively through pics/ (supports nested subdirectories)
3. For each sample folder containing 9 .bmp images:
   • Applies cv2.warpPerspective(image, H, image_size) with the matching calibration matrix
   • Saves aligned images to result/ with row-col naming (1-1.bmp, 1-2.bmp, ..., 3-3.bmp)

Parameters to adjust — in do_change.py:

a, b = (3, 3)          # Camera array dimensions (rows × cols)
folder = './pics/'      # Input directory
save_folder = './result/'  # Output directory

---

Step 3: Light Field Rendering (MATLAB GUI)

Open MATLAB and run:

Main4

The GUI window will appear with:

Control	Description
File path	Directory containing the 9 aligned images (from result/ subfolder)
Camera array rows (M)	Number of rows in your camera array (default: 3)
Camera array cols (N)	Number of columns in your camera array (default: 3)
Refocus parameter (L)	Initial focus slope — negative = focus in front, positive = focus behind
Run button	Compute refocused image at the given L value
Test button	Generate a full focal stack across all slider positions
Slider	Drag to smoothly sweep through focal depths
Click on image	Auto-select the refocus depth maximizing local Sobel edge sharpness

Workflow

1. Enter the full path to one sample's aligned images (e.g., C:\...\result\polyethylene+kieselguhr_1)
2. Set M and N to match your camera array dimensions
3. Click Run for a single refocused image, or Test to generate a focal stack
4. Use the slider to interactively sweep focal planes
5. Click anywhere on the image to auto-focus on that region using edge sharpness

The focal stack sequence is saved to {result_path}/重聚焦序列/ (Chinese for "refocus sequence") with filenames like -5.0.bmp, -4.9.bmp, ...

---

Algorithm Details

Homography-Based Calibration

The calibration finds a 3×3 projective transformation H for each camera such that:

p_ref ~ H · p_cam

where p_cam are the detected chessboard corners in camera i, and p_ref are the corresponding corners in the center (reference) camera.

OpenCV solves H using the Direct Linear Transform (DLT) algorithm with RANSAC outlier rejection (cv2.findHomography). This accounts for perspective distortion, camera tilt, and planar misalignment — producing a rectified light field where all sub-aperture images share a common epipolar geometry.

Shift-and-Sum Refocusing

Given a rectified light field L(u, v, x, y) where (u, v) indexes the camera and (x, y) indexes pixels, refocusing at depth alpha (slope) is performed as:

I_alpha(x, y) = sum_u sum_v L(u, v, x + alpha*u, y + alpha*v)

Each sub-aperture image is shifted by alpha * u along the horizontal and alpha * v along the vertical camera axis, then all shifted images are averaged. This is equivalent to focusing the synthetic aperture at depth 1/alpha.

In Main4.m, this is implemented as:

VVec = linspace(-0.5, 0.5, LFSize(1)) * Slope * LFSize(1);
UVec = linspace(-0.5, 0.5, LFSize(2)) * Slope * LFSize(2);
% Shift each sub-aperture with interpn, then sum

Click-to-focus works by evaluating the Sobel edge response in a local window around the click point across all refocused images in the focal stack, selecting the depth with maximum sharpness.

---

Sample Data

The repository includes calibration data and two sample scenes:

Scene	Description	Camera Count
CalibrationPic/	Chessboard (5×7 inner corners)	9
pics/聚乙烯颗粒加硅藻土1/	Polyethylene particles + diatomaceous earth, sample 1	9
pics/聚乙烯颗粒加硅藻土2/	Polyethylene particles + diatomaceous earth, sample 2	9

These scenes were captured with a 3×3 camera array for industrial particle analysis.

---

Configuration

File	Parameter	Default	Description
findHomography.py	cornersSize	(5, 7)	Chessboard inner corners (rows × cols)
do_change.py	a, b	(3, 3)	Camera array dimensions
do_change.py	folder	./pics/	Input raw images
do_change.py	save_folder	./result/	Output aligned images
Main4.m	Slider range	[-15, -5]	Refocus slope range
Main4.m	Window size	[0 0 1280 720]	GUI resolution

---

Troubleshooting

OpenCV cannot read images with Chinese file paths (Windows)

The project provides cv_imread() / cv_imwrite() wrappers using cv2.imdecode + np.fromfile to handle Unicode paths. If you add new image I/O code, use these wrappers instead of cv2.imread / cv2.imwrite.

Chessboard corners not detected

• Ensure the chessboard is fully visible and approximately planar in all 9 calibration images
• Increase lighting contrast — the algorithm needs clear black-white edges
• Verify cornersSize matches your chessboard inner corner count (not squares)
• Try adding cv2.CALIB_CB_ADAPTIVE_THRESH to the flags

Only some images are preprocessed

do_change.py skips folders where the image count is not equal to rows x cols (e.g., not equal to 9 for 3×3). Check if a sample folder has exactly the right number of BMP files.

MATLAB GUI displays black or shifted images

• M and N must exactly match your camera array dimensions
• Ensure the images in your result folder are properly aligned (run the calibration stage again if needed)
• If using a different camera layout, adjust the row-col naming convention in do_change.py

.npy files fail to load

• Confirm the calibration stage completed successfully (calibration/ should contain 0.npy through 8.npy)
• Verify numpy version compatibility

---

License

This project is shared for educational and research purposes. Please cite the repository if you use this code in published work.

---

Author: ofen1996
Email: 526083628@qq.com
Last Updated: 2026-08-13