Grow with the Flow: 4D Reconstruction of Growing Plants with Gaussian Flow Fields

TL;DR: 4D reconstruction of plant growth from multi-view timelapse measurements using neural ODE.

Weihan Luo, Lily Goli, Sherwin Bahmani, Felix Taubner, Andrea Tagliasacchi, David Lindell

 

Installation

1. Clone the repository

git clone https://github.com/weihan1/growflow.git
cd growflow/

2. Create a new conda environment (make sure miniconda3 is installed beforehand). We tested on python version 3.10.8

conda create -yn grow_env python=3.10.8
conda activate grow_env

3. Additionally please install PyTorch from (https://pytorch.org/get-started/previous-versions/).
4. Install the first set of requirements file with

pip install -r requirements.txt

5. Install the second set with

pip install -r requirements_no_iso.txt --no-build-isolation

Dataset and checkpoints

To download the dataset, run python -m scripts.download_data from the repository root.

The dataset structure is

├── captured
│   ├── pi_corn_full_subset4
│   ├── pi_paperwhite_full_subset4
│   └── pi_rose
└── synthetic
    ├── clematis_subsample_6
    ├── plant_1_subsample_6
    ├── plant_2_subsample_6
    ├── plant_3_subsample_6
    ├── plant_4_subsample_6
    ├── plant_5_subsample_6
    └── tulip_subsample_6

NOTE: the synthetic datasets are already subsampled with interpolation factor 6, whereas the captured datasets are not subsampled, so you need to manually set the subsample_factor argument (see the boundary reconstruction and global optimization stage).

To download the checkpoints, run python -m scripts.download_checkpoints from the repository root. The checkpoint structure is

├── ckpts
│   ├── gaussian_ckpt_29999_t0.pt
│   └── neural_ode_29999.pt
└── fixed_pc_traj
    └── full_traj_{some_number}.npy

neural_ode_29999.pt is the final neural ODE checkpoint, whereas gaussian_ckpt_29999_t0.pt and full_traj_{some_number}.npy are the cached Gaussians that you can use to directly train the neural ODE in the global optimization stage.

Usage

Training

Our model is trained in 3 stages (see sect. 3.3 of paper), the static reconstruction stage, the boundary reconstruction stage, and the global optimization stage.

Static reconstruction stage

Synthetic:

python -m commands.train synthetic default --data-dir <your_data_dir>

Captured:

python -m commands.train captured default --data-dir <your_data_dir>

Boundary reconstruction stage

Synthetic:

python -m commands.trajectory synthetic default --data-dir <your_data_dir> --static-ckpt <your_ckpt_from_static_stage> --no-adjoint

Captured:

python -m commands.trajectory captured default --data-dir <your_data_dir> --static-ckpt <your_ckpt_from_static_stage> --no-adjoint --subsample-factor <desired_subsample_factor>

NOTE: In the paper, for the rose scene, desired_subsample_factor=17 and for the corn scene, desired_subsample_factor=10. You can also choose your own subsample_factor, however, if it's not a divisor of the total number of timesteps, you need to add the --include-end flag. For the paperwhite scene, use the flag --encoding freq throughout.

Global optimization stage

Synthetic:

python -m commands.train synthetic default --data-dir <your_data_dir> --static-ckpt <your_ckpt_from_static_stage> --full-trajectory-path <your_ckpt_from_boundary_stage> --rtol 1e-5 --atol 1e-6

Captured:

python -m commands.train captured default --data-dir <your_data_dir> --static-ckpt <your_ckpt_from_last_stage> --unscaled-encoder-lr-init 5e-4 --subsample-factor <desired_subsample_factor> --full-trajectory-path <your_ckpt_from_boundary_stage>

Evaluation

Synthetic:

python -m commands.render synthetic --data-dir <your_data_dir> --dynamic-ckpt <your_final_checkpoint_from_global>

Captured:

python -m commands.render captured --data-dir <your_data_dir> --dynamic-ckpt <your_final_checkpoint_from_global>

Use --ground-truth with either render command to export ground-truth views instead. Training and trajectory commands also accept mcmc in place of default.

Optional color flow

By default, the Neural ODE evolves Gaussian position, rotation, and scale while keeping color and opacity fixed. To also learn time-varying color, add --learn-shs --sh-degree <degree> to boundary reconstruction, global optimization, and rendering. SH degrees 0 through 4 are supported. Both synthetic and captured commands accept these flags.

Start from a static checkpoint trained with the same SH degree; an existing matching checkpoint can be reused without static retraining. For example, with SH3:

python -m commands.trajectory synthetic default --data-dir <your_data_dir> --static-ckpt <your_SH3_static_ckpt> --no-adjoint --learn-shs --sh-degree 3

Keep the same resolution and background options as static training. This mode appends all 3 * (degree + 1)^2 spherical-harmonic coefficients to each Gaussian's ODE state: 13 total values for SH0, or 58 for SH3. Higher orders model view-dependent appearance as well as its change over time, and increase solver state/cache memory. Gaussian IDs stay persistent, opacity stays fixed, and every color derivative starts at zero. It does not introduce Gaussian birth/death or time-dependent visibility masks.

Boundary caches and dynamic checkpoints are mode-specific: regenerate the boundary trajectory when switching color flow on or off or changing SH degree, and use matching flags for global training and inference. Appearance optimization is not supported in color-flow mode. Omitting --learn-shs keeps the original geometry-only model and checkpoint format; existing SH0 color-flow checkpoints remain compatible with --learn-shs --sh-degree 0.

Inspect camera poses

The Viser viewer shows each camera's mature image in a side panel when its wireframe frustum is clicked. It also displays the mature synthetic mesh (or the latest COLMAP point cloud for captured scenes):

python -m scripts.view_camera_poses synthetic --data-dir data/synthetic/clematis_transparent_final_small_vase_70_timesteps_subsample_6
python -m scripts.view_camera_poses captured --data-dir data/captured/pi_paperwhite_full_subset4 --timestep 0

Open the URL printed by the script. Use the Plant dropdown to switch scenes without restarting; the viewer discovers sibling scenes automatically, or you can pass --data-dir data/synthetic (or data/captured) to start from the dataset root. Click a frustum to jump to its viewpoint and compare its image in the side panel. Blue frustums are training cameras and orange frustums are test cameras. Synthetic datasets support --split train|test|both; captured --timestep 0 selects the latest date, and --test-every defaults to 8. Captured datasets do not ship a mesh; use --mesh <aligned_mesh.ply> if you have one in COLMAP coordinates. Use --host and --port when viewing through a remote node or port forward.

Metrics

NOTE: You can only run the metrics code after running the eval code.

Synthetic:

python -m commands.metrics_synthetic --skip-dynamic3dgs --skip-4dgs --skip-4dgaussians

Captured:

python -m commands.metrics_captured --skip-dynamic3dgs --skip-4dgs --skip-4dgaussians

The results will be saved in ./growflow/final_results_{scene}.

Tips for training on custom data

• Use bounding box. For all of our experiments, we hand-craft custom bounding boxes to constrain the neural ODE to only learn the the flow field of the foreground Gaussians. On your custom data, simply add the bounds of a bounding box that roughly includes the foreground plant.
• Decrease subsample factor. For the real scenes, decreasing the temporal stride --subsample-factor and the image resolution --data-factor will lead to faster training and overall better reconstruction.
• Swap to MLP. For all experiments, we parametrize the neural ODE using an Hexplane as it performs slightly better in the synthetic scenes (see ablation studies in paper). However, on some real scenes, we find that using an MLP with fourier-encoding performs better. You can swap the representation in the boundary reconstruction stage and global optimization stage by using the flag --encoding freq.

Reproducing numbers

Download the dataset and checkpoints, then run python -m scripts.render_all to get the renderings and then run the metrics code. Run all commands from the repository root.

Credits

This code is built on top of gsplat and torchdiffeq. Thanks to the maintainers for their contribution to the community!

Citing

If you find growflow helpful, please consider citing:

@article{luo2026grow,
  title={Grow with the Flow: 4D Reconstruction of Growing Plants with Gaussian Flow Fields},
  author={Luo, Weihan and Goli, Lily and Bahmani, Sherwin and Taubner, Felix and Tagliasacchi, Andrea and Lindell, David B},
  journal={arXiv preprint arXiv:2602.08958},
  year={2026}
}