Skip to content

ENH: Add principal component flow regularizer to MultiscaleFlow - #17

Merged
ntustison merged 1 commit into
mainfrom
pcf_reg
Nov 12, 2026
Merged

ntustison merged 1 commit into
mainfrom
pcf_reg

Conversation

@ntustison

Copy link
Copy Markdown
Member

Add MultiscaleFlow.pcf_regularizer(x, num_dims=1, create_graph=None), the penalty of principal component flows (Cunningham, Cobb and Jha, ICML 2022) with single-coordinate partitions:

-I_hat(x) = sum_k log||dz_k/dx|| - log|det dz/dx| >= 0,

which is zero when the rows of the encoder Jacobian are orthogonal (Hadamard's inequality), i.e. when the flow's contours are its principal manifolds. Train with
loss = -log p(x) + alpha * pcf_regularizer(x).

The sum over latent coordinates is estimated without bias by stratified sampling: num_dims coordinates per level and per sample, each costing one vector-Jacobian product through the encoder, weighted by the level size. A level is summed exactly when num_dims >= its size. The graph is kept for double backward when the model is training.

Tests (tests/pcf_test.py): exact mode matches the full-Jacobian value and is non-negative; the stochastic estimate is unbiased; the penalty is zero for an orthogonal encoder; gradients are finite with and without gradient checkpointing; num_dims < 1 is rejected.

Add MultiscaleFlow.pcf_regularizer(x, num_dims=1, create_graph=None),
the penalty of principal component flows (Cunningham, Cobb and Jha,
ICML 2022) with single-coordinate partitions:

    -I_hat(x) = sum_k log||dz_k/dx|| - log|det dz/dx| >= 0,

which is zero when the rows of the encoder Jacobian are orthogonal
(Hadamard's inequality), i.e. when the flow's contours are its
principal manifolds. Train with
loss = -log p(x) + alpha * pcf_regularizer(x).

The sum over latent coordinates is estimated without bias by stratified
sampling: num_dims coordinates per level and per sample, each costing
one vector-Jacobian product through the encoder, weighted by the level
size. A level is summed exactly when num_dims >= its size. The graph is
kept for double backward when the model is training.

Tests (tests/pcf_test.py): exact mode matches the full-Jacobian value
and is non-negative; the stochastic estimate is unbiased; the penalty
is zero for an orthogonal encoder; gradients are finite with and
without gradient checkpointing; num_dims < 1 is rejected.
@ntustison
ntustison merged commit fa45dd1 into main Nov 12, 2026
4 checks passed
@ntustison
ntustison deleted the pcf_reg branch November 12, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant