Skip to content

Random intercept on the ZINB zero part - #13

Merged
dwachsmuth merged 8 commits into
cma-rfxfrom
claude/zinb-zero-rfx
Nov 9, 2026
Merged

dwachsmuth merged 8 commits into
cma-rfxfrom
claude/zinb-zero-rfx

Conversation

@dwachsmuth

Copy link
Copy Markdown

Gives the zero part of the zero-inflated negative-binomial family its own random intercept in bart() and bcf(). Draft against cma-rfx (84f5ccb). The branch was cut from cma-bcf-count (d8a0cb7), which cma-rfx contains with an identical tree now that #8–#10 are merged.

Model

With rfx_group_ids_train and family = "zinb":

  • bart(): pi_i = Phi(c0 + g(x_i) + w_i' v_k(i))
  • bcf(): pi_i = Phi(c0 + mu_z(x_i, pi_hat_i) + tau_z(x_i) z_i + w_i' v_k(i))

w_i is the count part's random-effects basis: a column of ones for model_spec = "intercept_only", or whatever spec or custom basis was supplied. v has the same groups and the same random_effects_params priors as the count part's random effects, but its own parameters. The count part is unchanged.

It is on by default whenever group ids are given. general_params$zero_random_effects = FALSE turns it off, in both functions.

Scheme

Each iteration does the following, in order:

  1. Draw s (structural zeros) with pi including v.
  2. Albert–Chib latent draw z ~ N(eta_zero, 1) truncated by s; the zero residual is z - c0 - forests - w'v.
  3. Sample the zero forest(s).
  4. New: zero_rfx_model$sample_random_effect() on that residual, with global_variance = 1. It uses its own RandomEffectsDataset with no variance weights (no Pólya-Gamma weights on this part) and its own tracker and RandomEffectSamples, reusing the probit rfx machinery.
  5. The count part's Pólya-Gamma step, forests, count rfx and r, as before.

Chain starts are handled like the count rfx:

  • GFR warm start: resetRandomEffectsModel/Tracker from the GFR draw.
  • Root start: rootReset*.
  • previous_model_json: reset from its zero-part samples, or a root reset with a warning when the previous model has none.

GFR draws are deleted with the rest.

v enters pi everywhere:

  • zero_hat_train / zero_hat_test are now the full index, so pnorm(zero_hat_*) are still draws of pi.
  • New outputs: zero_rfx_samples, zero_rfx_preds_train, zero_rfx_preds_test.
  • predict(): new term "zero_rfx", probit scale on every scale. y_hat on the "response" / "zero_probability" scales includes v, so it needs rfx_group_ids. "zero_forest" stays forests-only.
  • computeContrastBCFModel() on the count scales.
  • sampleBARTPosteriorPredictive() / sampleBCFPosteriorPredictive().
  • print().
  • JSON (string, file, object, combined, combined-string):
    • has_zero_rfx flag;
    • a second container, random_effect_container_1, with its own samples;
    • combining models with and without it is refused;
    • models saved without the field load as has_zero_rfx = FALSE.

Unseen groups:

  • bart(): follow random_effects_params$unseen_groups on both parts ("mean" gives the per-draw mean group effect).
  • bcf(): has no unseen-group policy, so the zero part refuses unseen labels, as the count part already did.

Recovery

Panel of 60 tracts × 30 rows, tract intercepts on both parts (count SD 0.4, zero-part SD 0.7 on the probit scale), smooth mu and pi, r = 5, 41–48% zeros. Calibration over 8 data sets, with and without the zero-part intercept (bart(): 10 GFR + 150 burn-in + 250 draws, 100 trees):

with without
bart() corr(posterior mean pi, truth) 0.882–0.927 0.535–0.677
bart() RMSE of pi 0.097–0.122 0.187–0.233
bart() response RMSE (with / without) ≤ 0.834 1
corr(zero-part tract intercepts, truth) 0.870–0.924 (bart()), 0.833–0.900 (bcf()) –
bcf() error of average zero-part effect (truth −0.5, treatment corr. 0.35–0.49 with the zero intercept) 0.076–0.296 0.556–0.824
bcf() error of average log rate ratio ≤ 0.107 ≤ 0.109

The intercept removes most of the confounding, but not all of it. With treatment drawn independently of the intercept, the zero-part effect error with the intercept is 0.01–0.2 (noise around 0), so the ~0.2 left under confounding is leakage through the partially pooled intercepts (posterior-mean intercept SD ≈ 0.6 vs 0.64–0.79 true, with 30 rows a tract and latent structural zeros). If that matters for the zoning model, a Mundlak term (the tract mean of z as a covariate) would be the usual fix.

tests/testthat/test-zinb-rfx.R uses the first calibration data set, with thresholds inside the calibrated ranges.

Mixing and timing

bcf() on the 60 × 30 recovery panel (treatment confounded with the zero intercept). 4 chains × 1,000 draws (no burn-in), 3 data sets, chains started from GFR (10 sweeps) or from the root. Split R-hat (posterior::rhat_basic) and bulk ESS of 4,000 draws; worst case over the 3 data sets:

quantity start with zero intercept without
ATE, response scale: split R-hat GFR / root ≤ 1.008 / ≤ 1.003 ≤ 1.005 / ≤ 1.004
ATE, response scale: bulk ESS GFR / root 566–660 / 468–642 520–851 / 700–720
average zero-part effect (probit): split R-hat GFR / root ≤ 1.035 / ≤ 1.026 ≤ 1.020 / ≤ 1.025
average zero-part effect (probit): bulk ESS GFR / root 134–183 / 133–192 173–212 / 178–243

No convergence problem in either direction. The cost is roughly 15–25% fewer effective draws for the ATE and the zero-part effect.

Posterior mean ATE (response scale): 1.28–1.35 with the intercept vs 0.79–0.96 without; truth 1.55–1.66. The zero-part effect is −0.27 to −0.39 with vs −0.00 to +0.28 without; truth −0.5.

Timing, n = 5,000 (100 tracts × 50 rows), default forest sizes, intercept-only rfx, 200 iterations from the root, 2 repetitions, ms per iteration:

NB ZINB, switch off ZINB, zero intercept
bart() 32.0 32.2 32.6
bcf() 65.2* 69.1–70.9 68.5–68.8

The zero-part random effects add about 1% in bart() and nothing measurable in bcf() (run-to-run noise is ±3 ms). *The first NB bcf() repetition, 90.8 ms, was a warm-up outlier.

Script: zinb-zero-rfx-mixing.R in the session notes folder. It uses the helpers in test-zinb-rfx.R.

What stays bitwise unchanged

Seeded references were captured on cma-bcf-count (d8a0cb7) before any edit: tests/testthat/fixtures/zinb-rfx-reference.rds and make-zinb-rfx-reference.R. They are checked with expect_identical in test-zinb-rfx.R:

  • (a) ZINB bart() without group ids;
  • (b) ZINB bart() with group ids and zero_random_effects = FALSE;
  • (c) ZINB bcf() with group ids and zero_random_effects = FALSE;
  • (d) NB bart() and bcf() with group ids.

Each covers draws, test-set predictions, predict scales, contrasts and posterior predictive draws, with GFR and two chains.

(b) and (c) are bitwise: the new sampler consumes no RNG draws and adds no arithmetic when off. All existing ZINB-with-groups tests in test-zinb.R / test-bcf-count.R now pass zero_random_effects = FALSE, so they keep testing exactly the old path.

The default behaviour of ZINB fits with group ids changes by design: they now include the zero-part intercept.

Tests

Full suite (NOT_CRAN=true):

  • Before (d8a0cb7): 1854 expectations, 1852 pass, 2 skip, 0 fail.
  • After: 1998 expectations, 1996 pass, 2 skip, 0 fail.

Every pre-existing file has the same count as before. The new test-zinb-rfx.R adds 144.

Interaction with #12

Draft #12 (claude/bcf-fixes, against cma-rfx) touches R/bcf.R (the tau_0 / adaptive-coding sums move into helpers, and the GFR leaf-scale line), test-bcf-count.R and NASH-FORK.md.

  • R/bcf.R and test-bcf-count.R: no shared lines; git merge-tree merges both cleanly.
  • NASH-FORK.md: conflicts, because both branches append a section at the end. Keep both sections.
  • Possible reference break: if Fix GFR treatment leaf scale and weighted tau_0 / adaptive coding in bcf() #12's helpers change the floating-point order of the count-path tau_0 / coding sums, the count-BCF reference cases here, (c) and the bcf() half of (d), would stop matching bitwise once both land, and should be regenerated from the merged base. I did not build the merged tree to check this, since the brief allowed one scratch library only.

Not verified / not done

  • bcf() unseen-group policy: not added. The count part's random effects were to stay as they are, so unseen labels still error on both parts.
  • Only model_spec = "intercept_only" is exercised for the zero part in tests. Random slopes and intercept_plus_treatment reuse the count basis and should work, but are untested on the zero part.
  • No separate prior for the zero-part random effects: it uses the count part's random_effects_params. Default IG(1, 1) on the variance, N(0, 1) working parameter.
  • extractParameter() / getRandomEffectSamples() have no zero-part entry; use model$zero_rfx_samples$extract_parameter_samples().
  • roxygen2 8.1.0 (the only version installed) cannot load this package's R6 classes, and the repo was generated with 7.3.3, so the four affected .Rd files were edited by hand to match the roxygen comments (tools::checkRd clean).
  • Only checked on aarch64-apple-darwin (the bitwise references skip on other platforms).

🤖 Generated with Claude Code

dwachsmuth and others added 7 commits November 8, 2026 18:32
…random intercept

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With rfx_group_ids_train and family = "zinb", the probit zero part now
gets its own random-effects model, dataset and tracker on the same groups,
basis and priors as the count part. It is sampled on the zero part's
latent residual with unit variance after the zero forest, and enters pi
in the structural-zero draw, the latent step, zero_hat_train/test,
predict() (new term "zero_rfx"), the posterior predictive and the JSON
round trips (a second random-effects container). Unseen groups follow
random_effects_params$unseen_groups. general_params$zero_random_effects =
FALSE restores the previous model, draw for draw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zero part's prognostic and treatment forests get the same
random-effects treatment as in bart(): a separate model on the count
part's groups and basis, sampled on the zero part's latent residual with
unit variance, entering pi everywhere (structural-zero draw, latent step,
zero_hat_train/test, predict() term "zero_rfx", posterior predictive,
contrasts on the response and zero-probability scales, JSON). The count
part's random effects are unchanged. general_params$zero_random_effects =
FALSE turns it off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Those tests describe the zero part as it was; zero_random_effects = FALSE
keeps them on exactly that path, draw for draw. The new random intercept is
tested in its own file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery on a 60 x 30 tract panel with and without the zero-part
intercept (pi, response and intercept recovery in bart(); the confounded
zero-part effect in bcf()), the bitwise references, prediction terms and
scales, posterior predictive draws, unseen groups, JSON round trips,
chains, combined models and warm starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dwachsmuth
dwachsmuth marked this pull request as ready for review November 9, 2026 00:10
@dwachsmuth
dwachsmuth merged commit 1273f12 into cma-rfx Nov 9, 2026
2 checks passed
@dwachsmuth
dwachsmuth deleted the claude/zinb-zero-rfx branch November 9, 2026 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant