Repository navigation
Random intercept on the ZINB zero part - #13
Merged
Merged
Conversation
…random intercept Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With rfx_group_ids_train and family = "zinb", the probit zero part now gets its own random-effects model, dataset and tracker on the same groups, basis and priors as the count part. It is sampled on the zero part's latent residual with unit variance after the zero forest, and enters pi in the structural-zero draw, the latent step, zero_hat_train/test, predict() (new term "zero_rfx"), the posterior predictive and the JSON round trips (a second random-effects container). Unseen groups follow random_effects_params$unseen_groups. general_params$zero_random_effects = FALSE restores the previous model, draw for draw. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zero part's prognostic and treatment forests get the same random-effects treatment as in bart(): a separate model on the count part's groups and basis, sampled on the zero part's latent residual with unit variance, entering pi everywhere (structural-zero draw, latent step, zero_hat_train/test, predict() term "zero_rfx", posterior predictive, contrasts on the response and zero-probability scales, JSON). The count part's random effects are unchanged. general_params$zero_random_effects = FALSE turns it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Those tests describe the zero part as it was; zero_random_effects = FALSE keeps them on exactly that path, draw for draw. The new random intercept is tested in its own file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery on a 60 x 30 tract panel with and without the zero-part intercept (pi, response and intercept recovery in bart(); the confounded zero-part effect in bcf()), the bitwise references, prediction terms and scales, posterior predictive draws, unseen groups, JSON round trips, chains, combined models and warm starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dwachsmuth
marked this pull request as ready for review
November 9, 2026 00:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gives the zero part of the zero-inflated negative-binomial family its own random intercept in
bart()andbcf(). Draft againstcma-rfx(84f5ccb). The branch was cut fromcma-bcf-count(d8a0cb7), whichcma-rfxcontains with an identical tree now that #8–#10 are merged.Model
With
rfx_group_ids_trainandfamily = "zinb":bart():pi_i = Phi(c0 + g(x_i) + w_i' v_k(i))bcf():pi_i = Phi(c0 + mu_z(x_i, pi_hat_i) + tau_z(x_i) z_i + w_i' v_k(i))w_iis the count part's random-effects basis: a column of ones formodel_spec = "intercept_only", or whatever spec or custom basis was supplied.vhas the same groups and the samerandom_effects_paramspriors as the count part's random effects, but its own parameters. The count part is unchanged.It is on by default whenever group ids are given.
general_params$zero_random_effects = FALSEturns it off, in both functions.Scheme
Each iteration does the following, in order:
s(structural zeros) withpiincludingv.z ~ N(eta_zero, 1)truncated bys; the zero residual isz - c0 - forests - w'v.zero_rfx_model$sample_random_effect()on that residual, withglobal_variance = 1. It uses its ownRandomEffectsDatasetwith no variance weights (no Pólya-Gamma weights on this part) and its own tracker andRandomEffectSamples, reusing the probit rfx machinery.r, as before.Chain starts are handled like the count rfx:
resetRandomEffectsModel/Trackerfrom the GFR draw.rootReset*.previous_model_json: reset from its zero-part samples, or a root reset with a warning when the previous model has none.GFR draws are deleted with the rest.
venterspieverywhere:zero_hat_train/zero_hat_testare now the full index, sopnorm(zero_hat_*)are still draws ofpi.zero_rfx_samples,zero_rfx_preds_train,zero_rfx_preds_test.predict(): new term"zero_rfx", probit scale on every scale.y_haton the"response"/"zero_probability"scales includesv, so it needsrfx_group_ids."zero_forest"stays forests-only.computeContrastBCFModel()on the count scales.sampleBARTPosteriorPredictive()/sampleBCFPosteriorPredictive().print().has_zero_rfxflag;random_effect_container_1, with its own samples;has_zero_rfx = FALSE.Unseen groups:
bart(): followrandom_effects_params$unseen_groupson both parts ("mean" gives the per-draw mean group effect).bcf(): has no unseen-group policy, so the zero part refuses unseen labels, as the count part already did.Recovery
Panel of 60 tracts × 30 rows, tract intercepts on both parts (count SD 0.4, zero-part SD 0.7 on the probit scale), smooth
muandpi,r = 5, 41–48% zeros. Calibration over 8 data sets, with and without the zero-part intercept (bart(): 10 GFR + 150 burn-in + 250 draws, 100 trees):bart()corr(posterior mean pi, truth)bart()RMSE of pibart()response RMSE (with / without)bart()), 0.833–0.900 (bcf())bcf()error of average zero-part effect (truth −0.5, treatment corr. 0.35–0.49 with the zero intercept)bcf()error of average log rate ratioThe intercept removes most of the confounding, but not all of it. With treatment drawn independently of the intercept, the zero-part effect error with the intercept is 0.01–0.2 (noise around 0), so the ~0.2 left under confounding is leakage through the partially pooled intercepts (posterior-mean intercept SD ≈ 0.6 vs 0.64–0.79 true, with 30 rows a tract and latent structural zeros). If that matters for the zoning model, a Mundlak term (the tract mean of
zas a covariate) would be the usual fix.tests/testthat/test-zinb-rfx.Ruses the first calibration data set, with thresholds inside the calibrated ranges.Mixing and timing
bcf()on the 60 × 30 recovery panel (treatment confounded with the zero intercept). 4 chains × 1,000 draws (no burn-in), 3 data sets, chains started from GFR (10 sweeps) or from the root. Split R-hat (posterior::rhat_basic) and bulk ESS of 4,000 draws; worst case over the 3 data sets:No convergence problem in either direction. The cost is roughly 15–25% fewer effective draws for the ATE and the zero-part effect.
Posterior mean ATE (response scale): 1.28–1.35 with the intercept vs 0.79–0.96 without; truth 1.55–1.66. The zero-part effect is −0.27 to −0.39 with vs −0.00 to +0.28 without; truth −0.5.
Timing, n = 5,000 (100 tracts × 50 rows), default forest sizes, intercept-only rfx, 200 iterations from the root, 2 repetitions, ms per iteration:
bart()bcf()The zero-part random effects add about 1% in
bart()and nothing measurable inbcf()(run-to-run noise is ±3 ms). *The first NBbcf()repetition, 90.8 ms, was a warm-up outlier.Script:
zinb-zero-rfx-mixing.Rin the session notes folder. It uses the helpers intest-zinb-rfx.R.What stays bitwise unchanged
Seeded references were captured on
cma-bcf-count(d8a0cb7) before any edit:tests/testthat/fixtures/zinb-rfx-reference.rdsandmake-zinb-rfx-reference.R. They are checked withexpect_identicalintest-zinb-rfx.R:bart()without group ids;bart()with group ids andzero_random_effects = FALSE;bcf()with group ids andzero_random_effects = FALSE;bart()andbcf()with group ids.Each covers draws, test-set predictions, predict scales, contrasts and posterior predictive draws, with GFR and two chains.
(b) and (c) are bitwise: the new sampler consumes no RNG draws and adds no arithmetic when off. All existing ZINB-with-groups tests in
test-zinb.R/test-bcf-count.Rnow passzero_random_effects = FALSE, so they keep testing exactly the old path.The default behaviour of ZINB fits with group ids changes by design: they now include the zero-part intercept.
Tests
Full suite (
NOT_CRAN=true):Every pre-existing file has the same count as before. The new
test-zinb-rfx.Radds 144.Interaction with #12
Draft #12 (
claude/bcf-fixes, againstcma-rfx) touchesR/bcf.R(thetau_0/ adaptive-coding sums move into helpers, and the GFR leaf-scale line),test-bcf-count.RandNASH-FORK.md.R/bcf.Randtest-bcf-count.R: no shared lines;git merge-treemerges both cleanly.NASH-FORK.md: conflicts, because both branches append a section at the end. Keep both sections.tau_0/ coding sums, the count-BCF reference cases here, (c) and thebcf()half of (d), would stop matching bitwise once both land, and should be regenerated from the merged base. I did not build the merged tree to check this, since the brief allowed one scratch library only.Not verified / not done
bcf()unseen-group policy: not added. The count part's random effects were to stay as they are, so unseen labels still error on both parts.model_spec = "intercept_only"is exercised for the zero part in tests. Random slopes andintercept_plus_treatmentreuse the count basis and should work, but are untested on the zero part.random_effects_params. DefaultIG(1, 1)on the variance,N(0, 1)working parameter.extractParameter()/getRandomEffectSamples()have no zero-part entry; usemodel$zero_rfx_samples$extract_parameter_samples()..Rdfiles were edited by hand to match the roxygen comments (tools::checkRdclean).🤖 Generated with Claude Code