Skip to content

Merge cma-rfx into main: count outcomes, bcf fixes, zero-part random effects - #15

Merged
dwachsmuth merged 39 commits into
mainfrom
merge/cma-rfx-into-main
Nov 12, 2026
Merged

dwachsmuth merged 39 commits into
mainfrom
merge/cma-rfx-into-main

Conversation

@dwachsmuth

Copy link
Copy Markdown

Brings the cma-rfx line into main so the fork has one line again. cma-rfx and main share the base bd5c5c4 (1 Nov); since then cma-rfx gained the count-outcome work and main gained the native forest writer, build receipts and the long-term-fork docs. This branch is cma-rfx at 19f70ab with origin/main (0345152) merged in.

What comes in from cma-rfx

Merge

  • Automatic for every code file (R/cpp11.R, src/cpp11.cpp auto-merged; main's changes are in the inference bindings and native writer, which the count work never touches).
  • One conflict, NASH-FORK.md: both sides appended sections after "Causal sampler correctness". Resolved by placing main's "nashtree identity and native forests (0.4.6.9010)" section first, then the cma-rfx sections in order (NB, ZINB, count BCF, bcf fixes, known upstream bugs, zero-part intercept, zero-part prior). No text interleaved.

Evidence

Merged tree installed into a scratch library, full suite with NOT_CRAN=true inside the package namespace (macOS arm64, R 4.6): FAIL 0, ERROR 0, WARN 13 (the usual upstream sampler notices), SKIP 2 (slow API-combination tests), PASS 2130. The bitwise fixtures from #8, #12, #13 and #14 all run and pass, so the Gaussian, probit, cloglog and NB paths are draw-for-draw unchanged against their references.

After merging

  • nash pins Curbcut/nashtree@ff3eb820 (main before Describe nashtree as the long-term Curbcut fork #7). A pin bump to the merge commit is a nash PR with its own library rebuilds; the fixtures say nash's fitting paths are unchanged, and the pin PR should confirm it on one real fit.
  • cma-rfx can be deleted once this is merged; upstream-main stays as the reference mirror.
  • Not run here: the native/ writer tests (cmake) and R CMD check.

🤖 Generated with Claude Code

dwachsmuth and others added 30 commits November 8, 2026 14:02
Records Gaussian BART (custom basis, two chains; intercept-only) and BCF
(intercept-only) random-effects fits from 0ac0e7d so that the coming
variance-weighted rfx conditional can be shown to leave the unweighted
path bitwise unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PG(1, z) is drawn exactly with Devroye's (2009) alternating-series
sampler for J*(1, z/2) as laid out by Polson, Scott and Windle (2013).
Integer b <= 50 sums exact draws; other b below 10,000 use the gamma
series truncated at 100 terms with the remainder replaced by its exact
mean; b >= 10,000 uses the normal approximation with exact moments.
Exposed to R as sample_polya_gamma_cpp(b, z, rng), vectorised over b and
z. Tests check means, variances and the Laplace transform against closed
forms, and that the exact and series paths agree in distribution.

No third-party implementation was consulted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Taken from bart() as of 0ac0e7d, before the count outcome is added, so
the existing outcome paths can be shown to stay bitwise unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a RandomEffectsDataset carries variance weights v_i, row i now
enters the working-parameter and group-parameter conditionals with
precision 1 / (sigma^2 v_i), i.e. X'WX and X'Wy with W = diag(1 / v_i).
Datasets without weights take the previous code path unchanged; a test
checks seeded BART and BCF random-effects draws bitwise against the
reference captured at 0ac0e7d. The variance-component conditional
depends only on the group parameters and needs no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OutcomeModel(outcome = 'count', link = 'log') fits y ~ NB(mu, r) with
log(mu) = offset + c + f(X) [+ random effects]. Each GFR and MCMC
iteration draws omega ~ PG(y + r, log(mu) - log(r)), passes the
pseudo-outcome (y - r) / (2 omega) net of the offset, c, -log(r) and the
current fits as the residual, and sets variance weights 1 / omega (times
any observation weights) on the forest and random-effects datasets, with
the global error variance fixed at 1. The forest, leaf-scale and rfx
samplers are unchanged.

The dispersion r is drawn from its exact full conditional given mu by
slice sampling on log(r) under a Gamma prior. The CRT-Gamma update
conditions on p = mu / (mu + r) and is not invariant for p(r | y, mu)
when the forest models log(mu); on simulated data it settled at 2.44
against an exact posterior mean of 2.05.

New offset_train / offset_test arguments (appended to the signature),
predict() scales 'rate' and 'response' and an offset argument, negative-
binomial posterior predictive draws, dispersion draws stored per sample
and carried through chain resets, warm starts, JSON save/load and the
combined loaders, print/summary/extractParameter support, and clear
refusals for variance forests, sample_sigma2_global = TRUE, offsets on
other outcomes and bcf().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery on simulated data (r in {2, 10}, with and without an offset and
a 20-group random intercept), bitwise checks of the Gaussian, probit,
cloglog and unweighted rfx paths against the 0ac0e7d references,
predict scales and offsets, posterior predictive draws, JSON round
trips (object, string, file, combined), multi-chain shapes, warm starts,
fixed dispersion, observation weights and input validation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Roxygen-generated pages for bart(), predict(), extractParameter(),
sampleBARTPosteriorPredictive() and OutcomeModel(); NEWS entry and
version 0.4.6.9011; NASH-FORK section on the model, the Gibbs scheme,
the Polya-Gamma sampler's provenance, the rfx weighting change, related
issues left unchanged and what is out of scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nflation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OutcomeModel('count', 'log') now carries family = 'negbin' by default and
accepts family = 'zinb'. Other outcome models keep their two-field object
and refuse a family.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With OutcomeModel('count', 'log', family = 'zinb') each iteration draws the
structural-zero indicators, samples a probit zero forest on them (its own
dataset, outcome, configuration and container, stored like the variance
forest), runs the Polya-Gamma count step on the non-structural rows only
(structural zeros get an infinite variance weight, so they drop out of the
leaf and random-effects sufficient statistics) and updates r over the
non-structural rows. The sampler starts from a constant-rate ZINB fit.

predict() gains the 'zero_probability' scale, 'response' = (1 - pi) mu and
the 'zero_forest' term; the posterior predictive includes the zero mass;
JSON (object, string, file, combined) carries the family and the zero
forest; print, summary and extractParameter report the zero part. Plain
negative-binomial draws are bitwise unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery in four scenarios with thresholds calibrated over 32 fits, exact
exclusion of infinite-weight rows from the forest and random-effects
samplers, the bitwise-unchanged plain NB path, predict() scales and terms,
the posterior predictive zero mass, JSON round trips, multiple chains,
warm starts and input validation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…counts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Polya-Gamma step turns the count part into the weighted Gaussian
regression bcf() already fits: prognostic forest, basis-regression
treatment forest (binary or continuous dose), propensity covariate,
adaptive coding, tau_0, random effects, GFR and multiple chains run on the
pseudo-outcome with variance weights 1 / omega. tau_0 and the coding
parameters, sampled in R, use the precision weights; tau_c is a log rate
ratio with a prior set by log_rate_ratio_prior_sd (default 0.5).
family = 'zinb' adds probit zero-part prognostic and treatment forests on
the structural-zero indicators, sharing the step of bart(). Variance
forests and a sampled global variance are refused.

predict.bcfmodel() gains the rate, response and zero_probability scales,
the zero-part terms and an offset; computeContrastBCFModel() and
sampleBCFPosteriorPredictive() handle counts; JSON carries the family,
dispersion draws and zero-part forests. Gaussian and probit draws are
bitwise unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery of a log rate ratio under confounded assignment (NB and ZINB) and
of the zero-part effect, with thresholds calibrated over 16 fits; Gaussian
and probit BCF draws bitwise against a reference; predict scales and terms,
contrasts, posterior predictive, JSON round trips, multiple chains, warm
starts, continuous dose, adaptive coding, weights and validation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Count outcomes in bcf() via Pólya-Gamma augmentation
…g fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With treatment_effect_forest_params$sample_sigma2_leaf = TRUE, each GFR
iteration drew a new treatment-forest leaf scale but then handed the
prognostic scale to the prognostic forest's config, so the treatment forest
kept its initial scale for every GFR sweep. Update the treatment config with
the treatment draw, as the MCMC loop already does. Fits that do not sample
the treatment leaf scale, or that run no GFR iterations, are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The C++ leaf statistics read observation_weights as variance multipliers
v_i (sum_w += 1 / v_i), but bcf() sampled tau_0 and the adaptive-coding
parameters in R from unweighted sums, so with weights the two halves of the
Gibbs sampler conditioned on different likelihoods and tau_0 drifted far from
the effect while the weighted forest offset it. Use precisions 1 / v_i in
Z'W r, Z'W Z and the s_tt / s_ty sums, through two helpers the count path
now shares (bcf_tau_0_posterior, bcf_coding_suff_stats). Without weights,
and for probit, the unweighted arithmetic is kept and draws are bitwise
unchanged. The count tests' weighted Gaussian reference is no longer
compared bitwise, since its draws change on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bugs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…random intercept

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With rfx_group_ids_train and family = "zinb", the probit zero part now
gets its own random-effects model, dataset and tracker on the same groups,
basis and priors as the count part. It is sampled on the zero part's
latent residual with unit variance after the zero forest, and enters pi
in the structural-zero draw, the latent step, zero_hat_train/test,
predict() (new term "zero_rfx"), the posterior predictive and the JSON
round trips (a second random-effects container). Unseen groups follow
random_effects_params$unseen_groups. general_params$zero_random_effects =
FALSE restores the previous model, draw for draw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zero part's prognostic and treatment forests get the same
random-effects treatment as in bart(): a separate model on the count
part's groups and basis, sampled on the zero part's latent residual with
unit variance, entering pi everywhere (structural-zero draw, latent step,
zero_hat_train/test, predict() term "zero_rfx", posterior predictive,
contrasts on the response and zero-probability scales, JSON). The count
part's random effects are unchanged. general_params$zero_random_effects =
FALSE turns it off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Those tests describe the zero part as it was; zero_random_effects = FALSE
keeps them on exactly that path, draw for draw. The new random intercept is
tested in its own file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery on a 60 x 30 tract panel with and without the zero-part
intercept (pi, response and intercept recovery in bart(); the confounded
zero-part effect in bcf()), the bitwise references, prediction terms and
scales, posterior predictive draws, unseen groups, JSON round trips,
chains, combined models and warm starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dwachsmuth and others added 9 commits November 8, 2026 19:10
Fix GFR treatment leaf scale and weighted tau_0 / adaptive coding in bcf()
Random intercept on the ZINB zero part
…m effects their own prior

Gaussian, probit and NB bart() / bcf() with group ids, cloglog bart(), and
default-prior ZINB bart() / bcf() with zero-part random effects (GFR and root
chain starts, warm starts), generated from a build of cma-rfx 1273f12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bart() and bcf() take zero_random_effects_params, with any of the six prior
keys of random_effects_params. Keys left out take the count part's values, so
the default list() keeps the shared prior draw for draw. The zero part's
starting values and priors are resolved by .zinbZeroRfxPrior() and used at
every place its RandomEffectsModel is set up or reset (initial set-up, GFR and
root chain starts, warm starts). The count part's code and the C++ sampler are
unchanged: the zero part already has its own model object.

Groups, basis and unseen-group policy stay shared and are refused here; a
non-empty list without family = 'zinb', group ids and zero_random_effects =
TRUE is an error. The roxygen documents the parameter expansion and the exact
half-Cauchy mapping (shape = scale = 1/2, working-parameter variance A^2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bitwise references for the unchanged paths; the count prior written out as the
zero prior reproduces the default draws; a tight zero prior holds the zero part
at every chain start while the count part stays free, and the reverse; a
half-Cauchy prior's scale against strong group effects; JSON round trips;
validation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A prior of its own for the ZINB zero-part random effects
@dwachsmuth
dwachsmuth merged commit 4ccb569 into main Nov 12, 2026
2 checks passed
@dwachsmuth
dwachsmuth deleted the merge/cma-rfx-into-main branch November 12, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant