Repository navigation
Merge cma-rfx into main: count outcomes, bcf fixes, zero-part random effects - #15
Merged
Merged
Conversation
Records Gaussian BART (custom basis, two chains; intercept-only) and BCF (intercept-only) random-effects fits from 0ac0e7d so that the coming variance-weighted rfx conditional can be shown to leave the unweighted path bitwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PG(1, z) is drawn exactly with Devroye's (2009) alternating-series sampler for J*(1, z/2) as laid out by Polson, Scott and Windle (2013). Integer b <= 50 sums exact draws; other b below 10,000 use the gamma series truncated at 100 terms with the remainder replaced by its exact mean; b >= 10,000 uses the normal approximation with exact moments. Exposed to R as sample_polya_gamma_cpp(b, z, rng), vectorised over b and z. Tests check means, variances and the Laplace transform against closed forms, and that the exact and series paths agree in distribution. No third-party implementation was consulted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Taken from bart() as of 0ac0e7d, before the count outcome is added, so the existing outcome paths can be shown to stay bitwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a RandomEffectsDataset carries variance weights v_i, row i now enters the working-parameter and group-parameter conditionals with precision 1 / (sigma^2 v_i), i.e. X'WX and X'Wy with W = diag(1 / v_i). Datasets without weights take the previous code path unchanged; a test checks seeded BART and BCF random-effects draws bitwise against the reference captured at 0ac0e7d. The variance-component conditional depends only on the group parameters and needs no change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OutcomeModel(outcome = 'count', link = 'log') fits y ~ NB(mu, r) with log(mu) = offset + c + f(X) [+ random effects]. Each GFR and MCMC iteration draws omega ~ PG(y + r, log(mu) - log(r)), passes the pseudo-outcome (y - r) / (2 omega) net of the offset, c, -log(r) and the current fits as the residual, and sets variance weights 1 / omega (times any observation weights) on the forest and random-effects datasets, with the global error variance fixed at 1. The forest, leaf-scale and rfx samplers are unchanged. The dispersion r is drawn from its exact full conditional given mu by slice sampling on log(r) under a Gamma prior. The CRT-Gamma update conditions on p = mu / (mu + r) and is not invariant for p(r | y, mu) when the forest models log(mu); on simulated data it settled at 2.44 against an exact posterior mean of 2.05. New offset_train / offset_test arguments (appended to the signature), predict() scales 'rate' and 'response' and an offset argument, negative- binomial posterior predictive draws, dispersion draws stored per sample and carried through chain resets, warm starts, JSON save/load and the combined loaders, print/summary/extractParameter support, and clear refusals for variance forests, sample_sigma2_global = TRUE, offsets on other outcomes and bcf(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery on simulated data (r in {2, 10}, with and without an offset and
a 20-group random intercept), bitwise checks of the Gaussian, probit,
cloglog and unweighted rfx paths against the 0ac0e7d references,
predict scales and offsets, posterior predictive draws, JSON round
trips (object, string, file, combined), multi-chain shapes, warm starts,
fixed dispersion, observation weights and input validation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Roxygen-generated pages for bart(), predict(), extractParameter(), sampleBARTPosteriorPredictive() and OutcomeModel(); NEWS entry and version 0.4.6.9011; NASH-FORK section on the model, the Gibbs scheme, the Polya-Gamma sampler's provenance, the rfx weighting change, related issues left unchanged and what is out of scope. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nflation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OutcomeModel('count', 'log') now carries family = 'negbin' by default and
accepts family = 'zinb'. Other outcome models keep their two-field object
and refuse a family.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With OutcomeModel('count', 'log', family = 'zinb') each iteration draws the
structural-zero indicators, samples a probit zero forest on them (its own
dataset, outcome, configuration and container, stored like the variance
forest), runs the Polya-Gamma count step on the non-structural rows only
(structural zeros get an infinite variance weight, so they drop out of the
leaf and random-effects sufficient statistics) and updates r over the
non-structural rows. The sampler starts from a constant-rate ZINB fit.
predict() gains the 'zero_probability' scale, 'response' = (1 - pi) mu and
the 'zero_forest' term; the posterior predictive includes the zero mass;
JSON (object, string, file, combined) carries the family and the zero
forest; print, summary and extractParameter report the zero part. Plain
negative-binomial draws are bitwise unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery in four scenarios with thresholds calibrated over 32 fits, exact exclusion of infinite-weight rows from the forest and random-effects samplers, the bitwise-unchanged plain NB path, predict() scales and terms, the posterior predictive zero mass, JSON round trips, multiple chains, warm starts and input validation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…counts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Polya-Gamma step turns the count part into the weighted Gaussian regression bcf() already fits: prognostic forest, basis-regression treatment forest (binary or continuous dose), propensity covariate, adaptive coding, tau_0, random effects, GFR and multiple chains run on the pseudo-outcome with variance weights 1 / omega. tau_0 and the coding parameters, sampled in R, use the precision weights; tau_c is a log rate ratio with a prior set by log_rate_ratio_prior_sd (default 0.5). family = 'zinb' adds probit zero-part prognostic and treatment forests on the structural-zero indicators, sharing the step of bart(). Variance forests and a sampled global variance are refused. predict.bcfmodel() gains the rate, response and zero_probability scales, the zero-part terms and an offset; computeContrastBCFModel() and sampleBCFPosteriorPredictive() handle counts; JSON carries the family, dispersion draws and zero-part forests. Gaussian and probit draws are bitwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery of a log rate ratio under confounded assignment (NB and ZINB) and of the zero-part effect, with thresholds calibrated over 16 fits; Gaussian and probit BCF draws bitwise against a reference; predict scales and terms, contrasts, posterior predictive, JSON round trips, multiple chains, warm starts, continuous dose, adaptive coding, weights and validation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Count outcomes in bcf() via Pólya-Gamma augmentation
…g fixes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With treatment_effect_forest_params$sample_sigma2_leaf = TRUE, each GFR iteration drew a new treatment-forest leaf scale but then handed the prognostic scale to the prognostic forest's config, so the treatment forest kept its initial scale for every GFR sweep. Update the treatment config with the treatment draw, as the MCMC loop already does. Fits that do not sample the treatment leaf scale, or that run no GFR iterations, are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The C++ leaf statistics read observation_weights as variance multipliers v_i (sum_w += 1 / v_i), but bcf() sampled tau_0 and the adaptive-coding parameters in R from unweighted sums, so with weights the two halves of the Gibbs sampler conditioned on different likelihoods and tau_0 drifted far from the effect while the weighted forest offset it. Use precisions 1 / v_i in Z'W r, Z'W Z and the s_tt / s_ty sums, through two helpers the count path now shares (bcf_tau_0_posterior, bcf_coding_suff_stats). Without weights, and for probit, the unweighted arithmetic is kept and draws are bitwise unchanged. The count tests' weighted Gaussian reference is no longer compared bitwise, since its draws change on purpose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bugs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…random intercept Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With rfx_group_ids_train and family = "zinb", the probit zero part now gets its own random-effects model, dataset and tracker on the same groups, basis and priors as the count part. It is sampled on the zero part's latent residual with unit variance after the zero forest, and enters pi in the structural-zero draw, the latent step, zero_hat_train/test, predict() (new term "zero_rfx"), the posterior predictive and the JSON round trips (a second random-effects container). Unseen groups follow random_effects_params$unseen_groups. general_params$zero_random_effects = FALSE restores the previous model, draw for draw. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zero part's prognostic and treatment forests get the same random-effects treatment as in bart(): a separate model on the count part's groups and basis, sampled on the zero part's latent residual with unit variance, entering pi everywhere (structural-zero draw, latent step, zero_hat_train/test, predict() term "zero_rfx", posterior predictive, contrasts on the response and zero-probability scales, JSON). The count part's random effects are unchanged. general_params$zero_random_effects = FALSE turns it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Those tests describe the zero part as it was; zero_random_effects = FALSE keeps them on exactly that path, draw for draw. The new random intercept is tested in its own file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery on a 60 x 30 tract panel with and without the zero-part intercept (pi, response and intercept recovery in bart(); the confounded zero-part effect in bcf()), the bitwise references, prediction terms and scales, posterior predictive draws, unseen groups, JSON round trips, chains, combined models and warm starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix GFR treatment leaf scale and weighted tau_0 / adaptive coding in bcf()
Random intercept on the ZINB zero part
…m effects their own prior Gaussian, probit and NB bart() / bcf() with group ids, cloglog bart(), and default-prior ZINB bart() / bcf() with zero-part random effects (GFR and root chain starts, warm starts), generated from a build of cma-rfx 1273f12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bart() and bcf() take zero_random_effects_params, with any of the six prior keys of random_effects_params. Keys left out take the count part's values, so the default list() keeps the shared prior draw for draw. The zero part's starting values and priors are resolved by .zinbZeroRfxPrior() and used at every place its RandomEffectsModel is set up or reset (initial set-up, GFR and root chain starts, warm starts). The count part's code and the C++ sampler are unchanged: the zero part already has its own model object. Groups, basis and unseen-group policy stay shared and are refused here; a non-empty list without family = 'zinb', group ids and zero_random_effects = TRUE is an error. The roxygen documents the parameter expansion and the exact half-Cauchy mapping (shape = scale = 1/2, working-parameter variance A^2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bitwise references for the unchanged paths; the count prior written out as the zero prior reproduces the default draws; a tight zero prior holds the zero part at every chain start while the count part stays free, and the reverse; a half-Cauchy prior's scale against strong group effects; JSON round trips; validation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A prior of its own for the ZINB zero-part random effects
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the
cma-rfxline intomainso the fork has one line again.cma-rfxandmainshare the base bd5c5c4 (1 Nov); since thencma-rfxgained the count-outcome work andmaingained the native forest writer, build receipts and the long-term-fork docs. This branch iscma-rfxat 19f70ab withorigin/main(0345152) merged in.What comes in from
cma-rfxOutcomeModel("count", "log"), offsets, PG sampler written from the published algorithms, random-effects conditional honours variance weights, slice-sampled dispersion).bart()(family = "zinb", probit zero forest).bcf()(four forests for ZINB, log-rate-ratio treatment prior).bcf(): the GFR loop now updates the treatment forest's leaf scale, and the Gaussiantau_0/ adaptive-coding updates honourobservation_weights.Merge
R/cpp11.R,src/cpp11.cppauto-merged;main's changes are in the inference bindings and native writer, which the count work never touches).NASH-FORK.md: both sides appended sections after "Causal sampler correctness". Resolved by placingmain's "nashtree identity and native forests (0.4.6.9010)" section first, then thecma-rfxsections in order (NB, ZINB, count BCF, bcf fixes, known upstream bugs, zero-part intercept, zero-part prior). No text interleaved.Evidence
Merged tree installed into a scratch library, full suite with
NOT_CRAN=trueinside the package namespace (macOS arm64, R 4.6): FAIL 0, ERROR 0, WARN 13 (the usual upstream sampler notices), SKIP 2 (slow API-combination tests), PASS 2130. The bitwise fixtures from #8, #12, #13 and #14 all run and pass, so the Gaussian, probit, cloglog and NB paths are draw-for-draw unchanged against their references.After merging
Curbcut/nashtree@ff3eb820(mainbefore Describe nashtree as the long-term Curbcut fork #7). A pin bump to the merge commit is a nash PR with its own library rebuilds; the fixtures say nash's fitting paths are unchanged, and the pin PR should confirm it on one real fit.cma-rfxcan be deleted once this is merged;upstream-mainstays as the reference mirror.native/writer tests (cmake) andR CMD check.🤖 Generated with Claude Code