diff --git a/NASH-FORK.md b/NASH-FORK.md index da403631..188e1514 100644 --- a/NASH-FORK.md +++ b/NASH-FORK.md @@ -256,3 +256,44 @@ Pre-existing in upstream code and not fixed here: - `observation_weights` of zero are infinite precision under the C++ convention: any zero weight makes Gaussian `bcf()` return `NaN` draws, although the input checks describe all-zero weights as prior sampling. + +## Random intercept on the ZINB zero part (0.4.6.9011) + +Branch `claude/zinb-zero-rfx`, on `cma-bcf-count` (d8a0cb7b), merged with +`cma-rfx` 2eab02bb (the #12 fixes). With +`rfx_group_ids_train` and `family = "zinb"`, the zero part of `bart()` and +`bcf()` gets its own random effects: `pi_i = Phi(c0 + g(x_i) [+ tau_z(x_i) z_i] ++ w_i' v_k(i))` on the count part's groups, basis and `random_effects_params` +priors, with separate parameters `v`. Default on; +`general_params$zero_random_effects = FALSE` gives the previous model draw for draw. + +**Scheme**: a second `RandomEffectsModel` / `RandomEffectsDataset` (no +variance weights) / `RandomEffectsTracker` / `RandomEffectSamples`, sampled +right after the zero forest(s) on the zero part's latent residual +`z - c0 - forests - w'v` with `global_variance = 1`, the way probit `bart()` +samples its rfx. `v` enters the structural-zero draw and the truncated-normal +mean. Chain resets, warm starts and GFR deletion mirror the count rfx. Output: +`zero_rfx_samples`, `zero_rfx_preds_train/test`; `zero_hat_*` are now the full +index. `predict()` term `"zero_rfx"` (probit scale on every scale); `y_hat` on +the response / zero-probability scales includes `v` and needs group ids. JSON: +flag `has_zero_rfx` and `random_effect_container_1`. Unseen groups follow +`unseen_groups` in `bart()`; `bcf()` refuses them on both parts (no policy +there, unchanged). + +**Recovery** (60 tracts x 30 rows, intercept SD 0.4 count / 0.7 zero, r = 5, +8 seeds): `bart()` pi correlation 0.882-0.927 with vs 0.535-0.677 without, +response RMSE ratio <= 0.834, intercept correlation 0.870-0.924. `bcf()` with +treatment correlated 0.35-0.49 with the zero intercept: zero-part effect error +0.076-0.296 with vs 0.556-0.824 without; the remainder is leakage through the +partially pooled intercepts (error 0.01-0.2 when treatment is unconfounded). + +**Mixing** (`bcf()`, same panel, 4 chains x 1,000 draws, 3 seeds, GFR and root +starts): split R-hat of the response-scale ATE <= 1.008 with vs <= 1.005 +without; bulk ESS 468-660 vs 520-851 of 4,000. Zero-part effect: R-hat <= 1.035 +vs <= 1.025, ESS 133-192 vs 173-243. **Timing** (n = 5,000, 100 tracts, default +trees, ms per iteration): `bart()` ZINB 32.2 -> 32.6; `bcf()` ZINB 69-71 -> +68.5-68.8 (within run-to-run noise). + +The seeded references `tests/testthat/fixtures/zinb-rfx-reference.rds` pin +ZINB without group ids, ZINB with the switch off (`bart()` and `bcf()`) and NB +with group ids (`bart()` and `bcf()`) to d8a0cb7b bitwise. diff --git a/NEWS.md b/NEWS.md index 830dc2a8..9e41d935 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,5 +1,23 @@ # nashtree 0.4.6.9011 +* The zero part of zero-inflated count models (`family = "zinb"`) in `bart()` + and `bcf()` gets random effects of its own whenever `rfx_group_ids_train` is + given: `pi = Phi(c0 + g(X) [+ tau_z(X) Z] + w' v_group)`, on the count part's + groups, basis (`model_spec`) and `random_effects_params` priors but with + separate parameters, sampled after the zero forest(s) on the zero part's + probit latent residual with unit variance. The intercept enters `pi` + everywhere: the structural-zero draw, the latent step, `zero_hat_train` / + `zero_hat_test` (now the full probit index), `predict()` (new term + `"zero_rfx"`, on the probit scale; `y_hat` on the `"response"` and + `"zero_probability"` scales now needs `rfx_group_ids`), the posterior + predictive, contrasts, and JSON (a second random-effects container). Unseen + groups follow `random_effects_params$unseen_groups` in `bart()`; `bcf()` + refuses them on both parts, as before. New outputs `zero_rfx_samples`, + `zero_rfx_preds_train`, `zero_rfx_preds_test`. + `general_params$zero_random_effects = FALSE` turns it off and reproduces the + previous draws bitwise; models without group ids, negative-binomial models + and saved models without the field are unchanged. + * `bcf()` fits count outcomes: `OutcomeModel("count", "log")` (negative binomial) and `OutcomeModel("count", "log", family = "zinb")`. The count part is `log(mu) = offset + c + mu(X, pi_hat) + [tau_0 + tau(X)] b(Z) [+ rfx]`, so diff --git a/R/bart.R b/R/bart.R index 6e85ffaf..763ba3cb 100644 --- a/R/bart.R +++ b/R/bart.R @@ -158,6 +158,7 @@ NULL #' - `negbin_dispersion_prior_rate` Count outcomes only. Rate `b0` of the same prior. Default: `0.01`, so the default prior is exponential with mean 100 and allows near-Poisson data. #' - `negbin_dispersion_init` Count outcomes only. Starting value of `r`. Defaults to a method-of-moments estimate around the constant-rate fit, clamped to `[0.1, 1000]`. #' - `sample_negbin_dispersion` Count outcomes only. Whether to sample `r`. If `FALSE`, `r` stays at `negbin_dispersion_init`. Default: `TRUE`. +#' - `zero_random_effects` Zero-inflated count outcomes with `rfx_group_ids_train` only. Whether the zero part gets its own random effects on the same groups and basis as the count part (see "Zero-inflated count outcomes"). `FALSE` gives the model without them, draw for draw. Default: `TRUE`. #' - `probit_outcome_model` Deprecated in favor of `outcome_model`. Whether or not the outcome should be modeled as explicitly binary via a probit link. If `TRUE`, `y` must only contain the values `0` and `1`. Default: `FALSE`. #' - `num_threads` Number of threads to use in the GFR and MCMC algorithms, as well as prediction. Defaults to `1` (single-threaded). Set to `-1` to use the maximum number of available threads, or a positive integer for a specific count. OpenMP must be available for values other than `1`. #' @@ -237,8 +238,18 @@ NULL #' variance weight, so `1 / w_i = 0` and they add nothing to the leaf sufficient statistics, to the #' leaf marginal likelihoods or to the random-effects conditional; a leaf that holds only such rows #' has a finite marginal likelihood equal to its prior and its value is drawn from the leaf prior; -#' (4) draws `r` from its conditional over the rows with `s_i = 0`; and (5) samples random effects, which -#' enter the count part only (the zero part has no random effects). +#' (4) draws `r` from its conditional over the rows with `s_i = 0`; and (5) samples the count part's +#' random effects. +#' +#' With `rfx_group_ids_train`, the zero part also gets random effects (unless +#' `general_params$zero_random_effects = FALSE`): `pi_i = Phi(c0 + g(X_i) + w_i' v_{k(i)})`, with the +#' basis `w_i` of the count part (a column of ones for `model_spec = "intercept_only"`), the same groups +#' and the same `random_effects_params` priors, but parameters `v` of their own. They are sampled right +#' after the zero forest, on the zero part's latent residual `z_i - c0 - g(X_i)` with unit variance (no +#' Pólya-Gamma weights), and enter `pi` everywhere: the draw of `s`, the latent step, `zero_hat_train` +#' and `zero_hat_test`, `predict()`, the posterior predictive and the serialized model (a second +#' random-effects container). Groups not seen in training follow `random_effects_params$unseen_groups` +#' on both parts. #' #' `min_samples_leaf` of the count forest counts all rows of a leaf, structural zeros included, so a #' count-forest leaf may hold fewer than `min_samples_leaf` rows that inform it. The bound is a guard @@ -254,10 +265,13 @@ NULL #' `sigma2_leaf_init = 2 / num_trees`, and is held fixed unless `zero_forest_params$sample_sigma2_leaf = TRUE`. #' `observation_weights` are not supported with this family. #' -#' The output adds `zero_forests`, `zero_hat_train` and `zero_hat_test` (draws of the probit index -#' `c0 + g(X)`, so `pnorm(zero_hat_train)` are draws of `pi`). `predict()` offers the scales `"linear"` -#' (`log(mu)`), `"rate"` (`mu`), `"zero_probability"` (`pi`) and `"response"` (`(1 - pi) mu`, the mean -#' of `y`), and the term `"zero_forest"`. [sampleBARTPosteriorPredictive()] includes the zero mass. +#' The output adds `zero_forests`, `zero_hat_train` and `zero_hat_test` (draws of the full probit index +#' `c0 + g(X)` plus any zero-part random effects, so `pnorm(zero_hat_train)` are draws of `pi`) and, with +#' zero-part random effects, `zero_rfx_samples`, `zero_rfx_preds_train` and `zero_rfx_preds_test`. +#' `predict()` offers the scales `"linear"` (`log(mu)`), `"rate"` (`mu`), `"zero_probability"` (`pi`) and +#' `"response"` (`(1 - pi) mu`, the mean of `y`), and the terms `"zero_forest"` (`c0 + g(X)`) and +#' `"zero_rfx"` (the zero-part random effects, on the probit scale on every prediction scale). +#' [sampleBARTPosteriorPredictive()] includes the zero mass. #' #' @return List of sampling outputs and a wrapper around the sampled forests (which can be used for in-memory prediction on new data, or serialized to JSON on disk). #' @export @@ -336,7 +350,8 @@ bart <- function( negbin_dispersion_prior_shape = 1, negbin_dispersion_prior_rate = 0.01, negbin_dispersion_init = NULL, - sample_negbin_dispersion = TRUE + sample_negbin_dispersion = TRUE, + zero_random_effects = TRUE ) general_params_updated <- preprocessParams( general_params_default, @@ -444,6 +459,7 @@ bart <- function( negbin_dispersion_prior_rate <- general_params_updated$negbin_dispersion_prior_rate negbin_dispersion_init <- general_params_updated$negbin_dispersion_init sample_negbin_dispersion <- general_params_updated$sample_negbin_dispersion + zero_random_effects <- general_params_updated$zero_random_effects # 2. Mean forest parameters num_trees_mean <- mean_forest_params_updated$num_trees @@ -588,6 +604,13 @@ bart <- function( "zero_forest_params is only used by zero-inflated count outcomes (OutcomeModel('count', 'log', family = 'zinb'))" ) } + if ( + !is.logical(zero_random_effects) || + length(zero_random_effects) != 1 || + is.na(zero_random_effects) + ) { + stop("general_params$zero_random_effects must be TRUE or FALSE") + } # Count (negative-binomial) outcome: validate before changing RNG state if (!link_is_log) { @@ -827,6 +850,13 @@ bart <- function( ) } previous_leaf_var_zero_samples <- previous_bart_model$sigma2_leaf_zero_samples + previous_zero_rfx_samples <- if ( + isTRUE(previous_bart_model$model_params$has_zero_rfx) + ) { + previous_bart_model$zero_rfx_samples + } else { + NULL + } if (previous_bart_model$model_params$outcome_model$link == "cloglog") { previous_cloglog_cutpoint_samples <- previous_bart_model$cloglog_cutpoint_samples previous_cloglog_num_categories <- previous_bart_model$cloglog_num_categories @@ -854,6 +884,7 @@ bart <- function( previous_negbin_dispersion_samples <- NULL previous_forest_samples_zero <- NULL previous_leaf_var_zero_samples <- NULL + previous_zero_rfx_samples <- NULL } # Determine whether conditional mean, variance, or both will be modeled @@ -1325,6 +1356,9 @@ bart <- function( ) } } + # Zero-inflated family: the zero part gets its own random effects on the + # same groups and basis unless general_params$zero_random_effects = FALSE + has_zero_rfx <- outcome_is_zinb && has_rfx && zero_random_effects if (has_rfx_test) { if (rfx_model_spec == "custom") { if (is.null(rfx_basis_test)) { @@ -1955,6 +1989,35 @@ bart <- function( ) } + # Zero-part random effects: same groups, basis and priors as the count + # part, sampled on the probit latent residual of the zero part with unit + # variance (no Polya-Gamma weights) + if (has_zero_rfx) { + zero_rfx_dataset_train <- createRandomEffectsDataset( + rfx_group_ids_train, + rfx_basis_train + ) + zero_rfx_tracker_train <- createRandomEffectsTracker(rfx_group_ids_train) + zero_rfx_model <- createRandomEffectsModel( + num_rfx_components, + num_rfx_groups + ) + rootResetRandomEffectsModel( + zero_rfx_model, + alpha_init, + xi_init, + sigma_alpha_init, + sigma_xi_init, + sigma_xi_shape, + sigma_xi_scale + ) + zero_rfx_samples <- createRandomEffectSamples( + num_rfx_components, + num_rfx_groups, + zero_rfx_tracker_train + ) + } + # Container of parameter samples num_actual_mcmc_iter <- num_mcmc * keep_every num_samples <- num_gfr + num_burnin + num_actual_mcmc_iter @@ -2220,6 +2283,13 @@ bart <- function( eta_count <- negbin_offset_train + y_bar_train + forest_pred + rfx_pred zero_pred <- active_forest_zero$predict(forest_dataset_zero) eta_zero <- zero_outcome_mean + zero_pred + if (has_zero_rfx) { + zero_rfx_pred <- zero_rfx_model$predict( + zero_rfx_dataset_train, + zero_rfx_tracker_train + ) + eta_zero <- eta_zero + zero_rfx_pred + } s <- .zinbSampleStructuralZeros( negbin_y, eta_count, @@ -2229,7 +2299,11 @@ bart <- function( # Albert and Chib latent z_i ~ N(eta_zero_i, 1) truncated to the side # given by s_i, as the probit outcome does z <- .probitLatentDraw(eta_zero, s) - outcome_zero$update_data(z - zero_outcome_mean - zero_pred) + if (has_zero_rfx) { + outcome_zero$update_data(z - zero_outcome_mean - zero_pred - zero_rfx_pred) + } else { + outcome_zero$update_data(z - zero_outcome_mean - zero_pred) + } forest_model_zero$sample_one_iteration( forest_dataset = forest_dataset_zero, residual = outcome_zero, @@ -2261,6 +2335,19 @@ bart <- function( as.matrix(leaf_scale_zero) ) } + # Zero-part random effects on the latent residual z - c0 - g(X) - u, + # with the latent error variance fixed at 1 + if (has_zero_rfx) { + zero_rfx_model$sample_random_effect( + zero_rfx_dataset_train, + outcome_zero, + zero_rfx_tracker_train, + zero_rfx_samples, + keep_sample, + 1.0, + rng + ) + } current_structural_zero <<- s list(forest_pred = forest_pred, rfx_pred = rfx_pred) } @@ -2581,6 +2668,21 @@ bart <- function( as.matrix(leaf_scale_zero_samples[forest_ind + 1]) ) } + if (has_zero_rfx) { + resetRandomEffectsModel( + zero_rfx_model, + zero_rfx_samples, + forest_ind, + sigma_alpha_init + ) + resetRandomEffectsTracker( + zero_rfx_tracker_train, + zero_rfx_model, + zero_rfx_dataset_train, + outcome_zero, + zero_rfx_samples + ) + } } } else if (has_prev_model) { warmstart_index <- ifelse( @@ -2732,6 +2834,42 @@ bart <- function( as.matrix(previous_leaf_var_zero_samples[warmstart_index]) ) } + if (has_zero_rfx) { + if (is.null(previous_zero_rfx_samples)) { + warning( + "`previous_model_json` did not have any zero-part random effects samples, so the zero-part RFX sampler will be run from scratch while the forests and any other parameters are warm started" + ) + rootResetRandomEffectsModel( + zero_rfx_model, + alpha_init, + xi_init, + sigma_alpha_init, + sigma_xi_init, + sigma_xi_shape, + sigma_xi_scale + ) + rootResetRandomEffectsTracker( + zero_rfx_tracker_train, + zero_rfx_model, + zero_rfx_dataset_train, + outcome_zero + ) + } else { + resetRandomEffectsModel( + zero_rfx_model, + previous_zero_rfx_samples, + warmstart_index - 1, + sigma_alpha_init + ) + resetRandomEffectsTracker( + zero_rfx_tracker_train, + zero_rfx_model, + zero_rfx_dataset_train, + outcome_zero, + zero_rfx_samples + ) + } + } } } else { if (include_mean_forest) { @@ -2831,6 +2969,23 @@ bart <- function( as.matrix(sigma2_leaf_zero_init) ) } + if (has_zero_rfx) { + rootResetRandomEffectsModel( + zero_rfx_model, + alpha_init, + xi_init, + sigma_alpha_init, + sigma_xi_init, + sigma_xi_shape, + sigma_xi_scale + ) + rootResetRandomEffectsTracker( + zero_rfx_tracker_train, + zero_rfx_model, + zero_rfx_dataset_train, + outcome_zero + ) + } } } for (i in (num_gfr + 1):num_samples) { @@ -3101,6 +3256,9 @@ bart <- function( if (has_rfx) { rfx_samples$delete_sample(0) } + if (has_zero_rfx) { + zero_rfx_samples$delete_sample(0) + } } if (include_mean_forest && !importance_only) { mean_forest_pred_train <- mean_forest_pred_train[, @@ -3208,6 +3366,24 @@ bart <- function( zero_outcome_mean } } + # Zero-part random effects enter the probit index on both sets + if (has_zero_rfx) { + zero_rfx_preds_train <- zero_rfx_samples$predict( + rfx_group_ids_train, + rfx_basis_train + ) + zero_hat_train <- zero_hat_train + zero_rfx_preds_train + if (has_rfx_test && has_test) { + zero_rfx_preds_test <- predictRandomEffectsForLabels( + zero_rfx_samples, + group_ids_factor_test, + rfx_basis_test, + unseen_groups = rfx_unseen_groups, + label = "rfx_group_ids_test" + ) + zero_hat_test <- zero_hat_test + zero_rfx_preds_test + } + } # Global error variance if (sample_sigma2_global) { @@ -3307,6 +3483,7 @@ bart <- function( model_params[["num_trees_zero"]] <- num_trees_zero model_params[["sample_sigma2_leaf_zero"]] <- isTRUE(sample_sigma2_leaf_zero) model_params[["sigma2_leaf_zero_init"]] <- as.numeric(sigma2_leaf_zero_init) + model_params[["has_zero_rfx"]] <- has_zero_rfx } result <- list( "model_params" = model_params, @@ -3345,6 +3522,13 @@ bart <- function( if (sample_sigma2_leaf_zero) { result[["sigma2_leaf_zero_samples"]] = leaf_scale_zero_samples } + if (has_zero_rfx) { + result[["zero_rfx_samples"]] = zero_rfx_samples + result[["zero_rfx_preds_train"]] = zero_rfx_preds_train + if (has_rfx_test && has_test) { + result[["zero_rfx_preds_test"]] = zero_rfx_preds_test + } + } } if (has_rfx) { result[["rfx_samples"]] = rfx_samples @@ -3403,6 +3587,9 @@ bart <- function( if (has_rfx) { rm(rfx_dataset_train, rfx_tracker_train, rfx_model) } + if (has_zero_rfx) { + rm(zero_rfx_dataset_train, zero_rfx_tracker_train, zero_rfx_model) + } rm(outcome_train) rm(rng) @@ -3430,8 +3617,8 @@ bart <- function( #' that were not in the training set. #' @param rfx_basis (Optional) Test set basis for "random-slope" regression in additive random effects model. #' @param type (Optional) Type of prediction to return. Options are "mean", which averages the predictions from every draw of a BART model, and "posterior", which returns the entire matrix of posterior predictions. Default: "posterior". -#' @param terms (Optional) Which model terms to include in the prediction. This can be a single term or a list of model terms. Options include "y_hat", "mean_forest", "rfx", "variance_forest", "zero_forest" (zero-inflated count models), or "all". If a model doesn't have mean forest, random effects, or variance forest predictions, but one of those terms is request, the request will simply be ignored. If none of the requested terms are present in a model, this function will return `NULL` along with a warning. Default: "all". -#' @param scale (Optional) Scale of mean function predictions. Options are "linear", which returns predictions on the original scale of the mean forest / RFX terms, "probability", which transforms predictions into class probabilities for models with discrete outcomes, and "class", which returns predicted outcome categories for discrete outcome models. "probability" is only valid for outcome models with `outcome == 'binary'` or `outcome == 'ordinal'`. For binary outcomes, this will return the probability that `y == 1`, and for ordinal outcomes, this will return probabilities for each outcome label. For count (negative-binomial) models, "linear" returns the log rate `eta = offset + c + f(X) [+ rfx]`, "rate" returns `mu = exp(eta)`, and "response" is an alias of "rate"; on the "rate" scale the mean-forest term is `exp(c + f(X))` and the random-effects term is the multiplicative rate ratio `exp(rfx)`. "rate" and "response" are only valid for count models, and "probability" and "class" are not valid for them. For zero-inflated count models (`family = "zinb"`), "linear" and "rate" describe the count part as above, "response" returns the mean `(1 - pi) * mu`, and "zero_probability" returns the structural-zero probability `pi = Phi(c0 + g(X))` (only the "y_hat" and "zero_forest" terms are defined on that scale, and "all" is narrowed to them); the "zero_forest" term is the probit index `c0 + g(X)` on the "linear" scale and `pi` on the other scales. With `type = "mean"`, transformed draws are averaged, so "rate" gives the posterior mean of `mu`. Default: "linear". +#' @param terms (Optional) Which model terms to include in the prediction. This can be a single term or a list of model terms. Options include "y_hat", "mean_forest", "rfx", "variance_forest", "zero_forest" and "zero_rfx" (zero-inflated count models), or "all". If a model doesn't have mean forest, random effects, or variance forest predictions, but one of those terms is request, the request will simply be ignored. If none of the requested terms are present in a model, this function will return `NULL` along with a warning. Default: "all". +#' @param scale (Optional) Scale of mean function predictions. Options are "linear", which returns predictions on the original scale of the mean forest / RFX terms, "probability", which transforms predictions into class probabilities for models with discrete outcomes, and "class", which returns predicted outcome categories for discrete outcome models. "probability" is only valid for outcome models with `outcome == 'binary'` or `outcome == 'ordinal'`. For binary outcomes, this will return the probability that `y == 1`, and for ordinal outcomes, this will return probabilities for each outcome label. For count (negative-binomial) models, "linear" returns the log rate `eta = offset + c + f(X) [+ rfx]`, "rate" returns `mu = exp(eta)`, and "response" is an alias of "rate"; on the "rate" scale the mean-forest term is `exp(c + f(X))` and the random-effects term is the multiplicative rate ratio `exp(rfx)`. "rate" and "response" are only valid for count models, and "probability" and "class" are not valid for them. For zero-inflated count models (`family = "zinb"`), "linear" and "rate" describe the count part as above, "response" returns the mean `(1 - pi) * mu`, and "zero_probability" returns the structural-zero probability `pi = Phi(c0 + g(X) [+ zero-part rfx])` (only the "y_hat", "zero_forest" and "zero_rfx" terms are defined on that scale, and "all" is narrowed to them); the "zero_forest" term is the forest's probit index `c0 + g(X)` on the "linear" scale and `Phi(c0 + g(X))` on the other scales, and the "zero_rfx" term (models with zero-part random effects) is the zero-part random effect on the probit scale on every scale. With zero-part random effects, `rfx_group_ids` are needed for "y_hat" on the "response" and "zero_probability" scales. With `type = "mean"`, transformed draws are averaged, so "rate" gives the posterior mean of `mu`. Default: "linear". #' @param offset (Optional) Count models only. Log-scale offset of length `nrow(X)` added to the linear predictor, as `offset_train` in [bart()]. If the model was fit with an offset and none is given here, an offset of 0 (unit exposure) is used with a warning. Default: `NULL`. #' @param ... (Optional) Other prediction parameters. #' @@ -3551,6 +3738,7 @@ predict.bartmodel <- function( has_variance_forest <- object$model_params$include_variance_forest has_rfx <- object$model_params$has_rfx has_zero_forest <- is_zinb + has_zero_rfx <- is_zinb && isTRUE(object$model_params$has_zero_rfx) has_y_hat <- has_mean_forest || has_rfx predict_y_hat <- (((has_y_hat) && ("y_hat" %in% terms)) || ((has_y_hat) && ("all" %in% terms))) @@ -3563,12 +3751,14 @@ predict.bartmodel <- function( ((has_variance_forest) && ("all" %in% terms))) predict_zero_forest <- has_zero_forest && (("zero_forest" %in% terms) || ("all" %in% terms)) + predict_zero_rfx <- has_zero_rfx && + (("zero_rfx" %in% terms) || ("all" %in% terms)) if (zero_probability_scale) { - # On the zero-probability scale only y_hat (= pi) and the zero forest are - # defined; "all" is narrowed to those two terms + # On the zero-probability scale only y_hat (= pi), the zero forest and the + # zero-part random effects are defined; "all" is narrowed to those terms if (any(c("mean_forest", "rfx") %in% terms)) { stop( - "scale = 'zero_probability' is only defined for the 'y_hat' and 'zero_forest' terms" + "scale = 'zero_probability' is only defined for the 'y_hat', 'zero_forest' and 'zero_rfx' terms" ) } predict_mean_forest <- FALSE @@ -3579,7 +3769,8 @@ predict.bartmodel <- function( predict_mean_forest, predict_rfx, predict_variance_forest, - predict_zero_forest + predict_zero_forest, + predict_zero_rfx )) if (predict_count == 0) { warning(paste0( @@ -3596,6 +3787,8 @@ predict.bartmodel <- function( # The zero part enters y_hat on the "response" and "zero_probability" scales predict_zero_forest_intermediate <- predict_y_hat && is_zinb && (scale %in% c("response", "zero_probability")) + predict_zero_rfx_intermediate <- predict_zero_forest_intermediate && + has_zero_rfx if (class_scale) { if (!((predict_count == 1) && (predict_y_hat))) { stop("Class scale can only be used with y_hat predictions") @@ -3624,7 +3817,9 @@ predict.bartmodel <- function( leaf_basis <- as.matrix(leaf_basis) } if ((is.null(dim(rfx_basis))) && (!is.null(rfx_basis))) { - if (predict_rfx) rfx_basis <- as.matrix(rfx_basis) + if (predict_rfx || predict_zero_rfx || predict_zero_rfx_intermediate) { + rfx_basis <- as.matrix(rfx_basis) + } } # Data checks @@ -3644,6 +3839,21 @@ predict.bartmodel <- function( "Random effect group labels (rfx_group_ids) must be provided for this model" ) } + if ( + (predict_zero_rfx || predict_zero_rfx_intermediate) && + is.null(rfx_group_ids) + ) { + stop( + "Random effect group labels (rfx_group_ids) must be provided for this model: its zero part has random effects" + ) + } + if ( + (predict_zero_rfx || predict_zero_rfx_intermediate) && + is.null(rfx_basis) && + !rfx_intercept + ) { + stop("Random effects basis (rfx_basis) must be provided for this model") + } if ((predict_rfx) && (is.null(rfx_basis)) && (!rfx_intercept)) { stop("Random effects basis (rfx_basis) must be provided for this model") } @@ -3773,6 +3983,21 @@ predict.bartmodel <- function( } } + # Zero-part random effects, on the probit scale on every prediction scale; + # they shift the probit index of pi (unseen groups follow the model's + # unseen-group policy, as the count part does) + if (predict_zero_rfx || predict_zero_rfx_intermediate) { + zero_rfx_predictions <- predictRandomEffectsForLabels( + object$zero_rfx_samples, + group_ids_factor, + rfx_basis, + unseen_groups = rfx_unseen_groups + ) + if (predict_zero_rfx_intermediate) { + zero_forest_linear <- zero_forest_linear + zero_rfx_predictions + } + } + # Compute rfx predictions (if needed) if (predict_rfx || predict_rfx_intermediate) { if (!is.null(rfx_basis)) { @@ -3930,6 +4155,9 @@ predict.bartmodel <- function( if (predict_zero_forest) { zero_forest_predictions <- rowMeans(zero_forest_predictions) } + if (predict_zero_rfx) { + zero_rfx_predictions <- rowMeans(zero_rfx_predictions) + } if (predict_y_hat) { if (is_ordinal_cloglog && probability_scale) { y_hat <- apply(y_hat, c(1, 2), mean) @@ -3959,6 +4187,8 @@ predict.bartmodel <- function( return(variance_forest_predictions) } else if (predict_zero_forest) { return(zero_forest_predictions) + } else if (predict_zero_rfx) { + return(zero_rfx_predictions) } } else { result <- list() @@ -3985,6 +4215,9 @@ predict.bartmodel <- function( if (predict_zero_forest) { result[["zero_forest_predictions"]] = zero_forest_predictions } + if (predict_zero_rfx) { + result[["zero_rfx_predictions"]] = zero_rfx_predictions + } return(result) } } @@ -4015,6 +4248,9 @@ print.bartmodel <- function(x, ...) { } if (identical(x$model_params$outcome_model$family, "zinb")) { model_terms <- c(model_terms, "zero forest") + if (isTRUE(x$model_params$has_zero_rfx)) { + model_terms <- c(model_terms, "zero-part random effects") + } } if (isTRUE(x$model_params$sample_negbin_dispersion)) { model_terms <- c(model_terms, "negative-binomial dispersion model") @@ -4066,7 +4302,9 @@ print.bartmodel <- function(x, ...) { ifelse(isTRUE(x$model_params$has_offset), " with an offset", ""), " and a probit zero part (", x$model_params$num_trees_zero, - " trees)" + " trees", + ifelse(isTRUE(x$model_params$has_zero_rfx), " and random effects", ""), + ")" ) } else if (identical(outcome_model$outcome, "count")) { outcome_model_summary <- paste0( @@ -4803,6 +5041,10 @@ saveBARTModelToJson <- function(object) { "parameters" ) } + jsonobj$add_boolean( + "has_zero_rfx", + isTRUE(object$model_params$has_zero_rfx) + ) } jsonobj$add_boolean("has_offset", isTRUE(object$model_params$has_offset)) jsonobj$add_scalar( @@ -4850,6 +5092,11 @@ saveBARTModelToJson <- function(object) { object$rfx_unique_group_ids ) } + # Zero-part random effects (zero-inflated family): the second random + # effects container, random_effect_container_1 + if (isTRUE(object$model_params$has_zero_rfx)) { + jsonobj$add_random_effects(object$zero_rfx_samples) + } # Add covariate preprocessor metadata preprocessor_metadata_string <- savePreprocessorToJsonString( diff --git a/R/bcf.R b/R/bcf.R index a42d06dc..bbe6f310 100644 --- a/R/bcf.R +++ b/R/bcf.R @@ -128,6 +128,7 @@ NULL #' - `probit_outcome_model` Deprecated in favor of `outcome_model`. Whether or not the outcome should be modeled as explicitly binary via a probit link. If `TRUE`, `y` must only contain the values `0` and `1`. Default: `FALSE`. #' - `num_threads` Number of threads to use in the GFR and MCMC algorithms, as well as prediction. Defaults to `1` (single-threaded). Set to `-1` to use the maximum number of available threads, or a positive integer for a specific count. OpenMP must be available for values other than `1`. #' - `negbin_dispersion_prior_shape`, `negbin_dispersion_prior_rate`, `negbin_dispersion_init`, `sample_negbin_dispersion` Count outcomes only: the `Gamma(shape, rate)` prior on the negative-binomial size `r` (defaults `1` and `0.01`), its starting value (method of moments, or the constant-rate ZINB fit, if `NULL`) and whether to sample it (default `TRUE`), as in [bart()]. +#' - `zero_random_effects` Zero-inflated count outcomes with `rfx_group_ids_train` only. Whether the zero part gets its own random effects on the same groups and basis as the count part (see "Count outcomes"). `FALSE` gives the model without them, draw for draw. Default: `TRUE`. #' #' @param prognostic_forest_params (Optional) A list of prognostic forest model parameters, each of which has a default value processed internally, so this argument list is optional. #' @@ -213,13 +214,19 @@ NULL #' With `family = "zinb"` a zero part is added: `pi_i = Phi(c0 + mu_z(X_i, pi_hat_i) + tau_z(X_i) Z_i)` is the #' probability of a structural zero, with a probit prognostic forest `mu_z` and a probit treatment forest #' `tau_z` (four forests in all), sampled on the structural-zero indicators as in [bart()]; structural zeros -#' drop out of the count part through infinite variance weights. The zero part has no random effects, no -#' intercept `tau_0` and no adaptive coding, and a multivariate treatment is refused. `observation_weights` are +#' drop out of the count part through infinite variance weights. The zero part has no intercept `tau_0` and +#' no adaptive coding, and a multivariate treatment is refused. With `rfx_group_ids_train` it gets random +#' effects of its own (unless `general_params$zero_random_effects = FALSE`): the count part's groups, basis +#' (`model_spec`) and priors, separate parameters, sampled after the zero forests on the zero part's latent +#' residual with unit variance, and entering `pi` everywhere, as in [bart()]. As for the count part, group +#' labels not seen in training are refused. `observation_weights` are #' refused with this family (with `family = "negbin"` they multiply the variance weights). #' #' `y_hat_train` holds draws of `log(mu)` including the offset, `mu_hat_train` of `c + mu(X)`, `tau_hat_train` of #' the log rate ratio. The zero-inflated family adds `zero_mu_hat_train` (`c0 + mu_z(X)`), `zero_tau_hat_train` -#' (`tau_z(X)`, probit scale) and `zero_hat_train` (the full probit index), and their test-set versions. +#' (`tau_z(X)`, probit scale) and `zero_hat_train` (the full probit index, zero-part random effects included), +#' and their test-set versions; with zero-part random effects also `zero_rfx_samples`, `zero_rfx_preds_train` +#' and `zero_rfx_preds_test`. #' [predict.bcfmodel()] offers the scales `"linear"`, `"rate"`, `"response"` (`(1 - pi) mu`) and `"zero_probability"`, #' [computeContrastBCFModel()] contrasts on those scales, and [sampleBCFPosteriorPredictive()] draws counts. #' @@ -326,7 +333,8 @@ bcf <- function( negbin_dispersion_prior_shape = 1, negbin_dispersion_prior_rate = 0.01, negbin_dispersion_init = NULL, - sample_negbin_dispersion = TRUE + sample_negbin_dispersion = TRUE, + zero_random_effects = TRUE ) general_params_updated <- preprocessParams( general_params_default, @@ -464,6 +472,7 @@ bcf <- function( negbin_dispersion_prior_rate <- general_params_updated$negbin_dispersion_prior_rate negbin_dispersion_init <- general_params_updated$negbin_dispersion_init sample_negbin_dispersion <- general_params_updated$sample_negbin_dispersion + zero_random_effects <- general_params_updated$zero_random_effects # 2. Mu forest parameters num_trees_mu <- prognostic_forest_params_updated$num_trees @@ -583,6 +592,13 @@ bcf <- function( "zero_prognostic_forest_params and zero_treatment_effect_forest_params are only used by zero-inflated count outcomes (OutcomeModel('count', 'log', family = 'zinb'))" ) } + if ( + !is.logical(zero_random_effects) || + length(zero_random_effects) != 1 || + is.na(zero_random_effects) + ) { + stop("general_params$zero_random_effects must be TRUE or FALSE") + } if (!link_is_log) { if (!is.null(offset_train) || !is.null(offset_test)) { stop( @@ -742,6 +758,13 @@ bcf <- function( previous_negbin_dispersion_samples <- previous_bcf_model$negbin_dispersion_samples previous_forest_samples_zero_mu <- previous_bcf_model$forests_zero_mu previous_forest_samples_zero_tau <- previous_bcf_model$forests_zero_tau + previous_zero_rfx_samples <- if ( + isTRUE(previous_bcf_model$model_params$has_zero_rfx) + ) { + previous_bcf_model$zero_rfx_samples + } else { + NULL + } if ( outcome_is_zinb && (previous_bcf_model$model_params$num_trees_zero_mu != num_trees_zero_mu || @@ -773,6 +796,7 @@ bcf <- function( previous_negbin_dispersion_samples <- NULL previous_forest_samples_zero_mu <- NULL previous_forest_samples_zero_tau <- NULL + previous_zero_rfx_samples <- NULL } # Determine whether conditional variance will be modeled @@ -1356,6 +1380,9 @@ bcf <- function( ) } } + # Zero-inflated family: the zero part gets its own random effects on the + # same groups and basis unless general_params$zero_random_effects = FALSE + has_zero_rfx <- outcome_is_zinb && has_rfx && zero_random_effects if (has_rfx_test) { if (rfx_model_spec == "custom") { if (is.null(rfx_basis_test)) { @@ -1956,6 +1983,35 @@ bcf <- function( ) } + # Zero-part random effects: same groups, basis and priors as the count + # part, sampled on the probit latent residual of the zero part with unit + # variance (no Polya-Gamma weights) + if (has_zero_rfx) { + zero_rfx_dataset_train <- createRandomEffectsDataset( + rfx_group_ids_train, + rfx_basis_train + ) + zero_rfx_tracker_train <- createRandomEffectsTracker(rfx_group_ids_train) + zero_rfx_model <- createRandomEffectsModel( + num_rfx_components, + num_rfx_groups + ) + rootResetRandomEffectsModel( + zero_rfx_model, + alpha_init, + xi_init, + sigma_alpha_init, + sigma_xi_init, + sigma_xi_shape, + sigma_xi_scale + ) + zero_rfx_samples <- createRandomEffectSamples( + num_rfx_components, + num_rfx_groups, + zero_rfx_tracker_train + ) + } + # Container of variance parameter samples num_actual_mcmc_iter <- num_mcmc * keep_every num_samples <- num_gfr + num_burnin + num_actual_mcmc_iter @@ -2371,6 +2427,10 @@ bcf <- function( zero_mu_pred <- active_forest_zero_mu$predict(forest_dataset_zero) zero_tau_pred <- active_forest_zero_tau$predict(forest_dataset_zero) eta_zero <- zero_outcome_mean + zero_mu_pred + zero_tau_pred + if (has_zero_rfx) { + eta_zero <- eta_zero + + zero_rfx_model$predict(zero_rfx_dataset_train, zero_rfx_tracker_train) + } s <- .zinbSampleStructuralZeros( negbin_y, negbin_offset_train + y_bar_train + fit, @@ -2408,6 +2468,19 @@ bcf <- function( keep_forest = keep_sample, gfr = gfr ) + # Zero-part random effects on the latent residual, with the latent + # error variance fixed at 1 + if (has_zero_rfx) { + zero_rfx_model$sample_random_effect( + zero_rfx_dataset_train, + outcome_zero, + zero_rfx_tracker_train, + zero_rfx_samples, + keep_sample, + 1.0, + rng + ) + } current_structural_zero <<- s fit } @@ -2437,6 +2510,41 @@ bcf <- function( TRUE ) } + # Reset the zero-part random effects at the start of a chain: from draw + # `index` of `samples`, or to their initial values when samples is NULL + bcf_zinb_reset_zero_rfx <- function(samples, index) { + if (is.null(samples)) { + rootResetRandomEffectsModel( + zero_rfx_model, + alpha_init, + xi_init, + sigma_alpha_init, + sigma_xi_init, + sigma_xi_shape, + sigma_xi_scale + ) + rootResetRandomEffectsTracker( + zero_rfx_tracker_train, + zero_rfx_model, + zero_rfx_dataset_train, + outcome_zero + ) + } else { + resetRandomEffectsModel( + zero_rfx_model, + samples, + index, + sigma_alpha_init + ) + resetRandomEffectsTracker( + zero_rfx_tracker_train, + zero_rfx_model, + zero_rfx_dataset_train, + outcome_zero, + samples + ) + } + } } # Run GFR (warm start) if specified @@ -2909,6 +3017,9 @@ bcf <- function( forest_samples_zero_tau, forest_ind ) + if (has_zero_rfx) { + bcf_zinb_reset_zero_rfx(zero_rfx_samples, forest_ind) + } } } else if (has_prev_model) { warmstart_index <- ifelse( @@ -3081,6 +3192,19 @@ bcf <- function( previous_forest_samples_zero_tau, warmstart_index - 1 ) + if (has_zero_rfx) { + if (is.null(previous_zero_rfx_samples)) { + warning( + "`previous_model_json` did not have any zero-part random effects samples, so the zero-part RFX sampler will be run from scratch while the forests and any other parameters are warm started" + ) + bcf_zinb_reset_zero_rfx(NULL, NULL) + } else { + bcf_zinb_reset_zero_rfx( + previous_zero_rfx_samples, + warmstart_index - 1 + ) + } + } } } else { resetActiveForest(active_forest_mu) @@ -3189,6 +3313,9 @@ bcf <- function( } if (outcome_is_zinb) { bcf_zinb_reset_zero_forests(NULL, NULL, NULL) + if (has_zero_rfx) { + bcf_zinb_reset_zero_rfx(NULL, NULL) + } } } for (i in (num_gfr + 1):num_samples) { @@ -3598,6 +3725,9 @@ bcf <- function( if (has_rfx) { rfx_samples$delete_sample(0) } + if (has_zero_rfx) { + zero_rfx_samples$delete_sample(0) + } } if (link_is_log) { negbin_dispersion_samples <- negbin_dispersion_samples[ @@ -3821,6 +3951,21 @@ bcf <- function( zero_tau_hat_test * as.numeric(Z_test) } } + # Zero-part random effects enter the full probit index on both sets + if (has_zero_rfx) { + zero_rfx_preds_train <- zero_rfx_samples$predict( + rfx_group_ids_train, + rfx_basis_train + ) + zero_hat_train <- zero_hat_train + zero_rfx_preds_train + if (has_rfx_test && has_test) { + zero_rfx_preds_test <- zero_rfx_samples$predict( + rfx_group_ids_test, + rfx_basis_test + ) + zero_hat_test <- zero_hat_test + zero_rfx_preds_test + } + } # Global error variance if (sample_sigma2_global) { @@ -3932,6 +4077,7 @@ bcf <- function( model_params[["num_trees_zero_tau"]] <- num_trees_zero_tau model_params[["sigma2_leaf_zero_mu"]] <- sigma2_leaf_zero_mu model_params[["sigma2_leaf_zero_tau"]] <- sigma2_leaf_zero_tau + model_params[["has_zero_rfx"]] <- has_zero_rfx } result <- list( "forests_mu" = forest_samples_mu, @@ -3997,6 +4143,13 @@ bcf <- function( result[["zero_tau_hat_test"]] = zero_tau_hat_test result[["zero_hat_test"]] = zero_hat_test } + if (has_zero_rfx) { + result[["zero_rfx_samples"]] = zero_rfx_samples + result[["zero_rfx_preds_train"]] = zero_rfx_preds_train + if (has_rfx_test && has_test) { + result[["zero_rfx_preds_test"]] = zero_rfx_preds_test + } + } } class(result) <- "bcfmodel" @@ -4025,7 +4178,7 @@ bcf <- function( #' that were not in the training set. #' @param rfx_basis (Optional) Test set basis for "random-slope" regression in additive random effects model. If the model was sampled with a random effects `model_spec` of "intercept_only" or "intercept_plus_treatment", this is optional, but if it is provided, it will be used. #' @param type (Optional) Type of prediction to return. Options are "mean", which averages the predictions from every draw of a BCF model, and "posterior", which returns the entire matrix of posterior predictions. Default: "posterior". -#' @param terms (Optional) Which model terms to include in the prediction. Options include `"y_hat"`, `"prognostic_function"`, `"mu"`, `"cate"`, `"tau"`, `"rfx"`, `"variance_forest"`, `"zero_prognostic"`, `"zero_tau"`, `"zero_forest"` (zero-inflated count models), or `"all"`. +#' @param terms (Optional) Which model terms to include in the prediction. Options include `"y_hat"`, `"prognostic_function"`, `"mu"`, `"cate"`, `"tau"`, `"rfx"`, `"variance_forest"`, `"zero_prognostic"`, `"zero_tau"`, `"zero_forest"`, `"zero_rfx"` (zero-inflated count models), or `"all"`. #' #' The treatment effect terms follow a three-level hierarchy: #' \itemize{ @@ -4037,7 +4190,7 @@ bcf <- function( #' Similarly for the prognostic term: `"mu"` returns the prognostic forest only, while `"prognostic_function"` additionally folds in the random intercept when `rfx_model_spec` is `"intercept_only"` or `"intercept_plus_treatment"`; otherwise the two are identical. #' #' If a model doesn't have random effects or variance forest predictions but one of those terms is requested, the request will simply be ignored. If none of the requested terms are present, this function will return `NULL` along with a warning. Default: `"all"`. -#' @param scale (Optional) Scale of mean function predictions. Options are "linear", which returns predictions on the original scale of the mean forest / RFX terms, and "probability", which transforms predictions into a probability of observing `y == 1`. "probability" is only valid for models fit with a probit outcome model. For count models, "linear" returns the log scale (`y_hat` is `log(mu)` with the offset, `mu` / `prognostic_function` are log baseline rates, `tau` / `cate` log rate ratios), "rate" exponentiates every term (`mu`, baseline rates, rate ratios, multiplicative random effects), and "response" is "rate" except that `y_hat` is the mean `(1 - pi) mu` for a zero-inflated model. For zero-inflated models, "zero_probability" returns `pi` as `y_hat` and the zero-part terms only. The zero-part terms `"zero_prognostic"` (`c0 + mu_z(X)`), `"zero_tau"` (`tau_z(X)`) and `"zero_forest"` (the full index) are probit indices on the "linear" scale; on the other scales `"zero_prognostic"` and `"zero_forest"` are probabilities and `"zero_tau"` stays a probit-scale coefficient. Default: "linear". +#' @param scale (Optional) Scale of mean function predictions. Options are "linear", which returns predictions on the original scale of the mean forest / RFX terms, and "probability", which transforms predictions into a probability of observing `y == 1`. "probability" is only valid for models fit with a probit outcome model. For count models, "linear" returns the log scale (`y_hat` is `log(mu)` with the offset, `mu` / `prognostic_function` are log baseline rates, `tau` / `cate` log rate ratios), "rate" exponentiates every term (`mu`, baseline rates, rate ratios, multiplicative random effects), and "response" is "rate" except that `y_hat` is the mean `(1 - pi) mu` for a zero-inflated model. For zero-inflated models, "zero_probability" returns `pi` as `y_hat` and the zero-part terms only. The zero-part terms `"zero_prognostic"` (`c0 + mu_z(X)`), `"zero_tau"` (`tau_z(X)`) and `"zero_forest"` (the forests' full index `c0 + mu_z(X) + tau_z(X) Z`) are probit indices on the "linear" scale; on the other scales `"zero_prognostic"` and `"zero_forest"` are probabilities and `"zero_tau"` stays a probit-scale coefficient. `"zero_rfx"` (models with zero-part random effects) is the zero part's random effect, on the probit scale on every scale; `pi` (and so `y_hat` on the "response" and "zero_probability" scales) includes it. Default: "linear". #' @param offset (Optional) Count models only. Log-scale offset of length `nrow(X)`, added to `y_hat`. If the model was fit with an offset and none is given, an offset of 0 is used with a warning. Default: `NULL`. #' @param ... (Optional) Other prediction parameters. #' @@ -4192,6 +4345,7 @@ predict.bcfmodel <- function( "zero_prognostic", "zero_tau", "zero_forest", + "zero_rfx", "all" )) ) { @@ -4231,7 +4385,7 @@ predict.bcfmodel <- function( if (zero_probability_scale) { if (any(c("mu", "tau", "prognostic_function", "cate", "rfx") %in% terms)) { stop( - "scale = 'zero_probability' is only defined for the 'y_hat', 'zero_prognostic', 'zero_tau' and 'zero_forest' terms" + "scale = 'zero_probability' is only defined for the 'y_hat', 'zero_prognostic', 'zero_tau', 'zero_forest' and 'zero_rfx' terms" ) } predict_mu_forest <- FALSE @@ -4243,6 +4397,8 @@ predict.bcfmodel <- function( predict_zero_mu <- is_zinb && any(c("zero_prognostic", "all") %in% terms) predict_zero_tau <- is_zinb && any(c("zero_tau", "all") %in% terms) predict_zero_forest <- is_zinb && any(c("zero_forest", "all") %in% terms) + has_zero_rfx <- is_zinb && isTRUE(object$model_params$has_zero_rfx) + predict_zero_rfx <- has_zero_rfx && any(c("zero_rfx", "all") %in% terms) predict_count <- sum(c( predict_y_hat, predict_mu_forest, @@ -4253,7 +4409,8 @@ predict.bcfmodel <- function( predict_variance_forest, predict_zero_mu, predict_zero_tau, - predict_zero_forest + predict_zero_forest, + predict_zero_rfx )) if (predict_count == 0) { warning(paste0( @@ -4655,11 +4812,14 @@ predict.bcfmodel <- function( rfx_predictions <- exp(rfx_predictions) } } + zero_part_enters_y_hat <- is_zinb && predict_y_hat && + scale %in% c("response", "zero_probability") needs_zero_part <- is_zinb && (predict_zero_mu || predict_zero_tau || predict_zero_forest || - (predict_y_hat && scale %in% c("response", "zero_probability"))) + predict_zero_rfx || + zero_part_enters_y_hat) if (needs_zero_part) { zero_mu_linear <- object$forests_zero_mu$predict(forest_dataset_pred) + object$model_params$zero_outcome_mean @@ -4667,11 +4827,21 @@ predict.bcfmodel <- function( forest_dataset_pred ) zero_linear <- zero_mu_linear + zero_tau_linear * as.numeric(Z) + # Zero-part random effects (probit scale on every prediction scale) + # shift the probit index of pi that enters y_hat + zero_index <- zero_linear + if (has_zero_rfx && (predict_zero_rfx || zero_part_enters_y_hat)) { + zero_rfx_predictions <- object$zero_rfx_samples$predict( + rfx_group_ids, + rfx_basis + ) + zero_index <- zero_linear + zero_rfx_predictions + } if (predict_y_hat && scale == "response") { - y_hat <- (1 - pnorm(zero_linear)) * y_hat + y_hat <- (1 - pnorm(zero_index)) * y_hat } if (y_hat_from_zero_part) { - y_hat <- pnorm(zero_linear) + y_hat <- pnorm(zero_index) } if (predict_zero_mu) { zero_prognostic <- if (scale == "linear") { @@ -4700,6 +4870,9 @@ predict.bcfmodel <- function( if (predict_zero_forest) { zero_forest <- rowMeans(zero_forest) } + if (predict_zero_rfx) { + zero_rfx_predictions <- rowMeans(zero_rfx_predictions) + } if (predict_mu_forest) { mu_hat <- rowMeans(mu_hat) } @@ -4750,6 +4923,8 @@ predict.bcfmodel <- function( return(zero_tau) } else if (predict_zero_forest) { return(zero_forest) + } else if (predict_zero_rfx) { + return(zero_rfx_predictions) } } else { result <- list() @@ -4797,6 +4972,9 @@ predict.bcfmodel <- function( if (predict_zero_forest) { result[["zero_forest"]] = zero_forest } + if (predict_zero_rfx) { + result[["zero_rfx_predictions"]] = zero_rfx_predictions + } } return(result) } @@ -4830,6 +5008,9 @@ print.bcfmodel <- function(x, ...) { } if (identical(x$model_params$outcome_model$family, "zinb")) { model_terms <- c(model_terms, "zero-part prognostic and treatment effect forests") + if (isTRUE(x$model_params$has_zero_rfx)) { + model_terms <- c(model_terms, "zero-part random effects") + } } if (isTRUE(x$model_params$sample_negbin_dispersion)) { model_terms <- c(model_terms, "negative-binomial dispersion model") @@ -4862,7 +5043,8 @@ print.bcfmodel <- function(x, ...) { paste0( "as zero-inflated negative binomial: a log-link count part (Polya-Gamma augmentation, treatment effect on the log-rate-ratio scale)", ifelse(isTRUE(x$model_params$has_offset), " with an offset", ""), - " and a probit zero part" + " and a probit zero part", + ifelse(isTRUE(x$model_params$has_zero_rfx), " with random effects", "") ) } else if (identical(x$model_params$outcome_model$outcome, "count")) { paste0( @@ -5808,6 +5990,10 @@ saveBCFModelToJson <- function(object) { )) { jsonobj$add_scalar(field, object$model_params[[field]]) } + jsonobj$add_boolean( + "has_zero_rfx", + isTRUE(object$model_params$has_zero_rfx) + ) } } @@ -5819,6 +6005,11 @@ saveBCFModelToJson <- function(object) { object$rfx_unique_group_ids ) } + # Zero-part random effects (zero-inflated family): the second random + # effects container, random_effect_container_1 + if (isTRUE(object$model_params$has_zero_rfx)) { + jsonobj$add_random_effects(object$zero_rfx_samples) + } jsonobj$add_string( "rfx_model_spec", object$model_params$rfx_model_spec diff --git a/R/bcf_count.R b/R/bcf_count.R index 6d665001..93832ba4 100644 --- a/R/bcf_count.R +++ b/R/bcf_count.R @@ -178,6 +178,12 @@ # has no variance forest output[["forests_zero_mu"]] <- load_forest("forest_2") output[["forests_zero_tau"]] <- load_forest("forest_3") + model_params[["has_zero_rfx"]] <- .readZeroRfxFlags(json_object_list) + if (model_params[["has_zero_rfx"]]) { + output[["zero_rfx_samples"]] <- .loadZeroRfxSamplesFromJson( + json_object_list + ) + } } list(model_params = model_params, output = output) } diff --git a/R/negbin.R b/R/negbin.R index 11fe5f20..f787e0ab 100644 --- a/R/negbin.R +++ b/R/negbin.R @@ -273,9 +273,36 @@ function(js) js$get_vector("sigma2_leaf_zero_samples", "parameters") )) } + zero_rfx <- .readZeroRfxFlags(json_object_list) + model_params[["has_zero_rfx"]] <- zero_rfx + if (zero_rfx) { + output[["zero_rfx_samples"]] <- .loadZeroRfxSamplesFromJson(json_object_list) + } list(model_params = model_params, output = output) } +# Whether the zero part of a list of zero-inflated JSON objects (one per chain +# or model) carries random effects. Models saved before zero-part random +# effects existed have no has_zero_rfx field and none. +.readZeroRfxFlags <- function(json_object_list) { + flags <- vapply(json_object_list, function(js) { + json_contains_field_cpp(js$json_ptr, "has_zero_rfx") && + js$get_boolean("has_zero_rfx") + }, logical(1)) + if (length(unique(flags)) > 1) { + stop( + "Cannot combine zero-inflated models with and without zero-part random effects" + ) + } + flags[1] +} + +# Zero-part random effects of a zero-inflated BART or BCF model: written after +# the count part's random effects, so they are random_effect_container_1 +.loadZeroRfxSamplesFromJson <- function(json_object_list) { + loadRandomEffectSamplesCombinedJson(json_object_list, 1) +} + # Column indices of X_train a forest may split on, from keep / drop lists of # names or indices (keep takes precedence), as bart() does for its mean and # variance forests. diff --git a/R/posterior_transformation.R b/R/posterior_transformation.R index 52908d30..a1c135c1 100644 --- a/R/posterior_transformation.R +++ b/R/posterior_transformation.R @@ -708,6 +708,7 @@ sampleBCFPosteriorPredictive <- function( mu = as.numeric(mu_draws) ) if (identical(model_object$model_params$outcome_model$family, "zinb")) { + # pi includes the zero-part random effects when the model has them pi_draws <- predict( model_object, X = X, @@ -716,7 +717,11 @@ sampleBCFPosteriorPredictive <- function( rfx_group_ids = rfx_group_ids, rfx_basis = rfx_basis, type = "posterior", - terms = "zero_forest", + terms = if (isTRUE(model_object$model_params$has_zero_rfx)) { + "y_hat" + } else { + "zero_forest" + }, scale = "zero_probability" ) ppd_vector[runif(length(ppd_vector)) < as.numeric(pi_draws)] <- 0 @@ -989,13 +994,27 @@ sampleBARTPosteriorPredictive <- function( # Zero-inflated family: a structural zero with probability pi, drawn # after the negative-binomial draws so that family = "negbin" is unchanged if (identical(model_object$model_params$outcome_model$family, "zinb")) { - pi_draws <- predict( - model_object, - X = X, - type = "posterior", - terms = "zero_forest", - scale = "zero_probability" - ) + pi_draws <- if (isTRUE(model_object$model_params$has_zero_rfx)) { + # pi includes the zero-part random effects of the rows' groups + predict( + model_object, + X = X, + leaf_basis = leaf_basis, + rfx_group_ids = rfx_group_ids, + rfx_basis = rfx_basis, + type = "posterior", + terms = "y_hat", + scale = "zero_probability" + ) + } else { + predict( + model_object, + X = X, + type = "posterior", + terms = "zero_forest", + scale = "zero_probability" + ) + } structural <- runif(length(ppd_vector)) < as.numeric(pi_draws) ppd_vector[structural] <- 0 } diff --git a/man/bart.Rd b/man/bart.Rd index 3de2d0af..ff25b243 100644 --- a/man/bart.Rd +++ b/man/bart.Rd @@ -101,6 +101,7 @@ referred to internally in the C++ layer as "variance weights" (\code{var_weights \item \code{negbin_dispersion_prior_rate} Count outcomes only. Rate \code{b0} of the same prior. Default: \code{0.01}, so the default prior is exponential with mean 100 and allows near-Poisson data. \item \code{negbin_dispersion_init} Count outcomes only. Starting value of \code{r}. Defaults to a method-of-moments estimate around the constant-rate fit, clamped to \verb{[0.1, 1000]}. \item \code{sample_negbin_dispersion} Count outcomes only. Whether to sample \code{r}. If \code{FALSE}, \code{r} stays at \code{negbin_dispersion_init}. Default: \code{TRUE}. +\item \code{zero_random_effects} Zero-inflated count outcomes with \code{rfx_group_ids_train} only. Whether the zero part gets its own random effects on the same groups and basis as the count part (see "Zero-inflated count outcomes"). \code{FALSE} gives the model without them, draw for draw. Default: \code{TRUE}. \item \code{probit_outcome_model} Deprecated in favor of \code{outcome_model}. Whether or not the outcome should be modeled as explicitly binary via a probit link. If \code{TRUE}, \code{y} must only contain the values \code{0} and \code{1}. Default: \code{FALSE}. \item \code{num_threads} Number of threads to use in the GFR and MCMC algorithms, as well as prediction. Defaults to \code{1} (single-threaded). Set to \code{-1} to use the maximum number of available threads, or a positive integer for a specific count. OpenMP must be available for values other than \code{1}. }} @@ -239,8 +240,18 @@ forest by the Pólya-Gamma step on the rows with \code{s_i = 0} only: rows with variance weight, so \code{1 / w_i = 0} and they add nothing to the leaf sufficient statistics, to the leaf marginal likelihoods or to the random-effects conditional; a leaf that holds only such rows has a finite marginal likelihood equal to its prior and its value is drawn from the leaf prior; -(4) draws \code{r} from its conditional over the rows with \code{s_i = 0}; and (5) samples random effects, which -enter the count part only (the zero part has no random effects). +(4) draws \code{r} from its conditional over the rows with \code{s_i = 0}; and (5) samples the count part's +random effects. + +With \code{rfx_group_ids_train}, the zero part also gets random effects (unless +\code{general_params$zero_random_effects = FALSE}): \verb{pi_i = Phi(c0 + g(X_i) + w_i' v_{k(i)})}, with the +basis \code{w_i} of the count part (a column of ones for \code{model_spec = "intercept_only"}), the same groups +and the same \code{random_effects_params} priors, but parameters \code{v} of their own. They are sampled right +after the zero forest, on the zero part's latent residual \code{z_i - c0 - g(X_i)} with unit variance (no +Pólya-Gamma weights), and enter \code{pi} everywhere: the draw of \code{s}, the latent step, \code{zero_hat_train} +and \code{zero_hat_test}, \code{predict()}, the posterior predictive and the serialized model (a second +random-effects container). Groups not seen in training follow \code{random_effects_params$unseen_groups} +on both parts. \code{min_samples_leaf} of the count forest counts all rows of a leaf, structural zeros included, so a count-forest leaf may hold fewer than \code{min_samples_leaf} rows that inform it. The bound is a guard @@ -256,10 +267,13 @@ fit. The zero-forest leaf scale follows the probit calibration of Chipman, Georg \code{sigma2_leaf_init = 2 / num_trees}, and is held fixed unless \code{zero_forest_params$sample_sigma2_leaf = TRUE}. \code{observation_weights} are not supported with this family. -The output adds \code{zero_forests}, \code{zero_hat_train} and \code{zero_hat_test} (draws of the probit index -\code{c0 + g(X)}, so \code{pnorm(zero_hat_train)} are draws of \code{pi}). \code{predict()} offers the scales \code{"linear"} -(\code{log(mu)}), \code{"rate"} (\code{mu}), \code{"zero_probability"} (\code{pi}) and \code{"response"} (\verb{(1 - pi) mu}, the mean -of \code{y}), and the term \code{"zero_forest"}. \code{\link[=sampleBARTPosteriorPredictive]{sampleBARTPosteriorPredictive()}} includes the zero mass. +The output adds \code{zero_forests}, \code{zero_hat_train} and \code{zero_hat_test} (draws of the full probit index +\code{c0 + g(X)} plus any zero-part random effects, so \code{pnorm(zero_hat_train)} are draws of \code{pi}) and, with +zero-part random effects, \code{zero_rfx_samples}, \code{zero_rfx_preds_train} and \code{zero_rfx_preds_test}. +\code{predict()} offers the scales \code{"linear"} (\code{log(mu)}), \code{"rate"} (\code{mu}), \code{"zero_probability"} (\code{pi}) and +\code{"response"} (\verb{(1 - pi) mu}, the mean of \code{y}), and the terms \code{"zero_forest"} (\code{c0 + g(X)}) and +\code{"zero_rfx"} (the zero-part random effects, on the probit scale on every prediction scale). +\code{\link[=sampleBARTPosteriorPredictive]{sampleBARTPosteriorPredictive()}} includes the zero mass. } \examples{ diff --git a/man/bcf.Rd b/man/bcf.Rd index 45907040..388d4db7 100644 --- a/man/bcf.Rd +++ b/man/bcf.Rd @@ -113,6 +113,7 @@ Default: \code{NULL}.} \item \code{probit_outcome_model} Deprecated in favor of \code{outcome_model}. Whether or not the outcome should be modeled as explicitly binary via a probit link. If \code{TRUE}, \code{y} must only contain the values \code{0} and \code{1}. Default: \code{FALSE}. \item \code{num_threads} Number of threads to use in the GFR and MCMC algorithms, as well as prediction. Defaults to \code{1} (single-threaded). Set to \code{-1} to use the maximum number of available threads, or a positive integer for a specific count. OpenMP must be available for values other than \code{1}. \item \code{negbin_dispersion_prior_shape}, \code{negbin_dispersion_prior_rate}, \code{negbin_dispersion_init}, \code{sample_negbin_dispersion} Count outcomes only: the \code{Gamma(shape, rate)} prior on the negative-binomial size \code{r} (defaults \code{1} and \code{0.01}), its starting value (method of moments, or the constant-rate ZINB fit, if \code{NULL}) and whether to sample it (default \code{TRUE}), as in \code{\link[=bart]{bart()}}. +\item \code{zero_random_effects} Zero-inflated count outcomes with \code{rfx_group_ids_train} only. Whether the zero part gets its own random effects on the same groups and basis as the count part (see "Count outcomes"). \code{FALSE} gives the model without them, draw for draw. Default: \code{TRUE}. }} \item{prognostic_forest_params}{(Optional) A list of prognostic forest model parameters, each of which has a default value processed internally, so this argument list is optional. @@ -213,13 +214,19 @@ precision weights \code{omega_i}. \code{r} is drawn by slice sampling as in \cod With \code{family = "zinb"} a zero part is added: \verb{pi_i = Phi(c0 + mu_z(X_i, pi_hat_i) + tau_z(X_i) Z_i)} is the probability of a structural zero, with a probit prognostic forest \code{mu_z} and a probit treatment forest \code{tau_z} (four forests in all), sampled on the structural-zero indicators as in \code{\link[=bart]{bart()}}; structural zeros -drop out of the count part through infinite variance weights. The zero part has no random effects, no -intercept \code{tau_0} and no adaptive coding, and a multivariate treatment is refused. \code{observation_weights} are +drop out of the count part through infinite variance weights. The zero part has no intercept \code{tau_0} and +no adaptive coding, and a multivariate treatment is refused. With \code{rfx_group_ids_train} it gets random +effects of its own (unless \code{general_params$zero_random_effects = FALSE}): the count part's groups, basis +(\code{model_spec}) and priors, separate parameters, sampled after the zero forests on the zero part's latent +residual with unit variance, and entering \code{pi} everywhere, as in \code{\link[=bart]{bart()}}. As for the count part, group +labels not seen in training are refused. \code{observation_weights} are refused with this family (with \code{family = "negbin"} they multiply the variance weights). \code{y_hat_train} holds draws of \code{log(mu)} including the offset, \code{mu_hat_train} of \code{c + mu(X)}, \code{tau_hat_train} of the log rate ratio. The zero-inflated family adds \code{zero_mu_hat_train} (\code{c0 + mu_z(X)}), \code{zero_tau_hat_train} -(\code{tau_z(X)}, probit scale) and \code{zero_hat_train} (the full probit index), and their test-set versions. +(\code{tau_z(X)}, probit scale) and \code{zero_hat_train} (the full probit index, zero-part random effects included), +and their test-set versions; with zero-part random effects also \code{zero_rfx_samples}, \code{zero_rfx_preds_train} +and \code{zero_rfx_preds_test}. \code{\link[=predict.bcfmodel]{predict.bcfmodel()}} offers the scales \code{"linear"}, \code{"rate"}, \code{"response"} (\verb{(1 - pi) mu}) and \code{"zero_probability"}, \code{\link[=computeContrastBCFModel]{computeContrastBCFModel()}} contrasts on those scales, and \code{\link[=sampleBCFPosteriorPredictive]{sampleBCFPosteriorPredictive()}} draws counts. } diff --git a/man/predict.bartmodel.Rd b/man/predict.bartmodel.Rd index 692dc5c7..5970ec22 100644 --- a/man/predict.bartmodel.Rd +++ b/man/predict.bartmodel.Rd @@ -32,9 +32,9 @@ that were not in the training set.} \item{type}{(Optional) Type of prediction to return. Options are "mean", which averages the predictions from every draw of a BART model, and "posterior", which returns the entire matrix of posterior predictions. Default: "posterior".} -\item{terms}{(Optional) Which model terms to include in the prediction. This can be a single term or a list of model terms. Options include "y_hat", "mean_forest", "rfx", "variance_forest", "zero_forest" (zero-inflated count models), or "all". If a model doesn't have mean forest, random effects, or variance forest predictions, but one of those terms is request, the request will simply be ignored. If none of the requested terms are present in a model, this function will return \code{NULL} along with a warning. Default: "all".} +\item{terms}{(Optional) Which model terms to include in the prediction. This can be a single term or a list of model terms. Options include "y_hat", "mean_forest", "rfx", "variance_forest", "zero_forest" and "zero_rfx" (zero-inflated count models), or "all". If a model doesn't have mean forest, random effects, or variance forest predictions, but one of those terms is request, the request will simply be ignored. If none of the requested terms are present in a model, this function will return \code{NULL} along with a warning. Default: "all".} -\item{scale}{(Optional) Scale of mean function predictions. Options are "linear", which returns predictions on the original scale of the mean forest / RFX terms, "probability", which transforms predictions into class probabilities for models with discrete outcomes, and "class", which returns predicted outcome categories for discrete outcome models. "probability" is only valid for outcome models with \code{outcome == 'binary'} or \code{outcome == 'ordinal'}. For binary outcomes, this will return the probability that \code{y == 1}, and for ordinal outcomes, this will return probabilities for each outcome label. For count (negative-binomial) models, "linear" returns the log rate \code{eta = offset + c + f(X) [+ rfx]}, "rate" returns \code{mu = exp(eta)}, and "response" is an alias of "rate"; on the "rate" scale the mean-forest term is \code{exp(c + f(X))} and the random-effects term is the multiplicative rate ratio \code{exp(rfx)}. "rate" and "response" are only valid for count models, and "probability" and "class" are not valid for them. For zero-inflated count models (\code{family = "zinb"}), "linear" and "rate" describe the count part as above, "response" returns the mean \code{(1 - pi) * mu}, and "zero_probability" returns the structural-zero probability \code{pi = Phi(c0 + g(X))} (only the "y_hat" and "zero_forest" terms are defined on that scale, and "all" is narrowed to them); the "zero_forest" term is the probit index \code{c0 + g(X)} on the "linear" scale and \code{pi} on the other scales. With \code{type = "mean"}, transformed draws are averaged, so "rate" gives the posterior mean of \code{mu}. Default: "linear".} +\item{scale}{(Optional) Scale of mean function predictions. Options are "linear", which returns predictions on the original scale of the mean forest / RFX terms, "probability", which transforms predictions into class probabilities for models with discrete outcomes, and "class", which returns predicted outcome categories for discrete outcome models. "probability" is only valid for outcome models with \code{outcome == 'binary'} or \code{outcome == 'ordinal'}. For binary outcomes, this will return the probability that \code{y == 1}, and for ordinal outcomes, this will return probabilities for each outcome label. For count (negative-binomial) models, "linear" returns the log rate \code{eta = offset + c + f(X) [+ rfx]}, "rate" returns \code{mu = exp(eta)}, and "response" is an alias of "rate"; on the "rate" scale the mean-forest term is \code{exp(c + f(X))} and the random-effects term is the multiplicative rate ratio \code{exp(rfx)}. "rate" and "response" are only valid for count models, and "probability" and "class" are not valid for them. For zero-inflated count models (\code{family = "zinb"}), "linear" and "rate" describe the count part as above, "response" returns the mean \code{(1 - pi) * mu}, and "zero_probability" returns the structural-zero probability \verb{pi = Phi(c0 + g(X) [+ zero-part rfx])} (only the "y_hat", "zero_forest" and "zero_rfx" terms are defined on that scale, and "all" is narrowed to them); the "zero_forest" term is the forest's probit index \code{c0 + g(X)} on the "linear" scale and \code{Phi(c0 + g(X))} on the other scales, and the "zero_rfx" term (models with zero-part random effects) is the zero-part random effect on the probit scale on every scale. With zero-part random effects, \code{rfx_group_ids} are needed for "y_hat" on the "response" and "zero_probability" scales. With \code{type = "mean"}, transformed draws are averaged, so "rate" gives the posterior mean of \code{mu}. Default: "linear".} \item{offset}{(Optional) Count models only. Log-scale offset of length \code{nrow(X)} added to the linear predictor, as \code{offset_train} in \code{\link[=bart]{bart()}}. If the model was fit with an offset and none is given here, an offset of 0 (unit exposure) is used with a warning. Default: \code{NULL}.} diff --git a/man/predict.bcfmodel.Rd b/man/predict.bcfmodel.Rd index 287e344d..520d78cb 100644 --- a/man/predict.bcfmodel.Rd +++ b/man/predict.bcfmodel.Rd @@ -35,7 +35,7 @@ that were not in the training set.} \item{type}{(Optional) Type of prediction to return. Options are "mean", which averages the predictions from every draw of a BCF model, and "posterior", which returns the entire matrix of posterior predictions. Default: "posterior".} -\item{terms}{(Optional) Which model terms to include in the prediction. Options include \code{"y_hat"}, \code{"prognostic_function"}, \code{"mu"}, \code{"cate"}, \code{"tau"}, \code{"rfx"}, \code{"variance_forest"}, \code{"zero_prognostic"}, \code{"zero_tau"}, \code{"zero_forest"} (zero-inflated count models), or \code{"all"}. +\item{terms}{(Optional) Which model terms to include in the prediction. Options include \code{"y_hat"}, \code{"prognostic_function"}, \code{"mu"}, \code{"cate"}, \code{"tau"}, \code{"rfx"}, \code{"variance_forest"}, \code{"zero_prognostic"}, \code{"zero_tau"}, \code{"zero_forest"}, \code{"zero_rfx"} (zero-inflated count models), or \code{"all"}. The treatment effect terms follow a three-level hierarchy: \itemize{ @@ -48,7 +48,7 @@ Similarly for the prognostic term: \code{"mu"} returns the prognostic forest onl If a model doesn't have random effects or variance forest predictions but one of those terms is requested, the request will simply be ignored. If none of the requested terms are present, this function will return \code{NULL} along with a warning. Default: \code{"all"}.} -\item{scale}{(Optional) Scale of mean function predictions. Options are "linear", which returns predictions on the original scale of the mean forest / RFX terms, and "probability", which transforms predictions into a probability of observing \code{y == 1}. "probability" is only valid for models fit with a probit outcome model. For count models, "linear" returns the log scale (\code{y_hat} is \code{log(mu)} with the offset, \code{mu} / \code{prognostic_function} are log baseline rates, \code{tau} / \code{cate} log rate ratios), "rate" exponentiates every term (\code{mu}, baseline rates, rate ratios, multiplicative random effects), and "response" is "rate" except that \code{y_hat} is the mean \verb{(1 - pi) mu} for a zero-inflated model. For zero-inflated models, "zero_probability" returns \code{pi} as \code{y_hat} and the zero-part terms only. The zero-part terms \code{"zero_prognostic"} (\code{c0 + mu_z(X)}), \code{"zero_tau"} (\code{tau_z(X)}) and \code{"zero_forest"} (the full index) are probit indices on the "linear" scale; on the other scales \code{"zero_prognostic"} and \code{"zero_forest"} are probabilities and \code{"zero_tau"} stays a probit-scale coefficient. Default: "linear".} +\item{scale}{(Optional) Scale of mean function predictions. Options are "linear", which returns predictions on the original scale of the mean forest / RFX terms, and "probability", which transforms predictions into a probability of observing \code{y == 1}. "probability" is only valid for models fit with a probit outcome model. For count models, "linear" returns the log scale (\code{y_hat} is \code{log(mu)} with the offset, \code{mu} / \code{prognostic_function} are log baseline rates, \code{tau} / \code{cate} log rate ratios), "rate" exponentiates every term (\code{mu}, baseline rates, rate ratios, multiplicative random effects), and "response" is "rate" except that \code{y_hat} is the mean \verb{(1 - pi) mu} for a zero-inflated model. For zero-inflated models, "zero_probability" returns \code{pi} as \code{y_hat} and the zero-part terms only. The zero-part terms \code{"zero_prognostic"} (\code{c0 + mu_z(X)}), \code{"zero_tau"} (\code{tau_z(X)}) and \code{"zero_forest"} (the forests' full index \verb{c0 + mu_z(X) + tau_z(X) Z}) are probit indices on the "linear" scale; on the other scales \code{"zero_prognostic"} and \code{"zero_forest"} are probabilities and \code{"zero_tau"} stays a probit-scale coefficient. \code{"zero_rfx"} (models with zero-part random effects) is the zero part's random effect, on the probit scale on every scale; \code{pi} (and so \code{y_hat} on the "response" and "zero_probability" scales) includes it. Default: "linear".} \item{offset}{(Optional) Count models only. Log-scale offset of length \code{nrow(X)}, added to \code{y_hat}. If the model was fit with an offset and none is given, an offset of 0 is used with a warning. Default: \code{NULL}.} diff --git a/tests/testthat/fixtures/make-zinb-rfx-reference.R b/tests/testthat/fixtures/make-zinb-rfx-reference.R new file mode 100644 index 00000000..747c1ab9 --- /dev/null +++ b/tests/testthat/fixtures/make-zinb-rfx-reference.R @@ -0,0 +1,157 @@ +# Generates zinb-rfx-reference.rds: seeded draws captured on branch +# cma-bcf-count (d8a0cb7b) before the zero part of the zero-inflated family got +# its own random intercept. test-zinb-rfx.R compares current draws with these +# bitwise to prove that the paths below did not change: +# (a) ZINB bart() without group ids, +# (b) ZINB bart() with group ids and general_params$zero_random_effects = FALSE, +# (c) ZINB bcf() with group ids and general_params$zero_random_effects = FALSE, +# (d) negative-binomial bart() and bcf() with group ids. +# d8a0cb7b ignores the unknown zero_random_effects entry of general_params, so +# the same calls run on both builds. +# +# Regenerate only from a build of d8a0cb7b: +# R_LIBS= Rscript tests/testthat/fixtures/make-zinb-rfx-reference.R +library(nashtree) + +zinb_rfx_reference_fits <- function() { + set.seed(7171) + n <- 220 + n_test <- 40 + p <- 4 + X <- matrix(runif((n + n_test) * p), ncol = p) + offset <- log(runif(n + n_test, 0.5, 2)) + groups <- sample(letters[1:6], n + n_test, replace = TRUE) + u_count <- c(a = -0.4, b = 0, c = 0.3, d = 0.1, e = -0.2, f = 0.5) + u_zero <- c(a = 0.5, b = -0.3, c = 0, d = 0.6, e = -0.6, f = 0.2) + ps <- 0.3 + 0.4 * X[, 1] + Z <- rbinom(n + n_test, 1, ps) + log_mu <- offset + 1 + X[, 1] - X[, 2] + 0.3 * Z + u_count[groups] + pi <- pnorm(-0.4 + X[, 3] - 0.3 * Z + u_zero[groups]) + y <- ifelse(runif(n + n_test) < pi, 0, + rnbinom(n + n_test, mu = exp(log_mu), size = 4)) + tr <- seq_len(n) + te <- n + seq_len(n_test) + zinb <- OutcomeModel("count", "log", family = "zinb") + negbin <- OutcomeModel("count", "log") + + keep_bart <- function(m, extra = list()) { + c(list( + y_hat_train = m$y_hat_train, y_hat_test = m$y_hat_test, + zero_hat_train = m$zero_hat_train, zero_hat_test = m$zero_hat_test, + dispersion = m$negbin_dispersion_samples, + sigma2_leaf = m$sigma2_leaf_samples, + rfx = m$rfx_preds_train, rfx_test = m$rfx_preds_test + ), extra) + } + keep_bcf <- function(m, extra = list()) { + c(list( + y_hat_train = m$y_hat_train, y_hat_test = m$y_hat_test, + mu_hat_train = m$mu_hat_train, tau_hat_train = m$tau_hat_train, + tau_hat_test = m$tau_hat_test, + zero_hat_train = m$zero_hat_train, zero_hat_test = m$zero_hat_test, + zero_mu_hat_train = m$zero_mu_hat_train, + zero_tau_hat_train = m$zero_tau_hat_train, + dispersion = m$negbin_dispersion_samples, + tau_0 = m$tau_0_samples, rfx = m$rfx_preds_train, + rfx_test = m$rfx_preds_test + ), extra) + } + out <- list() + + # (a) ZINB bart() without group ids: GFR warm start, two chains, test set + a <- bart(X[tr, ], y[tr], X_test = X[te, ], + offset_train = offset[tr], offset_test = offset[te], + num_gfr = 3, num_burnin = 3, num_mcmc = 6, + general_params = list(outcome_model = zinb, random_seed = 41L, + num_chains = 2), + mean_forest_params = list(num_trees = 30), + zero_forest_params = list(num_trees = 15)) + set.seed(42) + out$bart_zinb_no_rfx <- keep_bart(a, list( + response = predict(a, X[te, ], offset = offset[te], terms = "y_hat", + scale = "response"), + zero_probability = predict(a, X[te, ], terms = "y_hat", + scale = "zero_probability"), + ppd = sampleBARTPosteriorPredictive(a, X[te, ], offset = offset[te], + num_draws_per_sample = 2))) + + # (b) ZINB bart() with group ids, zero-part random effects switched off + b <- bart(X[tr, ], y[tr], X_test = X[te, ], + offset_train = offset[tr], offset_test = offset[te], + rfx_group_ids_train = groups[tr], rfx_group_ids_test = groups[te], + random_effects_params = list(model_spec = "intercept_only"), + num_gfr = 3, num_burnin = 3, num_mcmc = 6, + general_params = list(outcome_model = zinb, random_seed = 43L, + num_chains = 2, zero_random_effects = FALSE), + mean_forest_params = list(num_trees = 30), + zero_forest_params = list(num_trees = 15)) + set.seed(44) + out$bart_zinb_rfx_off <- keep_bart(b, list( + response = predict(b, X[te, ], rfx_group_ids = groups[te], + offset = offset[te], terms = "y_hat", scale = "response"), + zero_probability = predict(b, X[te, ], rfx_group_ids = groups[te], + terms = "y_hat", scale = "zero_probability"), + ppd = sampleBARTPosteriorPredictive(b, X[te, ], rfx_group_ids = groups[te], + offset = offset[te], num_draws_per_sample = 2))) + + # (c) ZINB bcf() with group ids, zero-part random effects switched off + cz <- bcf(X[tr, ], Z[tr], y[tr], propensity_train = ps[tr], + X_test = X[te, ], Z_test = Z[te], propensity_test = ps[te], + offset_train = offset[tr], offset_test = offset[te], + rfx_group_ids_train = groups[tr], rfx_group_ids_test = groups[te], + random_effects_params = list(model_spec = "intercept_only"), + num_gfr = 3, num_burnin = 3, num_mcmc = 6, + general_params = list(outcome_model = zinb, random_seed = 45L, + num_chains = 2, zero_random_effects = FALSE), + prognostic_forest_params = list(num_trees = 30), + treatment_effect_forest_params = list(num_trees = 15), + zero_prognostic_forest_params = list(num_trees = 15), + zero_treatment_effect_forest_params = list(num_trees = 10)) + set.seed(46) + out$bcf_zinb_rfx_off <- keep_bcf(cz, list( + response = predict(cz, X[te, ], Z[te], ps[te], rfx_group_ids = groups[te], + offset = offset[te], terms = "y_hat", scale = "response"), + zero_probability = predict(cz, X[te, ], Z[te], ps[te], + rfx_group_ids = groups[te], terms = "y_hat", scale = "zero_probability"), + contrast = computeContrastBCFModel(cz, X[te, ], X[te, ], rep(0, n_test), + rep(1, n_test), ps[te], ps[te], groups[te], groups[te], + matrix(1, n_test, 1), matrix(1, n_test, 1), scale = "response", offset_0 = offset[te], offset_1 = offset[te]), + ppd = sampleBCFPosteriorPredictive(cz, X[te, ], Z[te], ps[te], + rfx_group_ids = groups[te], offset = offset[te], + num_draws_per_sample = 2))) + + # (d) negative-binomial bart() and bcf() with group ids + d1 <- bart(X[tr, ], y[tr], X_test = X[te, ], + offset_train = offset[tr], offset_test = offset[te], + rfx_group_ids_train = groups[tr], rfx_group_ids_test = groups[te], + random_effects_params = list(model_spec = "intercept_only"), + num_gfr = 3, num_burnin = 3, num_mcmc = 6, + general_params = list(outcome_model = negbin, random_seed = 47L, + num_chains = 2), + mean_forest_params = list(num_trees = 30)) + out$bart_negbin_rfx <- keep_bart(d1, list( + rate = predict(d1, X[te, ], rfx_group_ids = groups[te], + offset = offset[te], terms = "y_hat", scale = "rate"))) + d2 <- bcf(X[tr, ], Z[tr], y[tr], propensity_train = ps[tr], + X_test = X[te, ], Z_test = Z[te], propensity_test = ps[te], + offset_train = offset[tr], offset_test = offset[te], + rfx_group_ids_train = groups[tr], rfx_group_ids_test = groups[te], + random_effects_params = list(model_spec = "intercept_only"), + num_gfr = 3, num_burnin = 3, num_mcmc = 6, + general_params = list(outcome_model = negbin, random_seed = 48L, + num_chains = 2), + prognostic_forest_params = list(num_trees = 30), + treatment_effect_forest_params = list(num_trees = 15)) + out$bcf_negbin_rfx <- keep_bcf(d2, list( + rate = predict(d2, X[te, ], Z[te], ps[te], rfx_group_ids = groups[te], + offset = offset[te], terms = "y_hat", scale = "rate"))) + out +} + +if (sys.nframe() == 0L) { + ref <- suppressWarnings(zinb_rfx_reference_fits()) + ref$platform <- R.version$platform + ref$generated_from <- "cma-bcf-count d8a0cb7b" + saveRDS(ref, file.path("tests", "testthat", "fixtures", "zinb-rfx-reference.rds")) + cat("wrote reference on", ref$platform, "\n") +} diff --git a/tests/testthat/fixtures/zinb-rfx-reference.rds b/tests/testthat/fixtures/zinb-rfx-reference.rds new file mode 100644 index 00000000..d5c9286d Binary files /dev/null and b/tests/testthat/fixtures/zinb-rfx-reference.rds differ diff --git a/tests/testthat/test-bcf-count.R b/tests/testthat/test-bcf-count.R index 2d066835..47551eff 100644 --- a/tests/testthat/test-bcf-count.R +++ b/tests/testthat/test-bcf-count.R @@ -48,6 +48,9 @@ small_bcf_count <- function(seed = 4, n = 300, family = "zinb") { d } +# For family = "zinb" the zero part has no random effects here +# (general_params$zero_random_effects = FALSE), as before they existed; +# test-zinb-rfx.R covers them. fit_small_bcf_count <- function(d, family = "zinb", seed = 1, num_chains = 1, num_gfr = 4, general = list(), ...) { bcf(d$X, d$Z, d$y, propensity_train = d$propensity, offset_train = d$offset, @@ -57,7 +60,7 @@ fit_small_bcf_count <- function(d, family = "zinb", seed = 1, num_chains = 1, random_effects_params = list(model_spec = "intercept_only"), num_gfr = num_gfr, num_burnin = 5, num_mcmc = 10, general_params = c(list(outcome_model = count_model(family), random_seed = seed, - num_chains = num_chains), general), + num_chains = num_chains, zero_random_effects = FALSE), general), prognostic_forest_params = list(num_trees = 40), treatment_effect_forest_params = list(num_trees = 20), ...) diff --git a/tests/testthat/test-zinb-rfx.R b/tests/testthat/test-zinb-rfx.R new file mode 100644 index 00000000..a7fe59c0 --- /dev/null +++ b/tests/testthat/test-zinb-rfx.R @@ -0,0 +1,592 @@ +# Random intercept on the zero part of the zero-inflated negative-binomial +# family (general_params$zero_random_effects, on by default whenever +# rfx_group_ids_train is given), in bart() and bcf(). See the "Zero-inflated +# count outcomes" sections of ?bart and ?bcf. + +zinb_model <- function() OutcomeModel(outcome = "count", link = "log", family = "zinb") + +# Panel of G groups (tracts) with m rows each. Count part: +# log mu = 0.9 + sin(2 pi x1) + 0.6 x2^2 + u_c[g], u_c ~ N(0, 0.4^2); zero +# part: pi = Phi(-0.6 + 0.8 cos(pi x3) + 0.4 (x4 - 0.5) + u_z[g]), +# u_z ~ N(0, 0.7^2); r = 5. Mean pi about 0.3. +# +# With treatment = TRUE a binary treatment is added whose assignment depends +# on the zero-part tract intercept, P(z = 1) = Phi(a(x) + 0.9 u_z[g]) with +# a(x) = 0.6 (x1 - 0.5), and has known effects: a log rate ratio of 0.3 on the +# count part and -0.5 on the probit scale of the zero part. The propensity +# handed to bcf() is the one an analyst could estimate from x alone, +# Phi(a(x) / sqrt(1 + 0.9^2 0.7^2)), so the forests cannot recover u_z from it. +simulate_zinb_panel <- function(seed, G = 60, m = 30, treatment = FALSE, + sd_count = 0.4, sd_zero = 0.7, r = 5) { + set.seed(seed) + n <- G * m + groups <- rep(seq_len(G), each = m) + X <- matrix(runif(n * 4), ncol = 4) + u_count <- rnorm(G, 0, sd_count) + u_zero <- rnorm(G, 0, sd_zero) + tau_count <- 0.3 + tau_zero <- -0.5 + a <- 0.6 * (X[, 1] - 0.5) + if (treatment) { + Z <- rbinom(n, 1, pnorm(a + 0.9 * u_zero[groups])) + propensity <- pnorm(a / sqrt(1 + 0.9^2 * sd_zero^2)) + } else { + Z <- rep(0, n) + propensity <- NULL + } + f <- 0.9 + sin(2 * pi * X[, 1]) + 0.6 * X[, 2]^2 + g <- -0.6 + 0.8 * cos(pi * X[, 3]) + 0.4 * (X[, 4] - 0.5) + log_mu0 <- f + u_count[groups] + zero_index0 <- g + u_zero[groups] + log_mu <- log_mu0 + tau_count * Z + pi <- pnorm(zero_index0 + tau_zero * Z) + y <- ifelse(runif(n) < pi, 0, rnbinom(n, mu = exp(log_mu), size = r)) + list( + X = X, y = y, Z = Z, propensity = propensity, groups = groups, + u_count = u_count, u_zero = u_zero, log_mu = log_mu, pi = pi, + response = (1 - pi) * exp(log_mu), tau_count = tau_count, tau_zero = tau_zero, + response0 = (1 - pnorm(zero_index0)) * exp(log_mu0), + response1 = (1 - pnorm(zero_index0 + tau_zero)) * exp(log_mu0 + tau_count) + ) +} + +fit_zinb_panel_bart <- function(d, zero_rfx, seed = 1, num_gfr = 10, + num_burnin = 150, num_mcmc = 250, + general = list(), ...) { + bart(d$X, d$y, + rfx_group_ids_train = d$groups, + random_effects_params = list(model_spec = "intercept_only"), + num_gfr = num_gfr, num_burnin = num_burnin, num_mcmc = num_mcmc, + general_params = c(list(outcome_model = zinb_model(), random_seed = seed, + zero_random_effects = zero_rfx), general), + mean_forest_params = list(num_trees = 100), + ... + ) +} + +fit_zinb_panel_bcf <- function(d, zero_rfx, seed = 1, num_gfr = 10, + num_burnin = 150, num_mcmc = 250, + general = list(), ...) { + bcf(d$X, d$Z, d$y, propensity_train = d$propensity, + rfx_group_ids_train = d$groups, + random_effects_params = list(model_spec = "intercept_only"), + num_gfr = num_gfr, num_burnin = num_burnin, num_mcmc = num_mcmc, + general_params = c(list(outcome_model = zinb_model(), random_seed = seed, + zero_random_effects = zero_rfx), general), + prognostic_forest_params = list(num_trees = 100), + treatment_effect_forest_params = list(num_trees = 40), + ... + ) +} + +# Recovery summaries of a bart() fit on simulate_zinb_panel() data +zinb_panel_bart_summary <- function(m, d) { + pi_hat <- rowMeans(pnorm(m$zero_hat_train)) + response_hat <- rowMeans(predict(m, d$X, rfx_group_ids = d$groups, + terms = "y_hat", scale = "response")) + out <- list( + pi_cor = cor(pi_hat, d$pi), + pi_rmse = sqrt(mean((pi_hat - d$pi)^2)), + response_rmse = sqrt(mean((response_hat - d$response)^2)) + ) + if (isTRUE(m$model_params$has_zero_rfx)) { + u_hat <- rowMeans(m$zero_rfx_samples$extract_parameter_samples()$beta_samples) + out$u_zero_cor <- cor(u_hat, d$u_zero) + } + out +} + +# Error of the posterior mean of the average zero-part effect (probit scale) +# of a bcf() fit, and of the average log rate ratio +zinb_panel_bcf_summary <- function(m, d) { + out <- list( + zero_effect_error = abs(mean(m$zero_tau_hat_train) - d$tau_zero), + count_effect_error = abs(mean(m$tau_hat_train) - d$tau_count) + ) + if (isTRUE(m$model_params$has_zero_rfx)) { + u_hat <- rowMeans(m$zero_rfx_samples$extract_parameter_samples()$beta_samples) + out$u_zero_cor <- cor(u_hat, d$u_zero) + } + out +} + +# --------------------------------------------------------------------------- +# Recovery with and without the zero-part random intercept +# +# Thresholds come from a calibration run of 8 data sets (simulation seeds +# 3001-3008, fit seeds 1-8, the settings of fit_zinb_panel_*(); 60 groups x 30 +# rows, 41-48% zeros). Over those 8 pairs of fits: +# bart(): correlation of the posterior mean pi with the truth 0.882-0.927 +# with the zero-part intercept and 0.535-0.677 without (gain >= 0.236); +# RMSE of the posterior mean response (1 - pi) mu with / without +# <= 0.834; correlation of the zero-part group intercepts with the truth +# 0.870-0.924. +# bcf() (treatment assignment correlated 0.35-0.49 with u_z): error of the +# posterior mean average zero-part effect 0.076-0.296 with and +# 0.556-0.824 without (difference >= 0.340); intercept correlation +# 0.833-0.900; error of the average log rate ratio <= 0.11 either way. +# The tests below use the first data set. +# --------------------------------------------------------------------------- + +test_that("The zero-part random intercept improves pi and the response in bart()", { + skip_on_cran() + d <- simulate_zinb_panel(3001) + with <- zinb_panel_bart_summary(fit_zinb_panel_bart(d, TRUE, seed = 1), d) + without <- zinb_panel_bart_summary(fit_zinb_panel_bart(d, FALSE, seed = 1), d) + expect_gt(with$pi_cor, 0.85) + expect_gt(with$pi_cor - without$pi_cor, 0.15) + expect_lt(with$pi_rmse, without$pi_rmse) + expect_lt(with$response_rmse, 0.92 * without$response_rmse) + expect_gt(with$u_zero_cor, 0.8) +}) + +test_that("The zero-part random intercept removes most of the confounding of the zero-part effect in bcf()", { + skip_on_cran() + d <- simulate_zinb_panel(3001, treatment = TRUE) + # Treatment assignment is correlated with the zero-part tract intercept + expect_gt(cor(d$Z, d$u_zero[d$groups]), 0.3) + with <- zinb_panel_bcf_summary(fit_zinb_panel_bcf(d, TRUE, seed = 1), d) + without <- zinb_panel_bcf_summary(fit_zinb_panel_bcf(d, FALSE, seed = 1), d) + expect_lt(with$zero_effect_error, 0.4) + expect_lt(with$zero_effect_error, without$zero_effect_error - 0.2) + expect_gt(with$u_zero_cor, 0.8) + expect_lt(with$count_effect_error, 0.15) + expect_lt(without$count_effect_error, 0.15) +}) + +# Small data set for the mechanics tests: 8 labelled groups, 25 rows each, +# a test set of 40 rows, an offset and a binary treatment +small_zinb_rfx_data <- function(seed = 5, G = 8, m = 25, n_test = 40) { + set.seed(seed) + n <- G * m + n_test + groups <- c(rep(letters[seq_len(G)], each = m), + sample(letters[seq_len(G)], n_test, replace = TRUE)) + X <- matrix(runif(n * 4), ncol = 4) + offset <- log(runif(n, 0.5, 2)) + u_count <- setNames(rnorm(G, 0, 0.3), letters[seq_len(G)]) + u_zero <- setNames(rnorm(G, 0, 0.7), letters[seq_len(G)]) + propensity <- 0.3 + 0.4 * X[, 1] + Z <- rbinom(n, 1, propensity) + log_mu <- offset + 1 + X[, 1] - X[, 2] + 0.3 * Z + u_count[groups] + pi <- pnorm(-0.4 + X[, 3] - 0.4 * Z + u_zero[groups]) + y <- ifelse(runif(n) < pi, 0, rnbinom(n, mu = exp(log_mu), size = 4)) + tr <- seq_len(G * m) + te <- G * m + seq_len(n_test) + list( + X = X[tr, ], y = y[tr], Z = Z[tr], propensity = propensity[tr], + offset = offset[tr], groups = groups[tr], + X_test = X[te, ], Z_test = Z[te], propensity_test = propensity[te], + offset_test = offset[te], groups_test = groups[te] + ) +} + +fit_small_bart <- function(d, seed = 1, num_chains = 1, num_gfr = 4, + num_mcmc = 15, general = list(), rfx = list(), ...) { + bart(d$X, d$y, X_test = d$X_test, + offset_train = d$offset, offset_test = d$offset_test, + rfx_group_ids_train = d$groups, rfx_group_ids_test = d$groups_test, + random_effects_params = c(list(model_spec = "intercept_only"), rfx), + num_gfr = num_gfr, num_burnin = 5, num_mcmc = num_mcmc, + general_params = c(list(outcome_model = zinb_model(), random_seed = seed, + num_chains = num_chains), general), + mean_forest_params = list(num_trees = 30), + zero_forest_params = list(num_trees = 15), + ... + ) +} + +fit_small_bcf <- function(d, seed = 1, num_chains = 1, num_gfr = 4, + num_mcmc = 15, general = list(), rfx = list(), ...) { + bcf(d$X, d$Z, d$y, propensity_train = d$propensity, + X_test = d$X_test, Z_test = d$Z_test, propensity_test = d$propensity_test, + offset_train = d$offset, offset_test = d$offset_test, + rfx_group_ids_train = d$groups, rfx_group_ids_test = d$groups_test, + random_effects_params = c(list(model_spec = "intercept_only"), rfx), + num_gfr = num_gfr, num_burnin = 5, num_mcmc = num_mcmc, + general_params = c(list(outcome_model = zinb_model(), random_seed = seed, + num_chains = num_chains), general), + prognostic_forest_params = list(num_trees = 30), + treatment_effect_forest_params = list(num_trees = 15), + zero_prognostic_forest_params = list(num_trees = 15), + zero_treatment_effect_forest_params = list(num_trees = 10), + ... + ) +} + +zero_rfx_beta <- function(m) { + beta <- m$zero_rfx_samples$extract_parameter_samples()$beta_samples + if (length(dim(beta)) == 3) beta[1, , ] else beta +} + +# --------------------------------------------------------------------------- +# Paths without a zero-part random intercept are bitwise unchanged +# --------------------------------------------------------------------------- + +test_that("ZINB without zero-part random effects and NB draws are bitwise identical to cma-bcf-count (d8a0cb7b)", { + skip_on_cran() + ref <- readRDS(test_path("fixtures", "zinb-rfx-reference.rds")) + skip_if_not( + identical(ref$platform, R.version$platform), + paste("reference draws were generated on", ref$platform) + ) + source(test_path("fixtures", "make-zinb-rfx-reference.R"), local = TRUE) + current <- suppressWarnings(zinb_rfx_reference_fits()) + for (name in setdiff(names(ref), c("platform", "generated_from"))) { + expect_identical(current[[name]], ref[[name]], label = name) + } +}) + +test_that("zero_random_effects is validated and only used with group ids and family = 'zinb'", { + skip_on_cran() + d <- small_zinb_rfx_data() + for (bad in list("yes", NA, c(TRUE, FALSE), 1)) { + expect_error( + fit_small_bart(d, num_gfr = 0, num_mcmc = 2, + general = list(zero_random_effects = bad)), + "zero_random_effects must be TRUE or FALSE" + ) + expect_error( + fit_small_bcf(d, num_gfr = 0, num_mcmc = 2, + general = list(zero_random_effects = bad)), + "zero_random_effects must be TRUE or FALSE" + ) + } + on <- fit_small_bart(d, num_gfr = 0, num_mcmc = 2) + expect_true(on$model_params$has_zero_rfx) + off <- fit_small_bart(d, num_gfr = 0, num_mcmc = 2, + general = list(zero_random_effects = FALSE)) + expect_false(off$model_params$has_zero_rfx) + expect_null(off$zero_rfx_samples) + no_groups <- bart(d$X, d$y, num_gfr = 0, num_burnin = 0, num_mcmc = 2, + general_params = list(outcome_model = zinb_model(), random_seed = 1), + mean_forest_params = list(num_trees = 10)) + expect_false(no_groups$model_params$has_zero_rfx) + nb <- bart(d$X, d$y, rfx_group_ids_train = d$groups, + random_effects_params = list(model_spec = "intercept_only"), + num_gfr = 0, num_burnin = 0, num_mcmc = 2, + general_params = list(outcome_model = OutcomeModel("count", "log"), + random_seed = 1), + mean_forest_params = list(num_trees = 10)) + expect_null(nb$model_params$has_zero_rfx) + expect_null(nb$zero_rfx_samples) + bcf_on <- fit_small_bcf(d, num_gfr = 0, num_mcmc = 2) + expect_true(bcf_on$model_params$has_zero_rfx) +}) + +# --------------------------------------------------------------------------- +# Prediction, posterior predictive, unseen groups +# --------------------------------------------------------------------------- + +test_that("ZINB bart() predictions put the zero-part random intercept in pi", { + skip_on_cran() + d <- small_zinb_rfx_data() + m <- fit_small_bart(d, seed = 2) + expect_equal(dim(m$zero_rfx_preds_train), c(200, 15)) + expect_equal(dim(m$zero_rfx_preds_test), c(40, 15)) + # Intercept-only: every row of a group carries that group's draw + beta <- zero_rfx_beta(m) + expect_equal(m$zero_rfx_preds_test, + beta[match(d$groups_test, m$rfx_unique_group_ids), ]) + + pred <- function(...) { + predict(m, d$X_test, rfx_group_ids = d$groups_test, offset = d$offset_test, ...) + } + linear <- pred() + expect_named(linear, c("y_hat", "mean_forest_predictions", "rfx_predictions", + "zero_forest_predictions", "zero_rfx_predictions")) + expect_equal(linear$zero_rfx_predictions, m$zero_rfx_preds_test) + # zero_hat_test is the full probit index c0 + g(x) + u_z + expect_equal(linear$zero_forest_predictions + linear$zero_rfx_predictions, + m$zero_hat_test) + pi <- pnorm(m$zero_hat_test) + + zero_probability <- pred(scale = "zero_probability") + expect_named(zero_probability, c("y_hat", "zero_forest_predictions", + "zero_rfx_predictions")) + expect_equal(zero_probability$y_hat, pi) + expect_equal(zero_probability$zero_forest_predictions, + pnorm(linear$zero_forest_predictions)) + # The zero-part term stays on the probit scale on every scale + expect_equal(zero_probability$zero_rfx_predictions, linear$zero_rfx_predictions) + + response <- pred(scale = "response") + expect_equal(response$y_hat, (1 - pi) * exp(linear$y_hat)) + expect_equal(pred(terms = "zero_rfx", type = "mean"), + rowMeans(linear$zero_rfx_predictions)) + expect_equal(extractParameter(m, "zero_probability_train"), pnorm(m$zero_hat_train)) + + # pi needs the group labels once the zero part has random effects + expect_error( + predict(m, d$X_test, terms = "y_hat", scale = "zero_probability"), + "zero part has random effects" + ) + # ... but the zero forest alone does not + expect_equal( + predict(m, d$X_test, terms = "zero_forest"), + linear$zero_forest_predictions + ) + expect_output(print(m), "zero-part random effects") + expect_output(print(m), "trees and random effects") +}) + +test_that("ZINB bart() posterior predictive draws use pi with the zero-part intercept", { + skip_on_cran() + d <- small_zinb_rfx_data() + m <- fit_small_bart(d, seed = 3, num_mcmc = 40) + ppd <- sampleBARTPosteriorPredictive(m, X = d$X_test, + rfx_group_ids = d$groups_test, offset = d$offset_test, + num_draws_per_sample = 50) + expect_equal(dim(ppd), c(40, 40, 50)) + pi <- predict(m, d$X_test, rfx_group_ids = d$groups_test, terms = "y_hat", + scale = "zero_probability") + mu <- predict(m, d$X_test, rfx_group_ids = d$groups_test, + offset = d$offset_test, terms = "y_hat", scale = "rate") + r <- matrix(m$negbin_dispersion_samples, nrow(mu), ncol(mu), byrow = TRUE) + expect_equal(mean(ppd), mean((1 - pi) * mu), tolerance = 0.05) + expect_equal( + mean(ppd == 0), + mean(pi + (1 - pi) * (r / (r + mu))^r), + tolerance = 0.03 + ) + # Rows in groups with a high zero-part intercept get more zeros + u <- rowMeans(m$zero_rfx_preds_test) + hi <- u > median(u) + p0 <- pi + (1 - pi) * (r / (r + mu))^r + ppd_gap <- mean(ppd[hi, , ] == 0) - mean(ppd[!hi, , ] == 0) + expect_gt(ppd_gap, 0) + expect_lt(abs(ppd_gap - (mean(p0[hi, ]) - mean(p0[!hi, ]))), 0.03) +}) + +test_that("Unseen groups get the mean random effect on both parts in bart()", { + skip_on_cran() + d <- small_zinb_rfx_data() + d$groups_test[1:5] <- "new" + m <- fit_small_bart(d, seed = 4, rfx = list(unseen_groups = "mean")) + count_beta <- m$rfx_samples$extract_parameter_samples()$beta_samples + zero_beta <- zero_rfx_beta(m) + for (i in 1:5) { + expect_equal(m$rfx_preds_test[i, ], colMeans(count_beta) * m$model_params$outcome_scale) + expect_equal(m$zero_rfx_preds_test[i, ], colMeans(zero_beta)) + } + p <- predict(m, d$X_test, rfx_group_ids = d$groups_test, offset = d$offset_test) + expect_equal(p$zero_rfx_predictions, m$zero_rfx_preds_test) + expect_equal(p$zero_forest_predictions + p$zero_rfx_predictions, m$zero_hat_test) + zp <- predict(m, d$X_test, rfx_group_ids = d$groups_test, terms = "y_hat", + scale = "zero_probability") + expect_equal(zp[1:5, ], pnorm(p$zero_forest_predictions[1:5, ] + + matrix(colMeans(zero_beta), 5, ncol(zero_beta), byrow = TRUE))) + + # With the default policy, unseen labels are refused for both parts + m_err <- fit_small_bart(small_zinb_rfx_data(), seed = 4) + expect_error( + predict(m_err, d$X_test, rfx_group_ids = d$groups_test, terms = "zero_rfx"), + "must have been present" + ) +}) + +test_that("ZINB bcf() predictions and contrasts put the zero-part random intercept in pi", { + skip_on_cran() + d <- small_zinb_rfx_data() + m <- fit_small_bcf(d, seed = 6) + expect_equal(dim(m$zero_rfx_preds_train), c(200, 15)) + expect_equal(dim(m$zero_rfx_preds_test), c(40, 15)) + expect_equal(m$zero_rfx_preds_test, + zero_rfx_beta(m)[match(d$groups_test, m$rfx_unique_group_ids), ]) + expect_equal( + m$zero_hat_test, + m$zero_mu_hat_test + m$zero_tau_hat_test * d$Z_test + m$zero_rfx_preds_test + ) + pred <- function(...) { + predict(m, d$X_test, d$Z_test, d$propensity_test, + rfx_group_ids = d$groups_test, offset = d$offset_test, ...) + } + zp <- pred(scale = "zero_probability") + expect_named(zp, c("y_hat", "zero_prognostic", "zero_tau", "zero_forest", + "zero_rfx_predictions")) + pi <- pnorm(m$zero_hat_test) + expect_equal(zp$y_hat, pi) + expect_equal(zp$zero_rfx_predictions, m$zero_rfx_preds_test) + rate <- pred(terms = "y_hat", scale = "rate") + expect_equal(pred(terms = "y_hat", scale = "response"), (1 - pi) * rate) + expect_equal(pred(terms = "zero_rfx", scale = "rate", type = "mean"), + rowMeans(m$zero_rfx_preds_test)) + + # Response-scale contrast: the group intercept enters pi in both arms + n_test <- nrow(d$X_test) + contrast <- computeContrastBCFModel(m, d$X_test, d$X_test, rep(0, n_test), + rep(1, n_test), d$propensity_test, d$propensity_test, d$groups_test, + d$groups_test, matrix(1, n_test, 1), matrix(1, n_test, 1), + scale = "response", offset_0 = d$offset_test, offset_1 = d$offset_test) + arm <- function(z) { + predict(m, d$X_test, rep(z, n_test), d$propensity_test, + rfx_group_ids = d$groups_test, offset = d$offset_test, terms = "y_hat", + scale = "response") + } + expect_equal(contrast, arm(1) - arm(0)) + pi0 <- pnorm(m$zero_mu_hat_test + m$zero_rfx_preds_test) + expect_equal( + arm(0), + (1 - pi0) * predict(m, d$X_test, rep(0, n_test), d$propensity_test, + rfx_group_ids = d$groups_test, offset = d$offset_test, terms = "y_hat", + scale = "rate") + ) + + ppd <- sampleBCFPosteriorPredictive(m, d$X_test, d$Z_test, d$propensity_test, + rfx_group_ids = d$groups_test, offset = d$offset_test, + num_draws_per_sample = 50) + expect_equal(dim(ppd), c(40, 15, 50)) + expect_equal(mean(ppd == 0), + mean(pi + (1 - pi) * (m$negbin_dispersion_samples[col(pi)] / + (m$negbin_dispersion_samples[col(pi)] + rate))^m$negbin_dispersion_samples[col(pi)]), + tolerance = 0.05) + + # bcf() has no unseen-group policy: unseen labels are refused, as for the + # count part + expect_error( + predict(m, d$X_test, d$Z_test, d$propensity_test, + rfx_group_ids = replace(d$groups_test, 1, "new"), terms = "zero_rfx"), + "must have been present" + ) + expect_output(print(m), "zero-part random effects") +}) + +# --------------------------------------------------------------------------- +# Serialization, multiple chains, warm starts +# --------------------------------------------------------------------------- + +test_that("ZINB bart() with zero-part random effects survives JSON round trips", { + skip_on_cran() + d <- small_zinb_rfx_data() + m <- fit_small_bart(d, seed = 7, num_chains = 2) + all_terms <- function(model) { + predict(model, d$X_test, rfx_group_ids = d$groups_test, offset = d$offset_test) + } + pi_of <- function(model) { + predict(model, d$X_test, rfx_group_ids = d$groups_test, terms = "y_hat", + scale = "zero_probability") + } + json_file <- tempfile(fileext = ".json") + on.exit(unlink(json_file)) + saveBARTModelToJsonFile(m, json_file) + round_trips <- list( + string = createBARTModelFromJsonString(saveBARTModelToJsonString(m)), + file = createBARTModelFromJsonFile(json_file), + object = createBARTModelFromJson(saveBARTModelToJson(m)), + combined = createBARTModelFromCombinedJson(list(saveBARTModelToJson(m))), + combined_string = createBARTModelFromCombinedJsonString( + list(saveBARTModelToJsonString(m)) + ) + ) + for (name in names(round_trips)) { + rt <- round_trips[[name]] + expect_true(rt$model_params$has_zero_rfx, label = name) + expect_identical( + rt$zero_rfx_samples$extract_parameter_samples(), + m$zero_rfx_samples$extract_parameter_samples(), + label = name + ) + expect_identical( + rt$rfx_samples$extract_parameter_samples(), + m$rfx_samples$extract_parameter_samples(), + label = name + ) + expect_identical(all_terms(rt), all_terms(m), label = name) + expect_identical(pi_of(rt), pi_of(m), label = name) + } + # The zero part's draws are not the count part's + expect_false(isTRUE(all.equal( + m$zero_rfx_samples$extract_parameter_samples()$beta_samples, + m$rfx_samples$extract_parameter_samples()$beta_samples + ))) + # Models saved without the field load as having no zero-part random effects + off <- fit_small_bart(d, seed = 7, general = list(zero_random_effects = FALSE)) + js <- saveBARTModelToJson(off) + expect_false(createBARTModelFromJson(js)$model_params$has_zero_rfx) +}) + +test_that("ZINB bcf() with zero-part random effects survives JSON round trips", { + skip_on_cran() + d <- small_zinb_rfx_data() + m <- fit_small_bcf(d, seed = 8, num_chains = 2) + all_terms <- function(model, scale) { + predict(model, d$X_test, d$Z_test, d$propensity_test, + rfx_group_ids = d$groups_test, offset = d$offset_test, scale = scale) + } + json_file <- tempfile(fileext = ".json") + on.exit(unlink(json_file)) + saveBCFModelToJsonFile(m, json_file) + round_trips <- list( + string = createBCFModelFromJsonString(saveBCFModelToJsonString(m)), + file = createBCFModelFromJsonFile(json_file), + object = createBCFModelFromJson(saveBCFModelToJson(m)), + combined = createBCFModelFromCombinedJson(list(saveBCFModelToJson(m))), + combined_string = createBCFModelFromCombinedJsonString( + list(saveBCFModelToJsonString(m)) + ) + ) + for (name in names(round_trips)) { + rt <- round_trips[[name]] + expect_true(rt$model_params$has_zero_rfx, label = name) + expect_identical( + rt$zero_rfx_samples$extract_parameter_samples(), + m$zero_rfx_samples$extract_parameter_samples(), + label = name + ) + expect_identical(all_terms(rt, "linear"), all_terms(m, "linear"), label = name) + expect_identical(all_terms(rt, "zero_probability"), + all_terms(m, "zero_probability"), label = name) + } +}) + +test_that("Zero-part random effects have the right shapes across chains, combined models and warm starts", { + skip_on_cran() + d <- small_zinb_rfx_data() + m3 <- fit_small_bart(d, seed = 9, num_chains = 3, num_gfr = 6, num_mcmc = 10) + expect_equal(m3$model_params$num_samples, 30) + expect_equal(dim(zero_rfx_beta(m3)), c(8, 30)) + expect_equal(dim(m3$zero_rfx_preds_train), c(200, 30)) + expect_equal(dim(m3$zero_hat_train), c(200, 30)) + m_root <- fit_small_bart(d, seed = 10, num_chains = 2, num_gfr = 0, num_mcmc = 10) + expect_equal(dim(zero_rfx_beta(m_root)), c(8, 20)) + + combined <- createBARTModelFromCombinedJsonString(list( + saveBARTModelToJsonString(m3), saveBARTModelToJsonString(m_root) + )) + expect_equal(combined$model_params$num_samples, 50) + expect_equal(zero_rfx_beta(combined), cbind(zero_rfx_beta(m3), zero_rfx_beta(m_root))) + expect_equal( + predict(combined, d$X_test, rfx_group_ids = d$groups_test, terms = "zero_rfx"), + cbind(m3$zero_rfx_preds_test, m_root$zero_rfx_preds_test) + ) + m_off <- fit_small_bart(d, seed = 11, general = list(zero_random_effects = FALSE)) + expect_error( + createBARTModelFromCombinedJsonString(list( + saveBARTModelToJsonString(m3), saveBARTModelToJsonString(m_off) + )), + "with and without zero-part random effects" + ) + + warm <- fit_small_bart(d, seed = 12, num_gfr = 0, num_mcmc = 5, + previous_model_json = saveBARTModelToJsonString(m3), + previous_model_warmstart_sample_num = 30) + expect_equal(dim(zero_rfx_beta(warm)), c(8, 5)) + expect_warning( + fit_small_bart(d, seed = 12, num_gfr = 0, num_mcmc = 5, + previous_model_json = saveBARTModelToJsonString(m_off), + previous_model_warmstart_sample_num = 10), + "zero-part random effects samples" + ) + + b3 <- fit_small_bcf(d, seed = 13, num_chains = 3, num_gfr = 6, num_mcmc = 10) + expect_equal(dim(zero_rfx_beta(b3)), c(8, 30)) + expect_equal(dim(b3$zero_rfx_preds_train), c(200, 30)) + b_root <- fit_small_bcf(d, seed = 14, num_chains = 2, num_gfr = 0, num_mcmc = 10) + b_combined <- createBCFModelFromCombinedJsonString(list( + saveBCFModelToJsonString(b3), saveBCFModelToJsonString(b_root) + )) + expect_equal(zero_rfx_beta(b_combined), cbind(zero_rfx_beta(b3), zero_rfx_beta(b_root))) + b_warm <- fit_small_bcf(d, seed = 15, num_gfr = 0, num_mcmc = 5, + previous_model_json = saveBCFModelToJsonString(b3), + previous_model_warmstart_sample_num = 30) + expect_equal(dim(zero_rfx_beta(b_warm)), c(8, 5)) +}) diff --git a/tests/testthat/test-zinb.R b/tests/testthat/test-zinb.R index 8e321f74..769ca2b3 100644 --- a/tests/testthat/test-zinb.R +++ b/tests/testthat/test-zinb.R @@ -57,6 +57,8 @@ small_zinb_data <- function(seed = 3, n = 300, n_test = 50) { ) } +# The zero part has no random effects here (general_params$zero_random_effects +# = FALSE), as before they existed; test-zinb-rfx.R covers them. fit_small_zinb <- function(d, seed, num_chains = 1, num_gfr = 5, ...) { bart( d$X, d$y, X_test = d$X_test, @@ -65,7 +67,7 @@ fit_small_zinb <- function(d, seed, num_chains = 1, num_gfr = 5, ...) { random_effects_params = list(model_spec = "intercept_only"), num_gfr = num_gfr, num_burnin = 10, num_mcmc = 20, general_params = list(outcome_model = zinb_model(), random_seed = seed, - num_chains = num_chains), + num_chains = num_chains, zero_random_effects = FALSE), zero_forest_params = list(num_trees = 20, ...) ) } @@ -453,7 +455,8 @@ test_that("ZINB multi-chain sampling, combined JSON and warm starts have the rig random_effects_params = list(model_spec = "intercept_only"), previous_model_json = saveBARTModelToJsonString(m3), previous_model_warmstart_sample_num = 60, - general_params = list(outcome_model = zinb_model(), random_seed = 14), + general_params = list(outcome_model = zinb_model(), random_seed = 14, + zero_random_effects = FALSE), zero_forest_params = list(num_trees = 20, sample_sigma2_leaf = TRUE)) expect_equal(dim(warm$zero_hat_train), c(300, 5)) expect_error(