From 9c1f4e9f29c5f1c6cb293c4d6800b1109faf2054 Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Thu, 5 Nov 2026 17:36:53 +1030 Subject: [PATCH 01/16] feat: known-truth controls (control_spec, controls, control_table) A control spec declares what a control removes and plants: the column to permute (default the single treatment), the design strata and unit of the restricted permutation, the unit-grained columns, the numeric targets and the per-level shift in target-SD units, the replicate count and the seed. controls() resolves it against any table with a role table and control_table() regenerates the null, a negative replicate or a positive replicate from its seed. Nothing is re-simulated. The null permutes every analysed column at its grain; the negative permutes the declared column within the strata at its unit; the positive is the same permutation plus the shift, so positive minus negative is the planted truth exactly. A whole-valued target takes a per-level whole shift and a shift that rounds away is refused. The planted vector comes from the spec's seed; a private run seed, for the owner's side, changes the permutations only. Refusals by class for a degenerate permutation, strata not nested above the unit, a target of the wrong kind or grain, an event column as target and a missing seed; both guards were falsified. 314 new expectations, full suite 3494/0. Vignette "Known-truth controls" on john_alpha, both sides. --- DESCRIPTION | 2 +- NAMESPACE | 3 + NEWS.md | 12 + R/control_spec.R | 293 +++++++++ R/controls.R | 753 ++++++++++++++++++++++++ README.md | 19 + _pkgdown.yml | 8 + inst/WORDLIST | 70 +-- man/control_spec.Rd | 83 +++ man/control_table.Rd | 43 ++ man/controls.Rd | 106 ++++ tests/testthat/helper-controls.R | 109 ++++ tests/testthat/test-controls-events.R | 33 ++ tests/testthat/test-controls-negative.R | 141 +++++ tests/testthat/test-controls-null.R | 111 ++++ tests/testthat/test-controls-positive.R | 138 +++++ tests/testthat/test-controls-spec.R | 83 +++ tests/testthat/test-controls-table.R | 73 +++ vignettes/controls.Rmd | 232 ++++++++ 19 files changed, 2277 insertions(+), 35 deletions(-) create mode 100644 R/control_spec.R create mode 100644 R/controls.R create mode 100644 man/control_spec.Rd create mode 100644 man/control_table.Rd create mode 100644 man/controls.Rd create mode 100644 tests/testthat/helper-controls.R create mode 100644 tests/testthat/test-controls-events.R create mode 100644 tests/testthat/test-controls-negative.R create mode 100644 tests/testthat/test-controls-null.R create mode 100644 tests/testthat/test-controls-positive.R create mode 100644 tests/testthat/test-controls-spec.R create mode 100644 tests/testthat/test-controls-table.R create mode 100644 vignettes/controls.Rmd diff --git a/DESCRIPTION b/DESCRIPTION index 67a301c..40c7f99 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: masque Title: Structurally Faithful Development Surrogates for Tabular Data -Version: 0.16.0 +Version: 0.16.0.9000 Authors@R: c( person("Max", "Moldovan", email = "max.moldovan@gmail.com", diff --git a/NAMESPACE b/NAMESPACE index 16104aa..6cf6da1 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -6,6 +6,9 @@ export(apply_recipe) export(audit_mask) export(clean_table) export(conform_table) +export(control_spec) +export(control_table) +export(controls) export(data_card) export(detect_design) export(jitter_coordinates) diff --git a/NEWS.md b/NEWS.md index c31d076..62a54b1 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,15 @@ +# masque (development version) + +## New features + +* Known-truth controls. `control_spec()` declares what a control removes and + plants; `controls()` derives a null, `n` negative and `n` positive control + datasets from any table with a role table, by restricted permutation at + the declared unit within declared design strata and a per-level shift in + target-SD units; `control_table()` regenerates any of them from its seed. + The same spec runs on the clone and, with a private seed, on the + original. Vignette "Known-truth controls". + # masque 0.16.0 ## Behaviour changes diff --git a/R/control_spec.R b/R/control_spec.R new file mode 100644 index 0000000..339fae9 --- /dev/null +++ b/R/control_spec.R @@ -0,0 +1,293 @@ +# control_spec.R -- control_spec(): the declaration behind controls(). +# +# A control spec names the column whose effect a negative control removes, +# the outcome a positive control plants an effect on, the effect per level in +# outcome-SD units, the design strata and unit of the restricted permutation, +# the unit-grained columns, the replicate count and the seed. It is declared +# before any control table exists and is the record of what was planted. + +#' Internal: the masque_control_spec S7 class +#' +#' A container for one validated declaration; constructed by +#' [control_spec()] only. +#' +#' @keywords internal +#' @noRd +masque_control_spec <- S7::new_class( + "masque_control_spec", + properties = list( + on = S7::new_property( + class = S7::new_union(S7::class_character, NULL), default = NULL + ), + target = S7::new_property( + class = S7::new_union(S7::class_character, NULL), default = NULL + ), + # A named numeric vector of level shifts in outcome-SD units, or + # `list(sd = )` until the spec meets a table and the vector is drawn. + effect = S7::new_property( + class = S7::new_union(S7::class_list, S7::class_numeric, NULL), + default = NULL + ), + strata = S7::new_property( + class = S7::class_character, default = character() + ), + unit = S7::new_property(class = S7::class_character, default = character()), + grain = S7::new_property(class = S7::class_list, default = list()), + n = S7::new_property(class = S7::class_integer, default = 1L), + seed = S7::class_integer + ) +) + +#' Declare the controls to derive from a table +#' +#' A control spec says what [controls()] removes and plants: the column +#' whose effect the negative control removes (`on`), the outcome the +#' positive control plants an effect on (`target`), the planted effect per +#' level of `on` in outcome-SD units (`effect`), the design strata within +#' which `on` is permuted and the unit it is permuted at, the other +#' unit-grained columns, the number of replicates and the seed. The spec is +#' declared once, before any control table exists, and is the record of +#' what was planted; print it to read the declaration. +#' +#' The null control needs nothing beyond the seed. The negative control +#' needs `on`, which defaults to the single treatment column of the table. +#' The positive control needs `target` and `effect` together. +#' +#' @param on The name of the column whose association with the rest of the +#' table the negative control removes, by permuting it within `strata` at +#' its `unit`. Must be a treatment or covariate column, categorical or +#' whole-valued. `NULL` (the default) means the single treatment column of +#' the table; a table with several or no treatment columns needs `on` +#' named. +#' @param target The name of one or more numeric outcome columns the +#' positive control plants the effect on, or `NULL` for no positive +#' control. A logical, factor, date, event or two-valued column is +#' refused when the spec meets the table. +#' @param effect The planted effect per level of `on`, in units of the +#' target's standard deviation in the table at hand: a named numeric +#' vector (`c(G01 = 0.5, G02 = -0.5)`; a level not named shifts by zero), +#' or `list(sd = s)` to draw one shift per level from a normal +#' distribution with standard deviation `s`, under the spec's seed, when +#' the spec first meets a table. Required with `target`. +#' @param strata Design columns within which `on` is permuted, or `NULL` +#' for no stratification. Every unit of `on` must lie within one stratum. +#' @param unit Design columns within which `on` is constant, so that it is +#' permuted across those units rather than across rows; `NULL` (the +#' default) permutes rows. +#' @param grain A named list giving, for other analysed columns constant +#' within a design unit, the unit columns: `list(rain = "trial")`. Such a +#' column is permuted across its units in the null control, so it stays +#' constant within each unit. +#' @param n The number of negative (and positive) replicates. +#' @param seed The seed of the control draws. Required, so that a control is +#' reproducible from its declaration. +#' +#' @returns A `masque_control_spec` object, with a print method. +#' +#' @examples +#' spec <- control_spec( +#' on = "gen", target = "yield", effect = c(G01 = 0.5, G02 = -0.5), +#' strata = c("rep", "block"), n = 19L, seed = 2026L +#' ) +#' spec +#' +#' @seealso [controls()], [control_table()]. +#' +#' @export +control_spec <- function( + on = NULL, + target = NULL, + effect = NULL, + strata = NULL, + unit = NULL, + grain = NULL, + n = 1L, + seed = NULL +) { + .check_control_seed(seed) + on <- .check_control_names(on, "on", single = TRUE) + target <- .check_control_names(target, "target", single = FALSE) + effect <- .check_control_effect(effect, target) + strata <- .check_control_cols(strata, "strata") + unit <- .check_control_cols(unit, "unit") + grain <- .check_control_grain(grain) + n <- .check_control_n(n) + masque_control_spec( + on = on, + target = target, + effect = effect, + strata = strata, + unit = unit, + grain = grain, + n = n, + seed = as.integer(seed) + ) +} + +# Refusal helper ------------------------------------------------------------- + +# A classed refusal: `masque_control__refusal` under the shared marker. +# The message is interpolated in the caller's frame, where its values live. +.control_refuse <- function(msg, what, .envir = parent.frame()) { + cli::cli_abort( + msg, + class = c(paste0("masque_control_", what, "_refusal"), "orchestra_refusal"), + .envir = .envir + ) +} + +# Argument checks ------------------------------------------------------------ + +# The seed is required: a control without a seed cannot be declared. +.check_control_seed <- function(seed) { + if (is.null(seed)) { + .control_refuse( + c( + "`seed` is required for a control spec.", + i = "A control is reproducible from its declaration, so the seed is part of it." + ), + "seed" + ) + } + .check_seed(seed) +} + +# `NULL`, or a character vector of column names, single when `single`. +.check_control_names <- function(x, arg, single) { + if (is.null(x)) { + return(NULL) + } + ok <- is.character(x) && length(x) >= 1L && !anyNA(x) && all(nzchar(x)) && + !anyDuplicated(x) && (!single || length(x) == 1L) + if (!ok) { + what <- if (single) "a single column name" else "one or more column names" + .control_refuse("`{arg}` must be {what}, or `NULL`.", arg) + } + x +} + +# `NULL` or a character vector of design column names. +.check_control_cols <- function(x, arg) { + if (is.null(x)) { + return(character()) + } + .check_control_names(x, arg, single = FALSE) +} + +# `effect` and `target` come together: a named numeric vector of finite +# level shifts, or `list(sd = s)` with a single positive `s`. +.check_control_effect <- function(effect, target) { + if (is.null(effect) && is.null(target)) { + return(NULL) + } + if (is.null(effect) || is.null(target)) { + .control_refuse( + "`target` and `effect` are declared together: a positive control needs both.", + "effect" + ) + } + if (is.numeric(effect)) { + nms <- names(effect) + ok <- length(effect) >= 1L && !is.null(nms) && all(nzchar(nms)) && + !anyDuplicated(nms) && all(is.finite(effect)) + if (!ok) { + .control_refuse( + "A numeric `effect` must be a named vector of finite level shifts, one name per level of `on`.", + "effect" + ) + } + return(stats::setNames(as.double(effect), nms)) + } + if (is.list(effect)) { + ok <- identical(names(effect), "sd") && is.numeric(effect$sd) && + length(effect$sd) == 1L && is.finite(effect$sd) && effect$sd > 0 + if (ok) { + return(list(sd = as.double(effect$sd))) + } + } + .control_refuse( + "`effect` must be a named numeric vector of level shifts or `list(sd = s)` with a single positive `s`.", + "effect" + ) +} + +# `NULL` or a named list of character vectors: column -> its unit columns. +.check_control_grain <- function(grain) { + if (is.null(grain)) { + return(list()) + } + nms <- names(grain) + ok <- is.list(grain) && length(grain) >= 1L && !is.null(nms) && + all(nzchar(nms)) && !anyDuplicated(nms) && + all(vapply(grain, function(u) { + is.character(u) && length(u) >= 1L && !anyNA(u) && all(nzchar(u)) + }, logical(1L))) + if (!ok) { + .control_refuse( + "`grain` must be a named list of unit column names, as in `list(rain = \"trial\")`.", + "grain" + ) + } + grain +} + +# A single whole number of replicates, at least one. +.check_control_n <- function(n) { + ok <- is.numeric(n) && length(n) == 1L && !is.na(n) && is.finite(n) && + n == round(n) && n >= 1L + if (!ok) { + .control_refuse("`n` must be a single whole number of replicates, at least 1.", "n") + } + as.integer(n) +} + +# Print ---------------------------------------------------------------------- + +# local(): see the plot method in detect_plot.R. +local(S7::method(print, masque_control_spec) <- function(x, ...) { + cli::cli_h1("masque_control_spec") + cli::cli_bullets(c( + "*" = sprintf("Permuted column (on): %s", .spec_on_line(x)), + "*" = sprintf("Planted-effect target: %s", .spec_target_line(x)), + "*" = sprintf("Effect (target-SD units): %s", .spec_effect_line(x)), + "*" = sprintf("Strata: %s", .spec_cols_line(x@strata, "none")), + "*" = sprintf("Unit of on: %s", .spec_cols_line(x@unit, "row")), + "*" = sprintf("Unit-grained columns: %s", .spec_grain_line(x)), + "*" = sprintf("Replicates: %d", x@n), + "*" = "Seed: present" + )) + invisible(x) +}) + +.spec_on_line <- function(x) { + if (is.null(x@on)) "the treatment column of the table" else x@on +} + +.spec_target_line <- function(x) { + if (is.null(x@target)) "none" else paste(x@target, collapse = ", ") +} + +.spec_effect_line <- function(x) { + if (is.null(x@effect)) { + return("none (no positive control)") + } + if (is.list(x@effect)) { + return(sprintf( + "one shift per level drawn from N(0, %s^2) when the spec meets a table", + format(x@effect$sd) + )) + } + paste(sprintf("%s %+.3g", names(x@effect), x@effect), collapse = ", ") +} + +.spec_cols_line <- function(cols, empty) { + if (length(cols)) paste(cols, collapse = " + ") else empty +} + +.spec_grain_line <- function(x) { + if (!length(x@grain)) { + return("none declared") + } + units <- vapply(x@grain, paste, character(1L), collapse = " + ") + paste(sprintf("%s within %s", names(x@grain), units), collapse = "; ") +} diff --git a/R/controls.R b/R/controls.R new file mode 100644 index 0000000..46a2d00 --- /dev/null +++ b/R/controls.R @@ -0,0 +1,753 @@ +# controls.R -- controls() and control_table(): null, negative and positive +# control datasets derived from a table by restricted permutation and planted +# effects. +# +# A control is a function of the table, its role table, a control spec and a +# seed; nothing is re-simulated. The null permutes every analysed column, the +# negative permutes one declared column within design strata at its unit, and +# the positive adds a declared per-level shift to that negative. The controls +# object holds the table, the resolved spec, one seed per dataset and the +# manifest; control_table() regenerates any dataset from its seed. + +#' Internal: the masque_controls S7 class +#' +#' The resolved declaration plus everything needed to regenerate a control +#' table: the owned table, the validated roles, the seeds, the analysed +#' columns, the event pairs, the per-row stratum and unit ids, the unit ids +#' of grained columns, the manifest, the truth table and the notes. +#' +#' @keywords internal +#' @noRd +masque_controls <- S7::new_class( + "masque_controls", + properties = list( + table = S7::class_data.frame, + roles = S7::class_data.frame, + spec = masque_control_spec, + seeds = S7::class_integer, + analysed = S7::class_character, + events = S7::class_list, + strata_id = S7::class_integer, + unit_id = S7::class_integer, + grain_units = S7::class_list, + manifest = S7::class_data.frame, + truth = S7::class_data.frame, + notes = S7::class_character + ) +) + +# Offset that roots the planted-effect draw away from the permutation stream. +.CONTROL_SALT <- 7919L + +#' Derive null, negative and positive controls from a table +#' +#' Builds the control datasets a [control_spec()] declares, from any table +#' with a role table: a synthetic clone, a comparator's clone, or, on the +#' owner's side, the original. Nothing is re-simulated. Three kinds of +#' control exist, each with a known truth: +#' +#' * `null`: every analysed column (outcome, covariate and date columns +#' with action `scramble` or `alias`) is permuted independently over its +#' non-missing rows; a column declared unit-grained is permuted across its +#' units and stays constant within each. Design, treatment and kept +#' columns, the NA mask and every column's multiset of values are kept. +#' No association among analysed columns, or with the treatment, remains. +#' Use it as a smoke test and to expose leakage through analysed columns. +#' * `negative`: the declared column `on` alone is permuted within the +#' declared design strata, at its unit, so its effect on everything is +#' exactly absent while the design structure and all other associations +#' are kept. With `n` replicates it gives a permutation null for a whole +#' pipeline: a false-positive rate, a permutation p-value. +#' * `positive`: the same permutation as the negative replicate, plus the +#' declared shift per level of `on` on each target, in units of the +#' target's standard deviation in the table at hand. Positive minus +#' negative is the planted truth, exactly. +#' +#' @section Declaration before data: +#' The spec is resolved against the table here: `on` defaults to the single +#' treatment column; strata and unit are checked for nesting; an `effect` +#' given as `list(sd = )` is drawn under the spec's seed, once, so the same +#' spec plants the same vector on every table it meets. The resolved spec +#' and the truth table are on the returned object and in its print. +#' +#' @section Two sides, two seeds: +#' An analyst who holds only the clone runs this on the clone. The owner +#' runs it on the original, offline, with a private `seed` that overrides +#' the spec's seed for the permutations (the planted vector still comes +#' from the spec's seed). A public seed would let anyone who knows the +#' design, which the clone shares, rebuild every permutation and read the +#' original outcome out of per-replicate statistics. Control tables built +#' from an original never leave the owner; what leaves is an aggregate +#' verdict about the pipeline, through the owner's usual output checking. +#' +#' @section Limits: +#' A kept column is structure and is never permuted, so leakage through +#' kept columns, ids or row order passes the null. A planted effect is +#' clean: uniform within a level, with no confounding and no interaction. +#' Nothing is clipped, so a shift can take a non-negative outcome below +#' zero; [control_table()] warns when it does. The null collapses the +#' design means of row-grained outcomes, so false-positive calibration +#' comes from the negative control, not the null. +#' +#' @param df The table to derive controls from: a data frame or data.table. +#' @param roles Its role table, as returned by [roles()] for a clone or by +#' [propose_roles()] and [set_role()] for any table. +#' @param spec A [control_spec()]. +#' @param events Linked time/status pairs among the outcome columns, in the +#' form [mask()] takes, so that the null permutes each pair as one block; +#' `NULL` when there are none. +#' @param seed A private seed for the permutations that overrides the +#' spec's seed, for the owner's side; `NULL` uses the spec's seed. +#' +#' @returns A `masque_controls` object: the resolved spec (`@spec`), the +#' manifest (`@manifest`, one row per dataset), the truth table +#' (`@truth`, one row per target and level of `on`) and the notes +#' (`@notes`). Datasets are produced by [control_table()]. +#' +#' @examples +#' trial <- read.csv(system.file("extdata", "john_alpha.csv", package = "masque")) +#' r <- propose_roles(trial, detect = FALSE) +#' r <- set_role(r, c("rep", "block"), role = "design", action = "keep") +#' r <- set_role(r, "gen", role = "treatment", action = "keep") +#' r <- set_role(r, "yield", role = "outcome", action = "scramble") +#' r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") +#' spec <- control_spec( +#' on = "gen", target = "yield", effect = list(sd = 0.5), +#' strata = c("rep", "block"), n = 5L, seed = 2026L +#' ) +#' ctl <- controls(trial, r, spec) +#' ctl +#' head(control_table(ctl, "positive", 1L)) +#' +#' @seealso [control_spec()], [control_table()], [mask()], [roles()]. +#' +#' @export +controls <- function(df, roles, spec, events = NULL, seed = NULL) { + withr::local_preserve_seed() + .check_control_table(df) + .check_control_spec_class(spec) + if (!is.null(seed)) { + .check_seed(seed) + } + df <- .as_owned_dt(df) + roles <- roles_validate(roles, df = df) + + # Resolve the declaration against the table ------------------------------ + res <- .resolve_control_spec(spec, df, roles) + events <- .normalise_events(events, df, roles) + .check_control_target_events(res$spec, events) + + # Seeds: one for the null, one per replicate ----------------------------- + run_seed <- if (is.null(seed)) spec@seed else as.integer(seed) + seeds <- as.integer(.sub_seeds(run_seed, res$spec@n + 1L)) + + manifest <- .control_manifest(res, seeds) + masque_controls( + table = df, + roles = .as_owned_dt(roles), + spec = res$spec, + seeds = seeds, + analysed = res$analysed, + events = events, + strata_id = res$strata_id, + unit_id = res$unit_id, + grain_units = res$grain_units, + manifest = manifest, + truth = res$truth, + notes = res$notes + ) +} + +#' Produce one control dataset +#' +#' Regenerates a control dataset of a [controls()] object from its seed: the +#' null, or negative or positive replicate `i`. Positive replicate `i` is +#' negative replicate `i` plus the planted shift. The result carries a +#' `masque_control` attribute naming its kind and replicate; its values +#' alone do not tell it from the table it came from, so keep the manifest +#' with it. +#' +#' @param ctl A `masque_controls` object from [controls()]. +#' @param kind One of `"null"`, `"negative"`, `"positive"`. +#' @param i The replicate, from 1 to `n` of the spec; ignored for the null. +#' +#' @returns A data.table with the columns, classes, NA mask and row order of +#' the input table. +#' +#' @examples +#' trial <- read.csv(system.file("extdata", "john_alpha.csv", package = "masque")) +#' r <- propose_roles(trial, detect = FALSE) +#' r <- set_role(r, c("rep", "block"), role = "design", action = "keep") +#' r <- set_role(r, "gen", role = "treatment", action = "keep") +#' r <- set_role(r, "yield", role = "outcome", action = "scramble") +#' r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") +#' spec <- control_spec(on = "gen", strata = c("rep", "block"), n = 3L, seed = 1L) +#' ctl <- controls(trial, r, spec) +#' neg <- control_table(ctl, "negative", 2L) +#' table(neg$block, neg$gen)[1:2, 1:6] +#' +#' @seealso [controls()], [control_spec()]. +#' +#' @export +control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) { + withr::local_preserve_seed() + .check_controls_class(ctl) + kind <- match.arg(kind) + i <- .check_control_index(i, ctl@spec@n) + if (identical(kind, "positive") && is.null(ctl@spec@target)) { + .control_refuse( + "The spec declares no `target`, so there is no positive control.", + "kind" + ) + } + + out <- data.table::copy(ctl@table) + if (identical(kind, "null")) { + with_rng_state(ctl@seeds[1L], .permute_analysed(out, ctl)) + } else { + with_rng_state(ctl@seeds[1L + i], .permute_on(out, ctl)) + if (identical(kind, "positive")) { + .plant_effect(out, ctl) + } + } + data.table::setattr(out, "masque_control", list( + kind = kind, + replicate = if (identical(kind, "null")) NA_integer_ else i + )) + out[] +} + +# Resolution ---------------------------------------------------------------- + +# Resolve a spec against a table and its roles: fill `on`, check every +# declared column against the roles, build the stratum and unit ids, draw or +# complete the effect vector and compute the truth table. +.resolve_control_spec <- function(spec, df, roles) { + n <- nrow(df) + notes <- character() + design <- intersect( + names(df), roles$col[roles$role == "design" & roles$action != "drop"] + ) + analysed <- intersect(names(df), roles$col[ + roles$role %in% c("outcome", "covariate", "date") & + roles$action %in% c("scramble", "alias") + ]) + if (!length(analysed)) { + .control_refuse( + "The table has no analysed column: an outcome, covariate or date column with action `scramble` or `alias`.", + "analysed" + ) + } + kept_outcome <- roles$col[roles$role == "outcome" & roles$action == "keep"] + if (length(kept_outcome)) { + warning(warningCondition( + sprintf( + "Kept outcome column%s %s %s structure and %s not permuted by any control.", + if (length(kept_outcome) > 1L) "s" else "", + paste(kept_outcome, collapse = ", "), + if (length(kept_outcome) > 1L) "are" else "is", + if (length(kept_outcome) > 1L) "are" else "is" + ), + class = "masque_control_kept_outcome" + )) + } + + # The permuted column and its unit ------------------------------------ + on <- .resolve_control_on(spec@on, df, roles, spec@target) + unit <- .check_design_cols(spec@unit, "unit", design) + unit_id <- if (length(unit)) { + as.integer(.unit_id(df, unit)) + } else { + seq_len(n) + } + if (length(unit) && !.constant_within(unit_id, df[[on]])) { + .control_refuse( + "`on` ({.field {on}}) varies within its declared unit ({.field {unit}}); a unit must hold one value of `on`.", + "unit" + ) + } + + # Strata: nested above the unit, and `on` must vary somewhere ------------ + strata <- .check_design_cols(spec@strata, "strata", design) + strata_id <- if (length(strata)) { + as.integer(.unit_id(df, strata)) + } else { + rep(1L, n) + } + if (length(strata) && !.constant_within(unit_id, strata_id)) { + .control_refuse( + "A unit of `on` ({.field {unit}}) spans more than one stratum ({.field {strata}}); strata must be nested above the unit.", + "strata" + ) + } + constant_share <- .strata_constant_share(df[[on]], strata_id) + if (constant_share >= 1) { + .control_refuse( + c( + "`on` ({.field {on}}) is constant within every stratum ({.field {strata}}), so a restricted permutation changes nothing.", + i = "Coarsen the strata or drop them." + ), + "degenerate" + ) + } + if (constant_share > 0.5) { + warning(warningCondition( + sprintf( + "`on` (%s) is constant within %.0f%% of the strata; the negative control permutes little.", + on, 100 * constant_share + ), + class = "masque_control_strata_weak" + )) + } + + # Grain of other analysed columns ---------------------------------------- + grain_units <- list() + for (cl in names(spec@grain)) { + if (!cl %in% analysed) { + .control_refuse( + "`grain` names {.field {cl}}, which is not an analysed column of the table.", + "grain" + ) + } + u <- .check_design_cols(spec@grain[[cl]], "grain", design) + gid <- as.integer(.unit_id(df, u)) + if (.constant_within(gid, df[[cl]])) { + grain_units[[cl]] <- gid + } else { + notes <- c(notes, sprintf( + "%s is not constant within %s in this table; permuted at the row.", + cl, paste(u, collapse = " + ") + )) + } + } + + # Targets and the planted truth ----------------------------------------- + target <- .check_control_targets(spec@target, df, roles, on, spec@grain) + levels <- .control_levels(df[[on]]) + effect <- .resolve_control_effect(spec@effect, levels, spec@seed) + truth <- .control_truth(df, target, effect, strata_id) + + spec_out <- masque_control_spec( + on = on, + target = target, + effect = effect, + strata = strata, + unit = unit, + grain = spec@grain, + n = spec@n, + seed = spec@seed + ) + list( + spec = spec_out, + analysed = analysed, + strata_id = strata_id, + unit_id = unit_id, + n_strata = max(strata_id), + constant_share = constant_share, + grain_units = grain_units, + truth = truth, + notes = notes + ) +} + +# `on`: the single treatment column when NULL; present, a treatment or +# covariate, categorical or whole-valued, and not a target. +.resolve_control_on <- function(on, df, roles, target) { + if (is.null(on)) { + trt <- intersect( + names(df), + roles$col[roles$role == "treatment" & roles$action != "drop"] + ) + if (length(trt) != 1L) { + .control_refuse( + c( + "`on` is `NULL` and the table has {length(trt)} treatment column{?s}; name `on`.", + i = "A negative control needs exactly one column to permute." + ), + "on" + ) + } + on <- trt + } + if (!on %in% names(df)) { + .control_refuse("`on` ({.field {on}}) is not a column of the table.", "on") + } + role <- roles$role[match(on, roles$col)] + kind <- roles$kind[match(on, roles$col)] + if (!role %in% c("treatment", "covariate")) { + .control_refuse( + "`on` ({.field {on}}) has role {.val {role}}; it must be a treatment or covariate column.", + "on" + ) + } + categorical <- kind %in% .categorical_kinds() + whole <- kind %in% .numeric_kinds() && .is_whole_valued(df[[on]]) + if (!categorical && !whole) { + .control_refuse( + "`on` ({.field {on}}) must be categorical or whole-valued; a continuous column has no levels to shift.", + "on" + ) + } + if (on %in% target) { + .control_refuse("`on` ({.field {on}}) cannot also be a `target`.", "on") + } + on +} + +# Columns named for strata, unit or grain must be retained design columns. +.check_design_cols <- function(cols, arg, design) { + if (!length(cols)) { + return(character()) + } + bad <- setdiff(cols, design) + if (length(bad)) { + .control_refuse( + "`{arg}` names {.field {bad}}, which {?is/are} not {?a/} retained design column{?s} of the table.", + arg + ) + } + cols +} + +# Targets: present, outcome or covariate, numeric with more than two values, +# not `on`, and not declared unit-grained. +.check_control_targets <- function(target, df, roles, on, grain) { + if (is.null(target)) { + return(NULL) + } + missing <- setdiff(target, names(df)) + if (length(missing)) { + .control_refuse( + "`target` names {.field {missing}}, not {?a/} column{?s} of the table.", + "target" + ) + } + for (tg in target) { + role <- roles$role[match(tg, roles$col)] + kind <- roles$kind[match(tg, roles$col)] + y <- df[[tg]] + ok <- role %in% c("outcome", "covariate") && kind %in% .numeric_kinds() && + data.table::uniqueN(y, na.rm = TRUE) > 2L + if (!ok) { + .control_refuse( + "`target` {.field {tg}} must be a numeric outcome or covariate with more than two distinct values.", + "target_kind" + ) + } + if (tg %in% names(grain)) { + .control_refuse( + "`target` {.field {tg}} is declared unit-grained; a shift by level of `on` would break its constancy within the unit.", + "target_grain" + ) + } + } + if (on %in% target) { + .control_refuse("`on` ({.field {on}}) cannot also be a `target`.", "target") + } + target +} + +# A target cannot be a declared event column. +.check_control_target_events <- function(spec, events) { + if (is.null(spec@target) || !length(events)) { + return(invisible(NULL)) + } + ev <- unlist(lapply(events, function(e) c(e$time, e$status))) + bad <- intersect(spec@target, ev) + if (length(bad)) { + .control_refuse( + "`target` {.field {bad}} is a linked-event column; events take no planted shift.", + "target_kind" + ) + } + invisible(NULL) +} + +# The levels of `on` as character, in sorted order. +.control_levels <- function(x) { + as.character(sort(unique(x[!is.na(x)]))) +} + +# Share of strata in which `on` holds at most one distinct value. +.strata_constant_share <- function(x, strata_id) { + g <- V1 <- NULL + obs <- !is.na(x) + pairs <- data.table::data.table(g = strata_id, x = x) + per <- pairs[obs, data.table::uniqueN(x), by = g] + n_strata <- data.table::uniqueN(strata_id) + constant <- n_strata - nrow(per) + sum(per$V1 <= 1L) + constant / n_strata +} + +# The effect vector over the levels: drawn from `list(sd = )` under the +# spec's seed, or a named vector completed with zeros. +.resolve_control_effect <- function(effect, levels, seed) { + if (is.null(effect)) { + return(NULL) + } + if (is.list(effect)) { + root <- (as.numeric(seed) + .CONTROL_SALT) %% .Machine$integer.max + draw <- with_rng_state(root, stats::rnorm(length(levels), 0, effect$sd)) + return(stats::setNames(draw, levels)) + } + unknown <- setdiff(names(effect), levels) + if (length(unknown)) { + .control_refuse( + "`effect` names {.val {unknown}}, not {?a/} level{?s} of `on`.", + "effect" + ) + } + out <- stats::setNames(rep(0, length(levels)), levels) + out[names(effect)] <- effect + out +} + +# One row per target and level: the effect in SD units, the target's SD, +# its within-strata residual SD, and the raw and realised shift. A +# whole-valued target takes a whole shift; a shift that rounds away is +# refused. +.control_truth <- function(df, target, effect, strata_id) { + empty <- data.table::data.table( + target = character(), level = character(), effect_sd = double(), + sd_y = double(), sd_within_strata = double(), shift_raw = double(), + shift_realised = double(), whole_valued = logical() + ) + if (is.null(target)) { + return(empty) + } + rows <- lapply(target, function(tg) { + y <- df[[tg]] + sd_y <- stats::sd(y, na.rm = TRUE) + resid <- y - stats::ave(y, strata_id, FUN = function(v) mean(v, na.rm = TRUE)) + whole <- .is_whole_valued(y) + shift_raw <- unname(effect) * sd_y + shift_realised <- if (whole) round(shift_raw) else shift_raw + vanished <- whole & effect != 0 & shift_realised == 0 + if (any(vanished)) { + .control_refuse( + c( + "On the whole-valued target {.field {tg}}, the shift for level{?s} {.val {names(effect)[vanished]}} rounds to zero.", + i = "Declare a larger effect, or a target with a finer resolution." + ), + "shift_vanishes" + ) + } + data.table::data.table( + target = tg, + level = names(effect), + effect_sd = unname(effect), + sd_y = sd_y, + sd_within_strata = stats::sd(resid, na.rm = TRUE), + shift_raw = shift_raw, + shift_realised = shift_realised, + whole_valued = whole + ) + }) + data.table::rbindlist(rows) +} + +# One manifest row per dataset: what was permuted, where, under which seed. +.control_manifest <- function(res, seeds) { + spec <- res$spec + n <- spec@n + kinds <- c("null", rep("negative", n), if (!is.null(spec@target)) rep("positive", n)) + reps <- c(NA_integer_, seq_len(n), if (!is.null(spec@target)) seq_len(n)) + dataset <- ifelse(is.na(reps), kinds, paste0(kinds, "_", reps)) + seed_of <- ifelse(is.na(reps), seeds[1L], seeds[1L + ifelse(is.na(reps), 0L, reps)]) + permuted <- ifelse( + kinds == "null", paste(res$analysed, collapse = ", "), spec@on + ) + data.table::data.table( + dataset = dataset, + kind = kinds, + replicate = reps, + seed = as.integer(seed_of), + permuted = permuted, + strata = if (length(spec@strata)) paste(spec@strata, collapse = " + ") else "none", + unit = if (length(spec@unit)) paste(spec@unit, collapse = " + ") else "row", + n_strata = as.integer(res$n_strata), + strata_constant_share = res$constant_share, + target = if (is.null(spec@target)) NA_character_ else paste(spec@target, collapse = ", ") + ) +} + +# Permutation --------------------------------------------------------------- + +# Permute the non-missing values of `x` across the units given by `unit_id`, +# within the groups given by `strata_id`. Each observed unit keeps one value +# (its first non-missing one) and receives another unit's value from the same +# stratum, broadcast to its non-missing rows; an all-missing unit stays +# missing. With `unit_id = seq_along(x)` this is a row permutation. +.permute_within <- function(x, unit_id, strata_id) { + obs <- which(!is.na(x)) + if (length(obs) < 2L) { + return(x) + } + u_obs <- unique(unit_id[obs]) + first <- obs[match(u_obs, unit_id[obs])] + u_val <- x[first] + groups <- split(seq_along(u_obs), strata_id[first]) + perm <- integer(length(u_obs)) + for (g in groups) { + perm[g] <- g[sample.int(length(g))] + } + new_val <- u_val[perm] + out <- x + out[obs] <- new_val[match(unit_id[obs], u_obs)] + out +} + +# The null: every analysed column on its own draw, at its grain, no strata; +# a linked-event pair moves as one row block. +.permute_analysed <- function(out, ctl) { + n <- nrow(out) + rows <- seq_len(n) + none <- rep(1L, n) + pair_cols <- unlist(lapply(ctl@events, function(e) c(e$time, e$status))) + for (cl in setdiff(ctl@analysed, pair_cols)) { + uid <- ctl@grain_units[[cl]] + if (is.null(uid)) { + uid <- rows + } + data.table::set(out, j = cl, value = .permute_within(out[[cl]], uid, none)) + } + for (e in ctl@events) { + perm <- .permute_within(rows, rows, none) + for (cl in c(e$time, e$status)) { + data.table::set(out, j = cl, value = out[[cl]][perm]) + } + } + invisible(out) +} + +# The negative: `on` alone, within strata, at its unit. +.permute_on <- function(out, ctl) { + on <- ctl@spec@on + data.table::set( + out, + j = on, + value = .permute_within(out[[on]], ctl@unit_id, ctl@strata_id) + ) + invisible(out) +} + +# The positive: add each target's realised shift to the rows of each level +# of the permuted `on`. Nothing is clipped; a non-negative target taken +# below zero is reported. +.plant_effect <- function(out, ctl) { + on_val <- as.character(out[[ctl@spec@on]]) + crossed <- character() + for (tg in unique(ctl@truth$target)) { + tr <- ctl@truth[ctl@truth$target == tg, ] + shift <- tr$shift_realised[match(on_val, tr$level)] + shift[is.na(shift)] <- 0 + y <- out[[tg]] + obs <- !is.na(y) + y_new <- y + y_new[obs] <- y[obs] + shift[obs] + if (is.integer(y)) { + y_new <- as.integer(round(y_new)) + } + if (all(y[obs] >= 0) && any(y_new[obs] < 0)) { + crossed <- c(crossed, tg) + } + data.table::set(out, j = tg, value = y_new) + } + if (length(crossed)) { + warning(warningCondition( + sprintf( + "The planted shift took %s below zero on a target with no negative values; nothing is clipped.", + paste(crossed, collapse = ", ") + ), + class = "masque_control_sign_crossed" + )) + } + invisible(out) +} + +# Argument checks ----------------------------------------------------------- + +.check_control_table <- function(df) { + if (!is.data.frame(df)) { + .control_refuse( + "`df` must be a data frame; got {.cls {class(df)[1]}}.", "table" + ) + } + if (nrow(df) < 2L) { + .control_refuse("`df` needs at least two rows to permute.", "table") + } + if (anyDuplicated(names(df))) { + .control_refuse("`df` has duplicated column names.", "table") + } + invisible(df) +} + +.check_control_spec_class <- function(spec) { + if (!S7::S7_inherits(spec, masque_control_spec)) { + .control_refuse( + "`spec` must be a {.cls masque_control_spec} from {.fn control_spec}; got {.cls {class(spec)[1]}}.", + "spec" + ) + } + invisible(spec) +} + +.check_controls_class <- function(ctl) { + if (!S7::S7_inherits(ctl, masque_controls)) { + .control_refuse( + "`ctl` must be a {.cls masque_controls} from {.fn controls}; got {.cls {class(ctl)[1]}}.", + "kind" + ) + } + invisible(ctl) +} + +.check_control_index <- function(i, n) { + ok <- is.numeric(i) && length(i) == 1L && !is.na(i) && i == round(i) && + i >= 1L && i <= n + if (!ok) { + .control_refuse( + "`i` must be a single replicate number from 1 to {n}.", "kind" + ) + } + as.integer(i) +} + +# Print --------------------------------------------------------------------- + +# local(): see the plot method in detect_plot.R. +local(S7::method(print, masque_controls) <- function(x, ...) { + spec <- x@spec + cli::cli_h1("masque_controls") + cli::cli_bullets(c( + "*" = sprintf( + "Table: %d rows, %d columns; %d analysed column%s", + nrow(x@table), ncol(x@table), length(x@analysed), + if (length(x@analysed) == 1L) "" else "s" + ), + "*" = sprintf("Permuted column (on): %s", spec@on), + "*" = sprintf( + "Strata: %s (%d strata; on constant in %.0f%% of them)", + .spec_cols_line(spec@strata, "none"), max(x@strata_id), + 100 * x@manifest$strata_constant_share[1L] + ), + "*" = sprintf("Unit of on: %s", .spec_cols_line(spec@unit, "row")), + "*" = sprintf( + "Datasets: 1 null, %d negative, %d positive", + spec@n, if (is.null(spec@target)) 0L else spec@n + ), + "*" = "Seeds: present" + )) + if (nrow(x@truth)) { + cli::cli_h2("Planted truth (positive minus negative, per level of on)") + print(x@truth[, c( + "target", "level", "effect_sd", "sd_y", "sd_within_strata", + "shift_realised" + )]) + } + if (length(x@notes)) { + cli::cli_h2("Notes") + cli::cli_bullets(stats::setNames(x@notes, rep("!", length(x@notes)))) + } + invisible(x) +}) diff --git a/README.md b/README.md index 04d36e8..badb2d9 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,25 @@ real data in scientific inference. --- +### Known-truth controls + +A pipeline developed on the clone can be checked against datasets in which +the answer is known. A control spec, declared once, names the column whose +effect is removed, the design strata it is permuted within, the outcome an +effect is planted on and the size of that effect. From it, `controls()` +derives a null (every analysed column permuted), negative replicates (one +column permuted within strata) and positive replicates (the negative plus +the planted shift), by permutation alone. The same spec runs on the clone +for the analyst and, with a private seed, on the original for the owner, +who returns a verdict about the pipeline rather than a dataset. + +```r +spec <- control_spec(on = "gen", target = "yield", effect = list(sd = 0.5), + strata = c("rep", "block"), n = 19L, seed = 2026L) +ctl <- controls(synthetic(m), roles(m), spec) +control_table(ctl, "positive", 1L) +``` + ## Threat model `masque` is **not** a privacy-preserving or differential-privacy tool. It is a diff --git a/_pkgdown.yml b/_pkgdown.yml index bf60104..17fa193 100644 --- a/_pkgdown.yml +++ b/_pkgdown.yml @@ -55,6 +55,13 @@ reference: contents: - audit_mask +- title: Known-truth controls + desc: Null, negative and positive control datasets derived from a table by a declared spec. + contents: + - control_spec + - controls + - control_table + - title: Geospatial post-processing desc: Coarsen or re-anchor synthetic coordinates while keeping them on land and plausible. contents: @@ -73,6 +80,7 @@ articles: - recipe_anatomy - confidentiality - design_preservation + - controls navbar: structure: diff --git a/inst/WORDLIST b/inst/WORDLIST index 5ffa407..a23a21d 100644 --- a/inst/WORDLIST +++ b/inst/WORDLIST @@ -1,31 +1,6 @@ -Allshouse -CRD -Doherty -Donut -Gesink -IBD -Leone -Lifecycle -NOTEs -NSW -ORCID -PII -POSIXct -POSIXlt -Pre -RCBD -README -S3 -S4 -S7 -SHA -Serre -TSV -X11 -XQuartz -Zandbergen agroclimatic al +Allshouse anonymisation anonymise anonymised @@ -37,33 +12,57 @@ categoricals centroid's changelog cli +CMD +Codecov +comparator's conductoR +covr +CRD cryptographic dataentry dev difftime +Doherty donut +Donut errored et +genotype's geocoded geolocations geomask geomasking geospatial +Gesink ggplot2 handoff +IBD interpretability jitter's joinable knitr +Leone +Lifecycle +lintr maintainer's masqued matchable +microdata multiset +NOTEs +NSW oldrel orchestrator +ORCID passthrough +PII +pkgdown +POSIXct +POSIXlt pre +Pre +RCBD +README regexes reproducibly retargeting @@ -72,13 +71,21 @@ roadmap roled roxygen runnable +S3 +S4 +S7 scriptable +sdcMicro +Serre +SHA shareability subprocess +synthpop tibble tokenises toolchains trippable +TSV unaliased unclassed underreport @@ -89,11 +96,6 @@ v0 v1 validator walkthrough -CMD -Codecov -covr -lintr -microdata -pkgdown -sdcMicro -synthpop +X11 +XQuartz +Zandbergen diff --git a/man/control_spec.Rd b/man/control_spec.Rd new file mode 100644 index 0000000..f9c4618 --- /dev/null +++ b/man/control_spec.Rd @@ -0,0 +1,83 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/control_spec.R +\name{control_spec} +\alias{control_spec} +\title{Declare the controls to derive from a table} +\usage{ +control_spec( + on = NULL, + target = NULL, + effect = NULL, + strata = NULL, + unit = NULL, + grain = NULL, + n = 1L, + seed = NULL +) +} +\arguments{ +\item{on}{The name of the column whose association with the rest of the +table the negative control removes, by permuting it within \code{strata} at +its \code{unit}. Must be a treatment or covariate column, categorical or +whole-valued. \code{NULL} (the default) means the single treatment column of +the table; a table with several or no treatment columns needs \code{on} +named.} + +\item{target}{The name of one or more numeric outcome columns the +positive control plants the effect on, or \code{NULL} for no positive +control. A logical, factor, date, event or two-valued column is +refused when the spec meets the table.} + +\item{effect}{The planted effect per level of \code{on}, in units of the +target's standard deviation in the table at hand: a named numeric +vector (\code{c(G01 = 0.5, G02 = -0.5)}; a level not named shifts by zero), +or \code{list(sd = s)} to draw one shift per level from a normal +distribution with standard deviation \code{s}, under the spec's seed, when +the spec first meets a table. Required with \code{target}.} + +\item{strata}{Design columns within which \code{on} is permuted, or \code{NULL} +for no stratification. Every unit of \code{on} must lie within one stratum.} + +\item{unit}{Design columns within which \code{on} is constant, so that it is +permuted across those units rather than across rows; \code{NULL} (the +default) permutes rows.} + +\item{grain}{A named list giving, for other analysed columns constant +within a design unit, the unit columns: \code{list(rain = "trial")}. Such a +column is permuted across its units in the null control, so it stays +constant within each unit.} + +\item{n}{The number of negative (and positive) replicates.} + +\item{seed}{The seed of the control draws. Required, so that a control is +reproducible from its declaration.} +} +\value{ +A \code{masque_control_spec} object, with a print method. +} +\description{ +A control spec says what \code{\link[=controls]{controls()}} removes and plants: the column +whose effect the negative control removes (\code{on}), the outcome the +positive control plants an effect on (\code{target}), the planted effect per +level of \code{on} in outcome-SD units (\code{effect}), the design strata within +which \code{on} is permuted and the unit it is permuted at, the other +unit-grained columns, the number of replicates and the seed. The spec is +declared once, before any control table exists, and is the record of +what was planted; print it to read the declaration. +} +\details{ +The null control needs nothing beyond the seed. The negative control +needs \code{on}, which defaults to the single treatment column of the table. +The positive control needs \code{target} and \code{effect} together. +} +\examples{ +spec <- control_spec( + on = "gen", target = "yield", effect = c(G01 = 0.5, G02 = -0.5), + strata = c("rep", "block"), n = 19L, seed = 2026L +) +spec + +} +\seealso{ +\code{\link[=controls]{controls()}}, \code{\link[=control_table]{control_table()}}. +} diff --git a/man/control_table.Rd b/man/control_table.Rd new file mode 100644 index 0000000..3796a6b --- /dev/null +++ b/man/control_table.Rd @@ -0,0 +1,43 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/controls.R +\name{control_table} +\alias{control_table} +\title{Produce one control dataset} +\usage{ +control_table(ctl, kind = c("null", "negative", "positive"), i = 1L) +} +\arguments{ +\item{ctl}{A \code{masque_controls} object from \code{\link[=controls]{controls()}}.} + +\item{kind}{One of \code{"null"}, \code{"negative"}, \code{"positive"}.} + +\item{i}{The replicate, from 1 to \code{n} of the spec; ignored for the null.} +} +\value{ +A data.table with the columns, classes, NA mask and row order of +the input table. +} +\description{ +Regenerates a control dataset of a \code{\link[=controls]{controls()}} object from its seed: the +null, or negative or positive replicate \code{i}. Positive replicate \code{i} is +negative replicate \code{i} plus the planted shift. The result carries a +\code{masque_control} attribute naming its kind and replicate; its values +alone do not tell it from the table it came from, so keep the manifest +with it. +} +\examples{ +trial <- read.csv(system.file("extdata", "john_alpha.csv", package = "masque")) +r <- propose_roles(trial, detect = FALSE) +r <- set_role(r, c("rep", "block"), role = "design", action = "keep") +r <- set_role(r, "gen", role = "treatment", action = "keep") +r <- set_role(r, "yield", role = "outcome", action = "scramble") +r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") +spec <- control_spec(on = "gen", strata = c("rep", "block"), n = 3L, seed = 1L) +ctl <- controls(trial, r, spec) +neg <- control_table(ctl, "negative", 2L) +table(neg$block, neg$gen)[1:2, 1:6] + +} +\seealso{ +\code{\link[=controls]{controls()}}, \code{\link[=control_spec]{control_spec()}}. +} diff --git a/man/controls.Rd b/man/controls.Rd new file mode 100644 index 0000000..b7c8a07 --- /dev/null +++ b/man/controls.Rd @@ -0,0 +1,106 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/controls.R +\name{controls} +\alias{controls} +\title{Derive null, negative and positive controls from a table} +\usage{ +controls(df, roles, spec, events = NULL, seed = NULL) +} +\arguments{ +\item{df}{The table to derive controls from: a data frame or data.table.} + +\item{roles}{Its role table, as returned by \code{\link[=roles]{roles()}} for a clone or by +\code{\link[=propose_roles]{propose_roles()}} and \code{\link[=set_role]{set_role()}} for any table.} + +\item{spec}{A \code{\link[=control_spec]{control_spec()}}.} + +\item{events}{Linked time/status pairs among the outcome columns, in the +form \code{\link[=mask]{mask()}} takes, so that the null permutes each pair as one block; +\code{NULL} when there are none.} + +\item{seed}{A private seed for the permutations that overrides the +spec's seed, for the owner's side; \code{NULL} uses the spec's seed.} +} +\value{ +A \code{masque_controls} object: the resolved spec (\verb{@spec}), the +manifest (\verb{@manifest}, one row per dataset), the truth table +(\verb{@truth}, one row per target and level of \code{on}) and the notes +(\verb{@notes}). Datasets are produced by \code{\link[=control_table]{control_table()}}. +} +\description{ +Builds the control datasets a \code{\link[=control_spec]{control_spec()}} declares, from any table +with a role table: a synthetic clone, a comparator's clone, or, on the +owner's side, the original. Nothing is re-simulated. Three kinds of +control exist, each with a known truth: +} +\details{ +\itemize{ +\item \code{null}: every analysed column (outcome, covariate and date columns +with action \code{scramble} or \code{alias}) is permuted independently over its +non-missing rows; a column declared unit-grained is permuted across its +units and stays constant within each. Design, treatment and kept +columns, the NA mask and every column's multiset of values are kept. +No association among analysed columns, or with the treatment, remains. +Use it as a smoke test and to expose leakage through analysed columns. +\item \code{negative}: the declared column \code{on} alone is permuted within the +declared design strata, at its unit, so its effect on everything is +exactly absent while the design structure and all other associations +are kept. With \code{n} replicates it gives a permutation null for a whole +pipeline: a false-positive rate, a permutation p-value. +\item \code{positive}: the same permutation as the negative replicate, plus the +declared shift per level of \code{on} on each target, in units of the +target's standard deviation in the table at hand. Positive minus +negative is the planted truth, exactly. +} +} +\section{Declaration before data}{ + +The spec is resolved against the table here: \code{on} defaults to the single +treatment column; strata and unit are checked for nesting; an \code{effect} +given as \code{list(sd = )} is drawn under the spec's seed, once, so the same +spec plants the same vector on every table it meets. The resolved spec +and the truth table are on the returned object and in its print. +} + +\section{Two sides, two seeds}{ + +An analyst who holds only the clone runs this on the clone. The owner +runs it on the original, offline, with a private \code{seed} that overrides +the spec's seed for the permutations (the planted vector still comes +from the spec's seed). A public seed would let anyone who knows the +design, which the clone shares, rebuild every permutation and read the +original outcome out of per-replicate statistics. Control tables built +from an original never leave the owner; what leaves is an aggregate +verdict about the pipeline, through the owner's usual output checking. +} + +\section{Limits}{ + +A kept column is structure and is never permuted, so leakage through +kept columns, ids or row order passes the null. A planted effect is +clean: uniform within a level, with no confounding and no interaction. +Nothing is clipped, so a shift can take a non-negative outcome below +zero; \code{\link[=control_table]{control_table()}} warns when it does. The null collapses the +design means of row-grained outcomes, so false-positive calibration +comes from the negative control, not the null. +} + +\examples{ +trial <- read.csv(system.file("extdata", "john_alpha.csv", package = "masque")) +r <- propose_roles(trial, detect = FALSE) +r <- set_role(r, c("rep", "block"), role = "design", action = "keep") +r <- set_role(r, "gen", role = "treatment", action = "keep") +r <- set_role(r, "yield", role = "outcome", action = "scramble") +r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") +spec <- control_spec( + on = "gen", target = "yield", effect = list(sd = 0.5), + strata = c("rep", "block"), n = 5L, seed = 2026L +) +ctl <- controls(trial, r, spec) +ctl +head(control_table(ctl, "positive", 1L)) + +} +\seealso{ +\code{\link[=control_spec]{control_spec()}}, \code{\link[=control_table]{control_table()}}, \code{\link[=mask]{mask()}}, \code{\link[=roles]{roles()}}. +} diff --git a/tests/testthat/helper-controls.R b/tests/testthat/helper-controls.R new file mode 100644 index 0000000..b7ab639 --- /dev/null +++ b/tests/testthat/helper-controls.R @@ -0,0 +1,109 @@ +# Fixtures for the control datasets: a small MET with a trial-level numeric, +# categorical and date covariate, a plot-level covariate correlated with +# yield, a whole-plot factor, a whole-valued outcome, and missing cells +# including one trial with no covariate value at all. +.ctl_met <- function(n_trial = 4L, n_rep = 3L, n_gen = 8L, seed = 11L) { + withr::with_seed(seed, { + trials <- sprintf("T%d", seq_len(n_trial)) + gens <- sprintf("G%02d", seq_len(n_gen)) + g <- expand.grid( + gen = gens, rep = seq_len(n_rep), trial = trials, + stringsAsFactors = FALSE, KEEP.OUT.ATTRS = FALSE + ) + g$block <- sprintf("B%d", (match(g$gen, gens) - 1L) %/% 4L + 1L) + g$irrig <- ifelse(g$rep %% 2L == 1L, "dry", "wet") + trial_eff <- stats::rnorm(n_trial, 0, 0.8) + gen_eff <- stats::rnorm(n_gen, 0, 0.4) + g$yield <- round( + 3 + trial_eff[match(g$trial, trials)] + gen_eff[match(g$gen, gens)] + + ifelse(g$irrig == "wet", 0.5, 0) + stats::rnorm(nrow(g), 0, 0.3), + 2 + ) + g$count <- as.integer(round(g$yield * 3)) + g$aet <- round(300 + 40 * trial_eff, 1)[match(g$trial, trials)] + g$plot_cov <- round(g$yield * 10 + stats::rnorm(nrow(g), 0, 1), 2) + g$soil <- rep_len(c("clay", "loam", "sand"), n_trial)[ + match(g$trial, trials) + ] + g$sow <- as.Date("2024-05-01") + 7L * match(g$trial, trials) + g$yield[c(3L, 17L, 40L)] <- NA + g$plot_cov[c(5L, 6L)] <- NA + g$aet[g$trial == trials[2L]] <- NA + g$rep <- sprintf("R%d", g$rep) + g[, c( + "trial", "rep", "block", "irrig", "gen", "yield", "count", "aet", + "plot_cov", "soil", "sow" + )] + }) +} + +.ctl_roles <- function(df) { + r <- suppressWarnings(propose_roles(df, detect = FALSE)) + r <- set_role(r, c("trial", "rep", "block"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "keep") + r <- set_role(r, "irrig", role = "covariate", action = "scramble") + r <- set_role(r, c("yield", "count"), role = "outcome", action = "scramble") + r <- set_role( + r, c("aet", "plot_cov", "soil"), role = "covariate", action = "scramble" + ) + set_role(r, "sow", role = "date", action = "scramble") +} + +# The standard spec on the MET: genotype permuted within blocks, the +# trial-level covariates declared unit-grained. +.ctl_spec <- function(..., n = 2L, seed = 3L) { + control_spec( + on = "gen", + strata = c("trial", "rep", "block"), + grain = list(aet = "trial", soil = "trial", sow = "trial"), + n = n, + seed = seed, + ... + ) +} + +# Two arms, no design: for effect recovery. +.ctl_two_arm <- function(n = 1000L, seed = 5L) { + withr::with_seed(seed, data.frame( + trt = rep(c("A", "B"), each = n / 2L), + y = round(stats::rnorm(n, 10, 2), 3), + stringsAsFactors = FALSE + )) +} + +.ctl_two_arm_roles <- function(df) { + r <- suppressWarnings(propose_roles(df, detect = FALSE)) + r <- set_role(r, "trt", role = "treatment", action = "keep") + set_role(r, "y", role = "outcome", action = "scramble") +} + +# Many levels, no design: for rank recovery. +.ctl_many_levels <- function(k = 20L, m = 30L, seed = 7L) { + withr::with_seed(seed, data.frame( + lev = rep(sprintf("L%02d", seq_len(k)), each = m), + y = round(stats::rnorm(k * m, 10, 2), 3), + stringsAsFactors = FALSE + )) +} + +.ctl_many_roles <- function(df) { + r <- suppressWarnings(propose_roles(df, detect = FALSE)) + r <- set_role(r, "lev", role = "treatment", action = "keep") + set_role(r, "y", role = "outcome", action = "scramble") +} + +# The non-missing values of two vectors are the same multiset. +.ctl_same_multiset <- function(a, b) { + identical( + sort(as.character(a[!is.na(a)])), + sort(as.character(b[!is.na(b)])) + ) +} + +# A refusal: an error of the given class under the shared marker, no warning. +.ctl_expect_refusal <- function(expr, class) { + err <- testthat::expect_no_warning(tryCatch(expr, error = function(e) e)) + testthat::expect_s3_class(err, "error") + testthat::expect_true(inherits(err, class)) + testthat::expect_true(inherits(err, "orchestra_refusal")) +} diff --git a/tests/testthat/test-controls-events.R b/tests/testthat/test-controls-events.R new file mode 100644 index 0000000..0838dbc --- /dev/null +++ b/tests/testthat/test-controls-events.R @@ -0,0 +1,33 @@ +# A declared linked-event pair moves as one row block in the null, so the +# time order within a row and the status patterns survive. + +test_that("a declared event pair moves as one block in the null", { + df <- .events_fixture(n = 300L) + r <- .events_roles(df) + spec <- control_spec(on = "arm", seed = 1L) + ctl <- controls(df, r, spec, events = .events_spec) + nul <- control_table(ctl, "null") + key_in <- paste(df$time_rec, df$status_rec, df$time_death, df$status_death) + key_out <- paste(nul$time_rec, nul$status_rec, nul$time_death, nul$status_death) + expect_identical(sort(key_in), sort(key_out)) + expect_false(identical(key_in, key_out)) + expect_true(all(nul$time_rec <= nul$time_death)) + expect_identical(nul$id, df$id) + expect_identical(nul$arm, df$arm) +}) + +test_that("without the declaration the pair breaks, and an event column is no target", { + df <- .events_fixture(n = 300L) + r <- .events_roles(df) + ctl <- controls(df, r, control_spec(on = "arm", seed = 1L)) + nul <- control_table(ctl, "null") + key_in <- paste(df$time_rec, df$status_rec, df$time_death, df$status_death) + key_out <- paste(nul$time_rec, nul$status_rec, nul$time_death, nul$status_death) + expect_false(identical(sort(key_in), sort(key_out))) + .ctl_expect_refusal( + controls(df, r, control_spec( + on = "arm", target = "time_rec", effect = c(B = 1), seed = 1L + ), events = .events_spec), + "masque_control_target_kind_refusal" + ) +}) diff --git a/tests/testthat/test-controls-negative.R b/tests/testthat/test-controls-negative.R new file mode 100644 index 0000000..5fdf5a6 --- /dev/null +++ b/tests/testthat/test-controls-negative.R @@ -0,0 +1,141 @@ +# The negative control: `on` permuted within strata at its unit, nothing +# else touched. + +test_that("the negative permutes on within strata and keeps everything else", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec()) + neg <- control_table(ctl, "negative", 1L) + for (cl in setdiff(names(df), "gen")) { + expect_identical(neg[[cl]], df[[cl]], info = cl) + } + stratum <- paste(df$trial, df$rep, df$block) + for (s in unique(stratum)) { + expect_identical(sort(neg$gen[stratum == s]), sort(df$gen[stratum == s])) + } + expect_false(identical(neg$gen, df$gen)) + expect_identical(ctl@manifest$n_strata[1L], 24L) + expect_identical(ctl@manifest$strata_constant_share[1L], 0) + expect_identical(attr(neg, "masque_control")$replicate, 1L) +}) + +test_that("replicates differ and the treatment-by-stratum incidence is kept", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec(n = 3L)) + n1 <- control_table(ctl, "negative", 1L) + n2 <- control_table(ctl, "negative", 2L) + expect_false(identical(n1$gen, n2$gen)) + expect_identical(table(n1$trial, n1$gen), table(df$trial, df$gen)) +}) + +test_that("a whole-plot on is permuted at its unit within coarser strata", { + df <- .ctl_met() + r <- .ctl_roles(df) + spec <- control_spec( + on = "irrig", unit = c("trial", "rep"), strata = "trial", n = 1L, seed = 9L + ) + ctl <- controls(df, r, spec) + neg <- control_table(ctl, "negative", 1L) + unit <- masque:::.unit_id(df, c("trial", "rep")) + expect_true(masque:::.constant_within(unit, neg$irrig)) + for (t in unique(df$trial)) { + u_in <- unique(df[df$trial == t, c("rep", "irrig")])$irrig + u_out <- unique(as.data.frame(neg)[neg$trial == t, c("rep", "irrig")])$irrig + expect_identical(sort(u_out), sort(u_in)) + } + expect_identical(ctl@manifest$unit[1L], "trial + rep") +}) + +test_that("a degenerate restricted permutation is refused", { + df <- .ctl_met() + r <- .ctl_roles(df) + spec <- control_spec( + on = "irrig", strata = c("trial", "rep", "block"), seed = 1L + ) + .ctl_expect_refusal(controls(df, r, spec), "masque_control_degenerate_refusal") +}) + +test_that("on constant in most strata warns", { + df <- data.frame( + site = rep(c("S1", "S2", "S3", "S4"), each = 4L), + trt = c(rep("a", 4L), rep("b", 4L), rep("a", 4L), c("a", "a", "b", "b")), + y = seq_len(16L) + 0.5, + stringsAsFactors = FALSE + ) + r <- suppressWarnings(propose_roles(df, detect = FALSE)) + r <- set_role(r, "site", role = "design", action = "keep") + r <- set_role(r, "trt", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + spec <- control_spec(on = "trt", strata = "site", seed = 1L) + expect_warning( + ctl <- controls(df, r, spec), class = "masque_control_strata_weak" + ) + expect_identical(ctl@manifest$strata_constant_share[1L], 0.75) +}) + +test_that("unit and strata must nest, be design columns, and hold on constant", { + df <- .ctl_met() + r <- .ctl_roles(df) + .ctl_expect_refusal( + controls(df, r, control_spec( + on = "irrig", unit = c("trial", "rep"), strata = "block", seed = 1L + )), + "masque_control_strata_refusal" + ) + .ctl_expect_refusal( + controls(df, r, control_spec(on = "gen", unit = "block", seed = 1L)), + "masque_control_unit_refusal" + ) + .ctl_expect_refusal( + controls(df, r, control_spec(on = "gen", strata = "gen", seed = 1L)), + "masque_control_strata_refusal" + ) + .ctl_expect_refusal( + controls(df, r, control_spec(on = "gen", unit = "soil", seed = 1L)), + "masque_control_unit_refusal" + ) +}) + +test_that("on defaults to the single treatment and is checked by role and kind", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, control_spec(strata = "trial", seed = 1L)) + expect_identical(ctl@spec@on, "gen") + + r_two <- set_role(r, "irrig", role = "treatment", action = "keep") + .ctl_expect_refusal( + controls(df, r_two, control_spec(strata = "trial", seed = 1L)), + "masque_control_on_refusal" + ) + .ctl_expect_refusal( + controls(df, r, control_spec(on = "trial", seed = 1L)), + "masque_control_on_refusal" + ) + .ctl_expect_refusal( + controls(df, r, control_spec(on = "yield", seed = 1L)), + "masque_control_on_refusal" + ) + .ctl_expect_refusal( + controls(df, r, control_spec(on = "plot_cov", seed = 1L)), + "masque_control_on_refusal" + ) + .ctl_expect_refusal( + controls(df, r, control_spec(on = "nope", seed = 1L)), + "masque_control_on_refusal" + ) +}) + +test_that("a whole-valued numeric treatment can be on", { + df <- .ctl_two_arm(n = 40L) + df$dose <- rep(c(50L, 100L), 20L) + r <- .ctl_two_arm_roles(df) + r <- set_role(r, "trt", role = "covariate", action = "scramble") + r <- set_role(r, "dose", role = "treatment", action = "keep") + ctl <- controls(df, r, control_spec( + target = "y", effect = c(`100` = 0.5), seed = 1L + )) + expect_identical(ctl@spec@on, "dose") + expect_identical(ctl@truth$level, c("50", "100")) + expect_identical(ctl@truth$effect_sd, c(0, 0.5)) +}) diff --git a/tests/testthat/test-controls-null.R b/tests/testthat/test-controls-null.R new file mode 100644 index 0000000..43cbe80 --- /dev/null +++ b/tests/testthat/test-controls-null.R @@ -0,0 +1,111 @@ +# The null control: every analysed column permuted at its grain, nothing else +# touched, every association removed. + +test_that("the null keeps structure, NA mask, classes and multisets", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec()) + nul <- control_table(ctl, "null") + + expect_identical(names(nul), names(df)) + expect_identical(nrow(nul), nrow(df)) + for (cl in c("trial", "rep", "block", "gen")) { + expect_identical(nul[[cl]], df[[cl]]) + } + expect_identical( + unname(is.na(as.data.frame(nul))), unname(is.na(df)) + ) + expect_identical( + vapply(nul, function(x) class(x)[1L], character(1L)), + vapply(df, function(x) class(x)[1L], character(1L)) + ) + for (cl in c("yield", "count", "plot_cov", "irrig", "aet", "soil", "sow")) { + expect_true(.ctl_same_multiset(nul[[cl]], df[[cl]]), info = cl) + } + expect_identical(attr(nul, "masque_control")$kind, "null") + expect_true(is.na(attr(nul, "masque_control")$replicate)) +}) + +test_that("a declared unit-grained column stays constant within its unit", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec()) + nul <- control_table(ctl, "null") + trial_id <- masque:::.unit_id(df, "trial") + for (cl in c("aet", "soil", "sow")) { + expect_true(masque:::.constant_within(trial_id, nul[[cl]]), info = cl) + } + # The all-missing trial stays missing; the unit values are the same set. + expect_true(all(is.na(nul$aet[nul$trial == "T2"]))) + expect_identical( + sort(unique(nul$aet[!is.na(nul$aet)])), + sort(unique(df$aet[!is.na(df$aet)])) + ) + # The unit values moved: at least one trial holds another trial's value. + expect_false(identical(nul$aet, df$aet) && identical(nul$soil, df$soil)) +}) + +test_that("a column not constant within its declared unit is row-permuted with a note", { + df <- .ctl_met() + r <- .ctl_roles(df) + spec <- control_spec( + on = "gen", strata = c("trial", "rep", "block"), + grain = list(plot_cov = "trial"), seed = 3L + ) + ctl <- controls(df, r, spec) + expect_length(ctl@notes, 1L) + expect_match(ctl@notes, "plot_cov is not constant within trial", fixed = TRUE) + nul <- control_table(ctl, "null") + expect_true(.ctl_same_multiset(nul$plot_cov, df$plot_cov)) +}) + +test_that("the null removes the association among analysed columns", { + df <- .ctl_met(n_trial = 5L, n_gen = 20L) + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec()) + nul <- control_table(ctl, "null") + before <- stats::cor(df$yield, df$plot_cov, use = "complete.obs") + after <- stats::cor(nul$yield, nul$plot_cov, use = "complete.obs") + expect_gt(before, 0.95) + expect_lt(abs(after), 0.3) +}) + +test_that("controls() and control_table() leave the caller's RNG untouched", { + df <- .ctl_met() + r <- .ctl_roles(df) + set.seed(123) + before <- .Random.seed + ctl <- controls(df, r, .ctl_spec()) + nul <- control_table(ctl, "null") + neg <- control_table(ctl, "negative", 1L) + expect_identical(.Random.seed, before) +}) + +test_that("the same seed gives the same null on data.frame and data.table input", { + df <- .ctl_met() + r <- .ctl_roles(df) + a <- control_table(controls(df, r, .ctl_spec()), "null") + b <- control_table(controls(data.table::as.data.table(df), r, .ctl_spec()), "null") + expect_identical(a, b) + c3 <- control_table(controls(df, r, .ctl_spec(seed = 4L)), "null") + expect_false(identical(a$yield, c3$yield)) +}) + +test_that("a kept outcome is reported and left alone; no analysed column is refused", { + df <- .ctl_met() + r <- .ctl_roles(df) + r_keep <- set_role(r, "yield", role = "outcome", action = "keep") + expect_warning( + ctl <- controls(df, r_keep, .ctl_spec()), + class = "masque_control_kept_outcome" + ) + expect_identical(control_table(ctl, "null")$yield, df$yield) + + r_all <- r + r_all$action <- "keep" + err <- tryCatch( + suppressWarnings(controls(df, r_all, .ctl_spec())), + error = function(e) e + ) + expect_true(inherits(err, "masque_control_analysed_refusal")) +}) diff --git a/tests/testthat/test-controls-positive.R b/tests/testthat/test-controls-positive.R new file mode 100644 index 0000000..9c8415e --- /dev/null +++ b/tests/testthat/test-controls-positive.R @@ -0,0 +1,138 @@ +# The positive control: the negative plus the declared per-level shift on +# each target. Positive minus negative is the planted truth exactly. + +test_that("positive minus negative equals the realised shift, row by row", { + df <- .ctl_met() + r <- .ctl_roles(df) + spec <- .ctl_spec(target = c("yield", "count"), effect = c(G01 = 0.5, G02 = -0.5)) + ctl <- controls(df, r, spec) + neg <- control_table(ctl, "negative", 2L) + pos <- control_table(ctl, "positive", 2L) + for (cl in setdiff(names(df), c("yield", "count"))) { + expect_identical(pos[[cl]], neg[[cl]], info = cl) + } + truth <- ctl@truth + for (tg in c("yield", "count")) { + shift <- truth$shift_realised[truth$target == tg][ + match(pos$gen, truth$level[truth$target == tg]) + ] + diff <- pos[[tg]] - neg[[tg]] + obs <- !is.na(neg[[tg]]) + expect_identical(is.na(pos[[tg]]), is.na(neg[[tg]])) + expect_equal(diff[obs], shift[obs], tolerance = 1e-12) + } + expect_true(is.integer(pos$count)) + expect_true(all(truth$whole_valued[truth$target == "count"])) + expect_identical(nrow(truth), 16L) + expect_identical( + truth$effect_sd[truth$target == "yield"], + c(0.5, -0.5, 0, 0, 0, 0, 0, 0) + ) + expect_equal( + truth$shift_raw[truth$target == "yield" & truth$level == "G01"], + 0.5 * stats::sd(df$yield, na.rm = TRUE) + ) +}) + +test_that("a shift that rounds to zero on a whole-valued target is refused", { + df <- .ctl_met() + r <- .ctl_roles(df) + spec <- .ctl_spec(target = "count", effect = c(G01 = 0.05)) + .ctl_expect_refusal(controls(df, r, spec), "masque_control_shift_vanishes_refusal") +}) + +test_that("a shift below zero on a non-negative target is reported", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec(target = "yield", effect = c(G01 = -30))) + expect_warning( + pos <- control_table(ctl, "positive", 1L), + class = "masque_control_sign_crossed" + ) + expect_true(any(pos$yield < 0, na.rm = TRUE)) +}) + +test_that("list(sd = ) draws the same vector whatever the run seed", { + df <- .ctl_met() + r <- .ctl_roles(df) + spec <- .ctl_spec(target = "yield", effect = list(sd = 0.5)) + a <- controls(df, r, spec) + b <- controls(df, r, spec, seed = 99L) + expect_identical(a@spec@effect, b@spec@effect) + expect_identical(names(a@spec@effect), sprintf("G%02d", 1:8)) + expect_false(identical( + control_table(a, "negative", 1L)$gen, control_table(b, "negative", 1L)$gen + )) + expect_identical(control_table(a, "null"), control_table(a, "null")) +}) + +test_that("a two-arm planted effect is recovered without bias", { + df <- .ctl_two_arm() + r <- .ctl_two_arm_roles(df) + n_rep <- 50L + ctl <- controls(df, r, control_spec( + on = "trt", target = "y", effect = c(B = 1), n = n_rep, seed = 21L + )) + shift <- ctl@truth$shift_realised[ctl@truth$level == "B"] + est <- vapply(seq_len(n_rep), function(i) { + d <- control_table(ctl, "positive", i) + unname(stats::coef(stats::lm(y ~ trt, data = d))["trtB"]) + }, numeric(1L)) + est_null <- vapply(seq_len(n_rep), function(i) { + d <- control_table(ctl, "negative", i) + unname(stats::coef(stats::lm(y ~ trt, data = d))["trtB"]) + }, numeric(1L)) + se <- stats::sd(est) / sqrt(n_rep) + expect_lt(abs(mean(est) - shift), 3 * se) + expect_lt(abs(mean(est_null)), 3 * stats::sd(est_null) / sqrt(n_rep)) +}) + +test_that("a many-level planted vector is recovered in rank", { + df <- .ctl_many_levels() + r <- .ctl_many_roles(df) + n_rep <- 50L + ctl <- controls(df, r, control_spec( + on = "lev", target = "y", effect = list(sd = 1), n = n_rep, seed = 22L + )) + truth <- ctl@truth$effect_sd + rho <- vapply(seq_len(n_rep), function(i) { + d <- control_table(ctl, "positive", i) + means <- tapply(d$y, d$lev, mean) + stats::cor(truth, means[ctl@truth$level], method = "spearman") + }, numeric(1L)) + expect_gte(sum(rho >= 0.9), 45L) +}) + +test_that("targets are checked by kind, grain, events and against on", { + df <- .ctl_met() + r <- .ctl_roles(df) + .ctl_expect_refusal( + controls(df, r, .ctl_spec(target = "soil", effect = c(G01 = 1))), + "masque_control_target_kind_refusal" + ) + .ctl_expect_refusal( + controls(df, r, .ctl_spec(target = "irrig", effect = c(G01 = 1))), + "masque_control_target_kind_refusal" + ) + .ctl_expect_refusal( + controls(df, r, control_spec( + on = "gen", target = "aet", effect = c(G01 = 1), grain = list(aet = "trial"), + seed = 1L + )), + "masque_control_target_grain_refusal" + ) + .ctl_expect_refusal( + controls(df, r, control_spec( + on = "irrig", target = "irrig", effect = c(wet = 1), seed = 1L + )), + "masque_control_on_refusal" + ) + .ctl_expect_refusal( + controls(df, r, .ctl_spec(target = "nope", effect = c(G01 = 1))), + "masque_control_target_refusal" + ) + .ctl_expect_refusal( + controls(df, r, .ctl_spec(target = "yield", effect = c(G99 = 1))), + "masque_control_effect_refusal" + ) +}) diff --git a/tests/testthat/test-controls-spec.R b/tests/testthat/test-controls-spec.R new file mode 100644 index 0000000..360b6c5 --- /dev/null +++ b/tests/testthat/test-controls-spec.R @@ -0,0 +1,83 @@ +# control_spec(): the declaration is validated before it meets a table. + +test_that("the seed is required", { + .ctl_expect_refusal(control_spec(on = "gen"), "masque_control_seed_refusal") + .ctl_expect_refusal( + control_spec(on = "gen", seed = NULL), "masque_control_seed_refusal" + ) + .ctl_expect_refusal( + control_spec(on = "gen", seed = 1.5), "masque_bad_seed_refusal" + ) +}) + +test_that("target and effect come together", { + .ctl_expect_refusal( + control_spec(target = "y", seed = 1L), "masque_control_effect_refusal" + ) + .ctl_expect_refusal( + control_spec(effect = c(A = 1), seed = 1L), "masque_control_effect_refusal" + ) +}) + +test_that("effect forms are a named vector or list(sd = )", { + .ctl_expect_refusal( + control_spec(target = "y", effect = 0.5, seed = 1L), + "masque_control_effect_refusal" + ) + .ctl_expect_refusal( + control_spec(target = "y", effect = c(A = 1, A = 2), seed = 1L), + "masque_control_effect_refusal" + ) + .ctl_expect_refusal( + control_spec(target = "y", effect = c(A = Inf), seed = 1L), + "masque_control_effect_refusal" + ) + .ctl_expect_refusal( + control_spec(target = "y", effect = list(sd = -1), seed = 1L), + "masque_control_effect_refusal" + ) + .ctl_expect_refusal( + control_spec(target = "y", effect = list(scale = 1), seed = 1L), + "masque_control_effect_refusal" + ) + s1 <- control_spec(target = "y", effect = c(A = 1L), seed = 1L) + expect_identical(s1@effect, c(A = 1)) + s2 <- control_spec(target = "y", effect = list(sd = 2L), seed = 1L) + expect_identical(s2@effect, list(sd = 2)) +}) + +test_that("n, on, strata and grain are checked", { + .ctl_expect_refusal(control_spec(n = 0L, seed = 1L), "masque_control_n_refusal") + .ctl_expect_refusal(control_spec(n = 1.5, seed = 1L), "masque_control_n_refusal") + .ctl_expect_refusal( + control_spec(on = c("a", "b"), seed = 1L), "masque_control_on_refusal" + ) + .ctl_expect_refusal( + control_spec(strata = c("a", "a"), seed = 1L), "masque_control_strata_refusal" + ) + .ctl_expect_refusal( + control_spec(grain = list("trial"), seed = 1L), "masque_control_grain_refusal" + ) + .ctl_expect_refusal( + control_spec(grain = list(aet = 1L), seed = 1L), "masque_control_grain_refusal" + ) + s <- control_spec(n = 19, seed = 2026, unit = "trial") + expect_identical(s@n, 19L) + expect_identical(s@seed, 2026L) + expect_identical(s@unit, "trial") + expect_identical(s@strata, character()) +}) + +test_that("the spec prints its declaration", { + s <- control_spec( + on = "gen", target = "yield", effect = c(G01 = 0.5), + strata = c("rep", "block"), grain = list(aet = "trial"), n = 19L, + seed = 1L + ) + out <- capture_full(print(s)) + expect_match(out, "Replicates: 19", fixed = TRUE) + expect_match(out, "G01 +0.5", fixed = TRUE) + expect_match(out, "aet within trial", fixed = TRUE) + s2 <- control_spec(target = "y", effect = list(sd = 0.3), seed = 1L) + expect_match(capture_full(print(s2)), "drawn from N(0, 0.3^2)", fixed = TRUE) +}) diff --git a/tests/testthat/test-controls-table.R b/tests/testthat/test-controls-table.R new file mode 100644 index 0000000..006901b --- /dev/null +++ b/tests/testthat/test-controls-table.R @@ -0,0 +1,73 @@ +# control_table(): regeneration from seed, argument checks, the manifest and +# the print. + +test_that("a control table is regenerated identically from its seed", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec(target = "yield", effect = c(G01 = 1))) + expect_identical(control_table(ctl, "null"), control_table(ctl, "null")) + expect_identical( + control_table(ctl, "negative", 2L), control_table(ctl, "negative", 2L) + ) + expect_identical( + control_table(ctl, "positive", 2L), control_table(ctl, "positive", 2L) + ) +}) + +test_that("kind and replicate are checked", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec()) + .ctl_expect_refusal(control_table(ctl, "negative", 3L), "masque_control_kind_refusal") + .ctl_expect_refusal(control_table(ctl, "negative", 0L), "masque_control_kind_refusal") + .ctl_expect_refusal(control_table(ctl, "positive", 1L), "masque_control_kind_refusal") + .ctl_expect_refusal(control_table(df, "null"), "masque_control_kind_refusal") + expect_error(control_table(ctl, "blank")) + .ctl_expect_refusal(controls(df, r, list(on = "gen")), "masque_control_spec_refusal") + .ctl_expect_refusal(controls(df[1L, ], r, .ctl_spec()), "masque_control_table_refusal") + .ctl_expect_refusal(controls(df, r, .ctl_spec(), seed = 1.5), "masque_bad_seed_refusal") +}) + +test_that("the manifest lists every dataset with its own seed", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec(n = 3L, target = "yield", effect = c(G01 = 1))) + m <- ctl@manifest + expect_identical(nrow(m), 7L) + expect_identical(m$kind, c("null", rep("negative", 3L), rep("positive", 3L))) + expect_identical(m$replicate, c(NA, 1L, 2L, 3L, 1L, 2L, 3L)) + expect_identical(m$seed[2:4], m$seed[5:7]) + expect_identical(anyDuplicated(m$seed[1:4]), 0L) + expect_identical(m$permuted[2L], "gen") + expect_match(m$permuted[1L], "yield", fixed = TRUE) + expect_identical(m$strata[1L], "trial + rep + block") + expect_identical(m$target[1L], "yield") + ctl0 <- controls(df, r, .ctl_spec()) + expect_identical(nrow(ctl0@manifest), 3L) + expect_true(is.na(ctl0@manifest$target[1L])) +}) + +test_that("the controls object prints its declaration and truth", { + df <- .ctl_met() + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec(target = "yield", effect = c(G01 = 1))) + out <- capture_full(print(ctl)) + expect_match(out, "Datasets: 1 null, 2 negative, 2 positive", fixed = TRUE) + expect_match(out, "Planted truth", fixed = TRUE) + expect_match(out, "24 strata", fixed = TRUE) + ctl2 <- controls(df, r, control_spec( + on = "gen", strata = "trial", grain = list(plot_cov = "trial"), seed = 1L + )) + expect_match(capture_full(print(ctl2)), "Notes", fixed = TRUE) +}) + +test_that("the input table is not modified", { + df <- data.table::as.data.table(.ctl_met()) + before <- data.table::copy(df) + r <- .ctl_roles(df) + ctl <- controls(df, r, .ctl_spec(target = "yield", effect = c(G01 = 1))) + invisible(control_table(ctl, "null")) + invisible(control_table(ctl, "positive", 1L)) + expect_identical(df, before) + expect_identical(ctl@table, before) +}) diff --git a/vignettes/controls.Rmd b/vignettes/controls.Rmd new file mode 100644 index 0000000..85a645f --- /dev/null +++ b/vignettes/controls.Rmd @@ -0,0 +1,232 @@ +--- +title: "Known-truth controls" +output: rmarkdown::html_vignette +vignette: > + %\VignetteIndexEntry{Known-truth controls} + %\VignetteEngine{knitr::rmarkdown} + %\VignetteEncoding{UTF-8} +--- + + + +```{r setup, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>" +) +old_options <- options(width = 100) +set.seed(1) +``` + +```{r library} +library(masque) +``` + +## Why + +A pipeline is an assay. Before an assay is trusted on a sample, it is run on +a blank, on a negative control and on a positive control, and its readings +on those three are what make the reading on the sample interpretable. +`masque` gives an analyst the structure of a confidential table without its +values. Controls add the other three: datasets derived from a table in +which the answer is known in advance. + +* The **null** control permutes every analysed column, so nothing is + associated with anything. A pipeline that still finds signal on it has a + leak or a bug. +* The **negative** control permutes one declared column, by default the + treatment, within the design strata. Its effect is exactly absent and + everything else is as it was. A pipeline that finds it has a false + positive, and over many replicates the rate of those is the pipeline's + false-positive rate on this very structure. +* The **positive** control adds a declared effect to the negative. The + pipeline should find it, and the size it reports can be compared with the + size that was planted. + +Nothing is re-simulated. Each control is a permutation of the table it was +built from, with a shift added for the positive, so the design columns, the +treatment allocation, the missing cells and every column's set of values are +exactly those of the input. + +## A clone and a declaration + +The example is the alpha-design trial shipped with the package: 24 +genotypes in 72 plots, three replicates, six blocks of four plots within +each replicate. + +```{r trial} +trial <- read.csv(system.file("extdata", "john_alpha.csv", package = "masque")) +r <- propose_roles(trial, detect = FALSE) +r <- set_role(r, c("rep", "block"), role = "design", action = "keep") +r <- set_role(r, "gen", role = "treatment", action = "keep") +r <- set_role(r, "yield", role = "outcome", action = "scramble") +r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") +m <- mask(trial, r, seed = 1L) +clone <- synthetic(m) +``` + +The control spec is declared once, before any control exists. It names the +column to permute, the strata to permute it within, the outcome to plant an +effect on, the effect per genotype in units of the outcome's standard +deviation, the number of replicates and the seed. Here the planted +genotype effects are drawn from a normal distribution with standard +deviation 0.5, under the spec's seed, so the same vector is planted on +every table the spec meets. + +```{r spec} +spec <- control_spec( + on = "gen", + target = "yield", + effect = list(sd = 0.5), + strata = c("rep", "block"), + n = 19L, + seed = 2026L +) +spec +``` + +## The analyst's side: controls from the clone + +The analyst holds the clone and its role table, and nothing else. The +controls are derived from the clone, so they disclose nothing the clone +does not. + +```{r controls-clone} +ctl <- controls(clone, roles(m), spec) +ctl +``` + +The print shows the resolved declaration and the planted truth: for each +genotype, the shift in standard-deviation units and the shift in yield +units that the positive control adds to that genotype's plots. The `@truth` +table holds the same, and `@manifest` lists every dataset with its seed. + +A small pipeline stands in for the analyst's: an analysis of variance of +yield on replicate, block within replicate and genotype, returning the +p-value of the genotype term. + +```{r pipeline} +genotype_p <- function(d) { + fit <- stats::aov(yield ~ rep + rep:block + gen, data = d) + tab <- summary(fit)[[1]] + tab[trimws(rownames(tab)) == "gen", "Pr(>F)"] +} +``` + +On the clone itself the genotype term is not expected to show, because the +default clone re-simulates yield without a genotype effect. On the null it +must not show. On the negatives it shows at the pipeline's false-positive +rate. On the positives it shows at the pipeline's power against the planted +size. + +```{r run} +p_clone <- genotype_p(clone) +p_null <- genotype_p(control_table(ctl, "null")) +p_neg <- vapply(seq_len(spec@n), function(i) { + genotype_p(control_table(ctl, "negative", i)) +}, numeric(1)) +p_pos <- vapply(seq_len(spec@n), function(i) { + genotype_p(control_table(ctl, "positive", i)) +}, numeric(1)) +data.frame( + dataset = c("clone", "null", "negative (19)", "positive (19)"), + p_or_share_below_0.05 = c( + round(p_clone, 3), round(p_null, 3), + mean(p_neg < 0.05), mean(p_pos < 0.05) + ) +) +``` + +The negatives give the pipeline's false-positive rate on this design, and +the positives its power against the planted effect. Both are properties of +the pipeline on this structure, learned without a single real yield. + +The planted size can be checked as well. The genotype means on a positive +replicate, less those on its negative replicate, are the planted shifts +exactly, because the two share one permutation. + +```{r recovery} +neg <- control_table(ctl, "negative", 1L) +pos <- control_table(ctl, "positive", 1L) +recovered <- tapply(pos$yield, pos$gen, mean) - tapply(neg$yield, neg$gen, mean) +planted <- ctl@truth$shift_realised[match(names(recovered), ctl@truth$level)] +all.equal(as.numeric(recovered), planted) +``` + +## The owner's side: the same spec on the original + +The owner runs the pipeline the analyst handed back on the original table, +and on controls derived from the original with the same spec. One thing +changes: the permutations are drawn from a private seed, given as `seed`, +which overrides the spec's seed. The planted vector still comes from the +spec's seed, so the truth is the same on both sides. A public seed would +let anyone who knows the design, which the clone shares, rebuild every +permutation and read the original outcome out of per-replicate results. + +```{r controls-original} +ctl_owner <- controls(trial, r, spec, seed = 40371L) +p_real <- genotype_p(trial) +p_neg_owner <- vapply(seq_len(spec@n), function(i) { + genotype_p(control_table(ctl_owner, "negative", i)) +}, numeric(1)) +p_pos_owner <- vapply(seq_len(spec@n), function(i) { + genotype_p(control_table(ctl_owner, "positive", i)) +}, numeric(1)) +``` + +What leaves the owner's side is a verdict about the pipeline, through the +owner's usual output checking: the false-positive rate over the negatives, +the power over the positives and, for the real result, a permutation +p-value, the share of negatives whose p-value is at or below the real one. + +```{r verdict} +data.frame( + verdict = c( + "false-positive rate (negatives)", "power at the planted effect (positives)", + "permutation p-value of the real result" + ), + value = c( + mean(p_neg_owner < 0.05), mean(p_pos_owner < 0.05), + (sum(p_neg_owner <= p_real) + 1) / (spec@n + 1) + ) +) +``` + +The power differs between the two sides. The default clone re-simulates +yield without the block structure, so the pipeline's residual variance is +larger there and the same planted size is harder to see. A clone conditioned +on the design would bring the two readings closer, and the gap between them +says what fidelity this pipeline needs from a clone. + +The control tables built from the original hold the real values and never +leave. Per-replicate results do not leave either: each replicate is one +more query against the original, and with the permutations known they are +linear equations in the real outcome. The replicate count in the spec is +the query budget, declared with everything else. + +## What a control cannot tell you + +* A kept column is structure and is never permuted. Leakage through kept + columns, ids or row order passes the null. +* The negative control removes the effect of one column under exchangeable + allocation within the strata. It cannot detect confounding, and on + observational data it is a permutation test with no randomisation + guarantee behind it. +* A planted effect is clean: the same shift for every plot of a genotype, + no confounding, no interaction. The verdict is sensitivity to a clean + effect of the declared size, not power against the real one. +* A shift is added in units of the outcome's standard deviation in the + table at hand, so the same spec plants the same nominal size on the clone + and on the original while the signal-to-noise ratio may differ. The + `@truth` table records the standard deviation used and the residual + standard deviation within the strata on each side. +* Nothing is clipped. A negative shift can take a non-negative outcome + below zero; `control_table()` warns when it does. +* The null collapses the design means of row-level outcomes, so a mixed + model sees a different variance structure. Calibrate false positives on + the negative control, not on the null. + +```{r teardown, include = FALSE} +options(old_options) +``` From 14d1922c657e85ad35031954ecb2459f5c8e8dc5 Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Thu, 5 Nov 2026 17:37:32 +1030 Subject: [PATCH 02/16] docs(controls): state what happens in the roxygen, trim comment blocks Rationale for the seed and the unit moved out of the parameter text into the commit record (prose guard P1, P4); file headers and helper comments cut to the contract. --- R/control_spec.R | 14 +++---------- R/controls.R | 35 +++++++++----------------------- man/control_spec.Rd | 8 +++----- man/controls.Rd | 4 ++-- tests/testthat/helper-controls.R | 6 ++---- 5 files changed, 20 insertions(+), 47 deletions(-) diff --git a/R/control_spec.R b/R/control_spec.R index 339fae9..89043d4 100644 --- a/R/control_spec.R +++ b/R/control_spec.R @@ -1,10 +1,4 @@ # control_spec.R -- control_spec(): the declaration behind controls(). -# -# A control spec names the column whose effect a negative control removes, -# the outcome a positive control plants an effect on, the effect per level in -# outcome-SD units, the design strata and unit of the restricted permutation, -# the unit-grained columns, the replicate count and the seed. It is declared -# before any control table exists and is the record of what was planted. #' Internal: the masque_control_spec S7 class #' @@ -71,16 +65,14 @@ masque_control_spec <- S7::new_class( #' the spec first meets a table. Required with `target`. #' @param strata Design columns within which `on` is permuted, or `NULL` #' for no stratification. Every unit of `on` must lie within one stratum. -#' @param unit Design columns within which `on` is constant, so that it is -#' permuted across those units rather than across rows; `NULL` (the -#' default) permutes rows. +#' @param unit Design columns within which `on` is constant. `on` is then +#' permuted across those units. `NULL` (the default) permutes rows. #' @param grain A named list giving, for other analysed columns constant #' within a design unit, the unit columns: `list(rain = "trial")`. Such a #' column is permuted across its units in the null control, so it stays #' constant within each unit. #' @param n The number of negative (and positive) replicates. -#' @param seed The seed of the control draws. Required, so that a control is -#' reproducible from its declaration. +#' @param seed The seed of the control draws. Required. #' #' @returns A `masque_control_spec` object, with a print method. #' diff --git a/R/controls.R b/R/controls.R index 46a2d00..366eb59 100644 --- a/R/controls.R +++ b/R/controls.R @@ -1,13 +1,6 @@ # controls.R -- controls() and control_table(): null, negative and positive # control datasets derived from a table by restricted permutation and planted -# effects. -# -# A control is a function of the table, its role table, a control spec and a -# seed; nothing is re-simulated. The null permutes every analysed column, the -# negative permutes one declared column within design strata at its unit, and -# the positive adds a declared per-level shift to that negative. The controls -# object holds the table, the resolved spec, one seed per dataset and the -# manifest; control_table() regenerates any dataset from its seed. +# effects. Nothing is re-simulated. #' Internal: the masque_controls S7 class #' @@ -94,8 +87,8 @@ masque_controls <- S7::new_class( #' [propose_roles()] and [set_role()] for any table. #' @param spec A [control_spec()]. #' @param events Linked time/status pairs among the outcome columns, in the -#' form [mask()] takes, so that the null permutes each pair as one block; -#' `NULL` when there are none. +#' form [mask()] takes. The null permutes each pair as one block. `NULL` +#' when there are none. #' @param seed A private seed for the permutations that overrides the #' spec's seed, for the owner's side; `NULL` uses the spec's seed. #' @@ -219,9 +212,7 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) # Resolution ---------------------------------------------------------------- -# Resolve a spec against a table and its roles: fill `on`, check every -# declared column against the roles, build the stratum and unit ids, draw or -# complete the effect vector and compute the truth table. +# Resolve a spec against a table and its roles. .resolve_control_spec <- function(spec, df, roles) { n <- nrow(df) notes <- character() @@ -502,10 +493,7 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) out } -# One row per target and level: the effect in SD units, the target's SD, -# its within-strata residual SD, and the raw and realised shift. A -# whole-valued target takes a whole shift; a shift that rounds away is -# refused. +# One row per target and level; a whole-valued target takes a whole shift. .control_truth <- function(df, target, effect, strata_id) { empty <- data.table::data.table( target = character(), level = character(), effect_sd = double(), @@ -573,11 +561,9 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) # Permutation --------------------------------------------------------------- -# Permute the non-missing values of `x` across the units given by `unit_id`, -# within the groups given by `strata_id`. Each observed unit keeps one value -# (its first non-missing one) and receives another unit's value from the same -# stratum, broadcast to its non-missing rows; an all-missing unit stays -# missing. With `unit_id = seq_along(x)` this is a row permutation. +# Permute the non-missing values of `x` across units within strata; an +# all-missing unit stays missing. With `unit_id = seq_along(x)` it is a row +# permutation. .permute_within <- function(x, unit_id, strata_id) { obs <- which(!is.na(x)) if (length(obs) < 2L) { @@ -631,9 +617,8 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) invisible(out) } -# The positive: add each target's realised shift to the rows of each level -# of the permuted `on`. Nothing is clipped; a non-negative target taken -# below zero is reported. +# The positive: each target's realised shift on the rows of each level of +# the permuted `on`; nothing is clipped. .plant_effect <- function(out, ctl) { on_val <- as.character(out[[ctl@spec@on]]) crossed <- character() diff --git a/man/control_spec.Rd b/man/control_spec.Rd index f9c4618..13c5e6b 100644 --- a/man/control_spec.Rd +++ b/man/control_spec.Rd @@ -38,9 +38,8 @@ the spec first meets a table. Required with \code{target}.} \item{strata}{Design columns within which \code{on} is permuted, or \code{NULL} for no stratification. Every unit of \code{on} must lie within one stratum.} -\item{unit}{Design columns within which \code{on} is constant, so that it is -permuted across those units rather than across rows; \code{NULL} (the -default) permutes rows.} +\item{unit}{Design columns within which \code{on} is constant. \code{on} is then +permuted across those units. \code{NULL} (the default) permutes rows.} \item{grain}{A named list giving, for other analysed columns constant within a design unit, the unit columns: \code{list(rain = "trial")}. Such a @@ -49,8 +48,7 @@ constant within each unit.} \item{n}{The number of negative (and positive) replicates.} -\item{seed}{The seed of the control draws. Required, so that a control is -reproducible from its declaration.} +\item{seed}{The seed of the control draws. Required.} } \value{ A \code{masque_control_spec} object, with a print method. diff --git a/man/controls.Rd b/man/controls.Rd index b7c8a07..47bc4e4 100644 --- a/man/controls.Rd +++ b/man/controls.Rd @@ -15,8 +15,8 @@ controls(df, roles, spec, events = NULL, seed = NULL) \item{spec}{A \code{\link[=control_spec]{control_spec()}}.} \item{events}{Linked time/status pairs among the outcome columns, in the -form \code{\link[=mask]{mask()}} takes, so that the null permutes each pair as one block; -\code{NULL} when there are none.} +form \code{\link[=mask]{mask()}} takes. The null permutes each pair as one block. \code{NULL} +when there are none.} \item{seed}{A private seed for the permutations that overrides the spec's seed, for the owner's side; \code{NULL} uses the spec's seed.} diff --git a/tests/testthat/helper-controls.R b/tests/testthat/helper-controls.R index b7ab639..f128545 100644 --- a/tests/testthat/helper-controls.R +++ b/tests/testthat/helper-controls.R @@ -1,7 +1,5 @@ -# Fixtures for the control datasets: a small MET with a trial-level numeric, -# categorical and date covariate, a plot-level covariate correlated with -# yield, a whole-plot factor, a whole-valued outcome, and missing cells -# including one trial with no covariate value at all. +# Fixtures for the control datasets: a small MET with trial-level covariates, +# a whole-plot factor, a whole-valued outcome and missing cells. .ctl_met <- function(n_trial = 4L, n_rep = 3L, n_gen = 8L, seed = 11L) { withr::with_seed(seed, { trials <- sprintf("T%d", seq_len(n_trial)) From 9666fc6d5a9c1b92d2c48ff31bd0a3203987fd7b Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Thu, 5 Nov 2026 22:49:41 +1030 Subject: [PATCH 03/16] fix(controls): vignette tests the treatment last; strata and limits corrected The vignette tested gen before the blocks, because aov orders interaction terms last, so its pipeline read a false-positive rate of 0.185 (MCSE 0.012) on john_alpha negatives against 0.047 (0.007) with gen last. The real-table p-value moves from 3.6e-7 to 1.5e-5. The docs said a negative control leaves the effect of `on` exactly absent. Within incomplete blocks each block keeps its set of levels, so part of the real effect survives. Examples now use complete replicates as strata, and the vignette and docs state both limits. --- R/control_spec.R | 3 ++- R/controls.R | 11 ++++++----- README.md | 2 +- man/control_spec.Rd | 5 +++-- man/control_table.Rd | 2 +- man/controls.Rd | 9 +++++---- vignettes/controls.Rmd | 30 +++++++++++++++++++----------- 7 files changed, 37 insertions(+), 25 deletions(-) diff --git a/R/control_spec.R b/R/control_spec.R index 89043d4..4db52f8 100644 --- a/R/control_spec.R +++ b/R/control_spec.R @@ -65,6 +65,7 @@ masque_control_spec <- S7::new_class( #' the spec first meets a table. Required with `target`. #' @param strata Design columns within which `on` is permuted, or `NULL` #' for no stratification. Every unit of `on` must lie within one stratum. +#' Prefer complete replicates to incomplete blocks; see [controls()]. #' @param unit Design columns within which `on` is constant. `on` is then #' permuted across those units. `NULL` (the default) permutes rows. #' @param grain A named list giving, for other analysed columns constant @@ -79,7 +80,7 @@ masque_control_spec <- S7::new_class( #' @examples #' spec <- control_spec( #' on = "gen", target = "yield", effect = c(G01 = 0.5, G02 = -0.5), -#' strata = c("rep", "block"), n = 19L, seed = 2026L +#' strata = "rep", n = 19L, seed = 2026L #' ) #' spec #' diff --git a/R/controls.R b/R/controls.R index 366eb59..857693d 100644 --- a/R/controls.R +++ b/R/controls.R @@ -47,9 +47,10 @@ masque_controls <- S7::new_class( #' No association among analysed columns, or with the treatment, remains. #' Use it as a smoke test and to expose leakage through analysed columns. #' * `negative`: the declared column `on` alone is permuted within the -#' declared design strata, at its unit, so its effect on everything is -#' exactly absent while the design structure and all other associations -#' are kept. With `n` replicates it gives a permutation null for a whole +#' declared design strata, at its unit, while the design structure and all +#' other associations are kept. Choose strata that every level of `on` can +#' reach, such as complete replicates: within incomplete blocks each block +#' keeps its set of levels, so part of the real effect of `on` survives. With `n` replicates it gives a permutation null for a whole #' pipeline: a false-positive rate, a permutation p-value. #' * `positive`: the same permutation as the negative replicate, plus the #' declared shift per level of `on` on each target, in units of the @@ -106,7 +107,7 @@ masque_controls <- S7::new_class( #' r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") #' spec <- control_spec( #' on = "gen", target = "yield", effect = list(sd = 0.5), -#' strata = c("rep", "block"), n = 5L, seed = 2026L +#' strata = "rep", n = 5L, seed = 2026L #' ) #' ctl <- controls(trial, r, spec) #' ctl @@ -174,7 +175,7 @@ controls <- function(df, roles, spec, events = NULL, seed = NULL) { #' r <- set_role(r, "gen", role = "treatment", action = "keep") #' r <- set_role(r, "yield", role = "outcome", action = "scramble") #' r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") -#' spec <- control_spec(on = "gen", strata = c("rep", "block"), n = 3L, seed = 1L) +#' spec <- control_spec(on = "gen", strata = "rep", n = 3L, seed = 1L) #' ctl <- controls(trial, r, spec) #' neg <- control_table(ctl, "negative", 2L) #' table(neg$block, neg$gen)[1:2, 1:6] diff --git a/README.md b/README.md index badb2d9..02c5cff 100644 --- a/README.md +++ b/README.md @@ -112,7 +112,7 @@ who returns a verdict about the pipeline rather than a dataset. ```r spec <- control_spec(on = "gen", target = "yield", effect = list(sd = 0.5), - strata = c("rep", "block"), n = 19L, seed = 2026L) + strata = "rep", n = 19L, seed = 2026L) ctl <- controls(synthetic(m), roles(m), spec) control_table(ctl, "positive", 1L) ``` diff --git a/man/control_spec.Rd b/man/control_spec.Rd index 13c5e6b..d7dd4f7 100644 --- a/man/control_spec.Rd +++ b/man/control_spec.Rd @@ -36,7 +36,8 @@ distribution with standard deviation \code{s}, under the spec's seed, when the spec first meets a table. Required with \code{target}.} \item{strata}{Design columns within which \code{on} is permuted, or \code{NULL} -for no stratification. Every unit of \code{on} must lie within one stratum.} +for no stratification. Every unit of \code{on} must lie within one stratum. +Prefer complete replicates to incomplete blocks; see \code{\link[=controls]{controls()}}.} \item{unit}{Design columns within which \code{on} is constant. \code{on} is then permuted across those units. \code{NULL} (the default) permutes rows.} @@ -71,7 +72,7 @@ The positive control needs \code{target} and \code{effect} together. \examples{ spec <- control_spec( on = "gen", target = "yield", effect = c(G01 = 0.5, G02 = -0.5), - strata = c("rep", "block"), n = 19L, seed = 2026L + strata = "rep", n = 19L, seed = 2026L ) spec diff --git a/man/control_table.Rd b/man/control_table.Rd index 3796a6b..6e1431b 100644 --- a/man/control_table.Rd +++ b/man/control_table.Rd @@ -32,7 +32,7 @@ r <- set_role(r, c("rep", "block"), role = "design", action = "keep") r <- set_role(r, "gen", role = "treatment", action = "keep") r <- set_role(r, "yield", role = "outcome", action = "scramble") r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") -spec <- control_spec(on = "gen", strata = c("rep", "block"), n = 3L, seed = 1L) +spec <- control_spec(on = "gen", strata = "rep", n = 3L, seed = 1L) ctl <- controls(trial, r, spec) neg <- control_table(ctl, "negative", 2L) table(neg$block, neg$gen)[1:2, 1:6] diff --git a/man/controls.Rd b/man/controls.Rd index 47bc4e4..f120934 100644 --- a/man/controls.Rd +++ b/man/controls.Rd @@ -43,9 +43,10 @@ columns, the NA mask and every column's multiset of values are kept. No association among analysed columns, or with the treatment, remains. Use it as a smoke test and to expose leakage through analysed columns. \item \code{negative}: the declared column \code{on} alone is permuted within the -declared design strata, at its unit, so its effect on everything is -exactly absent while the design structure and all other associations -are kept. With \code{n} replicates it gives a permutation null for a whole +declared design strata, at its unit, while the design structure and all +other associations are kept. Choose strata that every level of \code{on} can +reach, such as complete replicates: within incomplete blocks each block +keeps its set of levels, so part of the real effect of \code{on} survives. With \code{n} replicates it gives a permutation null for a whole pipeline: a false-positive rate, a permutation p-value. \item \code{positive}: the same permutation as the negative replicate, plus the declared shift per level of \code{on} on each target, in units of the @@ -94,7 +95,7 @@ r <- set_role(r, "yield", role = "outcome", action = "scramble") r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") spec <- control_spec( on = "gen", target = "yield", effect = list(sd = 0.5), - strata = c("rep", "block"), n = 5L, seed = 2026L + strata = "rep", n = 5L, seed = 2026L ) ctl <- controls(trial, r, spec) ctl diff --git a/vignettes/controls.Rmd b/vignettes/controls.Rmd index 85a645f..06620c0 100644 --- a/vignettes/controls.Rmd +++ b/vignettes/controls.Rmd @@ -36,8 +36,8 @@ which the answer is known in advance. associated with anything. A pipeline that still finds signal on it has a leak or a bug. * The **negative** control permutes one declared column, by default the - treatment, within the design strata. Its effect is exactly absent and - everything else is as it was. A pipeline that finds it has a false + treatment, within the design strata. Its allocation is random within them + and everything else is as it was. A pipeline that finds it has a false positive, and over many replicates the rate of those is the pipeline's false-positive rate on this very structure. * The **positive** control adds a declared effect to the negative. The @@ -79,7 +79,7 @@ spec <- control_spec( on = "gen", target = "yield", effect = list(sd = 0.5), - strata = c("rep", "block"), + strata = "rep", n = 19L, seed = 2026L ) @@ -108,9 +108,9 @@ p-value of the genotype term. ```{r pipeline} genotype_p <- function(d) { - fit <- stats::aov(yield ~ rep + rep:block + gen, data = d) - tab <- summary(fit)[[1]] - tab[trimws(rownames(tab)) == "gen", "Pr(>F)"] + blocks <- interaction(d$rep, d$block, drop = TRUE) + fit <- stats::lm(yield ~ blocks + gen, data = d) + stats::anova(fit)["gen", "Pr(>F)"] } ``` @@ -193,11 +193,11 @@ data.frame( ) ``` -The power differs between the two sides. The default clone re-simulates -yield without the block structure, so the pipeline's residual variance is -larger there and the same planted size is harder to see. A clone conditioned -on the design would bring the two readings closer, and the gap between them -says what fidelity this pipeline needs from a clone. +The two sides read close to each other here, within what 19 replicates can +resolve. They need not. The default clone re-simulates yield without the +block structure, so its residual variance differs from the original's, and +any gap between the readings says what fidelity this pipeline needs from a +clone. The control tables built from the original hold the real values and never leave. Per-replicate results do not leave either: each replicate is one @@ -209,6 +209,14 @@ the query budget, declared with everything else. * A kept column is structure and is never permuted. Leakage through kept columns, ids or row order passes the null. +* The strata should let every level of the column reach every stratum, as + complete replicates do. Permuting within incomplete blocks leaves each + block with its own set of genotypes, so part of the real genotype signal + survives on the negatives, and the false-positive rate reads high or low + depending on the layout. +* A sequential test depends on the order of its terms. A treatment entered + before the blocks carries their variance, and its false-positive rate on + this design is well above nominal. The negatives show this, as they should. * The negative control removes the effect of one column under exchangeable allocation within the strata. It cannot detect confounding, and on observational data it is a permutation test with no randomisation From 8998bb31dd23ec54390affcf7666b4ea70d42c47 Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Thu, 5 Nov 2026 23:43:03 +1030 Subject: [PATCH 04/16] fix(coords): key site displacement on the site value and seed A pinned seed reproduced a site's displacement only on a byte-identical table. Draws are now a pure hash of seed, pair, try and site value, so shuffled, added or removed rows leave other sites where they were. Corrects the rebuild table. --- DESCRIPTION | 2 +- NEWS.md | 13 ++ R/controls.R | 7 +- R/jitter_coordinates.R | 75 +++++++---- tests/testthat/test-coord-site-keyed.R | 166 +++++++++++++++++++++++++ 5 files changed, 238 insertions(+), 25 deletions(-) create mode 100644 tests/testthat/test-coord-site-keyed.R diff --git a/DESCRIPTION b/DESCRIPTION index 40c7f99..bfba185 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -29,7 +29,7 @@ Depends: Imports: cli, data.table (>= 1.15.0), - digest, + digest (>= 0.6.21), Matrix, S7, stats, diff --git a/NEWS.md b/NEWS.md index 62a54b1..8b8bc70 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,5 +1,18 @@ # masque (development version) +## Behaviour changes + +* The site displacement of `jitter_coordinates()` and `mask(coords = )` is now + keyed on the site's value and the seed, so one seed gives a site one + displacement even after rows are shuffled, added or removed. In 0.16.0 the + advice to reuse the seed held only for a byte-identical table. **A clone built + by an earlier version moves under the new draw, so rebuilding it counts as a + second release of its sites: retire the old clone.** +* The rebuild table in `?jitter_coordinates` is corrected: 14.6, 9.4, 6.3, 4.4 + and 3.1 km at 1, 2, 4, 8 and 16 kept rebuilds, and seven rebuilds, not eight, + fall below the 5 km floor. +* masque needs digest 0.6.21 or later. + ## New features * Known-truth controls. `control_spec()` declares what a control removes and diff --git a/R/controls.R b/R/controls.R index 857693d..431dd5d 100644 --- a/R/controls.R +++ b/R/controls.R @@ -49,9 +49,10 @@ masque_controls <- S7::new_class( #' * `negative`: the declared column `on` alone is permuted within the #' declared design strata, at its unit, while the design structure and all #' other associations are kept. Choose strata that every level of `on` can -#' reach, such as complete replicates: within incomplete blocks each block -#' keeps its set of levels, so part of the real effect of `on` survives. With `n` replicates it gives a permutation null for a whole -#' pipeline: a false-positive rate, a permutation p-value. +#' reach. Complete replicates qualify. Within incomplete blocks each block +#' keeps its set of levels, so part of the real effect of `on` survives. +#' With `n` replicates it gives a permutation null for a whole pipeline: a +#' false-positive rate, a permutation p-value. #' * `positive`: the same permutation as the negative replicate, plus the #' declared shift per level of `on` on each target, in units of the #' target's standard deviation in the table at hand. Positive minus diff --git a/R/jitter_coordinates.R b/R/jitter_coordinates.R index f862c1b..442092d 100644 --- a/R/jitter_coordinates.R +++ b/R/jitter_coordinates.R @@ -35,9 +35,13 @@ #' slightly between rows. Where such a group holds more than one distinct #' input coordinate, the group's row-weighted centroid is used and the #' consolidation is reported. -#' - `by = FALSE` restores the pre-0.10.0 behaviour of displacing every row -#' independently, and warns if the input carries repeated coordinate pairs. -#' It exists only so a table masked by an earlier version can be reproduced. +#' - `by = FALSE` displaces every row independently, as versions before 0.10.0 +#' did, and warns if the input carries repeated coordinate pairs. The draw +#' follows the row's position, so shuffled rows move differently. +#' +#' The draw is a function of the seed, the two column names and the site's +#' value: its coordinates, or its `by` values. Row order and the other sites +#' in the table play no part. #' #' Displacing each row of a repeated coordinate separately leaks the site. #' Donut displacement is isotropic, so the mean of many independent draws @@ -55,19 +59,24 @@ #' #' \tabular{lr}{ #' Rebuilds kept \tab Median recovery error \cr -#' 1 \tab 14.2 km \cr -#' 2 \tab 10.1 km \cr -#' 4 \tab 6.4 km \cr -#' 8 \tab 4.5 km \cr +#' 1 \tab 14.6 km \cr +#' 2 \tab 9.4 km \cr +#' 4 \tab 6.3 km \cr +#' 8 \tab 4.4 km \cr #' 16 \tab 3.1 km #' } #' -#' Eight kept rebuilds fall below the 5 km floor the donut was asked for. +#' Seven kept rebuilds fall below the 5 km floor the donut was asked for. #' -#' The remedy is to **reuse the seed**. The same `seed` reproduces the -#' same displacement for the same sites, so any number of rebuilds yields one -#' point and nothing to average. Draw a fresh seed only when you intend to -#' supersede every earlier version, and retire the ones you replace. +#' The remedy is to **reuse the seed**. The same `seed` gives a site the same +#' displacement in every rebuild, even when rows are shuffled, added or removed, +#' so any number of rebuilds yields one point and nothing to average. Three +#' things start a new draw for a site: a different seed, a changed site value +#' (a corrected coordinate, a renamed `by` label) and renamed coordinate +#' columns. A clone built before version 0.17.0 used a different draw, so a +#' rebuild under 0.17.0 is a new release of those sites even with the same seed. +#' Treat each of these as a fresh seed: retire the versions it replaces. The seed +#' reveals every displacement, so it stays as confidential as the recipe. #' #' Choose the unit carefully. It is the finest grouping that denotes **one #' physical place at one time**. For a multi-year trial series that is location @@ -198,7 +207,11 @@ jitter_coordinates <- function(df, lat_col, lon_col, land_fn <- .resolve_land_fn(on_land) withr::local_preserve_seed() - if (!is.null(seed)) set.seed(as.integer(.check_seed(seed))) + root <- if (is.null(seed)) { + sample.int(.Machine$integer.max, 1L) + } else { + as.integer(.check_seed(seed)) + } lat <- df[[lat_col]] lon <- df[[lon_col]] @@ -219,8 +232,9 @@ jitter_coordinates <- function(df, lat_col, lon_col, lat_v <- lat[valid] lon_v <- lon[valid] # Rows are one site only when their coordinates are identical as doubles. + # %.17g round-trips a double; adding zero merges -0 with 0. pair_key <- paste( - match(lat_v, unique(lat_v)), match(lon_v, unique(lon_v)), + sprintf("%.17g", lat_v + 0), sprintf("%.17g", lon_v + 0), sep = "\r" ) @@ -302,15 +316,16 @@ jitter_coordinates <- function(df, lat_col, lon_col, tries <- tries + 1L la <- src_lat[pending] lo <- src_lon[pending] + u <- .site_uniforms(root, c(lat_col, lon_col), ug[pending], tries) if (method == "donut") { # sqrt of a uniform on [min^2, max^2] is uniform by AREA over the annulus. - radius <- sqrt(stats::runif(length(pending), min_km^2, max_km^2)) - bearing <- stats::runif(length(pending), 0, 2 * pi) + radius <- sqrt(min_km^2 + u[, 1L] * (max_km^2 - min_km^2)) + bearing <- u[, 2L] * 2 * pi d_north <- radius * cos(bearing) d_east <- radius * sin(bearing) } else { - d_north <- stats::rnorm(length(pending), 0, sd_km) - d_east <- stats::rnorm(length(pending), 0, sd_km) + d_north <- stats::qnorm(u[, 1L], 0, sd_km) + d_east <- stats::qnorm(u[, 2L], 0, sd_km) } cand_lat <- la + d_north / 111.32 cand_lon <- lo + d_east / (111.32 * cos(la * pi / 180)) @@ -349,6 +364,24 @@ jitter_coordinates <- function(df, lat_col, lon_col, ) } +# Two uniforms in (0, 1) per site for one try: a pure function of the seed, +# the coordinate pair, the try and the site's value. Nothing depends on row +# order, on the other sites or on the RNG kind, so a pinned seed gives a site +# the same displacement in any table that holds it. +.site_uniforms <- function(root, pair, keys, try) { + hash <- digest::getVDigest("sha256") + h <- hash( + enc2utf8(paste(root, pair[1L], pair[2L], try, keys, sep = "\r")), + serialize = FALSE + ) + word <- function(from) { + high <- strtoi(substr(h, from, from + 5L), 16L) + low <- strtoi(substr(h, from + 6L, from + 12L), 16L) + (high * 16^7 + low + 0.5) / 16^13 + } + cbind(word(1L), word(14L)) +} + # A structured account of one coordinate-pair jitter, recorded on the recipe. .coord_report <- function(mode, n_sites, n_rows, consolidated = character(), unplaced = 0L) { @@ -579,13 +612,13 @@ jitter_coordinates <- function(df, lat_col, lon_col, } # Jitter each spec in place and return the frame with one report per spec. -# Sites are grouped on `original`; each spec draws from its own sub-stream. +# Sites are grouped on `original`; the pair's names separate its draws. .apply_coord_jitter <- function(synth, specs, seed, original = NULL) { n <- length(specs) if (!n) { return(list(df = synth, reports = list())) } - sub_seeds <- .sub_seeds(seed, n) + root <- if (is.null(seed)) .sub_seeds(NULL, 1L) else seed reports <- vector("list", n) for (i in seq_len(n)) { s <- specs[[i]] @@ -602,7 +635,7 @@ jitter_coordinates <- function(df, lat_col, lon_col, synth, lat_col = s$lat, lon_col = s$lon, by = if (isFALSE(s$by)) FALSE else NULL, method = s$method, min_km = s$min_km, max_km = s$max_km, - sd_km = s$sd_km, on_land = s$on_land, seed = sub_seeds[[i]], + sd_km = s$sd_km, on_land = s$on_land, seed = root, .group = grp ) synth <- res$df diff --git a/tests/testthat/test-coord-site-keyed.R b/tests/testthat/test-coord-site-keyed.R new file mode 100644 index 0000000..5030677 --- /dev/null +++ b/tests/testthat/test-coord-site-keyed.R @@ -0,0 +1,166 @@ +# A site's displacement is a function of the seed and the site's value, never +# of where its rows sit in the table. + +.fx_sites <- function(n_sites = 60L, each = 4L) { + lat <- -37 + (seq_len(n_sites) * 0.0773) %% 5 + lon <- 139 + (seq_len(n_sites) * 0.1319) %% 10 + data.frame( + site = rep(sprintf("S%03d", seq_len(n_sites)), each = each), + lat = rep(lat, each = each), + lon = rep(lon, each = each), + stringsAsFactors = FALSE + ) +} + +.fx_moved <- function(df, out, sites = unique(df$site)) { + i <- match(sites, df$site) + cbind(dlat = out$lat[i] - df$lat[i], dlon = out$lon[i] - df$lon[i]) +} + +test_that("shuffled rows keep every site's displacement", { + df <- .fx_sites() + a <- jitter_coordinates(df, "lat", "lon", on_land = FALSE, seed = 42L) + shuffled <- df[withr::with_seed(9L, sample.int(nrow(df))), ] + b <- jitter_coordinates(shuffled, "lat", "lon", on_land = FALSE, seed = 42L) + expect_identical(.fx_moved(df, a), .fx_moved(shuffled, b, unique(df$site))) +}) + +test_that("an added or removed site leaves the other sites alone", { + df <- .fx_sites() + a <- jitter_coordinates(df, "lat", "lon", on_land = FALSE, seed = 42L) + + extra <- data.frame(site = "NEW", lat = -30, lon = 150) + more <- rbind(extra, df) + b <- jitter_coordinates(more, "lat", "lon", on_land = FALSE, seed = 42L) + expect_identical(.fx_moved(df, a), .fx_moved(more, b, unique(df$site))) + + fewer <- df[df$site != "S001", ] + d <- jitter_coordinates(fewer, "lat", "lon", on_land = FALSE, seed = 42L) + kept <- unique(fewer$site) + expect_identical(.fx_moved(df, a, kept), .fx_moved(fewer, d, kept)) +}) + +test_that("a named site unit is keyed on its value, not its position", { + df <- .fx_sites() + a <- jitter_coordinates( + df, "lat", "lon", by = "site", on_land = FALSE, seed = 42L + ) + shuffled <- rbind(data.frame(site = "NEW", lat = -30, lon = 150), + df[rev(seq_len(nrow(df))), ]) + b <- jitter_coordinates( + shuffled, "lat", "lon", by = "site", on_land = FALSE, seed = 42L + ) + expect_identical(.fx_moved(df, a), .fx_moved(shuffled, b, unique(df$site))) +}) + +test_that("mask() keeps the displacement across a shuffled or extended table", { + df <- .fx_sites() + df$y <- withr::with_seed(1L, stats::rnorm(nrow(df))) + roles <- propose_roles(df, detect = FALSE) + roles$role[roles$col == "site"] <- "design" + roles$action[roles$col == "site"] <- "keep" + roles$role[roles$col == "y"] <- "outcome" + roles$action[roles$col == "y"] <- "scramble" + roles$role[roles$col %in% c("lat", "lon")] <- "covariate" + roles$action[roles$col %in% c("lat", "lon")] <- "keep" + spec <- list(list(lat = "lat", lon = "lon", on_land = FALSE)) + run <- function(d) { + synthetic(suppressWarnings(mask(d, roles, seed = 42L, coords = spec))) + } + a <- run(df) + sh <- df[withr::with_seed(3L, sample.int(nrow(df))), ] + b <- run(sh) + more <- rbind(df, transform(df[1L, ], site = "NEW", lat = -30, lon = 150)) + d <- run(more) + sites <- unique(df$site) + expect_identical(.fx_moved(df, a), .fx_moved(sh, b, sites)) + expect_identical(.fx_moved(df, a), .fx_moved(more, d, sites)) +}) + +test_that("kept rebuilds under one seed add nothing to average", { + df <- .fx_sites() + a <- jitter_coordinates(df, "lat", "lon", on_land = FALSE, seed = 42L) + rebuilds <- lapply(1:8, function(s) { + d <- df[withr::with_seed(s, sample.int(nrow(df))), ] + o <- jitter_coordinates(d, "lat", "lon", on_land = FALSE, seed = 42L) + .fx_moved(d, o, unique(df$site)) + }) + for (r in rebuilds) expect_identical(r, .fx_moved(df, a)) +}) + +test_that("the group mean over many seeds sits on the site, with the floor", { + # The same site under many fresh seeds; its rows share one displacement. + df <- data.frame(lat = rep(-34.9, 8L), lon = rep(138.6, 8L)) + seeds <- 1:4000 + moved <- t(vapply(seeds, function(s) { + o <- jitter_coordinates(df, "lat", "lon", on_land = FALSE, seed = s) + if (nrow(unique(o)) != 1L) stop("a site split apart") + c(o$lat[1L] - df$lat[1L], o$lon[1L] - df$lon[1L]) + }, numeric(2))) + north <- moved[, 1L] * 111.32 + east <- moved[, 2L] * 111.32 * cos(-34.9 * pi / 180) + r <- sqrt(north^2 + east^2) + # The per-axis SD of the 5-20 km donut is sqrt((R^2 + r^2) / 4) = 10.31 km. + mcse <- 10.31 / sqrt(length(seeds)) + expect_lt(abs(mean(north)), 3.5 * mcse) + expect_lt(abs(mean(east)), 3.5 * mcse) + expect_gte(min(r), 5 - 0.01) + expect_lte(max(r), 20 + 0.01) + # Area-uniform radius and uniform bearing, by Kolmogorov-Smirnov. + u_r <- (r^2 - 25) / (400 - 25) + expect_gt(stats::ks.test(pmin(pmax(u_r, 0), 1), "punif")$p.value, 0.001) + u_b <- (atan2(east, north) + pi) / (2 * pi) + expect_gt(stats::ks.test(u_b, "punif")$p.value, 0.001) +}) + +test_that("sites are displaced independently within a table", { + df <- .fx_sites(2000L, 1L) + df <- unique(df) + out <- jitter_coordinates(df, "lat", "lon", on_land = FALSE, seed = 7L) + m <- .fx_moved(df, out) + expect_lt(abs(stats::cor(m[-1L, "dlat"], m[-nrow(m), "dlat"])), 0.1) + expect_lt(abs(stats::cor(m[, "dlat"], m[, "dlon"])), 0.1) +}) + +test_that("the gaussian scheme is keyed the same way and has the right SD", { + df <- .fx_sites(3000L, 1L) + run <- function(d) { + jitter_coordinates(d, "lat", "lon", method = "gaussian", sd_km = 10, + on_land = FALSE, seed = 5L) + } + a <- run(df) + sh <- df[rev(seq_len(nrow(df))), ] + expect_identical(.fx_moved(df, a), .fx_moved(sh, run(sh), unique(df$site))) + expect_lt(abs(stats::sd(.fx_moved(df, a)[, "dlat"]) * 111.32 - 10), 0.8) +}) + +test_that("a land test re-draws from the site's own sequence", { + df <- .fx_sites() + # Reject everything west of 145 E, whatever the row order. + land <- function(lon, lat) lon > 145 + a <- suppressWarnings(jitter_coordinates( + df, "lat", "lon", on_land = land, seed = 42L, max_tries = 40L + )) + sh <- df[rev(seq_len(nrow(df))), ] + b <- suppressWarnings(jitter_coordinates( + sh, "lat", "lon", on_land = land, seed = 42L, max_tries = 40L + )) + expect_identical(.fx_moved(df, a), .fx_moved(sh, b, unique(df$site))) +}) + +test_that("two coordinate pairs at one place draw differently", { + df <- data.frame(a_lat = -34.9, a_lon = 138.6, b_lat = -34.9, b_lon = 138.6) + o <- jitter_coordinates(df, "a_lat", "a_lon", on_land = FALSE, seed = 1L) + o <- jitter_coordinates(o, "b_lat", "b_lon", on_land = FALSE, seed = 1L) + expect_false(identical(o$a_lat, o$b_lat)) +}) + +test_that("draws do not depend on the caller's RNG kind or state", { + df <- .fx_sites() + a <- jitter_coordinates(df, "lat", "lon", on_land = FALSE, seed = 42L) + withr::local_seed(3L, .rng_kind = "L'Ecuyer-CMRG") + before <- .Random.seed + b <- jitter_coordinates(df, "lat", "lon", on_land = FALSE, seed = 42L) + expect_identical(a, b) + expect_identical(before, .Random.seed) +}) From bb29978b6cdd7ffa4353d72a17a3faaf7514165c Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Thu, 5 Nov 2026 23:46:07 +1030 Subject: [PATCH 05/16] feat(mask): conditional = "design" and an explicit conditioning partition The design option keeps block and environment variance with no treatment effect, so a treatment test on the clone has the original's denominator and a true null. Column names give the partition; both run through the existing ladder. --- NEWS.md | 4 + R/mask.R | 51 +++--- R/mask_set.R | 2 +- R/masque.R | 3 +- R/recipe_print.R | 8 +- R/roles_actions.R | 45 ++++++ tests/testthat/test-conditional-partition.R | 167 ++++++++++++++++++++ 7 files changed, 255 insertions(+), 25 deletions(-) create mode 100644 tests/testthat/test-conditional-partition.R diff --git a/NEWS.md b/NEWS.md index 8b8bc70..0331bad 100644 --- a/NEWS.md +++ b/NEWS.md @@ -15,6 +15,10 @@ ## New features +* `mask(conditional = )` takes `"design"`, which conditions on the retained + design columns alone: the clone keeps design means and block variance and + carries no treatment effect. It also takes column names, an explicit + partition that runs through the same ladder. * Known-truth controls. `control_spec()` declares what a control removes and plants; `controls()` derives a null, `n` negative and `n` positive control datasets from any table with a role table, by restricted permutation at diff --git a/R/mask.R b/R/mask.R index 0bcc50c..cd6daf2 100644 --- a/R/mask.R +++ b/R/mask.R @@ -82,8 +82,9 @@ #' [apply_recipe()] / [unmask()], so a pipeline written against the #' aliased synthetic round-trips. Alias the names when the schema itself #' is sensitive. -#' @param conditional Logical scalar (default `FALSE`). The -#' *collaborate-grade conditional clone*. When `FALSE`, scrambled +#' @param conditional `FALSE` (default), `TRUE`, `"design"`, or a character +#' vector of column names. The *collaborate-grade conditional clone*. +#' When `FALSE`, scrambled #' numeric columns are re-simulated from one global Gaussian copula - #' marginals and global covariance survive, but the treatment-to-outcome #' relationship does not, so a causal model fitted on the clone recovers @@ -106,6 +107,15 @@ #' pooling raises a classed `masque_conditional_degraded` warning -- #' including the case where no treatment or design column survives at #' all, in which case the clone is the pooled copula. +#' +#' `"design"` conditions on the retained design columns alone. The clone +#' keeps the design means, block and environment variance, while a treatment +#' effect is not carried, so a test of the treatment on the clone has the +#' denominator it would have on the original and a true effect of zero. +#' A character vector of names gives the partition explicitly, from the +#' retained design and treatment columns, and runs through the same ladder; +#' treatment columns are still never dropped. The single word `"design"` +#' always means this option, even where a column has that name. #' @param ladder How the coarsening ladder orders the design columns it may #' drop, and what it keeps of a dropped one. `"hierarchy"` (the default) #' drops plot coordinates (`row`, `col`, `range`, `plot`) first, then @@ -287,13 +297,7 @@ mask <- function(df, class = c("masque_bad_df_refusal", "orchestra_refusal") ) } - if (!is.logical(conditional) || length(conditional) != 1L || - is.na(conditional)) { - cli::cli_abort( - "`conditional` must be a single `TRUE` or `FALSE`.", - class = c("masque_bad_conditional_refusal", "orchestra_refusal") - ) - } + .check_conditional_arg(conditional) if (!is.logical(grain) || length(grain) != 1L || is.na(grain)) { cli::cli_abort( "`grain` must be a single `TRUE` or `FALSE`.", @@ -429,15 +433,13 @@ mask <- function(df, opts <- mode_defaults(mode) # With nothing to condition on, a conditional clone uses the global copula. - conditioning_cols <- if (isTRUE(conditional)) { - .conditioning_cols(roles) - } else { - character() - } - if (isTRUE(conditional) && !length(conditioning_cols)) { + conditioning_cols <- .resolve_conditioning(conditional, roles) + conditional <- !isFALSE(conditional) + if (conditional && !length(conditioning_cols)) { msg <- paste0( - "conditional = TRUE but no treatment or design column survives to ", - "condition on; numeric synthesis falls back to the global copula." + "A conditional clone was asked for but no treatment or design column ", + "of its partition survives; numeric synthesis falls back to the global ", + "copula." ) # Classed, so a caller can catch the loss of conditional fidelity. warning(warningCondition(msg, class = "masque_conditional_degraded")) @@ -447,7 +449,8 @@ mask <- function(df, result <- with_rng_state( seed, .mask_orchestrate( - df, roles, mode, opts, .shared_maps, conditional, ladder, + df, roles, mode, opts, .shared_maps, conditional, conditioning_cols, + ladder, events = event_specs, grain = grain_res$groups ) ) @@ -703,7 +706,7 @@ mask <- function(df, if (length(dropped)) { parts <- c(parts, sprintf( paste0( - "conditional = TRUE: cells of the finest conditioning stratum held ", + "Conditional clone: cells of the finest conditioning stratum held ", "fewer than %d rows, so the ladder coarsened it by dropping %s. ", "Conditioning on: %s." ), @@ -739,7 +742,9 @@ mask <- function(df, # Internal: per-action synthesis, with the RNG already seeded. The action # column decides what happens to each column; mode only adds numeric jitter. .mask_orchestrate <- function(df, roles, mode, opts, shared_maps = list(), - conditional = FALSE, ladder = "hierarchy", + conditional = FALSE, + conditioning_cols = character(), + ladder = "hierarchy", events = list(), grain = list()) { synth <- df level_maps <- list() @@ -772,7 +777,7 @@ mask <- function(df, if (isTRUE(conditional)) { conditional_report <- .conditioning_ladder( df, - cond_cols = .conditioning_cols(roles), + cond_cols = conditioning_cols, protect_cols = roles$col[role == "treatment" & action != "drop"], min_stratum = .MIN_STRATUM, ladder = ladder, @@ -822,7 +827,9 @@ mask <- function(df, # Linked events: status patterns permuted, gaps re-simulated per pattern. event_reports <- list() if (length(events)) { - trt_cols <- roles$col[role == "treatment" & action != "drop"] + trt_cols <- intersect( + roles$col[role == "treatment" & action != "drop"], conditioning_cols + ) strata <- if (isTRUE(conditional) && length(trt_cols)) { do.call(paste, c(lapply(.cols(df, trt_cols), as.character), sep = "/")) } diff --git a/R/mask_set.R b/R/mask_set.R index 1ab5c72..9b6b940 100644 --- a/R/mask_set.R +++ b/R/mask_set.R @@ -41,7 +41,7 @@ #' @param alias_names Hide column names. `FALSE` (default) keeps them; #' `TRUE` aliases every non-link column (linked join keys keep their #' names so the synthetic set stays joinable). -#' @param conditional Logical scalar (default `FALSE`). Passed through to +#' @param conditional `FALSE` (default), `TRUE` or `"design"`. Passed through to #' each per-table [mask()] call: when `TRUE`, every table's numeric #' block is re-simulated within its own treatment-by-design strata so #' the treatment-to-outcome relationship survives the clone. See diff --git a/R/masque.R b/R/masque.R index 9a28f03..194c3f8 100644 --- a/R/masque.R +++ b/R/masque.R @@ -60,7 +60,8 @@ #' @param clean Hygiene mode passed to [clean_table()] (`"auto"`, #' `"report"`, `"off"`). #' @param alias_names Hide column names; see [mask()] / [mask_set()]. -#' @param conditional Logical scalar (default `FALSE`). The conditional +#' @param conditional `FALSE` (default), `TRUE`, `"design"` or column names. +#' The conditional #' clone mode passed through to [mask()] / [mask_set()]: when `TRUE`, #' numeric columns are re-simulated within each treatment-by-design #' stratum so the treatment-to-outcome relationship survives the clone. diff --git a/R/recipe_print.R b/R/recipe_print.R index c993fc4..7875b4b 100644 --- a/R/recipe_print.R +++ b/R/recipe_print.R @@ -369,8 +369,14 @@ reveal_maps <- function(rec, table = NULL) { if (!length(used)) { return("conditional requested (no conditioning column; pooled fallback)") } + r <- x@roles + has_treatment <- any(requested %in% r$col[r$role == "treatment"]) line <- sprintf( - "conditional (treatment -> outcome preserved; strata: %s)", + if (has_treatment) { + "conditional (treatment -> outcome preserved; strata: %s)" + } else { + "design-conditioned (treatment effect not carried; strata: %s)" + }, paste(used, collapse = ", ") ) dropped <- setdiff(requested, used) diff --git a/R/roles_actions.R b/R/roles_actions.R index d5fbc9a..9f0b1c7 100644 --- a/R/roles_actions.R +++ b/R/roles_actions.R @@ -138,6 +138,51 @@ set_role <- function(roles, cols, role = NULL, action = NULL) { roles$col[keep] } +# `conditional` is one flag, the word "design", or column names. +.check_conditional_arg <- function(conditional) { + ok <- if (is.character(conditional)) { + length(conditional) >= 1L && !anyNA(conditional) && + all(nzchar(conditional)) && !anyDuplicated(conditional) + } else { + is.logical(conditional) && length(conditional) == 1L && !is.na(conditional) + } + if (ok) { + return(invisible(conditional)) + } + cli::cli_abort( + paste0( + "`conditional` must be `TRUE`, `FALSE`, {.val design} or a character ", + "vector of retained design and treatment column names." + ), + class = c("masque_bad_conditional_refusal", "orchestra_refusal") + ) +} + +# The columns a conditional clone conditions on, before the ladder. +.resolve_conditioning <- function(conditional, roles) { + if (isFALSE(conditional)) { + return(character()) + } + retained <- .conditioning_cols(roles) + if (isTRUE(conditional)) { + return(retained) + } + if (identical(conditional, "design")) { + return(intersect(retained, roles$col[roles$role == "design"])) + } + bad <- setdiff(conditional, retained) + if (length(bad)) { + cli::cli_abort( + c( + "`conditional` names {.field {bad}}.", + x = "Only retained design and treatment columns can define a partition." + ), + class = c("masque_bad_conditional_refusal", "orchestra_refusal") + ) + } + conditional +} + # Per-mode default action for a (role, kind) pair, used by propose_roles(), # set_role(), the v1 upgrade and NA-action resolution. .default_action <- function(role, kind, mode) { diff --git a/tests/testthat/test-conditional-partition.R b/tests/testthat/test-conditional-partition.R new file mode 100644 index 0000000..1cd5dc9 --- /dev/null +++ b/tests/testthat/test-conditional-partition.R @@ -0,0 +1,167 @@ +# conditional = "design" and an explicit conditioning partition (N04). + +.fx_blocks <- function(seed = 1L) { + withr::with_seed(seed, { + d <- expand.grid( + rep = paste0("R", 1:3), block = paste0("B", 1:4), plot = 1:6, + stringsAsFactors = FALSE + ) + d$block <- paste(d$rep, d$block, sep = "_") + d$gen <- paste0("G", sample(rep(1:8, length.out = nrow(d)))) + blk <- stats::setNames(stats::rnorm(12L, 0, 3), unique(d$block)) + gen <- stats::setNames(stats::rnorm(8L, 0, 3), paste0("G", 1:8)) + d$y <- 50 + blk[d$block] + gen[d$gen] + stats::rnorm(nrow(d)) + d$plot <- NULL + d + }) +} + +.fx_block_roles <- function(d) { + r <- propose_roles(d, detect = FALSE) + r <- set_role(r, c("rep", "block"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "keep") + set_role(r, "y", role = "outcome", action = "scramble") +} + +.mask_q <- function(...) { + suppressWarnings(mask(..., mode = "local", quiet = TRUE)) +} + +test_that("conditional = 'design' carries design means and nulls treatment", { + d <- .fx_blocks() + r <- .fx_block_roles(d) + block_sd <- function(x) stats::sd(tapply(x$y, x$block, mean)) + gen_sd <- function(x) stats::sd(tapply(x$y, x$gen, mean)) + runs <- lapply(1:30, function(s) { + synthetic(.mask_q(d, r, seed = s, conditional = "design")) + }) + b <- vapply(runs, block_sd, numeric(1)) + g <- vapply(runs, gen_sd, numeric(1)) + expect_lt(abs(mean(b) / block_sd(d) - 1), 0.15) + # The real genotype spread is about 3; a null clone holds only sampling noise. + expect_lt(mean(g), 0.5 * gen_sd(d)) + + # Falsifier: TRUE keeps the genotype spread, FALSE loses the block spread. + cond <- synthetic(.mask_q(d, r, seed = 1L, conditional = TRUE)) + expect_gt(gen_sd(cond), 0.7 * gen_sd(d)) + marg <- lapply(1:10, function(s) { + synthetic(.mask_q(d, r, seed = s, conditional = FALSE)) + }) + # A pooled clone's block means hold only noise: sd(y) / sqrt(6 plots). + expect_lt(mean(vapply(marg, block_sd, numeric(1))), + 1.1 * stats::sd(d$y) / sqrt(6)) + expect_gt(block_sd(d), 1.5 * stats::sd(d$y) / sqrt(6)) +}) + +test_that("the recipe records the design partition without the treatment", { + d <- .fx_blocks() + m <- .mask_q(d, .fx_block_roles(d), seed = 2L, conditional = "design") + rec <- recipe(m) + expect_true(rec@conditional) + expect_setequal(rec@conditioning_cols, c("rep", "block")) + expect_false("gen" %in% rec@conditioning_used) + expect_match(.recipe_fidelity_line(rec), "treatment effect not carried") +}) + +test_that("an explicit partition is used, and TRUE is unchanged by the option", { + d <- .fx_blocks() + r <- .fx_block_roles(d) + a <- .mask_q(d, r, seed = 3L, conditional = c("rep", "block")) + b <- .mask_q(d, r, seed = 3L, conditional = "design") + expect_identical(synthetic(a), synthetic(b)) + + e <- .mask_q(d, r, seed = 3L, conditional = "rep") + expect_identical(recipe(e)@conditioning_used, "rep") + # gen x rep holds three rows a cell: the treatment stays, rep is given up. + f <- .mask_q(d, r, seed = 3L, conditional = c("gen", "rep")) + expect_setequal(recipe(f)@conditioning_cols, c("gen", "rep")) + expect_identical(recipe(f)@conditioning_used, "gen") + + t1 <- .mask_q(d, r, seed = 3L, conditional = TRUE) + t2 <- .mask_q(d, r, seed = 3L, conditional = c("rep", "block", "gen")) + expect_identical( + data.table::setcolorder(data.table::copy(synthetic(t1)), names(d)), + data.table::setcolorder(data.table::copy(synthetic(t2)), names(d)) + ) +}) + +test_that("structure is preserved and the recipe round-trips", { + d <- .fx_blocks() + d$y[c(3L, 40L)] <- NA + r <- .fx_block_roles(d) + m <- .mask_q(d, r, seed = 4L, conditional = "design") + s <- synthetic(m) + expect_identical(s$rep, d$rep) + expect_identical(s$block, d$block) + expect_identical(s$gen, d$gen) + expect_identical(is.na(s$y), is.na(d$y)) + expect_identical(apply_recipe(d, recipe(m))$gen, s$gen) +}) + +test_that("the ladder coarsens a thin design partition and warns", { + d <- .fx_blocks() + d$row <- seq_len(nrow(d)) + r <- propose_roles(d, detect = FALSE) + r <- set_role(r, c("rep", "block", "row"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + expect_warning( + m <- mask(d, r, mode = "local", seed = 5L, quiet = TRUE, + conditional = "design"), + class = "masque_conditional_degraded" + ) + expect_false("row" %in% recipe(m)@conditioning_used) +}) + +test_that("with no design column left the design option warns and pools", { + d <- .fx_blocks() + r <- propose_roles(d, detect = FALSE) + r <- set_role(r, c("rep", "block"), role = "covariate", action = "scramble") + r <- set_role(r, "gen", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + expect_warning( + m <- mask(d, r, mode = "local", seed = 6L, quiet = TRUE, + conditional = "design"), + class = "masque_conditional_degraded" + ) + expect_length(recipe(m)@conditioning_used, 0L) +}) + +test_that("a bad partition is refused by class", { + d <- .fx_blocks() + r <- .fx_block_roles(d) + cls <- "masque_bad_conditional_refusal" + expect_error(.mask_q(d, r, conditional = "nope"), class = cls) + expect_error(.mask_q(d, r, conditional = c("rep", "y")), class = cls) + expect_error(.mask_q(d, r, conditional = character()), class = cls) + expect_error(.mask_q(d, r, conditional = NA_character_), class = cls) + expect_error(.mask_q(d, r, conditional = c("rep", "rep")), class = cls) + expect_error(.mask_q(d, r, conditional = 1), class = cls) + r2 <- set_role(r, "rep", role = "design", action = "drop") + expect_error(.mask_q(d, r2, conditional = "rep"), class = cls) +}) + +test_that("the design option leaves linked events unstratified by treatment", { + n <- 200L + d <- withr::with_seed(8L, data.frame( + arm = rep(c("a", "b"), length.out = n), + centre = rep(c("c1", "c2", "c3", "c4"), each = n / 4L), + t1 = stats::rexp(n, 0.2) + )) + d$t2 <- d$t1 + withr::with_seed(9L, stats::rexp(n, 0.1)) + d$s1 <- withr::with_seed(10L, stats::rbinom(n, 1L, ifelse(d$arm == "a", 0.9, 0.1))) + d$s2 <- 1L + r <- propose_roles(d, detect = FALSE) + r <- set_role(r, "centre", role = "design", action = "keep") + r <- set_role(r, "arm", role = "treatment", action = "keep") + r <- set_role(r, c("t1", "t2", "s1", "s2"), role = "outcome", action = "scramble") + ev <- list(list(time = c("t1", "t2"), status = c("s1", "s2"))) + gap <- function(m) { + s <- synthetic(m) + abs(mean(s$s1[s$arm == "a"]) - mean(s$s1[s$arm == "b"])) + } + des <- .mask_q(d, r, seed = 1L, conditional = "design", events = ev) + tru <- .mask_q(d, r, seed = 1L, conditional = TRUE, events = ev) + expect_lt(gap(des), 0.25) + expect_gt(gap(tru), 0.6) +}) From 99846e925a5ab33503dd2fc3dc331999212ebd01 Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Fri, 6 Nov 2026 00:00:08 +1030 Subject: [PATCH 06/16] feat(plan): masque_plan, mask(df, plan), clone_plan, handover, owner-side controls A plan holds the roles and named specs. mask() resolves each control spec on the source and writes it back; the clone is unchanged by the spec. The analyst works from the clone-vocabulary plan; the owner replays cleaning, checks the table and the spec against the recipe and draws from a private seed. Version 0.17.0. --- API_STABILITY.md | 7 + CITATION.cff | 2 +- DESCRIPTION | 2 +- NAMESPACE | 5 + NEWS.md | 13 +- R/check_integrity.R | 2 +- R/controls.R | 32 +- R/mask.R | 34 +- R/masque.R | 6 + R/plan.R | 776 +++++++++++++++++++++++++++++++++++++ R/recipe.R | 5 + README.md | 5 + _pkgdown.yml | 8 + man/clone_plan.Rd | 30 ++ man/controls.Rd | 25 +- man/handover.Rd | 43 ++ man/jitter_coordinates.Rd | 35 +- man/mask.Rd | 21 +- man/mask_set.Rd | 2 +- man/masque.Rd | 3 +- man/masque_plan.Rd | 69 ++++ man/save_plan.Rd | 34 ++ tests/testthat/test-plan.R | 385 ++++++++++++++++++ vignettes/controls.Rmd | 45 +++ 24 files changed, 1553 insertions(+), 36 deletions(-) create mode 100644 R/plan.R create mode 100644 man/clone_plan.Rd create mode 100644 man/handover.Rd create mode 100644 man/masque_plan.Rd create mode 100644 man/save_plan.Rd create mode 100644 tests/testthat/test-plan.R diff --git a/API_STABILITY.md b/API_STABILITY.md index eafa88b..33c0c60 100644 --- a/API_STABILITY.md +++ b/API_STABILITY.md @@ -107,6 +107,13 @@ If you depend on `masque` before 1.0, set a version floor in `DESCRIPTION` (`Imports: masque (>= 0.14.0)`) and record the exact version with `renv` (`renv::snapshot()`). +- `0.17.0`: new exports `control_spec()`, `controls()`, `control_table()`, + `masque_plan()`, `clone_plan()`, `save_plan()`, `read_plan()`, `handover()`. + `mask(conditional = )` also takes `"design"` or column names, and `roles` + also takes a plan. **A site's coordinate displacement changes under the same + seed**, because the draw is now keyed on the site's value, so a rebuild is a + new release of those sites. No signature breaks. + ## 1.0 and after From `1.0.0`, `masque` adopts a **frozen API**: diff --git a/CITATION.cff b/CITATION.cff index 4c222eb..afcd24c 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -20,7 +20,7 @@ authors: repository-code: "https://github.com/max578/masque" url: "https://max578.github.io/masque/" license: MIT -version: 0.16.0 +version: 0.17.0 keywords: - synthetic-data - data-masking diff --git a/DESCRIPTION b/DESCRIPTION index bfba185..9b34df2 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: masque Title: Structurally Faithful Development Surrogates for Tabular Data -Version: 0.16.0.9000 +Version: 0.17.0 Authors@R: c( person("Max", "Moldovan", email = "max.moldovan@gmail.com", diff --git a/NAMESPACE b/NAMESPACE index 6cf6da1..f5f8df3 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -5,18 +5,22 @@ S3method(print,masque_conformance) export(apply_recipe) export(audit_mask) export(clean_table) +export(clone_plan) export(conform_table) export(control_spec) export(control_table) export(controls) export(data_card) export(detect_design) +export(handover) export(jitter_coordinates) export(mask) export(mask_set) export(masque) +export(masque_plan) export(plot_design_summary) export(propose_roles) +export(read_plan) export(read_recipe) export(read_set) export(recipe) @@ -24,6 +28,7 @@ export(reveal_maps) export(role_options) export(roles) export(roles_validate) +export(save_plan) export(save_recipe) export(set_role) export(synthesise_geospatial) diff --git a/NEWS.md b/NEWS.md index 0331bad..4bd5900 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,4 +1,4 @@ -# masque (development version) +# masque 0.17.0 ## Behaviour changes @@ -15,6 +15,17 @@ ## New features +* `masque_plan()` holds the roles and every named spec in one object. + `mask(df, plan)` reads them from it, an argument passed as well overrides the + plan, and each control spec is resolved on the source and written back to the + recipe. `save_plan()` and `read_plan()` keep a plan as YAML. +* `clone_plan()` gives the plan in clone vocabulary. `controls(clone, plan)` runs + the declared controls for the analyst. `controls(real, plan, recipe = rec)` + runs them for the owner: cleaning and conformance are replayed, a table with a + different schema or NA mask is refused, a spec changed since `mask()` is refused + and the seed is derived privately from the recipe. +* `handover(m, dir)` writes the clone, the clone plan, the data card and a note + to one folder after the release gate. It never writes the recipe. * `mask(conditional = )` takes `"design"`, which conditions on the retained design columns alone: the clone keeps design means and block variance and carries no treatment effect. It also takes column names, an explicit diff --git a/R/check_integrity.R b/R/check_integrity.R index 9028ac0..6264ad4 100644 --- a/R/check_integrity.R +++ b/R/check_integrity.R @@ -32,7 +32,7 @@ "changed since the recipe was built), pass ", "{.code check_integrity = FALSE} to {.fun apply_recipe}." ) - )) + ), class = c("masque_integrity_refusal", "orchestra_refusal")) } invisible(TRUE) } diff --git a/R/controls.R b/R/controls.R index 431dd5d..d16d9da 100644 --- a/R/controls.R +++ b/R/controls.R @@ -86,13 +86,22 @@ masque_controls <- S7::new_class( #' #' @param df The table to derive controls from: a data frame or data.table. #' @param roles Its role table, as returned by [roles()] for a clone or by -#' [propose_roles()] and [set_role()] for any table. -#' @param spec A [control_spec()]. +#' [propose_roles()] and [set_role()] for any table. Or a plan: the analyst +#' passes [clone_plan()] with the clone, the owner passes the plan and the +#' `recipe` with the real table. +#' @param spec A [control_spec()]. With a plan in `roles`, the name of one of +#' its specs, or `NULL` when it holds exactly one. #' @param events Linked time/status pairs among the outcome columns, in the #' form [mask()] takes. The null permutes each pair as one block. `NULL` #' when there are none. #' @param seed A private seed for the permutations that overrides the -#' spec's seed, for the owner's side; `NULL` uses the spec's seed. +#' spec's seed, for the owner's side; `NULL` uses the spec's seed, or with a +#' `recipe` a private seed derived from the recipe's. A value equal to the +#' spec's seed is refused there. +#' @param recipe With a plan in `roles`, the recipe of the clone. `df` is then +#' the real table: its cleaning and conformance are replayed as in +#' [apply_recipe()], a table with a different schema or NA mask is refused, +#' and a spec that differs from the one fixed at [mask()] is refused. #' #' @returns A `masque_controls` object: the resolved spec (`@spec`), the #' manifest (`@manifest`, one row per dataset), the truth table @@ -117,8 +126,21 @@ masque_controls <- S7::new_class( #' @seealso [control_spec()], [control_table()], [mask()], [roles()]. #' #' @export -controls <- function(df, roles, spec, events = NULL, seed = NULL) { +controls <- function(df, roles, spec = NULL, events = NULL, seed = NULL, + recipe = NULL) { withr::local_preserve_seed() + plan_notes <- character() + if (S7::S7_inherits(roles, .plan_class)) { + from_plan <- .controls_from_plan(df, roles, spec, events, seed, recipe) + df <- from_plan$df + roles <- from_plan$roles + spec <- from_plan$spec + events <- from_plan$events + seed <- from_plan$seed + plan_notes <- from_plan$notes + } else if (!is.null(recipe)) { + .control_refuse("`recipe` goes with a plan in `roles`: {.code controls(real, plan, recipe = rec)}.", "recipe") + } .check_control_table(df) .check_control_spec_class(spec) if (!is.null(seed)) { @@ -149,7 +171,7 @@ controls <- function(df, roles, spec, events = NULL, seed = NULL) { grain_units = res$grain_units, manifest = manifest, truth = res$truth, - notes = res$notes + notes = c(res$notes, plan_notes) ) } diff --git a/R/mask.R b/R/mask.R index cd6daf2..fba43a3 100644 --- a/R/mask.R +++ b/R/mask.R @@ -51,7 +51,10 @@ #' [conform_table()], whose `data` element is then masked. Its cleaning #' record and its merges and type changes go on the recipe, so #' [apply_recipe()] replays them; `clean` is not applied a second time. -#' @param roles A roles table from [propose_roles()] (possibly edited). +#' @param roles A roles table from [propose_roles()] (possibly edited), or a +#' [masque_plan()]. A plan supplies `coords`, `events`, `conditional`, +#' `ladder`, `grain` and `alias_names` where the call leaves them out, and its +#' control specs are resolved on `df` and written back to the recipe. #' Tables from masque <= 0.5.0 are upgraded with a deprecation #' warning; see [roles_validate()]. #' @param mode Either `"local"` or `"collaborate"`. When omitted, inherit @@ -283,6 +286,28 @@ mask <- function(df, # Undo any RNG change made inside mask() on exit, whichever path made it. withr::local_preserve_seed() + # A plan fills the arguments the call leaves out. + plan <- NULL + if (!missing(roles) && S7::S7_inherits(roles, .plan_class)) { + plan <- roles + if (!identical(plan@vocabulary, "source")) { + .plan_refuse( + c( + "This plan is in clone vocabulary.", + i = "It describes a clone for {.fn controls}, and cannot mask a table." + ), + "vocabulary" + ) + } + roles <- plan@roles + if (missing(coords) && length(plan@coords)) coords <- plan@coords + if (missing(events) && length(plan@events)) events <- plan@events + if (missing(conditional)) conditional <- plan@conditional + if (missing(ladder)) ladder <- plan@ladder + if (missing(grain)) grain <- plan@grain + if (missing(alias_names)) alias_names <- plan@alias_names + } + # A conformance object carries its own cleaning and the judged decisions; # both go on the recipe so apply_recipe() can replay them. conform_rec <- NULL @@ -434,6 +459,7 @@ mask <- function(df, # With nothing to condition on, a conditional clone uses the global copula. conditioning_cols <- .resolve_conditioning(conditional, roles) + conditional_declared <- conditional conditional <- !isFALSE(conditional) if (conditional && !length(conditioning_cols)) { msg <- paste0( @@ -646,6 +672,12 @@ mask <- function(df, warnings = warnings_acc, integrity_fp = digest::digest(is.na(df), algo = "sha256") ) + if (!is.null(plan)) { + rec@plan <- .plan_resolve(plan, df, roles, rec, list( + coords = coords, events = events, conditional = conditional_declared, + ladder = ladder, grain = grain, alias_names = alias_names + )) + } masque_obj( synthetic = .as_owned_dt(synth), diff --git a/R/masque.R b/R/masque.R index 194c3f8..9eaccb4 100644 --- a/R/masque.R +++ b/R/masque.R @@ -128,6 +128,12 @@ masque <- function(input, clean <- match.arg(clean) ladder <- match.arg(ladder) + if (S7::S7_inherits(roles, .plan_class)) { + .plan_refuse( + "{.fn masque} does not take a plan yet. Call {.code mask(df, plan)}.", + "unsupported" + ) + } is_set <- .masque_is_set(input) if (!is_set && (!is.null(sheets) || !is.null(info))) { cli::cli_abort(c( diff --git a/R/plan.R b/R/plan.R new file mode 100644 index 0000000..6cdf03e --- /dev/null +++ b/R/plan.R @@ -0,0 +1,776 @@ +# plan.R -- masque_plan(): the roles table and every named spec in one object. + +.plan_class <- S7::new_class( + "masque_plan", + properties = list( + roles = S7::class_data.frame, + coords = S7::class_list, + events = S7::class_list, + conditional = S7::new_union(S7::class_logical, S7::class_character), + ladder = S7::class_character, + grain = S7::class_logical, + alias_names = S7::new_union(S7::class_logical, S7::class_character), + controls = S7::class_list, + vocabulary = S7::class_character, + mode = S7::new_union(S7::class_character, NULL) + ) +) + +#' A masking plan: roles and named specs in one object +#' +#' Holds what a [mask()] call needs apart from the table: the roles, the +#' coordinate and event declarations, the conditioning choice and any named +#' [control_spec()]s. `mask(df, plan)` reads them from the plan; an argument +#' passed to [mask()] as well overrides the plan's value for that call. +#' +#' At `mask()` each control spec is resolved on the source table and written +#' back: its unit, its strata, the grain of its columns and its effect vector. +#' The clone does not depend on the specs. The resolved plan is on the recipe, +#' and [clone_plan()] gives the same plan in clone vocabulary for the analyst. +#' +#' A spec's `unit`, where not declared, resolves to the coarsest design +#' partition within which `on` is constant, or to the row. Its `strata`, where +#' not declared, resolve to the finest design partition above the unit in which +#' every stratum holds every level of `on`, or to none. A stratum that misses +#' levels of `on` would keep part of the real effect in a negative control, so +#' incomplete blocks are not chosen by default. +#' +#' @param roles A roles table from [propose_roles()]. +#' @param coords,events,conditional,ladder,grain,alias_names As in [mask()]. +#' @param controls `NULL`, one [control_spec()], or a named list of them. A +#' single spec is named `default`. +#' +#' @returns A `masque_plan` object. +#' +#' @examples +#' trial <- read.csv(system.file("extdata", "john_alpha.csv", package = "masque")) +#' r <- propose_roles(trial, detect = FALSE) +#' r <- set_role(r, c("rep", "block"), role = "design", action = "keep") +#' r <- set_role(r, "gen", role = "treatment", action = "alias") +#' r <- set_role(r, "yield", role = "outcome", action = "scramble") +#' r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") +#' p <- masque_plan( +#' r, +#' conditional = "design", +#' controls = control_spec( +#' on = "gen", target = "yield", effect = list(sd = 0.5), n = 5L, seed = 7L +#' ) +#' ) +#' p +#' m <- mask(trial, p, mode = "local", seed = 1L) +#' clone_plan(m) +#' +#' @seealso [mask()], [clone_plan()], [save_plan()], [controls()], [handover()]. +#' +#' @export +masque_plan <- function( + roles, + coords = NULL, + events = NULL, + conditional = FALSE, + ladder = c("hierarchy", "levels"), + grain = TRUE, + alias_names = FALSE, + controls = NULL +) { + if (!is.data.frame(roles)) { + .plan_refuse("`roles` must be a data frame; got {.cls {class(roles)[1]}}.", "roles") + } + need <- setdiff(c("col", "role", "action", "kind"), names(roles)) + if (length(need)) { + .plan_refuse("`roles` lacks column{?s} {.field {need}}.", "roles") + } + .check_conditional_arg(conditional) + ladder <- match.arg(ladder) + if (!is.logical(grain) || length(grain) != 1L || is.na(grain)) { + .plan_refuse("`grain` must be a single `TRUE` or `FALSE`.", "grain") + } + .plan_class( + roles = roles, + coords = .plan_wrap(coords, c("lat", "lon")), + events = .plan_wrap(events, c("time", "status")), + conditional = conditional, + ladder = ladder, + grain = grain, + alias_names = alias_names, + controls = .plan_controls(controls), + vocabulary = "source", + mode = attr(roles, "mode") + ) +} + +.plan_refuse <- function(msg, what, .envir = parent.frame()) { + cli::cli_abort( + msg, + class = c(paste0("masque_plan_", what, "_refusal"), "orchestra_refusal"), + .envir = .envir + ) +} + +# One spec, or a list of them, as a list of specs. +.plan_wrap <- function(x, keys) { + if (is.null(x)) { + return(list()) + } + if (is.character(x) && length(names(x))) { + x <- as.list(x) + } + if (is.list(x) && any(keys %in% names(x))) { + x <- list(x) + } + x +} + +.plan_controls <- function(controls) { + if (is.null(controls)) { + return(list()) + } + if (S7::S7_inherits(controls, masque_control_spec)) { + return(list(default = controls)) + } + nms <- names(controls) + ok <- is.list(controls) && length(controls) >= 1L && !is.null(nms) && + all(nzchar(nms)) && !anyDuplicated(nms) && + all(vapply(controls, S7::S7_inherits, logical(1L), masque_control_spec)) + if (!ok) { + .plan_refuse( + "`controls` must be one {.fn control_spec} or a named list of them.", + "controls" + ) + } + controls +} + +# Print ---------------------------------------------------------------------- + +# local(): see the plot method in detect_plot.R. +local(S7::method(print, .plan_class) <- function(x, ...) { + cli::cli_h1("masque_plan ({x@vocabulary} vocabulary)") + tally <- table(factor( + x@roles$role, + levels = c("design", "treatment", "outcome", "covariate", "date", "id", "ignore") + )) + tally <- tally[tally > 0L] + cli::cli_bullets(c( + "*" = sprintf( + "Roles: %s", + paste(sprintf("%d %s", tally, names(tally)), collapse = ", ") + ), + "*" = sprintf("Conditional: %s", paste(x@conditional, collapse = ", ")), + "*" = sprintf("Ladder: %s; grain: %s", x@ladder, x@grain), + "*" = sprintf("Column names aliased: %s", paste(x@alias_names, collapse = ", ")), + "*" = sprintf("Coordinate pairs: %d", length(x@coords)), + "*" = sprintf("Linked-event specs: %d", length(x@events)), + "*" = sprintf("Control specs: %d", length(x@controls)) + )) + for (nm in names(x@controls)) { + s <- x@controls[[nm]] + cli::cli_text("{.strong {nm}}") + cli::cli_bullets(c( + " " = sprintf("on %s, unit %s", .spec_on_line(s), .spec_cols_line(s@unit, "row")), + " " = sprintf("strata %s", .spec_cols_line(s@strata, "none")), + " " = sprintf("target %s; %s replicate(s)", .spec_target_line(s), s@n) + )) + } + invisible(x) +}) + +# Resolution on the source, at mask() ---------------------------------------- + +# The plan as it ran, with every spec resolved on the source table. +.plan_resolve <- function(plan, df, roles, rec, args) { + specs <- lapply(plan@controls, .plan_resolve_spec, df = df, roles = roles, rec = rec) + resolved_roles <- as.data.frame(rec@roles) + attr(resolved_roles, "mode") <- rec@mode + .plan_class( + roles = resolved_roles, + coords = .plan_wrap(args$coords, c("lat", "lon")), + events = .plan_wrap(args$events, c("time", "status")), + conditional = args$conditional, + ladder = args$ladder, + grain = args$grain, + alias_names = args$alias_names, + controls = specs, + vocabulary = "source", + mode = rec@mode + ) +} + +.plan_resolve_spec <- function(spec, df, roles, rec) { + design <- intersect( + names(df), roles$col[roles$role == "design" & roles$action != "drop"] + ) + on <- .resolve_control_on(spec@on, df, roles, spec@target) + unit <- if (length(spec@unit)) spec@unit else .default_control_unit(df, on, design) + strata <- if (length(spec@strata)) { + spec@strata + } else { + .default_control_strata(df, on, design, unit) + } + grain <- spec@grain + analysed <- roles$col[ + roles$role %in% c("outcome", "covariate", "date") & + roles$action %in% c("scramble", "alias") + ] + for (g in .prop_or(rec, "grain", list())) { + if (all(g$unit %in% design)) { + for (cl in setdiff(intersect(g$cols, analysed), c(names(grain), spec@target))) { + grain[[cl]] <- g$unit + } + } + } + draft <- masque_control_spec( + on = on, target = spec@target, effect = spec@effect, strata = strata, + unit = unit, grain = grain, n = spec@n, seed = spec@seed + ) + .resolve_control_spec(draft, df, roles)$spec +} + +# Coarsest design column, then pair of columns, within which `on` is constant. +# A partition of mostly single rows says nothing: `on` is constant there by +# construction, so it must average two rows a unit. +.default_control_unit <- function(df, on, design) { + n <- nrow(df) + sets <- c(as.list(design), if (length(design) >= 2L) { + utils::combn(design, 2L, simplify = FALSE) + }) + for (size in 1:2) { + pick <- Filter(function(s) length(s) == size, sets) + n_units <- vapply(pick, function(s) { + id <- as.integer(.unit_id(df, s)) + if (.constant_within(id, df[[on]])) max(id) else NA_integer_ + }, integer(1L)) + n_units[n_units > n / 2] <- NA_integer_ + if (any(!is.na(n_units))) { + return(pick[[which.min(n_units)]]) + } + } + character() +} + +# Finest design partition above the unit whose every stratum holds every level. +.default_control_strata <- function(df, on, design, unit) { + unit_id <- if (length(unit)) as.integer(.unit_id(df, unit)) else seq_len(nrow(df)) + all_levels <- length(unique(df[[on]][!is.na(df[[on]])])) + usable <- function(cols) { + id <- as.integer(.unit_id(df, cols)) + if (max(id) < 2L || !.constant_within(unit_id, id)) { + return(FALSE) + } + held <- tapply(df[[on]], id, function(v) length(unique(v[!is.na(v)]))) + all(held == all_levels) + } + passing <- Filter(usable, setdiff(design, c(unit, on))) + if (!length(passing)) { + return(character()) + } + if (usable(passing)) { + return(passing) + } + sizes <- vapply(passing, function(cl) max(as.integer(.unit_id(df, cl))), integer(1L)) + passing[which.max(sizes)] +} + +# Translation between source and clone vocabulary ---------------------------- + +.plan_col_translator <- function(rec, to) { + map <- rec@column_name_map + from_to <- if (length(map)) { + if (identical(to, "clone")) { + unlist(map) + } else { + stats::setNames(names(map), unlist(map)) + } + } else { + character() + } + function(x) { + if (!length(x)) { + return(x) + } + hits <- x %in% names(from_to) + x[hits] <- unname(from_to[x[hits]]) + x + } +} + +.plan_spec_translate <- function(spec, rec, to) { + rename <- .plan_col_translator(rec, to) + on_source <- if (identical(to, "clone")) spec@on else .plan_col_translator(rec, "source")(spec@on) + effect <- spec@effect + map <- rec@level_maps[[on_source]] + if (is.numeric(effect) && length(map)) { + relabel <- if (identical(to, "clone")) { + .apply_level_map_forward + } else { + .apply_level_map_inverse + } + names(effect) <- as.character(relabel(names(effect), map, col = on_source)) + } + grain <- lapply(spec@grain, rename) + names(grain) <- rename(names(grain)) + masque_control_spec( + on = rename(spec@on), + target = if (is.null(spec@target)) NULL else rename(spec@target), + effect = effect, + strata = rename(spec@strata), + unit = rename(spec@unit), + grain = grain, + n = spec@n, + seed = spec@seed + ) +} + +#' The plan in clone vocabulary, for the analyst +#' +#' The plan of a [mask()] result, in the names and labels of the clone. It +#' carries the roles as [roles()] gives them, the linked events and each control +#' spec as resolved on the source. It holds no coordinate declaration, and +#' nothing else from the recipe. With [controls()] it gives the analyst the +#' pre-declared controls on the clone. +#' +#' @param m A `masque` object from [mask()]. A clone masked without a plan gives +#' a plan with no control specs. +#' +#' @returns A `masque_plan` in clone vocabulary. +#' +#' @examples +#' r <- propose_roles(iris, detect = FALSE) +#' clone_plan(mask(iris, r, mode = "local", seed = 1L)) +#' +#' @seealso [masque_plan()], [controls()], [handover()]. +#' +#' @export +clone_plan <- function(m) { + if (!S7::S7_inherits(m, masque_obj)) { + cli::cli_abort( + "`m` must be a {.cls masque} object; got {.cls {class(m)[1]}}.", + class = c("masque_bad_masque_obj_refusal", "orchestra_refusal") + ) + } + rec <- m@recipe + source_plan <- .prop_or(rec, "plan", NULL) + public <- as.data.frame(.public_roles(rec, m@synthetic)) + attr(public, "mode") <- rec@mode + rename <- .plan_col_translator(rec, "clone") + conditional <- rec@conditional + grain <- TRUE + specs <- list() + if (!is.null(source_plan)) { + conditional <- source_plan@conditional + if (is.character(conditional) && !identical(conditional, "design")) { + conditional <- rename(conditional) + } + grain <- source_plan@grain + specs <- lapply(source_plan@controls, .plan_spec_translate, rec = rec, to = "clone") + } + .plan_class( + roles = public, + coords = list(), + events = .card_event_specs(rec), + conditional = conditional, + ladder = rec@ladder, + grain = grain, + alias_names = FALSE, + controls = specs, + vocabulary = "clone", + mode = rec@mode + ) +} + +# controls() from a plan ------------------------------------------------------ + +# Offset that roots the owner's permutations away from the spec's seed. +.CONTROL_SALT <- 15485863L + +# Arguments of controls() from a plan, on either side of the boundary. +.controls_from_plan <- function(df, plan, spec, events, seed, recipe) { + picked <- .plan_pick_spec(plan, spec) + if (is.null(recipe)) { + if (!identical(plan@vocabulary, "clone")) { + .control_refuse( + c( + "This plan is in source vocabulary, and no `recipe` was given.", + i = "The analyst uses {.fn clone_plan}; the owner passes `recipe =`, which draws from a private seed." + ), + "source_plan" + ) + } + return(list( + df = df, + roles = plan@roles, + spec = picked$spec, + events = if (is.null(events) && length(plan@events)) plan@events else events, + seed = seed, + notes = character() + )) + } + if (!S7::S7_inherits(recipe, masque_recipe)) { + .control_refuse("`recipe` must be a {.cls masque_recipe}; got {.cls {class(recipe)[1]}}.", "recipe") + } + source_spec <- if (identical(plan@vocabulary, "clone")) { + .plan_spec_translate(picked$spec, recipe, "source") + } else { + picked$spec + } + .check_spec_as_declared(source_spec, picked$name, recipe) + list( + df = .controls_replay_source(df, recipe), + roles = as.data.frame(recipe@roles), + spec = source_spec, + events = if (is.null(events) && length(recipe@events)) { + lapply(recipe@events, function(e) list(time = e$time, status = e$status)) + } else { + events + }, + seed = .control_source_seed(recipe@seed, source_spec@seed, seed), + notes = "Source side: the seed is private. Release aggregates only." + ) +} + +.plan_pick_spec <- function(plan, spec) { + specs <- plan@controls + if (!length(specs)) { + .control_refuse("This plan holds no control spec.", "spec") + } + if (is.null(spec)) { + if (length(specs) != 1L) { + nms <- names(specs) + .control_refuse( + "This plan holds {length(specs)} control specs; name one in `spec`: {.val {nms}}.", + "spec" + ) + } + return(list(name = names(specs), spec = specs[[1L]])) + } + if (!is.character(spec) || length(spec) != 1L || !spec %in% names(specs)) { + nms <- names(specs) + .control_refuse("With a plan, `spec` names one of its specs: {.val {nms}}.", "spec") + } + list(name = spec, spec = specs[[spec]]) +} + +# A spec fixed in the plan before any real result exists stays fixed. +.check_spec_as_declared <- function(spec, name, recipe) { + declared <- .prop_or(recipe, "plan", NULL) + if (is.null(declared)) { + return(invisible(TRUE)) + } + on_file <- declared@controls[[name]] + same <- !is.null(on_file) && all(vapply( + c("on", "target", "effect", "strata", "unit", "grain", "n", "seed"), + function(p) isTRUE(all.equal(S7::prop(spec, p), S7::prop(on_file, p))), + logical(1L) + )) + if (!same) { + .control_refuse( + c( + "Control spec {.val {name}} differs from the one fixed at {.fn mask}.", + i = "A control is declared before any real result exists, so the owner runs the spec on the recipe." + ), + "spec_changed" + ) + } + invisible(TRUE) +} + +# The table the clone was built on: cleaning and conformance replayed, the +# schema and NA mask checked. +.controls_replay_source <- function(df, recipe) { + .check_control_table(df) + df <- .apply_cleaning_forward(df, recipe@cleaning) + df <- .apply_conform_forward(df, .prop_or(recipe, "conform", NULL)) + retained <- setdiff(recipe@roles$col, .recipe_dropped_cols(recipe)) + gone <- setdiff(retained, names(df)) + if (length(gone)) { + .control_refuse( + "The table lacks column{?s} the recipe was built on: {.field {gone}}.", + "mismatch" + ) + } + .check_recipe_integrity(df, recipe) + df +} + +.control_source_seed <- function(recipe_seed, spec_seed, seed) { + if (!is.null(seed)) { + .check_seed(seed) + if (as.integer(seed) == spec_seed) { + .control_refuse( + "`seed` equals the spec's seed, which the analyst holds. The source side needs a private seed.", + "seed" + ) + } + return(as.integer(seed)) + } + if (is.null(recipe_seed)) { + .control_refuse( + "The recipe holds no seed to derive a private one from. Pass a private `seed`.", + "seed" + ) + } + root <- (as.numeric(recipe_seed) + as.numeric(spec_seed) + .CONTROL_SALT) %% + .Machine$integer.max + .sub_seeds(root, 1L) +} + +# YAML ------------------------------------------------------------------------ + +#' Save and read a masking plan as YAML +#' +#' A plan on disk is plain, reviewable and diffable. `save_plan()` needs the +#' `yaml` package. A custom `on_land` function cannot be written, and is refused. +#' A source-vocabulary plan names real columns and labels, so it stays with the +#' owner. The analyst's file comes from [clone_plan()] or [handover()]. +#' +#' @param plan A [masque_plan()]. +#' @param path Path of a `.yaml` or `.yml` file. +#' +#' @returns `save_plan()` returns `path` invisibly; `read_plan()` a `masque_plan`. +#' +#' @examples +#' if (requireNamespace("yaml", quietly = TRUE)) { +#' r <- propose_roles(iris, detect = FALSE) +#' f <- tempfile(fileext = ".yaml") +#' save_plan(masque_plan(r), f) +#' read_plan(f) +#' } +#' +#' @export +save_plan <- function(plan, path) { + if (!S7::S7_inherits(plan, .plan_class)) { + .plan_refuse("`plan` must be a {.cls masque_plan}; got {.cls {class(plan)[1]}}.", "class") + } + .check_plan_path(path) + .check_yaml_available() + yaml::write_yaml(.plan_to_list(plan), path, precision = 17L) + invisible(path) +} + +#' @rdname save_plan +#' @export +read_plan <- function(path) { + .check_plan_path(path) + .check_yaml_available() + if (!file.exists(path)) { + .plan_refuse("File not found: {.file {path}}.", "file") + } + x <- tryCatch( + yaml::read_yaml(path), + error = function(e) { + .plan_refuse("{.file {path}} is not valid YAML: {conditionMessage(e)}", "file") + } + ) + if (!is.list(x) || !identical(as.integer(x$masque_plan), 1L)) { + .plan_refuse("{.file {path}} is not a saved masque plan.", "file") + } + .plan_from_list(x) +} + +.check_plan_path <- function(path) { + ok <- is.character(path) && length(path) == 1L && !is.na(path) && + grepl("\\.ya?ml$", path, ignore.case = TRUE) + if (!ok) { + .plan_refuse("`path` must be a single path ending in {.file .yaml} or {.file .yml}.", "file") + } +} + +.check_yaml_available <- function() { + if (!requireNamespace("yaml", quietly = TRUE)) { + .plan_refuse("Writing and reading a plan needs the {.pkg yaml} package.", "yaml") + } +} + +.plan_to_list <- function(plan) { + r <- plan@roles + coords <- lapply(plan@coords, function(s) { + s <- as.list(s) + if (any(vapply(s, is.function, logical(1L)))) { + .plan_refuse("A coordinate spec with an `on_land` function cannot be written to a file.", "file") + } + s + }) + list( + masque_plan = 1L, + vocabulary = plan@vocabulary, + mode = plan@mode, + conditional = as.list(plan@conditional), + ladder = plan@ladder, + grain = plan@grain, + alias_names = as.list(plan@alias_names), + roles = lapply(seq_len(nrow(r)), function(i) { + list(col = r$col[i], role = r$role[i], action = r$action[i], kind = r$kind[i]) + }), + coords = coords, + events = lapply(plan@events, function(e) { + list(time = as.list(e$time), status = as.list(e$status)) + }), + controls = lapply(plan@controls, .spec_to_list) + ) +} + +.spec_to_list <- function(s) { + effect <- s@effect + list( + on = s@on, + target = as.list(s@target), + effect_sd = if (is.list(effect)) effect$sd, + effect = if (is.numeric(effect)) { + lapply(seq_along(effect), function(i) { + list(level = names(effect)[i], shift = effect[[i]]) + }) + }, + strata = as.list(s@strata), + unit = as.list(s@unit), + grain = lapply(seq_along(s@grain), function(i) { + list(col = names(s@grain)[i], unit = as.list(s@grain[[i]])) + }), + n = s@n, + seed = s@seed + ) +} + +.chr <- function(x) as.character(unlist(x)) + +.plan_from_list <- function(x) { + roles <- data.frame( + col = vapply(x$roles, function(r) as.character(r$col), character(1L)), + role = vapply(x$roles, function(r) as.character(r$role), character(1L)), + action = vapply(x$roles, function(r) as.character(r$action), character(1L)), + kind = vapply(x$roles, function(r) as.character(r$kind), character(1L)), + stringsAsFactors = FALSE + ) + attr(roles, "mode") <- x$mode + .plan_class( + roles = roles, + coords = lapply(x$coords, as.list), + events = lapply(x$events, function(e) { + list(time = .chr(e$time), status = .chr(e$status)) + }), + conditional = unlist(x$conditional), + ladder = as.character(x$ladder), + grain = isTRUE(x$grain), + alias_names = unlist(x$alias_names), + controls = lapply(x$controls, .spec_from_list), + vocabulary = as.character(x$vocabulary), + mode = x$mode + ) +} + +.spec_from_list <- function(s) { + effect <- if (!is.null(s$effect_sd)) { + list(sd = as.double(s$effect_sd)) + } else if (length(s$effect)) { + stats::setNames( + vapply(s$effect, function(e) as.double(e$shift), numeric(1L)), + vapply(s$effect, function(e) as.character(e$level), character(1L)) + ) + } + grain <- lapply(s$grain, function(g) .chr(g$unit)) + names(grain) <- vapply(s$grain, function(g) as.character(g$col), character(1L)) + masque_control_spec( + on = if (is.null(s$on)) NULL else as.character(s$on), + target = if (length(s$target)) .chr(s$target), + effect = effect, + strata = .chr(s$strata), + unit = .chr(s$unit), + grain = if (length(grain)) grain else list(), + n = as.integer(s$n), + seed = as.integer(s$seed) + ) +} + +# Handover -------------------------------------------------------------------- + +#' Hand a clone to an analyst +#' +#' Writes what the analyst works from into one folder: the clone (`clone.csv`), +#' the plan in clone vocabulary (`plan.yaml`), the data card (`card.yaml`) and a +#' plain note (`README.txt`). The recipe is never written. The release gate runs +#' first, as for [write_set()]: an unresolved HIGH finding blocks the write +#' unless `allow_high = TRUE`. +#' +#' @param m A `masque` object from [mask()]. +#' @param dir The folder to write. An existing folder that holds files is +#' refused unless `overwrite = TRUE`. +#' @param name Dataset name for the data card; `NULL` uses `"clone"`. +#' @param overwrite Replace the four files if they exist. +#' @param allow_high Write despite HIGH findings, with a warning. +#' +#' @returns The paths written, invisibly. +#' +#' @examples +#' if (requireNamespace("yaml", quietly = TRUE)) { +#' r <- propose_roles(iris, detect = FALSE) +#' m <- mask(iris, r, mode = "local", seed = 1L) +#' d <- tempfile() +#' handover(m, d) +#' list.files(d) +#' } +#' +#' @seealso [clone_plan()], [data_card()], [controls()]. +#' +#' @export +handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE) { + if (!S7::S7_inherits(m, masque_obj)) { + cli::cli_abort( + "`m` must be a {.cls masque} object; got {.cls {class(m)[1]}}.", + class = c("masque_bad_masque_obj_refusal", "orchestra_refusal") + ) + } + if (!is.character(dir) || length(dir) != 1L || is.na(dir) || !nzchar(dir)) { + cli::cli_abort( + "`dir` must be a single non-empty string.", + class = c("masque_bad_handover_dir_refusal", "orchestra_refusal") + ) + } + .check_yaml_available() + high <- .gate_release(m, allow_high) + if (dir.exists(dir) && length(list.files(dir, all.files = TRUE, no.. = TRUE)) && + !isTRUE(overwrite)) { + cli::cli_abort( + c( + "{.file {dir}} already holds files.", + i = "Choose an empty folder, or pass `overwrite = TRUE`." + ), + class = c("masque_handover_exists_refusal", "orchestra_refusal") + ) + } + dir.create(dir, recursive = TRUE, showWarnings = FALSE) + + paths <- file.path(dir, c("clone.csv", "plan.yaml", "card.yaml", "README.txt")) + data.table::fwrite(m@synthetic, paths[1L]) + save_plan(clone_plan(m), paths[2L]) + data_card(m, file = paths[3L], name = name) + writeLines(.handover_note(m, length(high)), paths[4L]) + invisible(paths) +} + +.handover_note <- function(m, n_high) { + plan <- clone_plan(m) + c( + "Masked clone for pipeline development", + "", + sprintf( + "clone.csv holds %d rows and %d columns. An empty cell is a missing value.", + nrow(m@synthetic), ncol(m@synthetic) + ), + "plan.yaml holds the column roles and any declared controls, in the clone's names.", + "card.yaml describes the table for downstream tools.", + "", + "The values are re-simulated. Do not read an effect or an estimate from this table.", + "Write and test your code here. The data owner runs the finished code on the real table.", + if (length(plan@controls)) { + paste0( + "In R, read_plan(\"plan.yaml\") and controls(clone, plan) rebuild the declared ", + "known-truth controls." + ) + }, + if (n_high > 0L) { + sprintf( + "The owner released this clone despite %d HIGH leakage finding(s), after review.", + n_high + ) + } + ) +} diff --git a/R/recipe.R b/R/recipe.R index c5b7236..122d2e8 100644 --- a/R/recipe.R +++ b/R/recipe.R @@ -88,6 +88,11 @@ masque_recipe <- S7::new_class( allow_kept_outcome = S7::new_property( class = S7::class_logical, default = FALSE ), + # The plan as it ran, specs resolved on the source; NULL without a plan. + # Absent before 0.17.0. + plan = S7::new_property( + class = S7::new_union(NULL, .plan_class), default = NULL + ), warnings = S7::new_property( class = S7::class_character, default = character() ), diff --git a/README.md b/README.md index 02c5cff..4fce54a 100644 --- a/README.md +++ b/README.md @@ -117,6 +117,11 @@ ctl <- controls(synthetic(m), roles(m), spec) control_table(ctl, "positive", 1L) ``` +A `masque_plan()` holds the roles and the spec together, so the spec is fixed +before any real result exists. `mask(df, plan)` resolves it on the source, +`handover()` gives the analyst the clone and the plan in clone vocabulary, and +`controls(real, plan, recipe = rec)` runs it on the owner's side. + ## Threat model `masque` is **not** a privacy-preserving or differential-privacy tool. It is a diff --git a/_pkgdown.yml b/_pkgdown.yml index 17fa193..6f53478 100644 --- a/_pkgdown.yml +++ b/_pkgdown.yml @@ -62,6 +62,14 @@ reference: - controls - control_table +- title: Plan and handover + desc: Roles and named specs in one object, and what an analyst receives. + contents: + - masque_plan + - clone_plan + - save_plan + - handover + - title: Geospatial post-processing desc: Coarsen or re-anchor synthetic coordinates while keeping them on land and plausible. contents: diff --git a/man/clone_plan.Rd b/man/clone_plan.Rd new file mode 100644 index 0000000..caec068 --- /dev/null +++ b/man/clone_plan.Rd @@ -0,0 +1,30 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/plan.R +\name{clone_plan} +\alias{clone_plan} +\title{The plan in clone vocabulary, for the analyst} +\usage{ +clone_plan(m) +} +\arguments{ +\item{m}{A \code{masque} object from \code{\link[=mask]{mask()}}. A clone masked without a plan gives +a plan with no control specs.} +} +\value{ +A \code{masque_plan} in clone vocabulary. +} +\description{ +The plan of a \code{\link[=mask]{mask()}} result, in the names and labels of the clone. It +carries the roles as \code{\link[=roles]{roles()}} gives them, the linked events and each control +spec as resolved on the source. It holds no coordinate declaration, and +nothing else from the recipe. With \code{\link[=controls]{controls()}} it gives the analyst the +pre-declared controls on the clone. +} +\examples{ +r <- propose_roles(iris, detect = FALSE) +clone_plan(mask(iris, r, mode = "local", seed = 1L)) + +} +\seealso{ +\code{\link[=masque_plan]{masque_plan()}}, \code{\link[=controls]{controls()}}, \code{\link[=handover]{handover()}}. +} diff --git a/man/controls.Rd b/man/controls.Rd index f120934..355e68f 100644 --- a/man/controls.Rd +++ b/man/controls.Rd @@ -4,22 +4,32 @@ \alias{controls} \title{Derive null, negative and positive controls from a table} \usage{ -controls(df, roles, spec, events = NULL, seed = NULL) +controls(df, roles, spec = NULL, events = NULL, seed = NULL, recipe = NULL) } \arguments{ \item{df}{The table to derive controls from: a data frame or data.table.} \item{roles}{Its role table, as returned by \code{\link[=roles]{roles()}} for a clone or by -\code{\link[=propose_roles]{propose_roles()}} and \code{\link[=set_role]{set_role()}} for any table.} +\code{\link[=propose_roles]{propose_roles()}} and \code{\link[=set_role]{set_role()}} for any table. Or a plan: the analyst +passes \code{\link[=clone_plan]{clone_plan()}} with the clone, the owner passes the plan and the +\code{recipe} with the real table.} -\item{spec}{A \code{\link[=control_spec]{control_spec()}}.} +\item{spec}{A \code{\link[=control_spec]{control_spec()}}. With a plan in \code{roles}, the name of one of +its specs, or \code{NULL} when it holds exactly one.} \item{events}{Linked time/status pairs among the outcome columns, in the form \code{\link[=mask]{mask()}} takes. The null permutes each pair as one block. \code{NULL} when there are none.} \item{seed}{A private seed for the permutations that overrides the -spec's seed, for the owner's side; \code{NULL} uses the spec's seed.} +spec's seed, for the owner's side; \code{NULL} uses the spec's seed, or with a +\code{recipe} a private seed derived from the recipe's. A value equal to the +spec's seed is refused there.} + +\item{recipe}{With a plan in \code{roles}, the recipe of the clone. \code{df} is then +the real table: its cleaning and conformance are replayed as in +\code{\link[=apply_recipe]{apply_recipe()}}, a table with a different schema or NA mask is refused, +and a spec that differs from the one fixed at \code{\link[=mask]{mask()}} is refused.} } \value{ A \code{masque_controls} object: the resolved spec (\verb{@spec}), the @@ -45,9 +55,10 @@ Use it as a smoke test and to expose leakage through analysed columns. \item \code{negative}: the declared column \code{on} alone is permuted within the declared design strata, at its unit, while the design structure and all other associations are kept. Choose strata that every level of \code{on} can -reach, such as complete replicates: within incomplete blocks each block -keeps its set of levels, so part of the real effect of \code{on} survives. With \code{n} replicates it gives a permutation null for a whole -pipeline: a false-positive rate, a permutation p-value. +reach. Complete replicates qualify. Within incomplete blocks each block +keeps its set of levels, so part of the real effect of \code{on} survives. +With \code{n} replicates it gives a permutation null for a whole pipeline: a +false-positive rate, a permutation p-value. \item \code{positive}: the same permutation as the negative replicate, plus the declared shift per level of \code{on} on each target, in units of the target's standard deviation in the table at hand. Positive minus diff --git a/man/handover.Rd b/man/handover.Rd new file mode 100644 index 0000000..985839b --- /dev/null +++ b/man/handover.Rd @@ -0,0 +1,43 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/plan.R +\name{handover} +\alias{handover} +\title{Hand a clone to an analyst} +\usage{ +handover(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE) +} +\arguments{ +\item{m}{A \code{masque} object from \code{\link[=mask]{mask()}}.} + +\item{dir}{The folder to write. An existing folder that holds files is +refused unless \code{overwrite = TRUE}.} + +\item{name}{Dataset name for the data card; \code{NULL} uses \code{"clone"}.} + +\item{overwrite}{Replace the four files if they exist.} + +\item{allow_high}{Write despite HIGH findings, with a warning.} +} +\value{ +The paths written, invisibly. +} +\description{ +Writes what the analyst works from into one folder: the clone (\code{clone.csv}), +the plan in clone vocabulary (\code{plan.yaml}), the data card (\code{card.yaml}) and a +plain note (\code{README.txt}). The recipe is never written. The release gate runs +first, as for \code{\link[=write_set]{write_set()}}: an unresolved HIGH finding blocks the write +unless \code{allow_high = TRUE}. +} +\examples{ +if (requireNamespace("yaml", quietly = TRUE)) { + r <- propose_roles(iris, detect = FALSE) + m <- mask(iris, r, mode = "local", seed = 1L) + d <- tempfile() + handover(m, d) + list.files(d) +} + +} +\seealso{ +\code{\link[=clone_plan]{clone_plan()}}, \code{\link[=data_card]{data_card()}}, \code{\link[=controls]{controls()}}. +} diff --git a/man/jitter_coordinates.Rd b/man/jitter_coordinates.Rd index 762b4f5..8efe5bb 100644 --- a/man/jitter_coordinates.Rd +++ b/man/jitter_coordinates.Rd @@ -100,11 +100,15 @@ values as one site, for the case where a site's recorded coordinates differ slightly between rows. Where such a group holds more than one distinct input coordinate, the group's row-weighted centroid is used and the consolidation is reported. -\item \code{by = FALSE} restores the pre-0.10.0 behaviour of displacing every row -independently, and warns if the input carries repeated coordinate pairs. -It exists only so a table masked by an earlier version can be reproduced. +\item \code{by = FALSE} displaces every row independently, as versions before 0.10.0 +did, and warns if the input carries repeated coordinate pairs. The draw +follows the row's position, so shuffled rows move differently. } +The draw is a function of the seed, the two column names and the site's +value: its coordinates, or its \code{by} values. Row order and the other sites +in the table play no part. + Displacing each row of a repeated coordinate separately leaks the site. Donut displacement is isotropic, so the mean of many independent draws around one true site converges on that site: averaging the 360 rows of @@ -123,19 +127,24 @@ out, exactly as its rows could be before. Measured at the 5-20 km default: \tabular{lr}{ Rebuilds kept \tab Median recovery error \cr -1 \tab 14.2 km \cr -2 \tab 10.1 km \cr -4 \tab 6.4 km \cr -8 \tab 4.5 km \cr +1 \tab 14.6 km \cr +2 \tab 9.4 km \cr +4 \tab 6.3 km \cr +8 \tab 4.4 km \cr 16 \tab 3.1 km } -Eight kept rebuilds fall below the 5 km floor the donut was asked for. - -The remedy is to \strong{reuse the seed}. The same \code{seed} reproduces the -same displacement for the same sites, so any number of rebuilds yields one -point and nothing to average. Draw a fresh seed only when you intend to -supersede every earlier version, and retire the ones you replace. +Seven kept rebuilds fall below the 5 km floor the donut was asked for. + +The remedy is to \strong{reuse the seed}. The same \code{seed} gives a site the same +displacement in every rebuild, even when rows are shuffled, added or removed, +so any number of rebuilds yields one point and nothing to average. Three +things start a new draw for a site: a different seed, a changed site value +(a corrected coordinate, a renamed \code{by} label) and renamed coordinate +columns. A clone built before version 0.17.0 used a different draw, so a +rebuild under 0.17.0 is a new release of those sites even with the same seed. +Treat each of these as a fresh seed: retire the versions it replaces. The seed +reveals every displacement, so it stays as confidential as the recipe. Choose the unit carefully. It is the finest grouping that denotes \strong{one physical place at one time}. For a multi-year trial series that is location diff --git a/man/mask.Rd b/man/mask.Rd index d382d92..067f11e 100644 --- a/man/mask.Rd +++ b/man/mask.Rd @@ -30,7 +30,10 @@ mask( record and its merges and type changes go on the recipe, so \code{\link[=apply_recipe]{apply_recipe()}} replays them; \code{clean} is not applied a second time.} -\item{roles}{A roles table from \code{\link[=propose_roles]{propose_roles()}} (possibly edited). +\item{roles}{A roles table from \code{\link[=propose_roles]{propose_roles()}} (possibly edited), or a +\code{\link[=masque_plan]{masque_plan()}}. A plan supplies \code{coords}, \code{events}, \code{conditional}, +\code{ladder}, \code{grain} and \code{alias_names} where the call leaves them out, and its +control specs are resolved on \code{df} and written back to the recipe. Tables from masque <= 0.5.0 are upgraded with a deprecation warning; see \code{\link[=roles_validate]{roles_validate()}}.} @@ -66,8 +69,9 @@ original-to-alias map is stored in the recipe and inverted by aliased synthetic round-trips. Alias the names when the schema itself is sensitive.} -\item{conditional}{Logical scalar (default \code{FALSE}). The -\emph{collaborate-grade conditional clone}. When \code{FALSE}, scrambled +\item{conditional}{\code{FALSE} (default), \code{TRUE}, \code{"design"}, or a character +vector of column names. The \emph{collaborate-grade conditional clone}. +When \code{FALSE}, scrambled numeric columns are re-simulated from one global Gaussian copula - marginals and global covariance survive, but the treatment-to-outcome relationship does not, so a causal model fitted on the clone recovers @@ -89,7 +93,16 @@ into a global fallback. The rung reached is recorded on the recipe as the pooled share as \code{fallback_frac}, and any coarsening or residual pooling raises a classed \code{masque_conditional_degraded} warning -- including the case where no treatment or design column survives at -all, in which case the clone is the pooled copula.} +all, in which case the clone is the pooled copula. + +\code{"design"} conditions on the retained design columns alone. The clone +keeps the design means, block and environment variance, while a treatment +effect is not carried, so a test of the treatment on the clone has the +denominator it would have on the original and a true effect of zero. +A character vector of names gives the partition explicitly, from the +retained design and treatment columns, and runs through the same ladder; +treatment columns are still never dropped. The single word \code{"design"} +always means this option, even where a column has that name.} \item{ladder}{How the coarsening ladder orders the design columns it may drop, and what it keeps of a dropped one. \code{"hierarchy"} (the default) diff --git a/man/mask_set.Rd b/man/mask_set.Rd index 9e0270e..b932f3f 100644 --- a/man/mask_set.Rd +++ b/man/mask_set.Rd @@ -51,7 +51,7 @@ modes must be reconciled explicitly. Otherwise default to \code{"local"}.} \code{TRUE} aliases every non-link column (linked join keys keep their names so the synthetic set stays joinable).} -\item{conditional}{Logical scalar (default \code{FALSE}). Passed through to +\item{conditional}{\code{FALSE} (default), \code{TRUE} or \code{"design"}. Passed through to each per-table \code{\link[=mask]{mask()}} call: when \code{TRUE}, every table's numeric block is re-simulated within its own treatment-by-design strata so the treatment-to-outcome relationship survives the clone. See diff --git a/man/masque.Rd b/man/masque.Rd index 95a64c1..306af40 100644 --- a/man/masque.Rd +++ b/man/masque.Rd @@ -52,7 +52,8 @@ in \code{\link[=mask]{mask()}} and \code{\link[=mask_set]{mask_set()}}; with no \item{alias_names}{Hide column names; see \code{\link[=mask]{mask()}} / \code{\link[=mask_set]{mask_set()}}.} -\item{conditional}{Logical scalar (default \code{FALSE}). The conditional +\item{conditional}{\code{FALSE} (default), \code{TRUE}, \code{"design"} or column names. +The conditional clone mode passed through to \code{\link[=mask]{mask()}} / \code{\link[=mask_set]{mask_set()}}: when \code{TRUE}, numeric columns are re-simulated within each treatment-by-design stratum so the treatment-to-outcome relationship survives the clone. diff --git a/man/masque_plan.Rd b/man/masque_plan.Rd new file mode 100644 index 0000000..92265b9 --- /dev/null +++ b/man/masque_plan.Rd @@ -0,0 +1,69 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/plan.R +\name{masque_plan} +\alias{masque_plan} +\title{A masking plan: roles and named specs in one object} +\usage{ +masque_plan( + roles, + coords = NULL, + events = NULL, + conditional = FALSE, + ladder = c("hierarchy", "levels"), + grain = TRUE, + alias_names = FALSE, + controls = NULL +) +} +\arguments{ +\item{roles}{A roles table from \code{\link[=propose_roles]{propose_roles()}}.} + +\item{coords, events, conditional, ladder, grain, alias_names}{As in \code{\link[=mask]{mask()}}.} + +\item{controls}{\code{NULL}, one \code{\link[=control_spec]{control_spec()}}, or a named list of them. A +single spec is named \code{default}.} +} +\value{ +A \code{masque_plan} object. +} +\description{ +Holds what a \code{\link[=mask]{mask()}} call needs apart from the table: the roles, the +coordinate and event declarations, the conditioning choice and any named +\code{\link[=control_spec]{control_spec()}}s. \code{mask(df, plan)} reads them from the plan; an argument +passed to \code{\link[=mask]{mask()}} as well overrides the plan's value for that call. +} +\details{ +At \code{mask()} each control spec is resolved on the source table and written +back: its unit, its strata, the grain of its columns and its effect vector. +The clone does not depend on the specs. The resolved plan is on the recipe, +and \code{\link[=clone_plan]{clone_plan()}} gives the same plan in clone vocabulary for the analyst. + +A spec's \code{unit}, where not declared, resolves to the coarsest design +partition within which \code{on} is constant, or to the row. Its \code{strata}, where +not declared, resolve to the finest design partition above the unit in which +every stratum holds every level of \code{on}, or to none. A stratum that misses +levels of \code{on} would keep part of the real effect in a negative control, so +incomplete blocks are not chosen by default. +} +\examples{ +trial <- read.csv(system.file("extdata", "john_alpha.csv", package = "masque")) +r <- propose_roles(trial, detect = FALSE) +r <- set_role(r, c("rep", "block"), role = "design", action = "keep") +r <- set_role(r, "gen", role = "treatment", action = "alias") +r <- set_role(r, "yield", role = "outcome", action = "scramble") +r <- set_role(r, c("plot", "row", "col"), role = "id", action = "keep") +p <- masque_plan( + r, + conditional = "design", + controls = control_spec( + on = "gen", target = "yield", effect = list(sd = 0.5), n = 5L, seed = 7L + ) +) +p +m <- mask(trial, p, mode = "local", seed = 1L) +clone_plan(m) + +} +\seealso{ +\code{\link[=mask]{mask()}}, \code{\link[=clone_plan]{clone_plan()}}, \code{\link[=save_plan]{save_plan()}}, \code{\link[=controls]{controls()}}, \code{\link[=handover]{handover()}}. +} diff --git a/man/save_plan.Rd b/man/save_plan.Rd new file mode 100644 index 0000000..077e660 --- /dev/null +++ b/man/save_plan.Rd @@ -0,0 +1,34 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/plan.R +\name{save_plan} +\alias{save_plan} +\alias{read_plan} +\title{Save and read a masking plan as YAML} +\usage{ +save_plan(plan, path) + +read_plan(path) +} +\arguments{ +\item{plan}{A \code{\link[=masque_plan]{masque_plan()}}.} + +\item{path}{Path of a \code{.yaml} or \code{.yml} file.} +} +\value{ +\code{save_plan()} returns \code{path} invisibly; \code{read_plan()} a \code{masque_plan}. +} +\description{ +A plan on disk is plain, reviewable and diffable. \code{save_plan()} needs the +\code{yaml} package. A custom \code{on_land} function cannot be written, and is refused. +A source-vocabulary plan names real columns and labels, so it stays with the +owner. The analyst's file comes from \code{\link[=clone_plan]{clone_plan()}} or \code{\link[=handover]{handover()}}. +} +\examples{ +if (requireNamespace("yaml", quietly = TRUE)) { + r <- propose_roles(iris, detect = FALSE) + f <- tempfile(fileext = ".yaml") + save_plan(masque_plan(r), f) + read_plan(f) +} + +} diff --git a/tests/testthat/test-plan.R b/tests/testthat/test-plan.R new file mode 100644 index 0000000..94a65ca --- /dev/null +++ b/tests/testthat/test-plan.R @@ -0,0 +1,385 @@ +# masque_plan(): mask(df, plan), resolution, translation, both control sides. + +.fx_plan_table <- function() { + withr::with_seed(11L, { + d <- expand.grid( + site = c("Adelong", "Booleroo"), rep = c("R1", "R2", "R3"), + block = paste0("B", 1:4), plot = 1:5, + stringsAsFactors = FALSE + ) + n <- nrow(d) + d$block <- paste(d$site, d$rep, d$block, sep = "_") + d$gen <- NA_character_ + for (k in unique(paste(d$site, d$rep))) { + i <- which(paste(d$site, d$rep) == k) + d$gen[i] <- sample(rep(paste0("Variety", 1:10), 2L)) + } + d$yield <- 10 + stats::rnorm(n) + d$yield[c(5L, 90L)] <- NA + d$plot <- NULL + d + }) +} + +.fx_plan_roles <- function(d) { + r <- propose_roles(d, mode = "local", detect = FALSE) + r <- set_role(r, c("site", "rep", "block"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + set_role(r, "yield", role = "outcome", action = "scramble") +} + +.fx_spec <- function(...) { + control_spec(on = "gen", target = "yield", effect = list(sd = 0.5), n = 4L, + seed = 77L, ...) +} + +test_that("mask(df, plan) equals mask() with the same arguments", { + d <- .fx_plan_table() + r <- .fx_plan_roles(d) + p <- masque_plan(r, conditional = "design", grain = FALSE) + a <- suppressWarnings(mask(d, p, seed = 3L, quiet = TRUE)) + b <- suppressWarnings(mask(d, r, seed = 3L, quiet = TRUE, + conditional = "design", grain = FALSE)) + expect_identical(synthetic(a), synthetic(b)) +}) + +test_that("an argument passed to mask() overrides the plan", { + d <- .fx_plan_table() + r <- .fx_plan_roles(d) + p <- masque_plan(r, conditional = TRUE) + a <- suppressWarnings(mask(d, p, seed = 3L, quiet = TRUE, conditional = FALSE)) + b <- suppressWarnings(mask(d, r, seed = 3L, quiet = TRUE)) + expect_identical(synthetic(a), synthetic(b)) + expect_false(recipe(a)@plan@conditional) +}) + +test_that("the clone is identical with and without a control spec", { + d <- .fx_plan_table() + r <- .fx_plan_roles(d) + a <- mask(d, masque_plan(r), seed = 5L, quiet = TRUE) + b <- mask(d, masque_plan(r, controls = .fx_spec()), seed = 5L, quiet = TRUE) + expect_identical(synthetic(a), synthetic(b)) +}) + +test_that("mask() resolves unit, strata, effect on the source and writes back", { + d <- .fx_plan_table() + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), + seed = 5L, quiet = TRUE) + s <- recipe(m)@plan@controls$default + # Every site and rep holds every variety; a block does not. + expect_setequal(s@strata, c("site", "rep")) + expect_length(s@unit, 0L) + expect_true(is.numeric(s@effect)) + expect_setequal(names(s@effect), paste0("Variety", 1:10)) + again <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), + seed = 99L, quiet = TRUE) + expect_identical(recipe(again)@plan@controls$default@effect, s@effect) +}) + +test_that("a whole-plot unit and declared strata resolve", { + d <- .fx_plan_table() + d$irrig <- ifelse(as.integer(factor(d$block)) %% 2L == 0L, "wet", "dry") + r <- .fx_plan_roles(d) + r <- set_role(r, "gen", role = "covariate", action = "scramble") + r <- set_role(r, "irrig", role = "treatment", action = "keep") + p <- masque_plan(r, controls = control_spec(on = "irrig", n = 2L, seed = 1L)) + m <- suppressWarnings(mask(d, p, seed = 5L, quiet = TRUE)) + s <- recipe(m)@plan@controls$default + expect_identical(s@unit, "block") + expect_setequal(s@strata, c("site", "rep")) + + p2 <- masque_plan(r, controls = control_spec( + on = "irrig", strata = "site", unit = "block", n = 2L, seed = 1L + )) + m2 <- suppressWarnings(mask(d, p2, seed = 5L, quiet = TRUE)) + expect_identical(recipe(m2)@plan@controls$default@strata, "site") +}) + +test_that("a spec the source cannot honour is refused at mask()", { + d <- .fx_plan_table() + r <- .fx_plan_roles(d) + bad <- masque_plan(r, controls = control_spec(on = "yield", n = 2L, seed = 1L)) + expect_error(mask(d, bad, seed = 1L, quiet = TRUE), class = "orchestra_refusal") + bad2 <- masque_plan(r, controls = control_spec( + on = "gen", strata = "nope", n = 2L, seed = 1L + )) + expect_error(mask(d, bad2, seed = 1L, quiet = TRUE), class = "orchestra_refusal") +}) + +test_that("the clone plan is in clone vocabulary, with nothing of the source", { + d <- .fx_plan_table() + r <- .fx_plan_roles(d) + m <- suppressWarnings(mask( + d, masque_plan(r, controls = .fx_spec(), alias_names = TRUE), + seed = 5L, quiet = TRUE + )) + cp <- clone_plan(m) + expect_identical(cp@vocabulary, "clone") + s <- cp@controls$default + expect_true(all(c(s@on, s@target, s@strata) %in% names(synthetic(m)))) + expect_false(any(c("gen", "yield", "site", "rep") %in% c(s@on, s@target, s@strata))) + clone_on <- as.character(synthetic(m)[[s@on]]) + expect_setequal(names(s@effect), unique(clone_on)) + expect_false(any(grepl("Variety", names(s@effect)))) + expect_length(cp@coords, 0L) + # Falsifier: the source plan does hold the source vocabulary. + expect_true(any(grepl("Variety", names(recipe(m)@plan@controls$default@effect)))) +}) + +test_that("the effect means the same variety on both sides", { + d <- .fx_plan_table() + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), + seed = 5L, quiet = TRUE) + src <- recipe(m)@plan@controls$default@effect + cln <- clone_plan(m)@controls$default@effect + map <- recipe(m)@level_maps$gen + expect_equal(unname(cln[unname(map[names(src)])]), unname(src)) +}) + +test_that("the analyst's controls run on the clone from the clone plan", { + d <- .fx_plan_table() + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), + seed = 5L, quiet = TRUE) + ctl <- controls(synthetic(m), clone_plan(m)) + neg <- control_table(ctl, "negative", 2L) + s <- synthetic(m) + expect_identical(neg$site, s$site) + expect_identical(is.na(neg$yield), is.na(s$yield)) + tab <- function(x) table(x$site, x$rep, x$gen) + expect_identical(tab(neg), tab(s)) + expect_false(identical(neg$gen, s$gen)) + pos <- control_table(ctl, "positive", 2L) + shift <- pos$yield - neg$yield + eff <- ctl@spec@effect + expect_equal( + tapply(shift, pos$gen, mean, na.rm = TRUE)[names(eff)], + eff * stats::sd(s$yield, na.rm = TRUE), + ignore_attr = TRUE + ) +}) + +test_that("the owner's controls take a private seed and the real table", { + d <- .fx_plan_table() + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), + seed = 5L, quiet = TRUE) + cp <- clone_plan(m) + rec <- recipe(m) + own <- controls(d, cp, recipe = rec) + ana <- controls(synthetic(m), cp) + # The seeds differ, and neither is the other's permutation. + expect_false(identical(own@seeds, ana@seeds)) + expect_false(identical(own@seeds, as.integer(.sub_seeds(77L, 5L)))) + # The owner's table is the real one, in real labels. + n1 <- control_table(own, "negative", 1L) + expect_setequal(unique(n1$gen), paste0("Variety", 1:10)) + expect_identical(sort(n1$yield), sort(d$yield)) + expect_true(any(grepl("private", own@notes))) + # Source plan and clone plan give the same owner's object. + own2 <- controls(d, rec@plan, recipe = rec) + expect_identical(own2@seeds, own@seeds) + expect_identical(own2@spec@effect, own@spec@effect) +}) + +test_that("the source side refuses what it should", { + d <- .fx_plan_table() + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), + seed = 5L, quiet = TRUE) + cp <- clone_plan(m) + rec <- recipe(m) + + # A source plan without a recipe would permute real rows under a public seed. + expect_error(controls(d, rec@plan), class = "masque_control_source_plan_refusal") + # The spec's seed is the analyst's. + expect_error(controls(d, cp, recipe = rec, seed = 77L), + class = "masque_control_seed_refusal") + # A different NA mask is a different table. + other <- d + other$yield[1L] <- NA + expect_error(controls(other, cp, recipe = rec), class = "masque_integrity_refusal") + # A missing column. + gone <- d + gone$rep <- NULL + expect_error(controls(gone, cp, recipe = rec), class = "masque_control_mismatch_refusal") + # A spec changed after the clone was made. + changed <- cp + s <- changed@controls$default + s@n <- 9L + changed@controls$default <- s + expect_error(controls(d, changed, recipe = rec), + class = "masque_control_spec_changed_refusal") + # A recipe with a plain roles table. + expect_error( + controls(d, .fx_plan_roles(d), .fx_spec(), recipe = rec), + class = "masque_control_recipe_refusal" + ) + # No recipe seed and no private seed. + m0 <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), quiet = TRUE) + expect_error( + controls(d, clone_plan(m0), recipe = recipe(m0)), + class = "masque_control_seed_refusal" + ) + own <- controls(d, clone_plan(m0), recipe = recipe(m0), seed = 123L) + expect_true(S7::S7_inherits(own, masque_controls)) +}) + +test_that("cleaning is replayed on the owner's table", { + d <- .fx_plan_table() + names(d)[names(d) == "yield"] <- "yield t/ha" + expect_warning( + cl <- clean_table(d, quiet = TRUE)$data, + class = "masque_name_repaired" + ) + yield_col <- setdiff(names(cl), c("site", "rep", "block", "gen")) + r <- propose_roles(cl, mode = "local", detect = FALSE) + r <- set_role(r, c("site", "rep", "block"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, yield_col, role = "outcome", action = "scramble") + spec <- control_spec(on = "gen", target = yield_col, effect = list(sd = 0.5), + n = 2L, seed = 77L) + m <- suppressWarnings(mask(d, masque_plan(r, controls = spec), seed = 5L, quiet = TRUE)) + own <- suppressWarnings(controls(d, clone_plan(m), recipe = recipe(m))) + expect_true(yield_col %in% names(control_table(own, "negative", 1L))) +}) + +test_that("several named specs are picked by name", { + d <- .fx_plan_table() + specs <- list( + a = .fx_spec(), + b = control_spec(on = "gen", n = 2L, seed = 8L) + ) + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = specs), seed = 5L, quiet = TRUE) + cp <- clone_plan(m) + expect_error(controls(synthetic(m), cp), class = "masque_control_spec_refusal") + expect_error(controls(synthetic(m), cp, "c"), class = "masque_control_spec_refusal") + expect_identical(controls(synthetic(m), cp, "b")@spec@n, 2L) +}) + +test_that("a plan refuses bad input and the wrong verb", { + d <- .fx_plan_table() + r <- .fx_plan_roles(d) + expect_error(masque_plan(1), class = "masque_plan_roles_refusal") + expect_error(masque_plan(r[, c("col", "role")]), class = "masque_plan_roles_refusal") + expect_error(masque_plan(r, conditional = 2), class = "masque_bad_conditional_refusal") + expect_error(masque_plan(r, grain = NA), class = "masque_plan_grain_refusal") + expect_error(masque_plan(r, controls = list(1)), class = "masque_plan_controls_refusal") + expect_error(masque(d, roles = masque_plan(r)), class = "masque_plan_unsupported_refusal") + m <- mask(d, masque_plan(r), seed = 1L, quiet = TRUE) + expect_error(mask(d, clone_plan(m)), class = "masque_plan_vocabulary_refusal") + expect_error(clone_plan(1), class = "masque_bad_masque_obj_refusal") +}) + +test_that("a plan with no spec still gives a clone plan and refuses controls", { + d <- .fx_plan_table() + m <- suppressWarnings(mask(d, .fx_plan_roles(d), seed = 1L, quiet = TRUE)) + cp <- clone_plan(m) + expect_length(cp@controls, 0L) + expect_error(controls(synthetic(m), cp), class = "masque_control_spec_refusal") +}) + +test_that("printing a plan shows no seed and no level", { + d <- .fx_plan_table() + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), + seed = 5L, quiet = TRUE) + out <- paste(cli::cli_fmt(print(clone_plan(m))), collapse = "\n") + expect_match(out, "clone vocabulary") + expect_match(out, "default") + expect_false(grepl("Variety|77", out)) +}) + +test_that("a plan round-trips through YAML, with awkward labels", { + skip_if_not_installed("yaml") + d <- .fx_plan_table() + awkward <- c("NA", "yes", "1", "0.5", "a: b", "true", "~", "null", "#x", "é") + labs <- stats::setNames(c(awkward), paste0("Variety", 1:10)) + d$gen <- unname(labs[d$gen]) + r <- .fx_plan_roles(d) + r <- set_role(r, "gen", role = "treatment", action = "keep") + m <- mask(d, masque_plan(r, conditional = "design", controls = .fx_spec()), + seed = 5L, quiet = TRUE) + src <- recipe(m)@plan + f <- withr::local_tempfile(fileext = ".yaml") + save_plan(src, f) + back <- read_plan(f) + expect_identical(back@controls$default@effect, src@controls$default@effect) + expect_identical(back@controls$default@strata, src@controls$default@strata) + expect_identical(back@conditional, src@conditional) + expect_identical(back@roles$col, src@roles$col) + expect_identical(back@roles$action, src@roles$action) + expect_identical(back@vocabulary, "source") + expect_identical(back@mode, "local") + a <- suppressWarnings(mask(d, back, seed = 5L, quiet = TRUE)) + expect_identical(synthetic(a), synthetic(m)) + + # The owner's spec check holds after a round trip of the clone plan. + g <- withr::local_tempfile(fileext = ".yaml") + save_plan(clone_plan(m), g) + own <- controls(d, read_plan(g), recipe = recipe(m)) + expect_true(S7::S7_inherits(own, masque_controls)) +}) + +test_that("reading and writing refuse what they cannot do", { + skip_if_not_installed("yaml") + d <- .fx_plan_table() + r <- .fx_plan_roles(d) + f <- withr::local_tempfile(fileext = ".yaml") + expect_error(save_plan(r, f), class = "masque_plan_class_refusal") + expect_error(save_plan(masque_plan(r), "a.csv"), class = "masque_plan_file_refusal") + expect_error(read_plan("nope.yaml"), class = "masque_plan_file_refusal") + writeLines("a: 1", f) + expect_error(read_plan(f), class = "masque_plan_file_refusal") + writeLines("a: [", f) + expect_error(read_plan(f), class = "masque_plan_file_refusal") + p <- masque_plan(r, coords = list(lat = "a", lon = "b", on_land = function(lon, lat) TRUE)) + expect_error(save_plan(p, f), class = "masque_plan_file_refusal") +}) + +test_that("handover writes four files and no source vocabulary", { + skip_if_not_installed("yaml") + d <- .fx_plan_table() + r <- set_role(.fx_plan_roles(d), c("site", "block"), action = "alias") + m <- suppressWarnings(mask( + d, masque_plan(r, controls = .fx_spec(), alias_names = TRUE), + seed = 5L, quiet = TRUE + )) + dir <- withr::local_tempdir() + out <- file.path(dir, "pack") + paths <- handover(m, out) + expect_setequal(basename(paths), c("clone.csv", "plan.yaml", "card.yaml", "README.txt")) + expect_setequal(list.files(out), basename(paths)) + text <- unlist(lapply(paths, readLines)) + for (w in c("Variety", "Adelong", "Booleroo", "yield")) { + expect_false(any(grepl(w, text, fixed = TRUE)), info = w) + } + # Falsifier: the same scan finds the plan's words in the source plan's file. + own_file <- withr::local_tempfile(fileext = ".yaml") + save_plan(recipe(m)@plan, own_file) + own_text <- readLines(own_file) + for (w in c("Variety", "yield")) { + expect_true(any(grepl(w, own_text, fixed = TRUE)), info = w) + } + expect_error(handover(m, out), class = "masque_handover_exists_refusal") + handover(m, out, overwrite = TRUE) + + # The file round trip the note describes. + clone <- data.table::fread(file.path(out, "clone.csv")) + ctl <- controls(clone, read_plan(file.path(out, "plan.yaml"))) + expect_true(S7::S7_inherits(ctl, masque_controls)) +}) + +test_that("handover runs the release gate", { + skip_if_not_installed("yaml") + d <- .fx_plan_table() + d$id_copy <- seq_len(nrow(d)) + r <- .fx_plan_roles(d[, setdiff(names(d), "id_copy")]) + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, c("site", "rep", "block"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, c("yield", "id_copy"), role = "outcome", action = "keep") + m <- suppressWarnings(mask(d, r, mode = "collaborate", seed = 1L, quiet = TRUE, + allow_kept_outcome = TRUE)) + out <- file.path(withr::local_tempdir(), "pack") + expect_error(handover(m, out), class = "masque_blocked_write") + expect_false(dir.exists(out)) + expect_warning(handover(m, out, allow_high = TRUE), class = "masque_high_override") + expect_true(any(grepl("HIGH", readLines(file.path(out, "README.txt"))))) +}) diff --git a/vignettes/controls.Rmd b/vignettes/controls.Rmd index 06620c0..1e011c2 100644 --- a/vignettes/controls.Rmd +++ b/vignettes/controls.Rmd @@ -205,6 +205,51 @@ more query against the original, and with the permutations known they are linear equations in the real outcome. The replicate count in the spec is the query budget, declared with everything else. +## Carrying the spec in a plan + +So far the spec has been a separate object. A plan fixes it in advance, with the +roles, and takes it through the boundary. At `mask()` the spec is resolved on +the source and recorded on the recipe. Where a unit or strata are not declared, +the plan picks them: here the replicate, since each replicate holds every +genotype and a block does not. The clone is unchanged by the spec. + +```{r plan-mask} +r_plan <- set_role(r, "gen", role = "treatment", action = "alias") +plan <- masque_plan( + r_plan, + conditional = "design", + controls = control_spec( + on = "gen", target = "yield", effect = list(sd = 0.5), n = 19L, seed = 2026L + ) +) +m_plan <- suppressWarnings(mask(trial, plan, mode = "local", seed = 1L)) +recipe(m_plan)@plan@controls$default@strata +``` + +`conditional = "design"` keeps the design means and block variance and carries +no genotype effect, so a test of genotype on this clone has the denominator it +would have on the original. + +The analyst gets the plan in the clone's own names and labels. `handover()` +writes it to a folder with the clone, the data card and a note, and leaves the +recipe out. From there the analyst rebuilds the declared controls. + +```{r plan-analyst} +analyst_plan <- clone_plan(m_plan) +ctl_analyst <- controls(synthetic(m_plan), analyst_plan) +ctl_analyst@spec@strata +``` + +The owner passes the same plan with the recipe and the real table. Cleaning is +replayed, a table that does not match the recipe is refused, a spec changed since +`mask()` is refused, and the seed is derived from the recipe, so the analyst +cannot rebuild these permutations. + +```{r plan-owner} +ctl_owner_plan <- controls(trial, analyst_plan, recipe = recipe(m_plan)) +identical(ctl_owner_plan@seeds, ctl_analyst@seeds) +``` + ## What a control cannot tell you * A kept column is structure and is never permuted. Leakage through kept From 7d5baab27870223a66a2fbb1f189fd9b9e38f4ea Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Fri, 6 Nov 2026 00:29:22 +1030 Subject: [PATCH 07/16] fix: close disclosure faults found in review before the 0.17.0 tag The clone plan's effect is drawn over sorted clone labels, so its order and the public seed give away no map. Dropped on or target, a spec seed equal to the mask seed, a plan passed to mask_set(), and a local clone at handover are refused. A design level needs three rows to carry a shift, and the audit grades a number that tracks the original row by row. Private seeds leave the manifest. --- NEWS.md | 12 + R/controls.R | 8 +- R/jitter_coordinates.R | 17 +- R/leakage.R | 54 ++++- R/mask.R | 13 +- R/mask_set.R | 6 + R/plan.R | 171 +++++++++++-- R/synthesise_numeric_conditional.R | 34 ++- man/handover.Rd | 22 +- man/jitter_coordinates.Rd | 10 +- man/mask.Rd | 9 +- man/save_plan.Rd | 2 +- tests/testthat/test-conditional-disclosure.R | 114 +++++++++ tests/testthat/test-coord-site-invariant.R | 2 +- tests/testthat/test-coord-site-keyed.R | 13 + tests/testthat/test-mask-conditional-ladder.R | 3 +- tests/testthat/test-plan.R | 229 +++++++++++++++++- .../testthat/test-synthesise_numeric_jitter.R | 6 +- vignettes/confidentiality.Rmd | 2 +- vignettes/controls.Rmd | 10 +- 20 files changed, 669 insertions(+), 68 deletions(-) create mode 100644 tests/testthat/test-conditional-disclosure.R diff --git a/NEWS.md b/NEWS.md index 4bd5900..9865caf 100644 --- a/NEWS.md +++ b/NEWS.md @@ -11,6 +11,18 @@ * The rebuild table in `?jitter_coordinates` is corrected: 14.6, 9.4, 6.3, 4.4 and 3.1 km at 1, 2, 4, 8 and 16 kept rebuilds, and seven rebuilds, not eight, fall below the 5 km floor. +* Rows are one site when their coordinates agree to nine decimal places, so a + table written to a file and read back keeps its draw. +* The audit grades a re-simulated number that tracks the original row by row + (r at or above 0.95): HIGH where its conditioning cell holds ten rows or fewer, + MEDIUM otherwise. A conditional clone of tiny strata can do this. +* Under the hierarchy ladder a design level must hold three rows to carry a shift. + A level of two rows gave a shift close to those rows' own values: a clone + correlated 0.99 with the original at its rows, and the audit said low. Conditional + clones with such a column change. +* A private `seed` in `controls()` no longer appears in the manifest. +* The averaged-row figure in `?jitter_coordinates` is corrected to 0.64 km for + 360 rows. * masque needs digest 0.6.21 or later. ## New features diff --git a/R/controls.R b/R/controls.R index d16d9da..ccd8312 100644 --- a/R/controls.R +++ b/R/controls.R @@ -159,6 +159,10 @@ controls <- function(df, roles, spec = NULL, events = NULL, seed = NULL, seeds <- as.integer(.sub_seeds(run_seed, res$spec@n + 1L)) manifest <- .control_manifest(res, seeds) + # A manifest is what the owner reads before a release: no private seed in it. + if (!is.null(seed)) { + manifest$seed <- NA_integer_ + } masque_controls( table = df, roles = .as_owned_dt(roles), @@ -395,7 +399,9 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) "on" ) } - categorical <- kind %in% .categorical_kinds() + # An aliased numeric is a set of labels, on both sides. + action <- roles$action[match(on, roles$col)] + categorical <- kind %in% .categorical_kinds() || action %in% c("alias", "permute") whole <- kind %in% .numeric_kinds() && .is_whole_valued(df[[on]]) if (!categorical && !whole) { .control_refuse( diff --git a/R/jitter_coordinates.R b/R/jitter_coordinates.R index 442092d..bbcc009 100644 --- a/R/jitter_coordinates.R +++ b/R/jitter_coordinates.R @@ -27,8 +27,8 @@ #' source table has one true coordinate per place. `by` chooses how a site is #' identified: #' -#' - `by = NULL` (default) treats rows that share an identical input coordinate -#' pair as one site. This needs no user action and cannot alter genuinely +#' - `by = NULL` (default) treats rows that share an input coordinate pair, to +#' nine decimal places, as one site. This needs no user action and cannot alter genuinely #' point-level data, where every input pair is already distinct. #' - `by = c("trial")` (one or more column names) treats rows sharing those #' values as one site, for the case where a site's recorded coordinates differ @@ -46,7 +46,7 @@ #' Displacing each row of a repeated coordinate separately leaks the site. #' Donut displacement is isotropic, so the mean of many independent draws #' around one true site converges on that site: averaging the 360 rows of -#' a single trial recovers the true position to a median 0.58 km, against the +#' a single trial recovers the true position to a median 0.64 km, against the #' 5 km floor the donut was asked for. One draw per site removes the estimator #' and leaves the full displacement in place. #' @@ -76,7 +76,9 @@ #' columns. A clone built before version 0.17.0 used a different draw, so a #' rebuild under 0.17.0 is a new release of those sites even with the same seed. #' Treat each of these as a fresh seed: retire the versions it replaces. The seed -#' reveals every displacement, so it stays as confidential as the recipe. +#' reveals every displacement, so it stays as confidential as the recipe. Use a +#' large random integer: where a clone shows site labels, or holds two pairs at +#' one place, a small seed can be found by search. #' #' Choose the unit carefully. It is the finest grouping that denotes **one #' physical place at one time**. For a multi-year trial series that is location @@ -231,10 +233,11 @@ jitter_coordinates <- function(df, lat_col, lon_col, lat_v <- lat[valid] lon_v <- lon[valid] - # Rows are one site only when their coordinates are identical as doubles. - # %.17g round-trips a double; adding zero merges -0 with 0. + # Rows are one site when their coordinates agree to nine decimal places, about + # 0.1 mm, so writing the table to a file and back does not start a new draw. + # Adding zero merges -0 with 0. pair_key <- paste( - sprintf("%.17g", lat_v + 0), sprintf("%.17g", lon_v + 0), + sprintf("%.9f", lat_v + 0), sprintf("%.9f", lon_v + 0), sep = "\r" ) diff --git a/R/leakage.R b/R/leakage.R index 6ddf582..1bf57a4 100644 --- a/R/leakage.R +++ b/R/leakage.R @@ -16,6 +16,7 @@ if (is.null(coarsened)) coarsened <- character() na_pattern_uniq <- .global_na_pattern_uniqueness(df) strata <- .audit_strata(df, rec) + row_cell <- .audit_row_cell(df, rec) design_cols <- intersect( rec@roles$col[rec@roles$role == "design"], names(df) ) @@ -75,6 +76,18 @@ } } + # A re-simulated number that tracks the original at its own row. + row_cor <- NA_real_ + if (in_synth && kind %in% c("numeric", "integer") && + .audit_numeric_resimulated(action) && !is_grained && !two_valued && + !(col %in% linked) && comparable_n >= 10L) { + y <- synth[[col]] + both <- !is.na(x) & !is.na(y) + if (stats::sd(x[both]) > 0 && stats::sd(y[both]) > 0) { + row_cor <- stats::cor(x[both], y[both]) + } + } + # Same-row match against a permutation baseline: categorical, date and # two-valued columns, compared in the original label space. same <- NULL @@ -122,7 +135,8 @@ two_valued = two_valued, linked = col %in% linked, by_excess = by_excess, grained = is_grained, allow_kept_outcome = allow_kept_outcome, - value_set_pct = if (is.null(value_set)) NA_real_ else value_set$pct + value_set_pct = if (is.null(value_set)) NA_real_ else value_set$pct, + row_cor = row_cor, row_cell = row_cell ) notes <- .audit_notes( @@ -134,6 +148,12 @@ row_grain = Filter(function(g) col %in% g$cols, row_grain), value_set = value_set ) + if (isTRUE(abs(row_cor) >= .AUDIT_THRESHOLDS$row_cor)) { + notes <- paste0( + notes, if (nzchar(notes)) "; ", + sprintf("tracks the original row by row (r = %.2f)", row_cor) + ) + } data.table::data.table( col = col, @@ -166,6 +186,8 @@ same_row_medium = list(excess = 10, z = 4), same_row_high = list(excess = 50, z = 4), value_set_high = 50, + row_cor = 0.95, + row_cor_cell = 10L, joint_medium = list(excess = 5, z = 4), joint_high = list(excess = 20, z = 4), rare_medium = list(excess = 1, z = 3), @@ -183,6 +205,20 @@ excess >= rule$z * sd) } +# Median rows per cell of the columns a clone's numbers were conditioned or +# shifted on; every row when there are none. +.audit_row_cell <- function(df, rec) { + terms <- unlist(strsplit( + c(rec@conditioning_used, .prop_or(rec, "conditioning_shifted", character())), + ":", fixed = TRUE + )) + cols <- intersect(unique(terms), names(df)) + if (!length(cols)) { + return(nrow(df)) + } + stats::median(tabulate(.row_group_id(.cols(df, cols)))) +} + # A scrambled numeric column is re-simulated through the copula, within the # conditioning strata when the clone is conditional. .audit_numeric_resimulated <- function(action) { @@ -583,7 +619,8 @@ exact_match_pct, freq_min, n_unique_levels, n_rows, coarsened = FALSE, same_row_excess = NA_real_, same_row_sd = NA_real_, two_valued = FALSE, linked = FALSE, by_excess = FALSE, grained = FALSE, - allow_kept_outcome = FALSE, value_set_pct = NA_real_ + allow_kept_outcome = FALSE, value_set_pct = NA_real_, row_cor = NA_real_, + row_cell = NA_integer_ ) { collab <- identical(mode, "collaborate") @@ -598,6 +635,13 @@ return(if (collab) "high" else "medium") } + # 0b. A re-simulated number that tracks the original row by row, in a small + # conditioning cell whose mean is close to the rows themselves: HIGH. + tracks <- collab && isTRUE(abs(row_cor) >= .AUDIT_THRESHOLDS$row_cor) + if (tracks && isTRUE(row_cell <= .AUDIT_THRESHOLDS$row_cor_cell)) { + return("high") + } + # 1b. Outcome kept as real values beside re-simulated columns in # collaborate: HIGH, MEDIUM under `allow_kept_outcome = TRUE`. if (role == "outcome" && collab && in_synth && !is.na(action) && @@ -628,6 +672,12 @@ return("high") } + # 3c. The same in a large cell, where the clone may only keep big environment + # means: MEDIUM. + if (tracks) { + return("medium") + } + # 4. Outcome with exact-match-pct > 1% in collaborate: MEDIUM. Two-valued # columns match by chance about half the time; rule 5b grades them. if (role == "outcome" && collab && !isTRUE(two_valued) && diff --git a/R/mask.R b/R/mask.R index fba43a3..008953b 100644 --- a/R/mask.R +++ b/R/mask.R @@ -113,8 +113,10 @@ #' #' `"design"` conditions on the retained design columns alone. The clone #' keeps the design means, block and environment variance, while a treatment -#' effect is not carried, so a test of the treatment on the clone has the -#' denominator it would have on the original and a true effect of zero. +#' effect is not carried: a real effect moves into the residual, as it does +#' when an owner permutes the treatment for a negative control, so the clone +#' stands in for those negatives. The residual is not the one a real fit has. +#' Linked `events` are not stratified by design under this option. #' A character vector of names gives the partition explicitly, from the #' retained design and treatment columns, and runs through the same ladder; #' treatment columns are still never dropped. The single word `"design"` @@ -159,7 +161,8 @@ #' grouping it was coarsened under, and [apply_recipe()] retargets to the real #' coordinates. Several published jitters of the same point average toward #' it: keep the seed, which the recipe records, and do not publish more than -#' one jittered release of the same sites. +#' one jittered release of the same sites. Use a large random seed: where a +#' clone shows site labels, a small one can be found by search. #' @param events Optional linked time-to-event outcomes. One spec per group of #' linked events, `list(time = c(...), status = c(...))`, with the time and #' status columns listed in the order the events occur; pass several specs @@ -583,6 +586,7 @@ mask <- function(df, roles = .strip_roles_provenance(as.data.frame(roles)), conditional = conditional, conditioning_used = conditioning_used, + conditioning_shifted = conditioning_shifted, column_name_map = NULL, level_maps = level_maps, storage_classes = list(), @@ -816,7 +820,8 @@ mask <- function(df, x_num = x_num ) shifts <- .design_shifts( - x_num, df, conditional_report$used, conditional_report$shifted + x_num, df, conditional_report$used, conditional_report$shifted, + treatment = roles$col[role == "treatment"] ) shift <- shifts$shift conditional_report$shifted <- shifts$shifted diff --git a/R/mask_set.R b/R/mask_set.R index 9b6b940..aad4ebb 100644 --- a/R/mask_set.R +++ b/R/mask_set.R @@ -109,6 +109,12 @@ mask_set <- function(input, .refuse_unused_dots("mask_set", ...) withr::local_preserve_seed() ladder <- match.arg(ladder) + if (is.list(roles) && any(vapply(roles, S7::S7_inherits, logical(1L), .plan_class))) { + .plan_refuse( + "{.fn mask_set} does not take a plan yet. Call {.code mask(df, plan)} on each table.", + "unsupported" + ) + } # Check the shape of `roles` before reading its mode, so a wrong argument # is reported as such. if (!is.null(roles)) .check_roles_shape(roles) diff --git a/R/plan.R b/R/plan.R index 6cdf03e..e92339d 100644 --- a/R/plan.R +++ b/R/plan.R @@ -85,6 +85,14 @@ masque_plan <- function( if (!is.logical(grain) || length(grain) != 1L || is.na(grain)) { .plan_refuse("`grain` must be a single `TRUE` or `FALSE`.", "grain") } + names_ok <- if (is.character(alias_names)) { + !anyNA(alias_names) + } else { + is.logical(alias_names) && length(alias_names) == 1L && !is.na(alias_names) + } + if (!names_ok) { + .plan_refuse("`alias_names` must be `TRUE`, `FALSE` or column names.", "alias_names") + } .plan_class( roles = roles, coords = .plan_wrap(coords, c("lat", "lon")), @@ -118,6 +126,13 @@ masque_plan <- function( if (is.list(x) && any(keys %in% names(x))) { x <- list(x) } + if (!is.list(x)) { + what <- keys[1L] + .plan_refuse( + "A spec needs named entries, as in {.code list({keys[1L]} = , {keys[2L]} = )}.", + if (identical(what, "lat")) "coords" else "events" + ) + } x } @@ -169,7 +184,7 @@ local(S7::method(print, .plan_class) <- function(x, ...) { cli::cli_bullets(c( " " = sprintf("on %s, unit %s", .spec_on_line(s), .spec_cols_line(s@unit, "row")), " " = sprintf("strata %s", .spec_cols_line(s@strata, "none")), - " " = sprintf("target %s; %s replicate(s)", .spec_target_line(s), s@n) + " " = sprintf("target %s; replicates: %s", .spec_target_line(s), s@n) )) } invisible(x) @@ -201,6 +216,18 @@ local(S7::method(print, .plan_class) <- function(x, ...) { names(df), roles$col[roles$role == "design" & roles$action != "drop"] ) on <- .resolve_control_on(spec@on, df, roles, spec@target) + .check_spec_retained(c(on, spec@target), roles) + .check_spec_on_labels(on, df, roles, rec) + if (!is.null(rec@seed) && rec@seed == spec@seed) { + .control_refuse( + c( + "The spec's seed equals the seed of {.fn mask}.", + x = "The analyst reads the spec's seed in the plan, and the mask seed reveals every displacement and map.", + i = "Use two different seeds." + ), + "seed" + ) + } unit <- if (length(spec@unit)) spec@unit else .default_control_unit(df, on, design) strata <- if (length(spec@strata)) { spec@strata @@ -220,15 +247,61 @@ local(S7::method(print, .plan_class) <- function(x, ...) { } } draft <- masque_control_spec( - on = on, target = spec@target, effect = spec@effect, strata = strata, - unit = unit, grain = grain, n = spec@n, seed = spec@seed + on = on, target = spec@target, + effect = .plan_draw_effect(spec, on, df, rec), + strata = strata, unit = unit, grain = grain, n = spec@n, seed = spec@seed ) .resolve_control_spec(draft, df, roles)$spec } +# A column gone from the clone cannot be named in a plan the analyst reads. +.check_spec_retained <- function(cols, roles) { + dropped <- intersect(cols, roles$col[roles$action == "drop"]) + if (length(dropped)) { + .control_refuse( + "A control spec names {.field {dropped}}, with action {.val drop}: it is not in the clone.", + "dropped" + ) + } + invisible(TRUE) +} + +# The levels of a numeric `on` are its real values, unless a map relabels them. +.check_spec_on_labels <- function(on, df, roles, rec) { + action <- roles$action[match(on, roles$col)] + if (is.numeric(df[[on]]) && !identical(action, "keep") && + !length(rec@level_maps[[on]])) { + .control_refuse( + c( + "`on` ({.field {on}}) is re-simulated numeric: its levels in a plan would be real values.", + i = "Give it the action {.val alias} or {.val keep}." + ), + "on" + ) + } + invisible(TRUE) +} + +# A drawn effect is drawn over the sorted clone labels. Drawn over source +# levels, the vector's order and the public seed would give away the map. +.plan_draw_effect <- function(spec, on, df, rec) { + effect <- spec@effect + if (!is.list(effect)) { + return(effect) + } + levels <- .control_levels(df[[on]]) + map <- rec@level_maps[[on]] + if (!length(map)) { + return(.resolve_control_effect(effect, levels, spec@seed)) + } + clone <- as.character(.apply_level_map_forward(levels, map, col = on)) + draw <- .resolve_control_effect(effect, sort(clone, method = "radix"), spec@seed) + stats::setNames(unname(draw[clone]), levels) +} + # Coarsest design column, then pair of columns, within which `on` is constant. -# A partition of mostly single rows says nothing: `on` is constant there by -# construction, so it must average two rows a unit. +# A row alone is constant by construction, so a quarter of the rows must share +# a unit for the constancy to mean anything. .default_control_unit <- function(df, on, design) { n <- nrow(df) sets <- c(as.list(design), if (length(design) >= 2L) { @@ -238,9 +311,10 @@ local(S7::method(print, .plan_class) <- function(x, ...) { pick <- Filter(function(s) length(s) == size, sets) n_units <- vapply(pick, function(s) { id <- as.integer(.unit_id(df, s)) - if (.constant_within(id, df[[on]])) max(id) else NA_integer_ + sizes <- tabulate(id) + shared <- sum(sizes[sizes >= 2L]) + if (shared >= n / 4 && .constant_within(id, df[[on]])) max(id) else NA_integer_ }, integer(1L)) - n_units[n_units > n / 2] <- NA_integer_ if (any(!is.na(n_units))) { return(pick[[which.min(n_units)]]) } @@ -307,6 +381,10 @@ local(S7::method(print, .plan_class) <- function(x, ...) { } names(effect) <- as.character(relabel(names(effect), map, col = on_source)) } + # Clone-label order on the way out, so the vector's order carries no map. + if (is.numeric(effect)) { + effect <- effect[order(names(effect), method = "radix")] + } grain <- lapply(spec@grain, rename) names(grain) <- rename(names(grain)) masque_control_spec( @@ -363,13 +441,14 @@ clone_plan <- function(m) { } grain <- source_plan@grain specs <- lapply(source_plan@controls, .plan_spec_translate, rec = rec, to = "clone") + .check_clone_vocabulary(specs, names(m@synthetic)) } .plan_class( roles = public, coords = list(), events = .card_event_specs(rec), conditional = conditional, - ladder = rec@ladder, + ladder = .prop_or(rec, "ladder", "levels"), grain = grain, alias_names = FALSE, controls = specs, @@ -378,6 +457,20 @@ clone_plan <- function(m) { ) } +# A spec bound for the analyst names clone columns only. +.check_clone_vocabulary <- function(specs, clone_cols) { + named <- unlist(lapply(specs, function(s) { + c(s@on, s@target, s@strata, s@unit, names(s@grain), unlist(s@grain)) + })) + if (!all(named %in% clone_cols)) { + .plan_refuse( + "A control spec names a column outside the clone, so no clone plan is written.", + "vocabulary" + ) + } + invisible(TRUE) +} + # controls() from a plan ------------------------------------------------------ # Offset that roots the owner's permutations away from the spec's seed. @@ -397,7 +490,7 @@ clone_plan <- function(m) { ) } return(list( - df = df, + df = .blank_labels_to_na(df), roles = plan@roles, spec = picked$spec, events = if (is.null(events) && length(plan@events)) plan@events else events, @@ -428,6 +521,19 @@ clone_plan <- function(m) { ) } +# A clone read back from CSV holds "" where it held NA. No label of a clone is +# an empty string, because the cleaning step reads one as missing. +.blank_labels_to_na <- function(df) { + .check_control_table(df) + df <- .as_owned_dt(df) + for (cl in names(df)) { + if (is.character(df[[cl]]) && any(df[[cl]] == "", na.rm = TRUE)) { + data.table::set(df, which(df[[cl]] == ""), cl, NA_character_) + } + } + df +} + .plan_pick_spec <- function(plan, spec) { specs <- plan@controls if (!length(specs)) { @@ -457,6 +563,12 @@ clone_plan <- function(m) { return(invisible(TRUE)) } on_file <- declared@controls[[name]] + if (!is.null(on_file) && is.numeric(on_file@effect)) { + on_file@effect <- on_file@effect[order(names(on_file@effect), method = "radix")] + } + if (is.numeric(spec@effect)) { + spec@effect <- spec@effect[order(names(spec@effect), method = "radix")] + } same <- !is.null(on_file) && all(vapply( c("on", "target", "effect", "strata", "unit", "grain", "n", "seed"), function(p) isTRUE(all.equal(S7::prop(spec, p), S7::prop(on_file, p))), @@ -479,7 +591,7 @@ clone_plan <- function(m) { .controls_replay_source <- function(df, recipe) { .check_control_table(df) df <- .apply_cleaning_forward(df, recipe@cleaning) - df <- .apply_conform_forward(df, .prop_or(recipe, "conform", NULL)) + df <- .as_owned_dt(.apply_conform_forward(df, .prop_or(recipe, "conform", NULL))) retained <- setdiff(recipe@roles$col, .recipe_dropped_cols(recipe)) gone <- setdiff(retained, names(df)) if (length(gone)) { @@ -518,7 +630,7 @@ clone_plan <- function(m) { #' Save and read a masking plan as YAML #' -#' A plan on disk is plain, reviewable and diffable. `save_plan()` needs the +#' A plan on disk can be read and compared as text. `save_plan()` needs the #' `yaml` package. A custom `on_land` function cannot be written, and is refused. #' A source-vocabulary plan names real columns and labels, so it stays with the #' owner. The analyst's file comes from [clone_plan()] or [handover()]. @@ -577,7 +689,7 @@ read_plan <- function(path) { .check_yaml_available <- function() { if (!requireNamespace("yaml", quietly = TRUE)) { - .plan_refuse("Writing and reading a plan needs the {.pkg yaml} package.", "yaml") + .plan_refuse("Writing and reading a plan needs {.pkg yaml}.", "yaml") } } @@ -688,7 +800,11 @@ read_plan <- function(path) { #' the plan in clone vocabulary (`plan.yaml`), the data card (`card.yaml`) and a #' plain note (`README.txt`). The recipe is never written. The release gate runs #' first, as for [write_set()]: an unresolved HIGH finding blocks the write -#' unless `allow_high = TRUE`. +#' unless `allow_high = TRUE`. A local-mode clone has not been audited and is +#' meant to stay with the owner, so it is refused unless `allow_local = TRUE`. +#' +#' `clone.csv` loses column classes. Read label columns as text where a label +#' looks like a number, or `controls()` will not find its levels. #' #' @param m A `masque` object from [mask()]. #' @param dir The folder to write. An existing folder that holds files is @@ -696,13 +812,14 @@ read_plan <- function(path) { #' @param name Dataset name for the data card; `NULL` uses `"clone"`. #' @param overwrite Replace the four files if they exist. #' @param allow_high Write despite HIGH findings, with a warning. +#' @param allow_local Write a local-mode clone, for a folder the owner keeps. #' #' @returns The paths written, invisibly. #' #' @examples #' if (requireNamespace("yaml", quietly = TRUE)) { -#' r <- propose_roles(iris, detect = FALSE) -#' m <- mask(iris, r, mode = "local", seed = 1L) +#' r <- propose_roles(iris, mode = "collaborate", detect = FALSE) +#' m <- mask(iris, r, mode = "collaborate", seed = 1L) #' d <- tempfile() #' handover(m, d) #' list.files(d) @@ -711,7 +828,8 @@ read_plan <- function(path) { #' @seealso [clone_plan()], [data_card()], [controls()]. #' #' @export -handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE) { +handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE, + allow_local = FALSE) { if (!S7::S7_inherits(m, masque_obj)) { cli::cli_abort( "`m` must be a {.cls masque} object; got {.cls {class(m)[1]}}.", @@ -725,6 +843,15 @@ handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE) ) } .check_yaml_available() + if (!identical(m@mode, "collaborate") && !isTRUE(allow_local)) { + cli::cli_abort( + c( + "This clone was masked in local mode: it has no audit and keeps real labels.", + i = "Mask with {.code mode = \"collaborate\"} to hand it over, or pass `allow_local = TRUE` for a folder you keep." + ), + class = c("masque_handover_local_refusal", "orchestra_refusal") + ) + } high <- .gate_release(m, allow_high) if (dir.exists(dir) && length(list.files(dir, all.files = TRUE, no.. = TRUE)) && !isTRUE(overwrite)) { @@ -748,6 +875,8 @@ handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE) .handover_note <- function(m, n_high) { plan <- clone_plan(m) + r <- plan@roles + kept <- r$col[r$role == "outcome" & r$action == "keep"] c( "Masked clone for pipeline development", "", @@ -758,7 +887,13 @@ handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE) "plan.yaml holds the column roles and any declared controls, in the clone's names.", "card.yaml describes the table for downstream tools.", "", - "The values are re-simulated. Do not read an effect or an estimate from this table.", + "Re-simulated columns hold no real values. Do not read an effect or an estimate from this table.", + if (length(kept)) { + sprintf("These outcome columns hold real values: %s.", paste(kept, collapse = ", ")) + }, + if (!identical(m@mode, "collaborate")) { + "This is a local-mode clone. It keeps real labels and stays with the data owner." + }, "Write and test your code here. The data owner runs the finished code on the real table.", if (length(plan@controls)) { paste0( @@ -768,7 +903,7 @@ handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE) }, if (n_high > 0L) { sprintf( - "The owner released this clone despite %d HIGH leakage finding(s), after review.", + "The owner released this clone after reviewing HIGH leakage findings: %d.", n_high ) } diff --git a/R/synthesise_numeric_conditional.R b/R/synthesise_numeric_conditional.R index 070375d..67697ba 100644 --- a/R/synthesise_numeric_conditional.R +++ b/R/synthesise_numeric_conditional.R @@ -218,13 +218,18 @@ synthesise_numeric_conditional <- function(x_num, groups, min_stratum = 5L) { }, numeric(1)) } -# A column can carry a shift when two or more levels hold two or more rows; -# single-row levels are pooled by `.collapse_singletons()`. -.shiftable <- function(v) { +# A column can carry a shift when two or more levels reach the floor; smaller +# levels are pooled by `.collapse_singletons()`. +.shiftable <- function(v, floor = 2L) { tab <- table(as.character(v)) - sum(tab >= 2L) >= 2L + sum(tab >= floor) >= 2L } +# Rows a level of a design term must hold to carry a shift. The shift of a level +# of two rows is close to those rows' own outcomes. A treatment keeps a floor of +# two: its means are what a conditional clone exists to carry. +.SHIFT_FLOOR_DESIGN <- 3L + # Pairs of dropped blocking and environment columns, as "a:b" (rep R1 in one # county is not R1 in another). Coordinates form no pairs. .nested_pairs <- function(cols) { @@ -245,12 +250,12 @@ synthesise_numeric_conditional <- function(x_num, groups, min_stratum = 5L) { interaction(lapply(.cols(df, parts), as.character), drop = TRUE, sep = ":") } -# Levels with a single row share one pooled level, so the shift a row -# receives is never that row's own outcome. -.collapse_singletons <- function(v) { +# Levels below the floor share one pooled level, so the shift a row receives is +# never that row's own outcome. +.collapse_singletons <- function(v, floor = 2L) { v <- as.character(v) tab <- table(v) - single <- names(tab)[tab < 2L] + single <- names(tab)[tab < floor] v[v %in% single] <- ".singleton" factor(v) } @@ -271,7 +276,7 @@ synthesise_numeric_conditional <- function(x_num, groups, min_stratum = 5L) { #' that restores the error variance the fit removed (`p` counts the stratum #' and each term's degrees of freedom, weighted by its shrinkage). #' @noRd -.design_shifts <- function(x_num, df, used, shifted) { +.design_shifts <- function(x_num, df, used, shifted, treatment = character()) { shift <- matrix( 0, nrow(x_num), ncol(x_num), dimnames = list(NULL, names(x_num)) ) @@ -286,7 +291,11 @@ synthesise_numeric_conditional <- function(x_num, groups, min_stratum = 5L) { with_stratum <- nlevels(key) >= 2L && !any(used %in% shifted) terms <- lapply(shifted, function(term) .shift_term(df, term)) names(terms) <- shifted - shifted <- shifted[vapply(terms, .shiftable, logical(1L))] + floors <- ifelse(shifted %in% treatment, 2L, .SHIFT_FLOOR_DESIGN) + names(floors) <- shifted + shifted <- shifted[vapply(shifted, function(term) { + .shiftable(terms[[term]], floors[[term]]) + }, logical(1L))] n_fit <- min(vapply(x_num, function(y) sum(!is.na(y)), integer(1L))) # Over the parameter budget, pairs go first (last pair first), then # columns in the ladder's drop order. @@ -294,7 +303,10 @@ synthesise_numeric_conditional <- function(x_num, groups, min_stratum = 5L) { if (!length(shifted)) { return(none) } - rhs <- lapply(terms[shifted], .collapse_singletons) + rhs <- lapply(shifted, function(term) { + .collapse_singletons(terms[[term]], floors[[term]]) + }) + names(rhs) <- shifted if (with_stratum) { rhs <- c(list(.stratum = key), rhs) } diff --git a/man/handover.Rd b/man/handover.Rd index 985839b..e5a9e50 100644 --- a/man/handover.Rd +++ b/man/handover.Rd @@ -4,7 +4,14 @@ \alias{handover} \title{Hand a clone to an analyst} \usage{ -handover(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE) +handover( + m, + dir, + name = NULL, + overwrite = FALSE, + allow_high = FALSE, + allow_local = FALSE +) } \arguments{ \item{m}{A \code{masque} object from \code{\link[=mask]{mask()}}.} @@ -17,6 +24,8 @@ refused unless \code{overwrite = TRUE}.} \item{overwrite}{Replace the four files if they exist.} \item{allow_high}{Write despite HIGH findings, with a warning.} + +\item{allow_local}{Write a local-mode clone, for a folder the owner keeps.} } \value{ The paths written, invisibly. @@ -26,12 +35,17 @@ Writes what the analyst works from into one folder: the clone (\code{clone.csv}) the plan in clone vocabulary (\code{plan.yaml}), the data card (\code{card.yaml}) and a plain note (\code{README.txt}). The recipe is never written. The release gate runs first, as for \code{\link[=write_set]{write_set()}}: an unresolved HIGH finding blocks the write -unless \code{allow_high = TRUE}. +unless \code{allow_high = TRUE}. A local-mode clone has not been audited and is +meant to stay with the owner, so it is refused unless \code{allow_local = TRUE}. +} +\details{ +\code{clone.csv} loses column classes. Read label columns as text where a label +looks like a number, or \code{controls()} will not find its levels. } \examples{ if (requireNamespace("yaml", quietly = TRUE)) { - r <- propose_roles(iris, detect = FALSE) - m <- mask(iris, r, mode = "local", seed = 1L) + r <- propose_roles(iris, mode = "collaborate", detect = FALSE) + m <- mask(iris, r, mode = "collaborate", seed = 1L) d <- tempfile() handover(m, d) list.files(d) diff --git a/man/jitter_coordinates.Rd b/man/jitter_coordinates.Rd index 8efe5bb..a5f04dd 100644 --- a/man/jitter_coordinates.Rd +++ b/man/jitter_coordinates.Rd @@ -92,8 +92,8 @@ therefore comes back with one masked coordinate per place, exactly as the source table has one true coordinate per place. \code{by} chooses how a site is identified: \itemize{ -\item \code{by = NULL} (default) treats rows that share an identical input coordinate -pair as one site. This needs no user action and cannot alter genuinely +\item \code{by = NULL} (default) treats rows that share an input coordinate pair, to +nine decimal places, as one site. This needs no user action and cannot alter genuinely point-level data, where every input pair is already distinct. \item \code{by = c("trial")} (one or more column names) treats rows sharing those values as one site, for the case where a site's recorded coordinates differ @@ -112,7 +112,7 @@ in the table play no part. Displacing each row of a repeated coordinate separately leaks the site. Donut displacement is isotropic, so the mean of many independent draws around one true site converges on that site: averaging the 360 rows of -a single trial recovers the true position to a median 0.58 km, against the +a single trial recovers the true position to a median 0.64 km, against the 5 km floor the donut was asked for. One draw per site removes the estimator and leaves the full displacement in place. } @@ -144,7 +144,9 @@ things start a new draw for a site: a different seed, a changed site value columns. A clone built before version 0.17.0 used a different draw, so a rebuild under 0.17.0 is a new release of those sites even with the same seed. Treat each of these as a fresh seed: retire the versions it replaces. The seed -reveals every displacement, so it stays as confidential as the recipe. +reveals every displacement, so it stays as confidential as the recipe. Use a +large random integer: where a clone shows site labels, or holds two pairs at +one place, a small seed can be found by search. Choose the unit carefully. It is the finest grouping that denotes \strong{one physical place at one time}. For a multi-year trial series that is location diff --git a/man/mask.Rd b/man/mask.Rd index 067f11e..c6db8a5 100644 --- a/man/mask.Rd +++ b/man/mask.Rd @@ -97,8 +97,10 @@ all, in which case the clone is the pooled copula. \code{"design"} conditions on the retained design columns alone. The clone keeps the design means, block and environment variance, while a treatment -effect is not carried, so a test of the treatment on the clone has the -denominator it would have on the original and a true effect of zero. +effect is not carried: a real effect moves into the residual, as it does +when an owner permutes the treatment for a negative control, so the clone +stands in for those negatives. The residual is not the one a real fit has. +Linked \code{events} are not stratified by design under this option. A character vector of names gives the partition explicitly, from the retained design and treatment columns, and runs through the same ladder; treatment columns are still never dropped. The single word \code{"design"} @@ -144,7 +146,8 @@ survives masking, coarsened; the recipe records the parameters and the site grouping it was coarsened under, and \code{\link[=apply_recipe]{apply_recipe()}} retargets to the real coordinates. Several published jitters of the same point average toward it: keep the seed, which the recipe records, and do not publish more than -one jittered release of the same sites.} +one jittered release of the same sites. Use a large random seed: where a +clone shows site labels, a small one can be found by search.} \item{events}{Optional linked time-to-event outcomes. One spec per group of linked events, \code{list(time = c(...), status = c(...))}, with the time and diff --git a/man/save_plan.Rd b/man/save_plan.Rd index 077e660..f30f392 100644 --- a/man/save_plan.Rd +++ b/man/save_plan.Rd @@ -18,7 +18,7 @@ read_plan(path) \code{save_plan()} returns \code{path} invisibly; \code{read_plan()} a \code{masque_plan}. } \description{ -A plan on disk is plain, reviewable and diffable. \code{save_plan()} needs the +A plan on disk can be read and compared as text. \code{save_plan()} needs the \code{yaml} package. A custom \code{on_land} function cannot be written, and is refused. A source-vocabulary plan names real columns and labels, so it stays with the owner. The analyst's file comes from \code{\link[=clone_plan]{clone_plan()}} or \code{\link[=handover]{handover()}}. diff --git a/tests/testthat/test-conditional-disclosure.R b/tests/testthat/test-conditional-disclosure.R new file mode 100644 index 0000000..888f7f1 --- /dev/null +++ b/tests/testthat/test-conditional-disclosure.R @@ -0,0 +1,114 @@ +# A conditional clone must not rebuild a row's value from a tiny design level, +# and the audit must see a clone that tracks the original row by row. + +.fx_pairs <- function(n = 400L, seed = 2L) { + withr::with_seed(seed, { + d <- data.frame( + gen = paste0("G", sample(20L, n, TRUE)), + rep = paste0("R", rep(1:4, length.out = n)), + pairid = paste0("P", rep(seq_len(n / 2L), each = 2L)) + ) + u <- rep(stats::rnorm(n / 2L, 0, 2), each = 2L) + d$y <- 10 + u + stats::rnorm(n, 0, 0.2) + d$z <- 3 + 0.5 * u + stats::rnorm(n, 0, 0.2) + d + }) +} + +.fx_pair_roles <- function(d) { + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, c("rep", "pairid"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + set_role(r, c("y", "z"), role = "outcome", action = "scramble") +} + +test_that("a two-row design level gives no shift that rebuilds the value", { + d <- .fx_pairs() + r <- .fx_pair_roles(d) + for (cnd in list(TRUE, "design")) { + m <- suppressWarnings(mask(d, r, mode = "collaborate", conditional = cnd, + seed = 1L, quiet = TRUE)) + expect_lt(abs(stats::cor(d$y, synthetic(m)$y)), 0.5) + } +}) + +test_that("a design level of ample rows still carries its shift", { + d <- .fx_pairs(400L) + withr::with_seed(5L, { + d$env <- rep(paste0("E", 1:8), each = 50L) + d$y <- rep(stats::rnorm(8L, 0, 3), each = 50L) + stats::rnorm(400L) + }) + d$pairid <- NULL + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, c("rep", "env"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, c("y", "z"), role = "outcome", action = "scramble") + m <- suppressWarnings(mask(d, r, mode = "collaborate", conditional = "design", + seed = 1L, quiet = TRUE)) + env_sd <- function(x) stats::sd(tapply(x$y, x$env, mean)) + expect_lt(abs(env_sd(synthetic(m)) / env_sd(d) - 1), 0.2) +}) + +test_that("the audit raises HIGH when a clone tracks the original row by row", { + withr::local_seed(4L) + n <- 300L + k <- 5L + d <- data.frame( + gen = paste0("G", sample(4L, n, TRUE)), + block = paste0("B", rep(seq_len(n / k), each = k)) + ) + d$y <- rep(stats::rnorm(n / k, 0, 5), each = k) + stats::rnorm(n, 0, 0.05) + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, "block", role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, "y", role = "outcome", action = "scramble") + m <- suppressWarnings(mask(d, r, mode = "collaborate", conditional = "design", + seed = 1L, quiet = TRUE)) + expect_gt(stats::cor(d$y, synthetic(m)$y), 0.95) + expect_identical(m@audit$leakage_class[m@audit$col == "y"], "high") + expect_error(handover(m, file.path(withr::local_tempdir(), "p")), + class = "masque_blocked_write") + # Falsifier: the marginal clone of the same table is not flagged. + g <- suppressWarnings(mask(d, r, mode = "collaborate", seed = 1L, quiet = TRUE)) + expect_false(identical(g@audit$leakage_class[g@audit$col == "y"], "high")) +}) + +test_that("the design clone's residual matches an owner's negative control", { + # A permutation moves a real treatment effect into the residual. The design + # clone does the same, so negatives on the two sides share a denominator. + withr::local_seed(8L) + d <- expand.grid(gen = paste0("G", 1:10), rep = paste0("R", 1:6), + stringsAsFactors = FALSE) + d$y <- as.numeric(factor(d$gen)) * 1.8 + as.numeric(factor(d$rep)) + + stats::rnorm(nrow(d)) + r <- propose_roles(d, mode = "local", detect = FALSE) + r <- set_role(r, "rep", role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + rms <- function(x) stats::anova(stats::lm(y ~ rep + gen, x))["Residuals", "Mean Sq"] + own <- controls(d, r, control_spec(on = "gen", strata = "rep", n = 40L, seed = 3L), + seed = 11L) + owner_rms <- vapply(1:40, function(i) rms(control_table(own, "negative", i)), numeric(1)) + clone_rms <- vapply(1:40, function(s) { + rms(synthetic(suppressWarnings(mask(d, r, seed = s, quiet = TRUE, conditional = "design")))) + }, numeric(1)) + expect_gt(rms(d) * 5, 1) + expect_gt(mean(owner_rms), 5 * rms(d)) + expect_lt(abs(mean(clone_rms) / mean(owner_rms) - 1), 0.25) +}) + +test_that("big environment means in large cells grade medium, not high", { + withr::local_seed(6L) + d <- expand.grid(env = paste0("E", 1:10), plot = 1:40, stringsAsFactors = FALSE) + d$gen <- paste0("G", sample(10L, nrow(d), TRUE)) + d$y <- rep(stats::rnorm(10L, 0, 30), times = 40L) + stats::rnorm(nrow(d)) + d$plot <- NULL + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, "env", role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, "y", role = "outcome", action = "scramble") + m <- suppressWarnings(mask(d, r, mode = "collaborate", conditional = "design", + seed = 1L, quiet = TRUE)) + expect_gt(stats::cor(d$y, synthetic(m)$y), 0.95) + expect_identical(m@audit$leakage_class[m@audit$col == "y"], "medium") +}) diff --git a/tests/testthat/test-coord-site-invariant.R b/tests/testthat/test-coord-site-invariant.R index ac2876c..5502890 100644 --- a/tests/testthat/test-coord-site-invariant.R +++ b/tests/testthat/test-coord-site-invariant.R @@ -48,7 +48,7 @@ test_that("the donut contract still holds, now per site", { test_that("a site's true position is not recoverable by averaging its rows", { # The privacy regression. Per-row jitter left the centroid of a 360-row site - # a median 0.58 km from the truth, against a 5 km floor. + # a median 0.64 km from the truth, against a 5 km floor. n <- 360L df <- data.frame(lat = rep(-34.9, n), lon = rep(138.6, n)) err <- vapply(1:10, function(s) { diff --git a/tests/testthat/test-coord-site-keyed.R b/tests/testthat/test-coord-site-keyed.R index 5030677..3ccf8b8 100644 --- a/tests/testthat/test-coord-site-keyed.R +++ b/tests/testthat/test-coord-site-keyed.R @@ -164,3 +164,16 @@ test_that("draws do not depend on the caller's RNG kind or state", { expect_identical(a, b) expect_identical(before, .Random.seed) }) + +test_that("a CSV round trip of computed coordinates keeps the draw", { + n <- 50L + set.seed(31L) + df <- data.frame(lat = stats::runif(n, -38, -33), lon = stats::runif(n, 139, 149)) + a <- jitter_coordinates(df, "lat", "lon", on_land = FALSE, seed = 4L) + f <- withr::local_tempfile(fileext = ".csv") + utils::write.csv(df, f, row.names = FALSE) + back <- utils::read.csv(f) + b <- jitter_coordinates(back, "lat", "lon", on_land = FALSE, seed = 4L) + expect_lt(max(abs((a$lat - df$lat) - (b$lat - back$lat))), 2e-5) + expect_lt(max(abs((a$lon - df$lon) - (b$lon - back$lon))), 2e-5) +}) diff --git a/tests/testthat/test-mask-conditional-ladder.R b/tests/testthat/test-mask-conditional-ladder.R index 938a22d..3aad604 100644 --- a/tests/testthat/test-mask-conditional-ladder.R +++ b/tests/testthat/test-mask-conditional-ladder.R @@ -293,7 +293,8 @@ test_that("mask(ladder = 'hierarchy') keeps the environment effect of a MET", { expect_identical(rec@ladder, "hierarchy") expect_identical(rec@conditioning_used, c("trt", "site")) expect_identical(rec@conditioning_dropped, c("row", "rep")) - expect_setequal(rec@conditioning_shifted, c("rep", "row")) + # `row` holds two rows a level here, below the floor for a design shift. + expect_identical(rec@conditioning_shifted, "rep") f_clone <- fit(as.data.frame(synthetic(m))) # Floor 0.4: the clone re-draws each cell's six-to-eight residuals, and # site:rep in the original is half replicate signal and half noise df. diff --git a/tests/testthat/test-plan.R b/tests/testthat/test-plan.R index 94a65ca..b646ded 100644 --- a/tests/testthat/test-plan.R +++ b/tests/testthat/test-plan.R @@ -72,7 +72,7 @@ test_that("mask() resolves unit, strata, effect on the source and writes back", expect_true(is.numeric(s@effect)) expect_setequal(names(s@effect), paste0("Variety", 1:10)) again <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), - seed = 99L, quiet = TRUE) + seed = 5L, quiet = TRUE) expect_identical(recipe(again)@plan@controls$default@effect, s@effect) }) @@ -343,7 +343,7 @@ test_that("handover writes four files and no source vocabulary", { )) dir <- withr::local_tempdir() out <- file.path(dir, "pack") - paths <- handover(m, out) + paths <- handover(m, out, allow_local = TRUE) expect_setequal(basename(paths), c("clone.csv", "plan.yaml", "card.yaml", "README.txt")) expect_setequal(list.files(out), basename(paths)) text <- unlist(lapply(paths, readLines)) @@ -357,8 +357,8 @@ test_that("handover writes four files and no source vocabulary", { for (w in c("Variety", "yield")) { expect_true(any(grepl(w, own_text, fixed = TRUE)), info = w) } - expect_error(handover(m, out), class = "masque_handover_exists_refusal") - handover(m, out, overwrite = TRUE) + expect_error(handover(m, out, allow_local = TRUE), class = "masque_handover_exists_refusal") + handover(m, out, overwrite = TRUE, allow_local = TRUE) # The file round trip the note describes. clone <- data.table::fread(file.path(out, "clone.csv")) @@ -383,3 +383,224 @@ test_that("handover runs the release gate", { expect_warning(handover(m, out, allow_high = TRUE), class = "masque_high_override") expect_true(any(grepl("HIGH", readLines(file.path(out, "README.txt"))))) }) + +# Faults found in review before the 0.17.0 tag -------------------------------- + +.fx_mapped <- function(action, seed = 12345L, spec_seed = 99L) { + d <- .fx_plan_table() + r <- set_role(.fx_plan_roles(d), "gen", role = "treatment", action = action) + spec <- control_spec(on = "gen", target = "yield", effect = list(sd = 0.3), + n = 3L, seed = spec_seed) + suppressWarnings(mask(d, masque_plan(r, controls = spec), seed = seed, quiet = TRUE)) +} + +test_that("the clone plan's effect does not reveal the level map", { + for (act in c("alias", "scramble")) { + leaks <- vapply(1:12, function(s) { + m <- .fx_mapped(act, seed = 1000L + s) + e <- clone_plan(m)@controls$default@effect + map <- recipe(m)@level_maps$gen + # Names in clone order; nothing tied to source order remains. + expect_identical(names(e), sort(names(e), method = "radix")) + # The attack: order of the vector read as the source-sorted map. + identical(unname(map[sort(names(map))]), names(e)) + }, logical(1L)) + expect_false(any(leaks), info = act) + + # The attack: recompute the draw from the public seed. It reaches clone + # labels only, and the same clone label gets the same draw in every build. + m <- .fx_mapped(act, seed = 7L) + e <- clone_plan(m)@controls$default@effect + root <- (99 + .CONTROL_SALT) %% .Machine$integer.max + draw <- withr::with_seed(root, stats::rnorm(length(e), 0, 0.3)) + expect_equal(unname(e), draw) + } +}) + +test_that("two maps give the same clone-side effect vector values by label", { + a <- clone_plan(.fx_mapped("alias", seed = 1L))@controls$default@effect + b <- clone_plan(.fx_mapped("alias", seed = 2L))@controls$default@effect + expect_identical(a, b) +}) + +test_that("the same variety still takes the same shift on both sides", { + m <- .fx_mapped("alias") + src <- recipe(m)@plan@controls$default@effect + cln <- clone_plan(m)@controls$default@effect + map <- recipe(m)@level_maps$gen + expect_equal(unname(cln[unname(map[names(src)])]), unname(src)) +}) + +test_that("a dropped on or target is refused and reaches no file", { + d <- .fx_plan_table() + d$protein <- withr::with_seed(2L, stats::rnorm(nrow(d))) + r <- .fx_plan_roles(d) + r <- set_role(r, "protein", role = "outcome", action = "scramble") + r1 <- set_role(r, "gen", role = "treatment", action = "drop") + expect_error( + mask(d, masque_plan(r1, controls = control_spec(on = "gen", n = 2L, seed = 1L)), + seed = 1L, quiet = TRUE), + class = "masque_control_dropped_refusal" + ) + r2 <- set_role(r, "protein", role = "outcome", action = "drop") + expect_error( + mask(d, masque_plan(r2, controls = control_spec( + on = "gen", target = "protein", effect = list(sd = 0.2), n = 2L, seed = 1L + )), seed = 1L, quiet = TRUE), + class = "masque_control_dropped_refusal" + ) +}) + +test_that("clone_plan fails closed on a name outside the clone", { + d <- .fx_plan_table() + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), seed = 5L, quiet = TRUE) + rec <- m@recipe + s <- rec@plan@controls$default + s@strata <- "a_real_name" + plan <- rec@plan + plan@controls$default <- s + rec@plan <- plan + m@recipe <- rec + expect_error(clone_plan(m), class = "masque_plan_vocabulary_refusal") +}) + +test_that("a spec seed equal to the mask seed is refused", { + d <- .fx_plan_table() + p <- masque_plan(.fx_plan_roles(d), controls = .fx_spec()) + expect_error(mask(d, p, seed = 77L, quiet = TRUE), class = "masque_control_seed_refusal") +}) + +test_that("mask_set() refuses a plan rather than ignoring it", { + d <- .fx_plan_table() + p <- masque_plan(.fx_plan_roles(d), alias_names = TRUE) + expect_error( + mask_set(list(a = d), roles = list(a = p), seed = 1L, quiet = TRUE), + class = "masque_plan_unsupported_refusal" + ) +}) + +test_that("handover refuses a local clone unless asked", { + skip_if_not_installed("yaml") + d <- .fx_plan_table() + m <- suppressWarnings(mask(d, .fx_plan_roles(d), seed = 1L, quiet = TRUE)) + out <- file.path(withr::local_tempdir(), "pack") + expect_error(handover(m, out), class = "masque_handover_local_refusal") + expect_false(dir.exists(out)) + handover(m, out, allow_local = TRUE) + expect_true(file.exists(file.path(out, "clone.csv"))) + expect_true(any(grepl("owner", readLines(file.path(out, "README.txt"))))) +}) + +test_that("a scrambled whole-valued on is refused: its levels are real values", { + d <- .fx_plan_table() + d$age <- rep(c(31L, 47L, 53L, 68L, 72L), length.out = nrow(d)) + r <- set_role(.fx_plan_roles(d), "gen", role = "treatment", action = "keep") + r <- set_role(r, "age", role = "covariate", action = "scramble") + expect_error( + mask(d, masque_plan(r, controls = control_spec(on = "age", n = 2L, seed = 1L)), + seed = 1L, quiet = TRUE), + class = "masque_control_on_refusal" + ) +}) + +test_that("a numeric or integer treatment with an alias runs on both sides", { + d <- .fx_plan_table() + for (vals in list(as.numeric(as.integer(factor(d$gen))) / 2, as.integer(factor(d$gen)))) { + d$dose <- vals + x <- d[, setdiff(names(d), "gen")] + r <- propose_roles(x, mode = "local", detect = FALSE) + r <- set_role(r, c("site", "rep", "block"), role = "design", action = "keep") + r <- set_role(r, "yield", role = "outcome", action = "scramble") + r <- set_role(r, "dose", role = "treatment", action = "alias") + spec <- control_spec(on = "dose", target = "yield", effect = list(sd = 0.3), + n = 2L, seed = 9L) + m <- suppressWarnings(mask(x, masque_plan(r, controls = spec), seed = 2L, quiet = TRUE)) + ana <- controls(synthetic(m), clone_plan(m)) + own <- controls(x, clone_plan(m), recipe = recipe(m)) + expect_true(S7::S7_inherits(ana, masque_controls)) + expect_true(S7::S7_inherits(own, masque_controls)) + map <- recipe(m)@level_maps$dose + expect_equal( + unname(ana@spec@effect[unname(map[names(own@spec@effect)])]), + unname(own@spec@effect) + ) + } +}) + +test_that("blank cells in a clone CSV stay missing under a clone plan", { + skip_if_not_installed("yaml") + d <- .fx_plan_table() + d$gen[c(2L, 50L, 90L)] <- NA + m <- suppressWarnings(mask(d, masque_plan(.fx_plan_roles(d), controls = control_spec( + on = "gen", target = "yield", effect = list(sd = 0.3), strata = "site", n = 3L, seed = 9L + )), seed = 2L, quiet = TRUE)) + out <- file.path(withr::local_tempdir(), "pack") + handover(m, out, allow_local = TRUE) + clone <- data.table::fread(file.path(out, "clone.csv")) + expect_identical(sum(clone$gen == "", na.rm = TRUE), 3L) + ctl <- suppressWarnings(controls(clone, read_plan(file.path(out, "plan.yaml")))) + neg <- control_table(ctl, "negative", 1L) + expect_identical(which(is.na(neg$gen)), c(2L, 50L, 90L)) + expect_false("" %in% ctl@truth$level) +}) + +test_that("the owner's manifest holds no private seed", { + d <- .fx_plan_table() + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), seed = 5L, quiet = TRUE) + own <- controls(d, clone_plan(m), recipe = recipe(m)) + expect_true(all(is.na(own@manifest$seed))) + ana <- controls(synthetic(m), clone_plan(m)) + expect_false(anyNA(ana@manifest$seed)) + r <- .fx_plan_roles(d) + priv <- controls(d, r, .fx_spec(), seed = 123L) + expect_true(all(is.na(priv@manifest$seed))) +}) + +test_that("an owner's table with row names from a filter is not refused", { + d <- .fx_plan_table() + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), seed = 5L, quiet = TRUE) + filt <- d + rownames(filt) <- paste0("row", seq_len(nrow(filt))) + expect_true(S7::S7_inherits( + controls(filt, clone_plan(m), recipe = recipe(m)), masque_controls + )) +}) + +test_that("a partly doubled whole plot still resolves as the unit", { + withr::local_seed(3L) + sizes <- rep(c(1L, 1L, 2L), length.out = 60L) + d <- data.frame(wp = rep(sprintf("P%02d", seq_along(sizes)), sizes)) + d$site <- rep(c("A", "B"), length.out = nrow(d)) + d$site <- rep(rep(c("A", "B"), length.out = 60L), sizes) + d$irr <- rep(sample(rep(c("wet", "dry"), 30L)), sizes) + d$y <- stats::rnorm(nrow(d)) + r <- propose_roles(d, mode = "local", detect = FALSE) + r <- set_role(r, c("wp", "site"), role = "design", action = "keep") + r <- set_role(r, "irr", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + m <- suppressWarnings(mask(d, masque_plan(r, controls = control_spec(on = "irr", n = 2L, seed = 1L)), + seed = 5L, quiet = TRUE)) + expect_identical(recipe(m)@plan@controls$default@unit, "wp") +}) + +test_that("a plan's bad coordinates or alias_names are refused at the plan", { + d <- .fx_plan_table() + r <- .fx_plan_roles(d) + expect_error(masque_plan(r, coords = c("a", "b")), class = "masque_plan_coords_refusal") + expect_error(masque_plan(r, alias_names = NA), class = "masque_plan_alias_names_refusal") +}) + +test_that("the note names a kept outcome instead of calling it re-simulated", { + skip_if_not_installed("yaml") + d <- .fx_plan_table() + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, c("site", "rep", "block"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, "yield", role = "outcome", action = "keep") + m <- suppressWarnings(mask(d, r, mode = "collaborate", seed = 1L, quiet = TRUE, + allow_kept_outcome = TRUE)) + out <- file.path(withr::local_tempdir(), "pack") + suppressWarnings(handover(m, out, allow_high = TRUE)) + note <- readLines(file.path(out, "README.txt")) + expect_true(any(grepl("real values", note))) +}) diff --git a/tests/testthat/test-synthesise_numeric_jitter.R b/tests/testthat/test-synthesise_numeric_jitter.R index cc7494e..d60f79b 100644 --- a/tests/testthat/test-synthesise_numeric_jitter.R +++ b/tests/testthat/test-synthesise_numeric_jitter.R @@ -75,8 +75,10 @@ test_that("Collaborate mode lowers numeric exact-match in mask()", { r <- propose_roles(df) r$role[r$col == "yield"] <- "outcome" - # Rep stays a kept design column (graded low), so no HIGH warning. - m <- mask(df, r, mode = "collaborate", seed = 1) + # Rep stays a kept design column (graded low), so no HIGH warning. The seed + # differs from the one that made the data: an equal seed draws the same + # normals, and the clone then tracks the table. + m <- mask(df, r, mode = "collaborate", seed = 2) audit <- audit_mask(m, print = FALSE) # Continuous outcome: exact_match_pct < 1 diff --git a/vignettes/confidentiality.Rmd b/vignettes/confidentiality.Rmd index 6410289..01fab03 100644 --- a/vignettes/confidentiality.Rmd +++ b/vignettes/confidentiality.Rmd @@ -753,7 +753,7 @@ cat("maps is not installed on this build, so the site-grouping demo is", Grouping is a confidentiality control, not only a fidelity one. Donut displacement is isotropic, so the mean of many independent draws around one true site converges on that site. Displacing a 360-row trial row by -row leaves its centroid a median 0.58 km from the truth, against the 5 km +row leaves its centroid a median 0.64 km from the truth, against the 5 km floor the donut was asked for. One draw per site removes that estimator and leaves the full displacement in place. The unit has to be chosen carefully: it is the finest grouping denoting one physical place at one diff --git a/vignettes/controls.Rmd b/vignettes/controls.Rmd index 1e011c2..9143a95 100644 --- a/vignettes/controls.Rmd +++ b/vignettes/controls.Rmd @@ -227,12 +227,14 @@ recipe(m_plan)@plan@controls$default@strata ``` `conditional = "design"` keeps the design means and block variance and carries -no genotype effect, so a test of genotype on this clone has the denominator it -would have on the original. +no genotype effect. A real effect moves into the residual, as it does when the +owner permutes genotype for a negative control, so the clone stands in for those +negatives. A fit on this clone does not show the residual of a fit on the +original. The analyst gets the plan in the clone's own names and labels. `handover()` -writes it to a folder with the clone, the data card and a note, and leaves the -recipe out. From there the analyst rebuilds the declared controls. +writes it to a folder with a collaborate-mode clone, the data card and a note, +and leaves the recipe out. From there the analyst rebuilds the declared controls. ```{r plan-analyst} analyst_plan <- clone_plan(m_plan) From 00c9b913d5dc81f9fc02c161fb224b7527275fe4 Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Fri, 6 Nov 2026 00:46:18 +1030 Subject: [PATCH 08/16] fix: second review round before the 0.17.0 tag The audit measures tracking on the rows of small levels, with the means of large terms taken out, so strong environments no longer block a clone and a tracking subset is seen. Owner control seeds derive from a random key on the recipe. Spec names holding hidden names are refused; a lone small level gets no shift; blank handling is limited to on; a numeric alias of any value works on both sides. --- NEWS.md | 17 +-- R/controls.R | 6 +- R/jitter_coordinates.R | 6 +- R/leakage.R | 115 +++++++++++++------ R/mask.R | 8 +- R/plan.R | 74 +++++++++--- R/recipe.R | 5 + R/recipe_print.R | 7 +- R/synthesise_numeric_conditional.R | 15 ++- man/handover.Rd | 3 +- man/jitter_coordinates.Rd | 6 +- man/mask.Rd | 7 +- man/masque_plan.Rd | 8 +- tests/testthat/test-conditional-disclosure.R | 93 +++++++++++++++ tests/testthat/test-plan.R | 79 ++++++++++++- 15 files changed, 361 insertions(+), 88 deletions(-) diff --git a/NEWS.md b/NEWS.md index 9865caf..37a60a6 100644 --- a/NEWS.md +++ b/NEWS.md @@ -12,14 +12,17 @@ and 3.1 km at 1, 2, 4, 8 and 16 kept rebuilds, and seven rebuilds, not eight, fall below the 5 km floor. * Rows are one site when their coordinates agree to nine decimal places, so a - table written to a file and read back keeps its draw. -* The audit grades a re-simulated number that tracks the original row by row - (r at or above 0.95): HIGH where its conditioning cell holds ten rows or fewer, - MEDIUM otherwise. A conditional clone of tiny strata can do this. -* Under the hierarchy ladder a design level must hold three rows to carry a shift. - A level of two rows gave a shift close to those rows' own values: a clone + table written to a file with 15 significant digits and read back keeps its draw + at nearly every site. +* The audit grades a re-simulated number that tracks the original row by row: + HIGH at r of 0.9 or more on the rows of small levels, with the means of large + terms taken out. MEDIUM at 0.7 or more there, or at 0.95 or more overall, which + big environment means alone can reach. +* Under the hierarchy ladder a design level must hold three rows to carry a + shift. A level of two rows gave a shift close to those rows' own values: a clone correlated 0.99 with the original at its rows, and the audit said low. Conditional - clones with such a column change. + clones with such a column change. Levels of three or four rows still carry a + shift, since real alpha blocks are that small, and the audit grades them. * A private `seed` in `controls()` no longer appears in the manifest. * The averaged-row figure in `?jitter_coordinates` is corrected to 0.64 km for 360 rows. diff --git a/R/controls.R b/R/controls.R index ccd8312..fe2c451 100644 --- a/R/controls.R +++ b/R/controls.R @@ -401,7 +401,7 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) } # An aliased numeric is a set of labels, on both sides. action <- roles$action[match(on, roles$col)] - categorical <- kind %in% .categorical_kinds() || action %in% c("alias", "permute") + categorical <- kind %in% .categorical_kinds() || identical(action, "alias") whole <- kind %in% .numeric_kinds() && .is_whole_valued(df[[on]]) if (!categorical && !whole) { .control_refuse( @@ -486,7 +486,7 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) # The levels of `on` as character, in sorted order. .control_levels <- function(x) { - as.character(sort(unique(x[!is.na(x)]))) + .level_key(sort(unique(x[!is.na(x)]))) } # Share of strata in which `on` holds at most one distinct value. @@ -650,7 +650,7 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) # The positive: each target's realised shift on the rows of each level of # the permuted `on`; nothing is clipped. .plant_effect <- function(out, ctl) { - on_val <- as.character(out[[ctl@spec@on]]) + on_val <- .level_key(out[[ctl@spec@on]]) crossed <- character() for (tg in unique(ctl@truth$target)) { tr <- ctl@truth[ctl@truth$target == tg, ] diff --git a/R/jitter_coordinates.R b/R/jitter_coordinates.R index bbcc009..9e993ef 100644 --- a/R/jitter_coordinates.R +++ b/R/jitter_coordinates.R @@ -76,9 +76,9 @@ #' columns. A clone built before version 0.17.0 used a different draw, so a #' rebuild under 0.17.0 is a new release of those sites even with the same seed. #' Treat each of these as a fresh seed: retire the versions it replaces. The seed -#' reveals every displacement, so it stays as confidential as the recipe. Use a -#' large random integer: where a clone shows site labels, or holds two pairs at -#' one place, a small seed can be found by search. +#' reveals every displacement, so it stays as confidential as the recipe. A seed +#' has 31 bits: where a clone shows site labels and holds two pairs at one place, +#' it can be found by search, so alias the label or declare one pair. #' #' Choose the unit carefully. It is the finest grouping that denotes **one #' physical place at one time**. For a multi-year trial series that is location diff --git a/R/leakage.R b/R/leakage.R index 1bf57a4..074708a 100644 --- a/R/leakage.R +++ b/R/leakage.R @@ -16,7 +16,7 @@ if (is.null(coarsened)) coarsened <- character() na_pattern_uniq <- .global_na_pattern_uniqueness(df) strata <- .audit_strata(df, rec) - row_cell <- .audit_row_cell(df, rec) + row_terms <- .audit_row_terms(df, rec) design_cols <- intersect( rec@roles$col[rec@roles$role == "design"], names(df) ) @@ -77,15 +77,11 @@ } # A re-simulated number that tracks the original at its own row. - row_cor <- NA_real_ + row_track <- list(all = NA_real_, within = NA_real_) if (in_synth && kind %in% c("numeric", "integer") && .audit_numeric_resimulated(action) && !is_grained && !two_valued && - !(col %in% linked) && comparable_n >= 10L) { - y <- synth[[col]] - both <- !is.na(x) & !is.na(y) - if (stats::sd(x[both]) > 0 && stats::sd(y[both]) > 0) { - row_cor <- stats::cor(x[both], y[both]) - } + !(col %in% linked) && comparable_n >= .MIN_STRATUM) { + row_track <- .row_tracking(x, synth[[col]], row_terms) } # Same-row match against a permutation baseline: categorical, date and @@ -136,7 +132,7 @@ by_excess = by_excess, grained = is_grained, allow_kept_outcome = allow_kept_outcome, value_set_pct = if (is.null(value_set)) NA_real_ else value_set$pct, - row_cor = row_cor, row_cell = row_cell + row_track = row_track ) notes <- .audit_notes( @@ -148,10 +144,12 @@ row_grain = Filter(function(g) col %in% g$cols, row_grain), value_set = value_set ) - if (isTRUE(abs(row_cor) >= .AUDIT_THRESHOLDS$row_cor)) { - notes <- paste0( - notes, if (nzchar(notes)) "; ", - sprintf("tracks the original row by row (r = %.2f)", row_cor) + tracked <- c(row_track$within, row_track$all) + if (isTRUE(abs(row_track$within) >= .AUDIT_THRESHOLDS$row_cor_within_medium) || + isTRUE(abs(row_track$all) >= .AUDIT_THRESHOLDS$row_cor)) { + notes <- sprintf( + "tracks the original row by row (r = %.2f)", + tracked[which(!is.na(tracked))[1L]] ) } @@ -186,8 +184,12 @@ same_row_medium = list(excess = 10, z = 4), same_row_high = list(excess = 50, z = 4), value_set_high = 50, + # Tracking of a number on its own row: overall, and inside small levels. A real + # alpha design conditioned on 24 genotypes of three rows reaches 0.81. row_cor = 0.95, - row_cor_cell = 10L, + row_cor_within_high = 0.9, + row_cor_within_medium = 0.7, + row_cell_small = 10L, joint_medium = list(excess = 5, z = 4), joint_high = list(excess = 20, z = 4), rare_medium = list(excess = 1, z = 3), @@ -205,18 +207,61 @@ excess >= rule$z * sd) } -# Median rows per cell of the columns a clone's numbers were conditioned or -# shifted on; every row when there are none. -.audit_row_cell <- function(df, rec) { - terms <- unlist(strsplit( - c(rec@conditioning_used, .prop_or(rec, "conditioning_shifted", character())), - ":", fixed = TRUE - )) - cols <- intersect(unique(terms), names(df)) - if (!length(cols)) { - return(nrow(df)) - } - stats::median(tabulate(.row_group_id(.cols(df, cols)))) +# The columns a clone's numbers were conditioned or shifted on, sorted into +# large terms, whose means a clone may keep, and the rows held by small levels, +# whose means sit close to those rows. A treatment's means are the point of a +# conditional clone, so its levels count as small only below the stratum floor. +.audit_row_terms <- function(df, rec) { + used <- intersect(.prop_or(rec, "conditioning_used", character()), names(df)) + shifted <- .prop_or(rec, "conditioning_shifted", character()) + cols <- intersect(unique(c(used, unlist(strsplit(shifted, ":", fixed = TRUE)))), names(df)) + treatment <- rec@roles$col[rec@roles$role == "treatment"] + small_rows <- rep(FALSE, nrow(df)) + large <- list() + # Rows per level, row by row, for a partition. + level_size <- function(partition) { + id <- .row_group_id(.cols(df, partition)) + tabulate(id)[id] + } + for (cl in cols) { + trt <- cl %in% treatment + size <- level_size(cl) + top <- if (trt) .MIN_STRATUM - 1L else .AUDIT_THRESHOLDS$row_cell_small + bottom <- if (trt) 2L else 3L + small_rows <- small_rows | (size >= bottom & size <= top) + if (stats::median(size) > top) { + large <- c(large, list(df[[cl]])) + } + } + if (length(used) >= 2L) { + size <- level_size(used) + small_rows <- small_rows | + (size >= .MIN_STRATUM & size <= .AUDIT_THRESHOLDS$row_cell_small) + } + list(large = large, small_rows = small_rows) +} + +# Correlation of a number with its original overall and, on the rows of small +# levels, after the means of the large terms are taken out. +.row_tracking <- function(x, y, terms) { + both <- !is.na(x) & !is.na(y) + r <- function(a, b) { + if (length(a) >= 2L && stats::sd(a) > 0 && stats::sd(b) > 0) stats::cor(a, b) else NA_real_ + } + a <- as.numeric(x[both]) + b <- as.numeric(y[both]) + overall <- r(a, b) + small <- terms$small_rows[both] + if (sum(small) < .MIN_STRATUM) { + return(list(all = overall, within = NA_real_)) + } + for (g in terms$large) { + g <- as.character(g[both]) + g[is.na(g)] <- ".na" + a <- a - stats::ave(a, g) + b <- b - stats::ave(b, g) + } + list(all = overall, within = r(a[small], b[small])) } # A scrambled numeric column is re-simulated through the copula, within the @@ -619,8 +664,8 @@ exact_match_pct, freq_min, n_unique_levels, n_rows, coarsened = FALSE, same_row_excess = NA_real_, same_row_sd = NA_real_, two_valued = FALSE, linked = FALSE, by_excess = FALSE, grained = FALSE, - allow_kept_outcome = FALSE, value_set_pct = NA_real_, row_cor = NA_real_, - row_cell = NA_integer_ + allow_kept_outcome = FALSE, value_set_pct = NA_real_, + row_track = list(all = NA_real_, within = NA_real_) ) { collab <- identical(mode, "collaborate") @@ -635,10 +680,9 @@ return(if (collab) "high" else "medium") } - # 0b. A re-simulated number that tracks the original row by row, in a small - # conditioning cell whose mean is close to the rows themselves: HIGH. - tracks <- collab && isTRUE(abs(row_cor) >= .AUDIT_THRESHOLDS$row_cor) - if (tracks && isTRUE(row_cell <= .AUDIT_THRESHOLDS$row_cor_cell)) { + # 0b. A re-simulated number that tracks the original inside small conditioning + # cells, whose means are close to the rows themselves: HIGH. + if (collab && isTRUE(abs(row_track$within) >= .AUDIT_THRESHOLDS$row_cor_within_high)) { return("high") } @@ -672,9 +716,10 @@ return("high") } - # 3c. The same in a large cell, where the clone may only keep big environment - # means: MEDIUM. - if (tracks) { + # 3c. Weaker tracking inside small levels, or overall only, where the clone may + # be keeping big environment means. + if (collab && (isTRUE(abs(row_track$within) >= .AUDIT_THRESHOLDS$row_cor_within_medium) || + isTRUE(abs(row_track$all) >= .AUDIT_THRESHOLDS$row_cor))) { return("medium") } diff --git a/R/mask.R b/R/mask.R index 008953b..0cc3bf5 100644 --- a/R/mask.R +++ b/R/mask.R @@ -116,7 +116,8 @@ #' effect is not carried: a real effect moves into the residual, as it does #' when an owner permutes the treatment for a negative control, so the clone #' stands in for those negatives. The residual is not the one a real fit has. -#' Linked `events` are not stratified by design under this option. +#' Linked `events` are stratified only by treatment columns of the partition, +#' so `"design"` and a design-only partition do not keep their design means. #' A character vector of names gives the partition explicitly, from the #' retained design and treatment columns, and runs through the same ladder; #' treatment columns are still never dropped. The single word `"design"` @@ -161,8 +162,8 @@ #' grouping it was coarsened under, and [apply_recipe()] retargets to the real #' coordinates. Several published jitters of the same point average toward #' it: keep the seed, which the recipe records, and do not publish more than -#' one jittered release of the same sites. Use a large random seed: where a -#' clone shows site labels, a small one can be found by search. +#' one jittered release of the same sites. A seed has 31 bits: where a clone +#' shows site labels and holds two pairs at one place, it can be found by search. #' @param events Optional linked time-to-event outcomes. One spec per group of #' linked events, `list(time = c(...), status = c(...))`, with the time and #' status columns listed in the order the events occur; pass several specs @@ -677,6 +678,7 @@ mask <- function(df, integrity_fp = digest::digest(is.na(df), algo = "sha256") ) if (!is.null(plan)) { + rec@owner_key <- .new_owner_key() rec@plan <- .plan_resolve(plan, df, roles, rec, list( coords = coords, events = events, conditional = conditional_declared, ladder = ladder, grain = grain, alias_names = alias_names diff --git a/R/plan.R b/R/plan.R index e92339d..d181294 100644 --- a/R/plan.R +++ b/R/plan.R @@ -33,7 +33,13 @@ #' not declared, resolve to the finest design partition above the unit in which #' every stratum holds every level of `on`, or to none. A stratum that misses #' levels of `on` would keep part of the real effect in a negative control, so -#' incomplete blocks are not chosen by default. +#' incomplete blocks are not chosen by default. Declare a whole-plot unit where +#' few plots hold more than one row: it is taken only when a quarter of the rows +#' share a unit. +#' +#' A drawn effect, `list(sd = )`, is drawn over the clone's labels and shows +#' nothing of the source. A declared effect is written to the clone plan as +#' given, so values that rise with a level reveal that level's rank. #' #' @param roles A roles table from [propose_roles()]. #' @param coords,events,conditional,ladder,grain,alias_names As in [mask()]. @@ -194,6 +200,7 @@ local(S7::method(print, .plan_class) <- function(x, ...) { # The plan as it ran, with every spec resolved on the source table. .plan_resolve <- function(plan, df, roles, rec, args) { + .check_spec_names(names(plan@controls), rec) specs <- lapply(plan@controls, .plan_resolve_spec, df = df, roles = roles, rec = rec) resolved_roles <- as.data.frame(rec@roles) attr(resolved_roles, "mode") <- rec@mode @@ -254,6 +261,23 @@ local(S7::method(print, .plan_class) <- function(x, ...) { .resolve_control_spec(draft, df, roles)$spec } +# A spec's name reaches the analyst as written, so it holds no hidden name. +.check_spec_names <- function(spec_names, rec) { + hidden <- c( + names(rec@column_name_map), + unlist(lapply(rec@level_maps, names), use.names = FALSE) + ) + tokens <- unlist(strsplit(as.character(spec_names), "[^[:alnum:]]+"), use.names = FALSE) + bad <- intersect(tokens, hidden) + if (length(bad)) { + .plan_refuse( + "A control spec's name holds {.val {bad}}, which the clone hides. Rename the spec.", + "controls" + ) + } + invisible(TRUE) +} + # A column gone from the clone cannot be named in a plan the analyst reads. .check_spec_retained <- function(cols, roles) { dropped <- intersect(cols, roles$col[roles$action == "drop"]) @@ -473,9 +497,17 @@ clone_plan <- function(m) { # controls() from a plan ------------------------------------------------------ -# Offset that roots the owner's permutations away from the spec's seed. +# Offset that roots a drawn effect away from the spec's seed. .CONTROL_SALT <- 15485863L +# Not from the R stream: a seeded mask() must not determine it. +.new_owner_key <- function() { + digest::digest( + list(Sys.time(), Sys.getpid(), tempfile()), + algo = "sha256" + ) +} + # Arguments of controls() from a plan, on either side of the boundary. .controls_from_plan <- function(df, plan, spec, events, seed, recipe) { picked <- .plan_pick_spec(plan, spec) @@ -490,7 +522,7 @@ clone_plan <- function(m) { ) } return(list( - df = .blank_labels_to_na(df), + df = .blank_on_to_na(df, picked$spec), roles = plan@roles, spec = picked$spec, events = if (is.null(events) && length(plan@events)) plan@events else events, @@ -516,20 +548,26 @@ clone_plan <- function(m) { } else { events }, - seed = .control_source_seed(recipe@seed, source_spec@seed, seed), + seed = .control_source_seed(.prop_or(recipe, "owner_key", character()), source_spec@seed, seed), notes = "Source side: the seed is private. Release aggregates only." ) } -# A clone read back from CSV holds "" where it held NA. No label of a clone is -# an empty string, because the cleaning step reads one as missing. -.blank_labels_to_na <- function(df) { +# A clone read back from CSV holds "" where `on` held NA. A plan whose effect +# names "" as a level has a real blank label, and it stays. +.blank_on_to_na <- function(df, spec) { .check_control_table(df) df <- .as_owned_dt(df) - for (cl in names(df)) { - if (is.character(df[[cl]]) && any(df[[cl]] == "", na.rm = TRUE)) { - data.table::set(df, which(df[[cl]] == ""), cl, NA_character_) - } + on <- spec@on + if (is.null(on) || !on %in% names(df) || "" %in% names(spec@effect)) { + return(df) + } + x <- df[[on]] + if (is.factor(x)) { + levels(x)[levels(x) == ""] <- NA + data.table::set(df, j = on, value = x) + } else if (is.character(x)) { + data.table::set(df, which(x == ""), on, NA_character_) } df } @@ -604,7 +642,7 @@ clone_plan <- function(m) { df } -.control_source_seed <- function(recipe_seed, spec_seed, seed) { +.control_source_seed <- function(owner_key, spec_seed, seed) { if (!is.null(seed)) { .check_seed(seed) if (as.integer(seed) == spec_seed) { @@ -615,15 +653,14 @@ clone_plan <- function(m) { } return(as.integer(seed)) } - if (is.null(recipe_seed)) { + if (!length(owner_key)) { .control_refuse( - "The recipe holds no seed to derive a private one from. Pass a private `seed`.", + "This recipe was masked without a plan, so it holds no key to derive a private seed from. Pass a private `seed`.", "seed" ) } - root <- (as.numeric(recipe_seed) + as.numeric(spec_seed) + .CONTROL_SALT) %% - .Machine$integer.max - .sub_seeds(root, 1L) + h <- digest::digest(paste(owner_key, spec_seed), algo = "sha256", serialize = FALSE) + strtoi(substr(h, 1L, 7L), 16L) } # YAML ------------------------------------------------------------------------ @@ -809,7 +846,8 @@ read_plan <- function(path) { #' @param m A `masque` object from [mask()]. #' @param dir The folder to write. An existing folder that holds files is #' refused unless `overwrite = TRUE`. -#' @param name Dataset name for the data card; `NULL` uses `"clone"`. +#' @param name Dataset name for the data card; `NULL` uses `"clone"`. It is +#' written as given, so it should say nothing the clone hides. #' @param overwrite Replace the four files if they exist. #' @param allow_high Write despite HIGH findings, with a warning. #' @param allow_local Write a local-mode clone, for a folder the owner keeps. diff --git a/R/recipe.R b/R/recipe.R index 122d2e8..b6c0a70 100644 --- a/R/recipe.R +++ b/R/recipe.R @@ -88,6 +88,11 @@ masque_recipe <- S7::new_class( allow_kept_outcome = S7::new_property( class = S7::class_logical, default = FALSE ), + # Fresh entropy behind the owner's control seeds, which no seed can guess. + # Empty without a plan. Absent before 0.17.0. + owner_key = S7::new_property( + class = S7::class_character, default = character() + ), # The plan as it ran, specs resolved on the source; NULL without a plan. # Absent before 0.17.0. plan = S7::new_property( diff --git a/R/recipe_print.R b/R/recipe_print.R index 7875b4b..19afc10 100644 --- a/R/recipe_print.R +++ b/R/recipe_print.R @@ -367,7 +367,12 @@ reveal_maps <- function(rec, table = NULL) { frac <- NA_real_ } if (!length(used)) { - return("conditional requested (no conditioning column; pooled fallback)") + shifted <- .prop_or(x, "conditioning_shifted", character()) + return(if (length(shifted)) { + sprintf("conditional: pooled copula with additive shifts on %s", paste(shifted, collapse = ", ")) + } else { + "conditional requested (no conditioning column; pooled fallback)" + }) } r <- x@roles has_treatment <- any(requested %in% r$col[r$role == "treatment"]) diff --git a/R/synthesise_numeric_conditional.R b/R/synthesise_numeric_conditional.R index 67697ba..af95c00 100644 --- a/R/synthesise_numeric_conditional.R +++ b/R/synthesise_numeric_conditional.R @@ -225,10 +225,6 @@ synthesise_numeric_conditional <- function(x_num, groups, min_stratum = 5L) { sum(tab >= floor) >= 2L } -# Rows a level of a design term must hold to carry a shift. The shift of a level -# of two rows is close to those rows' own outcomes. A treatment keeps a floor of -# two: its means are what a conditional clone exists to carry. -.SHIFT_FLOOR_DESIGN <- 3L # Pairs of dropped blocking and environment columns, as "a:b" (rep R1 in one # county is not R1 in another). Coordinates form no pairs. @@ -251,11 +247,15 @@ synthesise_numeric_conditional <- function(x_num, groups, min_stratum = 5L) { } # Levels below the floor share one pooled level, so the shift a row receives is -# never that row's own outcome. +# never that row's own outcome. A pool still below the floor gets no shift. .collapse_singletons <- function(v, floor = 2L) { v <- as.character(v) tab <- table(v) single <- names(tab)[tab < floor] + if (sum(tab[single]) < floor) { + v[v %in% single] <- NA_character_ + return(factor(v)) + } v[v %in% single] <- ".singleton" factor(v) } @@ -291,7 +291,10 @@ synthesise_numeric_conditional <- function(x_num, groups, min_stratum = 5L) { with_stratum <- nlevels(key) >= 2L && !any(used %in% shifted) terms <- lapply(shifted, function(term) .shift_term(df, term)) names(terms) <- shifted - floors <- ifelse(shifted %in% treatment, 2L, .SHIFT_FLOOR_DESIGN) + # A design level of two rows gives a shift close to its rows' own values. A + # level of three or four can too; the audit grades that, since real alpha blocks + # are this small. A treatment keeps a floor of two: its means are the point. + floors <- ifelse(shifted %in% treatment, 2L, 3L) names(floors) <- shifted shifted <- shifted[vapply(shifted, function(term) { .shiftable(terms[[term]], floors[[term]]) diff --git a/man/handover.Rd b/man/handover.Rd index e5a9e50..acef7e0 100644 --- a/man/handover.Rd +++ b/man/handover.Rd @@ -19,7 +19,8 @@ handover( \item{dir}{The folder to write. An existing folder that holds files is refused unless \code{overwrite = TRUE}.} -\item{name}{Dataset name for the data card; \code{NULL} uses \code{"clone"}.} +\item{name}{Dataset name for the data card; \code{NULL} uses \code{"clone"}. It is +written as given, so it should say nothing the clone hides.} \item{overwrite}{Replace the four files if they exist.} diff --git a/man/jitter_coordinates.Rd b/man/jitter_coordinates.Rd index a5f04dd..b8c3143 100644 --- a/man/jitter_coordinates.Rd +++ b/man/jitter_coordinates.Rd @@ -144,9 +144,9 @@ things start a new draw for a site: a different seed, a changed site value columns. A clone built before version 0.17.0 used a different draw, so a rebuild under 0.17.0 is a new release of those sites even with the same seed. Treat each of these as a fresh seed: retire the versions it replaces. The seed -reveals every displacement, so it stays as confidential as the recipe. Use a -large random integer: where a clone shows site labels, or holds two pairs at -one place, a small seed can be found by search. +reveals every displacement, so it stays as confidential as the recipe. A seed +has 31 bits: where a clone shows site labels and holds two pairs at one place, +it can be found by search, so alias the label or declare one pair. Choose the unit carefully. It is the finest grouping that denotes \strong{one physical place at one time}. For a multi-year trial series that is location diff --git a/man/mask.Rd b/man/mask.Rd index c6db8a5..62bc512 100644 --- a/man/mask.Rd +++ b/man/mask.Rd @@ -100,7 +100,8 @@ keeps the design means, block and environment variance, while a treatment effect is not carried: a real effect moves into the residual, as it does when an owner permutes the treatment for a negative control, so the clone stands in for those negatives. The residual is not the one a real fit has. -Linked \code{events} are not stratified by design under this option. +Linked \code{events} are stratified only by treatment columns of the partition, +so \code{"design"} and a design-only partition do not keep their design means. A character vector of names gives the partition explicitly, from the retained design and treatment columns, and runs through the same ladder; treatment columns are still never dropped. The single word \code{"design"} @@ -146,8 +147,8 @@ survives masking, coarsened; the recipe records the parameters and the site grouping it was coarsened under, and \code{\link[=apply_recipe]{apply_recipe()}} retargets to the real coordinates. Several published jitters of the same point average toward it: keep the seed, which the recipe records, and do not publish more than -one jittered release of the same sites. Use a large random seed: where a -clone shows site labels, a small one can be found by search.} +one jittered release of the same sites. A seed has 31 bits: where a clone +shows site labels and holds two pairs at one place, it can be found by search.} \item{events}{Optional linked time-to-event outcomes. One spec per group of linked events, \code{list(time = c(...), status = c(...))}, with the time and diff --git a/man/masque_plan.Rd b/man/masque_plan.Rd index 92265b9..736096a 100644 --- a/man/masque_plan.Rd +++ b/man/masque_plan.Rd @@ -43,7 +43,13 @@ partition within which \code{on} is constant, or to the row. Its \code{strata}, not declared, resolve to the finest design partition above the unit in which every stratum holds every level of \code{on}, or to none. A stratum that misses levels of \code{on} would keep part of the real effect in a negative control, so -incomplete blocks are not chosen by default. +incomplete blocks are not chosen by default. Declare a whole-plot unit where +few plots hold more than one row: it is taken only when a quarter of the rows +share a unit. + +A drawn effect, \code{list(sd = )}, is drawn over the clone's labels and shows +nothing of the source. A declared effect is written to the clone plan as +given, so values that rise with a level reveal that level's rank. } \examples{ trial <- read.csv(system.file("extdata", "john_alpha.csv", package = "masque")) diff --git a/tests/testthat/test-conditional-disclosure.R b/tests/testthat/test-conditional-disclosure.R index 888f7f1..3986555 100644 --- a/tests/testthat/test-conditional-disclosure.R +++ b/tests/testthat/test-conditional-disclosure.R @@ -112,3 +112,96 @@ test_that("big environment means in large cells grade medium, not high", { expect_gt(stats::cor(d$y, synthetic(m)$y), 0.95) expect_identical(m@audit$leakage_class[m@audit$col == "y"], "medium") }) + +test_that("three- and four-row levels that track their rows are HIGH and blocked", { + for (k in 3:4) { + n <- 240L + withr::with_seed(k, { + d <- data.frame( + gen = paste0("G", sample(10L, n, TRUE)), + lv = paste0("L", rep(seq_len(n / k), each = k)) + ) + d$y <- rep(stats::rnorm(n / k, 0, 3), each = k) + stats::rnorm(n, 0, 0.2) + }) + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, "lv", role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, "y", role = "outcome", action = "scramble") + m <- suppressWarnings(mask(d, r, mode = "collaborate", conditional = "design", + seed = 1L, quiet = TRUE)) + expect_gt(abs(stats::cor(d$y, synthetic(m)$y)), 0.9) + expect_identical(m@audit$leakage_class[m@audit$col == "y"], "high") + expect_error( + handover(m, file.path(withr::local_tempdir(), "p")), + class = "masque_blocked_write" + ) + } +}) + +test_that("a tracking subset of small levels is seen among many single rows", { + withr::local_seed(14L) + sizes <- c(rep(3L, 30L), rep(1L, 310L)) + d <- data.frame(lv = rep(sprintf("L%03d", seq_along(sizes)), sizes)) + d$gen <- paste0("G", sample(10L, nrow(d), TRUE)) + d$y <- rep(stats::rnorm(length(sizes), 0, 3), sizes) + stats::rnorm(nrow(d), 0, 0.2) + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, "lv", role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, "y", role = "outcome", action = "scramble") + m <- suppressWarnings(mask(d, r, mode = "collaborate", conditional = "design", + seed = 1L, quiet = TRUE)) + overall <- stats::cor(d$y, synthetic(m)$y) + expect_lt(abs(overall), 0.7) + # The overall figure alone reads low; the rows of the small levels do not. + expect_identical(m@audit$leakage_class[m@audit$col == "y"], "medium") +}) + +test_that("a lone small level is not shifted to its own value", { + withr::local_seed(12L) + n <- 300L + d <- data.frame( + gen = paste0("G", sample(5L, n + 1L, TRUE)), + lv = c(paste0("L", rep(seq_len(n / 10L), each = 10L)), "rare") + ) + d$y <- rep(stats::rnorm(n / 10L + 1L, 10, 1), c(rep(10L, n / 10L), 1L)) + + stats::rnorm(n + 1L, 0, 0.2) + d$y[n + 1L] <- 24 + d$env <- rep(c("a", "b", "c"), length.out = n + 1L) + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, c("lv", "env"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, "y", role = "outcome", action = "scramble") + rare <- vapply(1:12, function(s) { + synthetic(suppressWarnings(mask(d, r, mode = "collaborate", conditional = "design", + seed = s, quiet = TRUE)))$y[n + 1L] + }, numeric(1)) + expect_lt(abs(mean(rare) - 10), 6) +}) + +test_that("strong sites do not block a design clone; within-cell tracking does", { + sim <- function(site_sd, within_share) { + withr::with_seed(21L, { + d <- expand.grid(site = paste0("S", 1:6), rep = paste0("R", 1:3), block = paste0("B", 1:5), + plot = 1:6, stringsAsFactors = FALSE) + d$block <- paste(d$site, d$rep, d$block, sep = "_") + d$rep <- paste(d$site, d$rep, sep = "_") + d$plot <- NULL + d$gen <- paste0("G", sample(30L, nrow(d), TRUE)) + s <- stats::setNames(stats::rnorm(6L, 0, site_sd), paste0("S", 1:6)) + b <- stats::setNames(stats::rnorm(90L, 0, within_share), unique(d$block)) + d$y <- s[d$site] + b[d$block] + stats::rnorm(nrow(d), 0, 1) + d + }) + } + run <- function(d) { + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, c("site", "rep", "block"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "alias") + r <- set_role(r, "y", role = "outcome", action = "scramble") + m <- suppressWarnings(mask(d, r, mode = "collaborate", conditional = "design", + seed = 1L, quiet = TRUE)) + m@audit$leakage_class[m@audit$col == "y"] + } + expect_false(identical(run(sim(25, 0.3)), "high")) + expect_identical(run(sim(1, 15)), "high") +}) diff --git a/tests/testthat/test-plan.R b/tests/testthat/test-plan.R index b646ded..242d93d 100644 --- a/tests/testthat/test-plan.R +++ b/tests/testthat/test-plan.R @@ -212,13 +212,14 @@ test_that("the source side refuses what it should", { controls(d, .fx_plan_roles(d), .fx_spec(), recipe = rec), class = "masque_control_recipe_refusal" ) - # No recipe seed and no private seed. - m0 <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), quiet = TRUE) + # A plan-free recipe has no key: the owner must bring a private seed. + m0 <- suppressWarnings(mask(d, .fx_plan_roles(d), seed = 1L, quiet = TRUE)) + plan <- recipe(m)@plan expect_error( - controls(d, clone_plan(m0), recipe = recipe(m0)), + controls(d, plan, recipe = recipe(m0)), class = "masque_control_seed_refusal" ) - own <- controls(d, clone_plan(m0), recipe = recipe(m0), seed = 123L) + own <- controls(d, plan, recipe = recipe(m0), seed = 123L) expect_true(S7::S7_inherits(own, masque_controls)) }) @@ -604,3 +605,73 @@ test_that("the note names a kept outcome instead of calling it re-simulated", { note <- readLines(file.path(out, "README.txt")) expect_true(any(grepl("real values", note))) }) + +test_that("no value the analyst holds rebuilds the owner's seeds", { + d <- .fx_plan_table() + for (s in c(0L, 1L, 42L)) { + m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), + seed = s, quiet = TRUE) + own <- controls(d, clone_plan(m), recipe = recipe(m)) + guess <- (as.numeric(s) + 77 + .CONTROL_SALT) %% .Machine$integer.max + expect_false(identical(own@seeds, as.integer(.sub_seeds(.sub_seeds(guess, 1L), 5L)))) + expect_false(any(own@seeds %in% as.integer(.sub_seeds(77L, 5L)))) + } + # One seed, two recipes: the owner's key differs, the clone does not. + a <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), seed = 1L, quiet = TRUE) + b <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), seed = 1L, quiet = TRUE) + expect_identical(synthetic(a), synthetic(b)) + expect_false(identical(recipe(a)@owner_key, recipe(b)@owner_key)) + # The same recipe regenerates the owner's controls. + o1 <- controls(d, clone_plan(a), recipe = recipe(a)) + o2 <- controls(d, clone_plan(a), recipe = recipe(a)) + expect_identical(o1@seeds, o2@seeds) + # The key shows in no display route. + key <- recipe(a)@owner_key + shown <- c(utils::capture.output(print(recipe(a))), utils::capture.output(utils::str(recipe(a)))) + expect_false(any(grepl(key, shown, fixed = TRUE))) +}) + +test_that("a spec name holding a hidden name or label is refused", { + d <- .fx_plan_table() + r <- .fx_plan_roles(d) + bad <- list(Variety1_vs_yield = .fx_spec()) + expect_error( + mask(d, masque_plan(r, controls = bad, alias_names = TRUE), seed = 1L, quiet = TRUE), + class = "masque_plan_controls_refusal" + ) + ok <- list(main_test = .fx_spec()) + m <- mask(d, masque_plan(r, controls = ok, alias_names = TRUE), seed = 1L, quiet = TRUE) + expect_named(clone_plan(m)@controls, "main_test") +}) + +test_that("a non-terminating numeric alias works on both sides", { + d <- .fx_plan_table() + d$dose <- rep(c(1 / 3, 2 / 3, 1), length.out = nrow(d)) + x <- d[, setdiff(names(d), "gen")] + r <- propose_roles(x, mode = "local", detect = FALSE) + r <- set_role(r, c("site", "rep", "block"), role = "design", action = "keep") + r <- set_role(r, "yield", role = "outcome", action = "scramble") + r <- set_role(r, "dose", role = "treatment", action = "alias") + spec <- control_spec(on = "dose", target = "yield", effect = list(sd = 0.5), + n = 2L, seed = 9L) + m <- suppressWarnings(mask(x, masque_plan(r, controls = spec), seed = 2L, quiet = TRUE)) + own <- controls(x, clone_plan(m), recipe = recipe(m)) + neg <- control_table(own, "negative", 1L) + pos <- control_table(own, "positive", 1L) + d_y <- pos$yield - neg$yield + shift <- own@truth$shift_realised[match(.level_key(pos$dose), own@truth$level)] + expect_equal(d_y[!is.na(d_y)], shift[!is.na(d_y)]) + expect_true(all(abs(shift) > 0)) +}) + +test_that("a blank on means missing unless the plan names a blank level", { + d <- .fx_plan_table() + d$gen[3L] <- "" + r <- .fx_plan_roles(d) + r <- set_role(r, "gen", role = "treatment", action = "keep") + spec <- control_spec(on = "gen", target = "yield", effect = list(sd = 0.2), strata = "site", n = 2L, seed = 9L) + m <- suppressWarnings(mask(d, masque_plan(r, controls = spec), seed = 2L, quiet = TRUE)) + ctl <- suppressWarnings(controls(synthetic(m), clone_plan(m))) + expect_true("" %in% ctl@truth$level) + expect_false(anyNA(control_table(ctl, "negative", 1L)$gen)) +}) From 03283add5c9f49601f1abcfd2044d9e7b7d3d545 Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Fri, 6 Nov 2026 00:49:35 +1030 Subject: [PATCH 09/16] test: skip the handover checks without yaml --- tests/testthat/test-conditional-disclosure.R | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/testthat/test-conditional-disclosure.R b/tests/testthat/test-conditional-disclosure.R index 3986555..a9a8898 100644 --- a/tests/testthat/test-conditional-disclosure.R +++ b/tests/testthat/test-conditional-disclosure.R @@ -66,6 +66,7 @@ test_that("the audit raises HIGH when a clone tracks the original row by row", { seed = 1L, quiet = TRUE)) expect_gt(stats::cor(d$y, synthetic(m)$y), 0.95) expect_identical(m@audit$leakage_class[m@audit$col == "y"], "high") + skip_if_not_installed("yaml") expect_error(handover(m, file.path(withr::local_tempdir(), "p")), class = "masque_blocked_write") # Falsifier: the marginal clone of the same table is not flagged. @@ -131,6 +132,7 @@ test_that("three- and four-row levels that track their rows are HIGH and blocked seed = 1L, quiet = TRUE)) expect_gt(abs(stats::cor(d$y, synthetic(m)$y)), 0.9) expect_identical(m@audit$leakage_class[m@audit$col == "y"], "high") + skip_if_not_installed("yaml") expect_error( handover(m, file.path(withr::local_tempdir(), "p")), class = "masque_blocked_write" From fe3410d4b838448c8f068d1db6be51b9eccf926a Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Fri, 6 Nov 2026 09:37:58 +1030 Subject: [PATCH 10/16] fix: owner key, default unit, value integrity and write records before the 0.17.0 tag The owner's derived seed had 28 bits, and a per-replicate statistic verifies a guess. Permutations on the plan route now come from SHA-256 of a 256-bit system key. The default control unit is the finest legal column, not the coarsest. The recipe's integrity check binds values. The refusal text says what each writer records. Limits of the audit, the ladder and the land test are documented. --- DESCRIPTION | 1 + NEWS.md | 21 ++- R/audit.R | 6 + R/check_integrity.R | 56 ++++++- R/control_spec.R | 12 +- R/controls.R | 148 +++++++++++++++---- R/jitter_coordinates.R | 11 +- R/mask.R | 16 +- R/plan.R | 90 ++++++++--- R/recipe.R | 2 + R/recipe_apply.R | 3 +- R/release_gate.R | 5 +- R/write_set.R | 5 +- man/apply_recipe.Rd | 3 +- man/audit_mask.Rd | 6 + man/control_spec.Rd | 10 +- man/controls.Rd | 41 +++-- man/jitter_coordinates.Rd | 11 +- man/mask.Rd | 7 +- tests/testthat/test-conditional-disclosure.R | 1 - tests/testthat/test-controls-default-unit.R | 122 +++++++++++++++ tests/testthat/test-controls-negative.R | 3 +- tests/testthat/test-controls-owner-key.R | 103 +++++++++++++ tests/testthat/test-integrity-values.R | 83 +++++++++++ tests/testthat/test-plan.R | 30 +++- tests/testthat/test-safety-gate.R | 12 ++ vignettes/controls.Rmd | 22 ++- 27 files changed, 738 insertions(+), 92 deletions(-) create mode 100644 tests/testthat/test-controls-default-unit.R create mode 100644 tests/testthat/test-controls-owner-key.R create mode 100644 tests/testthat/test-integrity-values.R diff --git a/DESCRIPTION b/DESCRIPTION index 9b34df2..44bfd08 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -42,6 +42,7 @@ Suggests: ggplot2, knitr, maps, + openssl, readxl, rmarkdown, spelling, diff --git a/NEWS.md b/NEWS.md index 37a60a6..d6741b1 100644 --- a/NEWS.md +++ b/NEWS.md @@ -23,7 +23,26 @@ correlated 0.99 with the original at its rows, and the audit said low. Conditional clones with such a column change. Levels of three or four rows still carry a shift, since real alpha blocks are that small, and the audit grades them. -* A private `seed` in `controls()` no longer appears in the manifest. +* A private `seed` in `controls()` no longer appears in the manifest. With a + plan and its recipe, the owner's permutations now come from SHA-256 of a + 256-bit key from the system's random source, because a derived integer seed + could be searched against a per-replicate statistic. A private integer `seed` + keeps its 31 bits. Release the false-positive rate, the power and one + permutation p-value, and nothing computed from a single replicate. +* `apply_recipe()` and `controls()` with a recipe now refuse a table whose values + differ from those the recipe was built on, not only its NA mask. Numbers match + within what a file at 15 digits changes. +* In a plan, the default control unit is the finest design column on which `on` + is constant, not the coarsest, which can be a confounder. A column with too few + units for a 5 percent level is passed over with a warning. +* A warning now marks a unit and strata with too few arrangements for a 5 percent + level (`masque_control_unit_small`), strata holding one level of `on` above 10 + percent rather than 50 (`masque_control_strata_weak`), and a whole-valued + target whose rounding changes the declared shape + (`masque_control_shape_rounded`). +* `handover(allow_high = TRUE)` names the overridden findings in `README.txt`. + The refusal text now says what each writer records: only `masque()` writes to + the recipe. * The averaged-row figure in `?jitter_coordinates` is corrected to 0.64 km for 360 rows. * masque needs digest 0.6.21 or later. diff --git a/R/audit.R b/R/audit.R index d2eb710..1c6248e 100644 --- a/R/audit.R +++ b/R/audit.R @@ -89,6 +89,12 @@ #' (`grain = FALSE`, or a collaborate grain with fewer than five units) carries #' a note saying the clone does not keep that grain. #' +#' The grade of a re-simulated number compares rows. A shift on a design level +#' of three or four rows, with within-level noise as large as the signal, can +#' grade low while the correlation of the level means stays above 0.85. A low +#' class means the rows are not copies. It does not mean the level means are +#' hidden. +#' #' As a reference point, a collaborate-mode clone of `agridat::besag.met` #' (yield as the outcome, design columns kept) scores `low` on every #' column, and none of its yields equals the original value. diff --git a/R/check_integrity.R b/R/check_integrity.R index 6264ad4..00686b8 100644 --- a/R/check_integrity.R +++ b/R/check_integrity.R @@ -1,5 +1,5 @@ -#' Internal: verify the recipe's recorded NA-mask fingerprint against -#' the candidate `original` data frame. +#' Internal: verify the recipe's recorded NA-mask and value fingerprints +#' against the candidate `original` data frame. #' #' Called from [apply_recipe()] when `check_integrity = TRUE` (the #' default). Aborts with guidance if the fingerprint does not match. @@ -34,5 +34,57 @@ ) ), class = c("masque_integrity_refusal", "orchestra_refusal")) } + recorded <- .prop_or(rec, "content_fp", list()) + if (length(recorded) && !.content_matches(recorded, .content_fingerprint(original))) { + cli::cli_abort(c( + paste0( + "Recipe integrity check failed: the values of {.arg original} ", + "do not match the recipe's recorded fingerprint." + ), + i = "The recipe was built against a table with the same missingness and different values.", + "*" = paste0( + "If this is intentional, pass {.code check_integrity = FALSE} to ", + "{.fun apply_recipe}. {.fun controls} has no such switch: it runs on ", + "the table the clone was built on." + ) + ), class = c("masque_integrity_refusal", "orchestra_refusal")) + } invisible(TRUE) } + +# What the recipe records of the values. Numbers get one weighted sum per +# column, compared within a tolerance, because a table written at 15 digits and +# read back must still match; the weights make it depend on row order. Anything +# else gets a digest of its text, so a factor matches the character column a +# file gives back. Column order does not enter. +.content_fingerprint <- function(df) { + cols <- sort(names(df), method = "radix") + numeric_col <- vapply(cols, function(cl) is.numeric(df[[cl]]) && !is.object(df[[cl]]), logical(1L)) + sums <- lapply(cols[numeric_col], function(cl) { + x <- as.double(df[[cl]]) + ok <- is.finite(x) + w <- sin(seq_along(x) * sqrt(2)) + list( + value = sum(x[ok] * w[ok]), + scale = sum(abs(x[ok] * w[ok])), + other = digest::digest(list(which(!ok & !is.na(x)), x[!ok & !is.na(x)]), algo = "sha256") + ) + }) + names(sums) <- cols[numeric_col] + text <- lapply(cols[!numeric_col], function(cl) enc2utf8(as.character(df[[cl]]))) + names(text) <- cols[!numeric_col] + list(sums = sums, text = digest::digest(text, algo = "sha256")) +} + +.content_matches <- function(recorded, actual) { + if (!identical(recorded$text, actual$text) || + !setequal(names(recorded$sums), names(actual$sums))) { + return(FALSE) + } + all(vapply(names(recorded$sums), function(cl) { + a <- recorded$sums[[cl]] + b <- actual$sums[[cl]] + identical(a$other, b$other) && + abs(a$value - b$value) <= 1e-8 * max(a$scale, b$scale) + }, logical(1L))) +} diff --git a/R/control_spec.R b/R/control_spec.R index 4db52f8..0705de5 100644 --- a/R/control_spec.R +++ b/R/control_spec.R @@ -62,12 +62,18 @@ masque_control_spec <- S7::new_class( #' vector (`c(G01 = 0.5, G02 = -0.5)`; a level not named shifts by zero), #' or `list(sd = s)` to draw one shift per level from a normal #' distribution with standard deviation `s`, under the spec's seed, when -#' the spec first meets a table. Required with `target`. +#' the spec first meets a table. Required with `target`. On a whole-valued +#' target the shift is rounded, which can change the ratios between levels; +#' `shift_realised` in `@truth` holds the effect that is in the table, and a +#' warning (class `masque_control_shape_rounded`) marks a changed shape. #' @param strata Design columns within which `on` is permuted, or `NULL` #' for no stratification. Every unit of `on` must lie within one stratum. #' Prefer complete replicates to incomplete blocks; see [controls()]. #' @param unit Design columns within which `on` is constant. `on` is then -#' permuted across those units. `NULL` (the default) permutes rows. +#' permuted across those units. `NULL` (the default) permutes rows, and in a +#' plan picks the finest design column on which `on` is constant, if it has +#' enough units for a 5 percent level. In the plan's default a coarse column +#' is not preferred, since it can be a confounder of `on`. #' @param grain A named list giving, for other analysed columns constant #' within a design unit, the unit columns: `list(rain = "trial")`. Such a #' column is permuted across its units in the null control, so it stays @@ -161,7 +167,7 @@ control_spec <- function( # `NULL` or a character vector of design column names. .check_control_cols <- function(x, arg) { - if (is.null(x)) { + if (!length(x)) { return(character()) } .check_control_names(x, arg, single = FALSE) diff --git a/R/controls.R b/R/controls.R index fe2c451..c28388e 100644 --- a/R/controls.R +++ b/R/controls.R @@ -18,6 +18,8 @@ masque_controls <- S7::new_class( roles = S7::class_data.frame, spec = masque_control_spec, seeds = S7::class_integer, + # The owner's digest key; when set, `seeds` is NA and no integer is used. + key = S7::new_property(class = S7::class_character, default = character()), analysed = S7::class_character, events = S7::class_list, strata_id = S7::class_integer, @@ -67,13 +69,21 @@ masque_controls <- S7::new_class( #' #' @section Two sides, two seeds: #' An analyst who holds only the clone runs this on the clone. The owner -#' runs it on the original, offline, with a private `seed` that overrides -#' the spec's seed for the permutations (the planted vector still comes -#' from the spec's seed). A public seed would let anyone who knows the -#' design, which the clone shares, rebuild every permutation and read the -#' original outcome out of per-replicate statistics. Control tables built -#' from an original never leave the owner; what leaves is an aggregate -#' verdict about the pipeline, through the owner's usual output checking. +#' runs it on the original, offline, on a private draw (the planted vector +#' still comes from the spec's seed). A public seed would let anyone who knows +#' the design, which the clone shares, rebuild every permutation and read the +#' original outcome out of per-replicate statistics. +#' +#' With a plan and its `recipe`, the permutations come from SHA-256 of a +#' 256-bit key the recipe holds, so there is no seed to search. A private +#' integer `seed` is the weaker route: it has at most 31 bits, and anyone given +#' a per-replicate statistic can try each value, because the true seed fits the +#' statistic exactly and a wrong one does not. +#' +#' Control tables built from an original never leave the owner. What leaves is +#' the false-positive rate, the power and one permutation p-value, through the +#' owner's usual output checking. A vector of per-replicate means or contrasts +#' is not an aggregate, under either route. #' #' @section Limits: #' A kept column is structure and is never permuted, so leakage through @@ -84,6 +94,16 @@ masque_controls <- S7::new_class( #' design means of row-grained outcomes, so false-positive calibration #' comes from the negative control, not the null. #' +#' A negative control is a test of the stated size for a comparison that uses +#' only contrasts within the strata, so the analysis needs the strata in its +#' model. Permuting within replicates and fitting no replicate term puts the +#' replicate variance in the residual and rejects far less often than the +#' nominal level. A shared rate on the clone and the original shows that the two +#' agree, and a rate of zero on both says nothing about calibration. A warning +#' (class `masque_control_unit_small`) marks units and strata with too few +#' arrangements for a 5 percent level, and another (class +#' `masque_control_strata_weak`) marks strata that hold one level of `on`. +#' #' @param df The table to derive controls from: a data frame or data.table. #' @param roles Its role table, as returned by [roles()] for a clone or by #' [propose_roles()] and [set_role()] for any table. Or a plan: the analyst @@ -94,13 +114,14 @@ masque_controls <- S7::new_class( #' @param events Linked time/status pairs among the outcome columns, in the #' form [mask()] takes. The null permutes each pair as one block. `NULL` #' when there are none. -#' @param seed A private seed for the permutations that overrides the +#' @param seed A private integer seed for the permutations that overrides the #' spec's seed, for the owner's side; `NULL` uses the spec's seed, or with a -#' `recipe` a private seed derived from the recipe's. A value equal to the -#' spec's seed is refused there. +#' `recipe` the recipe's key. A value equal to the spec's seed is refused +#' there. See "Two sides, two seeds" for what an integer can and cannot hide. #' @param recipe With a plan in `roles`, the recipe of the clone. `df` is then #' the real table: its cleaning and conformance are replayed as in -#' [apply_recipe()], a table with a different schema or NA mask is refused, +#' [apply_recipe()], a table with a different schema, NA mask or values is +#' refused, #' and a spec that differs from the one fixed at [mask()] is refused. #' #' @returns A `masque_controls` object: the resolved spec (`@spec`), the @@ -130,6 +151,7 @@ controls <- function(df, roles, spec = NULL, events = NULL, seed = NULL, recipe = NULL) { withr::local_preserve_seed() plan_notes <- character() + key <- character() if (S7::S7_inherits(roles, .plan_class)) { from_plan <- .controls_from_plan(df, roles, spec, events, seed, recipe) df <- from_plan$df @@ -137,6 +159,7 @@ controls <- function(df, roles, spec = NULL, events = NULL, seed = NULL, spec <- from_plan$spec events <- from_plan$events seed <- from_plan$seed + key <- from_plan$key plan_notes <- from_plan$notes } else if (!is.null(recipe)) { .control_refuse("`recipe` goes with a plan in `roles`: {.code controls(real, plan, recipe = rec)}.", "recipe") @@ -155,12 +178,17 @@ controls <- function(df, roles, spec = NULL, events = NULL, seed = NULL, .check_control_target_events(res$spec, events) # Seeds: one for the null, one per replicate ----------------------------- - run_seed <- if (is.null(seed)) spec@seed else as.integer(seed) - seeds <- as.integer(.sub_seeds(run_seed, res$spec@n + 1L)) + seeds <- if (length(key)) { + rep(NA_integer_, res$spec@n + 1L) + } else { + as.integer(.sub_seeds( + if (is.null(seed)) spec@seed else as.integer(seed), res$spec@n + 1L + )) + } manifest <- .control_manifest(res, seeds) # A manifest is what the owner reads before a release: no private seed in it. - if (!is.null(seed)) { + if (!is.null(seed) || length(key)) { manifest$seed <- NA_integer_ } masque_controls( @@ -168,6 +196,7 @@ controls <- function(df, roles, spec = NULL, events = NULL, seed = NULL, roles = .as_owned_dt(roles), spec = res$spec, seeds = seeds, + key = key, analysed = res$analysed, events = events, strata_id = res$strata_id, @@ -223,10 +252,14 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) } out <- data.table::copy(ctl@table) - if (identical(kind, "null")) { - with_rng_state(ctl@seeds[1L], .permute_analysed(out, ctl)) + stream <- if (identical(kind, "null")) 0L else i + permute <- if (identical(kind, "null")) .permute_analysed else .permute_on + if (length(ctl@key)) { + permute(out, ctl, .keyed_draw(ctl@key, stream)) } else { - with_rng_state(ctl@seeds[1L + i], .permute_on(out, ctl)) + with_rng_state(ctl@seeds[1L + stream], permute(out, ctl, sample.int)) + } + if (!identical(kind, "null")) { if (identical(kind, "positive")) { .plant_effect(out, ctl) } @@ -309,15 +342,24 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) "degenerate" ) } - if (constant_share > 0.5) { + if (constant_share > 0.1) { warning(warningCondition( sprintf( - "`on` (%s) is constant within %.0f%% of the strata; the negative control permutes little.", + "`on` (%s) is constant within %.0f%% of the strata; the negative control does not permute those strata, and a model that ignores them can reject on every replicate.", on, 100 * constant_share ), class = "masque_control_strata_weak" )) } + if (!.level_attainable(unit_id, df[[on]], strata_id)) { + warning(warningCondition( + sprintf( + "`on` (%s) has too few distinct arrangements across its units and strata for a 5 percent randomisation level, so a rejection rate on the negative controls cannot read as a test size.", + on + ), + class = "masque_control_unit_small" + )) + } # Grain of other analysed columns ---------------------------------------- grain_units <- list() @@ -541,6 +583,17 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) shift_raw <- unname(effect) * sd_y shift_realised <- if (whole) round(shift_raw) else shift_raw vanished <- whole & effect != 0 & shift_realised == 0 + if (whole && !any(vanished) && any(shift_raw != 0) && max(abs( + shift_realised / sum(abs(shift_realised)) - shift_raw / sum(abs(shift_raw)) + )) > 0.1) { + warning(warningCondition( + sprintf( + "On the whole-valued target %s, rounding changes the shape of the declared effect. `shift_realised` in `@truth` holds the effect that is in the table.", + tg + ), + class = "masque_control_shape_rounded" + )) + } if (any(vanished)) { .control_refuse( c( @@ -591,10 +644,53 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) # Permutation --------------------------------------------------------------- +# Can a permutation of `on` across units within strata reach a 5 percent level? +# True when it has at least 20 arrangements; swapping the labels of equal-sized +# levels with no strata gives the same partition, so those count once. +.level_attainable <- function(unit_id, on_val, strata_id) { + keep <- !is.na(on_val) & !duplicated(unit_id) + val <- as.character(on_val[keep]) + log_n <- sum(vapply(split(val, strata_id[keep]), function(v) { + lfactorial(length(v)) - sum(lfactorial(table(v))) + }, numeric(1L))) + if (length(unique(strata_id[keep])) == 1L) { + log_n <- log_n - sum(lfactorial(table(table(val)))) + } + log_n >= log(20) - 1e-8 +} + +# One permutation of 1:n by Fisher-Yates, its uniforms taken from SHA-256 of +# the label and a counter, so the key is the whole digest and not an R seed. +.keyed_perm <- function(label, n) { + perm <- seq_len(n) + if (n < 2L) { + return(perm) + } + hash <- digest::getVDigest("sha256") + h <- hash(paste(label, seq_len(n - 1L), sep = "\r"), serialize = FALSE) + u <- (strtoi(substr(h, 1L, 6L), 16L) * 16^7 + strtoi(substr(h, 7L, 13L), 16L) + 0.5) / 16^13 + for (i in n:2L) { + j <- 1L + as.integer(floor(u[i - 1L] * i)) + held <- perm[i] + perm[i] <- perm[j] + perm[j] <- held + } + perm +} + +# A drawing function for one dataset: successive calls take successive labels. +.keyed_draw <- function(key, stream) { + calls <- 0L + function(n) { + calls <<- calls + 1L + .keyed_perm(paste(key, stream, calls, sep = "\r"), n) + } +} + # Permute the non-missing values of `x` across units within strata; an # all-missing unit stays missing. With `unit_id = seq_along(x)` it is a row # permutation. -.permute_within <- function(x, unit_id, strata_id) { +.permute_within <- function(x, unit_id, strata_id, draw = sample.int) { obs <- which(!is.na(x)) if (length(obs) < 2L) { return(x) @@ -605,7 +701,7 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) groups <- split(seq_along(u_obs), strata_id[first]) perm <- integer(length(u_obs)) for (g in groups) { - perm[g] <- g[sample.int(length(g))] + perm[g] <- g[draw(length(g))] } new_val <- u_val[perm] out <- x @@ -615,7 +711,7 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) # The null: every analysed column on its own draw, at its grain, no strata; # a linked-event pair moves as one row block. -.permute_analysed <- function(out, ctl) { +.permute_analysed <- function(out, ctl, draw) { n <- nrow(out) rows <- seq_len(n) none <- rep(1L, n) @@ -625,10 +721,10 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) if (is.null(uid)) { uid <- rows } - data.table::set(out, j = cl, value = .permute_within(out[[cl]], uid, none)) + data.table::set(out, j = cl, value = .permute_within(out[[cl]], uid, none, draw)) } for (e in ctl@events) { - perm <- .permute_within(rows, rows, none) + perm <- .permute_within(rows, rows, none, draw) for (cl in c(e$time, e$status)) { data.table::set(out, j = cl, value = out[[cl]][perm]) } @@ -637,12 +733,12 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) } # The negative: `on` alone, within strata, at its unit. -.permute_on <- function(out, ctl) { +.permute_on <- function(out, ctl, draw) { on <- ctl@spec@on data.table::set( out, j = on, - value = .permute_within(out[[on]], ctl@unit_id, ctl@strata_id) + value = .permute_within(out[[on]], ctl@unit_id, ctl@strata_id, draw) ) invisible(out) } diff --git a/R/jitter_coordinates.R b/R/jitter_coordinates.R index 9e993ef..4d14551 100644 --- a/R/jitter_coordinates.R +++ b/R/jitter_coordinates.R @@ -68,6 +68,11 @@ #' #' Seven kept rebuilds fall below the 5 km floor the donut was asked for. #' +#' A known region also cuts the annulus. Someone who knows a site lies in a +#' square a few times `max_km` wide or smaller can discard the part of the +#' annulus around the released point that falls outside it, with no second +#' release needed. +#' #' The remedy is to **reuse the seed**. The same `seed` gives a site the same #' displacement in every rebuild, even when rows are shuffled, added or removed, #' so any number of rebuilds yields one point and nothing to average. Three @@ -117,7 +122,11 @@ #' displacement that lands in the sea, using [maps::map.where()] (requires the #' `maps` package). A function `function(lon, lat)` returning a logical vector #' supplies your own test (for example an `sf`-based high-resolution -#' coastline). `FALSE` applies no constraint. +#' coastline). `FALSE` applies no constraint. The test conditions the annulus: +#' near a coast the accepted directions are the ones on land, so the mean +#' displacement points inland and the median distance can move by kilometres, +#' while no point falls inside `min_km`. A site that cannot reach land is +#' missing. #' @param max_tries Maximum re-draws per site before giving up. Default `100`. #' A site that cannot be placed on land is set to `NA` on both axes, with a #' warning: masque never emits the true coordinate as though it were masked. diff --git a/R/mask.R b/R/mask.R index 0cc3bf5..cb29232 100644 --- a/R/mask.R +++ b/R/mask.R @@ -140,7 +140,12 @@ #' how a three-replicate variety trial keeps its genotype means. A level #' with a single row is pooled with the other singletons before the fit, #' so no row's own outcome is carried. The terms carried are recorded on -#' the recipe as `conditioning_shifted`. `"levels"` +#' the recipe as `conditioning_shifted`. The interaction of a dropped column +#' with a kept treatment is not carried, so a multi-environment clone can keep +#' its genotype and environment means and lose the genotype-by-environment +#' pattern. Shrinkage of a shift uses the mean level size, so on unbalanced +#' columns the small levels are shrunk less than their own size would give. +#' `"levels"` #' is the ladder of masque 0.11.1 to 0.12.0: columns are dropped in #' decreasing order of distinct values and a dropped column leaves no #' trace in the clone. @@ -675,10 +680,17 @@ mask <- function(df, allow_unmasked_coords = isTRUE(allow_unmasked_coords), allow_kept_outcome = allow_kept_outcome, warnings = warnings_acc, + content_fp = .content_fingerprint(df), integrity_fp = digest::digest(is.na(df), algo = "sha256") ) if (!is.null(plan)) { rec@owner_key <- .new_owner_key() + if (!length(rec@owner_key)) { + warning(warningCondition( + "No system random source was found, so this recipe holds no key for the owner's controls. Install openssl, or pass a private `seed` to controls().", + class = "masque_owner_key_unavailable" + )) + } rec@plan <- .plan_resolve(plan, df, roles, rec, list( coords = coords, events = events, conditional = conditional_declared, ladder = ladder, grain = grain, alias_names = alias_names @@ -759,7 +771,7 @@ mask <- function(df, shifted <- report$shifted if (length(shifted)) { parts <- c(parts, sprintf( - "The main effect of %s is carried into the clone as an additive shift.", + "The main effect of %s is carried into the clone as an additive shift; its interaction with a kept treatment is not.", paste(shifted, collapse = ", ") )) } diff --git a/R/plan.R b/R/plan.R index d181294..e9cc877 100644 --- a/R/plan.R +++ b/R/plan.R @@ -323,11 +323,14 @@ local(S7::method(print, .plan_class) <- function(x, ...) { stats::setNames(unname(draw[clone]), levels) } -# Coarsest design column, then pair of columns, within which `on` is constant. -# A row alone is constant by construction, so a quarter of the rows must share -# a unit for the constancy to mean anything. +# Finest design column, then pair of columns, within which `on` is constant. A +# coarse column of this kind can be a confounder. A row alone is constant by +# construction, so a quarter of the rows must share a unit for the constancy to +# mean anything. A column whose units cannot reach a 5 percent level is passed +# over, and the unit falls to the row with a warning when nothing else remains. .default_control_unit <- function(df, on, design) { n <- nrow(df) + too_few <- character() sets <- c(as.list(design), if (length(design) >= 2L) { utils::combn(design, 2L, simplify = FALSE) }) @@ -337,12 +340,28 @@ local(S7::method(print, .plan_class) <- function(x, ...) { id <- as.integer(.unit_id(df, s)) sizes <- tabulate(id) shared <- sum(sizes[sizes >= 2L]) - if (shared >= n / 4 && .constant_within(id, df[[on]])) max(id) else NA_integer_ + if (shared < n / 4 || !.constant_within(id, df[[on]])) { + return(NA_integer_) + } + if (!.level_attainable(id, df[[on]], rep(1L, n))) { + too_few <<- c(too_few, paste(s, collapse = " + ")) + return(NA_integer_) + } + max(id) }, integer(1L)) if (any(!is.na(n_units))) { - return(pick[[which.min(n_units)]]) + return(pick[[which.max(n_units)]]) } } + if (length(too_few)) { + warning(warningCondition( + sprintf( + "`%s` is constant within %s, which has too few units for a 5 percent randomisation level. The default unit is the row; name a `unit` to choose otherwise.", + on, paste(too_few, collapse = ", ") + ), + class = "masque_control_unit_small" + )) + } character() } @@ -500,12 +519,27 @@ clone_plan <- function(m) { # Offset that roots a drawn effect away from the spec's seed. .CONTROL_SALT <- 15485863L -# Not from the R stream: a seeded mask() must not determine it. +# 256 bits from the operating system, or NULL where neither source exists. +.system_entropy <- function() { + if (file.exists("/dev/urandom")) { + con <- file("/dev/urandom", "rb", raw = TRUE) + on.exit(close(con)) + return(readBin(con, "raw", 32L)) + } + if (requireNamespace("openssl", quietly = TRUE)) { + return(openssl::rand_bytes(32L)) + } + NULL +} + +# Not from the R stream: a seeded mask() must not determine it. Time, process +# id and a temporary name can be enumerated, so they do not serve as a key. .new_owner_key <- function() { - digest::digest( - list(Sys.time(), Sys.getpid(), tempfile()), - algo = "sha256" - ) + bytes <- .system_entropy() + if (is.null(bytes)) { + return(character()) + } + paste(as.character(bytes), collapse = "") } # Arguments of controls() from a plan, on either side of the boundary. @@ -527,6 +561,7 @@ clone_plan <- function(m) { spec = picked$spec, events = if (is.null(events) && length(plan@events)) plan@events else events, seed = seed, + key = character(), notes = character() )) } @@ -539,6 +574,7 @@ clone_plan <- function(m) { picked$spec } .check_spec_as_declared(source_spec, picked$name, recipe) + draw <- .control_source_draw(.prop_or(recipe, "owner_key", character()), source_spec@seed, seed) list( df = .controls_replay_source(df, recipe), roles = as.data.frame(recipe@roles), @@ -548,7 +584,8 @@ clone_plan <- function(m) { } else { events }, - seed = .control_source_seed(.prop_or(recipe, "owner_key", character()), source_spec@seed, seed), + seed = draw$seed, + key = draw$key, notes = "Source side: the seed is private. Release aggregates only." ) } @@ -642,7 +679,9 @@ clone_plan <- function(m) { df } -.control_source_seed <- function(owner_key, spec_seed, seed) { +# The owner's draw: an explicit integer seed, or the whole digest of the +# recipe's key. An integer can be searched against a released statistic. +.control_source_draw <- function(owner_key, spec_seed, seed) { if (!is.null(seed)) { .check_seed(seed) if (as.integer(seed) == spec_seed) { @@ -651,16 +690,18 @@ clone_plan <- function(m) { "seed" ) } - return(as.integer(seed)) + return(list(seed = as.integer(seed), key = character())) } if (!length(owner_key)) { .control_refuse( - "This recipe was masked without a plan, so it holds no key to derive a private seed from. Pass a private `seed`.", + "This recipe holds no key: it was masked without a plan, or where no system random source was available. Pass a private `seed`.", "seed" ) } - h <- digest::digest(paste(owner_key, spec_seed), algo = "sha256", serialize = FALSE) - strtoi(substr(h, 1L, 7L), 16L) + list( + seed = NULL, + key = digest::digest(paste(owner_key, spec_seed), algo = "sha256", serialize = FALSE) + ) } # YAML ------------------------------------------------------------------------ @@ -890,7 +931,7 @@ handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE, class = c("masque_handover_local_refusal", "orchestra_refusal") ) } - high <- .gate_release(m, allow_high) + high <- .gate_release(m, allow_high, "the findings are named in README.txt") if (dir.exists(dir) && length(list.files(dir, all.files = TRUE, no.. = TRUE)) && !isTRUE(overwrite)) { cli::cli_abort( @@ -907,11 +948,16 @@ handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE, data.table::fwrite(m@synthetic, paths[1L]) save_plan(clone_plan(m), paths[2L]) data_card(m, file = paths[3L], name = name) - writeLines(.handover_note(m, length(high)), paths[4L]) + writeLines(.handover_note(m, high), paths[4L]) invisible(paths) } -.handover_note <- function(m, n_high) { +# Finding names in the clone's vocabulary: the note goes to the analyst. +.handover_high_names <- function(m, high) { + .plan_col_translator(m@recipe, "clone")(high) +} + +.handover_note <- function(m, high) { plan <- clone_plan(m) r <- plan@roles kept <- r$col[r$role == "outcome" & r$action == "keep"] @@ -939,10 +985,10 @@ handover <- function(m, dir, name = NULL, overwrite = FALSE, allow_high = FALSE, "known-truth controls." ) }, - if (n_high > 0L) { + if (length(high)) { sprintf( - "The owner released this clone after reviewing HIGH leakage findings: %d.", - n_high + "The owner released this clone after reviewing HIGH leakage findings on: %s.", + paste(.handover_high_names(m, high), collapse = ", ") ) } ) diff --git a/R/recipe.R b/R/recipe.R index b6c0a70..e382205 100644 --- a/R/recipe.R +++ b/R/recipe.R @@ -101,6 +101,8 @@ masque_recipe <- S7::new_class( warnings = S7::new_property( class = S7::class_character, default = character() ), + # A fingerprint of the values, beside the NA mask's. Absent before 0.17.0. + content_fp = S7::new_property(class = S7::class_list, default = list()), integrity_fp = S7::class_character ) ) diff --git a/R/recipe_apply.R b/R/recipe_apply.R index 48f0730..2ec21eb 100644 --- a/R/recipe_apply.R +++ b/R/recipe_apply.R @@ -34,7 +34,8 @@ #' @param rec A `masque_recipe` object (e.g. from `recipe(m)`). #' @param check_integrity Logical. When `TRUE` (default), verifies that #' the NA mask of `original` matches the recipe's recorded -#' `integrity_fp`. Mismatches error with guidance. Pass `FALSE` to +#' `integrity_fp` and values against the recipe's fingerprint of them, +#' numbers within rounding from a file. Mismatches error with guidance. Pass `FALSE` to #' bypass when the missingness has legitimately changed since the #' recipe was built. #' diff --git a/R/release_gate.R b/R/release_gate.R index 53d2dee..09d74a7 100644 --- a/R/release_gate.R +++ b/R/release_gate.R @@ -92,7 +92,8 @@ # Gate a package-managed write: abort ("masque_blocked_write") on unresolved # HIGH findings, or warn ("masque_high_override") under `allow_high = TRUE`. -.gate_release <- function(m, allow_high = FALSE) { +.gate_release <- function(m, allow_high = FALSE, + record = "the override is recorded on the recipe") { high <- if (identical(m@mode, "collaborate")) { .audit_high_cols(m) } else { @@ -113,7 +114,7 @@ i = "Re-role, alias, or drop each flagged column, then mask again.", i = paste0( "Or pass `allow_high = TRUE` to write anyway after your own ", - "review - the override is recorded." + "review - {record}." ) ), class = "masque_blocked_write" diff --git a/R/write_set.R b/R/write_set.R index a3dfb98..b186591 100644 --- a/R/write_set.R +++ b/R/write_set.R @@ -47,7 +47,10 @@ write_set <- function(m, path, overwrite = FALSE, allow_high = FALSE) { if (!is.character(path) || length(path) != 1L || !nzchar(path)) { cli::cli_abort("`path` must be a single non-empty string.") } - .gate_release(m, allow_high) + .gate_release( + m, allow_high, + "the override is only in that warning, so keep your own record" + ) .write_set_dispatch(m@synthetic, path, overwrite) invisible(path) } diff --git a/man/apply_recipe.Rd b/man/apply_recipe.Rd index e98a88c..099d4c4 100644 --- a/man/apply_recipe.Rd +++ b/man/apply_recipe.Rd @@ -13,7 +13,8 @@ apply_recipe(original, rec, check_integrity = TRUE) \item{check_integrity}{Logical. When \code{TRUE} (default), verifies that the NA mask of \code{original} matches the recipe's recorded -\code{integrity_fp}. Mismatches error with guidance. Pass \code{FALSE} to +\code{integrity_fp} and values against the recipe's fingerprint of them, +numbers within rounding from a file. Mismatches error with guidance. Pass \code{FALSE} to bypass when the missingness has legitimately changed since the recipe was built.} } diff --git a/man/audit_mask.Rd b/man/audit_mask.Rd index 06f1c00..9307aa2 100644 --- a/man/audit_mask.Rd +++ b/man/audit_mask.Rd @@ -112,6 +112,12 @@ A column constant within a design unit that was synthesised row by row (\code{grain = FALSE}, or a collaborate grain with fewer than five units) carries a note saying the clone does not keep that grain. +The grade of a re-simulated number compares rows. A shift on a design level +of three or four rows, with within-level noise as large as the signal, can +grade low while the correlation of the level means stays above 0.85. A low +class means the rows are not copies. It does not mean the level means are +hidden. + As a reference point, a collaborate-mode clone of \code{agridat::besag.met} (yield as the outcome, design columns kept) scores \code{low} on every column, and none of its yields equals the original value. diff --git a/man/control_spec.Rd b/man/control_spec.Rd index d7dd4f7..73d5042 100644 --- a/man/control_spec.Rd +++ b/man/control_spec.Rd @@ -33,14 +33,20 @@ target's standard deviation in the table at hand: a named numeric vector (\code{c(G01 = 0.5, G02 = -0.5)}; a level not named shifts by zero), or \code{list(sd = s)} to draw one shift per level from a normal distribution with standard deviation \code{s}, under the spec's seed, when -the spec first meets a table. Required with \code{target}.} +the spec first meets a table. Required with \code{target}. On a whole-valued +target the shift is rounded, which can change the ratios between levels; +\code{shift_realised} in \verb{@truth} holds the effect that is in the table, and a +warning (class \code{masque_control_shape_rounded}) marks a changed shape.} \item{strata}{Design columns within which \code{on} is permuted, or \code{NULL} for no stratification. Every unit of \code{on} must lie within one stratum. Prefer complete replicates to incomplete blocks; see \code{\link[=controls]{controls()}}.} \item{unit}{Design columns within which \code{on} is constant. \code{on} is then -permuted across those units. \code{NULL} (the default) permutes rows.} +permuted across those units. \code{NULL} (the default) permutes rows, and in a +plan picks the finest design column on which \code{on} is constant, if it has +enough units for a 5 percent level. In the plan's default a coarse column +is not preferred, since it can be a confounder of \code{on}.} \item{grain}{A named list giving, for other analysed columns constant within a design unit, the unit columns: \code{list(rain = "trial")}. Such a diff --git a/man/controls.Rd b/man/controls.Rd index 355e68f..d66376c 100644 --- a/man/controls.Rd +++ b/man/controls.Rd @@ -21,14 +21,15 @@ its specs, or \code{NULL} when it holds exactly one.} form \code{\link[=mask]{mask()}} takes. The null permutes each pair as one block. \code{NULL} when there are none.} -\item{seed}{A private seed for the permutations that overrides the +\item{seed}{A private integer seed for the permutations that overrides the spec's seed, for the owner's side; \code{NULL} uses the spec's seed, or with a -\code{recipe} a private seed derived from the recipe's. A value equal to the -spec's seed is refused there.} +\code{recipe} the recipe's key. A value equal to the spec's seed is refused +there. See "Two sides, two seeds" for what an integer can and cannot hide.} \item{recipe}{With a plan in \code{roles}, the recipe of the clone. \code{df} is then the real table: its cleaning and conformance are replayed as in -\code{\link[=apply_recipe]{apply_recipe()}}, a table with a different schema or NA mask is refused, +\code{\link[=apply_recipe]{apply_recipe()}}, a table with a different schema, NA mask or values is +refused, and a spec that differs from the one fixed at \code{\link[=mask]{mask()}} is refused.} } \value{ @@ -77,13 +78,21 @@ and the truth table are on the returned object and in its print. \section{Two sides, two seeds}{ An analyst who holds only the clone runs this on the clone. The owner -runs it on the original, offline, with a private \code{seed} that overrides -the spec's seed for the permutations (the planted vector still comes -from the spec's seed). A public seed would let anyone who knows the -design, which the clone shares, rebuild every permutation and read the -original outcome out of per-replicate statistics. Control tables built -from an original never leave the owner; what leaves is an aggregate -verdict about the pipeline, through the owner's usual output checking. +runs it on the original, offline, on a private draw (the planted vector +still comes from the spec's seed). A public seed would let anyone who knows +the design, which the clone shares, rebuild every permutation and read the +original outcome out of per-replicate statistics. + +With a plan and its \code{recipe}, the permutations come from SHA-256 of a +256-bit key the recipe holds, so there is no seed to search. A private +integer \code{seed} is the weaker route: it has at most 31 bits, and anyone given +a per-replicate statistic can try each value, because the true seed fits the +statistic exactly and a wrong one does not. + +Control tables built from an original never leave the owner. What leaves is +the false-positive rate, the power and one permutation p-value, through the +owner's usual output checking. A vector of per-replicate means or contrasts +is not an aggregate, under either route. } \section{Limits}{ @@ -95,6 +104,16 @@ Nothing is clipped, so a shift can take a non-negative outcome below zero; \code{\link[=control_table]{control_table()}} warns when it does. The null collapses the design means of row-grained outcomes, so false-positive calibration comes from the negative control, not the null. + +A negative control is a test of the stated size for a comparison that uses +only contrasts within the strata, so the analysis needs the strata in its +model. Permuting within replicates and fitting no replicate term puts the +replicate variance in the residual and rejects far less often than the +nominal level. A shared rate on the clone and the original shows that the two +agree, and a rate of zero on both says nothing about calibration. A warning +(class \code{masque_control_unit_small}) marks units and strata with too few +arrangements for a 5 percent level, and another (class +\code{masque_control_strata_weak}) marks strata that hold one level of \code{on}. } \examples{ diff --git a/man/jitter_coordinates.Rd b/man/jitter_coordinates.Rd index b8c3143..9b01055 100644 --- a/man/jitter_coordinates.Rd +++ b/man/jitter_coordinates.Rd @@ -43,7 +43,11 @@ most \code{max_km}.} displacement that lands in the sea, using \code{\link[maps:map.where]{maps::map.where()}} (requires the \code{maps} package). A function \verb{function(lon, lat)} returning a logical vector supplies your own test (for example an \code{sf}-based high-resolution -coastline). \code{FALSE} applies no constraint.} +coastline). \code{FALSE} applies no constraint. The test conditions the annulus: +near a coast the accepted directions are the ones on land, so the mean +displacement points inland and the median distance can move by kilometres, +while no point falls inside \code{min_km}. A site that cannot reach land is +missing.} \item{max_tries}{Maximum re-draws per site before giving up. Default \code{100}. A site that cannot be placed on land is set to \code{NA} on both axes, with a @@ -136,6 +140,11 @@ Rebuilds kept \tab Median recovery error \cr Seven kept rebuilds fall below the 5 km floor the donut was asked for. +A known region also cuts the annulus. Someone who knows a site lies in a +square a few times \code{max_km} wide or smaller can discard the part of the +annulus around the released point that falls outside it, with no second +release needed. + The remedy is to \strong{reuse the seed}. The same \code{seed} gives a site the same displacement in every rebuild, even when rows are shuffled, added or removed, so any number of rebuilds yields one point and nothing to average. Three diff --git a/man/mask.Rd b/man/mask.Rd index 62bc512..703ea73 100644 --- a/man/mask.Rd +++ b/man/mask.Rd @@ -125,7 +125,12 @@ stratum keep the stratum columns' main effects the same way, which is how a three-replicate variety trial keeps its genotype means. A level with a single row is pooled with the other singletons before the fit, so no row's own outcome is carried. The terms carried are recorded on -the recipe as \code{conditioning_shifted}. \code{"levels"} +the recipe as \code{conditioning_shifted}. The interaction of a dropped column +with a kept treatment is not carried, so a multi-environment clone can keep +its genotype and environment means and lose the genotype-by-environment +pattern. Shrinkage of a shift uses the mean level size, so on unbalanced +columns the small levels are shrunk less than their own size would give. +\code{"levels"} is the ladder of masque 0.11.1 to 0.12.0: columns are dropped in decreasing order of distinct values and a dropped column leaves no trace in the clone.} diff --git a/tests/testthat/test-conditional-disclosure.R b/tests/testthat/test-conditional-disclosure.R index a9a8898..d9f4beb 100644 --- a/tests/testthat/test-conditional-disclosure.R +++ b/tests/testthat/test-conditional-disclosure.R @@ -93,7 +93,6 @@ test_that("the design clone's residual matches an owner's negative control", { clone_rms <- vapply(1:40, function(s) { rms(synthetic(suppressWarnings(mask(d, r, seed = s, quiet = TRUE, conditional = "design")))) }, numeric(1)) - expect_gt(rms(d) * 5, 1) expect_gt(mean(owner_rms), 5 * rms(d)) expect_lt(abs(mean(clone_rms) / mean(owner_rms) - 1), 0.25) }) diff --git a/tests/testthat/test-controls-default-unit.R b/tests/testthat/test-controls-default-unit.R new file mode 100644 index 0000000..c793ad4 --- /dev/null +++ b/tests/testthat/test-controls-default-unit.R @@ -0,0 +1,122 @@ +# The plan's default unit, and the warnings on a control that cannot read as a +# test size. + +confounded <- function(tech = TRUE) { + set.seed(2) + d <- data.frame( + field = rep(paste0("f", 1:3), each = 30L), + tech = rep(paste0("t", 1:9), each = 10L), + plot = 1:90 + ) + d$gen <- paste0("G", match(d$field, paste0("f", 1:3))) + d$y <- c(0, 5, 1)[match(d$field, paste0("f", 1:3))] + stats::rnorm(90L) + design <- if (tech) c("field", "tech") else "field" + if (!tech) d$tech <- NULL + r <- suppressWarnings(propose_roles(d, detect = FALSE)) + r <- set_role(r, design, role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + r <- set_role(r, "plot", role = "id", action = "keep") + list(d = d, r = r) +} + +test_that("the default unit is the finest legal column, not the confounder", { + f <- confounded() + spec <- control_spec(on = "gen", target = "y", effect = list(sd = 0.5), n = 20L, seed = 3L) + m <- mask(f$d, masque_plan(f$r, controls = spec), seed = 4L, quiet = TRUE) + own <- controls(f$d, clone_plan(m), recipe = recipe(m)) + expect_identical(own@spec@unit, "tech") + p <- vapply(1:20, function(i) { + stats::anova(stats::lm(y ~ gen, control_table(own, "negative", i)))["gen", "Pr(>F)"] + }, numeric(1L)) + expect_gt(length(unique(signif(p, 6L))), 1L) + # Column order does not choose. + r2 <- f$r[rev(seq_len(nrow(f$r))), ] + d2 <- f$d[, rev(names(f$d))] + expect_identical( + recipe(mask(d2, masque_plan(r2, controls = spec), seed = 4L, quiet = TRUE))@plan@controls$default@unit, + "tech" + ) +}) + +test_that("between two legal columns the finer one is chosen", { + d <- data.frame( + field = rep(paste0("f", 1:8), each = 8L), + tech = rep(paste0("t", 1:16), each = 4L), + gen = rep(paste0("G", 1:4), each = 16L) + ) + expect_identical(masque:::.default_control_unit(d, "gen", c("field", "tech")), "tech") + expect_identical(masque:::.default_control_unit(d, "gen", c("tech", "field")), "tech") +}) + +test_that("a column with too few units is passed over, with a warning", { + f <- confounded(tech = FALSE) + spec <- control_spec(on = "gen", n = 20L, seed = 3L) + expect_warning( + m <- mask(f$d, masque_plan(f$r, controls = spec), seed = 4L, quiet = TRUE), + class = "masque_control_unit_small" + ) + expect_length(recipe(m)@plan@controls$default@unit, 0L) +}) + +test_that("a named unit with too few arrangements warns and still runs", { + f <- confounded(tech = FALSE) + spec <- control_spec(on = "gen", unit = "field", n = 2L, seed = 3L) + expect_warning(ctl <- controls(f$d, f$r, spec), class = "masque_control_unit_small") + expect_identical(ctl@spec@unit, "field") +}) + +test_that("arrangements are counted within strata and across equal labels", { + id <- 1:3 + expect_false(masque:::.level_attainable(id, c("a", "b", "c"), rep(1L, 3L))) + expect_false(masque:::.level_attainable(1:6, c("a", "b", "c", "d", "e", "f"), rep(1L, 6L))) + expect_true(masque:::.level_attainable(1:72, rep(paste0("G", 1:24), 3L), rep(1:3, each = 24L))) + # Two equal labels: choose(6, 3) = 20 arrangements are 10 partitions. + expect_false(masque:::.level_attainable(1:6, rep(c("a", "b"), each = 3L), rep(1L, 6L))) + expect_false(masque:::.level_attainable(1:6, c("a", "a", "b", "b", "b", "b"), rep(1:2, each = 3L))) + expect_true(masque:::.level_attainable(1:7, c(rep("a", 3L), rep("b", 4L)), rep(1L, 7L))) +}) + +test_that("strata holding one level warn from a tenth of the strata", { + d <- data.frame( + site = rep(paste0("S", 1:5), each = 8L), + trt = c(rep("a", 8L), rep("b", 8L), rep(c("a", "b"), 12L)), + y = seq_len(40L) + 0.5 + ) + r <- suppressWarnings(propose_roles(d, detect = FALSE)) + r <- set_role(r, "site", role = "design", action = "keep") + r <- set_role(r, "trt", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + expect_warning( + ctl <- controls(d, r, control_spec(on = "trt", strata = "site", seed = 1L)), + class = "masque_control_strata_weak" + ) + expect_identical(ctl@manifest$strata_constant_share[1L], 0.4) +}) + +test_that("the stored absence of strata goes back through the constructor", { + f <- confounded() + m <- mask(f$d, masque_plan(f$r, controls = control_spec(on = "gen", n = 2L, seed = 3L)), + seed = 4L, quiet = TRUE) + s <- recipe(m)@plan@controls$default + expect_length(s@strata, 0L) + again <- control_spec(on = "gen", strata = s@strata, unit = s@unit, n = 2L, seed = 3L) + expect_length(again@strata, 0L) +}) + +test_that("rounding that changes the declared shape warns", { + set.seed(4) + d <- data.frame(gen = rep(c("a", "b", "c"), 20L), y = as.integer(rpois(60L, 10))) + r <- suppressWarnings(propose_roles(d, detect = FALSE)) + r <- set_role(r, "gen", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + expect_warning( + ctl <- controls(d, r, control_spec(on = "gen", target = "y", effect = c(a = 0.4, b = -0.2), seed = 1L)), + class = "masque_control_shape_rounded" + ) + expect_identical(ctl@truth$shift_realised[ctl@truth$level == "a"], -ctl@truth$shift_realised[ctl@truth$level == "b"]) + expect_no_warning( + controls(d, r, control_spec(on = "gen", target = "y", effect = c(a = 2, b = -1), seed = 1L)), + class = "masque_control_shape_rounded" + ) +}) diff --git a/tests/testthat/test-controls-negative.R b/tests/testthat/test-controls-negative.R index 5fdf5a6..cfbcd80 100644 --- a/tests/testthat/test-controls-negative.R +++ b/tests/testthat/test-controls-negative.R @@ -69,7 +69,8 @@ test_that("on constant in most strata warns", { r <- set_role(r, "y", role = "outcome", action = "scramble") spec <- control_spec(on = "trt", strata = "site", seed = 1L) expect_warning( - ctl <- controls(df, r, spec), class = "masque_control_strata_weak" + expect_warning(ctl <- controls(df, r, spec), class = "masque_control_strata_weak"), + class = "masque_control_unit_small" ) expect_identical(ctl@manifest$strata_constant_share[1L], 0.75) }) diff --git a/tests/testthat/test-controls-owner-key.R b/tests/testthat/test-controls-owner-key.R new file mode 100644 index 0000000..f341843 --- /dev/null +++ b/tests/testthat/test-controls-owner-key.R @@ -0,0 +1,103 @@ +# The owner's draw: a 256-bit key, used whole, and no integer to search. + +owner_fixture <- function() { + set.seed(5) + d <- expand.grid(gen = paste0("g", 1:6), rep = paste0("r", 1:6), stringsAsFactors = FALSE) + d$y <- stats::rnorm(36L) + r <- suppressWarnings(propose_roles(d, detect = FALSE)) + r <- set_role(r, "rep", role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + list(d = d, r = r, spec = control_spec(on = "gen", strata = "rep", n = 3L, seed = 99L)) +} + +test_that("the owner key is 256 bits from the system source", { + key <- masque:::.new_owner_key() + expect_match(key, "^[0-9a-f]{64}$") + expect_false(identical(key, masque:::.new_owner_key())) + set.seed(1) + a <- masque:::.new_owner_key() + set.seed(1) + expect_false(identical(a, masque:::.new_owner_key())) +}) + +test_that("the key is the system's bytes and nothing else", { + testthat::local_mocked_bindings(.system_entropy = function() as.raw(1:32), .package = "masque") + expect_identical(masque:::.new_owner_key(), paste(as.character(as.raw(1:32)), collapse = "")) +}) + +test_that("a recipe draw is the whole digest and holds no integer seed", { + draw <- masque:::.control_source_draw(strrep("ab", 32L), 99L, NULL) + expect_null(draw$seed) + expect_match(draw$key, "^[0-9a-f]{64}$") + f <- owner_fixture() + m <- mask(f$d, masque_plan(f$r, controls = f$spec), seed = 1L, quiet = TRUE) + own <- controls(f$d, clone_plan(m), recipe = recipe(m)) + expect_true(all(is.na(own@seeds))) + expect_true(all(is.na(own@manifest$seed))) + # A key differing in its last digit gives other permutations. + other <- own + last <- substr(own@key, 64L, 64L) + other@key <- paste0(substr(own@key, 1L, 63L), if (last == "0") "1" else "0") + expect_false(identical( + control_table(own, "negative", 1L)$gen, control_table(other, "negative", 1L)$gen + )) +}) + +test_that("a keyed table ignores the R stream and leaves it alone", { + f <- owner_fixture() + m <- mask(f$d, masque_plan(f$r, controls = f$spec), seed = 1L, quiet = TRUE) + own <- controls(f$d, clone_plan(m), recipe = recipe(m)) + set.seed(1) + a <- control_table(own, "negative", 2L) + before <- .Random.seed + set.seed(2) + b <- control_table(own, "negative", 2L) + expect_identical(a$gen, b$gen) + set.seed(1) + invisible(control_table(own, "negative", 2L)) + expect_identical(.Random.seed, before) + expect_identical(sort(a$gen), sort(f$d$gen)) + expect_true(all(tapply(a$gen, a$rep, function(v) length(unique(v))) == 6L)) +}) + +test_that("keyed permutations are uniform", { + draws <- vapply(seq_len(6000L), function(i) { + paste(masque:::.keyed_perm(paste("k", i), 3L), collapse = "") + }, character(1L)) + counts <- table(draws) + expect_length(counts, 6L) + expect_gt(stats::chisq.test(counts)$p.value, 1e-4) + expect_identical(sort(masque:::.keyed_perm("k", 50L)), 1:50) + expect_identical(masque:::.keyed_perm("k", 1L), 1L) +}) + +test_that("with no system source the mask warns and controls ask for a seed", { + f <- owner_fixture() + testthat::local_mocked_bindings(.system_entropy = function() NULL, .package = "masque") + expect_identical(masque:::.new_owner_key(), character()) + expect_warning( + m <- mask(f$d, masque_plan(f$r, controls = f$spec), seed = 1L, quiet = TRUE), + class = "masque_owner_key_unavailable" + ) + expect_error( + controls(f$d, clone_plan(m), recipe = recipe(m)), class = "masque_control_seed_refusal" + ) + own <- controls(f$d, clone_plan(m), recipe = recipe(m), seed = 7L) + expect_false(anyNA(own@seeds)) +}) + +test_that("an explicit integer seed keeps its stream", { + f <- owner_fixture() + ctl <- controls(f$d, f$r, f$spec, seed = 7L) + seeds <- as.integer(masque:::.sub_seeds(7L, 4L)) + expect_identical(ctl@seeds, seeds) + out <- withr::with_seed(seeds[2L], { + perm <- integer(36L) + for (g in split(seq_len(36L), masque:::.unit_id(f$d, "rep"))) { + perm[g] <- g[sample.int(length(g))] + } + f$d$gen[perm] + }) + expect_identical(control_table(ctl, "negative", 1L)$gen, out) +}) diff --git a/tests/testthat/test-integrity-values.R b/tests/testthat/test-integrity-values.R new file mode 100644 index 0000000..828feab --- /dev/null +++ b/tests/testthat/test-integrity-values.R @@ -0,0 +1,83 @@ +# The recipe binds the table's values as well as its NA mask. + +integrity_fixture <- function() { + set.seed(8) + d <- data.frame( + gen = rep(paste0("G", 1:6), 6L), + rep = rep(paste0("R", 1:6), each = 6L), + y = stats::rnorm(36L, 5, 2), + stringsAsFactors = FALSE + ) + r <- suppressWarnings(propose_roles(d, detect = FALSE)) + r <- set_role(r, "rep", role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "keep") + r <- set_role(r, "y", role = "outcome", action = "scramble") + spec <- control_spec(on = "gen", strata = "rep", n = 2L, seed = 5L) + m <- mask(d, masque_plan(r, controls = spec), seed = 1L, quiet = TRUE) + list(d = d, m = m, plan = clone_plan(m), rec = recipe(m)) +} + +test_that("a table with other values and the same holes is refused", { + f <- integrity_fixture() + swapped <- f$d + swapped$y <- rev(swapped$y) + expect_error(controls(swapped, f$plan, recipe = f$rec), class = "masque_integrity_refusal") + expect_error(apply_recipe(swapped, f$rec), class = "masque_integrity_refusal") + labels <- f$d + labels$gen[1:2] <- labels$gen[2:1] + expect_error(controls(labels, f$plan, recipe = f$rec), class = "masque_integrity_refusal") + one <- f$d + one$y[20L] <- one$y[20L] + 1e-3 + expect_error(controls(one, f$plan, recipe = f$rec), class = "masque_integrity_refusal") + expect_no_error(apply_recipe(swapped, f$rec, check_integrity = FALSE)) +}) + +test_that("the table itself passes, through every route it can come back by", { + f <- integrity_fixture() + expect_no_error(controls(f$d, f$plan, recipe = f$rec)) + expect_no_error(controls(data.table::as.data.table(f$d), f$plan, recipe = f$rec)) + as_factor <- f$d + as_factor$gen <- factor(as_factor$gen) + expect_no_error(controls(as_factor, f$plan, recipe = f$rec)) + path <- withr::local_tempfile(fileext = ".csv") + utils::write.csv(f$d, path, row.names = FALSE) + expect_no_error(controls(utils::read.csv(path), f$plan, recipe = f$rec)) + data.table::fwrite(f$d, path) + expect_no_error(controls(data.table::fread(path), f$plan, recipe = f$rec)) +}) + +test_that("non-finite values and whole numbers read back as doubles match", { + d <- data.frame(a = c(1L, 2L, NA, 4L), b = c(1.5, Inf, -Inf, 2), s = as.Date("2026-01-01") + 0:3) + fp <- masque:::.content_fingerprint(d) + back <- d + back$a <- as.double(back$a) + back$s <- as.character(back$s) + expect_true(masque:::.content_matches(fp, masque:::.content_fingerprint(back))) + moved <- d + moved$b <- d$b[c(1L, 3L, 2L, 4L)] + expect_false(masque:::.content_matches(fp, masque:::.content_fingerprint(moved))) +}) + +test_that("a recipe without the value fingerprint still applies", { + f <- integrity_fixture() + rec <- f$rec + rec@content_fp <- list() + swapped <- f$d + swapped$y <- rev(swapped$y) + expect_no_error(apply_recipe(swapped, rec)) +}) + +test_that("the fingerprint stays with the owner", { + f <- integrity_fixture() + r <- set_role(roles(f$m), "gen", role = "treatment", action = "alias") + m <- mask(f$d, masque_plan(r, controls = control_spec(on = "gen", strata = "rep", n = 2L, seed = 5L)), + mode = "collaborate", seed = 1L, quiet = TRUE) + f$rec <- recipe(m) + expect_gt(length(f$rec@content_fp$sums), 0L) + out <- file.path(withr::local_tempdir(), "pack") + handover(m, out) + shipped <- unlist(lapply(list.files(out, full.names = TRUE), readLines)) + value <- format(f$rec@content_fp$sums$y$value, digits = 8L) + expect_false(any(grepl(value, shipped, fixed = TRUE))) + expect_false(any(grepl(f$rec@content_fp$text, shipped, fixed = TRUE))) +}) diff --git a/tests/testthat/test-plan.R b/tests/testthat/test-plan.R index 242d93d..343e42d 100644 --- a/tests/testthat/test-plan.R +++ b/tests/testthat/test-plan.R @@ -176,7 +176,7 @@ test_that("the owner's controls take a private seed and the real table", { expect_true(any(grepl("private", own@notes))) # Source plan and clone plan give the same owner's object. own2 <- controls(d, rec@plan, recipe = rec) - expect_identical(own2@seeds, own@seeds) + expect_identical(control_table(own2, "negative", 1L), control_table(own, "negative", 1L)) expect_identical(own2@spec@effect, own@spec@effect) }) @@ -382,7 +382,26 @@ test_that("handover runs the release gate", { expect_error(handover(m, out), class = "masque_blocked_write") expect_false(dir.exists(out)) expect_warning(handover(m, out, allow_high = TRUE), class = "masque_high_override") - expect_true(any(grepl("HIGH", readLines(file.path(out, "README.txt"))))) + note <- readLines(file.path(out, "README.txt")) + expect_true(any(grepl("HIGH", note) & grepl(.audit_high_cols(m)[1L], note, fixed = TRUE))) + expect_error(handover(m, tempfile()), "named in README.txt") +}) + +test_that("the override note uses the clone's column names", { + d <- .fx_plan_table() + r <- propose_roles(d, mode = "collaborate", detect = FALSE) + r <- set_role(r, c("site", "rep", "block"), role = "design", action = "keep") + r <- set_role(r, "gen", role = "treatment", action = "keep") + r <- set_role(r, "yield", role = "outcome", action = "scramble") + m <- suppressWarnings(mask(d, r, mode = "collaborate", seed = 1L, quiet = TRUE, alias_names = TRUE)) + expect_true("gen" %in% .audit_high_cols(m)) + out <- file.path(withr::local_tempdir(), "pack") + suppressWarnings(handover(m, out, allow_high = TRUE)) + note <- readLines(file.path(out, "README.txt")) + line <- note[grepl("HIGH", note)] + expect_length(line, 1L) + expect_false(grepl("\\bgen\\b", line)) + expect_true(grepl(recipe(m)@column_name_map[["gen"]], line, fixed = TRUE)) }) # Faults found in review before the 0.17.0 tag -------------------------------- @@ -612,9 +631,8 @@ test_that("no value the analyst holds rebuilds the owner's seeds", { m <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), seed = s, quiet = TRUE) own <- controls(d, clone_plan(m), recipe = recipe(m)) - guess <- (as.numeric(s) + 77 + .CONTROL_SALT) %% .Machine$integer.max - expect_false(identical(own@seeds, as.integer(.sub_seeds(.sub_seeds(guess, 1L), 5L)))) - expect_false(any(own@seeds %in% as.integer(.sub_seeds(77L, 5L)))) + expect_true(all(is.na(own@seeds))) + expect_match(own@key, "^[0-9a-f]{64}$") } # One seed, two recipes: the owner's key differs, the clone does not. a <- mask(d, masque_plan(.fx_plan_roles(d), controls = .fx_spec()), seed = 1L, quiet = TRUE) @@ -624,7 +642,7 @@ test_that("no value the analyst holds rebuilds the owner's seeds", { # The same recipe regenerates the owner's controls. o1 <- controls(d, clone_plan(a), recipe = recipe(a)) o2 <- controls(d, clone_plan(a), recipe = recipe(a)) - expect_identical(o1@seeds, o2@seeds) + expect_identical(control_table(o1, "negative", 1L), control_table(o2, "negative", 1L)) # The key shows in no display route. key <- recipe(a)@owner_key shown <- c(utils::capture.output(print(recipe(a))), utils::capture.output(utils::str(recipe(a)))) diff --git a/tests/testthat/test-safety-gate.R b/tests/testthat/test-safety-gate.R index f10b5f8..6b87881 100644 --- a/tests/testthat/test-safety-gate.R +++ b/tests/testthat/test-safety-gate.R @@ -226,3 +226,15 @@ test_that("a joint HIGH finding blocks write_set() and names the check", { ) expect_false(dir.exists(dir)) }) + +test_that("write_set's refusal says the override is not on the recipe", { + set.seed(6) + a <- data.frame(id = 1:20, y = stats::rnorm(20L)) + r <- suppressWarnings(propose_roles(a, detect = FALSE)) + r <- set_role(r, "y", role = "outcome", action = "keep") + r <- set_role(r, "id", role = "id", action = "keep") + m <- suppressWarnings(mask_set(list(t1 = a), list(t1 = r), mode = "collaborate", seed = 1L, + allow_kept_outcome = TRUE, quiet = TRUE)) + skip_if_not(length(.audit_high_cols(m)) > 0L) + expect_error(write_set(m, file.path(tempdir(), "no-write")), "keep your own record") +}) diff --git a/vignettes/controls.Rmd b/vignettes/controls.Rmd index 9143a95..ffcf4bd 100644 --- a/vignettes/controls.Rmd +++ b/vignettes/controls.Rmd @@ -158,11 +158,13 @@ all.equal(as.numeric(recovered), planted) The owner runs the pipeline the analyst handed back on the original table, and on controls derived from the original with the same spec. One thing -changes: the permutations are drawn from a private seed, given as `seed`, -which overrides the spec's seed. The planted vector still comes from the -spec's seed, so the truth is the same on both sides. A public seed would -let anyone who knows the design, which the clone shares, rebuild every -permutation and read the original outcome out of per-replicate results. +changes: the permutations are drawn privately. Here that is an integer given +as `seed`, which overrides the spec's seed; the plan route below uses a 256-bit +key instead. The planted vector still comes from the spec's seed, so the truth +is the same on both sides. A public seed would let anyone who knows the design, +which the clone shares, rebuild every permutation and read the original outcome +out of per-replicate results. An integer has at most 31 bits, so it hides the +permutations only while nothing per-replicate is released. ```{r controls-original} ctl_owner <- controls(trial, r, spec, seed = 40371L) @@ -244,8 +246,8 @@ ctl_analyst@spec@strata The owner passes the same plan with the recipe and the real table. Cleaning is replayed, a table that does not match the recipe is refused, a spec changed since -`mask()` is refused, and the seed is derived from the recipe, so the analyst -cannot rebuild these permutations. +`mask()` is refused, and the permutations come from the digest of a key on the +recipe, with no integer seed to search, so the analyst cannot rebuild them. ```{r plan-owner} ctl_owner_plan <- controls(trial, analyst_plan, recipe = recipe(m_plan)) @@ -278,6 +280,12 @@ identical(ctl_owner_plan@seeds, ctl_analyst@seeds) standard deviation within the strata on each side. * Nothing is clipped. A negative shift can take a non-negative outcome below zero; `control_table()` warns when it does. +* A negative control sets the size of a test that uses within-stratum + contrasts, so fit the strata. On this design, with replicate strata, `yield ~ + rep + gen` rejects near 5 percent and `yield ~ gen` rejects almost never, + because the replicate variance stays in its residual. A shared rate on both + sides is agreement between them, and zero on both says nothing about + calibration. * The null collapses the design means of row-level outcomes, so a mixed model sees a different variance structure. Calibrate false positives on the negative control, not on the null. From 5f4616f1069702e01e0df065127eb96ad4816a8b Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Fri, 6 Nov 2026 09:57:20 +1030 Subject: [PATCH 11/16] fix: second review of the pre-tag fixes Integrity tolerance tightened to the bound of a 15-digit file, with a second sum for small values beside a large one, and column types read back from a file now match. The default unit returns to the coarsest column that can reach a 5 percent level, with a warning when several qualify, because a finer unit hid pseudoreplication. Arrangement counts use label symmetry across strata and keep a unit whose first row is missing. The shape warning covers many levels. str() of controls hides the key. --- NEWS.md | 20 +++---- R/audit.R | 2 +- R/check_integrity.R | 60 +++++++++++++-------- R/control_spec.R | 9 ++-- R/controls.R | 26 ++++----- R/plan.R | 24 ++++++--- R/recipe_io.R | 4 +- R/recipe_print.R | 7 +++ inst/WORDLIST | 5 +- man/audit_mask.Rd | 2 +- man/control_spec.Rd | 9 ++-- man/save_recipe.Rd | 4 +- tests/testthat/test-controls-default-unit.R | 35 ++++++++++-- tests/testthat/test-controls-owner-key.R | 2 + tests/testthat/test-integrity-values.R | 42 +++++++++++++++ 15 files changed, 178 insertions(+), 73 deletions(-) diff --git a/NEWS.md b/NEWS.md index d6741b1..ff64c5e 100644 --- a/NEWS.md +++ b/NEWS.md @@ -30,16 +30,16 @@ keeps its 31 bits. Release the false-positive rate, the power and one permutation p-value, and nothing computed from a single replicate. * `apply_recipe()` and `controls()` with a recipe now refuse a table whose values - differ from those the recipe was built on, not only its NA mask. Numbers match - within what a file at 15 digits changes. -* In a plan, the default control unit is the finest design column on which `on` - is constant, not the coarsest, which can be a confounder. A column with too few - units for a 5 percent level is passed over with a warning. -* A warning now marks a unit and strata with too few arrangements for a 5 percent - level (`masque_control_unit_small`), strata holding one level of `on` above 10 - percent rather than 50 (`masque_control_strata_weak`), and a whole-valued - target whose rounding changes the declared shape - (`masque_control_shape_rounded`). + differ from those the recipe was built on, not only its NA mask. A table read + back from a file at 15 digits still matches, as do its column types. +* In a plan, the default control unit passes over a column with too few units for + a 5 percent level. It warns when several columns qualify + (`masque_control_unit_ambiguous`): name the unit `on` was assigned at. +* New warnings mark units and strata with too few arrangements for a 5 percent + level (`masque_control_unit_small`) and a whole-valued target whose rounding + changes the declared shape (`masque_control_shape_rounded`). + `masque_control_strata_weak` now fires when `on` is constant in over 10 percent + of the strata (was 50). * `handover(allow_high = TRUE)` names the overridden findings in `README.txt`. The refusal text now says what each writer records: only `masque()` writes to the recipe. diff --git a/R/audit.R b/R/audit.R index 1c6248e..92011eb 100644 --- a/R/audit.R +++ b/R/audit.R @@ -91,7 +91,7 @@ #' #' The grade of a re-simulated number compares rows. A shift on a design level #' of three or four rows, with within-level noise as large as the signal, can -#' grade low while the correlation of the level means stays above 0.85. A low +#' grade low while the correlation of the level means stays near 0.85. A low #' class means the rows are not copies. It does not mean the level means are #' hidden. #' diff --git a/R/check_integrity.R b/R/check_integrity.R index 00686b8..9bbc493 100644 --- a/R/check_integrity.R +++ b/R/check_integrity.R @@ -52,28 +52,45 @@ invisible(TRUE) } -# What the recipe records of the values. Numbers get one weighted sum per -# column, compared within a tolerance, because a table written at 15 digits and -# read back must still match; the weights make it depend on row order. Anything -# else gets a digest of its text, so a factor matches the character column a -# file gives back. Column order does not enter. +# What the recipe records of the values: two weighted sums per numeric column +# and a digest of the rest. The tolerance covers a file at 15 digits. The sum of +# logs guards small values beside a very large one. .content_fingerprint <- function(df) { cols <- sort(names(df), method = "radix") - numeric_col <- vapply(cols, function(cl) is.numeric(df[[cl]]) && !is.object(df[[cl]]), logical(1L)) - sums <- lapply(cols[numeric_col], function(cl) { - x <- as.double(df[[cl]]) - ok <- is.finite(x) - w <- sin(seq_along(x) * sqrt(2)) - list( - value = sum(x[ok] * w[ok]), - scale = sum(abs(x[ok] * w[ok])), - other = digest::digest(list(which(!ok & !is.na(x)), x[!ok & !is.na(x)]), algo = "sha256") - ) - }) - names(sums) <- cols[numeric_col] - text <- lapply(cols[!numeric_col], function(cl) enc2utf8(as.character(df[[cl]]))) - names(text) <- cols[!numeric_col] - list(sums = sums, text = digest::digest(text, algo = "sha256")) + canon <- lapply(cols, function(cl) .content_canonical(df[[cl]])) + names(canon) <- cols + numeric_col <- vapply(canon, is.numeric, logical(1L)) + list( + sums = lapply(canon[numeric_col], function(x) { + ok <- is.finite(x) + w <- sin(seq_along(x) * sqrt(2)) + list( + value = sum(x[ok] * w[ok]), + scale = sum(abs(x[ok] * w[ok])), + log_value = sum(sign(x[ok]) * log1p(abs(x[ok])) * w[ok]), + log_scale = sum(log1p(abs(x[ok])) * abs(w[ok])), + other = digest::digest(list(which(!ok & !is.na(x)), x[!ok & !is.na(x)]), algo = "sha256") + ) + }), + text = digest::digest(canon[!numeric_col], algo = "sha256") + ) +} + +# One form for a column whatever a file reader made of it: a number, where every +# value reads as one, and otherwise text. Logical values count as 0 and 1. +.content_canonical <- function(x) { + if (is.numeric(x) && !is.object(x) || inherits(x, "integer64")) { + return(as.double(x)) + } + if (is.logical(x)) { + return(as.double(x)) + } + text <- enc2utf8(as.character(x)) + if (all(text[!is.na(text)] %in% c("TRUE", "FALSE"))) { + return(as.double(text == "TRUE")) + } + num <- suppressWarnings(as.numeric(text)) + if (any(is.na(num) & !is.na(text))) text else num } .content_matches <- function(recorded, actual) { @@ -85,6 +102,7 @@ a <- recorded$sums[[cl]] b <- actual$sums[[cl]] identical(a$other, b$other) && - abs(a$value - b$value) <= 1e-8 * max(a$scale, b$scale) + abs(a$value - b$value) <= 1e-14 * max(a$scale, b$scale) && + abs(a$log_value - b$log_value) <= 1e-14 * max(a$log_scale, b$log_scale) }, logical(1L))) } diff --git a/R/control_spec.R b/R/control_spec.R index 0705de5..d06fddc 100644 --- a/R/control_spec.R +++ b/R/control_spec.R @@ -70,10 +70,11 @@ masque_control_spec <- S7::new_class( #' for no stratification. Every unit of `on` must lie within one stratum. #' Prefer complete replicates to incomplete blocks; see [controls()]. #' @param unit Design columns within which `on` is constant. `on` is then -#' permuted across those units. `NULL` (the default) permutes rows, and in a -#' plan picks the finest design column on which `on` is constant, if it has -#' enough units for a 5 percent level. In the plan's default a coarse column -#' is not preferred, since it can be a confounder of `on`. +#' permuted across those units, so name the unit `on` was assigned at. +#' `NULL` (the default) permutes rows, and in a plan picks the coarsest design +#' column on which `on` is constant and which has enough units for a 5 percent +#' level, with a warning (class `masque_control_unit_ambiguous`) when several +#' qualify. #' @param grain A named list giving, for other analysed columns constant #' within a design unit, the unit columns: `list(rain = "trial")`. Such a #' column is permuted across its units in the null control, so it stays diff --git a/R/controls.R b/R/controls.R index c28388e..122f3c3 100644 --- a/R/controls.R +++ b/R/controls.R @@ -583,9 +583,8 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) shift_raw <- unname(effect) * sd_y shift_realised <- if (whole) round(shift_raw) else shift_raw vanished <- whole & effect != 0 & shift_realised == 0 - if (whole && !any(vanished) && any(shift_raw != 0) && max(abs( - shift_realised / sum(abs(shift_realised)) - shift_raw / sum(abs(shift_raw)) - )) > 0.1) { + ratio <- (shift_realised / shift_raw)[shift_raw != 0] + if (whole && !any(vanished) && length(ratio) && diff(range(ratio)) > 0.1 * mean(abs(ratio))) { warning(warningCondition( sprintf( "On the whole-valued target %s, rounding changes the shape of the declared effect. `shift_realised` in `@truth` holds the effect that is in the table.", @@ -645,18 +644,15 @@ control_table <- function(ctl, kind = c("null", "negative", "positive"), i = 1L) # Permutation --------------------------------------------------------------- # Can a permutation of `on` across units within strata reach a 5 percent level? -# True when it has at least 20 arrangements; swapping the labels of equal-sized -# levels with no strata gives the same partition, so those count once. +# True when it has at least 20 arrangements. Swapping the labels of levels with +# the same count in every stratum gives the same partition, so those count once. .level_attainable <- function(unit_id, on_val, strata_id) { - keep <- !is.na(on_val) & !duplicated(unit_id) - val <- as.character(on_val[keep]) - log_n <- sum(vapply(split(val, strata_id[keep]), function(v) { - lfactorial(length(v)) - sum(lfactorial(table(v))) - }, numeric(1L))) - if (length(unique(strata_id[keep])) == 1L) { - log_n <- log_n - sum(lfactorial(table(table(val)))) - } - log_n >= log(20) - 1e-8 + keep <- !is.na(on_val) + keep[keep] <- !duplicated(unit_id[keep]) + counts <- table(as.character(on_val[keep]), strata_id[keep]) + log_n <- sum(lfactorial(colSums(counts))) - sum(lfactorial(counts)) + same_counts <- apply(counts, 1L, paste, collapse = " ") + log_n - sum(lfactorial(table(same_counts))) >= log(20) - 1e-8 } # One permutation of 1:n by Fisher-Yates, its uniforms taken from SHA-256 of @@ -847,7 +843,7 @@ local(S7::method(print, masque_controls) <- function(x, ...) { "Datasets: 1 null, %d negative, %d positive", spec@n, if (is.null(spec@target)) 0L else spec@n ), - "*" = "Seeds: present" + "*" = if (length(x@key)) "Draw: a private key" else "Seeds: present" )) if (nrow(x@truth)) { cli::cli_h2("Planted truth (positive minus negative, per level of on)") diff --git a/R/plan.R b/R/plan.R index e9cc877..92c84bc 100644 --- a/R/plan.R +++ b/R/plan.R @@ -323,11 +323,9 @@ local(S7::method(print, .plan_class) <- function(x, ...) { stats::setNames(unname(draw[clone]), levels) } -# Finest design column, then pair of columns, within which `on` is constant. A -# coarse column of this kind can be a confounder. A row alone is constant by -# construction, so a quarter of the rows must share a unit for the constancy to -# mean anything. A column whose units cannot reach a 5 percent level is passed -# over, and the unit falls to the row with a warning when nothing else remains. +# Coarsest design column, then pair, within which `on` is constant and whose +# units can reach a 5 percent level. A row alone is constant by construction, so +# a quarter of the rows must share a unit for the constancy to mean anything. .default_control_unit <- function(df, on, design) { n <- nrow(df) too_few <- character() @@ -350,7 +348,19 @@ local(S7::method(print, .plan_class) <- function(x, ...) { max(id) }, integer(1L)) if (any(!is.na(n_units))) { - return(pick[[which.max(n_units)]]) + chosen <- pick[[which.min(n_units)]] + if (sum(!is.na(n_units)) > 1L) { + warning(warningCondition( + sprintf( + "`%s` is constant within several design columns (%s). The default unit is %s, the coarsest. Name the unit `%s` was assigned at in `unit`: a finer one hides pseudoreplication in the negative controls.", + on, + paste(vapply(pick[!is.na(n_units)], paste, character(1L), collapse = " + "), collapse = ", "), + paste(chosen, collapse = " + "), on + ), + class = "masque_control_unit_ambiguous" + )) + } + return(chosen) } } if (length(too_few)) { @@ -586,7 +596,7 @@ clone_plan <- function(m) { }, seed = draw$seed, key = draw$key, - notes = "Source side: the seed is private. Release aggregates only." + notes = "Source side: the draw is private. Release the false-positive rate, the power and one permutation p-value, and nothing from a single replicate." ) } diff --git a/R/recipe_io.R b/R/recipe_io.R index 573ca75..0e7a2b3 100644 --- a/R/recipe_io.R +++ b/R/recipe_io.R @@ -3,8 +3,8 @@ #' Writes the recipe to a single `.rds` file. The default is #' **runtime-minimal**: #' no simulator state (copula covariance, raw margins) is written, only the -#' translation maps, factor metadata, storage classes, integrity fingerprint, -#' and warnings. This keeps the saved artefact small and reduces the +#' translation maps, factor metadata, storage classes, integrity fingerprints, +#' the key behind the owner's controls, and warnings. This keeps the saved artefact small and reduces the #' information that would leak if the recipe file alone were shared. #' #' `include_simulator = TRUE` is accepted and has no effect: the recipe diff --git a/R/recipe_print.R b/R/recipe_print.R index 19afc10..57957c4 100644 --- a/R/recipe_print.R +++ b/R/recipe_print.R @@ -206,6 +206,13 @@ local({ str <- utils::str S7::method(str, masque_recipe) <- .str_redacted S7::method(str, masque_obj) <- .str_redacted + S7::method(str, masque_controls) <- function(object, ...) { + cat(sprintf( + " %d rows, %d replicates; details not shown. See print().\n", + nrow(object@table), object@spec@n + )) + invisible(NULL) + } }) diff --git a/inst/WORDLIST b/inst/WORDLIST index a23a21d..7aaff98 100644 --- a/inst/WORDLIST +++ b/inst/WORDLIST @@ -16,6 +16,7 @@ CMD Codecov comparator's conductoR +confounder covr CRD cryptographic @@ -23,8 +24,8 @@ dataentry dev difftime Doherty -donut Donut +donut errored et genotype's @@ -59,8 +60,8 @@ PII pkgdown POSIXct POSIXlt -pre Pre +pre RCBD README regexes diff --git a/man/audit_mask.Rd b/man/audit_mask.Rd index 9307aa2..dedc7c4 100644 --- a/man/audit_mask.Rd +++ b/man/audit_mask.Rd @@ -114,7 +114,7 @@ a note saying the clone does not keep that grain. The grade of a re-simulated number compares rows. A shift on a design level of three or four rows, with within-level noise as large as the signal, can -grade low while the correlation of the level means stays above 0.85. A low +grade low while the correlation of the level means stays near 0.85. A low class means the rows are not copies. It does not mean the level means are hidden. diff --git a/man/control_spec.Rd b/man/control_spec.Rd index 73d5042..7d56295 100644 --- a/man/control_spec.Rd +++ b/man/control_spec.Rd @@ -43,10 +43,11 @@ for no stratification. Every unit of \code{on} must lie within one stratum. Prefer complete replicates to incomplete blocks; see \code{\link[=controls]{controls()}}.} \item{unit}{Design columns within which \code{on} is constant. \code{on} is then -permuted across those units. \code{NULL} (the default) permutes rows, and in a -plan picks the finest design column on which \code{on} is constant, if it has -enough units for a 5 percent level. In the plan's default a coarse column -is not preferred, since it can be a confounder of \code{on}.} +permuted across those units, so name the unit \code{on} was assigned at. +\code{NULL} (the default) permutes rows, and in a plan picks the coarsest design +column on which \code{on} is constant and which has enough units for a 5 percent +level, with a warning (class \code{masque_control_unit_ambiguous}) when several +qualify.} \item{grain}{A named list giving, for other analysed columns constant within a design unit, the unit columns: \code{list(rain = "trial")}. Such a diff --git a/man/save_recipe.Rd b/man/save_recipe.Rd index be23f7d..52f5788 100644 --- a/man/save_recipe.Rd +++ b/man/save_recipe.Rd @@ -23,8 +23,8 @@ written without simulator state.} Writes the recipe to a single \code{.rds} file. The default is \strong{runtime-minimal}: no simulator state (copula covariance, raw margins) is written, only the -translation maps, factor metadata, storage classes, integrity fingerprint, -and warnings. This keeps the saved artefact small and reduces the +translation maps, factor metadata, storage classes, integrity fingerprints, +the key behind the owner's controls, and warnings. This keeps the saved artefact small and reduces the information that would leak if the recipe file alone were shared. } \details{ diff --git a/tests/testthat/test-controls-default-unit.R b/tests/testthat/test-controls-default-unit.R index c793ad4..83f0e3c 100644 --- a/tests/testthat/test-controls-default-unit.R +++ b/tests/testthat/test-controls-default-unit.R @@ -20,7 +20,7 @@ confounded <- function(tech = TRUE) { list(d = d, r = r) } -test_that("the default unit is the finest legal column, not the confounder", { +test_that("a confounder too coarse for a 5 percent level is not the default", { f <- confounded() spec <- control_spec(on = "gen", target = "y", effect = list(sd = 0.5), n = 20L, seed = 3L) m <- mask(f$d, masque_plan(f$r, controls = spec), seed = 4L, quiet = TRUE) @@ -39,14 +39,20 @@ test_that("the default unit is the finest legal column, not the confounder", { ) }) -test_that("between two legal columns the finer one is chosen", { +test_that("between two legal columns the coarser is chosen, with a warning", { d <- data.frame( field = rep(paste0("f", 1:8), each = 8L), tech = rep(paste0("t", 1:16), each = 4L), gen = rep(paste0("G", 1:4), each = 16L) ) - expect_identical(masque:::.default_control_unit(d, "gen", c("field", "tech")), "tech") - expect_identical(masque:::.default_control_unit(d, "gen", c("tech", "field")), "tech") + expect_warning( + u1 <- masque:::.default_control_unit(d, "gen", c("field", "tech")), + class = "masque_control_unit_ambiguous" + ) + expect_identical(u1, "field") + expect_identical( + suppressWarnings(masque:::.default_control_unit(d, "gen", c("tech", "field"))), "field" + ) }) test_that("a column with too few units is passed over, with a warning", { @@ -77,6 +83,19 @@ test_that("arrangements are counted within strata and across equal labels", { expect_true(masque:::.level_attainable(1:7, c(rep("a", 3L), rep("b", 4L)), rep(1L, 7L))) }) +test_that("label symmetry counts across strata, and a unit's NA row does not drop it", { + # Paired, five blocks: 32 arrangements, 16 partitions. + pair <- rep(c("a", "b"), 5L) + expect_false(masque:::.level_attainable(1:10, pair, rep(1:5, each = 2L))) + # A randomised block of three treatments in two blocks: 36 arrangements, 6 partitions. + expect_false(masque:::.level_attainable(1:6, rep(c("a", "b", "c"), 2L), rep(1:2, each = 3L))) + # Eight units whose first rows hold NA are still eight units. + unit <- rep(1:8, each = 2L) + on <- rep(c("a", "a", "b", "b", "c", "c", "d", "d"), each = 2L) + on[seq(1L, 15L, by = 2L)] <- NA + expect_true(masque:::.level_attainable(unit, on, rep(1L, 16L))) +}) + test_that("strata holding one level warn from a tenth of the strata", { d <- data.frame( site = rep(paste0("S", 1:5), each = 8L), @@ -115,6 +134,14 @@ test_that("rounding that changes the declared shape warns", { class = "masque_control_shape_rounded" ) expect_identical(ctl@truth$shift_realised[ctl@truth$level == "a"], -ctl@truth$shift_realised[ctl@truth$level == "b"]) + many <- data.frame(gen = rep(paste0("G", 1:24), 3L), y = as.integer(rpois(72L, 10))) + rm <- suppressWarnings(propose_roles(many, detect = FALSE)) + rm <- set_role(rm, "gen", role = "treatment", action = "keep") + rm <- set_role(rm, "y", role = "outcome", action = "scramble") + expect_warning( + controls(many, rm, control_spec(on = "gen", target = "y", effect = stats::setNames(rep(c(0.4, -0.2), 12L), paste0("G", 1:24)), seed = 1L)), + class = "masque_control_shape_rounded" + ) expect_no_warning( controls(d, r, control_spec(on = "gen", target = "y", effect = c(a = 2, b = -1), seed = 1L)), class = "masque_control_shape_rounded" diff --git a/tests/testthat/test-controls-owner-key.R b/tests/testthat/test-controls-owner-key.R index f341843..75a3ec8 100644 --- a/tests/testthat/test-controls-owner-key.R +++ b/tests/testthat/test-controls-owner-key.R @@ -35,6 +35,8 @@ test_that("a recipe draw is the whole digest and holds no integer seed", { own <- controls(f$d, clone_plan(m), recipe = recipe(m)) expect_true(all(is.na(own@seeds))) expect_true(all(is.na(own@manifest$seed))) + shown <- c(utils::capture.output(print(own)), utils::capture.output(utils::str(own))) + expect_false(any(grepl(own@key, shown, fixed = TRUE))) # A key differing in its last digit gives other permutations. other <- own last <- substr(own@key, 64L, 64L) diff --git a/tests/testthat/test-integrity-values.R b/tests/testthat/test-integrity-values.R index 828feab..089ab34 100644 --- a/tests/testthat/test-integrity-values.R +++ b/tests/testthat/test-integrity-values.R @@ -58,6 +58,48 @@ test_that("non-finite values and whole numbers read back as doubles match", { expect_false(masque:::.content_matches(fp, masque:::.content_fingerprint(moved))) }) +test_that("a column's type on the way back from a file does not change the match", { + d <- data.frame( + empty = rep(NA_real_, 12L), lab = factor(1:12), whole = 3e9 + 1:12, + flag = rep(c(TRUE, FALSE), 6L), txt = as.character(1:12) + ) + fp <- masque:::.content_fingerprint(d) + back <- data.frame( + empty = rep(NA, 12L), lab = 1:12, whole = as.character(3e9 + 1:12), + flag = rep(c(1L, 0L), 6L), txt = 1:12 + ) + expect_true(masque:::.content_matches(fp, masque:::.content_fingerprint(back))) + back$lab[1L] <- 99L + expect_false(masque:::.content_matches(fp, masque:::.content_fingerprint(back))) +}) + +test_that("neither a large value nor a large table hides a change", { + set.seed(9) + n <- 1000L + y <- c(1e10, stats::rnorm(n - 1L)) + fp <- masque:::.content_fingerprint(data.frame(y = y)) + y2 <- c(y[1L], rev(y[-1L])) + expect_false(masque:::.content_matches(fp, masque:::.content_fingerprint(data.frame(y = y2)))) + path <- withr::local_tempfile(fileext = ".csv") + data.table::fwrite(data.frame(y = y), path) + expect_true(masque:::.content_matches(fp, masque:::.content_fingerprint(data.table::fread(path)))) + big <- stats::rnorm(1e5, 5000, 1000) + fp2 <- masque:::.content_fingerprint(data.frame(y = big)) + big[17L] <- big[17L] + 50 + expect_false(masque:::.content_matches(fp2, masque:::.content_fingerprint(data.frame(y = big)))) +}) + +test_that("each sum has its own tolerance", { + entry <- function(value = 100, log_value = 10) { + list(sums = list(y = list( + value = value, scale = 1e6, log_value = log_value, log_scale = 1e3, other = "x" + )), text = "t") + } + expect_true(masque:::.content_matches(entry(), entry(100 + 1e-9, 10 + 1e-12))) + expect_false(masque:::.content_matches(entry(), entry(value = 100 + 1e-7))) + expect_false(masque:::.content_matches(entry(), entry(log_value = 10 + 1e-10))) +}) + test_that("a recipe without the value fingerprint still applies", { f <- integrity_fixture() rec <- f$rec From f6a5366f6bd257a9c29aec4d1b34fef1593811fd Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Fri, 6 Nov 2026 09:57:51 +1030 Subject: [PATCH 12/16] test: the key's prefix shows in no display of controls --- tests/testthat/test-controls-owner-key.R | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/testthat/test-controls-owner-key.R b/tests/testthat/test-controls-owner-key.R index 75a3ec8..66fe7c6 100644 --- a/tests/testthat/test-controls-owner-key.R +++ b/tests/testthat/test-controls-owner-key.R @@ -36,7 +36,7 @@ test_that("a recipe draw is the whole digest and holds no integer seed", { expect_true(all(is.na(own@seeds))) expect_true(all(is.na(own@manifest$seed))) shown <- c(utils::capture.output(print(own)), utils::capture.output(utils::str(own))) - expect_false(any(grepl(own@key, shown, fixed = TRUE))) + expect_false(any(grepl(substr(own@key, 1L, 12L), shown, fixed = TRUE))) # A key differing in its last digit gives other permutations. other <- own last <- substr(own@key, 64L, 64L) From 0a7ddf18b8e9df9c188af7fc5cc9416ca863b071 Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Fri, 6 Nov 2026 10:04:03 +1030 Subject: [PATCH 13/16] test: skip handover checks without yaml; str of controls is one line --- tests/testthat/test-controls-owner-key.R | 4 ++-- tests/testthat/test-integrity-values.R | 1 + tests/testthat/test-plan.R | 1 + 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/testthat/test-controls-owner-key.R b/tests/testthat/test-controls-owner-key.R index 66fe7c6..7a24dc6 100644 --- a/tests/testthat/test-controls-owner-key.R +++ b/tests/testthat/test-controls-owner-key.R @@ -35,8 +35,8 @@ test_that("a recipe draw is the whole digest and holds no integer seed", { own <- controls(f$d, clone_plan(m), recipe = recipe(m)) expect_true(all(is.na(own@seeds))) expect_true(all(is.na(own@manifest$seed))) - shown <- c(utils::capture.output(print(own)), utils::capture.output(utils::str(own))) - expect_false(any(grepl(substr(own@key, 1L, 12L), shown, fixed = TRUE))) + expect_false(any(grepl(substr(own@key, 1L, 12L), utils::capture.output(print(own)), fixed = TRUE))) + expect_length(utils::capture.output(utils::str(own)), 1L) # A key differing in its last digit gives other permutations. other <- own last <- substr(own@key, 64L, 64L) diff --git a/tests/testthat/test-integrity-values.R b/tests/testthat/test-integrity-values.R index 089ab34..53c47f7 100644 --- a/tests/testthat/test-integrity-values.R +++ b/tests/testthat/test-integrity-values.R @@ -110,6 +110,7 @@ test_that("a recipe without the value fingerprint still applies", { }) test_that("the fingerprint stays with the owner", { + skip_if_not_installed("yaml") f <- integrity_fixture() r <- set_role(roles(f$m), "gen", role = "treatment", action = "alias") m <- mask(f$d, masque_plan(r, controls = control_spec(on = "gen", strata = "rep", n = 2L, seed = 5L)), diff --git a/tests/testthat/test-plan.R b/tests/testthat/test-plan.R index 343e42d..ce0abd0 100644 --- a/tests/testthat/test-plan.R +++ b/tests/testthat/test-plan.R @@ -388,6 +388,7 @@ test_that("handover runs the release gate", { }) test_that("the override note uses the clone's column names", { + skip_if_not_installed("yaml") d <- .fx_plan_table() r <- propose_roles(d, mode = "collaborate", detect = FALSE) r <- set_role(r, c("site", "rep", "block"), role = "design", action = "keep") From 901852f21032b61576d9fed9e6ff5aefc1861811 Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Fri, 6 Nov 2026 16:41:56 +1030 Subject: [PATCH 14/16] fix: value fingerprint ignores how text is held in memory; recipe size test checks growth --- R/check_integrity.R | 3 +- tests/testthat/test-integrity-values.R | 7 ++++ tests/testthat/test-recipe-io.R | 45 ++++++++++++++------------ 3 files changed, 33 insertions(+), 22 deletions(-) diff --git a/R/check_integrity.R b/R/check_integrity.R index 9bbc493..2604eca 100644 --- a/R/check_integrity.R +++ b/R/check_integrity.R @@ -85,7 +85,8 @@ if (is.logical(x)) { return(as.double(x)) } - text <- enc2utf8(as.character(x)) + # c() drops the growable bit data.table may set, which serialize() would hash. + text <- c(enc2utf8(as.character(x))) if (all(text[!is.na(text)] %in% c("TRUE", "FALSE"))) { return(as.double(text == "TRUE")) } diff --git a/tests/testthat/test-integrity-values.R b/tests/testthat/test-integrity-values.R index 53c47f7..8056a62 100644 --- a/tests/testthat/test-integrity-values.R +++ b/tests/testthat/test-integrity-values.R @@ -124,3 +124,10 @@ test_that("the fingerprint stays with the owner", { expect_false(any(grepl(value, shipped, fixed = TRUE))) expect_false(any(grepl(f$rec@content_fp$text, shipped, fixed = TRUE))) }) + +test_that("the value fingerprint does not depend on how text is held in memory", { + path <- tempfile(fileext = ".csv") + writeLines(c("site,yield", "a,1", "b,2", "c,3"), path) + read <- data.table::fread(path) + expect_identical(.content_fingerprint(read), .content_fingerprint(data.table::copy(read))) +}) diff --git a/tests/testthat/test-recipe-io.R b/tests/testthat/test-recipe-io.R index 151198e..49a37ff 100644 --- a/tests/testthat/test-recipe-io.R +++ b/tests/testthat/test-recipe-io.R @@ -27,27 +27,30 @@ test_that("save_recipe + read_recipe round-trip is identity", { expect_identical(rec@integrity_fp, rec2@integrity_fp) }) -test_that("save_recipe file is much smaller than the raw data (1000x20 fixture)", { - set.seed(0) - df <- data.frame( - Rep = rep(1:4, 250), - block = rep(1:5, each = 200), - Genotype = factor(rep(LETTERS[1:10], 100)), - yield = rnorm(1000) - ) - for (i in 1:16) df[[paste0("cov_", i)]] <- rnorm(1000) - r <- propose_roles(df) - r$role[r$col == "yield"] <- "outcome" - m <- mask(df, r, mode = "collaborate", seed = 1) - - tmp <- tempfile(fileext = ".rds") - save_recipe(recipe(m), tmp) - sz <- file.info(tmp)$size - # Compared with the data under the same serialiser: absolute sizes differ - # across R versions (R-devel is several times larger). - data_rds <- tempfile(fileext = ".rds") - saveRDS(df, data_rds) - expect_lt(sz, file.info(data_rds)$size) +test_that("save_recipe file does not grow with the rows of the data", { + sizes <- function(n) { + set.seed(0) + df <- data.frame( + Rep = rep(1:4, length.out = n), + block = rep(1:5, length.out = n), + Genotype = factor(rep(LETTERS[1:10], length.out = n)), + yield = rnorm(n) + ) + for (i in 1:16) df[[paste0("cov_", i)]] <- rnorm(n) + r <- propose_roles(df) + r$role[r$col == "yield"] <- "outcome" + m <- mask(df, r, mode = "collaborate", seed = 1) + recipe_rds <- tempfile(fileext = ".rds") + save_recipe(recipe(m), recipe_rds) + data_rds <- tempfile(fileext = ".rds") + saveRDS(df, data_rds) + file.info(c(recipe_rds, data_rds))$size + } + small <- sizes(1000L) + large <- sizes(10000L) + # The class definition sets the file's base size, which varies with the R + # version. A recipe holding rows would grow with the data. + expect_lt(large[1L] - small[1L], 0.01 * (large[2L] - small[2L])) }) test_that("save_recipe errors on invalid inputs", { From 7699ca29379f5568622a50d8bdf5f3e84d9d8605 Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Sat, 7 Nov 2026 16:50:28 +1030 Subject: [PATCH 15/16] docs: a whole number can fall one full step outside the observed range --- NEWS.md | 4 ++++ R/synthesise_numeric_jitter.R | 3 ++- tests/testthat/test-synthesise_numeric_jitter.R | 8 ++++++++ vignettes/confidentiality.Rmd | 2 +- 4 files changed, 15 insertions(+), 2 deletions(-) diff --git a/NEWS.md b/NEWS.md index ff64c5e..8feefcc 100644 --- a/NEWS.md +++ b/NEWS.md @@ -46,6 +46,10 @@ * The averaged-row figure in `?jitter_coordinates` is corrected to 0.64 km for 360 rows. * masque needs digest 0.6.21 or later. +* The range statement in `vignette("confidentiality")` is corrected. A whole + number in a collaborate clone can fall one full step outside the observed + range, not half a step as 0.16.0 stated, so a column whose minimum is 0 can + show -1. A conditional clone can go further. ## New features diff --git a/R/synthesise_numeric_jitter.R b/R/synthesise_numeric_jitter.R index 4d64820..37c44b4 100644 --- a/R/synthesise_numeric_jitter.R +++ b/R/synthesise_numeric_jitter.R @@ -8,7 +8,8 @@ #' clone; a value can fall up to half a resolution step outside it. #' Whole-numbered columns (integer class, or a double whose observed values #' are all whole) get stochastic rounding, so the output stays -#' whole-numbered in its original storage class. A column with at most two +#' whole-numbered in its original storage class and can fall one full step +#' outside the range. A column with at most two #' distinct observed values is #' left as drawn: the local copula draw already returns observed values, #' and jitter followed by rounding would flip a share of them. diff --git a/tests/testthat/test-synthesise_numeric_jitter.R b/tests/testthat/test-synthesise_numeric_jitter.R index d60f79b..3a742d7 100644 --- a/tests/testthat/test-synthesise_numeric_jitter.R +++ b/tests/testthat/test-synthesise_numeric_jitter.R @@ -39,6 +39,14 @@ test_that("synthesise_numeric_collaborate stays within half a step of range", { expect_true(all(out >= min(x_obs) - half) && all(out <= max(x_obs) + half)) }) +test_that("a whole-numbered column can reach one full step past the range", { + set.seed(1) + x_obs <- rep(0:20, 10L) + out <- masque:::synthesise_numeric_collaborate(x_obs, rep(0, length(x_obs))) + expect_identical(min(out), -1L) + expect_true(all(out >= -1L)) +}) + test_that("synthesise_numeric_collaborate preserves integer class", { set.seed(1) x_obs <- sample(1L:20L, 200, replace = TRUE) diff --git a/vignettes/confidentiality.Rmd b/vignettes/confidentiality.Rmd index 01fab03..e5c8275 100644 --- a/vignettes/confidentiality.Rmd +++ b/vignettes/confidentiality.Rmd @@ -72,7 +72,7 @@ per-column work: | Date / time columns | row-permuted with class preserved | row-permuted with class preserved | | Identifiers (`id`) | kept | dropped | | Free text (`text`: a character column in which more than half the values are distinct) | kept | dropped | -| Numeric synthesis | Gaussian copula with empirical-quantile marginals (may emit observed values) | the same, plus within-resolution jitter, with whole numbers stochastically rounded. A value can fall up to half a measurement step outside the observed range | +| Numeric synthesis | Gaussian copula with empirical-quantile marginals (may emit observed values) | the same, plus within-resolution jitter, with whole numbers stochastically rounded. A value can fall up to half a measurement step outside the observed range, and a whole number up to one full step, so a column whose minimum is 0 can show -1. A conditional clone can go further, because it adds level effects after the draw | | Columns constant within a design unit | one value per unit, drawn from the real unit values | one value per unit, jittered, or row by row when there are fewer than five units | | NA mask | preserved cell-by-cell | preserved cell-by-cell | | `audit_mask()` | on demand | automatic at `mask()` time | From e024b7e3be74fa2491c69ea27404208ab2468f3c Mon Sep 17 00:00:00 2001 From: Max Moldovan Date: Sat, 7 Nov 2026 20:41:27 +1030 Subject: [PATCH 16/16] docs: controls describe a sham assignment, not a removed effect The negative control leaves any real effect of the permuted column in the outcome. Each released reading is classed by what it estimates, the null is a smoke test that a leak already in the table passes, and the planted size is read against the residual of the pipeline's own model. --- NEWS.md | 6 +- R/control_spec.R | 15 +++-- R/controls.R | 26 +++++--- R/plan.R | 2 +- README.md | 4 +- man/control_spec.Rd | 15 +++-- man/controls.Rd | 26 +++++--- vignettes/controls.Rmd | 145 +++++++++++++++++++++++++++-------------- 8 files changed, 153 insertions(+), 86 deletions(-) diff --git a/NEWS.md b/NEWS.md index 8feefcc..750ff95 100644 --- a/NEWS.md +++ b/NEWS.md @@ -27,8 +27,8 @@ plan and its recipe, the owner's permutations now come from SHA-256 of a 256-bit key from the system's random source, because a derived integer seed could be searched against a per-replicate statistic. A private integer `seed` - keeps its 31 bits. Release the false-positive rate, the power and one - permutation p-value, and nothing computed from a single replicate. + keeps its 31 bits. Release nothing computed from a single replicate, and check + the permutation p-value as the result of the test on the real table. * `apply_recipe()` and `controls()` with a recipe now refuse a table whose values differ from those the recipe was built on, not only its NA mask. A table read back from a file at 15 digits still matches, as do its column types. @@ -68,7 +68,7 @@ design columns alone: the clone keeps design means and block variance and carries no treatment effect. It also takes column names, an explicit partition that runs through the same ladder. -* Known-truth controls. `control_spec()` declares what a control removes and +* Known-truth controls. `control_spec()` declares what a control permutes and plants; `controls()` derives a null, `n` negative and `n` positive control datasets from any table with a role table, by restricted permutation at the declared unit within declared design strata and a per-level shift in diff --git a/R/control_spec.R b/R/control_spec.R index d06fddc..af94102 100644 --- a/R/control_spec.R +++ b/R/control_spec.R @@ -34,8 +34,8 @@ masque_control_spec <- S7::new_class( #' Declare the controls to derive from a table #' -#' A control spec says what [controls()] removes and plants: the column -#' whose effect the negative control removes (`on`), the outcome the +#' A control spec says what [controls()] permutes and plants: the column +#' the negative control reassigns (`on`), the outcome the #' positive control plants an effect on (`target`), the planted effect per #' level of `on` in outcome-SD units (`effect`), the design strata within #' which `on` is permuted and the unit it is permuted at, the other @@ -47,9 +47,10 @@ masque_control_spec <- S7::new_class( #' needs `on`, which defaults to the single treatment column of the table. #' The positive control needs `target` and `effect` together. #' -#' @param on The name of the column whose association with the rest of the -#' table the negative control removes, by permuting it within `strata` at -#' its `unit`. Must be a treatment or covariate column, categorical or +#' @param on The name of the column the negative control reassigns, by +#' permuting it within `strata` at its `unit`. The outcome is not changed, +#' so any real effect of `on` stays in it as unexplained variation. Must be +#' a treatment or covariate column, categorical or #' whole-valued. `NULL` (the default) means the single treatment column of #' the table; a table with several or no treatment columns needs `on` #' named. @@ -58,7 +59,9 @@ masque_control_spec <- S7::new_class( #' control. A logical, factor, date, event or two-valued column is #' refused when the spec meets the table. #' @param effect The planted effect per level of `on`, in units of the -#' target's standard deviation in the table at hand: a named numeric +#' target's standard deviation in the table at hand. That standard deviation +#' includes the design variance and any real effect of `on`, so the size is +#' not a multiple of an analysis's residual. Give it as a named numeric #' vector (`c(G01 = 0.5, G02 = -0.5)`; a level not named shifts by zero), #' or `list(sd = s)` to draw one shift per level from a normal #' distribution with standard deviation `s`, under the spec's seed, when diff --git a/R/controls.R b/R/controls.R index 122f3c3..3ea8c6e 100644 --- a/R/controls.R +++ b/R/controls.R @@ -47,11 +47,14 @@ masque_controls <- S7::new_class( #' units and stays constant within each. Design, treatment and kept #' columns, the NA mask and every column's multiset of values are kept. #' No association among analysed columns, or with the treatment, remains. -#' Use it as a smoke test and to expose leakage through analysed columns. +#' Use it as a smoke test: a pipeline that finds signal here builds that +#' signal itself. A leak already in the table, such as a covariate that +#' copies the outcome, is shuffled away with the rest and passes. #' * `negative`: the declared column `on` alone is permuted within the #' declared design strata, at its unit, while the design structure and all -#' other associations are kept. Choose strata that every level of `on` can -#' reach. Complete replicates qualify. Within incomplete blocks each block +#' other associations are kept. The outcome is not changed, so any real +#' effect of `on` stays in it as unexplained variation. Choose strata that +#' every level of `on` can reach. Complete replicates qualify. Within incomplete blocks each block #' keeps its set of levels, so part of the real effect of `on` survives. #' With `n` replicates it gives a permutation null for a whole pipeline: a #' false-positive rate, a permutation p-value. @@ -80,15 +83,20 @@ masque_controls <- S7::new_class( #' a per-replicate statistic can try each value, because the true seed fits the #' statistic exactly and a wrong one does not. #' -#' Control tables built from an original never leave the owner. What leaves is -#' the false-positive rate, the power and one permutation p-value, through the -#' owner's usual output checking. A vector of per-replicate means or contrasts -#' is not an aggregate, under either route. +#' Control tables built from an original never leave the owner. What may leave +#' goes through the owner's usual output checking, classed by what it +#' estimates. The false-positive rate says little about the real outcome. The +#' rate at which the positives find the shift depends on the residual variance. +#' The permutation p-value is the result of the analyst's test on the real +#' table. A vector of per-replicate means or contrasts is not an aggregate, +#' under either route, and every further pipeline run on the original is +#' another set of queries. #' #' @section Limits: #' A kept column is structure and is never permuted, so leakage through -#' kept columns, ids or row order passes the null. A planted effect is -#' clean: uniform within a level, with no confounding and no interaction. +#' kept columns, ids or row order passes the null. No control can detect +#' confounding. A planted effect is clean: uniform within a level, with no +#' confounding and no interaction. #' Nothing is clipped, so a shift can take a non-negative outcome below #' zero; [control_table()] warns when it does. The null collapses the #' design means of row-grained outcomes, so false-positive calibration diff --git a/R/plan.R b/R/plan.R index 92c84bc..aff11c7 100644 --- a/R/plan.R +++ b/R/plan.R @@ -596,7 +596,7 @@ clone_plan <- function(m) { }, seed = draw$seed, key = draw$key, - notes = "Source side: the draw is private. Release the false-positive rate, the power and one permutation p-value, and nothing from a single replicate." + notes = "Source side: the draw is private. Nothing from a single replicate leaves. The permutation p-value is the result of the test on the real table, and the rate at which the positives find the shift reflects its residual variance: check both as results." ) } diff --git a/README.md b/README.md index 4fce54a..7148d36 100644 --- a/README.md +++ b/README.md @@ -101,8 +101,8 @@ real data in scientific inference. ### Known-truth controls A pipeline developed on the clone can be checked against datasets in which -the answer is known. A control spec, declared once, names the column whose -effect is removed, the design strata it is permuted within, the outcome an +the answer is known. A control spec, declared once, names the column to +reassign, the design strata it is permuted within, the outcome an effect is planted on and the size of that effect. From it, `controls()` derives a null (every analysed column permuted), negative replicates (one column permuted within strata) and positive replicates (the negative plus diff --git a/man/control_spec.Rd b/man/control_spec.Rd index 7d56295..271232f 100644 --- a/man/control_spec.Rd +++ b/man/control_spec.Rd @@ -16,9 +16,10 @@ control_spec( ) } \arguments{ -\item{on}{The name of the column whose association with the rest of the -table the negative control removes, by permuting it within \code{strata} at -its \code{unit}. Must be a treatment or covariate column, categorical or +\item{on}{The name of the column the negative control reassigns, by +permuting it within \code{strata} at its \code{unit}. The outcome is not changed, +so any real effect of \code{on} stays in it as unexplained variation. Must be +a treatment or covariate column, categorical or whole-valued. \code{NULL} (the default) means the single treatment column of the table; a table with several or no treatment columns needs \code{on} named.} @@ -29,7 +30,9 @@ control. A logical, factor, date, event or two-valued column is refused when the spec meets the table.} \item{effect}{The planted effect per level of \code{on}, in units of the -target's standard deviation in the table at hand: a named numeric +target's standard deviation in the table at hand. That standard deviation +includes the design variance and any real effect of \code{on}, so the size is +not a multiple of an analysis's residual. Give it as a named numeric vector (\code{c(G01 = 0.5, G02 = -0.5)}; a level not named shifts by zero), or \code{list(sd = s)} to draw one shift per level from a normal distribution with standard deviation \code{s}, under the spec's seed, when @@ -62,8 +65,8 @@ constant within each unit.} A \code{masque_control_spec} object, with a print method. } \description{ -A control spec says what \code{\link[=controls]{controls()}} removes and plants: the column -whose effect the negative control removes (\code{on}), the outcome the +A control spec says what \code{\link[=controls]{controls()}} permutes and plants: the column +the negative control reassigns (\code{on}), the outcome the positive control plants an effect on (\code{target}), the planted effect per level of \code{on} in outcome-SD units (\code{effect}), the design strata within which \code{on} is permuted and the unit it is permuted at, the other diff --git a/man/controls.Rd b/man/controls.Rd index d66376c..15af779 100644 --- a/man/controls.Rd +++ b/man/controls.Rd @@ -52,11 +52,14 @@ non-missing rows; a column declared unit-grained is permuted across its units and stays constant within each. Design, treatment and kept columns, the NA mask and every column's multiset of values are kept. No association among analysed columns, or with the treatment, remains. -Use it as a smoke test and to expose leakage through analysed columns. +Use it as a smoke test: a pipeline that finds signal here builds that +signal itself. A leak already in the table, such as a covariate that +copies the outcome, is shuffled away with the rest and passes. \item \code{negative}: the declared column \code{on} alone is permuted within the declared design strata, at its unit, while the design structure and all -other associations are kept. Choose strata that every level of \code{on} can -reach. Complete replicates qualify. Within incomplete blocks each block +other associations are kept. The outcome is not changed, so any real +effect of \code{on} stays in it as unexplained variation. Choose strata that +every level of \code{on} can reach. Complete replicates qualify. Within incomplete blocks each block keeps its set of levels, so part of the real effect of \code{on} survives. With \code{n} replicates it gives a permutation null for a whole pipeline: a false-positive rate, a permutation p-value. @@ -89,17 +92,22 @@ integer \code{seed} is the weaker route: it has at most 31 bits, and anyone give a per-replicate statistic can try each value, because the true seed fits the statistic exactly and a wrong one does not. -Control tables built from an original never leave the owner. What leaves is -the false-positive rate, the power and one permutation p-value, through the -owner's usual output checking. A vector of per-replicate means or contrasts -is not an aggregate, under either route. +Control tables built from an original never leave the owner. What may leave +goes through the owner's usual output checking, classed by what it +estimates. The false-positive rate says little about the real outcome. The +rate at which the positives find the shift depends on the residual variance. +The permutation p-value is the result of the analyst's test on the real +table. A vector of per-replicate means or contrasts is not an aggregate, +under either route, and every further pipeline run on the original is +another set of queries. } \section{Limits}{ A kept column is structure and is never permuted, so leakage through -kept columns, ids or row order passes the null. A planted effect is -clean: uniform within a level, with no confounding and no interaction. +kept columns, ids or row order passes the null. No control can detect +confounding. A planted effect is clean: uniform within a level, with no +confounding and no interaction. Nothing is clipped, so a shift can take a non-negative outcome below zero; \code{\link[=control_table]{control_table()}} warns when it does. The null collapses the design means of row-grained outcomes, so false-positive calibration diff --git a/vignettes/controls.Rmd b/vignettes/controls.Rmd index ffcf4bd..93c1f03 100644 --- a/vignettes/controls.Rmd +++ b/vignettes/controls.Rmd @@ -25,29 +25,37 @@ library(masque) ## Why -A pipeline is an assay. Before an assay is trusted on a sample, it is run on -a blank, on a negative control and on a positive control, and its readings -on those three are what make the reading on the sample interpretable. `masque` gives an analyst the structure of a confidential table without its -values. Controls add the other three: datasets derived from a table in -which the answer is known in advance. - -* The **null** control permutes every analysed column, so nothing is - associated with anything. A pipeline that still finds signal on it has a - leak or a bug. +values. Controls are tables derived from a table in which part of the answer is +fixed by construction, so a pipeline can be run where the right reading is +known. + +* The **null** control permutes every analysed column, so no analysed column + is associated with another or with the treatment. It is a smoke test. A + pipeline that finds strong signal on it has built that signal itself, from the + outcome or from a column the shuffle left alone. A leak that sits in the + table, such as a covariate that copies the outcome, is shuffled away with + everything else, so a quiet null does not clear a pipeline. * The **negative** control permutes one declared column, by default the treatment, within the design strata. Its allocation is random within them - and everything else is as it was. A pipeline that finds it has a false - positive, and over many replicates the rate of those is the pipeline's - false-positive rate on this very structure. -* The **positive** control adds a declared effect to the negative. The - pipeline should find it, and the size it reports can be compared with the - size that was planted. + and everything else is as it was, the outcome included. It is a sham + assignment: the permuted column is unrelated to the outcome, and any real + effect it had stays in the outcome as extra noise. Over many replicates the + share of negatives on which a pipeline rejects is its false-positive rate on + this table under this allocation. On a randomised experiment, the rank of the + real result among them is a randomisation p-value. +* The **positive** control adds a declared shift per level to the negative. + The share of positives on which a pipeline finds it is its sensitivity to a + clean shift of that size on this table. + +In epidemiology a negative control is a variable chosen to share the bias of +the question. These controls are null by permutation and share no bias, so none +of them can show confounding. Nothing is re-simulated. Each control is a permutation of the table it was -built from, with a shift added for the positive, so the design columns, the -treatment allocation, the missing cells and every column's set of values are -exactly those of the input. +built from, with a shift added for the positive. The design columns and the +missing cells are those of the input, and every column keeps its set of values, +apart from the shifted target of a positive. ## A clone and a declaration @@ -117,8 +125,8 @@ genotype_p <- function(d) { On the clone itself the genotype term is not expected to show, because the default clone re-simulates yield without a genotype effect. On the null it must not show. On the negatives it shows at the pipeline's false-positive -rate. On the positives it shows at the pipeline's power against the planted -size. +rate. On the positives it shows at the rate the pipeline finds the planted +shift. ```{r run} p_clone <- genotype_p(clone) @@ -138,13 +146,16 @@ data.frame( ) ``` -The negatives give the pipeline's false-positive rate on this design, and -the positives its power against the planted effect. Both are properties of -the pipeline on this structure, learned without a single real yield. +The negatives give the pipeline's false-positive rate, and the positives the +rate at which it finds the planted shift. Both describe this pipeline on this +table under these permutations, learned without a single real yield. Another +table with the same design can give other rates. -The planted size can be checked as well. The genotype means on a positive +The construction can be checked as well. The genotype means on a positive replicate, less those on its negative replicate, are the planted shifts -exactly, because the two share one permutation. +exactly, because the two share one permutation. This checks the controls, not +the pipeline: raw genotype means recover the shifts by construction, so only an +adjusted estimate can show a fault in the pipeline. ```{r recovery} neg <- control_table(ctl, "negative", 1L) @@ -154,6 +165,19 @@ planted <- ctl@truth$shift_realised[match(names(recovered), ctl@truth$level)] all.equal(as.numeric(recovered), planted) ``` +The planted size is in units of the yield's standard deviation in the table +at hand. That standard deviation holds the block variance and any real +genotype effect, so a size of 0.5 is not half the noise the pipeline sees. +Read it against the residual of the pipeline's own model: + +```{r residual-units} +residual_sd <- function(d) { + blocks <- interaction(d$rep, d$block, drop = TRUE) + summary(stats::lm(yield ~ blocks + gen, data = d))$sigma +} +0.5 * ctl@truth$sd_y[1] / residual_sd(clone) +``` + ## The owner's side: the same spec on the original The owner runs the pipeline the analyst handed back on the original table, @@ -177,15 +201,19 @@ p_pos_owner <- vapply(seq_len(spec@n), function(i) { }, numeric(1)) ``` -What leaves the owner's side is a verdict about the pipeline, through the -owner's usual output checking: the false-positive rate over the negatives, -the power over the positives and, for the real result, a permutation -p-value, the share of negatives whose p-value is at or below the real one. +Each reading is classed by what it estimates before it goes through the owner's +usual output checking. The false-positive rate over the negatives describes the +pipeline under sham assignment. For a test whose null distribution does not +depend on the yields, it says little about them. The rate +at which the positives find the shift depends on the residual variance, so it +tells something about the scale of the real noise. The permutation p-value, the +share of negatives whose p-value is at or below the real one, is the result of +the analyst's test on the real table, and it is checked as a result. ```{r verdict} data.frame( verdict = c( - "false-positive rate (negatives)", "power at the planted effect (positives)", + "false-positive rate (negatives)", "planted shift found (positives)", "permutation p-value of the real result" ), value = c( @@ -195,17 +223,24 @@ data.frame( ) ``` -The two sides read close to each other here, within what 19 replicates can -resolve. They need not. The default clone re-simulates yield without the -block structure, so its residual variance differs from the original's, and -any gap between the readings says what fidelity this pipeline needs from a -clone. +The owner's readings describe the pipeline on the real table, and they are the +ones that answer the analyst. The clone's readings are a rehearsal. Here the +two sides read close to each other, within what 19 replicates can resolve. +They need not. The default clone re-simulates yield without the block +structure, so its residual variance differs from the original's. A gap +between the readings is a statement about the clone. Agreement is not a check +on the pipeline, and a rate of zero on both sides agrees while saying nothing. +The same planted 0.5 is +`r round(0.5 * ctl_owner@truth$sd_y[1] / residual_sd(trial), 2)` residual +standard deviations of the pipeline's model on the original and +`r round(0.5 * ctl@truth$sd_y[1] / residual_sd(clone), 2)` on the clone. The control tables built from the original hold the real values and never leave. Per-replicate results do not leave either: each replicate is one more query against the original, and with the permutations known they are -linear equations in the real outcome. The replicate count in the spec is -the query budget, declared with everything else. +linear equations in the real outcome. The replicate count in the spec bounds +the queries for one pipeline, and every further pipeline or spec run on the +original is another set of them. ## Carrying the spec in a plan @@ -266,21 +301,28 @@ identical(ctl_owner_plan@seeds, ctl_analyst@seeds) * A sequential test depends on the order of its terms. A treatment entered before the blocks carries their variance, and its false-positive rate on this design is well above nominal. The negatives show this, as they should. -* The negative control removes the effect of one column under exchangeable - allocation within the strata. It cannot detect confounding, and on - observational data it is a permutation test with no randomisation - guarantee behind it. +* The negative control reassigns one column under exchangeable allocation + within the strata. The outcome is unchanged, so a large real effect of that + column makes the residual on a negative larger than on the original. It + cannot detect confounding, and on observational data it is a permutation + test with no randomisation guarantee behind it. * A planted effect is clean: the same shift for every plot of a genotype, - no confounding, no interaction. The verdict is sensitivity to a clean - effect of the declared size, not power against the real one. -* A shift is added in units of the outcome's standard deviation in the - table at hand, so the same spec plants the same nominal size on the clone - and on the original while the signal-to-noise ratio may differ. The - `@truth` table records the standard deviation used and the residual - standard deviation within the strata on each side. + no confounding, no interaction. The positives measure sensitivity to a + clean shift of the declared size, not power against the real effect, and a + pipeline that is biased under confounding still finds it. +* A shift is added in units of the outcome's standard deviation in the table + at hand. That standard deviation includes the design variance and any real + effect, so the same spec plants the same nominal size on the clone and on + the original while the shift relative to the residual may differ. The + `@truth` table records the standard deviation used and, as + `sd_within_strata`, the standard deviation left after the stratum means + are removed. That figure still holds the effect of `on` and any variance + within the strata, so it is larger than the residual of a model that fits + them. * Nothing is clipped. A negative shift can take a non-negative outcome - below zero; `control_table()` warns when it does. -* A negative control sets the size of a test that uses within-stratum + below zero; `control_table()` warns when it does. On a count the shift is + added, not a change of rate. +* A negative control estimates the size of a test that uses within-stratum contrasts, so fit the strata. On this design, with replicate strata, `yield ~ rep + gen` rejects near 5 percent and `yield ~ gen` rejects almost never, because the replicate variance stays in its residual. A shared rate on both @@ -289,6 +331,9 @@ identical(ctl_owner_plan@seeds, ctl_analyst@seeds) * The null collapses the design means of row-level outcomes, so a mixed model sees a different variance structure. Calibrate false positives on the negative control, not on the null. +* A pipeline adjusted until its controls read well has been fitted to this + null and this planted shift. It can still be wrong for every other + question. ```{r teardown, include = FALSE} options(old_options)