Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions API_STABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,13 @@ If you depend on `masque` before 1.0, set a version floor in `DESCRIPTION`
(`Imports: masque (>= 0.14.0)`) and record the exact version with `renv`
(`renv::snapshot()`).

- `0.17.0`: new exports `control_spec()`, `controls()`, `control_table()`,
`masque_plan()`, `clone_plan()`, `save_plan()`, `read_plan()`, `handover()`.
`mask(conditional = )` also takes `"design"` or column names, and `roles`
also takes a plan. **A site's coordinate displacement changes under the same
seed**, because the draw is now keyed on the site's value, so a rebuild is a
new release of those sites. No signature breaks.

## 1.0 and after

From `1.0.0`, `masque` adopts a **frozen API**:
Expand Down
2 changes: 1 addition & 1 deletion CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ authors:
repository-code: "https://github.com/max578/masque"
url: "https://max578.github.io/masque/"
license: MIT
version: 0.16.0
version: 0.17.0
keywords:
- synthetic-data
- data-masking
Expand Down
5 changes: 3 additions & 2 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: masque
Title: Structurally Faithful Development Surrogates for Tabular Data
Version: 0.16.0
Version: 0.17.0
Authors@R: c(
person("Max", "Moldovan",
email = "max.moldovan@gmail.com",
Expand Down Expand Up @@ -29,7 +29,7 @@ Depends:
Imports:
cli,
data.table (>= 1.15.0),
digest,
digest (>= 0.6.21),
Matrix,
S7,
stats,
Expand All @@ -42,6 +42,7 @@ Suggests:
ggplot2,
knitr,
maps,
openssl,
readxl,
rmarkdown,
spelling,
Expand Down
8 changes: 8 additions & 0 deletions NAMESPACE
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,30 @@ S3method(print,masque_conformance)
export(apply_recipe)
export(audit_mask)
export(clean_table)
export(clone_plan)
export(conform_table)
export(control_spec)
export(control_table)
export(controls)
export(data_card)
export(detect_design)
export(handover)
export(jitter_coordinates)
export(mask)
export(mask_set)
export(masque)
export(masque_plan)
export(plot_design_summary)
export(propose_roles)
export(read_plan)
export(read_recipe)
export(read_set)
export(recipe)
export(reveal_maps)
export(role_options)
export(roles)
export(roles_validate)
export(save_plan)
export(save_recipe)
export(set_role)
export(synthesise_geospatial)
Expand Down
78 changes: 78 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,81 @@
# masque 0.17.0

## Behaviour changes

* The site displacement of `jitter_coordinates()` and `mask(coords = )` is now
keyed on the site's value and the seed, so one seed gives a site one
displacement even after rows are shuffled, added or removed. In 0.16.0 the
advice to reuse the seed held only for a byte-identical table. **A clone built
by an earlier version moves under the new draw, so rebuilding it counts as a
second release of its sites: retire the old clone.**
* The rebuild table in `?jitter_coordinates` is corrected: 14.6, 9.4, 6.3, 4.4
and 3.1 km at 1, 2, 4, 8 and 16 kept rebuilds, and seven rebuilds, not eight,
fall below the 5 km floor.
* Rows are one site when their coordinates agree to nine decimal places, so a
table written to a file with 15 significant digits and read back keeps its draw
at nearly every site.
* The audit grades a re-simulated number that tracks the original row by row:
HIGH at r of 0.9 or more on the rows of small levels, with the means of large
terms taken out. MEDIUM at 0.7 or more there, or at 0.95 or more overall, which
big environment means alone can reach.
* Under the hierarchy ladder a design level must hold three rows to carry a
shift. A level of two rows gave a shift close to those rows' own values: a clone
correlated 0.99 with the original at its rows, and the audit said low. Conditional
clones with such a column change. Levels of three or four rows still carry a
shift, since real alpha blocks are that small, and the audit grades them.
* A private `seed` in `controls()` no longer appears in the manifest. With a
plan and its recipe, the owner's permutations now come from SHA-256 of a
256-bit key from the system's random source, because a derived integer seed
could be searched against a per-replicate statistic. A private integer `seed`
keeps its 31 bits. Release nothing computed from a single replicate, and check
the permutation p-value as the result of the test on the real table.
* `apply_recipe()` and `controls()` with a recipe now refuse a table whose values
differ from those the recipe was built on, not only its NA mask. A table read
back from a file at 15 digits still matches, as do its column types.
* In a plan, the default control unit passes over a column with too few units for
a 5 percent level. It warns when several columns qualify
(`masque_control_unit_ambiguous`): name the unit `on` was assigned at.
* New warnings mark units and strata with too few arrangements for a 5 percent
level (`masque_control_unit_small`) and a whole-valued target whose rounding
changes the declared shape (`masque_control_shape_rounded`).
`masque_control_strata_weak` now fires when `on` is constant in over 10 percent
of the strata (was 50).
* `handover(allow_high = TRUE)` names the overridden findings in `README.txt`.
The refusal text now says what each writer records: only `masque()` writes to
the recipe.
* The averaged-row figure in `?jitter_coordinates` is corrected to 0.64 km for
360 rows.
* masque needs digest 0.6.21 or later.
* The range statement in `vignette("confidentiality")` is corrected. A whole
number in a collaborate clone can fall one full step outside the observed
range, not half a step as 0.16.0 stated, so a column whose minimum is 0 can
show -1. A conditional clone can go further.

## New features

* `masque_plan()` holds the roles and every named spec in one object.
`mask(df, plan)` reads them from it, an argument passed as well overrides the
plan, and each control spec is resolved on the source and written back to the
recipe. `save_plan()` and `read_plan()` keep a plan as YAML.
* `clone_plan()` gives the plan in clone vocabulary. `controls(clone, plan)` runs
the declared controls for the analyst. `controls(real, plan, recipe = rec)`
runs them for the owner: cleaning and conformance are replayed, a table with a
different schema or NA mask is refused, a spec changed since `mask()` is refused
and the seed is derived privately from the recipe.
* `handover(m, dir)` writes the clone, the clone plan, the data card and a note
to one folder after the release gate. It never writes the recipe.
* `mask(conditional = )` takes `"design"`, which conditions on the retained
design columns alone: the clone keeps design means and block variance and
carries no treatment effect. It also takes column names, an explicit
partition that runs through the same ladder.
* Known-truth controls. `control_spec()` declares what a control permutes and
plants; `controls()` derives a null, `n` negative and `n` positive control
datasets from any table with a role table, by restricted permutation at
the declared unit within declared design strata and a per-level shift in
target-SD units; `control_table()` regenerates any of them from its seed.
The same spec runs on the clone and, with a private seed, on the
original. Vignette "Known-truth controls".

# masque 0.16.0

## Behaviour changes
Expand Down
6 changes: 6 additions & 0 deletions R/audit.R
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,12 @@
#' (`grain = FALSE`, or a collaborate grain with fewer than five units) carries
#' a note saying the clone does not keep that grain.
#'
#' The grade of a re-simulated number compares rows. A shift on a design level
#' of three or four rows, with within-level noise as large as the signal, can
#' grade low while the correlation of the level means stays near 0.85. A low
#' class means the rows are not copies. It does not mean the level means are
#' hidden.
#'
#' As a reference point, a collaborate-mode clone of `agridat::besag.met`
#' (yield as the outcome, design columns kept) scores `low` on every
#' column, and none of its yields equals the original value.
Expand Down
77 changes: 74 additions & 3 deletions R/check_integrity.R
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#' Internal: verify the recipe's recorded NA-mask fingerprint against
#' the candidate `original` data frame.
#' Internal: verify the recipe's recorded NA-mask and value fingerprints
#' against the candidate `original` data frame.
#'
#' Called from [apply_recipe()] when `check_integrity = TRUE` (the
#' default). Aborts with guidance if the fingerprint does not match.
Expand Down Expand Up @@ -32,7 +32,78 @@
"changed since the recipe was built), pass ",
"{.code check_integrity = FALSE} to {.fun apply_recipe}."
)
))
), class = c("masque_integrity_refusal", "orchestra_refusal"))
}
recorded <- .prop_or(rec, "content_fp", list())
if (length(recorded) && !.content_matches(recorded, .content_fingerprint(original))) {

Check warning on line 38 in R/check_integrity.R

View workflow job for this annotation

GitHub Actions / lint

file=R/check_integrity.R,line=38,col=81,[line_length_linter] Lines should not be more than 80 characters. This line is 88 characters.
cli::cli_abort(c(
paste0(
"Recipe integrity check failed: the values of {.arg original} ",
"do not match the recipe's recorded fingerprint."
),
i = "The recipe was built against a table with the same missingness and different values.",

Check warning on line 44 in R/check_integrity.R

View workflow job for this annotation

GitHub Actions / lint

file=R/check_integrity.R,line=44,col=81,[line_length_linter] Lines should not be more than 80 characters. This line is 97 characters.
"*" = paste0(
"If this is intentional, pass {.code check_integrity = FALSE} to ",
"{.fun apply_recipe}. {.fun controls} has no such switch: it runs on ",
"the table the clone was built on."
)
), class = c("masque_integrity_refusal", "orchestra_refusal"))
}
invisible(TRUE)
}

# What the recipe records of the values: two weighted sums per numeric column
# and a digest of the rest. The tolerance covers a file at 15 digits. The sum of
# logs guards small values beside a very large one.
.content_fingerprint <- function(df) {
cols <- sort(names(df), method = "radix")
canon <- lapply(cols, function(cl) .content_canonical(df[[cl]]))
names(canon) <- cols
numeric_col <- vapply(canon, is.numeric, logical(1L))
list(
sums = lapply(canon[numeric_col], function(x) {
ok <- is.finite(x)
w <- sin(seq_along(x) * sqrt(2))
list(
value = sum(x[ok] * w[ok]),
scale = sum(abs(x[ok] * w[ok])),
log_value = sum(sign(x[ok]) * log1p(abs(x[ok])) * w[ok]),
log_scale = sum(log1p(abs(x[ok])) * abs(w[ok])),
other = digest::digest(list(which(!ok & !is.na(x)), x[!ok & !is.na(x)]), algo = "sha256")

Check warning on line 72 in R/check_integrity.R

View workflow job for this annotation

GitHub Actions / lint

file=R/check_integrity.R,line=72,col=81,[line_length_linter] Lines should not be more than 80 characters. This line is 97 characters.
)
}),
text = digest::digest(canon[!numeric_col], algo = "sha256")
)
}

# One form for a column whatever a file reader made of it: a number, where every
# value reads as one, and otherwise text. Logical values count as 0 and 1.
.content_canonical <- function(x) {
if (is.numeric(x) && !is.object(x) || inherits(x, "integer64")) {
return(as.double(x))
}
if (is.logical(x)) {
return(as.double(x))
}
# c() drops the growable bit data.table may set, which serialize() would hash.
text <- c(enc2utf8(as.character(x)))
if (all(text[!is.na(text)] %in% c("TRUE", "FALSE"))) {
return(as.double(text == "TRUE"))
}
num <- suppressWarnings(as.numeric(text))
if (any(is.na(num) & !is.na(text))) text else num
}

.content_matches <- function(recorded, actual) {
if (!identical(recorded$text, actual$text) ||
!setequal(names(recorded$sums), names(actual$sums))) {
return(FALSE)
}
all(vapply(names(recorded$sums), function(cl) {
a <- recorded$sums[[cl]]
b <- actual$sums[[cl]]
identical(a$other, b$other) &&
abs(a$value - b$value) <= 1e-14 * max(a$scale, b$scale) &&
abs(a$log_value - b$log_value) <= 1e-14 * max(a$log_scale, b$log_scale)
}, logical(1L)))
}
Loading
Loading