v0.3.9

[mountainMath]

Release branch for v0.3.9.

Fixes #19.
Closes #21.

Major changes

• New experimental functions to detect and correct for likely geocoding anomalies in timelines on a common geography, where the same dwellings got assigned to different neighbouring regions in different years. This shows up as a surprising drop in one region that is offset by a jump in a neighbouring region (Incorporate functionality to detect anomalies in TongFen timelines #21).
  • tongfen_detect_anomalies lists the candidate regions with their best neighbour.
  • tongfen_anomaly_joins determines the regions to join, iterating until no more regions qualify.
  • tongfen_join_regions joins them in data that already is on a common geography.
  • tongfen_join_correspondence joins them in a correspondence for use with tongfen_aggregate.
  • New vignette "Geocoding anomalies in TongFen timelines", following the method from the blog post.
• StatCan correspondence files are now downloaded as parquet files from a mirror on S3. Statistics Canada put the original files behind a browser check that blocks programmatic downloads, which broke method = "statcan" (StatCan disabled programmatic download of correspondence files #19).
• Cached correspondence files are checked against the mirror via ETag once per session and only downloaded again if they changed. When the mirror can't be reached the cached files are used as they are.
• The mirror location can be changed via the tongfen.statcan_correspondence_url option.
• Previously cached statcan_correspondence_*.csv files in the tongfen cache directory are no longer used and can be removed.

Fixes with changed results

• proportional_reaggregate gave wrong results when the finer level data already has values for the categories to reaggregate. The values were compared to the parent total across all categories instead of per category, and a missing value in one child region discarded the existing values of all its siblings.
• tongfen_estimate underestimated values when the intersection of a source and a target region is a geometry collection.
• tongfen_aggregate and aggregate_data_with_meta scaled averages by their parent variable more than once when the metadata lists the same variable name for several datasets, as is common with US census data. tongfen_aggregate now only uses the metadata of the dataset being aggregated, conflicting aggregation rules for the same variable are an error.
• Averages aggregated with na.rm = TRUE are now taken over the regions that have a value. The parent variable of regions with a missing average still counted toward the total, pulling the result toward zero.
• "Average to" variables sharing a parent variable no longer overwrite each other's base. In that case the base columns in the result are named base_<variable> instead of after the parent.

Minor changes

• tongfen_estimate with na.rm = FALSE no longer returns NA for target regions that only share a boundary with a source region with missing values, returns NA for target regions that don't overlap the source instead of erroring out when none of them do, and gives a clear error when target already has a column named like one of the variables to estimate.
• Combining correspondences across three or more datasets no longer runs into a cross join when consecutive correspondences don't share a geographic identifier. This gave a dplyr deprecation warning and needlessly large intermediate tables, results are unchanged.
• estimate_tongfen_correspondence no longer requires the geometry column to be named geometry.
• refresh = TRUE in get_tongfen_correspondence_ca_census and get_tongfen_ca_census now also refreshes the cached StatCan correspondence files.
• US Census Bureau relationship files are downloaded to a temporary file first, an interrupted download no longer leaves a broken file in the cache. They are now cached in the same place as the StatCan correspondence files, also honouring the tongfen.cache_path environment variable and the custom_data_path option.
• tongfen_estimate_ca_census returns its result visibly.
• Documentation fixes: typos, missing \value sections, and the tongfen_aggregate example now passes a named list of datasets matching the metadata.
• The R-CMD-check workflow no longer triggers on pushes to the removed master branch.

Notes

• New dependency: nanoparquet (and tools) in Imports, dplyr requirement raised to 1.1.0, which the package already relied on.
• data-raw/statcan_correspondence.R builds the parquet files from the manually downloaded StatCan originals. The eight files are hosted under https://mountainmath.s3.ca-central-1.amazonaws.com/tongfen/statcan_correspondence/v1/.
• cran-comments.md is updated for the CRAN submission.

Testing

• New tests for the anomaly functions (including a comparison against the original implementation from the blog post), the ETag cache, the correspondence join order, and the reaggregation, estimation and averaging fixes.
• GitHub Actions R-CMD-check passes on macOS (release), Windows (release) and Ubuntu (devel, release, oldrel-1).
• Local R CMD check --as-cran on macOS with R 4.6.0, with all six vignettes rebuilt against live data: 0 errors, 0 warnings, 0 notes. This ran before the spelling and pkgdown commits, which only touch documentation.
• Live check against S3: cold download, cached repeat, unchanged and changed ETag, offline fallback, and get_tongfen_correspondence_ca_census with method = "statcan" for CT and DA levels across the 2001 to 2021 censuses.

🤖 Generated with Claude Code

https://claude.ai/code/session_01A4ixWqz6d5wpYg2DkoT4pm