Skip to content

Refresh the GBIF datasets: fix GBIF downloads for galah 2.3 (#121) - #142

Merged
wcornwell merged 1 commit into
masterfrom
121-gbif-refresh
Nov 16, 2026
Merged

wcornwell merged 1 commit into
masterfrom
121-gbif-refresh

Conversation

@wcornwell

@wcornwell wcornwell commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #121. Fixes the GBIF download code for galah 2.3 (what CI installs) and turns on the monthly refresh for the five GBIF datasets, which still have data from 2025-03-10.

What was broken

  • Filter field: GBIF's key is HAS_GEOSPATIAL_ISSUE, but galah 2.3 lists the field as hasGeospatialIssues, which GBIF rejects with a 400, and its field check refuses the correct hasGeospatialIssue. download_gbif_obs() now turns galah's field check off while it runs and restores it afterwards. The query itself is unchanged.
  • Splitting large downloads: a grouped-by-year GBIF count in galah returns only 10 years (612,933 of 1,167,782 records for Arachnida-FR), so splitting on it would silently skip years. Year ranges are now halved using total counts and then merged while under 1M records.
  • Silent losses: purrr::possibly() dropped any chunk that failed. Each chunk is now retried (the same with_retries() as the ALA code), the run fails if a chunk comes back with fewer than 95% of the records GBIF counted, and each chunk is processed as soon as it downloads.
  • Establishment Means type: written as bool when every value is NA and string otherwise. It's now always a string, and the validator accepts that one change from older files (build_tiles.R already converts it to a string).
  • Download reasons: not needed. galah 2.3 doesn't use download_reason_id for GBIF, only the username, password and email.

Tested locally (galah 2.3.0)

Dataset v2.0.6 rows New rows Old records still present Notes
Cephalopoda-PH 619 1,312 97% 1 chunk; specimens unchanged (131); passes validate_release.R
Arachnida-FR 77,959 138,213 99.2% 2 chunks (553k + 615k raw), both complete; 35 min

The growth is almost all iNaturalist: records from 2025–26 that the March 2025 download couldn't include, plus older records that have since reached research grade. Specimen counts barely change.

Added offline tests for the year-range splitting (tests/testthat/test-gbif_download.R).

Before merging

  • Repo secrets GBIF_USERNAME, GBIF_PWD, GBIF_EMAIL

  • Dry run of refresh-data from this branch with only set to the five GBIF ids; check the validation table (run): validation passed for all 10 datasets

    Dataset v2.0.6 rows New rows Change
    Gekkonidae-MG 5,838 9,288 +59.1%
    Lepidoptera-PE 23,090 37,887 +64.1%
    Cephalopoda-PH 619 1,312 +112.0%
    Arachnida-FR 77,959 138,213 +77.3%
    Squamata-ES 35,824 60,617 +69.2%

After merging, the next scheduled refresh (1 December) includes the GBIF datasets, or run refresh-data by hand with dry_run off to publish now.

🤖 Generated with Claude Code

- Skip galah's field check while downloading: galah lists the GBIF field
  as hasGeospatialIssues, which GBIF rejects (its key is
  HAS_GEOSPATIAL_ISSUE), and refuses the correct hasGeospatialIssue.
- Split downloads into year ranges of at most 1M records by halving on
  total counts (galah returns only 10 years from a grouped GBIF count),
  retry each chunk, fail if a chunk comes back short instead of dropping
  it, and process each chunk as it arrives.
- Always write Establishment Means as a string; the validator accepts
  the bool -> string change from older GBIF files.
- Turn on refresh for the five GBIF datasets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wcornwell

Copy link
Copy Markdown
Collaborator Author

Dry run 43769842186 (dry_run: true, the five GBIF ids only) passed, and nothing was published. Validation against v2.0.6:

Dataset Previous rows (2025-03-10) Rows Change
Gekkonidae-MG 5,838 9,288 +59.1%
Lepidoptera-PE 23,090 37,887 +64.1%
Cephalopoda-PH 619 1,312 +112.0%
Arachnida-FR 77,959 138,213 +77.3%
Squamata-ES 35,824 60,617 +69.2%

Columns and types are unchanged; the Establishment Means bool → string change was accepted as intended. Arachnida-FR (1.17M raw records) downloaded in two chunks, and both came back complete.

The growth is large, so I compared the files record by record:

  • No duplicates: no Link appears twice in the new files.
  • Little churn: 1–2% of the old records are gone, which is normal for GBIF (deleted or re-identified records).
  • The growth is real: it's almost all iNaturalist photographs. Collection records barely changed (e.g. Arachnida 12,355 → 12,447). Most of the new records date from 2025–26, which the March 2025 download predates. Earlier years also gained 5–20%, as older iNaturalist observations reached research grade.

@wcornwell
wcornwell merged commit 6fbc6f7 into master Nov 16, 2026
14 checks passed
@wcornwell
wcornwell deleted the 121-gbif-refresh branch November 16, 2026 01:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable GBIF datasets in the monthly data refresh

1 participant