Repository navigation
Refresh the GBIF datasets: fix GBIF downloads for galah 2.3 (#121) - #142
Merged
Merged
Conversation
- Skip galah's field check while downloading: galah lists the GBIF field as hasGeospatialIssues, which GBIF rejects (its key is HAS_GEOSPATIAL_ISSUE), and refuses the correct hasGeospatialIssue. - Split downloads into year ranges of at most 1M records by halving on total counts (galah returns only 10 years from a grouped GBIF count), retry each chunk, fail if a chunk comes back short instead of dropping it, and process each chunk as it arrives. - Always write Establishment Means as a string; the validator accepts the bool -> string change from older GBIF files. - Turn on refresh for the five GBIF datasets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Dry run 43769842186 (
Columns and types are unchanged; the The growth is large, so I compared the files record by record:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #121. Fixes the GBIF download code for galah 2.3 (what CI installs) and turns on the monthly refresh for the five GBIF datasets, which still have data from 2025-03-10.
What was broken
HAS_GEOSPATIAL_ISSUE, but galah 2.3 lists the field ashasGeospatialIssues, which GBIF rejects with a 400, and its field check refuses the correcthasGeospatialIssue.download_gbif_obs()now turns galah's field check off while it runs and restores it afterwards. The query itself is unchanged.purrr::possibly()dropped any chunk that failed. Each chunk is now retried (the samewith_retries()as the ALA code), the run fails if a chunk comes back with fewer than 95% of the records GBIF counted, and each chunk is processed as soon as it downloads.Establishment Meanstype: written asboolwhen every value is NA andstringotherwise. It's now always a string, and the validator accepts that one change from older files (build_tiles.Ralready converts it to a string).download_reason_idfor GBIF, only the username, password and email.Tested locally (galah 2.3.0)
validate_release.RThe growth is almost all iNaturalist: records from 2025–26 that the March 2025 download couldn't include, plus older records that have since reached research grade. Specimen counts barely change.
Added offline tests for the year-range splitting (
tests/testthat/test-gbif_download.R).Before merging
Repo secrets
GBIF_USERNAME,GBIF_PWD,GBIF_EMAILDry run of
refresh-datafrom this branch withonlyset to the five GBIF ids; check the validation table (run): validation passed for all 10 datasetsAfter merging, the next scheduled refresh (1 December) includes the GBIF datasets, or run
refresh-databy hand withdry_runoff to publish now.🤖 Generated with Claude Code