Repository navigation
feat: window per point, network check and tests - #1
Merged
Merged
Conversation
Weather scripts take --date-col with --days, so each point is read from its own date, such as a planting date, instead of one period for all points. Every script checks its store first and fails with the blocked host named. The HTTP stores honour HTTPS_PROXY. Points with missing columns fail clearly. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LpE9sWSPBk27r6awN4xP7W
scripts/check.py uses the standard library only, so it runs where uv cannot install packages. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LpE9sWSPBk27r6awN4xP7W
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LpE9sWSPBk27r6awN4xP7W
The tile-and-overlap grouping split reads at window gaps, not chunk edges, so groups in one tile fetched the same chunks again, one compute after another. A (point, day) time index in a single isel lets dask fetch each chunk once, in parallel, and drops TILE, windows() and clip(). On a synthetic 3,640-point store: 232 chunk reads instead of 759. When every point shares its dates, a plain time slice does the read. A per-point index costs dask work for every value, which made 4 years for 3,640 points take 22.7 s instead of 1.7 s.
pandas get_indexer with a half-cell tolerance returns -1 for off-grid and missing coordinates, replacing the hand-written bounds check. The point id rides along as the pt coordinate, so daily() no longer maps it back by position. A 1e-6 slack keeps points on a cell edge on the grid.
A failed read already fails fast and says why, so the extra HEAD request on every run added nothing. Wrapping the read names the store host and points to scripts/check.py, and also catches the CHIRTS download host that the pre-check never tested.
The host check only tested GitHub's network, which always reaches the stores, and the smoke test reads from them anyway. Docs-only pull requests no longer fail when a remote store is slow.
It ran outside the try, so a failure crashed the script instead of being reported with the rest.
The dependency list was copied into the README, the test docstring and CI. A PEP 723 header keeps it in one place, like the skill scripts.
The CHIRTS-only version guessed between three Zarr layouts and kept hosts in a class attribute. Listing one chunk URL per store lets the check name the download hosts of any store that redirects, such as upcoming copy of Hugging Face copy of SoilGrids.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A test of the maniva integration asked an agent in a cloud chat session for CHIRPS rain and CHIRTS Tmax/Tmin on 3,640 blocks, from planting date to 100 days after planting. It failed for two reasons inside this repo.
uvcould not reach PyPI and the stores were blocked, with no list of hosts, no check and no rule for the agent to follow.What changes
Window per point. The three weather scripts take
--date-col planting_date --days 100. Each point is read from its own date (day 0) to--dayslater and the output gains adaycolumn. Points in the same spatial chunk with overlapping windows share one read.--startand--endwork as before.fetch()takesdays=and adatecolumn.Network.
scripts/check.pytests all hosts with the standard library only, so it runs where nothing can be installed. For CHIRTS it follows one chunk to name the Hugging Face download host.HTTPS_PROXY(trust_env), which proxied networks need.SKILL.mdgains a "Network" section with the host table and a stop rule: report the blocked hosts, never estimate values or switch source, hand the user the exact command.Points.
validate_pointsfails clearly on missingid,lat,loncolumns.SKILL.mdgains a "Dates" section and the tricot-data mapping now keepsplanting_date.Tests and CI. Offline tests on synthetic stores, including a check that the copied helpers stay identical across skills. A live smoke test reads two points from every store on each pull request and every Monday, to catch a store that moved. AgERA5 runs only if a
CDSAPI_KEYsecret exists.Tested
livejob before merging.Not in this pull request
uv lock --scriptneeds PyPI. Runfor f in skills/*/scripts/fetch.py; do uv lock --script $f; doneon an open network if you want pinned environments.Generated by Claude Code