Skip to content

feat: window per point, network check and tests - #1

Merged
bjyberg merged 11 commits into
mainfrom
claude-kauedesousa
Nov 18, 2026
Merged

bjyberg merged 11 commits into
mainfrom
claude-kauedesousa

Conversation

@kauedesousa

Copy link
Copy Markdown
Member

Why

A test of the maniva integration asked an agent in a cloud chat session for CHIRPS rain and CHIRTS Tmax/Tmin on 3,640 blocks, from planting date to 100 days after planting. It failed for two reasons inside this repo.

  1. The repo assumed open internet and never said so. uv could not reach PyPI and the stores were blocked, with no list of hosts, no check and no rule for the agent to follow.
  2. The scripts took one period for all points. Plantings ran from October 2021 to October 2024, so one period meant about 4.4 million rows for 367,640 needed.

What changes

Window per point. The three weather scripts take --date-col planting_date --days 100. Each point is read from its own date (day 0) to --days later and the output gains a day column. Points in the same spatial chunk with overlapping windows share one read. --start and --end work as before. fetch() takes days= and a date column.

Network.

  • Every script checks its store first and fails with the blocked host named, instead of a library traceback.
  • scripts/check.py tests all hosts with the standard library only, so it runs where nothing can be installed. For CHIRTS it follows one chunk to name the Hugging Face download host.
  • The CHIRTS and AgERA5 readers now honour HTTPS_PROXY (trust_env), which proxied networks need.
  • SKILL.md gains a "Network" section with the host table and a stop rule: report the blocked hosts, never estimate values or switch source, hand the user the exact command.

Points. validate_points fails clearly on missing id, lat, lon columns. SKILL.md gains a "Dates" section and the tricot-data mapping now keeps planting_date.

Tests and CI. Offline tests on synthetic stores, including a check that the copied helpers stay identical across skills. A live smoke test reads two points from every store on each pull request and every Monday, to catch a store that moved. AgERA5 runs only if a CDSAPI_KEY secret exists.

Tested

  • Window grouping, clipping, validation, the host check and the CLI errors were run on the 3,640-block file in the blocked sandbox. Every block got 101 days and reads dropped from 4.4 million to about 1 million point-days.
  • The store reads themselves could not run there. The checks on this pull request are their first real run, so please read the live job before merging.

Not in this pull request

  • Lock files. uv lock --script needs PyPI. Run for f in skills/*/scripts/fetch.py; do uv lock --script $f; done on an open network if you want pinned environments.
  • Links from tricot-data and tricot-analysis-workflows, which follow once this layer is settled.
  • CHIRTS still depends on one personal Hugging Face account. The Monday test will flag a break but there is no second source yet.

Generated by Claude Code

Weather scripts take --date-col with --days, so each point is read from its
own date, such as a planting date, instead of one period for all points.
Every script checks its store first and fails with the blocked host named.
The HTTP stores honour HTTPS_PROXY. Points with missing columns fail clearly.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpE9sWSPBk27r6awN4xP7W
scripts/check.py uses the standard library only, so it runs where uv cannot
install packages.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpE9sWSPBk27r6awN4xP7W
@kauedesousa
kauedesousa requested a review from bjyberg November 10, 2026 13:02
The tile-and-overlap grouping split reads at window gaps, not chunk
edges, so groups in one tile fetched the same chunks again, one compute
after another. A (point, day) time index in a single isel lets dask
fetch each chunk once, in parallel, and drops TILE, windows() and clip().
On a synthetic 3,640-point store: 232 chunk reads instead of 759.

When every point shares its dates, a plain time slice does the read.
A per-point index costs dask work for every value, which made 4 years
for 3,640 points take 22.7 s instead of 1.7 s.
pandas get_indexer with a half-cell tolerance returns -1 for off-grid
and missing coordinates, replacing the hand-written bounds check. The
point id rides along as the pt coordinate, so daily() no longer maps it
back by position. A 1e-6 slack keeps points on a cell edge on the grid.
A failed read already fails fast and says why, so the extra HEAD request
on every run added nothing. Wrapping the read names the store host and
points to scripts/check.py, and also catches the CHIRTS download host
that the pre-check never tested.
The host check only tested GitHub's network, which always reaches the
stores, and the smoke test reads from them anyway. Docs-only pull
requests no longer fail when a remote store is slow.
It ran outside the try, so a failure crashed the script instead of being reported with the rest.
The dependency list was copied into the README, the test docstring and CI. A PEP 723 header keeps it in one place, like the skill scripts.
The CHIRTS-only version guessed between three Zarr layouts and kept
hosts in a class attribute. Listing one chunk URL per store lets the
check name the download hosts of any store that redirects, such as
upcoming copy of Hugging Face copy of SoilGrids.
@bjyberg
bjyberg added this pull request to stack #4 November 18, 2026 13:02
@bjyberg
bjyberg merged commit 091eef8 into main Nov 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants