Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/workflows/live.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: live

on:
pull_request:
paths: ["skills/**", "tests/**"]
push:
branches: [main]
paths: ["skills/**", "tests/**"]
schedule:
- cron: "23 5 * * 1" # Mondays, to catch a data store that moved or changed
workflow_dispatch:

permissions:
contents: read

jobs:
live:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- name: Two points from every store
run: python3 tests/smoke.py
env:
CDSAPI_KEY: ${{ secrets.CDSAPI_KEY }}
18 changes: 18 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: test

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

jobs:
offline:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- name: Unit tests on synthetic stores
run: uv run tests/test_fetch.py
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This repository holds skills that fetch weather, elevation and soil data for poi

1. Read `SKILL.md` first. It says which skill answers which need and the rules every skill shares.
2. Open only the skill folder the question needs, under `skills/`. Each `SKILL.md` is self-contained and its `scripts/` file does the fetch.
3. Run scripts with `uv run`. Do not install anything else without the user's agreement.
3. Run scripts with `uv run`. Do not install anything else without the user's agreement. If a script cannot reach its data store, or `uv` cannot install packages, follow "Network" in `SKILL.md`: run `python3 scripts/check.py`, name the blocked hosts, and hand the user the exact command. Never estimate values or switch source.
4. Keep the data's caveats with the data. Say which source, grid and dates a result came from.
5. Never invent a DOI, licence or date. Each skill ends with what to cite.
6. Write maniva and tricot in lowercase, always, including headings, table cells and the start of a sentence. Never write "Maniva" or "Tricot". tricot stands for triadic comparison of technology options. Quote published titles exactly as printed, even where they capitalise tricot or expand it as "triadic comparisons of technologies".
Expand Down
17 changes: 16 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,27 @@ ln -s "$PWD"/environment-data/skills/* ~/.claude/skills/

**Codex and Gemini CLI.** Same, into `~/.agents/skills/` or `~/.gemini/skills/`.

**Without an agent.** Run any script directly, e.g. `uv run skills/chirps-rainfall/scripts/fetch.py points.csv rain.parquet --start 2020-01-01`.
**Without an agent.** Run any script directly, e.g. `uv run skills/chirps-rainfall/scripts/fetch.py points.csv rain.parquet --start 2020-01-01`. When each point has its own date, such as a planting date, fetch a window per point: `--date-col planting_date --days 100`.

**R users.** Run the scripts as above, then read the parquet outputs in R with `nanoparquet::read_parquet()`, a small package with no dependencies, or `arrow::read_parquet()` if you already use arrow.

---

## Network

The scripts read remote stores and `uv` installs their packages on first run. Sandboxed agents, such as a cloud chat session, and some institutional networks block those hosts. `python3 scripts/check.py` lists what is blocked, using the standard library only. The hosts are in [SKILL.md](SKILL.md#network), with what an agent does when they are out of reach.

## Tests

```sh
uv run tests/test_fetch.py # offline, synthetic stores
python3 tests/smoke.py # two points from every live store
```

The offline tests run on every pull request. The live test runs on pull requests that change `skills/` or `tests/`, and every Monday to catch a store that moved.

---

## Cite

Cite with [CITATION.cff](CITATION.cff).
Expand Down
39 changes: 35 additions & 4 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,22 +26,53 @@ Write maniva and tricot in lowercase, always. That holds in headings, table cell

## Three ways to use a skill

1. **Run the script.** Resolve `scripts/fetch.py` from the selected skill directory and follow its CLI example (weather also requires `--start`). Use this by default. Output is parquet; name the output `.csv` for CSV, which suits elevation, soil and small weather pulls (daily weather for many points gets large).
2. **Import the script module.** Its `fetch()` takes a pandas data frame with `id`, `lat`, `lon` and returns a data frame. Use it inside a larger Python pipeline.
1. **Run the script.** Resolve `scripts/fetch.py` from the selected skill directory and follow its CLI example (weather also requires a period, see "Dates"). Use this by default. Output is parquet; name the output `.csv` for CSV, which suits elevation, soil and small weather pulls (daily weather for many points gets large).
2. **Import the script module.** Its `fetch()` takes a pandas data frame with `id`, `lat`, `lon` (and `date` for a window per point) and returns a data frame. Use it inside a larger Python pipeline.
3. **Write your own reader.** Each skill's "If you need to read ... yourself" section gives the store path and the traps (units, scaling, projection, grid order). Use it for R, other tools or needs the script does not cover.

## Points

Users give points in any form: a table with other column names, Excel, tricot data, or coordinates typed in chat. Build the `id`, `lat`, `lon` table yourself, using an existing identifier as `id` or the row number if there is none. Tell the user which columns you used. Use WGS84 decimal degrees with `lat` within −90 to 90 and `lon` within −180 to 180. Preserve IDs as strings and require unique, non-empty IDs for unambiguous joins. Investigate suspected swapped coordinates; never guess a correction or shift points to obtain values. Convert known missing-coordinate sentinels to blanks. Leave rows with missing coordinates in; the scripts return them as NaN.

For a tricot-data trial file, take `site_data[]`: `site_id` → `id`, `latitude` → `lat`, `longitude` → `lon`. Some sites have null coordinates; keep them as blanks. tricot site coordinates are displaced up to 1.5 km for privacy (`metadata.location_privacy`), and some are filled from the mean of their district (`coordinate_source`, `coordinate_uncertainty_m`). That is fine for 5 to 10 km weather grids. For elevation (90 m) and soil (250 m), values describe the published point, not the farm; say so when you report them.
For a tricot-data trial file, take `site_data[]`: `site_id` → `id`, `latitude` → `lat`, `longitude` → `lon`, and keep `planting_date` for a window per site (`--date-col planting_date`). Some sites have null coordinates or planting dates; keep them as blanks. Other trial tables follow the same pattern: one row per plot or block, its identifier as `id`, its coordinates and its planting date. tricot site coordinates are displaced up to 1.5 km for privacy (`metadata.location_privacy`), and some are filled from the mean of their district (`coordinate_source`, `coordinate_uncertainty_m`). That is fine for 5 to 10 km weather grids. For elevation (90 m) and soil (250 m), values describe the published point, not the farm; say so when you report them.

## Dates

The weather skills take a period in one of two ways. Use the same one for every source you combine.

- **One period for all points.** `--start 2020-01-01 --end 2020-12-31`. Use it for a season, a year or a climatology.
- **A window per point.** `--date-col planting_date --days 100` fetches each point from its own date (day 0) to 100 days later, 101 rows per point, with a `day` column counting from 0. Use it whenever points have their own dates, such as planting dates per plot or survey dates per household. Points planted weeks or years apart then cost only their own windows, not the whole span between the first and the last.

Keep the date column in the points file under its own name and pass that name to `--date-col`. Dates are ISO (`YYYY-MM-DD`). A point with a blank date gets no rows and the script lists it; report those IDs. A window that runs past the store's last day fails rather than being shortened. Ask the user for the number of days if they did not give one; the crop's season length is theirs to state.

## Network

The scripts read remote stores, and `uv` installs their packages on first run, so they need these hosts. Sandboxed agents and institutional networks often block them.

| Need | Hosts |
|---|---|
| Packages, first run of any script | `pypi.org`, `files.pythonhosted.org` |
| `chirps-rainfall` | `dynamical-ucsb-chc-chirps.s3.us-west-2.amazonaws.com` |
| `chirts-temperature` | `huggingface.co` and the download host it redirects to, which `scripts/check.py` names |
| `agera5-climate` | `arco.datastores.ecmwf.int` |
| `elevation` | `copernicus-dem-90m.s3.amazonaws.com` |
| `soilgrids-soil` | `files.isric.org` |

`python3 scripts/check.py [skill ...]` tests them with the standard library only, so it runs where nothing can be installed. When a script cannot read its store, its error names the host and points here.

When a host is blocked:

1. Run `scripts/check.py` once and report the blocked hosts as it prints them. Do not retry with other tools, mirrors or another data source the user did not ask for.
2. Never estimate, simulate or fill in the values. A blocked network is not missing data.
3. Prepare everything that needs no network: the points file and the exact command.
4. Give the user two ways forward: allow the hosts in the agent's network settings (the change may only apply to a new session), or run the command on their own machine and share the output file.

## Rules

1. Open the one skill the question needs. Each is self-contained. Pass the same points and dates between skills.
2. To run a script, use `uv run`. It needs only `uv`, which fetches Python and the packages on first run. If `uv` is missing, give the user the one-line install from https://docs.astral.sh/uv/getting-started/installation/ and ask before installing.
3. Return the output path, source/store, grid, units, actual dates (or soil depths/statistic), retrieval date and relevant caveats. Values are gridded estimates for a cell, not station readings for a point.
4. Report points and variables with missing values, including partial date gaps. Missing coordinates, coverage limits and source masks are different from network or authentication failures; do not label every failure as water or fill gaps silently.
4. Report points and variables with missing values, including partial date gaps. Missing coordinates, coverage limits and source masks are different from network or authentication failures; do not label every failure as water or fill gaps silently. For a network failure follow "Network".
5. Return daily data when a calling skill asked for it. Start hazard analysis only when the user asked for it.
6. Never invent a DOI, licence or date. Each skill ends with what to cite.

Expand Down
1 change: 1 addition & 0 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Read SKILL.md first. It says which skill answers which need and the rules they s
- [SKILL.md](https://raw.githubusercontent.com/AgrDataSci/environment-data/main/SKILL.md): index of skills and shared rules
- [AGENTS.md](https://raw.githubusercontent.com/AgrDataSci/environment-data/main/AGENTS.md): short instructions for AI agents
- [README.md](https://raw.githubusercontent.com/AgrDataSci/environment-data/main/README.md): sources, install and licence
- [scripts/check.py](https://raw.githubusercontent.com/AgrDataSci/environment-data/main/scripts/check.py): tests whether this network reaches the data stores; standard library only

## Skills

Expand Down
80 changes: 80 additions & 0 deletions scripts/check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
"""Can this machine reach what the skills need? Standard library only, so it runs where packages cannot be installed.

python3 scripts/check.py [skill ...]

Prints one line per host and exits 1 if any is blocked. Run it first in a sandbox, or after a fetch fails to connect."""

import shutil
import sys
import urllib.error
import urllib.request
from urllib.parse import urljoin, urlsplit

# host lists mirror HOSTS in each skill's scripts/fetch.py
NEEDS = {
"packages": ["pypi.org", "files.pythonhosted.org"], # uv installs the Python packages from here on first run
"chirps-rainfall": ["dynamical-ucsb-chc-chirps.s3.us-west-2.amazonaws.com"],
"chirts-temperature": ["huggingface.co"],
"agera5-climate": ["arco.datastores.ecmwf.int"],
"elevation": ["copernicus-dem-90m.s3.amazonaws.com"],
"soilgrids-soil": ["files.isric.org"],
}
# one data chunk per store that redirects downloads to other hosts, such as Hugging Face; the check follows it to name them
FOLLOW = {
"chirts-temperature": ["https://huggingface.co/datasets/aaguilar90/chirts-era5-daily/resolve/main/chirts-era5-daily.zarr/Tmax/c/0/0/0"],
}


class NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *args): # hand back each 3xx so its Location can be read
return None


def blocked(host: str) -> str | None:
"""Why the host is out of reach, or None. Any HTTP answer, even 403 or 404, means the host is reachable."""
try:
urllib.request.urlopen(urllib.request.Request(f"https://{host}/", method="HEAD"), timeout=20)
except urllib.error.HTTPError:
return None
except OSError as exc:
return str(getattr(exc, "reason", exc))
return None


def redirect_hosts(url: str) -> list[str]:
"""Each host a download is sent on to, hop by hop. A host that does not answer ends the chain and is still listed."""
hosts, opener = [], urllib.request.build_opener(NoRedirect)
for _ in range(5): # real chains are one or two hops; this stops a loop
try:
opener.open(urllib.request.Request(url, method="HEAD"), timeout=20)
except urllib.error.HTTPError as exc:
if location := exc.headers.get("Location"):
url = urljoin(url, location)
hosts.append(urlsplit(url).hostname)
continue
except OSError:
pass
break
return list(dict.fromkeys(hosts))


if __name__ == "__main__":
skills = sys.argv[1:] or [s for s in NEEDS if s != "packages"]
if unknown := set(skills) - set(NEEDS):
sys.exit(f"unknown skill {sorted(unknown)}; choose from {', '.join(s for s in NEEDS if s != 'packages')}")
print(f"uv: {shutil.which('uv') or 'not found, see https://docs.astral.sh/uv/getting-started/installation/'}")
if sys.version_info < (3, 12):
print(f"python {sys.version.split()[0]}: older than 3.12, so uv will also download Python from github.com")
failed = []
for need in ["packages", *skills]:
results = {h: blocked(h) for h in NEEDS[need]}
if need in FOLLOW and not any(results.values()): # redirects can only be followed once the store's host answers
for url in FOLLOW[need]:
results |= {h: blocked(h) for h in redirect_hosts(url) if h not in results}
for host, why in results.items():
print(f"{'BLOCKED' if why else 'ok':8}{need:20}{host}{f' ({why})' if why else ''}")
failed += [host] if why else []
if failed:
print(f"\nThis network blocks {len(failed)} host(s). Allow them, or run the fetch on a machine with open internet:")
sys.exit("\n".join(failed))
print("\nAll hosts reachable.")
7 changes: 5 additions & 2 deletions skills/agera5-climate/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,10 @@ Resolve `scripts/fetch.py` relative to this skill's directory, not the user's wo

```sh
uv run scripts/fetch.py points.csv weather.parquet --start 2020-01-01 --end 2025-12-31
uv run scripts/fetch.py points.csv weather.parquet --date-col planting_date --days 100
```

`--start` is required; use the requested period, asking if it is unspecified. `--end` defaults to the latest day actually in the store. Explicit dates outside store coverage fail with the available range; do not silently shorten a requested period. Needs only `uv` ([install](https://docs.astral.sh/uv/getting-started/installation/)) and a key (below). Output is one row per point and day: `id`, `date`, the variables, `cell_lat`, `cell_lon`. The script prints any point whose series is all NaN (missing coordinates, outside coverage or source nodata). Report these IDs without moving the points.
Give one period for all points with `--start`, or a window per point with `--date-col` and `--days`: each point is fetched from the date in that column (day 0) to `--days` later, and the output gains a `day` column counting from 0. Use the per-point form when points have their own dates, such as planting dates; it reads only the dates each point needs. A point with a blank date gets no rows and is listed. Use the requested period, asking if it is unspecified. `--end` defaults to the latest day actually in the store. Explicit dates outside store coverage fail with the available range; do not silently shorten a requested period. Needs only `uv` ([install](https://docs.astral.sh/uv/getting-started/installation/)) and a key (below). Output is one row per point and day: `id`, `date`, the variables, `cell_lat`, `cell_lon`. The script prints any point whose series is all NaN (missing coordinates, outside coverage or source nodata). Report these IDs without moving the points.

Default variables are the ones CHIRPS and CHIRTS don't have. Pick others with `--vars`:

Expand Down Expand Up @@ -46,6 +47,8 @@ The script looks for `CDSAPI_KEY` in the environment, then the `key:` line of `~
key: <token>
```

The script needs `pypi.org` and `files.pythonhosted.org` on first run and `arco.datastores.ecmwf.int` for the data. If it reports a host it cannot reach, the network is blocked, not the data missing: say so, name the host, and give the user the command to run on an open network. Do not switch source or estimate values.

## If you need to read the store yourself

```python
Expand All @@ -57,7 +60,7 @@ ds = xr.open_zarr("https://arco.datastores.ecmwf.int/cadl-arco-geo-001/arco/sis_
- 27 variables, `ds.data_vars` lists them. Temperatures in K, radiation in J m⁻² d⁻¹, missing is `-9999` and decodes to NaN.
- Latitude ascending from -90, longitude from -180. Chunks are 2112 days × 16 × 32 cells, so long periods at a few places are cheap and one day over a region is slow.
- Each day runs midnight to midnight local time, not UTC.
- Select variables and dates first, pick all points in one vectorised `sel(method="nearest")`, `.compute()` once, raise dask's thread count.
- Select variables and dates first, pick all points in one vectorised `sel(method="nearest")`, `.compute()` once, raise dask's thread count. For a different window per point, index time with a (point, day) array of positions in the same vectorised `isel`; dask still fetches each chunk once. `select_daily` in `scripts/fetch.py` shows how.

## Caveats

Expand Down
Loading
Loading