Add the Transport Canada aircraft register as a registry source

AGENTS.md

@@ -1,121 +1,82 @@
-## Never interpolate `${{ github.event.* }}` into a `run:` block
+## Never interpolate `${{ github.event.* }}` or `${{ inputs.* }}` into a `run:` block
 
-Actions substitutes `${{ }}` as raw text before the shell parses it, and issue bodies here are
-public and unauthenticated. Pass untrusted values through `env:` and quote them.
+Pass them via `env:` and quote the shell variable. A quoted heredoc does not help — the body can
+contain the delimiter and close it early. Never fix this by escaping or renaming the delimiter.
 
-A quoted heredoc delimiter does not save you: the body can *contain* the delimiter, close the
-heredoc early, and execute every line after it. `validate-community-submission.yaml` had that shape.
-Never fix this class of bug by escaping, sanitizing, or renaming the delimiter — move it to `env:`.
+## Invocation
 
-## Run everything from the repo root
-
-`src/create_daily_faa_release.py` must be invoked as a **script** (`python src/create_daily_faa_release.py`).
-It uses bare sibling imports, so `-m src.create_daily_faa_release` raises `ModuleNotFoundError`.
-Everything under `src/adsb/` and `src/contributions/` is the opposite — `python -m`, package-relative.
-
-Output paths are CWD-relative.
+- `src/*.py` at the root of `src/` are scripts: `python src/create_daily_faa_release.py`. Bare
+  sibling imports, so `-m` raises `ModuleNotFoundError`.
+- `src/adsb/*`, `src/contributions/*` are packages: `python -m`.
+- Run from the repo root; output paths are CWD-relative.
 
 ## Verification
 
-There is no test framework, linter, or packaging config. **Do not add one unprompted**, and do not
-treat "nothing broke" as verification.
-
-The ADS-B path has no cheap end-to-end check — one day of input is tens of GB. Exercise
-`compress_multi_icao_df` / `compress_df_polars` directly against a small hand-built Polars frame.
-
-**Never `gh workflow run` to test a change.** Every dispatch pulls tens of GB and fans out over date
-matrices. Reason about the YAML statically.
-
-**Never commit generated data.** The product is a GitHub Release; jobs pass state as artifacts.
-
-## ADS-B invariants
-
-- `FINAL_COLUMN_ORDER` (`compress_adsb_to_aircraft_data.py`) is the only definition of the released
-  column contract. `pl.concat` matches by **position** after `.select()`, and
-  `get_latest_release.get_latest_aircraft_adsb_csv_df` parses released CSVs against the same order —
-  so a forked copy corrupts the release with no error anywhere.
-- `load_parquet_part()` deleting its source parquet is **deliberate**: the raw part is many GB and
-  the runner would otherwise exhaust disk. Do not defer the delete to make reruns easier; that raises
-  peak disk by the size of the part.
-- A released row means "most informative observation for this ICAO on this UTC day" — non-empty
-  fields not a subset of another row's, tie-broken by signature frequency. It is not a registry record.
-- HTTP 404 is terminal in the release fetch. Restoring the retry makes the Dec-31 next-year-repo
-  probe stall ~45 minutes on a repo that does not exist yet.
-
-## Fork and upstream
-
-`src/get_latest_release.py` pins `REPO = "PlaneQuery/openairframes"` on purpose: this fork reads
-**upstream's** releases wherever it runs. Do not repoint it at `github.repository` without being asked.
-
-Upstream develops on `develop` and PRs into `main`. The daily release **deletes the existing release
-and tag** before recreating them.
+- No test framework, linter, or packaging config. Do not add one unprompted.
+- Never `gh workflow run` to test a change — every dispatch pulls tens of GB.
+- Never commit generated data. The product is a GitHub Release; jobs pass state as artifacts.
+- ADS-B has no cheap end-to-end check. Exercise `compress_multi_icao_df` on a hand-built frame.
+
+## Release invariants
+
+- `FINAL_COLUMN_ORDER` (`compress_adsb_to_aircraft_data.py`) is the only definition of the ADS-B
+  column contract. `pl.concat` matches by position after `.select()`; a forked copy corrupts the
+  release with no error.
+- Empty string, never null, in every released frame.
+- `openairframes_id` = `normalize(manufacturer)|normalize(model)|normalize(serial)`. Reuse
+  `derive_from_faa_master_txt.normalize()`.
+- Each source's daily build reads its own previous release asset and appends. Falling back to a
+  single-day rebuild on anything but `FileNotFoundError` republishes one day as the whole dataset,
+  which the next run then reads back as its base. Keep the fallback narrow.
+- Python `3.14` for FAA/community/vendor jobs, `3.12` for ADS-B. Match the surrounding job.
+
+## Registry sources
+
+- Judge **redistribution**, not access. A public licence travels to this project; a bilateral
+  permission granted to another project does not — that alone disqualifies Taiwan, Estonia, Chile.
+  Non-commercial-only terms are a separate, independent bar.
+- `NOTICE` carries the terms that make each asset redistributable and is a required release file.
+  Never edit or drop an entry. Transport Canada requires both its notices together.
+- `LICENSE` is MIT and covers code only. Claim nothing about released data.
+- Owner/registrant mailing addresses are published for every registry. FAA and TC must not diverge.
+- CCARCS `ACTIVE_FLAG` is not "current owner": 1,932 Registered marks carry only `I` parties, and
+  those rows are the `MAIL_RECIPIENT`. Prefer `A`, fall back to all.
+- CCARCS addresses come from the single `MAIL_RECIPIENT == "Y"` row, never merged across co-owners.
+
+## ADS-B
+
+- `load_parquet_part()` deleting its source parquet is deliberate — disk pressure. Do not defer it.
+- A released row is the most informative observation for that ICAO on that UTC day, not a registry
+  record.
+- HTTP 404 is terminal in the release fetch; restoring the retry stalls the Dec-31 probe ~45 min.
 
 ## Community submissions are automation-owned
 
-Merging to `community/**` or `schemas/**` force-pushes every open `community`-labeled PR branch back
-onto main. Anything you hand-edit on such a branch is destroyed on the next merge.
-
-- Never hand-author files in `community/` — the filename encodes `sha256(content)[:8]`, so an edit
-  orphans the hash and duplicates on re-approval.
-- Never invent or copy a `contributor_uuid`; it is derived from the GitHub user id.
-- Do not reintroduce a hardcoded `"main"` or `v1` filename. Both are resolved at runtime now.
-
-**A tag's JSON type is fixed by its first-ever submission and enforced forever** — emergent from
-`build_tag_type_registry` + `validate_submission`, and written nowhere in the schema. Retyping or
-renaming an existing tag breaks every future contributor, not just the current one.
-
-Dropping a `community_submission.v2.schema.json` into `schemas/` promotes it atomically across every
-reader and writer. That is a one-way door for contributors — only on explicit request.
-
-## Conventions
-
-- **Empty string, not null**, everywhere in released frames.
-- Reuse `derive_from_faa_master_txt.normalize()` for `openairframes_id`; never re-derive the format.
-- Python `3.14` for FAA/community/vendor jobs, `3.12` for ADS-B jobs (pyarrow pin + multiprocessing).
-  Deliberate. Match the surrounding job; do not unify.
-
-## References with no target — do not chase as regressions
-
-| Reference | Missing |
-|---|---|
-| `process-historical-faa.yaml` | `src/get_historical_faa.py`, `scripts/concat_csvs.py` |
-| `af-klm-fleet/package.json` → `npm run validate` | `af-klm-fleet/scripts/validate.js` |
-
-`process-historical-faa.yaml` is dead, not stale — it also uses the disabled `::set-output`.
-Repair-vs-delete is the owner's call; leave it alone unprompted.
-
-## `af-klm-fleet/` and `community-routes/` are unwired
-
-Nothing in CI touches either, and nothing consumes `community-routes/`. `af-klm-fleet/` is a vendored
-project by a different author with its own license — its aircraft model is unrelated to
-`schemas/community_submission.*`, so do not merge the two. Its `README.md` is generated by
-`generate-readme.js`; hand edits are overwritten.
-
-## Warts left standing — flag, do not silently fix
+- Merging `community/**` or `schemas/**` force-pushes every open `community` PR branch onto main.
+  Hand edits there are destroyed.
+- Never hand-author `community/` files — the filename encodes `sha256(content)[:8]`.
+- A tag's JSON type is fixed by its first submission and enforced forever. Emergent from
+  `build_tag_type_registry` + `validate_submission`; written nowhere in the schema.
+- Adding `community_submission.v2.schema.json` promotes it atomically across every reader and
+  writer. One-way door — only on request.
 
-- `NUMBER_PARTS` is restated by the matrix and four hand-written upload steps in
-  `adsb-to-aircraft-for-day.yaml`; changing the constant alone silently drops data. YAML cannot loop
-  upload steps and one merged artifact would force every map job to download all parts, so any real
-  fix is a restructure.
-- `MAX_WORKERS = OS_CPU_COUNT if OS_CPU_COUNT > 4 else 1` collapses to a single worker on a ≤4-core
-  runner, shrinking `files_per_batch` with it. Possibly intentional memory control — do not raise it
-  without measuring peak RSS on the target runner.
-- `update-community-prs.yaml` runs `regenerate_pr_schema || true` then force-pushes, so a
-  regeneration failure ships anyway. Making it fatal leaves PRs un-rebased instead — a judgment call.
-- `approve_submission.py` wraps its schema update in a bare `except Exception`, so a submission can
-  merge without its new tags reaching the schema.
+## Fork
 
-## Workflow authoring
+`get_latest_release.REPO` pins upstream `PlaneQuery/openairframes` on purpose. Do not repoint it.
+Upstream develops on `develop`. The daily release deletes the existing release and tag first.
 
-The user's global GitHub Actions rules apply. Existing workflows violate most of them.
-**Do not bulk-remediate** — bring only the file you were asked to touch up to standard, and surface
-the rest in chat.
+## Do not chase
 
-## External sources
+- `process-historical-faa.yaml` is dead: missing `src/get_historical_faa.py`,
+  `scripts/concat_csvs.py`, and uses the disabled `::set-output`.
+- `af-klm-fleet/package.json` → `npm run validate` has no `scripts/validate.js`.
+- `af-klm-fleet/` and `community-routes/` are unwired; nothing in CI touches them.
+  `af-klm-fleet/README.md` is generated.
 
-`registry.faa.gov` and ADS-B Exchange are **required** — the release fails without them. Mictronics is
-**tolerated**: it retries, then the job continues without it. adsb.lol may simply not have published a
-given day, in which case the previous CSV is re-released rather than failing.
+## Flag, do not silently fix
 
-FAA refreshes at 05:30 UTC; the release cron fires at 06:00 UTC. That 30-minute margin is the reason
-for the schedule.
+- `NUMBER_PARTS` is restated by the matrix and four upload steps in `adsb-to-aircraft-for-day.yaml`.
+- `MAX_WORKERS = ... if OS_CPU_COUNT > 4 else 1` collapses to one worker on a ≤4-core runner.
+- `update-community-prs.yaml` runs `regenerate_pr_schema || true`, then force-pushes.
+- `approve_submission.py` wraps its schema update in a bare `except Exception`.
+- Existing workflows violate the global GHA rules. Fix only the file you were asked to touch.

NOTICE

@@ -0,0 +1,66 @@
+OpenAirframes — Data Source Notices
+===================================
+
+The LICENSE file covers the *code* in this repository. It does not cover the
+*data* published in releases. Several upstream registries permit redistribution
+only on condition that specific notices travel with the data. Those conditions
+are reproduced below. This file is published as an asset on every release; anyone
+redistributing a release asset further must carry the corresponding notice with it.
+
+Removing or altering a notice in this file removes the permission that makes the
+corresponding asset redistributable.
+
+
+Transport Canada — Canadian Civil Aircraft Register (CCARCS)
+------------------------------------------------------------
+Asset: openairframes_tc_*.csv
+Source: https://wwwapps.tc.gc.ca/saf-sec-sur/2/ccarcs-riacc/download/ccarcsdb.zip
+
+Redistribution is permitted under the Government of Canada terms recorded for this
+dataset. They require both of the following notices, verbatim, and require that the
+two reach the consumer together. The governing instrument is not published on the
+CCARCS download page, so no licence name or URL is asserted here.
+
+    Reproduced and distributed with the permission of the Government of Canada.
+
+    This product has been produced by or for the OpenAirframes project and
+    includes data provided by the Government of Canada. The incorporation of
+    data sourced from the Government of Canada within this product shall not be
+    construed as constituting an endorsement by the Government of Canada of our
+    product.
+
+Registered-owner mailing addresses are redistributed, matching the registrant
+addresses the FAA asset already carries. Because CCARCS lists one row per party,
+the published address is that of the single designated mail recipient rather than
+a merge across co-owners.
+
+
+FAA — Releasable Aircraft Database
+-----------------------------------
+Source: https://registry.faa.gov/database/ReleasableAircraft.zip
+
+ReleasableAircraft_*.zip is redistributed unmodified. It is a work of the United
+States federal government, not subject to copyright protection in the United
+States (17 U.S.C. § 105). No notice is required; it is credited for provenance.
+
+openairframes_faa_*.csv is a derived product built by this repository —
+normalized, joined, deduplicated, and extended with an identifier this project
+defines. Section 105 disclaims copyright in the government's own work and says
+nothing about a derivative, so no claim is made here about the CSV's status.
+
+
+Other redistributed assets
+---------------------------
+The following assets are republished from third parties whose terms have not
+been assessed in this repository. They are listed for provenance only; nothing
+here asserts a licence over them.
+
+    openairframes_adsb_*.csv.gz   derived from adsb.lol daily globe history
+                                  (github.com/adsblol), with registration data
+                                  from tar1090-db
+    basic-ac-db_*.json.gz         ADS-B Exchange, downloads.adsbexchange.com
+    mictronics-db_*.zip           Mictronics, www.mictronics.de
+
+Community submissions under community/ are contributed by their authors through
+the repository's submission workflow and are published with the attribution each
+contributor selected.

src/create_daily_tc_release.py

@@ -0,0 +1,73 @@
+from pathlib import Path
+from datetime import datetime, timezone
+import argparse
+
+parser = argparse.ArgumentParser(description="Create daily Transport Canada release")
+parser.add_argument("--date", type=str, help="Date to process (YYYY-MM-DD format, default: today)")
+args = parser.parse_args()
+
+if args.date:
+    date_str = args.date
+else:
+    date_str = datetime.now(timezone.utc).strftime("%Y-%m-%d")
+
+out_dir = Path("data/tc_ccarcs")
+out_dir.mkdir(parents=True, exist_ok=True)
+zip_name = f"ccarcsdb_{date_str}.zip"
+
+zip_path = out_dir / zip_name
+if not zip_path.exists():
+    url = "https://wwwapps.tc.gc.ca/saf-sec-sur/2/ccarcs-riacc/download/ccarcsdb.zip"
+    from urllib.request import Request, urlopen
+
+    # CCARCS 403s a default urllib agent. Any browser-like UA works; the exact
+    # version string is not load-bearing.
+    req = Request(
+        url,
+        headers={
+            "User-Agent": (
+                "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
+                "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
+            )
+        },
+        method="GET",
+    )
+
+    with urlopen(req, timeout=120) as r:
+        body = r.read()
+    # TC serves an HTML maintenance page with a 200, which would otherwise be cached
+    # under a .zip name and re-read on every later run.
+    if body[:2] != b"PK":
+        raise RuntimeError(f"{url} did not return a zip (got {body[:40]!r})")
+    tmp_path = zip_path.with_suffix(".part")
+    tmp_path.write_bytes(body)
+    tmp_path.replace(zip_path)
+
+OUT_ROOT = Path("data/openairframes")
+OUT_ROOT.mkdir(parents=True, exist_ok=True)
+from derive_from_tc_ccarcs import convert_tc_ccarcs_to_df
+# Named for FAA but column-agnostic: fingerprints every column except download_date.
+from derive_from_faa_master_txt import concat_faa_historical_df
+from get_latest_release import get_latest_aircraft_tc_csv_df
+df_new = convert_tc_ccarcs_to_df(zip_path, date_str)
+
+# Only a genuine first run may rebuild from a single day. Every other failure -- a rate
+# limit, a schema change, a truncated download -- must stop the run, because this file
+# becomes tomorrow's base and silently republishing one day erases the whole history.
+try:
+    df_base, start_date_str = get_latest_aircraft_tc_csv_df()
+except FileNotFoundError as e:
+    print(f"No existing Transport Canada release found, bootstrapping from today only: {e}")
+    df_base = None
+    start_date_str = date_str
+
+if df_base is not None:
+    missing = set(df_base.columns) ^ set(df_new.columns)
+    if missing:
+        raise SystemExit(f"Column set changed since the last release: {sorted(missing)}")
+    df_base = concat_faa_historical_df(df_base, df_new)
+    assert df_base['download_date'].is_monotonic_increasing, "download_date is not monotonic increasing"
+else:
+    df_base = df_new
+
+df_base.to_csv(OUT_ROOT / f"openairframes_tc_{start_date_str}_{date_str}.csv", index=False)