Add Transport Canada, build registry sources in parallel, and join them into one asset

[anchildress1]

Draft. 11 atomic commits, 8 files. Each commit builds and imports cleanly at its own SHA.

Two related changes that only make sense together: a second registry source, and a pipeline shape
that can actually hold more than one.

---

1. Transport Canada as a registry source

Picked against two constraints — the licence must permit redistribution in a public release, and the
data must carry ICAO 24-bit hex, since the ADS-B pipeline joins on it.

Most redistributable registries fail the second test: Australia, Brazil, Spain, Singapore and
Lithuania publish no hex at all. Taiwan, Estonia and Chile are cleared only by bilateral permission
granted to a different project, which does not travel here. Switzerland has hex but the same
bilateral problem. Transport Canada satisfies both — a standing public open licence, and 100% Mode
S coverage across all 34,913 records.

Two details worth a close look

• Registrations have two eras. Three-character marks are vintage CF-; everything else takes the
  modern C- prefix. The split is 2,451 / 32,462.
• ACTIVE_FLAG does not mean "current owner". 1,932 currently-Registered marks carry only I
  parties, and those rows are the MAIL_RECIPIENT. Filtering on A alone publishes registered
  aircraft with no owner at all, so the code prefers A and falls back to all parties.

Owner mailing addresses are published, matching the registrant_* addresses the FAA asset already
carries for 99.7% of US registrants. Because CCARCS lists one row per party, the address comes from
the single MAIL_RECIPIENT row (exactly one per mark) rather than being merged across co-owners —
a co-owned mark otherwise produces two streets and two cities with no way to pair them.

Column naming

Columns use the FAA vocabulary rather than CCARCS's own: registrant_* not owner_*, status not
registration_status, and source = "TC" matching the discriminator the FAA frame already carries.
That raises the column names shared with the FAA frame from 9 to 21, which is what makes part 2
possible. registrant_zip_code holds the Canadian postal code — the name is the FAA's on purpose,
because a union table wants one column per concept, not one per country's vocabulary.

---

2. Parallel source threads, joined once

Before this, every source was a hand-written job: adding one meant copy-pasting a job and editing
create-release in four places, and each source published its own asset — so a consumer looking up
one aircraft had to know which country's file to open, which is exactly what openairframes_id and
transponder_code_hex exist to avoid.

build-registry-source  (matrix: faa required, tc tolerated)   parallel threads
        |  each calls the reusable registry-source.yaml
        v
join-registry          waits for all, merges on the union of columns
        v
create-release

Adding a registry is now one matrix entry plus src/create_daily_<source>_release.py. The
reusable workflow derives the script path from the source id and fails loudly if it is missing;
join-registry discovers sources by artifact pattern, so nothing downstream changes.
required: true|false per matrix entry replaces ad-hoc continue-on-error, so whether a source can
block the release is declared in one place instead of inferred from several.

New asset

openairframes_registry_{start}_{end}.csv — every registry source on the union of columns,
discriminated by source. Each frame is reindexed to the union before concatenating, so a source
missing a column yields an empty cell rather than a shifted row.

openairframes_faa_*.csv keeps publishing unchanged. It is documented in the README and people
pull it, so it should be deprecated deliberately rather than disappearing under them.

Verified on real data

Joined the live 275 MB FAA asset with a live CCARCS build:

faa: 625,257 rows x 50 cols
tc:   34,913 rows x 41 cols
->   660,170 rows x 70 cols

• 100% coverage of transponder_code_hex, registration_number and openairframes_id on both sources
• shared fields populate on both sides (registrant_name FAA 96.7% / TC 100%)
• zero hex collisions between sources

---

NOTICE

LICENSE covers code; released data has separate terms. Canada requires two notices to reach the
consumer together, so NOTICE carries them and is validated as a required release file —
shipping the data without it removes the permission that makes it redistributable. It also had to be
added to the create-release sparse checkout, which otherwise checks out only .github.

NOTICE also records that the derived openairframes_faa_*.csv is not itself a US government work:
17 U.S.C. §105 disclaims copyright in the FAA's own data and says nothing about a derivative.

Also here

AGENTS.md is trimmed to operative constraints only (151 lines to 85) and updated for the above.

Happy to split any of this back out if you would rather take the pipeline change separately.

🤖 Generated with Claude Code