Add the Transport Canada aircraft register as a registry source - #71
Closed
anchildress1 wants to merge 9 commits into
Closed
Add the Transport Canada aircraft register as a registry source#71anchildress1 wants to merge 9 commits into
anchildress1 wants to merge 9 commits into
Conversation
- carry the two Government of Canada notices the open licence requires verbatim - record FAA public-domain provenance and unassessed third-party republished assets - LICENSE covers code only; release data carries its own conditions Generated-by: Claude Opus 5 <noreply@anthropic.com>
- parse the headerless latin1 CCARCS export against its declared column layout - derive transponder_code_hex from the 24-bit Mode S binary, populated for all 34,913 rows - expand marks to C- and vintage CF- registrations and drop owner mailing addresses - mirror the FAA build: same concat-with-latest-release dedup and output conventions Generated-by: Claude Opus 5 <noreply@anthropic.com>
- attribution is a licence condition and NOTICE must travel with release assets - a public licence travels to this project; a bilateral permission does not - non-commercial-only registries are incompatible with the MIT-licensed releases Generated-by: Claude Opus 5 <noreply@anthropic.com>
- reject Mode S fields that are not 24 binary digits; a short field zero-padded into a plausible address belonging to a different aircraft - check the "N rows selected." footer against the parsed row count and enforce a row floor, so an upstream short export fails instead of publishing as a smaller register - prefer ACTIVE_FLAG "A" parties but fall back to all: 1,932 Registered marks carry only "I" rows, and those are the MAIL_RECIPIENT, so filtering on "A" alone drops real owners - count distinct owner names rather than rows, fixing 154 marks labelled Co-owner in error - match the spool footer by pattern instead of a brittle ragged-row count Generated-by: Claude Opus 5 <noreply@anthropic.com>
- fall back to a single-day rebuild only on FileNotFoundError; a rate limit, schema change or truncated download previously took the same path and republished one day as the whole dataset, which the next run then read back as its base - move the monotonic download_date assert out of the try so corruption cannot select the destructive branch - walk back through releases like the ADS-B reader, so one missing optional asset does not strand the accumulation - authenticate release reads and verify downloaded asset size - read the previous CSV with keep_default_na=False so literal NA values round-trip - write the download atomically and reject non-zip responses Generated-by: Claude Opus 5 <noreply@anthropic.com>
- stop asserting US public-domain status for the derived FAA CSV; section 105 covers the government's own work, not this repository's derivative - name no licence instrument for Transport Canada, which does not publish one on the download page, rather than citing one that cannot be verified - state that MIT covers code only, resolving a contradiction with AGENTS.md - lead the source-eligibility rule with the bilateral-permission bar; Taiwan is OGDL licensed and excluded for that reason, not for commercial terms Generated-by: Claude Opus 5 <noreply@anthropic.com>
- publish street, city, postal code and care-of, which the FAA asset already carries as registrant_* for 99.7% of US registrants; dropping them here left one repository with two different postures on the same class of data - take the address from the single MAIL_RECIPIENT row rather than merging across parties, since a co-owned mark lists several people in different cities Generated-by: Claude Opus 5 <noreply@anthropic.com>
- registrant_* rather than owner_*, status rather than registration_status, so both registries describe the same concept with the same column name - registrant_zip_code carries the Canadian postal code: a union table needs one column per concept, not one per country's vocabulary - set source="TC", matching the discriminator the FAA frame already carries - raises the column names shared with the FAA frame from 9 to 21 Generated-by: Claude Opus 5 <noreply@anthropic.com>
- drop narrative rationale, restated rules and background a reader can grep for - one rule per line; the file is read by a model about to act, not by a person catching up - 151 lines to 85 Generated-by: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Code only — this PR adds no workflow and publishes no asset. The pipeline change that
wires it in is stacked on top of it (see the follow-up PR).
Adds the Canadian Civil Aircraft Register (CCARCS) as a registry source, built to mirror the
existing FAA path.
Why this source
Two constraints: the licence must permit redistribution in a public release, and the data must
carry ICAO 24-bit hex, since the ADS-B pipeline joins on it.
Most redistributable registries fail the second test — Australia, Brazil, Spain, Singapore and
Lithuania publish no hex. Taiwan, Estonia and Chile are cleared only by bilateral permission
granted to another project, which does not travel here. Switzerland has hex but the same bilateral
problem. Transport Canada satisfies both: a standing public open licence, and 100% Mode S
coverage across all 34,913 records.
Column naming
The columns use the FAA vocabulary rather than CCARCS's own —
registrant_*notowner_*,statusnotregistration_status, andsource = "TC"matching the discriminator the FAA framealready carries. That raises the column names shared with the FAA frame from 9 to 21, which is what
makes a single joined registry possible in the follow-up.
registrant_zip_codeholds the Canadian postal code. The name is the FAA's on purpose: a uniontable wants one column per concept, not one per country's vocabulary.
Two details worth review
CF-; everything else takesthe modern
C-prefix. The split is 2,451 / 32,462.ACTIVE_FLAGis not "current owner". 1,932 currently-Registered marks carry onlyIparties,and those rows are the
MAIL_RECIPIENT. Filtering onAalone publishes registered aircraft withno owner at all, so the code prefers
Aand falls back to all parties.Owner mailing addresses are published, matching the
registrant_*addresses the FAA asset alreadycarries for 99.7% of US registrants. Because CCARCS lists one row per party, the address comes from
the single
MAIL_RECIPIENTrow rather than being merged across co-owners.NOTICE
LICENSEcovers code; the data has separate terms. Canada requires two notices to reach theconsumer together, so
NOTICEnow carries them. It also records that the derivedopenairframes_faa_*.csvis not itself a US government work — §105 covers the FAA's own data, nota derivative.
Verified against a live CCARCS export: 34,913 rows, hex valid and unique for every one, all inside
the Canadian C-block, 2,650 co-owned marks merged with every party preserved.
🤖 Generated with Claude Code