[Alexcchip]

Summary

Every ingestion source now writes to Postgres through the repository layer. Weather and flights gained normalizers and a runner; the hotel parser stopped writing its own SQL.

Refs #36, not Closes: flight_arrivals is wired and tested but needs OpenSky and RapidAPI credentials to populate. The other two criteria are met.

Type of change

• feat — new feature
• fix — bug fix
• docs — documentation only
• refactor — code change that neither fixes a bug nor adds a feature
• perf — performance improvement
• test — adding or updating tests
• chore / build / ci — tooling, deps, or pipeline changes

Changes

• repository/ gains weather, flights and hotel repositories. upsert_rows is the shared primitive, repositories never commit, the caller owns the transaction.
• upsert_rows collapses duplicate keys within a batch, and stamps captured_at only where the column exists. Both were crashes.
• Weather moved from backend/src/data to foresight/ingest. Behaviour fix: Open-Meteo's Time() is local midnight expressed in UTC, so every reading landed a day early east of Greenwich. weather_date joins stay_date, so this mis-attributed every weather feature by a night.
• New ingest/flights.py, per-flight rows to daily counts, and ingest/external.py, a runner taking --source weather|opensky|aerodatabox.
• The hotel parser no longer calls create_engine or to_sql, which was the last adapter writing SQL directly. A separate persist() writes through the repository.
• upsert_hotel and upsert_room_type let a reload correct base_rate and coordinates. Coordinates coalesce, so a loader that omits them does not erase them.
• docs/external-signals.md covers the contract, both CLIs and the known gaps.

Database migrations

• No schema changes
• Includes an Alembic migration (just migrate-create "...")
• Migration runs cleanly (just migrate) and rolls back (just migrate-down)

b9719be9df40 adds natural keys for the external tables, c97a8a991e88 for the hotel tables. Verified up, down, up, and from scratch on an empty database.

Checklist

• Commits follow Conventional Commits
• Lint passes (just lint) and code is formatted (just format)
• Pre-commit hooks run clean (just hooks installed) — pre-commit run --all-files passes ruff and ruff-format; every commit checked against the pinned conventional-pre-commit==4.1.0
• Updated docs / README / .env.example where relevant — new docs/external-signals.md, plus ingest/ added to the README structure. No new env var
• Tested locally (just dev) and verified the API at http://localhost:8000/docs — booted via uvicorn rather than compose, since Postgres is local here. /health 200, /docs 200, no startup errors. No route changed, but the parser gained an import edge into repository/ worth proving

How to test

1. just migrate
2. cd backend && uv run python -m foresight.ingest.external --source weather --city Dublin --lat 53.3498 --lon -6.2603. Expect 16 new, then 16 unchanged on a re-run.
3. uv run --group ml pytest -q. Expect 195 passing. Bare just test fails on a missing statsmodels, which is pre-existing: it lives in the ml group and CI installs it.

Notes for reviewers

• Overlaps feat: api to serve hotel signal data #45, which also adds repository/hotel.py with create_hotel and create_room_type. Different functions so the merge is additive, but whoever lands second resolves the file. Those versions commit inside the repository while this path commits in the caller, which is worth settling as a team.
• daily_performance.rate is null on every row this writes. It is the training target in compare_models.py, and the export's unmapped ARR column could fill it, but that is a modelling decision rather than a persistence one.