Repo hygiene: license, CI, pinned deps, complete env example, no personal data in source

.env.example

@@ -1,5 +1,24 @@
+# Copy to .env and fill in. Every variable the code reads is listed here.
+
+# Required — the app will not import without these
 ANTHROPIC_API_KEY=sk-ant-...
 TWILIO_ACCOUNT_SID=AC...
 TWILIO_AUTH_TOKEN=...
-TWILIO_PHONE_NUMBER=+1...
-SERP_API_KEY=...
+TWILIO_PHONE_NUMBER=+15551234567
+TAVILY_API_KEY=tvly-...            # news search; read at import time
+
+# Deployment
+APP_URL=https://your-app.herokuapp.com   # Twilio status callbacks, Palmer Home links
+# DATABASE_URL=postgres://...            # set by Heroku; omit locally to use SQLite (palmer.db)
+PALMER_CONTACT_EMAIL=you@example.com     # sent to the National Weather Service in the User-Agent, which NWS requires
+
+# Features — each degrades gracefully when its key is absent
+TOMTOM_API_KEY=...            # traffic, drive times, the commute
+SERP_API_KEY=...              # product price watches, shopping, flights, hotels (SerpAPI)
+GIPHY_API_KEY=...             # GIFs
+TMDB_API_KEY=...              # movies and shows in the Opening section (free, non-commercial)
+TICKETMASTER_API_KEY=...      # events in the Opening section
+
+# Development flags
+# PALMER_NO_SCHEDULER=1       # import main without starting the job loop (tests, shells, one-off scripts)
+# DATA_ASK_DRY_RUN=1          # preview the missing-city asks without sending

.github/workflows/ci.yml

@@ -0,0 +1,30 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    env:
+      # Placeholders only. The suite is fully offline; a few modules read their
+      # key at import time and simply need the variable to exist.
+      ANTHROPIC_API_KEY: test
+      TWILIO_ACCOUNT_SID: ACtest
+      TWILIO_AUTH_TOKEN: test
+      TWILIO_PHONE_NUMBER: "+15550000000"
+      TAVILY_API_KEY: test
+      TOMTOM_API_KEY: test
+      SERP_API_KEY: test
+      PALMER_NO_SCHEDULER: "1"
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version-file: .python-version
+          cache: pip
+      - run: pip install -r requirements-dev.txt
+      - run: ruff check .
+      - run: pytest -q

.gitignore

@@ -14,3 +14,7 @@ package-lock.json
 # .env backups — these carry live keys and must never be committed
 .env.bak*
 .env.*.bak
+
+# tool caches
+.pytest_cache/
+.ruff_cache/

CLAUDE.md

@@ -9,7 +9,7 @@ Palmer is a personal AI delivered entirely over SMS via Twilio. A FastAPI web dy
 ## Running / commands
 
 ```bash
-# Install (uses .python-version, currently 3.12)
+# Install (uses .python-version, currently 3.11)
 python3 -m venv venv && source venv/bin/activate
 pip install -r requirements.txt
 
@@ -79,7 +79,7 @@ Dependencies run strictly downward: `llm`/`netutil`/`sources` ← `smstext`/`wea
 
 Underscore prefixes still mean "internal to Palmer", not "private to this module" — `smstext._sms_clean` is imported by six modules. Grep before renaming.
 
-**Patching in tests follows the code, not the name.** `patch("agent.client")` stopped working when functions moved out; patch the module the function actually lives in (`patch("userprofile.client")`). A dead patch target does not fail loudly — it lets the test make real API calls. Watch the suite runtime: 1058 tests in ~10s, and a jump means something is hitting the network.
+**Patching in tests follows the code, not the name.** `patch("agent.client")` stopped working when functions moved out; patch the module the function actually lives in (`patch("userprofile.client")`). A dead patch target does not fail loudly — it lets the test make real API calls. Watch the suite runtime: ~1,400 tests in ~6s, and a jump means something is hitting the network.
 
 ### Scheduler cadence (main.py)
 ```
@@ -666,7 +666,7 @@ Three rules came from the spec and each has a test that catches its reversal:
   (`MAX_EPISODES` takes its slots from `MAX_SCREENS`). A show you actually watch
   is worth more than a film chosen for you, and the row count stays put.
 
-Resolution runs on the **write** path (`resolve_show`, one TMDB search when the
+Resolution runs on the **write** path (`find_shows`, one TMDB search when the
 user follows), never on read — same terms as `_normalize_price_topic` and
 `_city_from_weather_topic`. An unresolvable title asks the user to confirm; it
 never guesses one and never sends them elsewhere to look it up.
@@ -966,7 +966,7 @@ code reading it gets strings where it expects dicts.
 `migrate_profile_prune.py` cleans rows that grew before the allow-list existed. It folds the stray keys into canonical fields with a Sonnet pass before dropping them, so real facts survive. Dry run by default; `--apply` writes.
 
 ### DB access patterns
-- `get_all_profiles()` returns every `(phone, profile)` in ONE query. The scheduler jobs use it. Do not write `for phone in get_all_phones(): get_profile(phone)` — `_conn()` opens a fresh connection per call, so that is N+1 per tick.
+- `get_all_profiles()` returns every `(phone, profile)` in ONE query. The scheduler jobs use it. Do not loop over phones calling `get_profile(phone)` per user — `_conn()` opens a fresh connection per call, so that is N+1 per tick.
 - `upsert_profile()` does its read and write on one connection, and takes a row lock on Postgres. It used to be two connections with an unsynchronised gap, so concurrent writers could drop each other's fields.
 - Pass profiles down rather than re-reading them. The reaction path once cost five `get_profile` calls for a single inbound tapback.
 
@@ -983,7 +983,7 @@ The system prompt in `agent.py` hard-routes user asks to specific tools. Never m
 - `get_price` → CoinGecko (crypto) / yfinance (stocks) only
 - `get_travel_time` / `get_city_traffic` → TomTom only
 - `set_commute` / `clear_commute` → the user's REGULAR drive; TomTom geocode on the write path, routed for their leave time by `home._fetch_traffic`
-- `get_my_page` → the caller's own Palmer Home URL, via `home.ensure_fresh` (never `home_url` — that can hand out a link to a page that was never built)
+- `get_my_page` → the caller's own Palmer Home URL, via `home.ensure_fresh` (never a bare URL builder — that can hand out a link to a page that was never built)
 - `add_price_watch` / `run_price_watches` → SerpAPI Google Shopping only (product prices, distinct from `get_price` for crypto/stocks)
 - `web_search` → Tavily news mode only, never for weather or prices
 
@@ -1426,7 +1426,7 @@ to doubt a confirmation.
   `weather.ambiguous_location`, used on the **write** paths only. Note
   `resolve_weather_location`'s docstring has always claimed "None means the model
   should ask rather than guess" — that was only ever true when *nothing* matched.
-- **Shows.** `resolve_show` took `results[0]` too, and `sports.py`'s own comment
+- **Shows.** The show resolver took `results[0]` too, and `sports.py`'s own comment
   says naming a show is not ambiguous the way naming a team is. True of Reacher,
   false of The Office, Shameless, Skins and Ghosts. `shows.find_shows` returns
   matches with year and origin country; only a genuinely shared title is a

LICENSE

@@ -0,0 +1,14 @@
+Copyright (c) 2026 Jeff Larson. All rights reserved.
+
+This source code is made available for reading, reference and evaluation only.
+
+No license is granted, express or implied, to copy, modify, merge, publish,
+distribute, sublicense, sell, host, or run this software or any derivative of
+it, in whole or in part, for any purpose, without prior written permission
+from the copyright holder.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE.

README.md

@@ -1,5 +1,7 @@
 # Palmer
 
+[![CI](https://github.com/pythonjeff/Palmer/actions/workflows/ci.yml/badge.svg)](https://github.com/pythonjeff/Palmer/actions/workflows/ci.yml)
+
 **The AI that texts like a friend who actually pays attention.**
 
 Palmer is a personal AI delivered entirely over SMS. No app to download, no interface to learn — just text it. It knows who you are, remembers what you care about, and reaches out when something worth knowing happens.
@@ -92,7 +94,8 @@ cp .env.example .env
 | `TOMTOM_API_KEY` | developer.tomtom.com (free tier, ~2,500 requests/day) |
 | `SERP_API_KEY` | serpapi.com (free plan is 250 searches/mo; paid from $50/mo for 5,000) |
 | `APP_URL` | Your deployed app URL (for Twilio status callbacks) |
-| `DATABASE_URL` | Postgres connection string (auto-set by Heroku) |
+| `DATABASE_URL` | Postgres connection string (auto-set by Heroku; omit locally to use SQLite) |
+| `PALMER_CONTACT_EMAIL` | Contact address sent in the User-Agent to the National Weather Service, which requires one |
 
 **3. Run locally**
 ```bash
@@ -134,15 +137,35 @@ GET /preview?phone=+15551234567         # generate morning briefing without send
 
 ---
 
+## Development
+
+```bash
+pip install -r requirements-dev.txt
+pytest                      # ~1,400 tests, all offline, a few seconds
+ruff check .                # lint
+```
+
+The tests need no real keys. A few modules read their API key at import time, so set placeholder values (`TAVILY_API_KEY=x`, `TOMTOM_API_KEY=x`, `SERP_API_KEY=x`, the Twilio and Anthropic vars) or copy `.env.example` to `.env`. `PALMER_NO_SCHEDULER=1` keeps the job loop from starting when `main` is imported in a shell or a one-off script. CI runs the same two commands on every push.
+
+Design notes — the reasoning behind each subsystem and the production incidents that shaped it — live in [`CLAUDE.md`](CLAUDE.md).
+
+---
+
 ## Architecture
 
 - `WEB_CONCURRENCY=1` is required. In-memory phone locks and APScheduler only work correctly in a single process.
 - Per-phone `threading.Lock` serializes inbound messages so conversation history never interleaves under concurrent requests from the same number.
 - **Source quality is one gate, applied at the search call** (`sources.py`). Every news fact and link Palmer sends — watch alerts, the morning briefing, Palmer Home, and conversational web search — goes through `datafeeds._search_raw` / `_search`, which filter before returning, so no surface can drift to its own standard. Four gates: a **blocklist** dropping press-release wires (`prnewswire`, `globenewswire`) and republishing aggregators (`msn.com`, `biztoc`, `news.google.com`) outright; a **relevance floor** that gives trusted sources 0.15 of slack, because Tavily's score measures query-text match and that is exactly what an SEO content farm is built to win; **tier ordering** from `trusted_sources.json` (tier 1 = wires and premier newsrooms plus `.gov`/`.edu`, tier 2 = mainstream and reputable specialists, tier 3 = everything else) sorting `(tier, -score)` so a wire report beats a higher-scoring blog; and **corroboration** requiring ≥ 2 distinct domains OR ≥ 1 tier-1 source before any unprompted alert fires. Palmer Home additionally drops tier 3 entirely — an untrusted row is worse than no row on a short curated list. Edit `trusted_sources.json` to add or remove sources; no code change needed.
 - Watches run every 30 minutes via APScheduler but only alert on major breaking developments — dated results from the last 12 hours, a strict criticality gate, a HEAD reachability check so a dead top link falls through to the next result, per-watch cooldown (default 4 hours), and dedup against the last alerted event.
 - Traffic uses TomTom. Morning briefings auto-include a short city snapshot (`get_city_traffic`): geocode → parallel Traffic Flow + Traffic Incidents in a city bounding box → Haiku drafts one natural line, or skips silently on failure. On demand, users can ask for city conditions or point-to-point drive times (`get_travel_time`, live traffic vs. free-flow). Landmark destinations (White House, Fenway, LAX) get resolved to street addresses by Sonnet before geocoding — TomTom's geocoder is a mapping API, not a search engine, and mis-ranks landmark names. A regular commute is saved with `set_commute` (geocoded once on save, with an optional leave time); the page's Commute card and the morning text are then routed for that departure via TomTom `departAt`, and say whether the number is a forecast for it or live traffic.
-- Price watches use SerpAPI Google Shopping (`shopping.py`): user texts "watch these sneakers under $80" → `add_price_watch` tool saves to `price_watches` → `run_price_watches` job runs every 6h. Each tick: SerpAPI Google Shopping query on the product name, Haiku picks the cheapest genuine match from the top candidates (guards against firing on unrelated accessories/refurbs), first successful check establishes a baseline silently, subsequent checks alert when the target is hit or the price drops ≥ 15% from baseline. Cooldown defaults to 12h. Fails silently on any SerpAPI error — never surfaces "shopping tool failed" to the user.
+- Price watches use SerpAPI Google Shopping (`shopping.py`): user texts "watch these sneakers under $80" → `add_price_watch` tool saves to `price_watches` → `run_price_watches` job runs twice a day on a cron (00:00 and 16:00 UTC — a cron rather than an interval so a deploy cannot reset the clock). Each run: SerpAPI Google Shopping query on the product name, Haiku picks the cheapest genuine match from the top candidates (guards against firing on unrelated accessories/refurbs), first successful check establishes a baseline silently, subsequent checks alert when the target is hit or the price has moved more than $2 in either direction (a flat bar, deliberately not a percentage, so a $12 grocery item and a $1,200 laptop are held to the same standard), then re-baseline. Cooldown defaults to 12h. Fails silently on any SerpAPI error — never surfaces "shopping tool failed" to the user.
 - Proactive outbound is scheduled: **mornings** at each user's local time (5-min tick, catch-up window, per-day guard), **check-ins** on a 2h tick but paced in days (`followup.GAP_DAYS`; Haiku picks one subject from their threads, their team's game or the page's headlines by echoing it exactly, Sonnet drafts, 1–7pm window), and **missing-data asks** every 60 min for users onboarded without a city so mornings can target them (7-day cooldown, US-daytime UTC window; `DATA_ASK_DRY_RUN=1` to preview).
 - Reminder delivery uses `FOR UPDATE SKIP LOCKED` on Postgres — safe for multiple scheduler ticks, no double-sends.
 - Twilio webhook signatures (HMAC-SHA1) validated on every inbound request. All DB queries parameterized and scoped to phone number.
 - Tool routing is hard: `get_weather` → NWS/Open-Meteo only, `get_price` → CoinGecko/yfinance only, traffic tools → TomTom only, product price watches → SerpAPI only, web search → Tavily news mode. No overlap, no hallucinated data.
+
+---
+
+## License
+
+Copyright © 2026 Jeff Larson. All rights reserved. The source is published for reading and evaluation; see [`LICENSE`](LICENSE).

agent.py

@@ -14,8 +14,7 @@
 from datetime import datetime, timedelta, timezone
 
 from db import (
-    init_db, get_history, save_message, get_profile, upsert_profile, save_reminder, cancel_reminders,
-    HISTORY_LIMIT,
+    init_db, get_history, save_message, get_profile, upsert_profile, save_reminder, HISTORY_LIMIT,
     save_watch, get_user_watches, cancel_watches,
     save_price_watch, get_user_price_watches, cancel_price_watches, set_price_watch_baseline,
 )
@@ -745,7 +744,7 @@ def _dispatch(b):
                 current = list(profile.get("weather_locations") or [])
                 asked = (b.input.get("location") or "").strip()
                 # Resolve on the WRITE path, once — never on read, which runs
-                # on every page view. Same terms as resolve_show/_normalize_price_topic.
+                # on every page view. Same terms as shows.find_shows/_normalize_price_topic.
                 resolved = resolve_weather_location(asked)
                 choices = ambiguous_location(asked)
                 if not resolved:

artifacts.py

@@ -30,7 +30,7 @@
 import secrets
 import threading
 
-from db import save_artifact, get_artifact
+from db import get_artifact
 
 _APP_URL = os.environ.get("APP_URL", "").rstrip("/")
 
@@ -48,13 +48,6 @@ def new_token() -> str:
     return secrets.token_urlsafe(16)
 
 
-def publish(payload: dict, ttl_hours: int = TTL_HOURS) -> tuple[str, str]:
-    """Store a briefing payload. Returns (token, page_url)."""
-    token = new_token()
-    save_artifact(token, "briefing", json.dumps(payload).encode(), ttl_hours=ttl_hours)
-    return token, page_url(token)
-
-
 def page_url(token: str) -> str:
     return f"{_APP_URL}/c/{token}"
 

cards.py

@@ -39,7 +39,6 @@
 UP, DOWN = (31, 110, 58), (163, 39, 31)
 WARM = (168, 70, 26)
 COOL = (31, 90, 140)
-AMBER = (138, 90, 16)
 
 _FONT_DIRS = (
     "/usr/share/fonts/truetype/dejavu",                  # heroku slug

datafeeds.py

@@ -7,7 +7,7 @@
 import base64
 import random
 import concurrent.futures
-from datetime import datetime, timezone, timedelta, date as _date
+from datetime import datetime, timezone, timedelta
 
 import requests as _requests
 from tavily import TavilyClient

db.py

@@ -316,15 +316,6 @@ def get_profile(phone: str) -> dict:
     return json.loads(row["profile"]) if row else {}
 
 
-def get_all_phones() -> list[str]:
-    conn = _conn()
-    cur = conn.cursor()
-    cur.execute("SELECT phone FROM users")
-    phones = [r["phone"] for r in cur.fetchall()]
-    conn.close()
-    return phones
-
-
 def upsert_profile(phone: str, updates: dict):
     """Merge `updates` into the stored profile.
 
@@ -577,7 +568,7 @@ def normalize_due_at_rows() -> int:
     which is worth seeing rather than silently rewriting."""
     conn = _conn()
     cur = conn.cursor()
-    cur.execute(f"SELECT id, due_at FROM reminders WHERE sent = 0")
+    cur.execute("SELECT id, due_at FROM reminders WHERE sent = 0")
     fixed = 0
     for row in cur.fetchall():
         parsed = _parse_due(row["due_at"])
@@ -771,7 +762,7 @@ def update_watch_alerted(watch_id: int, summary: str, recent_summaries: list[str
     back by watches._daily_ok, and both used to key on the dyno's UTC day —
     so the cap window rolled at 17:00 Pacific, mid-evening, and a user could
     take the whole allowance across one local evening and be capped by lunch
-    the next day. Same defect alerts.py was fixed for."""
+    the next day. Same defect the retired daily-alert job was fixed for."""
     now = datetime.now(timezone.utc).isoformat()
     today = today or datetime.now(timezone.utc).date().isoformat()
     conn = _conn()

flightwatch.py

@@ -16,10 +16,9 @@
 """
 from __future__ import annotations
 
-from datetime import date, datetime, timezone
+from datetime import date
 
-from db import (get_active_flight_watches, update_flight_watch_price,
-                get_profile)
+from db import (get_active_flight_watches, update_flight_watch_price)
 
 # A fare has to move by more than this to be worth a text. Flights are volatile
 # by tens of dollars daily, so the flat $2 rule that governs product watches