September release candidate (DO NOT MERGE) - #1112
Draft
ChristianPavilonis wants to merge 411 commits into
Draft
Conversation
Harden the request-scoped EC KV snapshot work (#851) with the test coverage the spec's test strategy called for, plus small cleanups the plan requested. Tests: - ec::kv: snapshot upsert (write-without-read, unchanged-preserves-gen, refresh-once, CAS re-merge, tombstone-rejects, store-fail) and conditional tombstone (CAS conflict, store failure, disappear-on-retry) - ec::mod: generate_if_needed collision retry + exhaustion; default and read-path recovery-ineligibility (non-Fastly adapter contract) - ec::prebid_eids: collect_eid_cookie_updates merge and empty-registry - ec::finalize: NotRead/Failed/tombstone/subresource no-rotate paths and two-ID existing-only withdrawal - ec::pull_sync: request-wide aggregation into one bulk write across concurrency batches, plus no-dispatch for non-present snapshots - publisher: concurrent-vs-eager origin scheduling order and origin-start failure, using recording HTTP/KV collaborators Cleanups: - Extract rewrite_origin_request to remove the duplicated origin-rewrite logic across the concurrent and eager paths - Bind the orphaned EC ID once in recover_orphaned_ec Docs: - Note the fastly-ssl vendor-header layering wart in core scheme detection and the origin-forwarding strip (comments only; behavior unchanged)
A Failed request-scoped snapshot no longer short-circuits tombstone_existing_from_snapshot. A transient read error earlier in the request must not silently drop a consent withdrawal, so a non-authoritative snapshot is re-read (bounded by MAX_CAS_RETRIES) and the row is tombstoned when present. An authoritative Missing snapshot stays a no-op.
Address PR review findings: - Gate orphan-recovery eligibility to the publisher fallback after a successful origin start. Named routes, integration proxies, and request-filter short circuits no longer reach EC finalization with recovery authorized, so a blocked or non-publisher response cannot rotate an identity. - Never downgrade an in-request Add-confirmed Present snapshot on a preload refresh miss, and confirm an authoritative miss with a second read (after the origin round trip) before rotating. A single eventually-consistent edge miss can no longer rotate a valid identity; a now-visible row is adopted instead. - Preload the origin-overlapped snapshot with the unfiltered active EC ID while keeping the consent-filtered ID for auction identity, so consent-withdrawn navigations keep the withdrawal CAS off the post-origin latency path. - Resolve the initial usable snapshot outside the CAS retry counter in both partner upsert and conditional tombstoning, so a generation-unavailable or refreshed snapshot keeps all five write attempts. - Defer /auction and page-bids identity-graph reads until a live auction actually runs with a partner registry, avoiding billable KV reads that cannot be consumed. Add tests for four-conflicts-then-fifth-write CAS, transient Add->Missing->Present confirmation, and recovery-eligibility lifecycle across named routes, filter short circuits, and origin-start failures.
Ad-heavy publisher pages (video players, continuous ad refresh, anti-bot scripts) may never fire the `load` event, so `page.goto` would block until the navigation timeout and the audit failed before scraping any slots. Article pages consistently timed out this way while lighter listing pages succeeded. Navigate without hard-failing on the load wait: a load-wait or main-document-response timeout is downgraded to a "results may be partial" warning, and the existing settle loop is the real readiness signal. The settle loop now also accepts `interactive` readyState, since these pages define their GPT slots before (or without ever) reaching `complete`. Load wait is bounded separately at 12s and the settle cap is raised to 12s so lazily-defined slots are captured.
`render_slots` prepends a `# Slots managed by ...` header, but the in-place splice preserved the previous copy in the scalar block and inserted a fresh one, so each `ts audit ad-templates generate` run against an already-managed config appended another duplicate comment block. Extract the two header lines to constants and strip any prior copy (and the blank lines it leaves) from the preserved head before re-inserting the rendered slots, so repeated runs keep exactly one header. Add a regression test that splices three times and asserts a single header.
…o feature/ts-cli-ad-templates
CI's rustfmt wraps the single method-chain argument of this assert! onto its own lines; the compact form the merge brought in passed locally but failed the format gate. Match CI's canonical form.
…apshot-ec-ttl # Conflicts: # crates/trusted-server-adapter-fastly/src/app.rs # crates/trusted-server-core/src/auction/endpoints.rs # crates/trusted-server-core/src/ec/finalize.rs # crates/trusted-server-core/src/ec/prebid_eids.rs # crates/trusted-server-core/src/ec/pull_sync.rs # crates/trusted-server-core/src/http_util.rs
The CLI still called `resolved_gam_unit_path`, which core replaced with the
path-aware `render_gam_unit_path` when `{section}` templating landed, so the
crate no longer compiled. Both call sites now derive the section through
`CreativeOpportunitiesConfig::section_for_path` and render the template, and
`ExpectedSlot`/`ConfiguredJson` carry an optional unit path so an over-limit
dynamic render is reported rather than silently matched against the wrong unit.
Also resolves the outstanding review findings on these paths:
- Write the operator config through a same-directory temp file, fsync, and
rename, so a failed write cannot truncate `trusted-server.toml`.
- Validate TLS certificates in both audit browser sessions; opting out now
requires `--danger-accept-invalid-certs`.
- Refuse a redirect that leaves the requested origin during verify unless
`--allow-cross-origin-redirect` is passed, so another origin's evidence
cannot satisfy `--strict`.
- Reject page patterns the runtime cannot compile before they reach the file,
through a new shared `compile_page_pattern` in core.
- Reject `creative_opportunities` declared in a form the line-based splice
cannot edit, instead of appending a duplicate table.
- Drop non-integer GPT sizes in the collector so one fluid size cannot fail
deserialization of the whole evidence payload.
- Escape control characters in page-controlled text written to the terminal.
`ts audit ad-templates generate` derived everything it wrote from a live,
page-controlled ad stack and never checked the result, so several reachable
inputs produced a config that cannot load. An unloadable `trusted-server.toml`
is not a degraded ad stack: `build_state` fails and the adapter answers every
route from the startup error router, so the whole site returns 500 once pushed.
Add a write-side gate that runs the candidate through `Settings::from_toml`,
the same `finalize_deserialized` chain the runtime uses at startup. It runs on
the `--dry-run` path too, so a clean preview is now evidence the config loads.
When the target config was already unloadable before the run, the gate reports
that as a warning instead of blaming this run, so a freshly bootstrapped file
carrying placeholder secrets can still be updated.
Close the three reachable paths at their source as well:
- Skip a scraped slot whose ad-unit path contains `{` or `}`. The path is a
template and there is no escape syntax, so a literal brace either fails
config load or is silently reinterpreted as a placeholder.
- Skip a slot whose div id normalizes to nothing (a wholly ephemeral id such
as a React SSR marker). An empty `div_id` fails config load, and as a runtime
prefix it would bind the slot to the first id-bearing element on the page.
- Refuse to create a `[creative_opportunities]` section with no GAM network id
rather than writing one that omits the required key. This is reachable
because the network id is only recovered from an all-digit leading segment,
which an MCM child-network path does not have.
Groundwork for discovering ad slots across a site's sections rather than from a single page. Nothing calls this yet; `run_update_slots` is unchanged. `AuditCollector` gains a defaulted `collect_pages` that streams each page to a sink, so every existing implementor keeps working and the caller can fold a page into its evidence and drop the DOM immediately instead of holding every serialization at once. The browser collector overrides it to launch Chrome once for the whole crawl: a cold start plus a fresh profile dominates the cost of a multi-page run, and the shared profile carries a bot-protection clearance cookie earned on the first page across the rest of the walk. Page discovery reads the hydrated DOM rather than the served markup, because an app-router page keeps its link graph in the framework payload — parsing raw HTML finds only a fraction of a site's sections. Sitemaps are fetched from inside the open page via `fetch` plus `DOMParser`, which inherits the session's cookies and Chrome's TLS fingerprint, gets transparent gzip and XML parsing, and so needs no new Rust dependency. `crawl_plan` turns links and sitemap entries into a bounded page set: one landing page and one article per section, ranked by whether navigation and the sitemap corroborate each other, capped by section and page budgets. Sections dropped for budget are reported rather than silently omitted. Same-origin is enforced on links and on sitemap entries alike, since a `Sitemap:` directive can name any host and the crawl carries operator cookies.
Template inference needs the set of observations per slot, not one snapshot: a single page cannot distinguish a literal ad-unit path from a templated one, so the divergence across pages is the only signal available. Add the table that holds it. Nothing calls this yet. Slots are keyed on the normalized div stem, since raw GPT div ids carry per-render framework hashes and would otherwise look like a new slot on every page. Three reconciliations happen here and nowhere else: - Formats union across pages. A size that renders only on article pages, such as a 300x600 rail, has to survive alongside the homepage's sizes; taking the first page's list would silently narrow the slot. - Divergent unit paths are retained as separate rows rather than collapsed, because discarding them is what makes templating impossible. - Network ids must agree. Two GAM networks in one crawl means the pages are not one property, so this is a hard error naming both rather than a guess that would bid against the wrong inventory. Pages that yield no slots are recorded rather than dropped, so a caller can recognise a bot challenge serving interstitials and refuse to write a half-empty config.
Adds the inference that turns literal scraped ad-unit paths into a
`{network_id}`/`{section}` template plus the section policy it depends on.
Nothing calls this yet.
A wrong template makes a publisher bid against inventory that does not exist,
which is worse than a narrow literal path, so this refuses rather than guesses.
Three rules carry that:
- `{network_id}` binds positionally to unit segment 0 and only when that segment
already equals the resolved id. Substring replacement would rewrite
`/123/sports123/home` into `/{network_id}/sports{network_id}/home`.
- Exactly one unit segment may vary. Zero proves nothing and stays literal; two
means the unit tracks a dimension the request path cannot supply, such as a
device or geo split, and is refused with that reason.
- Two pages must witness both a different derived section and a different unit
segment before anything is templated. Round-trip verification cannot supply
this: a single observation is reproduced equally well by a literal path, a
`{network_id}`-only template, and a `{section}` template, so only variation
distinguishes them.
`section_segment` is chosen by partitioning observations into pages that have a
section segment and pages that do not, the latter fixing `section_root`. An
index that cannot be witnessed is rejected, an unwitnessed root leaves the path
literal rather than guessing, and two indices that both fit are ambiguous and
template nothing.
Every accepted template is then replayed through the runtime's own
`render_gam_unit_path` and `derive_section` against every observation, so a
section slug the path cannot reproduce is caught and downgraded. `derive_section`
becomes public for exactly this: the check has to use the runtime's derivation
rather than a second implementation that could drift from it.
…terns Two gaps between what inference produces and what the writer could put on disk. Nothing calls the new code yet. `page_patterns` expands the paths a slot was observed on into globs. Each witnessed section contributes a pair, because one glob cannot cover both halves: `*` crosses `/` in this dialect, so `/news/*` matches `/news/a/b` but not the bare `/news` landing page, and emitting only the star form would silently drop the landing page from the slot. Nothing extrapolates past a witnessed section, so a crawl that never visited `/reviews` never claims it. `replace_key_in_section` can only rewrite a key that is already present, so it could not add `section_root` or `section_segment` to a config that predates them, which is every config a first templated run touches. Add `upsert_key_in_section`, which inserts immediately after the section header so the new key lands in the section's scalar block rather than after a subtable, where TOML would read it as belonging to that subtable instead. `splice_creative_slots` now takes the section keys as a struct rather than a bare network id. It omits `section_root` and `section_segment` entirely unless a slot actually templated: both are `deny_unknown_fields` additions, so writing them into a config that does not need them would make it unloadable by an older binary for no benefit.
Connects the crawl, evidence, inference and writer pieces: a bare
`ts audit ad-templates generate <url>` now samples the site's sections,
reconciles each slot across them, infers a `{section}` ad-unit template where
the evidence proves one, and writes the section policy alongside the slots.
The flow is collect root, plan the crawl from its links and sitemap, walk the
planned pages on one browser, fold each into the evidence table, infer, then
merge, render, splice and validate as before. Page patterns now come from the
sections a slot was actually seen on, so a slot scraped from one article serves
its whole section instead of that single URL.
Failure handling follows what the evidence can support. A page that will not
collect is reported and skipped, because one blocked page should not discard
the sections that worked. But if more than a quarter of crawled pages yield no
slots the run refuses outright: that is the signature of bot protection serving
challenge interstitials, and writing from it would silently narrow the
operator's slot set. Pages disagreeing about the GAM network id is likewise a
refusal rather than a guess.
A run that templates prints the deploy-ordering contract, because the config it
just wrote is not rollback-safe: `section_root` and `section_segment` are
`deny_unknown_fields` additions, so an older binary rejects the whole config and
serves an error on every route.
`--max-pages` and `--max-sections` bound the crawl; `--max-pages 1` restores
single-page behavior exactly, and an explicit `--page-pattern` still applies to
every slot and skips pattern inference. `run_update_slots` takes a request
struct, since a nine-argument signature could not absorb the crawl bounds.
Removes `default_page_pattern`, superseded by section-derived patterns, and
narrows the single-page `merge_slots` path to test scaffolding.
Publishers routinely serve a different GAM ad unit per device (`/network/desktop/news` against `/network/mobile/news`). A single-profile crawl cannot see that: it infers a template that is correct for the profile it used and silently wrong for every other impression, with nothing in the data to say so. This was the one unmitigated risk in the inference design. `--profiles desktop,mobile` walks every planned page once per profile, each with its own viewport and user agent, folding all of it into one evidence table. The user agent matters as much as the viewport here — ad stacks branch on it, so emulating size alone can still return desktop ad units on a phone-sized page. No new refusal logic was needed. Two profiles disagreeing produce two ad-unit paths for a single page, which is already the structural refusal inference applies to a unit that varies by something the request path cannot derive. The slot is still written, with its div and formats intact, but with no `gam_unit_path`: no path at all is better than one that is wrong on mobile, and the runtime falls back to the default unit rather than bidding on a unit that does not exist. Desktop-only stays the default, so the extra crawl is opt-in.
`ts audit ad-templates generate` had no documentation at all. Cover what the crawl does, what it writes, and the two things an operator cannot discover from the output alone. The first is when the command declines to generalize. A wrong ad-unit template makes a publisher bid against inventory that does not exist, so the command prefers a narrow literal path over a plausible guess, and the table says which situations produce which outcome — including the cases that fail the run outright, such as a crawl where bot protection served mostly challenge pages. The second is deploy ordering. A config carrying `section_root` or `section_segment` is not rollback-safe: a binary predating ad-unit templating rejects those keys, and the rejection fails the whole configuration load rather than just the ad-template section, so every route serves an error. Ship the template-aware binary first, push second, and do not roll back while that config is live.
A static configuration choice should not emit a warning during every request-time settings load. Removing the log keeps runtime output focused without changing validation or creative processing behavior. Resolves: #1089
# Conflicts: # crates/trusted-server-adapter-fastly/src/app.rs
# Conflicts: # crates/trusted-server-adapter-fastly/src/app.rs # crates/trusted-server-core/src/config.rs # scripts/template-cache-local-test.sh # trusted-server.example.toml
# Conflicts: # docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md
…spec/auction-timeline-offsets
# Conflicts: # crates/trusted-server-cli/src/commands/audit/mod.rs # crates/trusted-server-cli/src/commands/config/init.rs # crates/trusted-server-core/src/config.rs # crates/trusted-server-core/src/integrations/js_asset_proxy.rs # crates/trusted-server-core/src/integrations/mod.rs # crates/trusted-server-core/src/platform/test_support.rs # docs/guide/getting-started.md # docs/superpowers/specs/2026-04-01-js-asset-proxy-design.md # docs/superpowers/specs/2026-06-22-ts-audit-js-asset-proxy-config-design.md # trusted-server.example.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
September release candidate
This PR is an integration branch for release-candidate validation. Do not merge it into
main.Already included through main:
Included PRs