Skip to content

fix: first-run fill rate (limiter, cooldown policy, re-run keep, Email Verify, reasons) - #182

Merged
velixio merged 49 commits into
mainfrom
fix/first-run-fill-rate
Nov 19, 2026
Merged

velixio merged 49 commits into
mainfrom
fix/first-run-fill-rate

Conversation

@velixio

@velixio velixio commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Builds on #175 (train/2026-11-19) and #178 (the benchmark). The diff includes both until they merge. Fixes the causes of the low first-run fill rate of the sample workbook found in #178. Commits are small and test first.

What changed

Browser and providers

  • Process-wide browser limiter sized from BROWSER_MAX_CONCURRENCY; the queue wait is bounded by the run context and does not count against the provider timeout.
  • Every provider declares what a timeout from it means (upstream, target site, local). Target-site providers get a 30 s budget; deep_scraper and website_scraper walk a site once per run, not once per column.

Cooldown

  • One policy in internal/cooldown for both write paths: a rate limit keeps the 300 s cooldown; a timeout cools down only after 3 consecutive timeouts; a timeout of a target-site provider never cools it down; concurrent timeouts count as one event. Migrations 00006 and 00007 (consecutive timeouts, last timeout time).
  • Search backends are left alone while they cool down; WebSearch honours the keyless backoff.

Cells and re-runs

  • A failed re-run keeps the cell's earlier value and notes the failed attempt.
  • Empty cells say why (no LLM key, rate limit, cooling down, browser service unavailable).
  • Search in records matches field values, not JSON keys.

Email Verify

  • Reads the row's Email cell through the column graph, runs after Email, stores the verifier's answer, never reports a guess as verified.
  • Root cause of the zero fill: the cascade's SMTP verifier was built with the empty Deps of NewDeps, so no SMTP probe was ever sent (fix 9efeb94; test sends a probe through the cascade).

Outcome and status model (internal/cellstate)

  • A cell ends as filled, no_data, needs_setup or error. A run ends as complete (no errors; cells with no data are not failures), completed_with_errors (some real errors) or failed (nothing completed). Run history and the status message carry the counts. The web app shows no_data and needs_setup as neutral outcomes and uses red only for real errors.

Demo and benchmark

  • The seed description no longer claims every column is zero-key; the research column is marked as needing an LLM key.
  • scripts/bench/first-run-fill-rate.sh reports four separate rates (all filled; excluding guessed emails; excluding key-requiring columns; excluding environment-limited columns). A cell is filled only when it carries an informative value.

Numbers (200 enrichment cells, zero keys, two fresh installs each)

Rate Original Round 1 Round 2 (this head)
All filled 53.5% / 44.0% 60.5% / 67.0% 82.0% / 82.5%
Excluding guessed emails 44.5% / not computed 57.5% / 65.5% 77.0% / 77.0%
Excluding key columns (175 cells) 61.1% / 50.3% 69.1% / 76.6% 93.7% / 94.3%
Excluding environment-limited not computed not computed 82.0% / 82.5% (no column left out)

Round 2 Email Verify statuses (install 1 / 2): valid 3 / 5, catch_all 19 / 19, invalid 1 / 0, empty (no_mx) 2 / 1. Without the catch_all cells the all-filled rate would be 72.5% / 73.0% (from the cell files, not a script number). Worker provider timed out warnings: 3 and 7 (round 1: 137 and 144). The filled rule was tightened between rounds, so round 1 is not strictly like for like. Remaining empty cells: HQ Location (25, needs an LLM key), phone 6 / 5, size 2 / 2, hiring 1 / 2, Email Verify 2 / 1. Details, per-column tables and evidence: docs/benchmarks/2026-11-first-run-fill-rate.md.

Tests run (apps/backend-go, head: the last commit on the branch)

  • gofmt -l . (clean), go vet ./..., go vet -tags integration ./..., go build ./...: clean.
  • go test -count=1 on the 37 Go packages changed on the branch: all ok.
  • go test -tags integration -count=1 -p 2 on the same packages except internal/api, against a fresh migrated database and Redis DB 9: all ok.
  • go test -tags integration -count=1 -timeout 40m ./internal/api (595 s): 2 tests fail, TestIntegrationRoutesMatchFixtures and TestSettingsRoutesMatchRecordedFixtures, both on the integrations list ($.integrations[2] is Attio, the fixture expects Airtable). Not caused by this branch: the branch does not touch those fixtures or code, and the failing CI job on this PR shows the same two. The fix is in train/2026-11-19 (commits 205f179 and 31190f4 add Attio, HeyReach and lemlist to the recorded fixtures); this branch only carries an older partial merge of the train. It goes away when Release train 2026-11-19: PRs #166 to #181 combined #175 merges and this branch is updated. All other api tests passed, including TestPersonWSNoInternalErrorTextLeaked.
  • apps/web: bun install --frozen-lockfile, bun run lint, bun test tests (208 pass), bun run build, bun run test:a11y (52 passed): all clean.
  • Browser service (apps/browser-service): the CI check had failed on a ruff import-order error in tests/test_search_backend_health.py; fixed in a new commit. Then uv run --frozen ruff check . and ruff format --check . clean, uv run --frozen pytest -q 249 passed.
  • Benchmark: scripts/bench/first-run-fill-rate.sh twice in a row on commit baa6b66 (docs only after that).

Not verified

  • The two integration failures above are not fixed here (inherited from the train, outside this branch).
  • Results come from two installs from one network on one day; two installs are not a confidence interval. SMTP results depend on port 25 being open on the host's network.
  • No run with an LLM key, paid keys or a proxy; no re-run in round 2; no accuracy audit of the filled values; catch_all verify cells count as filled by the script's rule.
  • Why the remaining phone and size cells are empty was read from provider messages, not traced per call (for example, whether a page over the 1 MiB body cap holds a phone).

…tegy

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Nine declarative manifests (Anymail Finder, Bouncer, MillionVerifier,
Findymail, ContactOut) with docs-derived fixtures and one generic test,
a wiring test across catalog, workbook and settings registries, and a
rewritten manifest format guide.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Replace every native confirm/prompt with useConfirm()/usePrompt() (focus managed, destructive style, named objects). Fix target-size at shared level via --gtm-target-min (accounts for the shell's CSS zoom), fix contrast and unnamed controls, add six routes to the axe gate, and empty the baseline.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…ls; changelog for the Go cutover

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…_verify words

response.value_maps translates a mapped value with an explicit default,
request.omit_empty drops empty headers, query parameters and body keys,
and values substituted into request.url are escaped. The three verifier
manifests now emit only valid, invalid, catch_all, risky, unknown.
Adds Settings key cards for LeadMagic and Prospeo.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Audience and automation delivery adds a lead to a HeyReach (LinkedIn) or lemlist (email) campaign with the workspace's own API key. Workbook output columns are not part of this change.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…ints

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Upserts each member as an Attio person matched on email and links or creates its company by domain, with the workspace's own token.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… train/2026-11-19

# Conflicts:
#	apps/backend-go/internal/settings/registry.json
…11-19

# Conflicts:
#	apps/backend-go/internal/settings/registry.json
…rain/2026-11-19

# Conflicts:
#	apps/web/src/components/command-menu-content.tsx
…ucing script

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…eep the queue wait off the provider timeout

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…s, one shared policy for both write paths

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… notes the failed attempt

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… column graph and stores the verifier's answer

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… cooling down, browser service

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…olumn, link Email Verify to Email

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…luding key-requiring columns as separate numbers

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…mbers, remaining causes

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
@velixio
velixio requested a review from debpalash as a code owner November 19, 2026 00:41
@github-actions

Copy link
Copy Markdown

All contributors on this pull request have signed the OpenGTM CLA. Thank you!

Agreement: OpenGTM CLA 1.0

@github-actions github-actions Bot added the cla-signed All contributors signed the CLA label Nov 19, 2026
… (upstream, target site, local)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…n; concurrent timeouts count as one event

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Drives the built app against a running stack (setup, Get started, sample run,
every route in light and dark, one action per area), records API statuses,
console output and error states, and writes screenshots and findings.json.
Separate config and script; not part of test:a11y and not run in CI.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…nalytics trend contrast

The mocked suite answered unmatched endpoints with [], so /overview, /analytics and
/settings rendered their error states (or a skeleton) and were never audited with
content. Add shapes captured from the real API, fail any page that shows an
error state, and wait for the query retry before auditing. The analytics page,
now rendered with data, exposed low-contrast trend text (emerald-500 and red-400
on white); use the 700/400 pair.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Loading the sample completes step 3, and done steps carry no actions, so
Open it from step 3 pointed at nothing. The workbook link is step 4's.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
The list search cast accepted_fields to text, so domain or name (or any
short term) matched every record. Search the values with jsonb_each_text.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… and website_scraper walk a site once per run, not once per column

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…, needs_setup, error)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…d; cells are counted as filled, no_data, needs_setup or error

Provider failures that the cell used to hide as no_data (an HTTP 5xx, a missing key) now say so
(provider_error, provider_key_missing). Retry passes pick up error cells only. The receipt stores the
counts. Tests that encode the old status and count semantics are updated in the next commit.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…statuses

The wbexec parity vectors (testdata/wbexec_vectors.json) are unchanged. They record Python's row_result,
which counted every cell without a value as an error: provenance_error_cell (1), condition_skip_blocks_dependent
(2, one of them a skip) and email_verify_no_value_no_cascade (1). Go now splits those into no_data, needs_setup and
error, so the harness compares Python's count with their sum. The run_integration and receipt tests that asserted
errors=2 / status=failed for cells that found no data, and the exact receipt key set, now assert the new counts.

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… for real errors

The run label reads '86 filled · 89 no data · 25 need setup'. The Run needs attention banner shows only for failed or completed_with_errors workbooks. Cell icons: no data is neutral, needs setup links to Settings. A shared outcome table is checked by both the Go and the TS classifier.
…message

The run-history result returns no_data and needs_setup when the receipt has them (older receipts keep their shape); the workbook_status message carries them too.
…ours the keyless backoff

ddgs builds a fresh engine set per query and swallows non-200 answers, so a backend that had just said 429 or 403 was asked again on the next query and a dead one cost a full timeout each time. The browser service now remembers each backend's last answer and skips it while it cools down; when all are cooling down the query fails at once as rate_limited. WebSearch now shares the Go keyless backoff. No change to how requests look or where they come from.
docs/operations/search-backends.md says the honest answer is your own search key or SearXNG.
… of NewDeps, so no probe was ever sent

NewDeps registered the SMTP verifier already bound to the Deps of that moment, before NewLive set NewSession, SMTPDisabled and the HELO identity. Every cascade verification therefore ended 'unknown / unreachable' without opening a connection (port 25 was open; a direct probe returns catch_all or invalid). The verifier is now registered unbound and the cascade binds it, and checks availability, against the Deps it runs with.

Also: the probe records where it stopped (Why), and a run of distinct mail servers that all fail to answer while none does marks port 25 as blocked (smtp_port_blocked). An unknown verdict is no longer stored as the cell value: it is no_data (email_inconclusive), needs_setup (blocked port 25, with the text to use a verification provider key; or nothing to ask), or an error (timeout, open breaker).
…st environment-limited columns

Email Verify 'unknown' is no fill. Columns whose cells say SMTP port 25 is blocked are reported as environment-limited, with their own rate (excluding_environment_limited) beside the other three.
…h installs

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…alth test (ruff)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
@velixio
velixio merged commit 5a99feb into main Nov 19, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed All contributors signed the CLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant