Skip to content

Remove donor leftovers: twin engines' --mode help, access badge, issue templates - #14

Merged
jehrr merged 3 commits into
mainfrom
fix/donor-leftovers
Sep 28, 2026
Merged

jehrr merged 3 commits into
mainfrom
fix/donor-leftovers

Conversation

@jehrr

@jehrr jehrr commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Leftovers from the repos etsy-scraper was bootstrapped from (mediamarkt, farfetch) that describe another site as if it were this one.

Fixed

  • selenium_scraper.py, puppeteer_scraper.py --help (user-visible) — --mode said "listing (default) or product. product reads one /product/ page and adds brand, EAN…", which is MediaMarkt's product mode. All three engines take listing|product|shop and Etsy's detail page is /listing/<id>. The twins now carry the Playwright engine's help text and module-docstring mode list verbatim.
  • README access badge — "needs a residential exit" → "needs Scraping Browser API". The README's own table measures Playwright Chromium via a residential US exit as 403, t=bv, says "a residential exit was not sufficient on its own here", and names the Scraping Browser API as the path that works. Badge order was already the family order.
  • bug_report.yml — example --out grills → --out mugs; expected placeholder was Farfetch's "96 products, as the README says a category page yields" → the README's 184-row 3-page search run.
  • site_changed.yml — "the parser tries JSON-LD first", an original_price or lowest_price_30d overlay option (no such column; TROUBLESHOOTING says so) and "DOM price confirmation below 90%" → data-listing-id primary anchor with JSON-LD enriching (8 entries for 64 tiles, from product_parser.py), and the per-page-kind CONFIRMATION_FLOOR.
  • SECURITY.md — "how Akamai or reCAPTCHA behave" → DataDome.
  • output_writer.py — comment's example of a no-listings page was MediaMarkt's /de/category/tv-audio-202.html → a taxonomy hub (TROUBLESHOOTING's own wording).
  • smoke_test.py — privacy-pattern comment said "MediaMarkt's pages embed…" without naming it a sibling; now "On a sibling site (mediamarkt-scraper)…".
  • CONTRIBUTING.md — "the registration form" → "a registration or login form".

Kept, deliberately

  • README "Unlike farfetch-scraper, where a plain local Chromium… returned 96 products" — names the sibling as a sibling.
  • diff_runs.py "The sibling repos guard this with source… mediamarkt.at against mediamarkt.de" and product_parser.py "Unlike … the sibling mediamarkt-scraper" / "unlike in farfetch-scraper" — explicit provenance with a lesson.
  • smoke_test.py: detect_page_state("<html><body>MediaMarkt</body></html>") (a deliberately foreign page asserting "built out of none of Etsy's assets is blocked"), is_supported_host("https://www.amazon.de/x") (another marketplace is refused), and the "dash-cents is a MediaMarkt convention, measured absent on Etsy" pin — all intentional.
  • akamai entry in the vendor marker table; captcha_solver farfetch notes; craigslist .gitignore note; released CHANGELOG sections.
  • Repo description/topics: no donor words, unchanged.

Verification

python3 smoke_test.py exit 0 before and after (same two engine groups skipped locally); ci_checks.py --secret-check and --help-check --sample-check exit 0; issue templates parse as YAML.

🤖 Generated with Claude Code

jehrr and others added 3 commits September 28, 2026 19:48
…lates

- selenium/puppeteer: --mode help and docstring described MediaMarkt's
  "/product/ page with brand, EAN"; now mirror Playwright's listing /
  product (/listing/<id>) / shop text.
- README: access badge "needs a residential exit" -> "needs Scraping
  Browser API" (README measures residential alone as refused, t=bv).
- bug_report.yml: --out grills -> mugs; Farfetch's "96 products" -> the
  README's 184-row run.
- site_changed.yml: JSON-LD-first / lowest_price_30d / 90% options
  replaced with data-listing-id anchor and per-page-kind floors.
- SECURITY.md: Akamai -> DataDome. output_writer.py: MediaMarkt
  tv-audio URL example -> a taxonomy hub. CONTRIBUTING: "the registration
  form" generalised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tern comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- SECURITY.md said the project has no releases or version tags; it has both
  (gh release list). Supported versions now names the latest release and main,
  in the wording bbb-scraper and foodpanda-scraper already use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr force-pushed the fix/donor-leftovers branch from 8e80775 to a6e97d2 Compare September 28, 2026 19:51
@jehrr
jehrr merged commit d8660fa into main Sep 28, 2026
7 checks passed
@jehrr
jehrr deleted the fix/donor-leftovers branch September 28, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant