Zendriver Example

A passive web scraper built on Zendriver and the Chrome DevTools Protocol. You use the browser as normal, and the scraper reads the JSON responses and page content in the background.

What it shows

Most scrapers drive the browser: they click, wait, and parse. This one only listens. The Chrome DevTools Protocol, or CDP, sends an event for each network request, so the scraper can read every JSON response that the page itself loads:

1. On Network.requestWillBeSent, the scraper notes the ID of each request with Accept: application/json.
2. On Network.loadingFinished, it fetches the body of a noted request with Network.getResponseBody and prints it.
3. A loop watches tab.url. When the page changes to a known URL, it reads the HTML with a CSS selector.

The page does the requests, with its own cookies and headers. The scraper never sends one.

A request ID stays in a TTLCache for 30 seconds, so the IDs of requests that never finish do not pile up.

Run it

1. Install mise and Chrome or Chromium.

2. Install the tools and dependencies, then start the scraper:

   git clone git@github.com:tabmadi/zendriver-example.git
   cd zendriver-example
   mise install
   mise run setup
   mise run run

3. A browser opens on https://httpbin.org/, and the scraper prints the page title. Click an endpoint in the page and run it. The scraper prints each JSON response.

Variable	Description	Default
CHROME_PATH	The Chrome or Chromium binary	the one that Zendriver finds

The browser profile is in ~/.config/google-chrome-nodriver, so a sign-in stays between runs.

How it works

Function in src/main.py	Does
on_request	Notes the ID of each JSON request
on_loading_finished	Fetches and prints the body of a noted request
parse_html	Prints the text of the .title element
start	Starts the browser, adds the two CDP handlers, and watches the URL

What is deliberately missing

• Storage: the scraper prints. A real collector writes to a file or a queue.
• Page rules: only one URL has an HTML parser. A real collector maps each URL pattern to its own parser.
• A stop condition: the loop runs until you stop the process.

Tasks

Task	Does
mise run setup	Installs the locked dependencies and the git hooks
mise run run	Opens the browser and prints what the scraper reads
mise run lint	Runs lint:py, lint:md, and lint:secrets
mise run format	Applies every formatter
mise run test	Runs the tests
mise run check	Runs lint and test. Run it before you finish a change
mise run ci	Runs exactly what CI runs. The pre-push hook runs it too

License

Apache License 2.0. See LICENSE.