A complete Python web scraping kit with 3 practical strategies: BeautifulSoup for static HTML, Selenium for legacy JavaScript-heavy pages, and Playwright for modern, asynchronous scraping. Includes rate limiting, retry with exponential backoff, User-Agent rotation, and export to CSV, JSON, and Parquet.
- ✅ 3 scraping strategies — static, Selenium, Playwright
- ✅ Configurable rate limiting to avoid blocks
- ✅ Automatic retry with exponential backoff
- ✅ User-Agent rotation for ethical scraping
- ✅ Flexible export — CSV, JSON, and Parquet
- ✅ Polars-based processing — faster than Pandas
- ✅ Structured logging for debugging
| Tool | When to use |
|---|---|
| BeautifulSoup + Requests | Static HTML, simple sites |
| Selenium | JavaScript-heavy sites, browser automation |
| Playwright | Modern, asynchronous, multi-browser support |
| Polars | Ultra-fast data processing |
| Parquet | Efficient columnar storage |
web-scraper-toolkit/
├── scrapers/
│ ├── bs4_scraper.py # BeautifulSoup (static HTML)
│ ├── selenium_scraper.py # Selenium (JavaScript)
│ └── playwright_scraper.py # Playwright (asynchronous)
├── utils/
│ ├── exporters.py # CSV/JSON/Parquet exporters
│ └── rate_limiter.py # Configurable rate limiting
├── examples/
│ ├── quotes_bs4.py # Scraping quotes.toscrape.com
│ ├── books_selenium.py # Scraping books.toscrape.com
│ └── news_playwright.py # News scraping (async)
├── tests/
├── .github/workflows/ci.yml
├── requirements.txt
├── requirements-dev.txt
└── README.md
git clone https://github.com/LacerdaTraderCode/web-scraper-toolkit.git
cd web-scraper-toolkit
python -m venv venv
source venv/bin/activate # Linux/Mac
# venv\Scripts\activate # Windows
pip install -r requirements.txt
# Install browsers for Playwright (first time only)
playwright install chromiumpython -m examples.quotes_bs4Extracts quotes from quotes.toscrape.com, saves to CSV and Parquet.
python -m examples.books_seleniumNavigates the books.toscrape.com catalog simulating a real user.
python -m examples.news_playwrightCollects headlines asynchronously (faster than Selenium).
| Tool | Time | CPU | Recommended for |
|---|---|---|---|
| BeautifulSoup | ~15s | Low | Simple sites |
| Selenium | ~90s | High | Legacy JS-heavy sites |
| Playwright (async) | ~25s | Medium | Modern projects |
pip install -r requirements-dev.txt
ruff check . && ruff format --check .
pytest -vBrowser drivers and HTTP sessions are mocked, so the test suite needs no browser or network access. The same checks run on every push and pull request via GitHub Actions.
This toolkit is for educational purposes. When scraping:
- ✅ Respect the site's
robots.txt - ✅ Use rate limiting to avoid overloading servers
- ✅ Correctly identify your User-Agent
- ✅ Respect the site's Terms of Use
- ❌ Do not collect personal data without consent
- Python 3.11 or higher
Wagner Lacerda — Senior Software Engineer | Python, Backend, AI Apps, Automation & Systems
📍 Rio Grande do Sul, Brazil
Distributed under the MIT license. See LICENSE for more details.