Skip to content
@spire-rs

Spire

Crawl the web from Rust: the Spire framework and the libraries it's built on.
README.md

Crates

Crate Version What it does
spire spire The crawler and scraper framework: frontiers, HTTP and browser backends, typed routing and extractors.
uagent uagent Browser profiles with user agents and matching navigation headers for Chrome, Edge, Firefox and Safari.
robotxt robotxt Parses and builds robots.txt files following RFC 9309, with the crawl-delay and sitemap extensions.
sitemapo sitemapo Parses and builds text and XML sitemaps and sitemap indexes, with the image, video, news and hreflang extensions.
countio countio Byte counting for readers and writers, for std::io, tokio and futures.

Getting started

The Spire Book is the guide: handling pages, discovering them, fetching over HTTP or with a browser, running and resuming a crawl, and extending spire. The API reference lives on docs.rs.

Pinned Loading

  1. spire spire Public

    🗼 The flexible crawler and scraper framework for Rust: route pages to typed handlers, crawl over HTTP or real browsers, built on tokio.

    Rust 5

  2. kit kit Public

    📚 The robots.txt and sitemap protocols for Rust: parse and build robots.txt files per RFC 9309, and text and XML sitemaps with the image, video, news and hreflang extensions.

    Rust 3 1

  3. countio countio Public

    💡 Byte counting for Rust I/O: wrap any std, tokio or futures reader or writer to track the bytes read and written.

    Rust 6

  4. uagent uagent Public

    🌐 Browser profiles for Rust: user agents, matching navigation headers and client hints, with deterministic selection for web crawlers.

    Rust 1

Repositories

Showing 5 of 5 repositories

Top languages

Loading…

Most used topics

Loading…