Repository navigation
Skip Explorer renders for bulk training crawlers - #29
Merged
Merged
Conversation
droplister
marked this pull request as ready for review
September 21, 2026 02:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Training crawlers currently trigger full Explorer renders and their API reads. Add GPTBot and ClaudeBot to the existing crawler restriction and publish the same names in robots.txt. Keep OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, Google, Bing, and social previews allowed.
For September 19–20 UTC, Cloudflare's sampled HTTP analytics counted 495,944 matching external requests to xcp.io (495,825 successful). This is a workload opportunity, not an invoice saving: a middleware rejection still incurs a Worker invocation, and the long-term effect of robots.txt has not yet been measured. No cache TTL, page content, navigation behavior, or data freshness changes.
Validation: root npm run check passes. All 16 isolated browser/accessibility tests pass, including targeted 403 responses, readable robots.txt, and rendered pages for eight search/user/preview agents. Production build validation is recorded in the accompanying audit.
Bot roles: https://developers.openai.com/api/docs/bots and https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler . The tradeoff is intentional: these two bulk training agents lose access; search and user-initiated agents retain it.