This is a small package to make parsing robots.txt rules easier.
-
Updated
Mar 31, 2017 - PHP
#
disallowed-paths
Here is 1 public repository matching this topic...
Add this topic to your repo
To associate your repository with the disallowed-paths topic, visit your repo's landing page and select "manage topics."