Perl stand alone robots.txt parser. With real world experience crawling the web. Googlebot like interpretation of robots.txt.
-
Notifications
You must be signed in to change notification settings - Fork 4
Perl stand alone robots.txt parser. With real world experience crawling the web. Googlebot like interpretation of robots.txt.
License
randomstring/ParseRobotsTXT
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Perl stand alone robots.txt parser. With real world experience crawling the web. Googlebot like interpretation of robots.txt.
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published