Robots.txt php parsing
PHP Library for robots.txt parsing. Latest library version is available for download from github.
Example of library using:
$robotsTxtContent = file_get_contents('http://example.com/robots.txt');
$parser = new RobotsTxtParser($robotsTxtContent);
print_r($parser->getRules());
Note: please use v2.0+ (the latest master) which is significantly faster.
The latest release notes:
- Added RobotsTxtValidator to check isUrlAllow, isUrlDisallow.
- RobotsTxtParser works by same rules but is more than 100 times faster.
