PHP Library for robots.txt parsing. Latest library version is available for download from github.
Example of library using:
$robotsTxtContent = file_get_contents('http://example.com/robots.txt'); $parser = new RobotsTxtParser($robotsTxtContent); print_r($parser->getRules());