new-rutor.org robots.txt
New-rutor.org has robots.txt file. Http response code is 200.
Robots size is small. It is about 122 kb and 23 rows.
new-rutor.org has great amount of sitemap files — 12: http://new-rutor.org/sitemaps/new-rutor.org/new.xml, http://new-rutor.org/sitemaps/new-rutor.org/1.xml, http://new-rutor.org/sitemaps/new-rutor.org/2.xml and others. Robots.txt for User-Agent=* contains only disallow directives.
Robots size is small. It is about 122 kb and 23 rows.
new-rutor.org has great amount of sitemap files — 12: http://new-rutor.org/sitemaps/new-rutor.org/new.xml, http://new-rutor.org/sitemaps/new-rutor.org/1.xml, http://new-rutor.org/sitemaps/new-rutor.org/2.xml and others. Robots.txt for User-Agent=* contains only disallow directives.
Array
(
[*] => Array
(
[crawl-delay] => 0.3
[disallow] => Array
(
[0] => /new11/
[1] => /newCache/
[2] => /topCache/
[3] => /inc/checkcache/
[4] => /inc/recache/
[5] => /redirect/
[6] => /parse/d.rutor.org/download/
)
[clean-param] => Array
(
[0] => osCsid&recache&checkcache
)
[host] => new-rutor.org
[sitemap] => Array
(
[0] => http://new-rutor.org/sitemaps/new-rutor.org/new.xml
[1] => http://new-rutor.org/sitemaps/new-rutor.org/1.xml
[2] => http://new-rutor.org/sitemaps/new-rutor.org/2.xml
[3] => http://new-rutor.org/sitemaps/new-rutor.org/3.xml
[4] => http://new-rutor.org/sitemaps/new-rutor.org/4.xml
[5] => http://new-rutor.org/sitemaps/new-rutor.org/5.xml
[6] => http://new-rutor.org/sitemaps/new-rutor.org/6.xml
[7] => http://new-rutor.org/sitemaps/new-rutor.org/7.xml
[8] => http://new-rutor.org/sitemaps/new-rutor.org/8.xml
[9] => http://new-rutor.org/sitemaps/new-rutor.org/9.xml
[10] => http://new-rutor.org/sitemaps/new-rutor.org/10.xml
[11] => http://new-rutor.org/sitemaps/new-rutor.org/11.xml
)
)
)
User-agent: * Crawl-delay: 0.3 Disallow: /new11/ Disallow: /newCache/ Disallow: /topCache/ Disallow: /inc/checkcache/ Disallow: /inc/recache/ Disallow: /redirect/ Disallow: /parse/d.rutor.org/download/ Clean-param: osCsid&recache&checkcache Host: new-rutor.org Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/new.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/1.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/2.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/3.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/4.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/5.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/6.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/7.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/8.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/9.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/10.xml Sitemap: http://new-rutor.org/sitemaps/new-rutor.org/11.xml
- код 200 http://new-rutor.org/robots.txt