Htmlbook.ru has robots.txt file. Http response status is 200.
Robots size is very small. It is about 52 kb and 21 rows.

Robots.txt for User-Agent=* contains only disallow directives.

Host directive is not presented for domain htmlbook.ru in robots.txt file.
Array
(
    [*] => Array
        (
            [disallow] => Array
                (
                    [0] => /example/
                    [1] => /includes/
                    [2] => /misc/
                    [3] => /modules/
                    [4] => /profiles/
                    [5] => /scripts/
                    [6] => /themes/
                    [7] => /node
                    [8] => /cron.php
                    [9] => /update.php
                    [10] => /xmlrpc.php
                    [11] => /boost_stats.php
                    [12] => /admin/
                    [13] => /comment/reply/
                    [14] => /contact
                    [15] => /logout/
                    [16] => /node/add/
                    [17] => /search/
                )

        )

    [yadirectbot] => Array
        (
        )

)
User-agent: *

Disallow: /example/
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/

Disallow: /node
Disallow: /cron.php
Disallow: /update.php
Disallow: /xmlrpc.php
Disallow: /boost_stats.php

Disallow: /admin/
Disallow: /comment/reply/
Disallow: /contact
Disallow: /logout/
Disallow: /node/add/
Disallow: /search/

User-Agent: YadirectBot
Disallow:
  • код 200 http://htmlbook.ru/robots.txt