steelsmart.shop robots.txt

Steelsmart.shop has robots.txt file. Http response code is 200.
Robots size is small. It is about 110 kb and 39 rows.

steelsmart.shop has sitemap file https://steelsmart.shop/sitemap.xml. Robots.txt for User-Agent=* contains only disallow directives.

Main host for steelsmart.shop is https://steelsmart.shop.

steelsmart.shop has separate rules for googlebot and yandexbot and also has rules for all bots (User-Agent=*).
Array
(
    [googlebot] => Array
        (
            [disallow] => Array
                (
                    [0] => /?
                    [1] => /notfound/
                    [2] => /admin
                    [3] => /index.php
                    [4] => /emarket/addToCompare
                    [5] => /emarket/basket
                    [6] => /emarket/gateway
                    [7] => /go-out.php
                    [8] => /cron.php
                    [9] => /filemonitor.php
                    [10] => /search
                )

        )

    [yandex] => Array
        (
            [disallow] => Array
                (
                    [0] => /?
                    [1] => /notfound/
                    [2] => /admin
                    [3] => /index.php
                    [4] => /emarket/addToCompare
                    [5] => /emarket/basket
                    [6] => /emarket/gateway
                    [7] => /go-out.php
                    [8] => /cron.php
                    [9] => /filemonitor.php
                    [10] => /search
                )

        )

    [*] => Array
        (
            [disallow] => Array
                (
                    [0] => /?
                    [1] => /notfound/
                    [2] => /admin
                    [3] => /index.php
                    [4] => /emarket/addToCompare
                    [5] => /emarket/basket
                    [6] => /emarket/gateway
                    [7] => /go-out.php
                    [8] => /cron.php
                    [9] => /filemonitor.php
                    [10] => /search
                )

            [host] => https://steelsmart.shop
            [sitemap] => Array
                (
                    [0] => https://steelsmart.shop/sitemap.xml
                )

            [crawl-delay] => 3
        )

)

User-Agent: Googlebot
Disallow: /?
Disallow: /notfound/
Disallow: /admin
Disallow: /index.php
Disallow: /emarket/addToCompare
Disallow: /emarket/basket
Disallow: /emarket/gateway
Disallow: /go-out.php
Disallow: /cron.php
Disallow: /filemonitor.php
Disallow: /search

User-Agent: Yandex
Disallow: /?
Disallow: /notfound/
Disallow: /admin
Disallow: /index.php
Disallow: /emarket/addToCompare
Disallow: /emarket/basket
Disallow: /emarket/gateway
Disallow: /go-out.php
Disallow: /cron.php
Disallow: /filemonitor.php
Disallow: /search

User-Agent: *
Disallow: /?
Disallow: /notfound/
Disallow: /admin
Disallow: /index.php
Disallow: /emarket/addToCompare
Disallow: /emarket/basket
Disallow: /emarket/gateway
Disallow: /go-out.php
Disallow: /cron.php
Disallow: /filemonitor.php
Disallow: /search

Host: https://steelsmart.shop
Sitemap: https://steelsmart.shop/sitemap.xml
Crawl-delay: 3
  • код 301 http://steelsmart.shop/robots.txt
  • код 200 https://steelsmart.shop/robots.txt