Newsru.co.il has robots.txt file. Http response status is 200.
Robots size is very small. It is about 36 kb and 15 rows.

Robots.txt for User-Agent=* contains only disallow directives.

Host directive is not presented for domain newsru.co.il in robots.txt file.
Array
(
    [*] => Array
        (
            [disallow] => Array
                (
                    [0] => /search.html/*
                    [1] => /bin
                    [2] => /cgi
                    [3] => /css
                    [4] => /plain
                    [5] => /arch
                    [6] => /*_save.html$
                    [7] => /*_blog.html$
                    [8] => /*_print.html$
                    [9] => /*_email.html$
                )

        )

    [yandexnews] => Array
        (
            [allow] => Array
                (
                    [0] => /bin/export/yandex/
                    [1] => /pict/
                )

            [disallow] => Array
                (
                    [0] => /
                )

        )

)
User-agent: *
Disallow: /search.html/*
Disallow: /bin
Disallow: /cgi
Disallow: /css
Disallow: /plain
Disallow: /arch
Disallow: /*_save.html$
Disallow: /*_blog.html$
Disallow: /*_print.html$
Disallow: /*_email.html$

User-agent: YandexNews
Allow: /bin/export/yandex/
Allow: /pict/
Disallow: /
  • код 301 http://newsru.co.il/robots.txt
  • код 200 https://www.newsru.co.il/robots.txt