Dailyexpress.com.my has robots.txt file. Http response code is 200.
Robots size is very small. It is about 25 kb and 11 rows.

Robots.txt for User-Agent=* contains only disallow directives.

Host directive is not presented for domain dailyexpress.com.my in robots.txt file.
Array
(
    [*] => Array
        (
            [crawl-delay] => 10
            [disallow] => Array
                (
                    [0] => /cgi-bin/
                    [1] => /cgi/
                    [2] => /thumbs/
                    [3] => /movie/
                    [4] => /admin/
                    [5] => /manage/
                )

        )

)
# Robots.txt

# Disallow the crawling for cgi-bin dir
User-agent: *
Crawl-delay: 10
Disallow: /cgi-bin/
Disallow: /cgi/
Disallow: /thumbs/
Disallow: /movie/
Disallow: /admin/
Disallow: /manage/
  • код 200 http://dailyexpress.com.my/robots.txt