Web.de has robots.txt file. Http response status is 200.
Robots size is very small. It is about 21 kb and 9 rows.

Robots.txt for User-Agent=* contains only disallow directives.

Host directive is not presented for domain web.de in robots.txt file.
Array
(
    [*] => Array
        (
            [disallow] => Array
                (
                    [0] => /test/
                )

        )

    [googlebot-news] => Array
        (
            [disallow] => Array
                (
                    [0] => /
                    [1] => /magazine/*/thema/
                )

            [allow] => Array
                (
                    [0] => /magazine/
                    [1] => /amp/
                    [2] => /$
                )

        )

)
User-agent: *
Disallow: /test/

User-agent: googlebot-news
Disallow: /
Disallow: /magazine/*/thema/
Allow: /magazine/
Allow: /amp/
Allow: /$

#https://web.de/robots.txt
  • код 301 http://web.de/robots.txt
  • код 200 https://web.de/robots.txt