Haz.de has robots.txt file. Http response code is 200.
Robots size is small. It is about 113 kb and 39 rows.

haz.de has sitemap file https://www.haz.de/sitemap/index. Robots.txt for User-Agent=* contains only disallow directives.

Host directive is not presented for domain haz.de in robots.txt file.
Array
(
    [*] => Array
        (
            [disallow] => Array
                (
                    [0] => /bilder
                    [1] => /content/download
                    [2] => /content/view
                    [3] => /datalayer/entitlement
                    [4] => /E-Mediabox
                    [5] => /erasmo
                    [6] => /esi/javascript
                    [7] => /Externe-Weiterleitung
                    [8] => /Import/
                    [9] => /Mediabox
                    [10] => /mol/
                    [11] => /molnode/
                    [12] => /mostread
                    [13] => /newsroom
                    [14] => /newsticker/calendar/
                    [15] => /paid-content
                    [16] => /paidcontent
                    [17] => /Portal-Service/App-Service
                    [18] => /Promotion
                    [19] => /security
                    [20] => /storage
                    [21] => /suche
                    [22] => /Testbereich
                    [23] => /Testbereich2
                    [24] => /Weiterleitung
                )

            [sitemap] => Array
                (
                    [0] => https://www.haz.de/sitemap/index
                )

        )

    [dataengine] => Array
        (
        )

    [jobs.de-robot] => Array
        (
        )

    [nachtschatten] => Array
        (
        )

    [neofonie] => Array
        (
        )

    [webreaper] => Array
        (
            [disallow] => Array
                (
                    [0] => /
                )

        )

    [slurp] => Array
        (
            [crawl-delay] => 18
        )

    [googlebot-news] => Array
        (
            [disallow] => Array
                (
                    [0] => /Mehr/Bilder/Galerien
                )

        )

    [mj12bot] => Array
        (
            [crawl-delay] => 20
        )

)
User-agent: *
Disallow: /bilder
Disallow: /content/download
Disallow: /content/view
Disallow: /datalayer/entitlement
Disallow: /E-Mediabox
Disallow: /erasmo
Disallow: /esi/javascript
Disallow: /Externe-Weiterleitung
Disallow: /Import/
Disallow: /Mediabox
Disallow: /mol/
Disallow: /molnode/
Disallow: /mostread
Disallow: /newsroom
Disallow: /newsticker/calendar/
Disallow: /paid-content
Disallow: /paidcontent
Disallow: /Portal-Service/App-Service
Disallow: /Promotion
Disallow: /security
Disallow: /storage
Disallow: /suche
Disallow: /Testbereich
Disallow: /Testbereich2
Disallow: /Weiterleitung

User-agent: DataEngine
User-agent: jobs.de-Robot
User-agent: nachtschatten
User-agent: neofonie
User-agent: WebReaper
Disallow: /

User-agent: Slurp
Crawl-delay: 18

User-agent: Googlebot-News
Disallow: /Mehr/Bilder/Galerien

User-Agent: MJ12bot
Crawl-Delay: 20

Sitemap: https://www.haz.de/sitemap/index
  • код 301 http://haz.de/robots.txt
  • код 301 http://www.haz.de/robots.txt
  • код 200 https://www.haz.de/robots.txt