Ynet.co.il has robots.txt file. Http response code is 200.
Robots size is small. It is about 189 kb and 37 rows.

Robots.txt for User-Agent=* contains only disallow directives.

Host directive is not presented for domain ynet.co.il in robots.txt file.
Array
(
    [*] => Array
        (
            [disallow] => Array
                (
                    [0] => /english/
                    [1] => /Ext/App/Thumbnails/CdaThumbnails_OpenWin/
                    [2] => /Ext/App/Billing/Login/CdaRegBill_UserIdentificationComponentIFrame/
                    [3] => /Ext/Comp/ArticleLayout/CdaSendArticleResponse/
                    [4] => /YediothPortal/Ext/TalkBack/CdaViewTalkBack/
                    [5] => /YediothPortal/Ext/TalkBack/CdaTalkBack/
                    [6] => /YediothPortal/Ext/Chn/Astrology/CdaPrevewPrint/
                    [7] => /home/*,7340,L-313,00.html
                    [8] => /home/*,7340,L-847,00.html
                    [9] => /home/*,7340,L-300,00.html
                    [10] => /home/*,7340,L-5020,00.html
                    [11] => /home/*,7340,L-7854,00.html
                    [12] => /home/*,7340,L-5019,00.html
                    [13] => /home/*,7340,L-5022,00.html
                    [14] => /home/*,7340,L-5025,00.html
                    [15] => /home/*,7340,L-5026,00.html
                    [16] => /home/*,7340,L-5028,00.html
                    [17] => /home/*,7340,L-5032,00.html
                    [18] => /Ext/Comp/ArticleLayout/CdaArticlePrintPreview
                    [19] => /Ext/Comp/CdaNewsFlash/0,2297,L-4174879_184,00.html
                    [20] => /home/0,7340,L-10613,00.html
                    [21] => /Ext/App/Ency/Items/CdaAppEncyTVProgram/
                    [22] => /YediothPortal/Ext/TalkBack/CdaTalkBackTrans/
                    [23] => /YediothPortal/Ext/App/TDG/CdaSendTo_Discgrp/
                    [24] => /YediothPortal/Ext/Chn/Astrology/AstrologyPopSend_message/
                    [25] => /novgnpre/Ext/Comp/ArticleLayout/CdaArticleSendtoForm1024/
                    [26] => /Ext/App/TalkBack/CdaAddTalkbackRecommendation/
                    [27] => /YediothPortal/Ext/App/TDG/CdaPrintDiscgrp/
                    [28] => /articlesnc/
                    [29] => /homenc/
                    [30] => /tags/JDate
                )

        )

    [msnbot] => Array
        (
            [crawl-delay] => 1
        )

    [bingbot] => Array
        (
            [crawl-delay] => 1
        )

)
User-agent: *
Disallow: /english/ 
Disallow: /Ext/App/Thumbnails/CdaThumbnails_OpenWin/ 
Disallow: /Ext/App/Billing/Login/CdaRegBill_UserIdentificationComponentIFrame/
Disallow: /Ext/Comp/ArticleLayout/CdaSendArticleResponse/
Disallow: /YediothPortal/Ext/TalkBack/CdaViewTalkBack/
Disallow: /YediothPortal/Ext/TalkBack/CdaTalkBack/
Disallow: /YediothPortal/Ext/Chn/Astrology/CdaPrevewPrint/
Disallow: /home/*,7340,L-313,00.html
Disallow: /home/*,7340,L-847,00.html
Disallow: /home/*,7340,L-300,00.html
Disallow: /home/*,7340,L-5020,00.html
Disallow: /home/*,7340,L-7854,00.html
Disallow: /home/*,7340,L-5019,00.html
Disallow: /home/*,7340,L-5022,00.html
Disallow: /home/*,7340,L-5025,00.html
Disallow: /home/*,7340,L-5026,00.html
Disallow: /home/*,7340,L-5028,00.html
Disallow: /home/*,7340,L-5032,00.html
Disallow: /Ext/Comp/ArticleLayout/CdaArticlePrintPreview
Disallow: /Ext/Comp/CdaNewsFlash/0,2297,L-4174879_184,00.html
Disallow: /home/0,7340,L-10613,00.html
Disallow: /Ext/App/Ency/Items/CdaAppEncyTVProgram/
Disallow: /YediothPortal/Ext/TalkBack/CdaTalkBackTrans/
Disallow: /YediothPortal/Ext/App/TDG/CdaSendTo_Discgrp/
Disallow: /YediothPortal/Ext/Chn/Astrology/AstrologyPopSend_message/
Disallow: /novgnpre/Ext/Comp/ArticleLayout/CdaArticleSendtoForm1024/
Disallow: /Ext/App/TalkBack/CdaAddTalkbackRecommendation/
Disallow: /YediothPortal/Ext/App/TDG/CdaPrintDiscgrp/
Disallow: /articlesnc/
Disallow: /homenc/
Disallow: /tags/JDate

User-agent: msnbot
Crawl-delay: 1
 
User-agent: bingbot 
Crawl-delay: 1
  • код 302 http://ynet.co.il/robots.txt
  • код 301 http://www.ynet.co.il/robots.txt
  • код 200 https://www.ynet.co.il/robots.txt