repubblica.it robots.txt
Repubblica.it has robots.txt file. Http response code is 200.
Robots size is medium. It is about 974 kb and 291 rows.
repubblica.it has great amount of sitemap files — 13: https://www.repubblica.it/sitemap-n.xml, https://video.repubblica.it/sitemap-v-day.xml, https://video.repubblica.it/sitemap-v.xml and others. Host directive is not presented for domain repubblica.it in robots.txt file.
Robots size is medium. It is about 974 kb and 291 rows.
repubblica.it has great amount of sitemap files — 13: https://www.repubblica.it/sitemap-n.xml, https://video.repubblica.it/sitemap-v-day.xml, https://video.repubblica.it/sitemap-v.xml and others. Host directive is not presented for domain repubblica.it in robots.txt file.
Array
(
[*] => Array
(
[allow] => Array
(
[0] => /
)
[disallow] => Array
(
[0] => /ultimora/24ore/nazionale/news-dettaglio/3688417
[1] => /ultimora/24ore/AUTO-IN-CAMBIO-DI-DROGA-13-ARRESTI-DELLA-POLIZIA-DI-PESCARA-(2)/news-dettaglio/3746479
[2] => /sport/ciclismo/2023/10/24/news/doping_ciclismo_virtuale_azzurro_zanasca_rischia_una_lunga_squalifica-418660589/
[3] => /spettacoli/people/2023/09/07/news/kanye_west_e_bianca_censori_in_italia_la_lunga_estate_calda_degli_scandali-413629176/<
[4] => /speciale/
[5] => /social/sites/repubblica/d/boxes/shares/
[6] => /servizi/
[7] => /ricerca/
[8] => /politica/2017/01/19/news/il_fratello_del_ministro_alfano_rischia_il_posto_e_costera_all_erario_mezzo_milione_di_euro[1]156371739/
[9] => /politica/2/10/13/news/da_maroni_a_fontana_c_e_sempre_quel_notaio_nelle_indagini_sulla_lega-301070805/
[10] => /persone/
[11] => /online/cronaca/veronaecstasy/bologna/bologna.html
[12] => /online/
[13] => /news/ired/ultimora/cronaca/rep_cronaca_n_3487677.html
[14] => /java
[15] => /ipad/
[16] => /interstitial/
[17] => /indici/
[18] => /il-gusto/2022/04/27/news/frode_fiscale_bancarotta_fraudolenta_arrestato_giovanni_maspero_ristorante_i_tigli_in_theoria-347110162/
[19] => /esteri/2024/07/24/diretta/kamala_harris_nel_mirino_dei_repubblicani_usa_ricorso_contro_il_trasferimento_dei_fondi_dalla_campagna_di_biden-423411481/
[20] => /embed/*
[21] => /economia/miojob/
[22] => /economia/2013/01/31/news/mps_verso_sequestro_conservativo- 51615470/
[23] => /dal-quotidiano/
[24] => /corporate/privacy/index_static.html?
[25] => /cgi-bin/back.cgi
[26] => /blaize/datalayer
[27] => /audio/i-daily-di-repubblica
[28] => /audio-test-app
[29] => /api/checkpw-v1/premium/amp/pingback*
[30] => /api/*
[31] => /annunci/
[32] => /2010/
[33] => /2009/
[34] => /2008/
[35] => /2007/
[36] => /2006/05/gallerie/politica/offese-da-premier-102/esterne130937331310093801_big.jpg
[37] => /2006/05/gallerie/politica/farabutti-foto-lettori-95/esterne191030071909103240_big.jpg
[38] => /2006/05/gallerie/politica/farabutti-foto-lettori-95/esterne191030071909103240.jpg
[39] => /2006/
[40] => /2005/
[41] => /2004/
[42] => /2003/
[43] => /coccodrilli/
[44] => /swf
[45] => /serietv/schede/cerca/?query=
[46] => /online3
[47] => /mobile/
[48] => /ztest/
[49] => /dossier/sapori/guide-espresso*
[50] => /dossier/motori*
[51] => /topics/*?p=
[52] => /mobile-rep/
[53] => /tv/
[54] => /static/__sdemo__/
[55] => /quotidiano
)
[sitemap] => Array
(
[0] => https://www.repubblica.it/sitemap-n.xml
[1] => https://video.repubblica.it/sitemap-v-day.xml
[2] => https://video.repubblica.it/sitemap-v.xml
[3] => https://www.repubblica.it/sitemap-italiantech-n.xml
[4] => https://www.repubblica.it/sitemap-il-gusto-n.xml
[5] => https://www.repubblica.it/sitemap-salute-n.xml
[6] => https://www.repubblica.it/sitemap-green-and-blue-n.xml
[7] => https://www.repubblica.it/sitemap-dirette-sport-n.xml
[8] => https://www.repubblica.it/sitemap.xml
[9] => https://www.repubblica.it/sitemap-italiantech.xml
[10] => https://www.repubblica.it/sitemap-il-gusto.xml
[11] => https://www.repubblica.it/sitemap-salute.xml
[12] => https://www.repubblica.it/sitemap-green-and-blue.xml
)
)
[googlebot-news] => Array
(
[disallow] => Array
(
[0] => /la-zampa/
)
)
[gptbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[google-extended] => Array
(
[disallow] => Array
(
[0] => /
)
)
[ccbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[anthropic-ai] => Array
(
[disallow] => Array
(
[0] => /
)
)
[omgilibot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[facebookbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[cohere-ai] => Array
(
[disallow] => Array
(
[0] => /
)
)
[petalbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[yandex] => Array
(
[disallow] => Array
(
[0] => /
)
)
[perplexitybot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[amazonbot] => Array
(
[allow] => Array
(
[0] => /
)
)
[ahrefsbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[archive.org_bot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[archivebot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[arquivo-web-crawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[awariorssbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[awariosmartbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[backlinkcrawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[bixocrawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[blexbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[bytespider] => Array
(
[disallow] => Array
(
[0] => /
)
)
[cliqzbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[converacrawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[crawler4j] => Array
(
[disallow] => Array
(
[0] => /
)
)
[crystalsemanticsbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[dataforseobot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[diffbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[discobot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[discoverybot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[dloader(naverrobot)] => Array
(
[disallow] => Array
(
[0] => /
)
)
[dotbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[download ninja] => Array
(
[disallow] => Array
(
[0] => /
)
)
[duckassistbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[europarchive.org] => Array
(
[disallow] => Array
(
[0] => /
)
)
[exabot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[extractorpro] => Array
(
[disallow] => Array
(
[0] => /
)
)
[fetch] => Array
(
[disallow] => Array
(
[0] => /
)
)
[flamingo_searchengine] => Array
(
[disallow] => Array
(
[0] => /
)
)
[friendlycrawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[gigabot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[gumgum bot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[ia_archiver] => Array
(
[disallow] => Array
(
[0] => /
)
)
[ia_archiver-web.archive.org] => Array
(
[disallow] => Array
(
[0] => /
)
)
[imagesiftbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[istellabot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[jetbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[jikespider] => Array
(
[disallow] => Array
(
[0] => /
)
)
[jyxobot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[k2spider] => Array
(
[disallow] => Array
(
[0] => /
)
)
[kangaroo bot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[kbcrawl] => Array
(
[disallow] => Array
(
[0] => /
)
)
[lexxebot/1.0] => Array
(
[disallow] => Array
(
[0] => /
)
)
[linkarchiver] => Array
(
[disallow] => Array
(
[0] => /
)
)
[linkextractorpro] => Array
(
[disallow] => Array
(
[0] => /
)
)
[livelapbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[magpie-crawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[meta-externalfetcher] => Array
(
[disallow] => Array
(
[0] => /
)
)
[mj12bot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[moreoverbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[msiecrawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[nabot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[naverbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[nerdbynature.bot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[netestate ne crawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[nnetseer crawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[nextgensearchbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[nicecrawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[npbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[nutch] => Array
(
[disallow] => Array
(
[0] => /
)
)
[openbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[pangubot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[peer39_crawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[peer39_crawler/1.0] => Array
(
[disallow] => Array
(
[0] => /
)
)
[piplbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[pixray-seeker] => Array
(
[disallow] => Array
(
[0] => /
)
)
[primalbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[psbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[queryseekerspider] => Array
(
[disallow] => Array
(
[0] => /
)
)
[quora-bot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[rogerbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[scrapy] => Array
(
[disallow] => Array
(
[0] => /
)
)
[searchpreview] => Array
(
[disallow] => Array
(
[0] => /
)
)
[semrushbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[seoengbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[seokicks-robot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[sitebot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[sitebot/0.1] => Array
(
[disallow] => Array
(
[0] => /
)
)
[slurp] => Array
(
[disallow] => Array
(
[0] => /
)
)
[sosospider] => Array
(
[disallow] => Array
(
[0] => /
)
)
[spbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[timpibot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[trendictionbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[trovitbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[true_robot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[turnitinbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[ubicrawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[umbot-ln] => Array
(
[disallow] => Array
(
[0] => /
)
)
[url_spider_pro] => Array
(
[disallow] => Array
(
[0] => /
)
)
[verticalsearch] => Array
(
[disallow] => Array
(
[0] => /
)
)
[wbsearchbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[web-archive-net.com.bot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[wesee:search] => Array
(
[disallow] => Array
(
[0] => /
)
)
[wotbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[youbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[zealbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[applebot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[applebot-extended] => Array
(
[disallow] => Array
(
[0] => /
)
)
)
User-agent: * Allow: / Disallow: /ultimora/24ore/nazionale/news-dettaglio/3688417 Disallow: /ultimora/24ore/AUTO-IN-CAMBIO-DI-DROGA-13-ARRESTI-DELLA-POLIZIA-DI-PESCARA-(2)/news-dettaglio/3746479 Disallow: /sport/ciclismo/2023/10/24/news/doping_ciclismo_virtuale_azzurro_zanasca_rischia_una_lunga_squalifica-418660589/ Disallow: /spettacoli/people/2023/09/07/news/kanye_west_e_bianca_censori_in_italia_la_lunga_estate_calda_degli_scandali-413629176/< Disallow: /speciale/ Disallow: /social/sites/repubblica/d/boxes/shares/ Disallow: /servizi/ Disallow: /ricerca/ Disallow: /politica/2017/01/19/news/il_fratello_del_ministro_alfano_rischia_il_posto_e_costera_all_erario_mezzo_milione_di_euro[1]156371739/ Disallow: /politica/2/10/13/news/da_maroni_a_fontana_c_e_sempre_quel_notaio_nelle_indagini_sulla_lega-301070805/ Disallow: /persone/ Disallow: /online/cronaca/veronaecstasy/bologna/bologna.html Disallow: /online/ Disallow: /news/ired/ultimora/cronaca/rep_cronaca_n_3487677.html Disallow: /java Disallow: /ipad/ Disallow: /interstitial/ Disallow: /indici/ Disallow: /il-gusto/2022/04/27/news/frode_fiscale_bancarotta_fraudolenta_arrestato_giovanni_maspero_ristorante_i_tigli_in_theoria-347110162/ Disallow: /esteri/2024/07/24/diretta/kamala_harris_nel_mirino_dei_repubblicani_usa_ricorso_contro_il_trasferimento_dei_fondi_dalla_campagna_di_biden-423411481/ Disallow: /embed/* Disallow: /economia/miojob/ Disallow: /economia/2013/01/31/news/mps_verso_sequestro_conservativo- 51615470/ Disallow: /dal-quotidiano/ Disallow: /corporate/privacy/index_static.html? Disallow: /cgi-bin/back.cgi Disallow: /blaize/datalayer Disallow: /audio/i-daily-di-repubblica Disallow: /audio-test-app Disallow: /api/checkpw-v1/premium/amp/pingback* Disallow: /api/* Disallow: /annunci/ Disallow: /2010/ Disallow: /2009/ Disallow: /2008/ Disallow: /2007/ Disallow: /2006/05/gallerie/politica/offese-da-premier-102/esterne130937331310093801_big.jpg Disallow: /2006/05/gallerie/politica/farabutti-foto-lettori-95/esterne191030071909103240_big.jpg Disallow: /2006/05/gallerie/politica/farabutti-foto-lettori-95/esterne191030071909103240.jpg Disallow: /2006/ Disallow: /2005/ Disallow: /2004/ Disallow: /2003/ Disallow: /coccodrilli/ Disallow: /swf Disallow: /serietv/schede/cerca/?query= Disallow: /online3 Disallow: /mobile/ Disallow: /ztest/ Disallow: /dossier/sapori/guide-espresso* Disallow: /dossier/motori* Disallow: /topics/*?p= Disallow: /mobile-rep/ Disallow: /tv/ Disallow: /static/__sdemo__/ Disallow: /quotidiano User-agent: Googlebot-News Disallow: /la-zampa/ # Sitemap: https://www.repubblica.it/sitemap-n.xml # Sitemap: https://video.repubblica.it/sitemap-v-day.xml Sitemap: https://video.repubblica.it/sitemap-v.xml Sitemap: https://www.repubblica.it/sitemap-italiantech-n.xml Sitemap: https://www.repubblica.it/sitemap-il-gusto-n.xml Sitemap: https://www.repubblica.it/sitemap-salute-n.xml Sitemap: https://www.repubblica.it/sitemap-green-and-blue-n.xml # Sitemap: https://www.repubblica.it/sitemap-dirette-sport-n.xml # Sitemap: https://www.repubblica.it/sitemap.xml Sitemap: https://www.repubblica.it/sitemap-italiantech.xml Sitemap: https://www.repubblica.it/sitemap-il-gusto.xml Sitemap: https://www.repubblica.it/sitemap-salute.xml Sitemap: https://www.repubblica.it/sitemap-green-and-blue.xml User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Omgilibot Disallow: / User-agent: FacebookBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: PetalBot Disallow: / User-agent: Yandex Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Amazonbot Allow: / User-agent: AhrefsBot Disallow: / User-agent: archive.org_bot Disallow: / User-agent: ArchiveBot Disallow: / User-agent: Arquivo-web-crawler Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: BIXOCRAWLER Disallow: / User-agent: BLEXBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Cliqzbot Disallow: / User-agent: ConveraCrawler Disallow: / User-agent: crawler4j Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Diffbot Disallow: / User-agent: discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: dloader(NaverRobot) Disallow: / User-agent: dotbot Disallow: / User-agent: Download Ninja Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: europarchive.org Disallow: / User-agent: Exabot Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Fetch Disallow: / User-agent: Flamingo_SearchEngine Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: Gigabot Disallow: / User-agent: GumGum Bot Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver-web.archive.org Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: IstellaBot Disallow: / User-agent: Jetbot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Jyxobot Disallow: / User-agent: k2spider Disallow: / User-agent: Kangaroo Bot Disallow: / User-agent: kbcrawl Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: LinkArchiver Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: Livelapbot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: MJ12bot Disallow: / User-agent: Moreoverbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: nabot Disallow: / User-agent: NaverBot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: nNetSeer crawler Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: Nicecrawler Disallow: / User-agent: NPBot Disallow: / User-agent: nutch Disallow: / User-agent: Openbot Disallow: / User-agent: PanguBot Disallow: / User-agent: peer39_crawler Disallow: / User-agent: peer39_crawler/1.0 Disallow: / User-agent: PiplBot Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: Primalbot Disallow: / User-agent: psbot Disallow: / User-agent: QuerySeekerSpider Disallow: / User-agent: Quora-Bot Disallow: / User-agent: rogerbot Disallow: / User-agent: Scrapy Disallow: / User-agent: searchpreview Disallow: / User-agent: SemrushBot Disallow: / User-agent: SEOENGBot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: SiteBot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: Slurp Disallow: / User-agent: Sosospider Disallow: / User-agent: spbot Disallow: / User-agent: Timpibot Disallow: / User-agent: trendictionbot Disallow: / User-agent: trovitBot Disallow: / User-agent: True_Robot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: UbiCrawler Disallow: / User-agent: uMBot-LN Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: verticalsearch Disallow: / User-agent: WBSearchBot Disallow: / User-agent: web-archive-net.com.bot Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: Wotbot Disallow: / User-agent: YouBot Disallow: / User-agent: Zealbot Disallow: / User-agent: AppleBot Disallow: / User-agent: Applebot-Extended Disallow: /
- код 301 http://repubblica.it/robots.txt
- код 200 https://www.repubblica.it/robots.txt