bbc.co.uk robots.txt
Bbc.co.uk has robots.txt file. Http response status is 200.
Robots size is small. It is about 613 kb and 172 rows.
bbc.co.uk has great amount of sitemap files — 13: https://www.bbc.co.uk/sitemap.xml, https://www.bbc.co.uk/sitemaps/https-index-uk-archive.xml, https://www.bbc.co.uk/sitemaps/https-index-uk-news.xml and others. Robots.txt for User-Agent=* contains only disallow directives.
Host directive is not presented for domain bbc.co.uk in robots.txt file.
Robots size is small. It is about 613 kb and 172 rows.
bbc.co.uk has great amount of sitemap files — 13: https://www.bbc.co.uk/sitemap.xml, https://www.bbc.co.uk/sitemaps/https-index-uk-archive.xml, https://www.bbc.co.uk/sitemaps/https-index-uk-news.xml and others. Robots.txt for User-Agent=* contains only disallow directives.
Host directive is not presented for domain bbc.co.uk in robots.txt file.
Array
(
[*] => Array
(
[sitemap] => Array
(
[0] => https://www.bbc.co.uk/sitemap.xml
[1] => https://www.bbc.co.uk/sitemaps/https-index-uk-archive.xml
[2] => https://www.bbc.co.uk/sitemaps/https-index-uk-news.xml
[3] => https://www.bbc.co.uk/food/sitemap.xml
[4] => https://www.bbc.co.uk/bitesize/sitemap/sitemapindex.xml
[5] => https://www.bbc.co.uk/teach/sitemap/sitemapindex.xml
[6] => https://www.bbc.co.uk/sitemaps/https-index-uk-archive_video.xml
[7] => https://www.bbc.co.uk/sitemaps/https-index-uk-video.xml
[8] => https://www.bbc.co.uk/sitemaps/sitemap-uk-ws-topics.xml
[9] => https://www.bbc.co.uk/sport/sitemap.xml
[10] => https://www.bbc.co.uk/sitemaps/sitemap-uk-topics.xml
[11] => https://www.bbc.co.uk/ideas/sitemap.xml
[12] => https://www.bbc.co.uk/tiny-happy-people/sitemap/sitemapindex.xml
)
[disallow] => Array
(
[0] => /asset/
[1] => /backstage/bbc-login-help/
[2] => /backstage/bbc-login-help$
[3] => /bitesize/search$
[4] => /bitesize/search/
[5] => /bitesize/search?
[6] => /cbbc/search$
[7] => /cbbc/search/
[8] => /cbbc/search?
[9] => /cbeebies/search$
[10] => /cbeebies/search/
[11] => /cbeebies/search?
[12] => /chwilio/
[13] => /chwilio$
[14] => /chwilio?
[15] => /iplayer/bigscreen/
[16] => /iplayer/cbbc/episodes/
[17] => /iplayer/cbbc/search
[18] => /iplayer/cbeebies/episodes/
[19] => /iplayer/cbeebies/search
[20] => /iplayer/search
[21] => /indepthtoolkit/smallprox$
[22] => /indepthtoolkit/smallprox/
[23] => /moderation/reports/
[24] => /modules/musicnav/language/
[25] => /news/0
[26] => /radio/aod/
[27] => /radio/aod$
[28] => /radio/imda
[29] => /radio/player/
[30] => /radio/player$
[31] => /search/
[32] => /search$
[33] => /search?
[34] => /sport/alpha/
[35] => /sounds/player/
[36] => /sounds/player$
[37] => /ugc$
[38] => /ugc/
[39] => /ugcsupport$
[40] => /ugcsupport/
[41] => /userinfo/
[42] => /userinfo
[43] => /food/favourites
[44] => /food/menus/*/shopping-list
[45] => /food/recipes/*/shopping-list
[46] => /food/search*?*
[47] => /sounds/search$
[48] => /sounds/search/
[49] => /sounds/search?
[50] => /ws/includes
[51] => /rd/search$
[52] => /rd/search/
[53] => /rd/search?
[54] => /things/search$
[55] => /things/search/
[56] => /things/search?
)
)
[amazonbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[magpie-crawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[ccbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[omgili] => Array
(
[disallow] => Array
(
[0] => /
)
)
[omgilibot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[claudebot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[claude-web] => Array
(
[disallow] => Array
(
[0] => /
)
)
[anthropic-ai] => Array
(
[disallow] => Array
(
[0] => /
)
)
[cohere-ai] => Array
(
[disallow] => Array
(
[0] => /
)
)
[bytespider] => Array
(
[disallow] => Array
(
[0] => /
)
)
[petalbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[scrapy] => Array
(
[disallow] => Array
(
[0] => /
)
)
[applebot-extended] => Array
(
[disallow] => Array
(
[0] => /
)
)
[gptbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[chatgpt-user] => Array
(
[disallow] => Array
(
[0] => /
)
)
[google-cloudvertexbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[google-extended] => Array
(
[disallow] => Array
(
[0] => /
)
)
[perplexitybot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[perplexity-user] => Array
(
[disallow] => Array
(
[0] => /
)
)
[meta-externalagent] => Array
(
[disallow] => Array
(
[0] => /
)
)
[oai-searchbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[yandexadditional] => Array
(
[disallow] => Array
(
[0] => /
)
)
[yandexadditionalbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[turnitinbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[brightbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[apifybot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[apifywebsitecontentcrawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[diffbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[diffbot-user] => Array
(
[disallow] => Array
(
[0] => /
)
)
[exabot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[tavilybot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[shapbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[youbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[firecrawlagent] => Array
(
[disallow] => Array
(
[0] => /
)
)
[amzn-searchbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[amzn-user] => Array
(
[disallow] => Array
(
[0] => /
)
)
[proratainc] => Array
(
[disallow] => Array
(
[0] => /
)
)
[cloudflarebrowserrenderingcrawler] => Array
(
[disallow] => Array
(
[0] => /
)
)
[ahrefsbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
)
# version: 36756f545af9144e59780f65727af4ba98093b3b # The BBC's Terms of Use: https://www.bbc.co.uk/terms # - Explain the rules for using our services # - Tell you what you can do with our content # # In short: Please use our site like a human, not a robot. # That means: # - No scraping, crawling, or systematic extraction of content # - No use of BBC content for training or fine-tuning AI models, including large language models (LLMs) # - No retrieval-augmented generation (RAG), AI-powered search, agentic AI or grounding using BBC content # - No creating datasets from BBC content # - No text and data mining (TDM) under Article 4 of the EU Directive on Copyright in the Digital Single Market # - No using BBC content to create summaries for your own use # - No business use without permission (details: https://www.bbc.co.uk/usingthebbc/terms/can-i-use-bbc-content-for-my-business/) # - The BBC reserves all rights in its content and expressly opts out of any statutory exceptions in any jurisdiction for text and data mining, as permitted by law # TL;DR: Browse, read, watch, enjoy - like a human. # # HTTPS www.bbc.co.uk User-agent: * Sitemap: https://www.bbc.co.uk/sitemap.xml Sitemap: https://www.bbc.co.uk/sitemaps/https-index-uk-archive.xml Sitemap: https://www.bbc.co.uk/sitemaps/https-index-uk-news.xml Sitemap: https://www.bbc.co.uk/food/sitemap.xml Sitemap: https://www.bbc.co.uk/bitesize/sitemap/sitemapindex.xml Sitemap: https://www.bbc.co.uk/teach/sitemap/sitemapindex.xml Sitemap: https://www.bbc.co.uk/sitemaps/https-index-uk-archive_video.xml Sitemap: https://www.bbc.co.uk/sitemaps/https-index-uk-video.xml Sitemap: https://www.bbc.co.uk/sitemaps/sitemap-uk-ws-topics.xml Sitemap: https://www.bbc.co.uk/sport/sitemap.xml Sitemap: https://www.bbc.co.uk/sitemaps/sitemap-uk-topics.xml Sitemap: https://www.bbc.co.uk/ideas/sitemap.xml Sitemap: https://www.bbc.co.uk/tiny-happy-people/sitemap/sitemapindex.xml Disallow: /asset/ Disallow: /backstage/bbc-login-help/ Disallow: /backstage/bbc-login-help$ Disallow: /bitesize/search$ Disallow: /bitesize/search/ Disallow: /bitesize/search? Disallow: /cbbc/search$ Disallow: /cbbc/search/ Disallow: /cbbc/search? Disallow: /cbeebies/search$ Disallow: /cbeebies/search/ Disallow: /cbeebies/search? Disallow: /chwilio/ Disallow: /chwilio$ Disallow: /chwilio? Disallow: /iplayer/bigscreen/ Disallow: /iplayer/cbbc/episodes/ Disallow: /iplayer/cbbc/search Disallow: /iplayer/cbeebies/episodes/ Disallow: /iplayer/cbeebies/search Disallow: /iplayer/search Disallow: /indepthtoolkit/smallprox$ Disallow: /indepthtoolkit/smallprox/ Disallow: /moderation/reports/ Disallow: /modules/musicnav/language/ Disallow: /news/0 Disallow: /radio/aod/ Disallow: /radio/aod$ Disallow: /radio/imda Disallow: /radio/player/ Disallow: /radio/player$ Disallow: /search/ Disallow: /search$ Disallow: /search? Disallow: /sport/alpha/ Disallow: /sounds/player/ Disallow: /sounds/player$ Disallow: /ugc$ Disallow: /ugc/ Disallow: /ugcsupport$ Disallow: /ugcsupport/ Disallow: /userinfo/ Disallow: /userinfo Disallow: /food/favourites Disallow: /food/menus/*/shopping-list Disallow: /food/recipes/*/shopping-list Disallow: /food/search*?* Disallow: /sounds/search$ Disallow: /sounds/search/ Disallow: /sounds/search? Disallow: /ws/includes Disallow: /rd/search$ Disallow: /rd/search/ Disallow: /rd/search? Disallow: /things/search$ Disallow: /things/search/ Disallow: /things/search? User-agent: Amazonbot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: CCBot Disallow: / User-Agent: omgili Disallow: / User-Agent: omgilibot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: anthropic-ai Disallow: / User-agent: cohere-ai Disallow: / User-agent: Bytespider Disallow: / User-agent: PetalBot Disallow: / User-agent: Scrapy Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: Google-Extended Disallow: / User-Agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: meta-externalagent Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: YandexAdditional Disallow: / User-agent: YandexAdditionalBot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Brightbot Disallow: / User-agent: ApifyBot Disallow: / User-agent: ApifyWebsiteContentCrawler Disallow: / User-agent: Diffbot Disallow: / User-agent: Diffbot-User Disallow: / User-agent: ExaBot Disallow: / User-agent: TavilyBot Disallow: / User-agent: ShapBot Disallow: / User-agent: YouBot Disallow: / User-agent: FirecrawlAgent Disallow:/ User-agent: Amzn-SearchBot Disallow: / User-agent: Amzn-User Disallow: / User-agent: ProRataInc Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: AhrefsBot Disallow: /
- код 301 http://bbc.co.uk/robots.txt
- код 301 https://bbc.co.uk/robots.txt
- код 200 https://www.bbc.co.uk/robots.txt