apnews.com robots.txt
Apnews.com has robots.txt file. Http response status is 200.
Robots size is small. It is about 151 kb and 50 rows.
apnews.com has great amount of sitemap files — 5: https://apnews.com/ap-sitemap.xml, https://apnews.com/news-sitemap-content.xml, https://apnews.com/hubs-sitemap-content.xml and others. Robots.txt for User-Agent=* contains only disallow directives.
Host directive is not presented for domain apnews.com in robots.txt file.
Robots size is small. It is about 151 kb and 50 rows.
apnews.com has great amount of sitemap files — 5: https://apnews.com/ap-sitemap.xml, https://apnews.com/news-sitemap-content.xml, https://apnews.com/hubs-sitemap-content.xml and others. Robots.txt for User-Agent=* contains only disallow directives.
Host directive is not presented for domain apnews.com in robots.txt file.
Array
(
[*] => Array
(
[disallow] => Array
(
[0] => *_ptid=*
[1] => *?prx_t=*
[2] => /zephr*
[3] => /blaize*
[4] => /search?q=*
[5] => /*.rss
[6] => /buyline-shopping/*
[7] => /buyline-personal-finance/*
[8] => *?jw_start
[9] => *&jw_start
[10] => /gallery/
[11] => /config/newrelic/prod
[12] => /parsely-api-endpoint
[13] => /apdata/
[14] => /article/border-immigration-election-2182fec11fa008ec96c4202e5656d19e
[15] => /api/v2/feed/
)
[sitemap] => Array
(
[0] => https://apnews.com/ap-sitemap.xml
[1] => https://apnews.com/news-sitemap-content.xml
[2] => https://apnews.com/hubs-sitemap-content.xml
[3] => https://apnews.com/video-sitemap.xml
[4] => https://apnews.com/projects/elections-2026/sitemap-index.xml
)
)
[ccbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[gptbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[anthropic-ai] => Array
(
[disallow] => Array
(
[0] => /
)
)
[claudebot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[claude-web] => Array
(
[disallow] => Array
(
[0] => /
)
)
[cohere-ai] => Array
(
[disallow] => Array
(
[0] => /
)
)
[perplexitybot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[amazonbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[applebot-extended] => Array
(
[disallow] => Array
(
[0] => /
)
)
[bytedance] => Array
(
[disallow] => Array
(
[0] => /
)
)
[timpibot] => Array
(
[disallow] => Array
(
[0] => /
)
)
[claude-user] => Array
(
[disallow] => Array
(
[0] => /
)
)
[claude-searchbot] => Array
(
[disallow] => Array
(
[0] => /
)
)
)
User-Agent: * Disallow: Disallow: *_ptid=* Disallow: *?prx_t=* Disallow: /zephr* Disallow: /blaize* Disallow: /search?q=* Disallow: /*.rss Disallow: /buyline-shopping/* Disallow: /buyline-personal-finance/* Disallow: *?jw_start Disallow: *&jw_start Disallow: /gallery/ Disallow: /config/newrelic/prod Disallow: /parsely-api-endpoint Disallow: /apdata/ Disallow: /article/border-immigration-election-2182fec11fa008ec96c4202e5656d19e Disallow: /api/v2/feed/ Sitemap: https://apnews.com/ap-sitemap.xml Sitemap: https://apnews.com/news-sitemap-content.xml Sitemap: https://apnews.com/hubs-sitemap-content.xml Sitemap: https://apnews.com/video-sitemap.xml Sitemap: https://apnews.com/projects/elections-2026/sitemap-index.xml # Disallow Rules User-agent: CCBot Disallow: / User-agent: GPTBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytedance Disallow: / User-agent: Timpibot Disallow: / User-agent: Claude-User Disallow: / User-agent: Claude-SearchBot Disallow: /
- код 301 http://apnews.com/robots.txt
- код 200 https://apnews.com/robots.txt