External Publication
Visit Post

GPTBot ignoring robots.txt and hammering single URL in a loop — potential infinite crawl bug

OpenAI Developer Community February 21, 2026
Source
Yes it is verified by Cloudflare themselves as the real GPT Bot. It seems that I fixed this issue by blocking access to those nocache strings completely. GPT Bot got blocked about 4k in 24 hours and disappeared for a while. It is back now and crawling as expected.

Discussion in the ATmosphere

Loading comments...