External Publication
Visit Post

GPTBot ignoring robots.txt and hammering single URL in a loop — potential infinite crawl bug

OpenAI Developer Community February 18, 2026
Source

It’s entirely plausible that someone is pretending to be GPTBot. Have you checked the IP Address(es) attached to the requests?

You can match it here, and in the future ensure that any user agent as GPTBOT is valid

https://openai.com/gptbot.json

Lastly, you could be more explicit and include this in your robots.txt (not sure if this would help)

User-agent: GPTBot
Disallow: /*?*nocache=*

Discussion in the ATmosphere

Loading comments...