openai

OpenAI documents four crawlers, and one of them exists to judge ad landing pages. What each does, which ignores robots.txt, and how to verify rather than guess.