AI assistants fetch pages on demand when a user asks a chatbot or an agent to read, summarise or act on a URL.
Each request is tied to a person, but the visit carries no ad impressions or analytics. Decide per agent whether the value is worth the load.
Of the 1 AI assistants and agents with a stated policy, 0 follow robots.txt and 1 do not. Bots that ignore robots.txt can only be stopped by user agent or IP at your server, CDN or WAF.
Transfon Security identifies every bot in this directory in your traffic reports, flags fake search crawlers, and lets you allow, block or rate limit each bot.
Privacy, security, monetization, and performance insights. A few emails a month, no spam.