Articles tagged with AI Bots

How to Block AI Training Crawlers in robots.txt: The New __usage_ai-training Directive
The new __usage_ai-training robots.txt directive lets you block every AI training crawler, including Google-Extended, GPTBot and ClaudeBot, with a single entry instead of one per company. Here is what it does, what it does not cover, who honours it and when, and why you still need bot management alongside it.

Agentic Browsers and Meta-ExternalAgent: What to Block Before Meta's 20 November Terms
AI agents now reach forms and checkout without declaring themselves, and Meta's new crawling terms start on 20 November. How Transfon Security lets you see, limit and block agentic browser and Meta crawler traffic.
Privacy, security, monetization, and performance insights. A few emails a month, no spam.