Top Web Crawlers & Bot Traffic Stats for 2025
Transfon Team
14 min read
Total Bot Traffic
51.37%
Is Dominated by Just 3 Bots
Search Engine Share
~45%
of top bot traffic is from major search engines.
The Rise of AI
>12%
of bot activity is now driven by AI training and queries.
The Digital Giants: Traffic Share
The internet's bot traffic is highly concentrated. A handful of major players from search and AI account for the huge majority of all automated hits, leaving a small fraction for everyone else.
Top 5 Bots by Traffic
While Google and Bing lead, the rise of AI crawlers like ChatGPT-User and GPTBot is undeniable. This chart compares the traffic percentage of the most active bots on the web.
Who's Crawling? A Field Guide to Bots
Search Engines
The foundation of the web, these bots index content for search results.
- Googlebot (Smartphone & Desktop)
- Bingbot
- Applebot
- YandexBot
- Yahoo! Slurp
AI & Data Crawlers
The new generation of bots gathering data to train large language models.
- ChatGPT-User
- GPTBot
- OAI-SearchBot
- ClaudeBot
- PerplexityBot
Social & Link Preview
These bots fetch data to create the link previews you see in apps.
- Meta-ExternalAgent
- FacebookExternalHit
- LinkedInBot
- Twitterbot
- DiscordBot
SEO & Analytics
Tools used by marketers to analyze websites and track rankings.
- Barkrowler
- BLEXBot
- DataForSeoBot
- AhrefsBot
- SemrushBot
The following list is detected by the Transfon Security intelligent bot management system.
Top Bot List 2025
| Rank | Bot Name | % of Total | Description |
|---|---|---|---|
| 1 | Googlebot Mobile | 19.08% | Google's mobile-first crawler simulating a smartphone user |
| 2 | Bingbot | 18.91% | Microsoft's main web crawler for Bing Search |
| 3 | Meta-ExternalAgent | 13.38% | Meta's bot for fetching external content, likely for link previews |
| 4 | ChatGPT-User | 7.54% | Identifier for ChatGPT traffic fetching site data |
| 5 | Googlebot Desktop | 5.80% | Google's crawler simulating a desktop browser |
| 6 | FacebookExternalHit | 4.58% | Facebook's bot for generating link previews |
| 8 | GPTBot | 3.03% | OpenAI's crawler for AI training and search |
| 10 | Applebot | 2.66% | Apple's crawler for Siri and Spotlight search |
| 11 | OAI-SearchBot | 2.14% | OpenAI's bot for search-related content retrieval |
| 12 | Barkrowler | 1.52% | SEO crawling bot from Barkrowler services |
| 13 | YandexAccessibilityBot | 1.16% | Yandex bot testing accessibility and rendering |
| 14 | Meta-ExternalAds | 1.15% | Meta's bot for validating ads and previews |
| 15 | GoogleOther | 0.94% | Miscellaneous Google crawler activity |
| 16 | Yahoo! Slurp | 0.90% | Yahoo's historical search crawler |
| 17 | YandexBot | 0.86% | Main search crawler for Yandex |
| 18 | LinkedInBot | 0.84% | LinkedIn's link preview generator |
| 19 | BLEXBot | 0.74% | SEO analytics bot for backlink analysis |
| 20 | DataForSeoBot | 0.71% | DataForSEO's bot for SERP and keyword analysis |
| 21 | Twitterbot | 0.64% | Twitter's crawler for link cards and previews |
| 22 | Amazonbot | 0.61% | Amazon's web crawler for various indexing uses |
| 23 | MojeekBot | 0.38% | Independent search engine crawler |
| 24 | ClaudeBot | 0.37% | Anthropic's AI-related crawler |
| 25 | Googlebot-Image | 0.36% | Specialized Google crawler for images |
| 26 | PerplexityBot | 0.34% | Perplexity AI's content-fetching crawler |
| 27 | Cortex | 0.31% | Bot associated with Cortex AI or analytics tools |
| 28 | FlipboardRSS | 0.29% | Flipboard's bot for pulling RSS content |
| 29 | Sogou Web Spider | 0.28% | Chinese search engine Sogou's crawler |
| 30 | Feedly | 0.22% | RSS reader Feedly's content fetcher |
| 31 | BingPreview | 0.18% | Bing's page preview generator |
| 32 | Protopage | 0.18% | Protopage's bot for RSS and widget updates |
| 33 | Also | 0.16% | News aggregation bot from Also |
| 34 | CCBot | 0.16% | Common Crawl's open web indexing bot |
| 35 | TrendictionBot | 0.15% | Trendiction's content aggregation crawler |
| 36 | YaDirectFetcher | 0.15% | Yandex Direct ads fetching bot |
| 37 | AdsBot-Google | 0.13% | Google's bot for verifying ads landing pages |
| 38 | Feeder.co | 0.13% | Feeder RSS reader's crawler |
| 39 | PetalBot | 0.13% | Huawei's search engine bot |
| 40 | Turnitin | 0.12% | Plagiarism detection service crawler |
| 41 | AdsBot-Google-Mobile | 0.12% | Mobile version of Google's ads verification bot |
| 42 | Google-Site-Verification | 0.11% | Bot checking site ownership for Google Search Console |
| 43 | SeznamBot | 0.11% | Crawler for Czech search engine Seznam |
| 44 | AmazonAdBot | 0.11% | Amazon's ad verification bot |
| 45 | Site24x7 | 0.09% | Website uptime monitoring bot |
| 46 | Yeti | 0.09% | Naver's crawler (South Korea) |
| 47 | Google Favicon | 0.08% | Google's bot fetching favicons |
| 48 | YaK | 0.07% | Yandex's image or content fetching bot |
| 49 | Inoreader | 0.07% | RSS reader Inoreader's bot |
| 50 | Google Web Preview Analytics | 0.07% | Google's analytics preview bot |
| 51 | Known Attack M | 0.06% | Malicious traffic variant flagged by system |
| 52 | Omgilibot | 0.06% | Bot used for indexing online libraries |
| 53 | Google Web Preview | 0.06% | Google's preview fetching bot |
| 54 | Adreview | 0.05% | Ad content quality check bot |
| 55 | PlayStore-Google | 0.05% | Google Play Store's content verification bot |
| 56 | Peer39 Crawler | 0.05% | Peer39's semantic ad targeting bot |
| 57 | Photon | 0.05% | Image proxy and optimization bot |
| 58 | LivelapBot | 0.05% | Bot for sports/live event data updates |
| 59 | Google-Read-Aloud | 0.05% | Bot for Google's read-aloud service |
| 60 | BUbiNG | 0.05% | High-speed web crawler research project |
| 61 | DotBot 1.1 | 0.05% | Moz's SEO crawler |
| 63 | Google-Ads-Conversions | 0.04% | Bot for tracking ad conversions |
| 64 | Linkbot | 0.04% | SEO link-checking bot |
| 65 | SentiBot | 0.04% | Sentiment analysis crawler |
| 66 | StatusCake | 0.04% | Uptime monitoring bot |
| 67 | Feedspot | 0.04% | Feedspot RSS aggregator bot |
| 68 | Semantic-Visions | 0.04% | Semantic Visions data collection bot |
| 69 | OnlineOrNot Bot | 0.04% | Website uptime and performance monitoring bot |
| 70 | AhrefsBot | 0.04% | Popular SEO backlink analysis crawler |
| 71 | Google-PageRenderer | 0.04% | Google's rendering bot for JavaScript-heavy pages |
| 72 | Chrome-Lighthouse | 0.03% | Google's performance and accessibility audit bot |
| 73 | YisouSpider | 0.03% | Crawler for Yisou search engine (China) |
| 74 | FlipboardProxy | 0.03% | Flipboard's proxy fetching bot |
| 75 | GoogleWebLight | 0.03% | Google's mobile data-saving crawler |
| 76 | Brightbot | 0.03% | Bot for content marketing or SEO testing |
| 77 | Neticle Crawler | 0.03% | Neticle's online monitoring crawler |
| 78 | DotBot 1.2 | 0.03% | Moz's updated SEO crawling bot |
| 79 | Coccocbot | 0.03% | Vietnamese search engine Cốc Cốc's bot |
| 80 | DuckDuckBot | 0.03% | DuckDuckGo's web crawler |
| 81 | ClaudeBot | 0.03% | Anthropic's AI crawler (duplicate variant) |
| 82 | Adbeat Bot | 0.02% | Competitive ad intelligence crawler |
| 83 | Uptime Bot | 0.02% | Website uptime monitoring bot |
| 84 | YandexRenderResources | 0.02% | Yandex bot fetching render resources |
| 85 | DuckAssistBot | 0.02% | DuckDuckGo's AI answer bot |
| 86 | SkypeUriPreview | 0.02% | Skype's link preview bot |
| 87 | Baiduspider | 0.02% | Baidu's main search engine crawler |
| 88 | Google-Safety | 0.02% | Google's safety verification bot |
| 89 | 360Spider | 0.02% | Qihoo 360 search engine crawler |
| 90 | FeedBurner | 0.02% | Google's FeedBurner RSS fetcher |
| 91 | Outbrain | 0.02% | Outbrain's content recommendation bot |
| 92 | SemrushBot | 0.02% | SEO analysis crawler from Semrush |
| 93 | DiscordBot | 0.01% | Discord's link preview bot |
| 94 | UptimeRobot | 0.01% | Popular uptime monitoring service bot |
| 95 | SiteAuditBot | 0.01% | SEO site audit crawler |
| 96 | Storebot-Google | 0.01% | Google Play Store's indexing bot |
| 97 | MJ12bot | 0.01% | Distributed web crawler project |
| 98 | Imgproxy | 0.01% | Image proxy and transformation service |
| 99 | Perplexity-User | 0.01% | Perplexity AI user agent for fetching content |
| 100 | Slack-ImgProxy | 0.01% | Slack's image proxy bot |
| 101 | Google-Shopping-Quality | 0.01% | Google Shopping quality crawler |
| 102 | WadooImageBot/1.0 | 0.01% | Image crawling bot from Wadoo |
| 103 | Mediapartners-Google | 0.01% | Google AdSense crawler |
| 104 | MJ12bot/v2.0.2 | 0.01% | MJ12bot distributed crawler (v2.0.2) |
| 105 | theoldreader.com | 0.01% | RSS reader bot |
| 106 | agentslug | 0.01% | Bot used for content aggregation |
| 107 | DongleEmulatorBot/0.1 | 0.01% | Emulator bot for testing |
| 108 | GumGum-Bot/1.0 | 0.01% | Ad/visual recognition bot |
| 109 | GoogleDocs; documents | 0.01% | Google Docs content fetcher |
| 110 | AhrefsBot/7.0 SA | 0.01% | SEO backlink crawler |
| 111 | Embedly/0.2 | 0.01% | Link preview and embedding bot |
| 112 | archive.org_bot | 0.01% | Internet Archive crawler |
| 113 | TelegramBot | 0.01% | Telegram link preview bot |
| 114 | allOrigins/3 | 0.01% | Data fetching bot for web origins |
| 115 | IABot/2.0 | 0.01% | IAB (Interactive Advertising Bureau) crawler |
| 116 | SentiBot | 0.01% | Sentiment analysis crawler |
| 117 | Googlebot snippet | 0.01% | Specialized Google snippet crawler |
| 118 | proximic | 0.01% | Contextual ad analytics crawler |
| 119 | dlvr.it/1.0 | 0.01% | Social media content distribution bot |
| 120 | IbouBot/1.0 | 0.01% | Generic web content crawler |
| 121 | 0.01% | Miscellaneous Google crawling activity | |
| 122 | PubMatic Crawler Bot | 0.01% | Ad verification crawler |
| 123 | Chrome Privacy Preserving Prefetch Proxy | 0.01% | Chrome prefetch proxy bot |
| 124 | GoogleAdwordsExpress | 0.01% | Adwords Express crawler |
| 125 | Qwantbot/1.0 | 0.01% | Qwant search engine crawler |
| 126 | coccocbot-image/1.0 | 0.01% | Cốc Cốc image crawler |
| 127 | YandexImages/3.0 | 0.01% | Yandex image crawler |
| 128 | um-IC/1.0 | 0.01% | Unknown bot, user agent um-IC |
| 129 | kakaotalk-scrap/1.0 | 0.01% | KakaoTalk content scraping bot |
| 130 | ReverseEngineeringBot/0.1 | 0.01% | Bot for reverse engineering/testing |
| 131 | Verity/1.1 | 0.01% | Web crawling/monitoring bot |
| 132 | VirusTotalBot | 0.01% | Malware analysis bot |
| 133 | TextRazor Downloader | 0.01% | TextRazor content fetching bot |
| 134 | YisouSpider | 0.01% | Yisou search engine crawler |
| 135 | Qwantbot/1.0_3151071 | 0.01% | Variant of Qwantbot crawler |
| 136 | CommaFeed/4.4.0 | 0.01% | RSS reader crawler |
| 137 | DecompilationBot/0.1 | 0.01% | Decompilation testing bot |
| 138 | DuckDuckGo-Favicons-Bot/1.0 | 0.01% | DuckDuckGo favicon fetcher |
| 139 | Paqlebot/2.0 | 0.01% | Content aggregation bot |
| 140 | l9scan | 0.01% | Security scanning bot |
| 141 | Dataprovider | 0.01% | Content/data collection bot |
| 142 | LinkpadBot/2.4 | 0.01% | SEO link checking bot |
| 143 | imageSpider | 0.01% | Image scraping bot |
| 144 | HatenaBlog-bot/0.02 | 0.01% | Hatena Blog crawler |
| 145 | NetSystemsResearch | 0.01% | Monitoring/security bot |
| 146 | GPTBot/1.0 | 0.01% | OpenAI GPTBot variant |
| 147 | YandexMarket/2.0 | 0.01% | Yandex Market crawler |
| 148 | OrgProbe/2.0.0 | 0.01% | Web probing bot |
| 149 | VelenPublicWebCrawler/1.0 | 0.01% | Public web crawler |
| 150 | Google PP Default | 0.01% | Google PageSpeed default crawler |
| 151 | BrightEdge Crawler/1.0 | 0.01% | SEO crawler for content marketing |
| 152 | KOCMOHABT Web Explorer | 0.01% | Web exploration bot |
| 153 | um-LN/1.0 | 0.01% | Unknown bot user agent |
| 154 | Nekstbot | 0.01% | Web crawling bot |
| 155 | PocketParser/2.0 | 0.01% | RSS parser crawler |
| 156 | Bytespider | 0.01% | SEO/content crawler |
| 157 | Whibse | 0.01% | Web crawling bot |
| 158 | SemrushBot-BA | 0.01% | SEO analysis crawler |
| 159 | Expanse | 0.01% | Web asset discovery bot |
| 160 | Custom-AsyncHttpClient | 0.01% | HTTP fetching bot |
| 161 | YandexMobileBot/3.0 | 0.01% | Yandex mobile crawler |
| 162 | Keydrop.io/1.0 | 0.01% | Link and SEO crawler |
| 163 | Instapaper/4.0 | 0.01% | Instapaper content fetcher |
| 164 | linkdex.com/v2.0 | 0.01% | SEO and link analysis bot |
| 165 | Y!J-ASR/1.0 | 0.01% | Yahoo Japan search crawler |
| 166 | GoogleDocs; drawings | 0.01% | Google Docs drawing fetcher |
| 167 | line-poker/1.0 | 0.01% | Unknown content fetcher |
| 168 | OrgProbe/2.2.0 | 0.01% | Web probing bot variant |
| 169 | Qwantbot/1.0_crawler-fetcher-25 | 0.01% | Qwantbot variant for crawling |
| 170 | SemrushBot-SI/0.97 | 0.01% | SEO crawler variant |
| 171 | TestLocally/1.0 | 0.01% | Testing bot |
| 172 | SemrushBot | 0.01% | SEO crawler |
| 173 | bl.uk_ldfc_bot/3.4.0 | 0.01% | British Library content bot |
| 174 | GCore Labs Cyberthreat Research | 0.01% | Threat research crawler |
| 175 | google-xrawler | 0.01% | Google scraping/testing bot |
| 176 | gocrawl v0.4 | 0.01% | Web crawling bot |
| 177 | adidxbot/2.0 | 0.01% | Ad indexing crawler |
| 178 | Xpanse | 0.01% | Web scanning bot |
| 179 | Baiduspider/2.0 mobile | 0.01% | Baidu mobile crawler |
Transformation Strategies for Publishers and Marketers
Data Privacy and Security, Performance and User Experience, Regulation Compliance, User and Revenue Growth
Stay ahead of the publishing curve
Privacy, security, monetization, and performance insights. A few emails a month, no spam.