What is fasthttp?

fasthttp is software in the HTTP clients and libraries category.

Bot name
fasthttp
Developer
Unknown
Type
Software
Respects robots.txt
No
robots.txt user agent
fasthttp
Main crawl location
The Netherlands (71%)
Signs requests (Web Bot Auth)
No

About fasthttp

fasthttp is a fast HTTP library for Go. Its default user agent is fasthttp, sent by Go programs that do not set their own, often scanners and scrapers.

HTTP clients are programming libraries and command line tools. Developers use them for scripts, integrations and scrapers. Real browsers never send these user agents, so treat them as automated traffic and block them on pages meant for people.

fasthttp user agent

fasthttp identifies itself with this user agent string:

  • fasthttp

Anyone can copy a user agent string. Transfon Security also checks where each request comes from, so a request that claims to be a known crawler but comes from the wrong network is treated as a fake, not as the real crawler.

Where fasthttp crawls from

Share of fasthttp requests to sites protected by Transfon Security in the 7 days to 10 November 2026.

Countries

The Netherlands71%
Poland13%
Singapore8%
Belarus8%

Networks

TechTies Inc. AS19717071%
IP Volume inc AS20242513%
Akamai Technologies, Inc. AS639498%
Republican Unitary Telecommunication Enterprise Beltelecom AS66978%

How to block fasthttp in robots.txt

fasthttp does not read robots.txt. The rules below record your policy, but they will not stop it. To actually block fasthttp, use a WAF rule as shown in the next section.

Block fasthttp from the whole site

User-agent: fasthttp
Disallow: /

Block fasthttp from some folders

User-agent: fasthttp
Disallow: /private/
Disallow: /search

Allow fasthttp only on some folders

User-agent: fasthttp
Allow: /blog/
Disallow: /

Allow fasthttp and block other crawlers

User-agent: fasthttp
Allow: /

User-agent: *
Disallow: /

Things to know

  • Each host needs its own file. Rules on www.example.com do not apply to shop.example.com.
  • A crawler obeys only the most specific group that names it. Once you add a User-agent: fasthttp group, fasthttp ignores your User-agent: * rules, so repeat any that should still apply.
  • The user agent name is not case sensitive. We take fasthttp from fasthttp's user agent; check the operator's own documentation if it publishes a different robots.txt name.
  • Crawlers cache robots.txt, so changes can take up to a day to apply. Disallow stops crawling but does not remove pages already collected or indexed.

Limits of robots.txt

  1. fasthttp may not follow the rule. fasthttp does not read robots.txt at all, so a rule for it has no effect.
  2. The user agent could be fake. robots.txt matches the name a request claims. Anyone can send fasthttp's user agent, and a scraper using it is not bound by the operator's robots.txt policy. Only checking the source network tells a real fasthttp from a fake one.

Treat robots.txt as a statement of your policy, and enforce it where requests arrive.

Who should care about fasthttp

Publishers

fasthttp is a common tool for scrapers. Real readers never use it, so block it on article pages and keep it out of ad requests.

Transfon Security for publishers

Media buyers and advertisers

Traffic from fasthttp on landing pages is automated. It inflates clicks and visits, so filter it from campaign data and raise it with the ad network.

Transfon Security for advertisers

E-commerce

Price scrapers, stock checkers and card testing scripts often run on fasthttp. Block it on product, cart and checkout pages.

Transfon Security for e-commerce

SaaS and APIs

fasthttp is normal for API clients but not for your web app. Allow it only on API routes that need a key, and rate limit it there.

Transfon Security for SaaS and APIs

Block fasthttp with Transfon Security

fasthttp ignores robots.txt, so it has to be stopped when the request arrives. Transfon Security shows fasthttp by name in the Top Bots report, so you can see how often it visits and which pages it reads. Block or allow it with a User Agent rule, or slow it down with a rate limit decision rule.

Transfon Security

Control fasthttp on your site

  • See every fasthttp visit and the pages it reads
  • Spot bots that fake trusted crawler names
  • Block, allow or rate limit fasthttp with one rule
  • Works with or without robots.txt support

Frequently asked questions

What is fasthttp?

fasthttp is software in the HTTP clients and libraries category. fasthttp is a fast HTTP library for Go. Its default user agent is fasthttp, sent by Go programs that do not set their own, often scanners and scrapers.

Does fasthttp respect robots.txt?

No. fasthttp does not follow robots.txt, so a Disallow rule will not stop it. Block it by user agent at your server, CDN or WAF instead.

How do I block fasthttp?

Block requests whose user agent contains "fasthttp" at your server, CDN or WAF, because fasthttp does not read robots.txt. In Transfon Security that is one User Agent block rule.

Is robots.txt enough to block fasthttp?

No. robots.txt is voluntary, so fasthttp may not follow it, and anyone can send fasthttp's user agent, so a fake fasthttp ignores your rule. Enforce the rule where requests arrive, at your WAF, and check the source network.

Should I block fasthttp?

HTTP clients are programming libraries and command line tools. Developers use them for scripts, integrations and scrapers. Real browsers never send these user agents, so treat them as automated traffic and block them on pages meant for people.

Related bots

All HTTP clients and libraries · Full bot directory

See Which Bots Visit Your Site

Transfon Security identifies every bot in this directory in your traffic reports, flags fake search crawlers, and lets you allow, block or rate limit each bot.

Stay ahead of the publishing curve

Privacy, security, monetization, and performance insights. A few emails a month, no spam.

By subscribing you agree to our privacy policy.