Build a valid robots.txt file for your website in seconds. Add allow and disallow rules per crawler, set a crawl delay, point to your sitemap, then copy or download the file and upload it to your site root.
How to Use
- Click Add rule for each crawler: choose the user agent, Allow or Disallow, and the path.
- Set an optional crawl delay in seconds.
- Enter your site URL — the sitemap address is suggested automatically.
- Click Generate robots.txt, then copy or download the file and upload it to your site root.
Key Features
- Grouped crawler rules — rules are organised per user agent automatically.
- Common crawlers listed — Googlebot, Bingbot, DuckDuckBot, Baidu and Yandex included.
- Crawl delay and sitemap — optional directives supported.
- Copy and download — one-click clipboard copy and robots.txt file download.
- 100% private — runs entirely in your browser; nothing is sent or stored.
FAQs
Where do I put the robots.txt file?
Upload it to your website’s root folder so it is served at example.com/robots.txt. Crawlers only look in that one location.
Do all crawlers obey robots.txt?
Reputable search engine crawlers do. Malicious bots may ignore it, so never rely on robots.txt to hide sensitive content — use proper access controls instead.
What is a crawl delay?
It asks crawlers to wait a number of seconds between requests, reducing load on your server. Not all crawlers support it.
What is the difference between Allow and Disallow?
Disallow blocks crawlers from a path; Allow explicitly permits one — useful for opening an exception inside a blocked folder.
Related Tools
Try these related tools: Meta Tag Generator, Sitemap Generator and SEO Content Checker.