Robots.txt Generator
Build a valid robots.txt file with as many user-agent groups as you need. Add allow and disallow rules, a crawl delay and your sitemap, then copy or download the result.
Runs entirely in your browser — nothing is uploaded
Rule group 1
* means every crawler. Or name one, such as Googlebot.
Google ignores this. Bing and Yandex honour it.
Your robots.txt
User-agent: * Disallow: /admin/ Disallow: /api/ Sitemap: https://example.com/sitemap.xml
Upload this to the root of your domain so it is reachable at /robots.txt. It will not work anywhere else.
How to use this tool
- Set the user-agent — * covers every crawler.
- Add the paths you want to disallow, one per line.
- Add your sitemap URL.
- Download the file and upload it to the root of your domain.
What is Robots.txt Generator?
robots.txt tells crawlers which parts of your site they may request. It is a crawling instruction, not a security control and not an indexing control — anyone can read it, and a blocked URL can still appear in results if other pages link to it.
To keep a page out of the index, use a noindex meta tag instead. Blocking it in robots.txt actually prevents Google from seeing that noindex, which is a common and self-defeating mistake.
Best practices
- The file must live at the root: example.com/robots.txt. Nowhere else works.
- Never block CSS or JavaScript — Google needs them to render your pages.
- Do not use robots.txt to hide private content. It is a public file that lists exactly what you wanted hidden.
- Add your sitemap URL; it is the one directive that helps discovery rather than restricting it.
- Test changes in Search Console before deploying — one careless line can deindex a site.
Frequently asked questions
Does robots.txt stop a page appearing in Google?
No. It stops crawling, not indexing. A blocked URL can still be listed — usually without a description — if other sites link to it. Use a noindex meta tag to keep a page out of results.
Where do I upload the file?
The root of your domain, so it resolves at example.com/robots.txt. Crawlers look nowhere else, and a file in a subdirectory is simply ignored.
Does Google respect crawl-delay?
No. Google ignores it entirely — crawl rate is managed in Search Console instead. Bing and Yandex do honour it.
What happens if I have no robots.txt at all?
Crawlers assume everything is allowed, which is usually fine. A missing file is not an error.
Related tools
Want this done properly on your site?
These tools handle the checks. If you would rather someone did the work — the audit, the fixes and the content — that is what I do.
Results are provided for informational purposes and may vary depending on the website, browser, network and available data.