Robots.txt Generator
Build a robots.txt from user-agent rules and a sitemap URL.
Rule Groups
Sitemap
robots.txt
Add a user-agent or a sitemap URL to generate a robots.txt.
More in this kitAll of Writing & SEO Tools
About Robots.txt Generator
The Robots.txt Generator builds a robots.txt file, which tells search engine crawlers which parts of your site they may visit. Fill in rules for each crawler and optionally point to your sitemap, then copy the finished file.
How to use it
- Enter a user-agent for the first group, such as * for all crawlers.
- List paths to block under Disallow and paths to permit under Allow, one per line.
- Use + Add Group to create separate rules for specific crawlers.
- Optionally add your sitemap URL, then copy the result and save it as robots.txt at the root of your site.
Good to know
- robots.txt must live at the top level of a domain, for example https://example.com/robots.txt.
- Blocking a page in robots.txt stops crawling but does not guarantee the page is removed from search results. Use a noindex tag for that.
- The file is public, so never list private paths in it expecting them to stay hidden.
Common questions
What does "Disallow: /" do?
It asks the matching crawler not to visit any page on the site. Use it with care, because applying it to all crawlers can remove a site from search.
Do I need a robots.txt at all?
Not strictly. Without one, crawlers assume they may visit everything. It becomes useful when you want to keep crawlers out of certain areas or point them to your sitemap.