urlkit[ CREATE / [bot] ]
Robots.txt generator
Create a simple robots.txt file with clear crawler path rules.
02 / Your result
Your result will appear here.
Start with your own input or try an example.
What it does
Set up a small crawler policy with an allow-crawling preset, a request to stop crawling, or your own Allow and Disallow paths. Add a sitemap address if you have one, then download plain text for your site root. The file is crawler guidance, so it must not be used to protect private pages.
Useful when
- Write a first robots.txt file for a small site.
- Ask matching crawlers to skip internal search paths.
- Add a sitemap location to a reviewed crawler policy.
How to use it
- Choose a preset or enter custom paths for the User-agent: * group.
- Optionally add the full URL of your sitemap, then review the generated rules.
- Download robots.txt and review any existing file before replacing it at your site root.
A few useful details
Does a Disallow rule hide or secure a page?
No. Robots.txt is public and crawlers can ignore it. A disallowed URL may still appear in search results. Use real authentication for private content and appropriate indexing controls when you need to manage search visibility.
Which custom rules are supported?
One User-agent: * group, with up to 100 combined Allow and Disallow lines and 32 KiB of rule input. Paths start with / and are case-sensitive. Asterisks match text and a final dollar sign anchors the end. Use %20 for spaces; comments and custom user-agent groups are not accepted in the path fields.
Where do I put the file?
Publish it as /robots.txt at the root of the exact protocol, host, and port it should apply to. A file on one subdomain does not govern another. This generator does not upload the file, test crawler behavior, or verify your sitemap URL.