Robots.txt Generator

Build a robots.txt file for one user-agent token. Enter Allow and Disallow paths line by line and add an optional sitemap URL.

Draft crawl instructions as plain text. The generator validates the user-agent token and requires rule paths to begin with a slash.

Sitemap URL (optional)

https://example.com/page

User agent

Allow paths (one per line)

Disallow paths (one per line)

Crawl rules are public instructions, not access controls. Protect private pages with authentication.

Your input stays on this device for this operation. We don’t upload or store it.

About Robots.txt Generator

Draft crawl instructions as plain text. The generator validates the user-agent token and requires rule paths to begin with a slash.

Generate separate Allow and Disallow lines.

Add an optional absolute sitemap URL.

1
Choose the crawler token
Set User agent and review the default Disallow path
2
Enter crawl paths
Enter one path per line and an absolute Sitemap URL if needed
3
Save robots.txt
Generate, review and download robots.txt

User-agent, rule paths, optional sitemap URL → robots.txt

Each generated file contains one user-agent group. Inline hash comments are rejected in rule fields.
Crawl directives do not protect private content or guarantee removal from search results. Review scope before placing the file at the site root.

For user-agent *, disallow /admin/ and add https://example.com/sitemap.xml as the sitemap URL.

1 answers

Does Disallow keep a page out of search results?

It is a crawl instruction. Use the appropriate indexing or access control for your actual goal.