Does robots.txt protect private pages?

No. It is a public file and only asks well-behaved crawlers to stay away. Anyone can read it, and it does not secure anything. Use authentication to protect private content.

Where do I upload the file?

At the root of your site, so it is available at https://yourdomain.com/robots.txt. Each subdomain needs its own file.

Why is there no Crawl-delay option?

Google ignores Crawl-delay and other crawlers interpret it differently, so the generator only writes widely supported directives: User-agent, Allow, Disallow and Sitemap.

Do all crawlers understand * and $ in paths?

Major search engines do, but not every crawler. Test important rules with the tools offered by the search engines you care about.

Robots.txt Generator

Build a clean, valid robots.txt file with crawler groups, allow and disallow rules and your sitemap — then copy or download it.

Free
No signup
Private

Good to know: robots.txt is a request that well-behaved crawlers follow. It is not a security feature, it does not hide pages, and a blocked URL can still appear in search results if other sites link to it. To keep a page out of search results, use a noindex tag or password protection.

Start from a preset

Sitemap URLs (one per line, full URL)

robots.txt

The file must be placed at the root of your domain (for example https://example.com/robots.txt). * and $ wildcards in paths are understood by major search engines, but not by every crawler. Crawl-delay is not included because Google ignores it.

Overview

Build a clean, valid robots.txt file with crawler groups, allow and disallow rules and your sitemap — then copy or download it.

Features

•Multiple crawler groups
•Allow, Disallow and Sitemap lines
•Presets for common site types
•Live preview with format validation
•Copy or download the finished file

Benefits

Valid syntax with instant error checking
Presets that give you a sensible starting point
Protects against accidental formatting mistakes
Free, private and runs in your browser

Use Cases

Launching a new website
Blocking admin, cart or search result pages from crawling
Keeping a staging site from being crawled
Declaring the sitemap location for search engines

Users Also Use

How to Use

Start from a preset (general website, blog, e-commerce or staging) or build your own groups.
For each crawler group enter the user-agent, the paths to disallow and any allow exceptions.
Add your sitemap URLs, then copy or download robots.txt and upload it to the root of your domain.

FAQ

Does robots.txt protect private pages?: No. It is a public file and only asks well-behaved crawlers to stay away. Anyone can read it, and it does not secure anything. Use authentication to protect private content.
Will a Disallow rule remove a page from Google?: Not reliably. A blocked URL can still be listed if other pages link to it. To keep a page out of search results use a noindex directive and leave the page crawlable so the directive can be seen.
Where do I upload the file?: At the root of your site, so it is available at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
Why is there no Crawl-delay option?: Google ignores Crawl-delay and other crawlers interpret it differently, so the generator only writes widely supported directives: User-agent, Allow, Disallow and Sitemap.
Do all crawlers understand * and $ in paths?: Major search engines do, but not every crawler. Test important rules with the tools offered by the search engines you care about.

About Robots.txt Generator

Create a robots.txt file without memorising the syntax. Add crawler groups, block sections you do not want crawled, allow exceptions, and point crawlers to your XML sitemap.