A public exception
Disallow /private/ and allow /private/public-guide/ to describe a public section inside a broader excluded path.
One crawler group
Use * for all crawlers, or product tokens such as ExampleBot. Every agent listed here shares the same rules.
robots.txt preview
Publish at the origin root, for example https://example.com/robots.txt. Rules are case-sensitive path patterns. * matches any sequence; $ anchors the end. The most specific matching rule applies, with Allow winning an equal-length tie.
Robots rules are crawler guidance, not access control. Disallowed URLs may still appear in search results. No crawl-delay or nonstandard directives are generated; sitemap discovery is an extension supported by major crawlers.
Rules and sitemap URLs stay in the page. The downloaded file contains exactly the preview text.
How to use the Robots.txt Generator
- 01
Start with the site example, allow-crawling preset, or disallow-all preset.
- 02
Enter crawler product tokens and one allow or disallow path per line. Add absolute sitemap URLs if needed.
- 03
Review the preview and publish the UTF-8 file at your origin’s root /robots.txt path.
Examples
Block every path
Disallow: / asks a matching compliant crawler not to crawl any path on the origin. It does not require authentication or prevent access.
Allow everything by default
An empty Disallow value is the conventional allow-all form. Blank rule input produces that line instead of accidentally generating Disallow: /.
One group with shared rules
Each user-agent line in the generated group receives the same allow and disallow rules. Use separate manually reviewed groups if different crawlers need different policies. This interface deliberately avoids silently merging different policies into one group.
Path matching requires care
Paths are case-sensitive. Prefix matches can cover more URLs than an exact page: /news may also match /newsletter. Use $ for an end anchor when appropriate and * for a wildcard. A more specific matching rule takes precedence; equally specific allow and disallow rules favor allow.
Crawling, indexing, and access control
robots.txt is a voluntary crawl protocol. It cannot protect private files, and a disallowed URL can still be known to a search engine through links. Use authentication for private content. Indexing directives such as noindex belong in appropriate page metadata or response headers and require the crawler to see them.
What to know
- Generates one shared crawler group. No automatic crawler simulation, remote fetch, crawl-delay, or access protection.
Common questions
Where should I upload the file?
At /robots.txt on the relevant origin, not inside the content directory you want to exclude. A different hostname, scheme, or port can have its own policy.
Why is there no crawl-delay field?
Crawl-delay is not part of RFC 9309 and support varies. This generator limits its core output to the standard group and path rules, plus the widely used Sitemap extension.
Are sitemap URLs checked online?
No. Syntax is checked locally, but the tool does not fetch a sitemap, confirm that it exists, or inspect its contents.