Robots.txt Generator

Templates
Choosing a template will override your current custom rules.
Basic Settings
Helps search engines find your sitemap.
Custom Rules

Output: robots.txt

About Robots.txt Generator

Robots.txt Generator builds the directives that tell search-engine and AI crawlers which areas of a website they may request.

Configure a default policy, add rules for individual user agents, publish a sitemap location and export a ready-to-upload robots.txt file.

Key features

  • Ready-made templates: Start with presets for WordPress, Laravel, e-commerce platforms, Blogger and other common website setups.
  • Crawler-specific rules: Create multiple User-agent groups and combine Allow, Disallow and Crawl-delay directives for precise crawl control.
  • Sitemap and AI crawler controls: Add the sitemap URL and optionally append rules for widely used AI crawlers before copying or downloading the file.

How to use Robots.txt Generator

Follow these steps to use Robots.txt Generator.

  1. Choose a starting policy Select a template or choose whether the default * user agent may crawl the entire website.
  2. Add crawler rules Add User-agent groups, select Allow or Disallow and enter the directory paths each rule applies to.
  3. Add optional directives Set a crawl delay if needed, enter the absolute sitemap URL and decide whether to block common AI crawlers.
  4. Publish and validate the file Review the generated directives, copy or download robots.txt, upload it to the website root and test the live URL.

Usage tips

  • Upload the file at the website root so it is available at https://example.com/robots.txt.
  • Test important rules with the robots.txt tester in your search-engine webmaster console before publishing them.
  • Do not put passwords or sensitive paths in robots.txt: the file is public and compliant crawlers may still expose blocked URLs.
  • Use a noindex directive or access control when a page must not appear in search results; Disallow only controls crawling.

Frequently asked questions

Place it in the website root. For example.com, crawlers must be able to request https://example.com/robots.txt.
No. Disallow asks compliant bots not to crawl a path, but its URL can still be indexed. Use noindex or authentication when removal is required.
No. The file is public and malicious bots can ignore it. Protect private content with authentication and server-side authorization.
Disallow blocks crawling of a matching path for the selected user agent. Allow can reopen a more specific subpath inside a broader blocked path.