About this tool
Robots.txt Generator
Generate a simple robots.txt starting point for a website. Use it to communicate crawl preferences, while keeping important pages and assets accessible to search engines.
How to use it
- Enter the information requested in the tool above.
- Run the check and review the result carefully.
- Apply the useful recommendations, then check the page again after making changes.