Robots.txt Generator
Choose which user-agents to allow or block — including AI crawlers like GPTBot, ClaudeBot, PerplexityBot and Google-Extended — add disallow paths, and reference your sitemap. Copy the finished file.
Place the file at the root of your domain: https://example.com/robots.txt
How it works
- 1Pick a starting template (allow all, block staging, etc.).
- 2Add allow/disallow rules per user-agent.
- 3Decide your policy for AI crawlers.
- 4Copy the file to your site root as /robots.txt.
Use cases
- Launching a new site
- Blocking staging or faceted URLs
- Setting an explicit AI crawler policy
- Fixing an over-restrictive robots file
Frequently asked questions
Yes. Every tool in the Nix Inside free tools platform is free to use with no signup. Tools that run entirely in your browser never send your data anywhere.
Related tools
SEO Checker
Run a quick on-page SEO check on any URL — title, meta, headings, links, schema and more.
Try toolKeyword Research Tool
Find keyword ideas with volume, difficulty and intent for a seed term.
Try toolKeyword Density Checker
Analyze word and phrase frequency and density in your content.
Try toolKeyword Rank Checker
Check where a domain ranks for a keyword in a given location.
Try toolNeed expert help? Get a free consultation
These tools handle the quick wins. For a prioritized plan across technical SEO, content, authority and AI visibility, talk to Nix Inside.