Robots.txt Generator – WordPress, Next.js & Block AI Bots

This robots.txt generator writes the file that tells search engines and other crawlers which parts of your site to stay out of. Start from a preset for your platform, add your own rules, choose whether AI crawlers may use your content, and download a file that is ready to upload. Then add page tags with the meta tag generator and structured data with the schema markup generator.

How to use the robots.txt generator

Pick a starting point to create robots.txt: allow everything, WordPress, Next.js or block everything (for staging sites).
Edit the paths to disallow or allow, one per line, and add rules for specific crawlers if needed.
Choose whether to block AI training crawlers, or all AI crawlers.
Add your sitemap URL, then copy or download robots.txt and upload it to your site's root folder.

Tips for best results

Never block your CSS or JavaScript files: Google needs them to see your pages properly.
robots.txt is public and only a request. Don't use it to hide private pages; protect them with a login instead.
To remove a page from Google, use a noindex tag, not robots.txt. A blocked page can still appear in results without a description.
After uploading, open yoursite.com/robots.txt in a browser and check it in Google Search Console's robots.txt report.

AI crawlers this tool can block

Model training	GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai
AI search and assistants	OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot

Frequently asked questions

Will blocking AI bots hurt my Google ranking?

No. Google-Extended only controls whether Google may use your content for Gemini training; normal Googlebot crawling and ranking are unaffected. Bing's search crawler is also left alone.

Should I block AI search assistants too?

That is a business choice. Blocking them keeps your content out of AI answers, but also removes the chance of being cited and getting visits from tools like ChatGPT search and Perplexity. Many sites block training bots but allow assistants.

Do all bots obey robots.txt?

Reputable search engines and the AI companies listed do. Scrapers and malicious bots often ignore it. For those, you need firewall rules, such as Cloudflare's bot settings.

What should a WordPress robots.txt contain?

Usually just a block on /wp-admin/ with an allow for /wp-admin/admin-ajax.php (needed by many themes and plugins), plus your sitemap. Avoid blocking /wp-content/ or /wp-includes/, which hold files Google needs.

Where do I upload robots.txt?

To the top level of your domain, so it opens at https://yoursite.com/robots.txt. A robots.txt inside a folder is ignored. Each subdomain needs its own file.

All seo tools