What is a robots.txt file?
A robots.txt file sits at the root of your site and tells search engine crawlers which pages or folders they may or may not crawl. It is one of the first files Google looks for when it visits your site.
SEO Tools
Robots.txt Generator
Create a robots.txt file to control search crawlers.
User-agent: *
Save this as robots.txt in the root of your website (example.com/robots.txt). Generated in your browser.
How it works
1
Start with a preset
Choose allow all, block all, or a common CMS default, or start from scratch.
2
Add your rules
Set the user-agent, list the paths to disallow or allow, and add your sitemap URL.
3
Download robots.txt
Copy or download the file and upload it to the root of your website.
Robots.txt Generator features
- Quick presets for allow all, block all, and WordPress defaults.
- Add any number of Disallow and Allow paths.
- Set the user-agent to target specific crawlers.
- Optional crawl-delay and sitemap URL.
- Copy the output or download a ready-to-use robots.txt file.
- Runs in your browser. Free with no signup and no limits.
Robots.txt Generator FAQ
How do I create a robots.txt file for free?
Pick a preset or add your own allow and disallow rules, then download the file. Upload it to the root of your site so it is reachable at yoursite.com/robots.txt. It is free and generated in your browser.
Where do I put the robots.txt file?
It must live in the root directory of your domain, so it loads at yoursite.com/robots.txt. It will not work in a subfolder. Most hosts let you upload it via your file manager or FTP.
What is the difference between Disallow and Allow?
Disallow tells crawlers not to crawl a path. Allow explicitly permits a path, which is useful to open up a single file inside an otherwise disallowed folder. An empty Disallow means everything is allowed.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap line pointing to your sitemap.xml helps search engines discover all your pages faster. This tool adds it for you when you enter the URL.
Do I need to sign up?
No. The robots.txt generator is free with no account and no limits.
Related SEO Tools
Website SEO AuditSoonCheck key on-page SEO signals and get an actionable report.
Meta Tag GeneratorGenerate title, description, and meta tags for your pages.Schema GeneratorCreate JSON-LD structured data markup for rich results.SERP Snippet PreviewPreview how your page looks in Google search results.Recent from the blog
JPG vs PNG vs WebP: Which Image Format Should You Use? (2026 Guide)
JPG, PNG, and WebP all store images, but picking the wrong one gives you blurry photos, huge files, or lost transparency. Here is exactly when to use each, in plain English.
Read guide →How to Compress a PDF Without Losing Quality (2026 Guide)
A large PDF is a pain to email or upload, but most compressors either barely shrink it or turn your crisp text into a blurry mess. Here is how to reduce PDF file size while keeping it looking clean.
Read guide →How to Merge PDF Files for Free (2026 Guide)
Combining several PDFs into one file used to mean paid software or uploading private documents to a stranger server. Here is the faster, safer way to merge PDF files for free.
Read guide →