NoticeTechnical
Missing robots.txt File: How to Fix It
robots.txt is a text file at the root of your domain that tells crawlers which areas they may visit and where your sitemap is.
Quick fix: Add a robots.txt file at the root of your site that allows crawling and lists your sitemap.
Why it matters for SEO
It is not required, but it is the standard place to list your sitemap and to keep crawlers out of admin, search, and cart pages.
How to fix it
- 1Create a file named robots.txt in your site root.
- 2Allow crawling of public pages and block private areas.
- 3Add a Sitemap: line with the full sitemap URL.
- 4Test it at yourdomain.com/robots.txt.
Example
User-agent: *
Disallow: /admin/
Disallow: /cart
Sitemap: https://example.com/sitemap.xmlFree tools to fix it
Frequently asked questions
Can robots.txt remove pages from Google?
No. It blocks crawling, not indexing. Use noindex to remove pages from results.
Where must robots.txt be?
At the root of the domain, for example https://example.com/robots.txt.
Does your website have this issue?
Run a free SEO audit to check this and 40+ other issues in seconds.