Reference guide for resolving the most frequent configuration errors and warnings.
Your robots.txt blocks the entire site
A `Disallow: /` rule in the group that applies to Googlebot tells every crawler to stay off every URL. This is the single most damaging robots.txt mistake, and it is usually left behind after a staging deploy.
The Fix: Remove the `Disallow: /` line, or narrow it to the paths you actually want kept out. If you want everything crawlable, an empty Disallow value is the explicit way to say so.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
The file is empty
An empty robots.txt is valid and means "crawl everything". That may be exactly right, but an empty file is far more often a build that wrote the file without its contents.
The Fix: If you meant to allow everything, add an explicit group and a sitemap reference so the intent is visible to the next person who reads it.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
An HTML page is being served instead of robots.txt
The response is a web page, not a text file. Crawlers cannot parse it, so none of your rules apply. This normally means the server has no robots.txt and is returning a soft 404 — a "not found" page with a 200 status.
The Fix: Serve a real text file at /robots.txt with the content type text/plain. If the file genuinely does not exist, return a 404 rather than an HTML page.
No Sitemap directive
Your robots.txt does not tell crawlers where your sitemap is. This is not an error, but it is the cheapest discovery win available — every major crawler reads it.
The Fix: Add a Sitemap line with the absolute URL. It can appear anywhere in the file and applies to all crawlers, regardless of which group it sits in.
Sitemap: https://example.com/sitemap.xml
A directive is misspelled
Crawlers ignore any line whose field name they do not recognise. They do not warn you and they do not guess — a misspelled `Disallow` simply does nothing, so a rule you believe is protecting a path is not.
The Fix: Correct the spelling to the exact directive name. The valid fields are User-agent, Allow, Disallow, Sitemap and Crawl-delay.
A line has no colon
Every robots.txt directive is `Field: value`. A line without a colon is not a directive and is skipped in full.
The Fix: Add the colon. `Disallow /admin` must be `Disallow: /admin`.
A path does not start with a slash
Allow and Disallow values are matched against the URL path, which always begins with `/`. A value without one can never match anything.
The Fix: Add the leading slash: `Disallow: admin` becomes `Disallow: /admin`.
Rules appear before any User-agent line
Allow and Disallow rules only exist inside a group, and a group starts with a User-agent line. Rules written before the first User-agent belong to no group and are discarded.
The Fix: Move the rules below a User-agent line.
User-agent: *
Disallow: /admin