FlinkTools

Sitemap and robots.txt validator

Check that search engines and AI crawlers can find your pages, and only the ones you want.

Checked by our server in Frankfurt
Try an example

Our server checks this from outside your network. We keep only what you save as a report.

10 checks an hour, 30 signed in.

Your robots.txt with errors marked in place, the sitemaps found, URL counts and which AI crawlers are blocked appear here.

What it does

This check reads robots.txt and the sitemaps it points to, validates them against the rules search engines use, and tries a sample of the listed URLs. It also shows which AI crawlers your robots.txt allows or blocks.

How to use it

  1. 1Type your website address.
  2. 2Read the errors, each with its line number.
  3. 3Fix the files and check again.

Questions

What does robots.txt do?

It asks crawlers which parts of your site to stay out of. It is a request, not a lock.

Why does a sitemap need valid dates?

Search engines ignore a lastmod they cannot read, and trust the rest of the file less.

Can I block AI crawlers?

Add a User-agent group for the crawler with Disallow: /. Well-behaved crawlers obey it.

AD320 × 100