FlinkTools

robots.txt generator

Build a robots.txt with rules, AI crawler presets and a sitemap, and test a path against it.

Processed on your device, never uploaded

Rules

One per line. * and $ work as in Google.
AI crawlers

robots.txt

Upload it to the top folder of your site, at /robots.txt. Rules are a request: well-behaved crawlers follow them, others may not. Do not use it to hide private pages.

A robots.txt with 0 blocked paths and 9 blocked AI crawlers.Processed on your device, never uploaded

Test a path

Blocked
  • GPTBot (OpenAI): model training
  • ClaudeBot (Anthropic): model training
  • Google-Extended (Google): model training
  • Applebot-Extended (Apple): model training
  • CCBot (Common Crawl): model training
  • Bytespider (ByteDance): model training
  • Meta-ExternalAgent (Meta): model training
  • cohere-ai (Cohere): model training
  • Diffbot (Diffbot): model training

What it does

The generator writes a robots.txt from your rules, can block AI crawlers such as GPTBot and ClaudeBot, adds your sitemap, and tells you whether a path is allowed for a crawler.

How to use it

  1. 1Add the paths to keep out.
  2. 2Choose how to treat AI crawlers.
  3. 3Test a path, then upload the file to /robots.txt.

Questions

Does robots.txt stop crawlers?

It is a request. Well-behaved crawlers follow it. It does not hide private pages.

Will blocking AI crawlers hurt my search ranking?

Blocking training crawlers does not affect normal search. Blocking a search crawler can reduce your reach in that search.

How is a path tested?

With the usual rules: the crawler’s own group wins, the longest matching rule wins, and Allow wins a tie.

AD320 × 100