whitelist the good bots, block the costly ai scrapers
Tick the crawlers you want, switch off the AI scrapers you don't, add your own rules, and download a clean, standards-compliant robots.txt. Everything below is live — no account needed.
▋quick start
whitelist good bots
toggle on the search engines, archives, and social crawlers you want to keep.
block AI scrapers
toggle off the training crawlers. the live preview on the right updates instantly.
download your file
hit copy or download robots.txt — drop the file at your site root and you're done.
configs are saved in this browser. go pro to sync them to your account across every device.
pro configs get a hosted robots.txt url — always live, updates itself — and can auto-block new ai scrapers as we add them. go pro
whitelist good bots
search, social, and archive crawlers worth keeping.
block ai scrapers
a curated list of training and answer-engine crawlers.
custom rules
no custom rules. add one to allow or disallow a specific path (e.g. disallow /admin).
robots.txt · live valid# robots.txt — generated by robot.guard # robotguard.ogbuilds.ai # allowed crawlers User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / User-agent: ia_archiver Allow: / User-agent: facebookexternalhit Allow: / User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / # blocked ai scrapers User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Bytespider Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: /
download the file and place it at your site root (yoursite.com/robots.txt). robots.txt is a request compliant crawlers honour — pair it with a firewall for bots that ignore it.
one file, three jobs.
robot·guard turns the most overlooked file on your server into a control panel. pick what crawls you, skip what doesn't.
whitelist good bots
keep googlebot, bingbot, applebot, and the social crawlers that build your presence. tick once, done.
block AI scrapers
opt out of gptbot, claudebot, bytespider, and the rest. 16 training crawlers tracked and updated.
add custom rules
need to protect a specific path or block a crawler not on the list? write the directive yourself.
three steps to a clean file.
pick your bots
tick the crawlers you want to keep. untick the AI scrapers you don't.
preview live
see the exact robots.txt as you configure — every directive, in order, instantly.
download and ship
copy the file or hit download. drop it at your site root. done — no account needed.
bots tracked and maintained
sign-up required
click to download
start for free. go pro when you're ready.
the editor is free forever. pro adds a hosted robots.txt url that updates itself, auto-blocking of new ai scrapers, and cloud saves synced across every device.