robots.txt
document-trust • 8/14/2026
robots.txt
Policy: ALLOW ALL crawlers (search engines, SEO tools, AI bots)
Clients verify backlinks via Ahrefs/Semrush/Majestic — blocking these defeats the purpose
Only pure bulk training harvesters with zero citation/indexing value are blocked
--- DEFAULT: ALLOW ALL CRAWLERS ---
User-agent: *
Allow: /
--- PURE TRAINING HARVESTERS ONLY (blocked — no citation or SEO value) ---
User-agent: CCBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: Bytespider
Disallow: /
Sitemap: https://cabcdn.workers.dev/r/czla-6r3-t1-1