User-agent: * Allow: / # Internal working/preview pages — not for indexing Disallow: /og-images.html Disallow: /og-gen.html Disallow: /about-og.html Disallow: /about-gen.html Disallow: /drafts/ Disallow: /graphics/ Disallow: /components.html Disallow: /snippets/ Disallow: /AI%20Receptionist%20Graphics.html Disallow: /AI%20Agents%20Design%20Explorations.html Disallow: /Wrapstart%20Pitch%20Deck.html # --------------------------------------------------------------------------- # AI / answer-engine crawlers that send traffic back — explicitly welcome. # These power ChatGPT Search, Claude, Perplexity and Google AI answers, which # are a real acquisition channel for us. Blocking them would cost more than it # protects. # --------------------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # --------------------------------------------------------------------------- # Bulk scrapers, dataset harvesters and site-cloning agents — denied. # These send no traffic back and exist to copy pages wholesale. # --------------------------------------------------------------------------- User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: FacebookBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: AI2Bot Disallow: / User-agent: cohere-ai Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: PanguBot Disallow: / User-agent: Kangaroo Bot Disallow: / User-agent: YouBot Disallow: / # SEO crawlers left allowed on purpose — you use Ahrefs/Semrush to audit # wrapstart.com, and blocking them would break your own reports. User-agent: DotBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: Scrapy Disallow: / User-agent: HTTrack Disallow: / User-agent: SiteSucker Disallow: / User-agent: WebCopier Disallow: / User-agent: WebZIP Disallow: / User-agent: Wget Disallow: / User-agent: curl Disallow: / # No text and data mining reservation. Wrapstart content is open to the # answer engines listed above — being cited in an AI answer is a channel we # want, not one we defend against. See /.well-known/tdmrep.json Sitemap: https://www.wrapstart.com/sitemap.xml Disallow: /features/phone.html Disallow: /office-hours.html