# Allow all search engines # # Content-Signal (contentsignals.org, Cloudflare's Content Signals Policy) is a # group-member record: it applies to the `User-agent` group it sits in, exactly # like `Allow` and `Disallow`. It is therefore declared HERE, in the `*` group, # so it covers every crawler — the reference implementation Cloudflare serves on # its own properties puts it last inside `User-agent: *`. # # The three signals say what a crawler may do with the content once fetched: # search=yes index it and return links and short excerpts # ai-input=yes feed it to a model at answer time (RAG, grounding, citations) # ai-train=yes train or fine-tune on it # # All three are `yes` by the decision of 2026-08-15 (see the training-corpus # group below): a marketing site wants to be in the corpus that answers # questions about tools like this one. TO FLIP ONE, change the value on the # `Content-Signal:` line below AND the matching `User-agent` group further down # — a signal that says `ai-train=no` while `CCBot` is still `Allow: /` states # two contradictory policies, and a crawler is free to believe either. User-agent: * Allow: / Disallow: /api/ Disallow: /404 Content-Signal: search=yes, ai-input=yes, ai-train=yes # AI search engine crawlers User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Google-Extended Allow: / # Training-corpus crawlers, deliberately let in (decision of 2026-08-15). # CCBot is the notable one: it is not a search engine, but its corpus feeds # a large share of the models that answer questions about tools like this # one. Excluding it removed Evalmee from training data without removing it # from anywhere a buyer looks. User-agent: CCBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: meta-externalagent Allow: / User-agent: Amazonbot Allow: / # Sitemap Sitemap: https://evalmee.com/sitemap-index.xml # Schema Map — NLWeb Schema Feeds (github.com/nlweb-ai/website, SCHEMA_SPEC.md) # An index of the aggregated structured-data feeds this site publishes, so a # consumer collects them without crawling 181 pages for their markup. Generated # at build from the same locale list the `` uses: # src/pages/schemamap.xml.ts schemamap: https://evalmee.com/schemamap.xml # LLMs.txt for AI crawlers # See: https://evalmee.com/llms.txt