# Coil — robots
User-agent: *
Allow: /
# Content Signals — declare AI content-usage preferences (contentsignals.org / IETF aipref).
# All-yes on purpose: Coil's growth IS being searched, cited in AI answers (ai-input), and
# recalled by models (ai-train). Mirrors Cloudflare's own robots.txt. Must stay inside the
# User-agent: * group — a Content-Signal line binds only to the group it sits in.
Content-Signal: search=yes, ai-input=yes, ai-train=yes
# ^ Bing Webmaster's robots Tester flags this line "Syntax not understood" — EXPECTED and
# HARMLESS. Content-Signal is a newer directive Bing's linter doesn't know; RFC 9309 requires
# crawlers to ignore unknown lines, and Bing's own tester shows Bingbot: Allowed on the same
# screen. Cloudflare's robots.txt trips the identical flag. Do NOT "fix" this by deleting it.

# NB (2026-07-29): 22 per-bot 'Allow: /' groups used to sit here as an explicit welcome.
# Removed — not because the welcome changed, but because RFC 9309 specific-group precedence
# meant GPTBot, ClaudeBot, PerplexityBot, CCBot and Bingbot matched their own group and
# therefore NEVER READ the Content-Signal line above: the affirmative invitation was inert
# for exactly the crawlers it was written for. One universal group, everyone reads it.

# Googlebot-only group (2026-09-06, operator-approved). GSC coverage export 09-06: 33 pages indexed,
# 662 "Discovered - currently not indexed" — the 564 noindex,follow /stocks/ mesh pages plus ~90 real
# sitemap pages Google has found and never crawled. On a near-zero-backlink domain the crawl budget is
# tiny, and the mesh was competing for it. RFC 9309 precedence: Googlebot reads ONLY this group, so it
# must be complete on its own — everything not listed here stays allowed. The '*' group above (and its
# Content-Signal line) still binds every other crawler, AI crawlers included; /stocks/ pages stay
# noindex for them as before. Bing is NOT in this group on purpose: it indexes the site fine.
User-agent: Googlebot
Disallow: /stocks
Disallow: /sitemap-stocks.xml

Sitemap: https://coil.trade/sitemap.xml
# sitemap-stocks.xml intentionally NOT advertised (2026-07-18): those 564 pages are
# noindex,follow — advertising them as crawl-worthy just spends crawl budget we need
# on the pages that actually rank and sell.