# ============================================================================
# robots.txt — QNT Sport
# ============================================================================
# Current live robots.txt is just "User-agent: * / Allow: *" with a sitemap
# line — that's not the problem. The problem (confirmed 2026-07-24 by
# fetching live with GPTBot/ClaudeBot/PerplexityBot UAs) is that the site's
# *dynamic rendering* only recognizes Googlebot's UA, so every AI crawler
# below is technically "allowed" but receives a 10KB empty shell instead of
# the ~690KB server-rendered page Googlebot gets. Fixing that server-side
# allowlist matters far more than anything in this file — but this file is
# still where you make an explicit, auditable policy decision per bot,
# rather than leaving it to a blanket wildcard.
# ============================================================================

# --- Retrieval / answer-engine crawlers -------------------------------------
# These fetch pages at query time (or near it) to ground an AI-generated
# answer with a citation back to your site — functionally closer to a
# search engine than a training scraper. For a public storefront that wants
# shopping visibility inside ChatGPT/Perplexity/Copilot answers, allow these.
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Amazonbot
Allow: /

# --- Training-only crawlers --------------------------------------------------
# These scrape for model pre-training corpora, not live answer retrieval.
# Allowing them doesn't drive traffic back to you the way retrieval bots
# do — but for a public product catalog (not paywalled/proprietary content),
# there's little downside, and being present in model training data is
# itself part of how models "know" your brand exists when a user asks a
# product question with no live browsing involved. Block only if you have a
# specific reason to withhold catalog content from third-party model training.
User-agent: GPTBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: cohere-ai
Allow: /

# --- Everyone else ------------------------------------------------------------
User-agent: *
Allow: /

Sitemap: https://www.qntsport.in/sitemap.xml