# robots.txt for Game Dev Outsourcing # https://www.gamedevoutsourcing.com # # ONE group, deliberately, and with NO BLANK LINES inside it. Both of those are # load-bearing — see the two notes at the bottom of this file before you tidy # anything here. Comment lines between rules are safe; empty ones are not. # # Content Signals Policy (Cloudflare, 2025): an advisory statement of intent # that sits alongside the access rules, not a replacement for them. All three # are yes — being read, trained on and cited by AI assistants is this # directory's largest referral channel, so there is nothing here we want # withheld. Advisory only: it is not part of RFC 9309, most crawlers do not # parse it (they must ignore the unknown field), and it neither grants access # the rules below deny nor licenses anything. User-agent: * Content-Signal: search=yes, ai-train=yes, ai-input=yes # Three surfaces live under /api/ and must stay crawlable despite the blanket # /api/ block below: the MCP server, the public JSON API, and the badge # artwork. RFC 9309 resolves conflicts by longest matching path, so these beat # "Disallow: /api/" for their prefixes only — /api/mcp is 9 characters against # /api/'s 5, /api/public/ is 12, /api/badge/ is 11. They are also written # before the Disallow so that older first-match-wins parsers reach the same # answer. Everything else under /api/ stays blocked. # # /api/badge/ is the one path here that is deliberately embedded on OTHER # people's domains: the image embed snippet ships # . Blocked, a # crawler rendering a studio's page cannot fetch the artwork it is being shown, # and the image is ineligible for image search. The anchor around it always # passed equity either way — this is about the artwork, not the link. Allow: /api/mcp Allow: /api/public/ Allow: /api/badge/ # Private and non-indexable areas (admin, API, auth, authenticated app) Disallow: /admin Disallow: /admin/* Disallow: /api/ Disallow: /api/* Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /verify-email Disallow: /dashboard Disallow: /messages Disallow: /profile Disallow: /profile/* Disallow: /projects Disallow: /projects/* Disallow: /onboarding # Landing page A/B test variants (not for indexing) Disallow: /landing/ # The catch-all goes LAST, not first. Under RFC 9309 (Google, Bing, OpenAI, # Anthropic, Perplexity) the longest matching rule wins and the position of # this line is irrelevant. Under the older first-match-wins convention — still # implemented by Python's stdlib urllib.robotparser and anything built on it — # a leading "Allow: /" matches every URL and nullifies every Disallow above it, # which is what this file did until August 2026. Last, both conventions agree. Allow: / # --------------------------------------------------------------------------- # Note 1: rule order and blank lines inside the group above (fixed Aug 2026) # # Two separate legacy-parser hazards were fixed here, and both had made the # whole private-path list a no-op for anything not using an RFC 9309 parser: # # 1. Blank lines. RFC 9309 and Google end a group only at the next user-agent # line and ignore blank lines. The older convention ends a group at the # first BLANK line and then discards every rule after it, since those rules # belong to no group. This file used to have a blank line between the # "Allow: /" and the Disallow list, so the Disallow list vanished. # 2. A leading catch-all "Allow: /", which under first-match-wins matches every # URL before any Disallow is ever consulted. It now sits last. # # Keep the group contiguous: comment lines are safe as separators (they are # stripped after the blank-line check), empty lines are not. Verified with # Python's urllib.robotparser, which implements both legacy behaviours — before # the fix it reported /admin, /dashboard, /profile and /landing/ as crawlable; # after it reports them blocked, with /api/mcp and /api/public/ allowed. # # --------------------------------------------------------------------------- # Note 2: why there is no list of AI crawler tokens here (removed Aug 2026) # # This file used to carry nine named groups — GPTBot, OAI-SearchBot, # ChatGPT-User, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, # Google-Extended, CCBot — each containing nothing but "Allow: /". That was a # security bug, not a courtesy. # # Per RFC 9309 and Google's robots.txt spec, a crawler obeys exactly ONE group: # the most specific one whose user-agent token it matches. Named groups do NOT # combine with the "*" group, they REPLACE it. So a named group containing only # "Allow: /" does not "also allow" that bot — it discards every Disallow above # for that bot. Those nine crawlers were being told they could crawl /admin, # /admin/*, /dashboard, /login, /register, /forgot-password, /reset-password, # /verify-email, /messages, /profile, /profile/*, /projects, /projects/*, # /onboarding, /api/ and /landing/ (the A/B variants that must never be # indexed). Every token added to such a list widens that hole by one crawler. # # The fix is to have no named groups at all. The "*" group above is already # exactly the policy we want for every crawler, AI ones included: allowed # everywhere public, blocked on the private paths, plus the two agent # endpoints. One group, one policy, no exemption surface, and nothing to keep # in sync as vendors rename their tokens. # # If you are here because an article told you to add AI crawler tokens: # # - You do not need them in order to be maximally citable. Nothing is blocked. # Adding "User-agent: SomeBot / Allow: /" changes nothing except to unblock # the private paths for SomeBot. # - The tokens that decide whether we get CITED, as opposed to merely trained # on, are the search and user-fetch agents: OAI-SearchBot, Claude-SearchBot, # Claude-User, PerplexityBot, Perplexity-User and DuckAssistBot. Blocking a # training crawler costs nothing in AI search; blocking one of these costs # the citation itself — Anthropic states that blocking Claude-SearchBot "may # reduce your site's visibility and accuracy in user search results" and that # blocking Claude-User "may reduce your visibility for user-directed web # search". None of them is blocked here, because "*" allows them. # - anthropic-ai and Claude-Web are RETIRED tokens that predate ClaudeBot and # match no live crawler. Any advice naming them is stale; they did nothing # but widen the hole described above. # - Google-Extended does NOT control AI Overviews or AI Mode. Google's own # documentation says those are generated from the ordinary Googlebot crawl # and that "there are no additional requirements to appear in AI Overviews or # AI Mode". Google-Extended covers Gemini model training and Vertex AI # grounding only, and the sole real opt-out is a Search Console toggle, not a # robots.txt token. Blocking it would not remove us from AI Overviews; # allowing it is not what puts us there. # # The only legitimate reason to add a named group is to give one crawler a # DIFFERENT policy from "*". If you do that, you must restate the full Disallow # list inside it — a named group inherits nothing. # --------------------------------------------------------------------------- # Sitemap Sitemap: https://www.gamedevoutsourcing.com/sitemap.xml