# Allow AdSense and Google bots User-agent: Mediapartners-Google Allow: / # Crawlers that collect content to train AI models. Blocking them is a # deliberate choice: the reader gets nothing back when a model is trained on # this catalogue, unlike a search engine that sends visitors here. # # Cloudflare's managed robots.txt used to prepend exactly this list. It only # reached the apex, so www.dofusfashionista.gg answered with the managed stub # alone -- nine rules and Allow: / -- and every rule below went unseen by any # crawler that arrived there. Keeping the list in this file makes the answer # the same on both hostnames, and puts it under review like the rest. # # Assistants that fetch a page because a reader asked for it (ChatGPT-User, # Claude-User, PerplexityBot, OAI-SearchBot) are deliberately absent: those # send readers back, and blocking them would cost visits. User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: * # search=yes : indexing and linking here is welcome. # ai-train=no : no training or fine-tuning on this catalogue. # use=reference : cite it, do not reproduce it wholesale. Content-Signal: search=yes,ai-train=no,use=reference # The public content lives at /, /guides/, /encyclopedia/, /sharedbuilds/, # /s//, /user//, /compare_sets/... : all crawlable. # Everything below is an action endpoint or a per-project/private page: not # indexable content. Crawling them only wastes crawl budget and fills Search # Console with 403 / soft-404 / noindex "errors". The */path/ form also covers # the version-prefixed copies (/beta/…, /dofus2/…, /retro/…, /touch/…). # Keep the public "create a project" landing, block the per-project editors. # Every version has its own landing and the sitemap submits all five, so the # Allow has to cover the prefixed ones too: */setup/ alone was telling crawlers # to skip four pages we ask them to index. Allow: /setup/$ Allow: /*/setup/$ Disallow: /setup/ Disallow: */setup/ Disallow: /understandbuild/ # A different build every time it is asked, so there is nothing here to index # and nothing stable to come back to. It is not a discovery path either: the # 1 980 shared builds are submitted by name in sitemap-pages.xml, each with its # own lastmod. Crawling this only produces redirect entries in Search Console. Disallow: /random/ Disallow: */random/ # Project workflow (solver, wizard, edition, per-project pages) Disallow: */wizard/ Disallow: */wizardpost/ Disallow: */wizardgetsliders/ Disallow: */fashion/ Disallow: */solution/ Disallow: */spells/ Disallow: */loadproject/ Disallow: */saveproject/ Disallow: */saveprojecttouser/ Disallow: */createproject/ Disallow: */duplicateproject/ Disallow: */duplicatemyproject/ Disallow: */duplicatesomeonesproject/ Disallow: */deleteprojects/ Disallow: */project/ Disallow: */initbasestats/ Disallow: */initbasestatspost/ Disallow: */statspost/ Disallow: */minstatspost/ Disallow: */inclusions/ Disallow: */inclusionspost/ Disallow: */exclusions/ Disallow: */exclusionspost/ Disallow: */options/ Disallow: */optionspost/ # Item / set editing and switching Disallow: */exchange/ Disallow: */itemexchange/ Disallow: */remove/ Disallow: */setitemlocked/ Disallow: */setitemforbidden/ Disallow: */setslotlockempty/ Disallow: */setitemstatoverride/ Disallow: */getitemdetails/ Disallow: */get_item_stats_compare/ Disallow: */getsharinglink/ Disallow: */get_compare_sharing_link/ Disallow: */hidesharinglink/ Disallow: */choose_item/ Disallow: */choose_set/ Disallow: */edit_item/ Disallow: */edit_set/ Disallow: */update_item/ Disallow: */update_set/ Disallow: */delete_item/ Disallow: */delete_set/ Disallow: */itemadd/ # Community / social actions Disallow: */addtag/ Disallow: */removetag/ Disallow: */postcomment/ Disallow: */deletecomment/ Disallow: */reportcomment/ Disallow: */votebuild/ Disallow: */follow/ Disallow: */unfollow/ Disallow: */workshop/ Disallow: */inventory/ Disallow: */feed/ # Staff-only tools Disallow: /admin-tools/ Disallow: /out/ Disallow: */out/ Disallow: /admin-comment-action/ # Account / auth Disallow: */manageaccount/ Disallow: */saveaccount/ Disallow: */save_char/ Disallow: */login/ Disallow: */login_page/ Disallow: */local_login/ Disallow: */register/ Disallow: */confirm_email/ Disallow: */email_confirmed/ Disallow: */check_username/ Disallow: */check_your_email/ Disallow: */change_password/ Disallow: */recover_password Disallow: */do_recover_password/ Disallow: */send/ # Sitemap location Sitemap: https://dofusfashionista.gg/sitemap.xml