# App-aquatic robots.txt # # CRAWLER POLICY. Every search and AI crawler is deliberately allowed. This site # wants to be retrieved, quoted and named, and there is no paywall to protect. # That includes the training crawlers (GPTBot, ClaudeBot, CCBot): presence in # model weights is what produces unprompted brand recall, and blocking them does # not improve citation rates. Google-Extended and Applebot-Extended are likewise # NOT blocked - both govern training and grounding only, so blocking them would # forfeit Gemini grounding and Apple model presence while gaining nothing in # Search or in AI Overviews. # # There are deliberately NO per-crawler "Allow" groups below. A robots.txt group # does not inherit: a crawler obeys only the single most specific group that # matches its name. Writing "User-agent: Googlebot / Allow: /" therefore exempts # Googlebot from every Disallow in the "*" group - which, when tested, silently # re-exposed the internal documents and /archive/ to the one crawler that # matters most. The allow-everything policy is the default, so it needs no rule. # Bulk scraper with no published policy, repeatedly reported as ignoring # robots.txt. Blocking is bandwidth hygiene, not a citation decision. User-agent: Bytespider Disallow: / User-agent: * # Disallow lines come BEFORE "Allow: /" deliberately. Google resolves conflicts # by longest match, so order would not matter to it - but several parsers # (including Python's urllib.robotparser, which is what a lot of tooling uses) # take the first matching rule, and under those "Allow: /" swallows everything # below it. Written this way, both interpretations produce the same answer. # # Internal working documents and the generation pipeline. These are served only # because they sit in the published branch; nothing on the site links to them, # but they are crawlable and guessable. They include a link-acquisition # playbook, Search Console performance tables and competitor keyword research - # the documents most likely to turn an algorithmic question into a manual one. # Disallow is the stopgap; moving them out of the published branch is the fix. Disallow: /archive/ Disallow: /scripts/ Disallow: /plagiarism-report.txt # Wildcard and end-anchor are an extension, supported by Google, Bing and the # major AI crawlers but not by every parser. The documents it covers are also # listed individually below so the rule does not depend on that support. Disallow: /*.md$ Disallow: /APP_AQUATIC_SEO_BUILD.md Disallow: /AUDIT-GUIDES-COMPLIANCE.md Disallow: /BACKLINK_PLAYBOOK.md Disallow: /CONTENT_ROADMAP.md Disallow: /GA4-CONVERSION-SETUP.md Disallow: /GUIDE-TEMPLATE.md Disallow: /LOCALISATION_AGENTS.md Disallow: /PRODUCT_HUNT_LAUNCH.md Disallow: /README.md Disallow: /SEO_AND_AI_RECOMMENDATIONS.md Disallow: /SEO_AQUARIUM_APP_WEEKLY_LOOP.md Disallow: /SEO_RESEARCH_TOP_SITES_AND_KEYWORDS.md Disallow: /SITE_AUDIT_AND_NEXT_STEPS.md Disallow: /WEBSITE_AGENT.md Disallow: /crowd-thread-digest-blueprint.md Disallow: /stocking-calculator-roadmap.md Allow: / Sitemap: https://app-aquatic.com/sitemap.xml