# ─── Search engine crawlers ─── # A crawler obeys ONLY its own named group, so the blocks of the * group below are repeated here (15-Nov-2026: # these two groups held only "Allow: /", so Google and Bing crawled admin, search and filter pages). Pagination # (?page=) stays open for them so they can reach products beyond the first page of a category or brand. # 17-Nov-2026: every search/filter rule is written twice. "/*?sort=" only matches when the parameter comes # first in the address; "/*&sort=" catches it when it comes later (?page=2&sort=x was not blocked before). # Keep in sync with the * group below. User-agent: Googlebot User-agent: Bingbot Allow: / Disallow: /admin/ Disallow: /secure-admin/ Disallow: /st-admin/ Disallow: /dashboard/ Disallow: /d/ Disallow: /accounts/ Disallow: /u/ Disallow: /api/ Disallow: /static/admin/ Disallow: /inv/ Disallow: /ops/ Disallow: /ideas/ Disallow: /review/ Disallow: /design/ Disallow: /procurement/ Disallow: /employees/ Disallow: /hr/ Disallow: /finance/ Disallow: /tasks/ Disallow: /leads/ Disallow: /suppliers/ Disallow: /scrap/ Disallow: /forecast/ Disallow: /label/ Disallow: /erp/ Disallow: /social/ Disallow: /p/manage/ Disallow: /p/api/ Disallow: /p/add/ Disallow: /p/new/ Disallow: /training/create/ Disallow: /training/*/edit/ Disallow: /training/*/modules/ Disallow: /training/*/lessons/ Disallow: /training/*/learn/ Disallow: /training/my-courses/ Disallow: /training/my-teaching/ Disallow: /manage/ Disallow: /products/manage/ Disallow: /products/ajax/ Disallow: /field/ Disallow: /p/search/ Disallow: /*?q= Disallow: /*&q= Disallow: /*?sort= Disallow: /*&sort= Disallow: /*?filter= Disallow: /*&filter= Disallow: /*?brand= Disallow: /*&brand= Disallow: /*?category= Disallow: /*&category= Disallow: /*?performance= Disallow: /*&performance= Disallow: /*?from=social # Google ignores Crawl-delay; Bing honours it (kept at 1 s for the small server) Crawl-delay: 1 # ─── SEO-tool crawlers — heavily rate-limited (no ranking benefit). # NOTE: bots matching a named group ignore the * group, so the disallow # list is repeated here. 17-Nov-2026: the list is now complete (18 lines of the # * group were missing) and has the "&" form of every search/filter rule. # Pagination (?page=) stays closed for these bots. Keep in sync with the * group below. User-agent: AhrefsBot User-agent: SemrushBot User-agent: DotBot User-agent: MJ12bot User-agent: PetalBot Crawl-delay: 30 Disallow: /admin/ Disallow: /secure-admin/ Disallow: /st-admin/ Disallow: /dashboard/ Disallow: /d/ Disallow: /accounts/ Disallow: /u/ Disallow: /api/ Disallow: /static/admin/ Disallow: /inv/ Disallow: /ops/ Disallow: /ideas/ Disallow: /review/ Disallow: /design/ Disallow: /procurement/ Disallow: /employees/ Disallow: /hr/ Disallow: /finance/ Disallow: /tasks/ Disallow: /leads/ Disallow: /suppliers/ Disallow: /scrap/ Disallow: /forecast/ Disallow: /label/ Disallow: /erp/ Disallow: /social/ Disallow: /p/manage/ Disallow: /p/api/ Disallow: /p/add/ Disallow: /p/new/ Disallow: /training/create/ Disallow: /training/*/edit/ Disallow: /training/*/modules/ Disallow: /training/*/lessons/ Disallow: /training/*/learn/ Disallow: /training/my-courses/ Disallow: /training/my-teaching/ Disallow: /manage/ Disallow: /products/manage/ Disallow: /products/ajax/ Disallow: /field/ Disallow: /p/search/ Disallow: /*?q= Disallow: /*&q= Disallow: /*?sort= Disallow: /*&sort= Disallow: /*?filter= Disallow: /*&filter= Disallow: /*?page= Disallow: /*?brand= Disallow: /*&brand= Disallow: /*?category= Disallow: /*&category= Disallow: /*?performance= Disallow: /*&performance= Disallow: /*?from=social # ─── AI search & assistant crawlers — explicitly welcome (2026-08-18). # These feed ChatGPT Search, Claude, Perplexity, Gemini, Copilot, Meta AI # and Apple Intelligence answers — a growing source of product discovery. # NOTE: named groups ignore the * group, so the disallow list is repeated. # 17-Nov-2026: the list is now complete (18 lines of the * group were missing, so these robots could open # management and filtered-list addresses) and has the "&" form of every search/filter rule. Pagination # (?page=) is now open for them, as for Googlebot and Bingbot, so they can reach every product of a # category or brand. # Keep in sync with the * group below. Also see /llms.txt. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: Google-Extended User-agent: Applebot User-agent: Applebot-Extended User-agent: Amazonbot User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: DuckAssistBot User-agent: YouBot User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /secure-admin/ Disallow: /st-admin/ Disallow: /dashboard/ Disallow: /d/ Disallow: /accounts/ Disallow: /u/ Disallow: /api/ Disallow: /static/admin/ Disallow: /inv/ Disallow: /ops/ Disallow: /ideas/ Disallow: /review/ Disallow: /design/ Disallow: /procurement/ Disallow: /employees/ Disallow: /hr/ Disallow: /finance/ Disallow: /tasks/ Disallow: /leads/ Disallow: /suppliers/ Disallow: /scrap/ Disallow: /forecast/ Disallow: /label/ Disallow: /erp/ Disallow: /social/ Disallow: /p/manage/ Disallow: /p/api/ Disallow: /p/add/ Disallow: /p/new/ Disallow: /training/create/ Disallow: /training/*/edit/ Disallow: /training/*/modules/ Disallow: /training/*/lessons/ Disallow: /training/*/learn/ Disallow: /training/my-courses/ Disallow: /training/my-teaching/ Disallow: /manage/ Disallow: /products/manage/ Disallow: /products/ajax/ Disallow: /field/ Disallow: /p/search/ Disallow: /*?q= Disallow: /*&q= Disallow: /*?sort= Disallow: /*&sort= Disallow: /*?filter= Disallow: /*&filter= Disallow: /*?brand= Disallow: /*&brand= Disallow: /*?category= Disallow: /*&category= Disallow: /*?performance= Disallow: /*&performance= Disallow: /*?from=social User-agent: * Allow: / Allow: /training/ Allow: /training/$ # Block admin & management areas Disallow: /admin/ Disallow: /secure-admin/ Disallow: /st-admin/ Disallow: /dashboard/ Disallow: /d/ Disallow: /accounts/ Disallow: /u/ Disallow: /api/ Disallow: /static/admin/ # Block internal tools and ERP modules Disallow: /inv/ Disallow: /ops/ Disallow: /ideas/ Disallow: /review/ Disallow: /design/ Disallow: /procurement/ Disallow: /employees/ Disallow: /hr/ Disallow: /finance/ Disallow: /tasks/ Disallow: /leads/ Disallow: /suppliers/ Disallow: /scrap/ Disallow: /forecast/ Disallow: /label/ Disallow: /erp/ # Block social hub (private/auth-only content) Disallow: /social/ # Block product management subroutes (but allow product pages) Disallow: /p/manage/ Disallow: /p/api/ Disallow: /p/add/ Disallow: /p/new/ # Block training management subroutes (but allow course pages) Disallow: /training/create/ Disallow: /training/*/edit/ Disallow: /training/*/modules/ Disallow: /training/*/lessons/ Disallow: /training/*/learn/ Disallow: /training/my-courses/ Disallow: /training/my-teaching/ # Block internal management pages Disallow: /manage/ Disallow: /products/manage/ Disallow: /products/ajax/ # Block field marketing (auth-only) Disallow: /field/ # Block search results and faceted navigation (thin content / duplicate) # 17-Nov-2026: every search/filter rule also in its "&" form (the parameter is not the first one in the address) Disallow: /p/search/ Disallow: /*?q= Disallow: /*&q= Disallow: /*?sort= Disallow: /*&sort= Disallow: /*?filter= Disallow: /*&filter= Disallow: /*?page= Disallow: /*?brand= Disallow: /*&brand= Disallow: /*?category= Disallow: /*&category= Disallow: /*?performance= Disallow: /*&performance= Disallow: /*?from=social # Crawl-delay for other bots Crawl-delay: 1 # Sitemaps Sitemap: https://www.west-velocity.com/sitemap.xml Sitemap: https://www.west-velocity.com/sitemap-images.xml