# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / # Training crawlers User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: omgili Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: MistralAI-Training Disallow: / User-agent: meta-externalagent Disallow: / User-agent: GoogleOther Disallow: / User-agent: Baiduspider Disallow: / User-agent: PetalBot Disallow: / User-agent: AwarioSmartBot User-agent: AwarioRssBot Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: QualifiedBot Disallow: / User-agent: Cotoyogi Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: atlassian-bot Disallow: / User-agent: FishBot Disallow: / User-agent: BorderxBot Disallow: / User-agent: NavuBot Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: WARDBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: KimiBot Disallow: / User-agent: CitibotSiteCrawler Disallow: / # END Cloudflare Managed Content # =========================================================================== # robots.txt for https://ramabijapurkar.com/ # CMS: October CMS (Laravel-based) # =========================================================================== # SECTION 1: PRIMARY SEARCH ENGINE BOTS # Full crawl access to all public content User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-Video User-agent: Googlebot-News User-agent: Storebot-Google User-agent: Bingbot User-agent: Slurp # Yahoo Search User-agent: DuckDuckBot User-agent: Baiduspider # Baidu (China) User-agent: Yandexbot # Yandex (Russia/Global) User-agent: Exabot # Exalead User-agent: facebot # Facebook crawler User-agent: facebookexternalhit User-agent: Twitterbot # Twitter/X link previews User-agent: LinkedInBot # LinkedIn previews User-agent: WhatsApp # WhatsApp link previews User-agent: Applebot # Apple Search / Siri Allow: / # --- October CMS: Block backend & system directories --- Disallow: /backend/ # October CMS admin panel Disallow: /backend/backend/ # Fallback nested backend route Disallow: /october/ # Legacy October CMS system folder # --- October CMS: Block Laravel/framework internals --- Disallow: /storage/ # App storage (logs, cache, sessions) Disallow: /vendor/ # Composer vendor packages Disallow: /bootstrap/ # Laravel bootstrap files Disallow: /config/ # App configuration files Disallow: /modules/ # October CMS core modules Disallow: /plugins/ # Plugin source files (not public assets) # --- Standard private/sensitive routes --- Disallow: /admin/ Disallow: /login/ Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /staging/ Disallow: /internal/ Disallow: /dev/ Disallow: /test/ Disallow: /tmp/ Disallow: /private/ Disallow: /api/private/ Disallow: /.env # Environment config (must never be exposed) Disallow: /artisan # Laravel CLI entry point # --- Block duplicate/noisy URL parameters --- Disallow: /*?s= # Internal search result pages Disallow: /*?session_id= # Session-specific URLs Disallow: /*?ref= # Referral parameter duplicates Disallow: /*?_token= # Laravel CSRF token URLs # --- Explicitly allow public assets (October CMS structure) --- Allow: /themes/ # Public theme assets (CSS, JS, images) Allow: /storage/app/uploads/ # Public user-uploaded media Allow: /*.css$ Allow: /*.js$ Allow: /*.jpg$ Allow: /*.jpeg$ Allow: /*.png$ Allow: /*.gif$ Allow: /*.svg$ Allow: /*.webp$ Allow: /*.woff$ Allow: /*.woff2$ Allow: /*.ico$ # =========================================================================== # SECTION 2: AI CRAWLERS & LLM BOTS (AEO & GEO Optimisation) # Explicitly permit AI crawlers for indexing, training, and answer generation # Improves visibility in AI-powered answers (ChatGPT, Gemini, Perplexity, etc.) # =========================================================================== User-agent: GPTBot # OpenAI / ChatGPT User-agent: ChatGPT-User # ChatGPT browsing plugin User-agent: OAI-SearchBot # OpenAI SearchBot User-agent: Google-Extended # Google Gemini / Bard training User-agent: Gemini-AI # Google Gemini (alternative UA) User-agent: ClaudeBot # Anthropic Claude User-agent: Claude-Web # Anthropic Claude browsing User-agent: anthropic-ai # Anthropic general crawler User-agent: PerplexityBot # Perplexity AI User-agent: YouBot # You.com AI search User-agent: cohere-ai # Cohere AI training User-agent: Amazonbot # Amazon Alexa / AWS AI User-agent: Meta-ExternalAgent # Meta AI (Llama) User-agent: Meta-ExternalFetcher # Meta AI content fetcher User-agent: Diffbot # Diffbot knowledge graph User-agent: Bytespider # ByteDance / TikTok AI User-agent: Applebot-Extended # Apple AI / Siri extended User-agent: PetalBot # Huawei AI search Allow: / # --- October CMS: Block backend from AI crawlers --- Disallow: /backend/ Disallow: /backend/backend/ Disallow: /october/ Disallow: /storage/ Disallow: /vendor/ Disallow: /bootstrap/ Disallow: /config/ Disallow: /modules/ Disallow: /plugins/ # --- Standard private routes --- Disallow: /admin/ Disallow: /login/ Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /staging/ Disallow: /internal/ Disallow: /private/ Disallow: /.env Disallow: /artisan Disallow: /*?_token= # =========================================================================== # SECTION 3: REGIONAL SEARCH BOTS # Explicitly allow regional crawlers for geo-targeted indexing User-agent: SeznamBot # Seznam User-agent: Qwantify # Qwant (Privacy-focused) User-agent: ia_archiver # Internet Archive / Wayback Machine Allow: / Disallow: /backend/ Disallow: /backend/backend/ Disallow: /october/ Disallow: /storage/ Disallow: /vendor/ Disallow: /bootstrap/ Disallow: /config/ Disallow: /modules/ Disallow: /plugins/ Disallow: /admin/ Disallow: /login/ Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /staging/ Disallow: /internal/ Disallow: /private/ Disallow: /.env Disallow: /artisan # =========================================================================== # SECTION 4: KNOWN MALICIOUS & UNWANTED BOTS # Block scrapers, content thieves, and vulnerability scanners # Note: Pair this with WAF/firewall rules — bad bots often ignore robots.txt # =========================================================================== User-agent: DotBot User-agent: BLEXBot User-agent: Turnitin # Plagiarism scraper User-agent: SiteAuditBot User-agent: DataForSeoBot User-agent: Scrapy # Generic scraping framework User-agent: python-requests # Raw Python scraper User-agent: curl # cURL-based scrapers User-agent: wget # wget-based scrapers User-agent: libwww-perl # Perl scraper User-agent: HTTrack # Website copier User-agent: WebCopier # Website copier User-agent: WebBandit # Aggressive crawler User-agent: EmailCollector # Email harvester User-agent: EmailSiphon # Email harvester User-agent: Zeus # DDoS / scraper bot User-agent: Zmeu # Vulnerability scanner (targets Laravel/.env) User-agent: masscan # Mass port/content scanner User-agent: nikto # Security vulnerability scanner User-agent: sqlmap # SQL injection scanner User-agent: WPScan # CMS vulnerability scanner (also targets OctoberCMS) User-agent: SiteExplorer # Aggressive indexer User-agent: MegaIndex # Aggressive link crawler Disallow: / # =========================================================================== # SECTION 5: GENERIC / UNKNOWN BOTS WITH CRAWL DELAY # Allow public pages but throttle to protect server performance # October CMS on Laravel can be resource-intensive under heavy crawl load User-agent: * Allow: / # --- October CMS system directories --- Disallow: /backend/ Disallow: /backend/backend/ Disallow: /october/ Disallow: /storage/ Disallow: /vendor/ Disallow: /bootstrap/ Disallow: /config/ Disallow: /modules/ Disallow: /plugins/ # --- Standard private routes --- Disallow: /admin/ Disallow: /login/ Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /staging/ Disallow: /internal/ Disallow: /dev/ Disallow: /test/ Disallow: /tmp/ Disallow: /private/ Disallow: /api/private/ Disallow: /.env Disallow: /artisan # --- Noisy URL parameters --- Disallow: /*?s= Disallow: /*?session_id= Disallow: /*?ref= Disallow: /*?_token= Crawl-delay: 10 # =========================================================================== # SECTION 6: SITEMAP REFERENCE # =========================================================================== Sitemap: https://ramabijapurkar.com/sitemap.xml