• ɐɥO
    link
    fedilink
    83 months ago

    Nope. Search engines should follow the robots.txt

    • @[email protected]
      link
      fedilink
      -23 months ago

      You misunderstand. Sometimes you want your public website to be indexed by search engines but not scraped for the next LLM model. If you disallow scraping alltogether, then you won’t be indexed on the internet. That can be a problem.

      • ɐɥO
        link
        fedilink
        73 months ago

        I know that. Thats why I dont ban everyone but only those who dont follow the rules inside my robots.txt. All “sane” search engine crawlers should follow those so its no problem