badrs,
@badrs@universeodon.com avatar

@liztai Scraping the internet is how Google functions, it's how Google has functioned for nearly 30 years, that's how they know where everything is. There are robots crawling over everything that's ever been posted.

This isn't the first time this has been a problem. Obviously there are lots of places on the internet that don't want to be scraped by Google.

The solution was something called robots.txt. A file in the root of your website listing what directories and pages you don't want scraped. Thing is it's only an option for a website owner, you can't put a robots.txt on your Instagram for example, maybe we need a new version?

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • uselessserver093
  • Food
  • aaaaaaacccccccce
  • test
  • CafeMeta
  • testmag
  • MUD
  • RhythmGameZone
  • RSS
  • dabs
  • KamenRider
  • Ask_kbincafe
  • TheResearchGuardian
  • KbinCafe
  • Socialism
  • oklahoma
  • SuperSentai
  • feritale
  • All magazines