I suppose I shouldn't be surprised by the lack of empathy some #tech bros have about the whole "#Google is scraping content to train their AI" thing. Lots of people defending Google in tech forums, saying this has been done forever so why are we being such a baby about it?
Maybe because all they can think about is $$ they can get from us content producers & we're standing in the way of that.
I feel like the #Internet is devolving faster than my emotions can keep up. :blobfoxshocked:
@liztai Scraping the internet is how Google functions, it's how Google has functioned for nearly 30 years, that's how they know where everything is. There are robots crawling over everything that's ever been posted.
This isn't the first time this has been a problem. Obviously there are lots of places on the internet that don't want to be scraped by Google.
The solution was something called robots.txt. A file in the root of your website listing what directories and pages you don't want scraped. Thing is it's only an option for a website owner, you can't put a robots.txt on your Instagram for example, maybe we need a new version?
Add comment