lukecooperatus, 7 months ago

FYI, bots and crawlers can simply ignore your robots.txt entirely. This is probably common knowledge around these parts, but I’ve run into clients at work who thought it was a law or something.

I do like the idea of intentionally polluting the data robots will see, as suggested by this comment. There’s no reliable way to block them without also blocking humans, so making the crawled data as useless as possible is a good option.

Just be careful not to also confuse screen readers with that tactic, so that accessibility is maintained for humans. It should be easy enough if you keep your aria attributes filled out appropriately, I imagine.