I hate to sound like such a darn boot licker, but if NYT doesn’t want Archive crawler on thier domain, what reason can you possibly come up with (legitimately, not based on your feelings) as to why they should have to allow this?
Private business gets to do private business things, nothing to see here other than that tbh.