How about a user-specific option to hide votes? Like a NSFW toggle.
That way, people who don’t want to deal with votes don’t have to, while those that like anonymous voting can retain that privacy.
There are plenty of times when people want to express disapproval without getting roped into a discussion (see: bad faith actors and other trolls). Sure, there’s always going to be people downvoting things for any reason under the sun, but I see that as noise in an otherwise often useful graph. If you see more noise, then it might help to not see anything.