I disagree, a lot of people troll-downvote meaning they downvote content that they don’t like, or just downvote because it’s already low score. Same also happens with upvotes and content that shouldn’t be upvoted. If you’re concerned with other people seeing your vote history you probably shouldn’t be making those votes.
Semi public voting discourages this form of misconduct because people who want to can audit their history and see that they’re a troll.