The downvote button is not supposed to be used as an “I disagree” (although that is how many people use it). Nor are you supposed to downvote something just because you personally don’t like it or the person that wrote it.
Downvotes are for posts that are irrelevant to the community/discussion at hand, trolling, arguements that are made in bad faith and that sort of thing.
If you really want a commenter to know that you disagree with them or that you think they’re stupid, you should reply to the comment with a well thought out counter argument of your own, not just “downvote and hope the comment gets hidden”.