This is community-evaluated content, and downvotes are a tool used for evaluation. So I think they make sense.
That being said, I don't believe they should be public by default. People are nuts these days, especially online, and I don't want to catch an online stalker or some nazi sliding aggro into my DMs because I downvoted their post.