The thing is that people usually won’t take the time to write that reply and in case of trolls, they shouldn’t be wasting their time.
YouTube’s dislikes used to be great for telling useful content apart from useless spam at a glance without even clicking on it. They removed them, so now we have to read the comments before watching a video if we want to avoid wasting time.
In this day and age, information flows so fast there isn’t always time for nuance. The downvote isn’t perfect and misused often, but it does serve a purpose.