In that case, downvotes should be invisible. The sorting algorithm can see them but people can’t.
I don’t believe that tiny communities and instances should have them on until a threshold is reached and they become “sortable”.
Being visible is an aggressive moderation tool. It doesn’t foster discussion. reddit devolved into downvote heavy as time went on and I hated it, most of the time it didn’t make sense why things were downvoted. They work better for memes and pics, not comments (unless they are horrible) and discussions. Bad actors use the downvote for bad acting.