It seems like a tricky thing to deal with technically due to how the Fediverse works, and I know far too little about ActivityPub to feel comfortable with coming with any ideas. I would however like to point out what I think we'll be seeing more of with this trajectory. Here's someone on kbin getting called out for downvoting and also explicitly named in a comment: https://kbin.social/m/wholesome/t/80838/To-the-one-guy-who-keeps-downvoting-m-wholesome-threads