It’s really not brigading by definition. Brigading is a conscious and coordinated effort by many people to target a post or community. There is nothing conscious or coordinated about random people randomly seeing a post and downvoting it.
Brigading is not the right term. Unpopular is.
This is likely unavoidable because the majority of users of sites like this are not conservative. They trend toward the younger demographics. Millennials overall in the U.S. are 60% left-leaning. Gen Z are almost 70% left-leaning. What you're noticing may only become more pronounced after kbin re-federates. The fediverse is far more international then Reddit ever was and international and non-English posts are not filtered out by region like on Reddit. Conservative US politics are generally farther right than most right-leaning parties in the developed world. So you may be downvoted by international users as well that actually intentionally view magazines or posts with the conservative tag.
Really the only way to avoid this is to create your own instance and de-federate. This is precisely why the far-right social media platforms (like Truth Social) have had to isolate themselves. They’re simply a significant minority on the internet.