You’re a housing provider, not a landlord. If you aren’t making anything off of the houses you lease then you aren’t the subject of the ire of renters.
Ignore those goons saying you’re a bad investor. It’s noble of you to not leech off of the people who you rent to, and at the end of the day, the equity of the house is still yours.