Same story here. I self hosted my email with a Linode for a year and a half and it fucking sucked. Gmail almost always filters inbound email from what they deem as untrustworthy IP addresses (which is pretty much any mail server other than the big players), and even if you never plan on mailing Gmail addresses your server’s IP will show up on some “critical spam” blacklist somewhere simply because you’re running a mail server, which basically spells doom for mail deliverability.
You also need to diligently harden your mail server because bots indiscriminately try to find loopholes in mail server configs all the time, and once they do they start spamming their poor victims through your server. It’s a fool’s errand that varies wildly based on what distro you installed, which mail/postfix/dovecot/fail2bam/dkim/spf etc configuration you have, and a lot of the time the information doesn’t actually exist on the Internet so you have to figure it all out yourself.
Ever since I switched to Tutanota + redirection with my domain I’ve never had any of these issues, and I’m never going to look back. It’s unfortunate, but the days of self hosting mail servers are over. It’s simply not worth the struggle and it just becomes an uphill battle in the end.