For a server with only a few users, the hard part is outgoing mail, ensuring your mails get delivered.
It is not particularly difficult from a technical point of view.
But if you get blocked by big tech even when doing everything right (reverse DNS, SPF, DMARC, DKIM, RFC compliant MTA), you have to beg them to unblock you. This part is time consuming.
I’ve read horror stories where it went well for years until suddenly Gmail started flagging well-behaved servers as spam without any clear reason. Sometimes mail got through, sometimes it didn’t, without any clear pattern or explanation.
I simply don’t have that kind of time and nerves to deal with this. “hard” may be the wrong word, but it is nerve-wrecking.
Reflections on running my own mail server for 10 years (quitesimple.org)