I agree. Though I think 1v1 on Matrix is fine if encrypted prior, but I mostly use Matrix as tech news/forum. Federated services are not good for privacy anyway, they are not meant to be.
Currently I tell people to install Signal if they want to message me, I guess it would be a couple people for SimpleX. Most of my Signal list came when WhatsApp ToS made it to the news, some of them uninstalled Signal after a while but I’m okay with it. Well, I’m happy with Molly-FOSS for now. I would love to use SimpleX but it’s something.