I am hosting a two-person XMPP server now, and it’s pretty light on resources. Matrix, however, I am not even sure my VPS would even handle: I’ve seen multiple people talk about how their servers would explode when someone tries to join a large room. And also there’s an issue of every participating server storing chat history/media: my disk is small, I need it for the media on my site!
I am also concerned about how overly prevalent the central matrix.org server is.