There are certain things that are memory intensive and CPU intensive. If you have 10k on one server doing that it really adds up. However having them across a wide range of smaller servers, its not such a big deal.
As a user, you literally lose out on nothing not being on lemmy.world. You can partake in all the same conversations, communities and everything. In fact when lemmy.world is down, you can still see everything and when it comes back up, your posts will synchronize. There’s genuinely no upside to being on lemmy.world. That’s the way the system was designed.