I think a big part of the problem is the sign up page, which is being worked on. Also, lemmy.world goes offline just about every day and is one of the larger instances so lots of new users go there and then have a bad experience. And it needs to be made easier for users to find out about all the third party clients available to browse Lemmy because the default client leaves much to be desired.
Most people won’t go out of their way to switch instances when lemmy.world goes offline enough to make them quit the fediverse, or try navigating around the base client and become upset at all the unnecessary button bloat and missing basic features like blocking an instance.