For me, a native speaker, I’ve been confused maybe twice by the use of singular they, and all it took was a quick clarification to remedy. This is because it seems pretty rare in English to use a pronoun without some context surrounding it.
If someone came to me and said, “they went to the store” my confusion would be due to a lack of context. Who are we even talking about in the first place? In that scenario gender neutral he, which is confusing for a myriad of other reasons and can lead to false assumptions, would be just as confusing.
However in an exchange like, “where did this person go? They went to the store” or, “where did Alex and Bob go? They went to the store” the context provides whether they is singular or plural. Revisiting the first example with zero context, “they” would normally be replaced by a proper noun. This sets the required context and makes future uses of “they” make perfect sense.
Whereas with gender neutral he I’d assume you knew the gender of who you were referring to. I grew up using they in the singular form constantly, and it’s not like I was surrounded by queer culture, it’s just a function of how English is spoken in some places even outside of the UK (I’m from California).
I can definitely see it being confusing though if you were taught “proper” formal English. No one I’ve ever encountered speaks that way and it’s largely reserved for academic works. Hell, should’nt’ve and wheredya might as well be in the dictionary by now.