You say that but non-hispanic white people make up “only” 60% of the population, and many of those people think that conservativism is insane and also want to support the remaining 40%. Almost any LGBT+ white people will be on that side as well.
That also begs the question that with nearly half the country being visible minorities and huge chunks of the majority being other forms of minority what do you mean by “those groups that should be representing your core base”?
I’ll also agree that a party’s core base should be broader than “just enough to win” and there’s no reason that salt-of-the-earth labourers and such can’t be taken care of too, especially since there’s a fair amount of overlap.