This isn’t true at all. IQ isn’t some magical catch-all measure of a person’s intellectual ability, but it’s not entirely total quackery either.
I suspect that academic success would be very strongly correlated with having a supportive home life, but IQ not so much. Maybe the gifted kids you refer to were the academically successful ones and not the high-IQ ones?