Someone who has 100,000 times the average (median) wealth of a US resident would have more than 12 billion dollars. There are only a few people in the world with that kind of money. Even the richest person alive doesn't have a million times the average wealth (120 billion).
By and large, if you're talking about 'millionaires', you're talking about people who have 100-200 times the average wealth. Which, not irrelevantly, is comparable to the average wealth of an American to the average wealth of someone from South Sudan.
I'm not playing apologist for the ultra-wealthy. It's pretty clear that, as a class, they're fucking our society. But ownership of 100-200 times the wealth of the average American is no more wrong than ownership of 100-200 times the wealth of the average South Sudanese. What makes wealth exceptionally wrong is the way one acquires and maintains it, not its existence/possession.