Relativism is a cop out argument for or against anything. It's a hail Mary, it's a sign you're scraping the bottom of the rhetorical barrel.
But yes, I'd say that the simple fact of having a billion dollars makes you immoral. All other parts of your life aside. It is immoral simply to have that much money and not be using it to help people.