The neo-babylonians sacked Jerusalem, among many other cities and temples. Temples are where much of the wealth and power was kept, sacking the first temple had little to do with the potency of their specific religion. At that time the religion was just the normal Canaanite pantheon.
Judaism as we think of it, with the covenant between the special people and a single all powerful god - that only begins after the first temple is destroyed and Judah is largely depopulated, around the 500-200BCE time period.
What happened to the 2nd (actually 3rd) temple
The Romans destroyed it 70CE. ~600 years between major sacks of your city shows it's not that important.
Don’t forget, Crusades anyone?
Yes, during the medieval period Jerusalem finally starts to become an important goal of religious conflict - 2-2.5 thousand years later than the building of the temple of Solomon