But the existence of a "geometric center" breaks several of the base assumptions of astrophysics. If there were a geometric center as you say, then there would have to be "edges" of the universe, or the inflation would need a center.
Both of which we think are not the case, due to lots of battle tested theories that tell us space is generally of flat curvature and homogenous, meaning one random chunk of space (the actual stuff of space, not like looking at stars and other structures of the universe) should be indistinguishable from another.
These days those assumptions are considered pretty fundamental to any understanding of space itself and the universe at large that we have.