I considered getting a Mini a couple years ago. I tried one. It was not small enough for the reasons you describe. I have a regular sized iPhone for that reason.
I’ve long said the mini flopped not because people don’t want small phones, but rather because it wasn’t small enough to be worth the tradeoff of a smaller screen.