I’m not sure what market you’re in or what your budget is, but the pains that you’re talking about (other than the actual price of the houses) can be mitigated by working with a competent realtor. You want someone who both knows that particular market and also knows how to identify comparable sales. That means knowing the neighborhoods you’re looking at as well as prices and trends.
I’m not a realtor, but I did buy a house fairly recently in an extremely hot market. People here will bid $300k over asking and waive all contingencies. That’s just the nature of the market right now - at least in some areas.
Basically, look at the asking price but use something like redfin or zillow to get their estimation of what the selling price of the house will be. The sites will also (try) to show you comparable houses. The sites tend to err on the high side with estimated values, but if you’re not in a market that’s open to bargain hunters it will be close.
It’s a big step - probably bigger than it should be. I believe that the majority of American families have the majority of their net worth invested in their house.
Closed bidding is how it’s always worked, though. Generally, the people selling the house will pick the top two or three offers and make a counter-offer or otherwise solicit additional offers. When a market is hot, they can additionally push for a quick closing. As a rule of thumb, you’ll probably spend less for houses that have been on the market for a while, but you can likely expect the same kind of issue when it comes time to sell it. If you buy in a desirable market where houses sell within a few weeks, you’ll again probably see that kind of thing when you sell. Bargain hunting in real estate - waiting for the chance to get a smart offer approved - can lead to you being frozen out - or worse, getting taken for a ride that more informed buyers are staying away from. It can be like buying stocks in that way, except with hundreds of thousands to millions on the line with a single investment.
Anyway, find a realtor you can trust - possibly through a referral. Ask around at work or something if none of your friends has someone. It makes the process a lot smoother. This depends on the state, but the seller may pay the realtor’s commission, and in any case should be considered part of the cost of buying a house.