“We” - as in the government or the taxpayer - don’t build investment or luxury homes at all. Private businesses build them. All they need is permission to build them.
And they want to build them because there’s demand for them. If you don’t let them meet that demand, it will be met in a worse way: by rich people buying the homes of poor people and converting those into luxury investment housing. You can see that everywhere - flips, teardowns, gentrification, etc. That’s what happens if you don’t let enough new luxury housing get built: you think the luxury buyers are gonna stop buying?
So to solve the housing crisis, 2 things need to happen:
government needs to invest heavily in affordable housing and public construction infrastructure.
government needs to get the hell out of the way of the private sector that will happily profit from stopping the bleeding.