Trust me I’m more on your side than you think, what you described should indeed work as you expected. If I selected the Window Managers group during install and it specifically installed things that need Xorg, I’d be surprised if they weren’t working on first boot because of missing Xorg. The Anaconda installer that Fedora uses has generally been very reliable for me, but I haven’t installed Fedora in that specific way, I’ve always chosen a graphical desktop and then installed other window managers or DEs on top of it, or gone with a minimal server install for headless deployments with no GUI at all.
The fact that this was hard to troubleshoot is not a good look for Fedora either, even if this is a somewhat non-standard setup. It’s bad UX that base-x is not documented or easy to find, though on the positive side, I’ve never needed to know that base-x contains all the Xorg packages because Fedora has, for me, seemed to manage this on its own without needing me to know this detail. One more implementation detail I don’t have to deal with is a positive in my book, right up until you have to deal with it, then it’s super frustrating.