I’d recommend going with the vanilla Raspberry Pi OS then. Sure, it’s not as lightweight as one would usually hope from a SBC OS, and it has the usual problems that apt has, but it general, it works. It has the firmware stuff ready, so no hassle with that. It has device trees set up in a generally-usable way from the get go, etc.
I didn’t go that route myself and spent couple of days trying to get hardware acceleration to work where I wanted with the VideoCore chip, after which I gave up. VideoCore just isn’t that well supported by the general software stacks, but this was a year or so ago, so it might’ve improved.
Also note that this is all RPi4 specific. Older RPis work quite well.