Surprisingly this isnt actually needed. It makes it easier to setup, but isnt required beyond that (and some bios need one plugged in to boot but that won’t be an issue).
My current setup is 2 nvidias both passed through to separate VMS, then I just use another PC to access the web console. My ideal setup would be a VM that launches itself when none else are running, to allow a graphical output even with the host headless