Keep in mind that RPI is good for containers, but not for virtualisation, so if you want to run some VMs or have proper expandability I would advise you against RPI.
Plus disabling the ports won’t make a big difference I presume, a couple of Watts at best.
I recently bought an N100 powered firewall appliance and the idle consumption is around 13 Watts, with two SSDs and 32Gb of RAM, a bunch of USB and video ports and 4x2.5Gbps ports all switched on. People are even reporting that this can be pushed down to 7-8W, comparable to what you would get with RPI.
The i3-N305 has 8 physical cores and TDP of 15Watts, which can easily be reduced to around 9-10.