If you took a system that uses D4, replaced that with D6 according to my table and didn’t tell your players that the system was actually meant to be played with D4, do you really think they’d notice any imbalance? Would it affect their fun?
No, not at all. You only notice a difference if you calculate it, it won’t suddenly make your character class or weapon imbalanced.