Have you tried installing the respective agents inside the VMs that assists with the guest interfacing?
In VMware: VMware Tools
In Hyper-V: Linux Integration Services (LIS) or hyperv-daemons
In VirtualBox: Guest Additions
These generally help with the mouse input lag and host-guest interfacing that can be sometimes slow in virtual consoles, etc. I don’t know to what degree you consider ‘perfectly smooth’, so you may be talking about something beyond that but I hope it helps.