You should be safe if you are the only one with access to that computer.
If you are sharing the computer with another human being; please Unplug your Yubikey and take it with you when you are not using the computer and it is likely that another human being could be using the computer. Just to be safe; Do Not Leave Your Yubikey Plugged In If Another Person Is Using It…unless you’re authorizing them to access something.
Your Yubikey can’t tell who clicked it’s button as it is NOT a Fingerprint Reader.