There are a bunch of vulnerabilities for VLC for example. Some of them are based on modified .avi or .mkv files.
Note that those are all known and already patched but there are certainly some vulnerabilities out there that are unknown and/or unpatched. You’re quite unlikely to get one of those though.
The biggest security issue probably is an unpatched system so don’t forget to keep your software up to date.