There's plenty of reasons why you would not want to have a Jellyfin server be publicly available (even behind authentication). It's simply not a well-secured system at this point (and may not get there for a long time, because it's not a focus).
I strongly suggest keeping it accessed via VPN.
But note that VPN access is not necessarily any slower than "publicly" serving the HTTPs directly, at least not by much.
If you don't already use Wireguard as the protocol, then maybe consider running a wireguard VPN instead, that tends to be quicker than classic OpenVPN.
And last but not least: a major restricting factor in performance of media servers from afar is the upload speed of your ISP connection, which is very often much lower than your download (100Mbit/10Mbit are common here, for example, so only 10% of the speed up than down).