Are transfer speeds tied to internet speed?

When i connect to my jellyfin server to stream/download video/audio the speeds are tied to my internet speed. If my internet speed drops so does the transfer rate from my server. However it seems tied to my internet download speed (which varies from 0.5 to 80 mb/s), not the upload speed(which is usually 2 mb/s), and if i...