It depends on the source. Lemmy uses “pict-rs” as the built in image hosting mechanism, this includes a way to fetch small thumbnails. But other hosts, like catbox.moe or Imgur or anywhere else might not be being fetched efficiently.
Additionally, pict-rs uses imagemagik to transform the image on request. If that transformation doesn’t already exist (cached), or the host is over-loaded or under-spec’d, then you might see a delay in the request response
Your best bet is to use your developer tools to check network usage and response times if you are interested