Yeah no shit if you train your program wrong you’re going to get bad results. But that’s all overfitting is, a training failure. So instead of your bot being able to evaluate your image based on how it fits the tags, it evaluates it based on how it fits the overffitted data. So while it still doesn’t actually remember the overfilled data you still end up converging on it.
Though I’ll give you that you can make the argument that if you train your program on a single image, then it is just remembering that one imagine and essentially copying it. But that’s literally no different to hardcoding a program that produces that one image.