Pretty sure https is a requirement for federation. Not sure about non-standard port number, but I would try and move to 443. I know you mention in a comment port 443 is already in use, in which case, I recommend setting up a reverse proxy (I like nginx-proxy-manager), and configure a subdomain for the lemmy instance.