I fixed it. It seems to work now. For future reference:
Due to my reverse proxy setup the php container identified it’s own URL as being accessed via HTTP. Settings HTTPS=on in the config did the trick. This will force symfony to assume HTTPS for all communication.
Edit: it seems my comments are not being federated so I could still use some help. Edit 2: it seems all it needed was some patience