Lemmy instance broken after upgrading to 0.18.0

This is a docker setup, so to update all I did was change the lemmy-ui and lemmy versions in docker-compose.yml. Note: downgrading to 0.17.4 results in an API error instead, and still a broken site, so downgrading does not appear to be an option.

Upgraded my instance to 0.18.0, and now there are errors in both lemmy-ui and lemmy backend.

I see federation messages processing as usual, however trying to load the UI generates a weird error in lemmy-ui, and returns “Server Error” instead of the main page.

The error in the lemmy-ui logs looks like it is trying to load the site icon via pictrs from the public facing domain, but instead trying to connect to 127.0.1.1:443 (for pictrs) and getting refused.

lemmy-ui log

<span style="color:#323232;">FetchError: request to https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp failed, reason: connect ECONNREFUSED 127.0.1.1:443
</span><span style="color:#323232;">    at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
</span><span style="color:#323232;">    at ClientRequest.emit (node:events:511:28)
</span><span style="color:#323232;">    at TLSSocket.socketErrorListener (node:_http_client:495:9)
</span><span style="color:#323232;">    at TLSSocket.emit (node:events:511:28)
</span><span style="color:#323232;">    at emitErrorNT (node:internal/streams/destroy:151:8)
</span><span style="color:#323232;">    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
</span><span style="color:#323232;">    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
</span><span style="color:#323232;">  type: 'system',
</span><span style="color:#323232;">  errno: 'ECONNREFUSED',
</span><span style="color:#323232;">  code: 'ECONNREFUSED'
</span><span style="color:#323232;">}
</span>

lemmy-ui and pictrs are on the same default lemmyinternal network.

lemmy log errors

<span style="color:#323232;">2023-06-23T21:10:03.153597Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities
</span><span style="color:#323232;">   0: lemmy_apub::activities::community::announce::receive
</span><span style="color:#323232;">             at crates/apub/src/activities/community/announce.rs:46
</span><span style="color:#323232;">   1: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">           with http.method=POST http.scheme="http" http.host=hakbox.social http.target=/inbox otel.kind="server" request_id=35c58bff-dc83-40f7-b7f0-d885072958ab http.status_code=400 otel.status_code="OK"
</span><span style="color:#323232;">             at src/root_span_builder.rs:16
</span><span style="color:#323232;">LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "u{1b}[3mhttp.methodu{1b}[0mu{1b}[2m=u{1b}[0mPOST u{1b}[3mhttp.schemeu{1b}[0mu{1b}[2m=u{1b}[0m"http" u{1b}[3mhttp.hostu{1b}[0mu{1b}[2m=u{1b}[0mhakbox.social u{1b}[3mhttp.targetu{1b}[0mu{1b}[2m=u{1b}[0m/inbox u{1b}[3motel.kindu{1b}[0mu{1b}[2m=u{1b}[0m"server" u{1b}[3mrequest_idu{1b}[0mu{1b}[2m=u{1b}[0m35c58bff-dc83-40f7-b7f0-d885072958ab u{1b}[3mhttp.status_codeu{1b}[0mu{1b}[2m=u{1b}[0m400 u{1b}[3motel.status_codeu{1b}[0mu{1b}[2m=u{1b}[0m"OK"", file: "src/root_span_builder.rs", line: 16 }] }
</span>

<span style="color:#323232;">2023-06-23T21:09:14.740187Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Other errors which are not explicitly handled
</span><span style="color:#323232;">   0: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">           with http.method=POST http.scheme="http" http.host=SITE_URL_REDACTED http.target=/inbox otel.kind="server" request_id=83feb464-5402-4d88-b98a-98bc0a76913d http.status_code=400 otel.status_code="OK"
</span><span style="color:#323232;">             at src/root_span_builder.rs:16
</span><span style="color:#323232;">LemmyError { message: None, inner: Other errors which are not explicitly handled
</span><span style="color:#323232;">
</span><span style="color:#323232;">Caused by:
</span><span style="color:#323232;">    Http Signature is expired, checked Date header, checked at Fri, 23 Jun 2023 21:09:14 GMT, expired at Fri, 23 Jun 2023 21:08:14 GMT, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "u{1b}[3mhttp.methodu{1b}[0mu{1b}[2m=u{1b}[0mPOST u{1b}[3mhttp.schemeu{1b}[0mu{1b}[2m=u{1b}[0m"http" u{1b}[3mhttp.hostu{1b}[0mu{1b}[2m=u{1b}[0mSITE_URL_REDACTED u{1b}[3mhttp.targetu{1b}[0mu{1b}[2m=u{1b}[0m/inbox u{1b}[3motel.kindu{1b}[0mu{1b}[2m=u{1b}[0m"server" u{1b}[3mrequest_idu{1b}[0mu{1b}[2m=u{1b}[0m83feb464-5402-4d88-b98a-98bc0a76913d u{1b}[3mhttp.status_codeu{1b}[0mu{1b}[2m=u{1b}[0m400 u{1b}[3motel.status_codeu{1b}[0mu{1b}[2m=u{1b}[0m"OK"", file: "src/root_span_builder.rs", line: 16 }] }
</span>

I’ve also filed a bug, because I’ve been trying to troubleshoot this, but haven’t found a solution yet.

Any help is appreciated.