[Solved] My lemmy server just died over night without me doing anything

Woke up in the morning and my selfhosted Lemmy server was basically braindead. I installed it in a Proxmox lxct container using the ansible playbook. It spams my logs with this:

<span style="color:#323232;">target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=8206186d-eaf9-486d-99ad-d9c5def188c8", file: "src/root_span_builder.rs", line: 16 }] }
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:08.001260Z  WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::activities::community::announce::receive
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:46
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    1: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=dee66940-4048-4424-88c8-51cb58851eb0
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=dee66940-4048-4424-88c8-51cb58851eb0", file: "src/root_span_builder.rs", line: 16 }] }
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:08.442900Z  WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::activities::community::announce::receive
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:46
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    1: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=ba977ad6-03ea-46b3-a7d8-c0eb05355358
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=ba977ad6-03ea-46b3-a7d8-c0eb05355358", file: "src/root_span_builder.rs", line: 16 }] }
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:10.626474Z  WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum PageOrNote
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::objects::comment::verify
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/objects/comment.rs:127
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    1: lemmy_apub::fetcher::post_or_comment::verify
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/fetcher/post_or_comment.rs:68
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    2: lemmy_apub::activities::voting::vote::verify
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/voting/vote.rs:57
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    3: lemmy_apub::activities::community::announce::receive
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:144
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    4: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0aeb2c01-23ac-492c-9693-3a9bb171a900
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum PageOrNote, context: SpanTrace [{ target: "lemmy_apub::objects::comment", name: "verify", file: "crates/apub/src/objects/comment.rs", line: 127 }, { target: "lemmy_apub::fetcher::post_or_comment", name: "verify", file: "crates/apub/src/fetcher/post_or_comment.rs", line: 68 }, { target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0aeb2c01-23ac-492c-9693-3a9bb171a900", file: "src/root_span_builder.rs", line: 16 }] }
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:14.796561Z  WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum PageOrNote
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::activities::voting::vote::verify
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/voting/vote.rs:57
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    1: lemmy_apub::activities::community::announce::receive
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:144
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    2: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=552c4fb1-c65c-4220-8919-430243d720cd
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum PageOrNote, context: SpanTrace [{ target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=552c4fb1-c65c-4220-8919-430243d720cd", file: "src/root_span_builder.rs", line: 16 }] }
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:27:00.779726Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:30:09.026861Z  WARN lemmy_server::root_span_builder: Timeout occurred while waiting for a slot to become available
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::objects::person::read_from_id
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/objects/person.rs:66
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    1: lemmy_apub::fetcher::user_or_community::read_from_id
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/fetcher/user_or_community.rs:47
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    2: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0a4c7741-ce73-4108-a323-e7f5cc5b92ae
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_apub::objects::person", name: "read_from_id", file: "crates/apub/src/objects/person.rs", line: 66 }, { target: "lemmy_apub::fetcher::user_or_community", name: "read_from_id", file: "crates/apub/src/fetcher/user_or_community.rs", line: 47 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0a4c7741-ce73-4108-a323-e7f5cc5b92ae", file: "src/root_span_builder.rs", line: 16 }] }
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:35:46.568991Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for hot ranks update: could not translate host name "postgres" to address: Try again
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:42:37.481187Z  WARN activitypub_federation::activity_queue: Queueing activity https://lemmy.ohaa.xyz/activities/announce/df1ccbd5-3971-49e1-96b4-df6e73f39420 to https://pcglinks.com/inbox for retry after connection failure: Request error: error sending request for url (https://pcglinks.com/inbox): operation timed out.  Sleeping for 216000s and trying again
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:42:51.370663Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:46:36.393798Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T05:53:44.661098Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for hot ranks update: could not translate host name "postgres" to address: Try again
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T06:00:21.684837Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T06:10:18.172432Z  WARN lemmy_server::root_span_builder: Timeout occurred while waiting for a slot to become available
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::objects::person::from_json
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/objects/person.rs:134
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    1: lemmy_apub::activities::verify_person_in_community
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/mod.rs:62
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    2: lemmy_apub::activities::voting::vote::verify
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/voting/vote.rs:57
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    3: lemmy_apub::activities::community::announce::receive
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:144
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    4: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=b7159d57-eaae-4992-8d37-0ab87f9f3adb
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_apub::objects::person", name: "from_json", file: "crates/apub/src/objects/person.rs", line: 134 }, { target: "lemmy_apub::activities", name: "verify_person_in_community", file: "crates/apub/src/activities/mod.rs", line: 62 }, { target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=b7159d57-eaae-4992-8d37-0ab87f9f3adb", file: "src/root_span_builder.rs", line: 16 }] }
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T06:16:50.839271Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for active counts update: could not translate host name "postgres" to address: Try again
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 2023-08-22T06:26:51.471879Z  WARN lemmy_server::root_span_builder: Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | Caused by:
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |     Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |    0: lemmy_server::root_span_builder::HTTP request
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=57b8b802-fc7e-4f14-87ac-89de2b0d7770
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | 
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | Caused by:
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 |     Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=57b8b802-fc7e-4f14-87ac-89de2b0d7770", file: "src/root_span_builder.rs", line: 16 }] }
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | thread 'actix-server worker 5' panicked at 'read local site data: LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=ab4b5ccf-f8b9-4acd-9e4b-3c7966a17fd5", file: "src/root_span_builder.rs", line: 16 }] }', crates/apub/src/lib.rs:45:8
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | thread 'actix-server worker 3' panicked at 'read local site data: LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=4e4ba6d1-d716-4d48-93de-e3d7772243d7", file: "src/root_span_builder.rs", line: 16 }] }', crates/apub/src/lib.rs:45:8
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | connection error: db error: FATAL: terminating connection due to administrator command
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | federation enabled, host is lemmy.ohaa.xyz
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | Starting http server at 0.0.0.0:8536
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | connection error: db error: FATAL: terminating connection due to administrator command
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | federation enabled, host is lemmy.ohaa.xyz
</span><span style="color:#323232;">lemmyohaaxyz-lemmy-1 | Starting http server at 0.0.0.0:8536
</span>