I had the exact same issues when I ran matrix with my friends and family. I see this as a major bug with the encryption.
AKA it’s not just you. I’ll probably encounter it again soon when I setup the server again soon and open a bug report.
I don’t care if I setup an encryption key backup at sign up. If my session should still be valid I’m not entering that long thing, I’m just quitting the use of the app. The encryption had to work and not just randomly bug out.