Recent Outages

Hey everyone, so as I’m sure everyone is aware Lemmy.World has been experiencing several outages throughout the last few days.

We have been investigating the root cause of these outages but believe that they are related to our current hosting provider (Hetzner) blocking access from ClouldFlare as (we think) they believe that our CDN is a DDoS’er, and is causing these disconnects to our backend server, problematic for sure.

We’ve opened support tickets with our current provider and are awaiting a response. We have no issue with being as transparent as possible with downtime. Anyone that is curious, can feel free to check out status.lemmy.world and dash.lemmy.world for up to the minute outage information. We are also looking into other fediverse friendly methods of posting status and outage updates

In the meantime, we are evaluating alternative hosting options and solutions to provide a high level of reliability to you, our users. Really, we want to say thanks to everyone for soldiering through all our technical growing pains.

Cheers

  • LW Infra Team
badcommandorfilename,

I just wanted to say that Lemmy.world has been much more stable for me lately, thanks!

I know a lot of people are putting in hard work to get here, so I hope you know your effort is being noticed.

rafa,

Hey, hire people that understands how the fediverse works. You’re welcome

Zuberi1,

deleted_by_moderator

    •
    Zuberi1,

    Why are still people on lemmy.world is beyond me

    Go to an instance that isn’t flooded w/ propaganda

    olafurp,

    I brows all, never noticed as outage lol.

    drewthejedi,

    Everyone talking about the downtime including lemme and me just enjoying lemmy like never before. I’ve experienced no downtime so far

    malloc,

    The beauty of a federated app is that I do not have to rely on lw or lemme. Use another instance and go on with my day 😃

    webhead, (edited )
    @webhead@lemmy.world avatar

    Could look into Dacentec if you need more cheap servers. I use them for my stuff. YMMV since you’re getting a hell of a lot more traffic than I am but they haven’t blocked Cloudflare on mine yet so that’s a plus. :)

    psion1369,

    Try Liquid Web for the hosting.

    TurnItOff_OnAgain,

    The dashboard requires a login to view. At least from here it does. Is there a way to view without making an account?

    KazuyaDarklight,
    @KazuyaDarklight@lemmy.world avatar

    I was able to access it without a login just now.

    TurnItOff_OnAgain,

    Seems to work now. Maybe it was a setting on their side?

    Shadywack,
    @Shadywack@lemmy.world avatar

    Only thing that needs to be said here is: thank you for the best Lemmy instance (in my opinion) out there!

    Boozilla,
    @Boozilla@lemmy.world avatar

    Whenever I get frustrated by the outages I remind myself: still better than reddit.

    infyrin,
    @infyrin@lemmy.world avatar

    Yeah because Reddit blames YOU and not itself.

    NocturnalMorning,

    It’s your fault for not pitching in and making lemmy more stable. There, happy, you’ve been blamed :P

    grue,

    It’s your fault for not pitching in and making lemmy more stable.

    Plot twist: unlike Reddit, with Lemmy that would be a legitimate option! Ain’t Free Software and federated networks grand?

    shastaxc,

    What is the burden if I wanted to host a node and limit access just to myself? Is it just a portal into the rest of the fediverse or is there a large maintenance burden or storage requirement?

    maor,

    Yeah, pretty big storage requirement due to the way pictrs works. Pictrs is the piece of software Lemmy relies upon to manage image storage, uploads, and most importantly: caches pictures from other instances. This takes up a HUGE amount of storage space, and there’s no official way to clear this up, see these posts I recently made: first one, second one. The solution I resorted to is renting a 1TB storage box from Hetzner for 3 euros per month, pretty sweet deal but I was kinda annoying by it. So the cheapest deal I could find costs me 6 euros per month: 3 for an Alma Linux ARM VPS from Hetzner, and 3 for that storage box. If you’re in for the fun in tinkering (I sure as hell am in), then get ready for a good time. Other than that, if your main line of reasoning is to take burden off of lemmy.world, then I think just go ahead and join another instance. Better yet: join croud funding of another instnace:)

    Draconic_NEO,
    @Draconic_NEO@lemmy.world avatar

    I don’t think this is correct, the images from one instance are hosted on that instance, they don’t get copied over. What’s federated and copied over is text and references to those images (like this ![](https://image.url/image.filename)). The only images that are hosted on your own Lemmy instance are the ones that you and your users upload, which sure if you have open sign ups and a bunch of people using your instance uploading images then it will become a problem very quickly, but for personal use it shouldn’t unless you’re uploading a lot of images or even videos.

    Just use external image hosts for posts and comments, and only use image uploads for your own profile and banner and everything should be fine (if you want extra assurance you can disable uploads in case you forget).

    maor,

    Nah I thought the same but then I manually checked it. In most of the image posts I see, the image URL starts with lemmy.org.il, which made me wonder whether they’re actually downloaded or it’s some kind of whacky proxy. So I downloaded some of these pics and looked for files of identical size and hash digest, and indeed they were on my disk!

    It’s not a bad decision to cache pics, because it does make the experience really smooth, and I’m not complaining about it. Mastodon does this as well

    Draconic_NEO,
    @Draconic_NEO@lemmy.world avatar

    Weird because I’ve checked it before and it doesn’t seem to, one notable example is that on the lemmy.zip communities I can’t see any of the images at all because something on my network or with my ISP blocks zip domains. The posts, comments and all their text gets copied over to lemmy.world but the images don’t seem to and thus they don’t display for me due to their domains being blocked.

    (Alternate DNS doesn’t seem to bypass whatever they’re doing to block it, I can only connect to .zip domains through Tor or VPN).

    It’s not a bad decision to cache pics, because it does make the experience really smooth, and I’m not complaining about it. Mastodon does this as well

    I agree it’s definitely a good idea to create a more smooth user experience, would also eliminate the problems that I pointed out with the images not loading due to their domain being blocked.

    Swim,

    Yup

    Shady_Shiroe,
    @Shady_Shiroe@lemmy.world avatar

    Thanks to lemmy I don’t doom scroll anymore because either there is an outage or I read all the new content I’m subscribed to in 10-20 minutes

    kratoz29,

    Huh, I could keep up with my daily feeds before, now not so much.

    GONADS125,

    That’s mostly how I redditted for years. It was mostly for those moments in between things, on the toilet, laying in bed at night. Not something I did for long periods of time.

    chulo_sinhatche,

    Look at this guy with his reasonable social media use. YOU THINK YOU’RE BETTER THAN ME??

    ethman42,

    Scrolling a few miles or so a day isn’t too bad, is it?

    malloc,

    I fallback to one of the many instances that federate with lw. Don’t get frustrated at all bro

    thantik,

    I mean, I’d help if I didn’t have to have the educational background of a rich upper class white man. Teach me what I need to know and I’ll sysop. But it looked like you had lots of people already come forward with experience “serving at the edge”, etc. All I’ve done is play with Kubernetes and Proxmox at home on a residential IP with a domain pointed at my residence. Unfortunately, with no college education, I’ve had to teach myself all I know and it probably isn’t up to what’s wanted. I’ve heard “A bad sysop can be worse than just having nobody”, so :(

    remotelove,

    I quit highschool and was force-fed some kind of formal education in the Navy. I think I have an associates degree of some kind? I dunno and I personally don’t care if I do.

    However, I built my career doing what I love and that was doing anything in IT… I settled on IT Security and just kept getting real world experience. Proving your skills and abilities can be a challenge at first, but its no different than anyone else fresh out of college.

    Don’t sell yourself short because you didn’t go to college. I likely saved thousands of dollars and years of my life by actually learning and applying useful skills. Many people in this world have done the same.

    College is not for everyone. As a matter of fact, I tend to stear away from candidates during interviews that have just memorized everything and don’t know how to apply it. Case in point, I spent a week trying to teach someone how to troubleshoot a problem only to find out he had no idea how TCP and UDP actually worked. (I am a hands off teacher. I explain a problem and the concepts needed to fix it, but it is up to that person to do the leg work.) Sure, he could quote what he was told to memorize at school, but for many tasks he is basically useless. Ugh.

    If someone reading this has gone to college and enjoyed it, awesome! Good on ya! You learn differently than I do.

    Geert,
    @Geert@lemmy.world avatar

    You can also help by posting content and reporting people that break LW rules. You don’t need to be a tech wizard to help out.

    thantik,

    Being a sysop and reporting mean comments are two WILDLY different things.

    Geert,
    @Geert@lemmy.world avatar

    I’m just saying there are plenty of things you can do if you’re not technical.

    RanchOnPancakes,
    @RanchOnPancakes@lemmy.world avatar

    Maybe its just the times I’m accessing but its seem better this week to me compared to the last few ones.

    gsa32,

    lol

    perfectra1n,

    On your Cloudflare account, if there was a change in the CNAME/A record being proxied vs. DNS only, that could cause an issue, as Cloudflare would then strip headers off the request that your Apache/Nginx would be looking for.

    If you enabled HTTP DDoS protection in your Security -> WAF tab (I think that’s where it is) that could do this too. Might be worth disabling.

    Also check for any headers your HTTP load balancer might be expecting, that Cloudflare could be stripping.

    Might be worth tailing the webserver logs to see what happens to requests coming in from Cloudflare.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • random
  • uselessserver093
  • Food
  • aaaaaaacccccccce
  • test
  • [email protected]
  • CafeMeta
  • testmag
  • MUD
  • RhythmGameZone
  • RSS
  • dabs
  • KbinCafe
  • TheResearchGuardian
  • Socialism
  • feritale
  • oklahoma
  • SuperSentai
  • KamenRider
  • All magazines
    •