jerry,
@jerry@infosec.exchange avatar

I also ran into a really odd iptables problem... when I meshed in the second app server node using wireguard, my egress filter rule was blocking outbound traffic on both hosts. After some fiddling, out of frustration, I flushed the rules out of iptables on both hosts - so no rules. And... iptables was still blocking the outbound traffic. I ended up having to reboot each host - and once I did that, things worked ok. Has anyone seen iptables go into zombie mode before?

khorwood,
@khorwood@infosec.exchange avatar

@jerry I'm glad you were able to figure it out, despite the brain fog. I swear it took months before I could properly use my brain to solve complex puzzles after recovering from COVID.

SpaceLifeForm,

@jerry

Interesting.

Netfilter data corruption?

Are you trying magic with dynamic rule changes?

jerry,
@jerry@infosec.exchange avatar

@SpaceLifeForm I do use dynamic rule changes.

SpaceLifeForm,

@jerry

Is eBPF in the picture?

jerry,
@jerry@infosec.exchange avatar

@SpaceLifeForm no, just a script that adds rules based on certain events

SpaceLifeForm,

@jerry

Those events can occur at supposedly random times and frequently, correct?

That the events are externally driven outside of your control, correct?

Is SMT Disabled?

SpaceLifeForm,

@jerry

Bottom line, it sure looks like a bug in the kernel code, either Netfilter or WireGuard.

I will bet you that it is a Race Condition bug due to having SMT enabled.

Either way, it is important that we find it.

jtk,
@jtk@infosec.exchange avatar

@jerry connection table state full?

frehi,
@frehi@fosstodon.org avatar

@jerry
I once had problems because newer iptables on is basically a compatibility layer using in the background. Flushing all rules with iptables would remove the nftables rules but not the rules. I had to use iptables-legacy to flush the rules.

starchy,
@starchy@infosec.exchange avatar

@jerry did you maybe hit a netfilter conntrack limit? That’s bit me before when ramping up traffic

jerry,
@jerry@infosec.exchange avatar

@starchy that sounds like a possibility. Do you know how you fixed that?

starchy,
@starchy@infosec.exchange avatar

@jerry iirc bumping the limit hilariously high (into the billions)

starchy,
@starchy@infosec.exchange avatar

@jerry Just getting back to this now. Mind you this was on a less recent Debian version, so ymmv, but I had the line

net.netfilter.nf_conntrack_max = 4000000

in /etc/sysctl.d/netfilter.conf. You can also echo the value to /proc/net/netfilter/nf_conntrack_max if you want to set it without a reboot.

If the values of /proc/net/netfilter/nf_conntrack_count and /proc/net/netfilter/nf_conntrack_max are ever equal, you're in trouble for sure.

Also note this will be under /proc/net/ipv4 or /proc/net/ipv6 on some distros.

hadret,
@hadret@fosstodon.org avatar

@jerry Yep, happened to me as well, number of times

jerry,
@jerry@infosec.exchange avatar

and let me tell you - this was fun to solve with covid brain fog...

NorCal_Lynne,
@NorCal_Lynne@mstdn.social avatar

@jerry I haven’t had Covid yet, that I know of, but a lot of people have reported what sounds kinda scary “brain fog “ , hope it goes away and you start feeling better

Fog GIF

redezem,
@redezem@aus.social avatar

@jerry Oh dude, vibe. When I had it last it was hard enough to make instant ramen, can't imagine doing network infra debugging. I feel for you man, get well soon.

gangrif,
@gangrif@social.undrground.org avatar

@jerry your torment is appreciated.. or i assume it is.. i appreciate you whether i’m on your instance or not.

alex,
@alex@yopp.me avatar

@jerry hugs! I hope you won’t get the post covid fatigue. I still have some after almost 2 years since my last covid :(

uzayran,
@uzayran@cyberplace.social avatar

@jerry oof, no idea how you're getting any work done. Going through the same rn. I hope you'll get well soon

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • uselessserver093
  • Food
  • aaaaaaacccccccce
  • test
  • CafeMeta
  • testmag
  • MUD
  • RhythmGameZone
  • RSS
  • dabs
  • KamenRider
  • Ask_kbincafe
  • TheResearchGuardian
  • KbinCafe
  • Socialism
  • oklahoma
  • SuperSentai
  • feritale
  • All magazines