I am really just using some very cheap off brand controller I found at some store. It does work on my RetroPie, but not my Debian pc. I have now switched over to my steam controller (since it works without any problems) and playing with it feels fine too.
So I guess I the problem has been solved for me, but the mystery of what causes this issue remains.