If you don't already: Try inspecting/sorting your beans and blowing out chaff before brewing.
I thought I had my brewing routine down to pretty much perfect, before I saw a Lance Hendrick video where he recommends sorting out faulty beans and getting rid of most of the chaff....