You could call your pastor a heretic because he doesnt follow your doctrine correctly(jesus loves everyone unconditionally, including sinners) but I doubt it would do anything.
Glod luck though
I know I shouldnt feel guilty about not caring, but the fact is caring and empathy are in my nature