Not a good rationale not to ever again visit a dentist. Have you considered changing your dentist?
They can give you a second opinion. Maybe it’s in fact scientific consensus that you should get them removed to omit the caries risk?
It’s important to get everything else checked on a regular basis. Stuff gets nasty real quickly if not detected early on, e.g. pardontitis, caries, etc. Don’t take that risk. Find a dentist who respects your opinion :).