The Waffen-SS had 900,000 people by the end of WWII and it included conscripts. Recruitment standards loosened significantly as the war progressed. After the war, the Nuremberg trials specifically exempted conscripts who had been forced into the SS and who had not committed war crimes from any criminal responsibility.
World War II was complicated, especially on the eastern front. Don't assume the "good guys" and "bad guys" were clearly delimited by a simple badge or insignia.