I’m not speaking from any position of authority here, but I think it has more to do with being in the right culture, not the right skin color like some people imply.
There are urban white kids who grew up in the projects who use the same language as their black peers, and I think generally no one cares. However, when someone outside that culture uses that language, it’s not something they use for normal speech and there’s probably some other motivation behind it. It’s just like in a lot of people’s speech they say “bro” but it doesn’t mean brother. To people outside the culture it means black person, but they use it like others use “bro”. It doesn’t really mean anything.
Reiterating, I have very little experience with this. It’s just my observations. I’ve known black people who it’d be weird for them to use this language and seen white people who it’s just a normal part of their speech.