You’d likely know where they grew up by the manner of their speaking. The white kids from the projects are going to sound “urban” and the black affluent people are going to sound stereotypically “white”. I wouldn’t be offended (I’m white) but I would be surprised if the black people I know who do not use that language started using the n-word suddenly. It’s not a part of their normal language, so what caused them to suddenly start using it? Probably nothing good I would assume.
It also depends on the time and place. Those white kids from the projects would probably need to code switch when going to a job interview or something, for example, just like their black peers do. That language is context-dependents on whether it’s acceptable. It’s more than just being black.