I’d say it’s more of a context thing. If you’re hanging out in a group of people chatting together and you code switch to speak to someone so nobody else can understand, that’s rude. If you’re just speaking to someone in another language on your own, nobody cares (except xenophobic bigots).