Privacy and anonimity are different things. As long as nobody besides you and the indented destination(s) has access to the content of your communication, that communication maintains privacy, even if everyone sees that it’s you talking.
Also, and this is something I mention all the time, the only information this gives is that you use signal. Besides that, as soon as anybody else registered your phone in their contact list, your phone number is already known and associated with you considering that many apps (like all the meta ones) gain access to the contact list and the chance that anybody who has your phone number uses one of those is almost 100%.