Signal notifies users who have your phone number in their contact list; it doesn’t matter if you have them in theirs. It’s an unfortunate side effect of their using phone numbers as identifiers instead of usernames or friend codes or something.
Still a problem, and you basically need a new number as a workaround if you’re trying to conceal that you’re using it. On the upside, they don’t get access to your profile without your permission, so if an ex sees that you signed up they don’t know if it’s you or if you changed your number