You dont even need google to access your emails for that. You dont even need to be a google user at all, unfortunately.
I think the phone number is easily found by google, by all their users synching their contact list… If you’re google and you have 100 people Synching John B. Smith with number 123 in Region A of the world, you’re pretty confident that that the person and the phone number are linked.