No, sex is a biological characteristic and the doctor is describing your phenotypic sex based on observable characteristics. This really isn't that complicated. There are two* combinations of chromosomes that determine sex, so there are two sexes. This is basic biology and has absolutely fuckall to do with gender as a social construct.
Really there are around a half dozen sex chromosome combinations because they occasionally get duplicated. Functionally there are two because all of the combinations except 1 have a y chromosome and are male