Oh yes, I know the one.
First off, to expand on the topic of how sex is different from gender, I wrote a brief explanation on this page.
Next, from a medical perspective, when it comes to determining physical "sex", there are several levels:
1) Your sex is defined by your genetic structure. Specifically your sex chromosomes. If you're XY, you're male, if you're XX, you're female. A better way to put this really is "if you have a Y sex chromosome, you're male".
Note, however, that not all people are simply XX or XY. There are various continuums of syndromes and presentations that range from trisomy of sex chromosomes, for example, Klinefelter's syndrome. Note- it affects approximately 1 in every 500 males.
There is also Turner syndrome (one functional X chromosome, second X chromosome is absent/defunct, incidence ~1:2500 live female births), Triple X syndrome (affects ~1:1000 females), (XYY syndrome, which affects ~1:1000 males, although whether this abnormality is in fact a pathology is debatable.
There are other presentations which involve mutations where some cells express one set of chromosomes, and others another (mosaicism).
2) Primary sexual characteristics are generally your gonads and genitals. In general, if you have a penis and testicles, you have male primary sexual characteristics, and if you have mammaries and a vagina, you have female primary sexual characteristics.
However, again, this isn't clear cut- if anything it's less clear cut than the genetics. Specifically one's physical attributes may not match their chromosomal attributes, and when this applies to gonads, this is known as an intersex condition. These include partial and complete hermaphrodites, among others- incidence according to the definition above is approximately 0.018%.
One of the best examples I can think of are those who have Androgen Insensitivity Syndrome. These are males (i.e. XY), but some aspect of their endocrine system is such that their bodies do not respond at all to testosterone. The result? The 'erfect' woman: naturally hairless limbs and bodies (but normal head hair), slim hourglass figure, with graceful, refined features, etc. you get the idea. Except...they don't have ovaries, nor a uterus, so they'll never have a period, and they'll never bear children.
It is known that a number of high-profile Hollywood actresses were in face males with AIS. More recently, AIS has been featured on a number of medical shows, including House. Usually, revelation of their true characteristics is rather distressing as for all intents and purposes they're female but are actually male! Identity crisis much?
3) Secondary sexual characteristics are the peripheral features that tend to accompany either sex. For example if you're male, you'll tend to have squarer hips, a more rugged build, a deeper voice, more body hair, and you'll tend to think in certain ways. If you're female, you'll tend to have rounder hips, less body hair (note, not none- women produce testosterone too!), a higher voice, and you'll tend to think/feel in other ways.
It is in distinguishing between primary and secondary sexual characteristics that most people are ignorant. If I had a dollar for every story I've heard of a guy trying to make it with a girl only to discover she was in fact...a shemale, well, I could probably have a nice pub lunch. It is in the domain of secondary sexual characteristics that most teasing and gendered persecution occurs- the conceptions of one's physical attributes directly conferring "manliness" or "feminimity".
All this should demonstrate clearly one thing: that your sex is not as clear cut as most people are taught to think. Male and female are, on all levels, averages and not a complete reality. This is all equally a part of nature, and really, does it have to matter that much? I don't think so.
