I'm nearly 65 years old -- and white -- and it still happens to me.
We seem driven to instantly categorize the gender of people we meet. I guess that once had a sort of evolutionary value, having the question "can I make babies with this one?" at the top of our consciousness. And for women, it also had (sometimes has) safety implications -- unknown males are more likely to be dangerous than other females.
Please, if you are a woman and perceive a person whose gender seems in doubt in a women's bathroom, look again before you shriek. You might simply be seeing a person whose way of presenting herself is different from your expectations.