I often use both (I refuse to wait in a five-person line for the women's single-stall locking bathroom when the men's single-stall locking bathroom is empty) and I suspect it has something to do with differing cleanliness expectations. (The men's room always has a filthier floor. The women's room always has a filthier toilet.)
In New Hampshire there is (or was when I lived there, and it probably hasn't changed) a law to the effect that there had to be separate facilities. There was a restaurant in the town where I lived that was located in what used to be a house. The women's room was on the main floor, the men's a floor up.
They had a sign on the door saying something like "The NH Liquor Control Board requires that we have separate facilities for men and women. We think the LCB should keep its noses out of our bathrooms. Use whichever one you want."
That "Live Free or Die" motto didn't come out of nowhere. Ayup.