The more interesting category, to me at least, is black men. When a black man calls me ma'am, especially if he's older, it makes me feel like I'm personally responsible for all of the racial oppression that he's suffered throughout his life.
I became conscious of this today as I was leaving the office, and our security guard said, "have a nice day, ma'am" to me with just a hint of a southern accent... or maybe I was imagining the accent. I'm sure he meant nothing by it besides basic politeness, but I felt like I was forcing him to sit on the back of the bus and drink out of a different water fountain. It was very strange.