It depends a lot on gender.

Keen observation my anonymous friend. What you said is deeply and truly funny. My story is slightly different. I never knew how to address people (what "to refer to them as") because I do not know who they are. For the same reason, whatever I address people as, they are always offended because they are like "shut up you do not know me you little piece of *" And as a result, I never speak at all. When I do refer to people as sir or ma'am, I'd be only mocking and parroting what people do in the movie, in the theatre. Once upon a time, I liked it when people address me as "sir". I am a guy but at that time, I was using a female portrait.

"It's funny, how perception can affect someone's response to a simple phrase so drastically." No, it's funny how you think you've said something deep and/or true.

Would you really expect everyone to respond to something in the exact same way? Really? What a dim perception of the world you must have.