Pardon My French

Whenever someone utters some profanity – almost always words we have heard before – and follows it by “pardon my French,” my first thought is that what they said is not French. Sometimes I even say that. This is not just something that bugs me, but I wanted to know where it came from. It seems to have evolved from a general animosity between the English and the French. The context for this expression is that anything coarse or vulgar is French. But it makes no sense that Americans say this.