The non-religious are used to having our lack of religion dismissed, ridiculed, rebutted. Many of us grew up in religious communities, and are accustomed to displays of religiosity. We rarely take offense at the expression of religious opinions, except to the extent that they are offensive for other reasons. Only the attempt to make religious ritual a part of state functions is directly odious. I am personally offended by anyone saying that "religious murders are wrong, but...". I fly into a rage when I hear that. Of course, nobody notices this because I am only at home in front of my computer. Of course, my rage does not mean that I have a right to commit physical violence against people expressing that opinion. By their own logic, though, they should expect me, and others like me, to be violent. After all, they have offended me! Why wouldn't they expect a punch in the nose? The Pope made the analogy just yesterday, that if someone cursed his mother in the Argentine fashion, he would punch them.
The difference is that that non-religion, for the non-religious, is not a sacred icon. You insult my lack of religion, you don't insult me at all, because that is not a part of me. You can burn effigies of Bertrand Russell or Darwin all day long, I don't care. You might as well insult my love of cilantro or my lack of affection for cats.
Nobody cares about offending a secular humanist, and perhaps they are right not to care, because we are slow to offend and slow to violence.
What I have learned from my own post, then, is that the claim to offense is a powerful tool that non-religious people don't use a whole lot. By claiming this power you automatically gain a mother who can be cursed at obscenely, and hence the right to punch people in the face. The non-religious are orphans. You can say "chinga tu madre" all day and our answer is that we have no mother to be fucked.