You often hear this argument: the speech of the weaker in society should be more protected than the speech of the strong. So if you are powerful, privileged, etc.. your speech should be less protected.
There are a few things to consider here:
1) This means that someone, in order to be heard, must claim victim status. (The merits of the arguments aren't meaningful anymore.) So you will find that even people belonging, ostensibly, to powerful, privileged, or influential groups, will start to claim that they are the true victims, once they figure out that that is the way to be heard.
2) In practical terms, the weak or underprivileged need freedom more, because they don't have power in other ways. Hence they have a vested interest in keeping freedom for everyone.
3) It will be the liberal professor, the one who often sympathizes with the less privileged, who will get in trouble with the censors first, before the hardcore racists. I'm sure people who have made these kind of "free speech ... but..." arguments have themselves found themselves censored, more than some conservative economics prof who just keeps his head down.