The Fallacy Behind David and Goliath

1. We know the favorite wins more often than the underdog. Here, favorite is defined as the team more likely to win, in almost everyone's opinion. When the favorite wins, no explanation is needed, and the story behind their win is not interesting. There is no "story" there; no narrative impulse. There is not story in which the hare beats the tortoise, because, well, that is too obvious. A story must have some twist to it. The story of David and Goliath where Goliath wins is a story that doesn't get passed on in David's family. The reason David beats Goliath is that that is the raison d'être of the damned story. It's a story that exists for the sole reason of having David to do something improbably heroic.

2. So when the expected does not happen, then there must be some explanation, because the result is puzzling by its very nature. These explanations can be interesting (almost by definition), involving factors that people did not see ahead of time. But they usually have little predictive value. Why not? Well, because the underdog still loses more often than it doesn't, and the factors that can foul up predictions are unforeseeable by their very nature. A predictable upset is not an upset at all. If we factored in the factors that make certain underdogs win more often than not, then they would no longer be underdogs: those factors would just, from now on, be included in future predictions. Put another way, we can't use Gladwell's insights to win bets on football games. We can't say the underdog will always beat the point spread, or that favorite will always cover it. Of course, ex post facto reasoning is always wonderful. That's why, after the event, we can also give lists of reasons why things happened the way they did.

3. Since upsets are more interesting narratively, than expected results, and spectacular insights are even more interesting, they are more memorable. The explanations thus seem more significant. But really, they are not; unless they fall into some significant patterns, that can be made the basis of predictions. It is not particularly predictive to say: sometimes the underdog wins, because, say the chances of a 2% event occurring are still 2%, not zero. We don't remember the 98% of times when things happened like we thought they would.

[After reading some comments on Thomas's blog today I was thinking of Gladwell's book on this subject, in which he goes to great lengths to explain why things don't always occur the way we think they would.]