Question

Explain how during the past Presidential election how so many statisticians got the projections of the elections wrong?

Answer 1

Predicting elections is hard. For various reasons, most importantly their relatively low frequency and high rate of change among predictive variables, predicting the outcome of elections with high certainty is a near impossible task. There is also uncertainty about sampling error, because when you rely mostly on telephone survey sampling (current practice), this introduces a bias when the possession of phones in the population changes over time.

As in any statistical forecast, there are three possibilities:

The models were wrong:

No model is perfect, but it seemed to me at least that the various forecasts, despite their differing methodologies, all captured the essential mechanisms of being elected President: the electoral college; the similar behaviors of some states; the influence of economic and demographic statistics; the relationship between polls and votes. Clearly, something was missed, but these models have been good enough before, and it's not clear why they weren't this time.

The models were right, and this is a fluke:

Even a 95% probability isn't a guaranteed outcome. It's entirely possible that the electorate behaved in exactly the way the models described, just at the most extreme Trump-favouring end of the predicted spectrum. Looking at the "residuals" of the model should give us some clue, like the county-by-country swings from the 2012 election shown above. To me, though, that looks like something systematic was missed in the model.

The data were bad:

The US election process, because of its haphazard nature and inconsistent processes across the country, makes it unlikely that the actual election results were incorrect. That leaves the data going into the models. I see no reason why the economic and demographic data shouldn't be considered solid. One possibility that comes to my mind is that the "feedback effect" -- the influence on voters from the poll results and projections themselves - behaved unexpectedly this year, thanks to the power of personality of the candidates, and the increased influence of the network effects of social media.