State Poll Results: Mason-Dixon, Rasmussen Duke it Out for Last Place

So which pollsters were the best and which were the worst? That depends on how you want to analyze things. 

I'm not a statistician or anything, so my analysis can only be very rudimentary. But here it is.

I took only the final state polls from every pollster I had in my database*.  The most recent final poll was from the day before election, the oldest from September 13 (Quinnipiac poll of New York), but most were from October and November.  I used the pollster's spread as the metric by which to measure it against the actual election's percentage spread as of Friday morning.  I calculated a house effect for each poll (predicted minus actual) and an error for each poll (absolute value of predicted minus actual); averages were the sum of each poll's house effect / error divided by the number of final polls that pollster did.

I analyzed every pollster's final poll, but I focus on the eleven pollsters with five or more states in this analysis. 

By the lowest average error metric, Marist won, with PPP second.  Mason-Dixon was by far the worst, with Rasmussen being the second worst.














The most centrist national pollster ended up being Pharos.  Pharos was also the only pollster with a left-leaning house effect on average over all presidential and senate polls.  Everyone else tilted right of actual results, even PPP.  The most partisan pollster on the right?  ARG (4 points).

______________________________________

You could also say a quality pollster is the one which has the most number of guesses within some given number of points. 





















The spread in every single final Marist poll was within 5 points of the actual result in both the presidential race and the senate race, which is a pretty amazing accomplishment.  By contrast, fewer than half of Mason-Dixon's polls were within five, also an amazing accomplishment really.

Quinnipiac had the best senate poll on-the-nose mark (3 of 8 within a point), Gravis (amazingly) had the best presidential on-the-nose mark (3 of 11 within a point), but PPP had the best combined on-the-nose mark (9 of 35).  Mason-Dixon was not within a point in any of its 15 final polls (president or senate), though Rasmussen's 2-for-38 mark deserves some special scorn as well just because they had so many more chances than the rest. 

Pharos was within three points of the final margin 9 out of 14 times.  For comparison, Mason-Dixon was within three points only 2 out of 15 times.

_________________________________________

Another way you can take the measure of the best and the worst pollsters is by the number of battleground races that they call better than anyone else, because really it's the battleground races that most people focus on.  So I looked at the presidential races in the battleground 9 states plus the expanded map 3 states of Minnesota, Michigan and Pennsylvania.  I also looked at the 14 closest senate races plus Nebraska. 

The overwhelming winner by this standard: PPP.

(1 * indicates a pollster was the best or tied for the best national pollster in the given race, but not the best pollster. 2 ** indicate the best or tied for the best pollster. 1 ? indicates a pollster was the worst or tied for the worst national pollster in a given race, but not the worst pollster. 2 ?? indicate the worst or tied for the worst pollster.)

PPP was tops among national firms in 15 races: Colorado, Florida, Michigan, Minnesota, New Hampshire, Nevada, Nevada senate, Virginia, Virginia senate, Wisconsin senate, Arizona senate, Connecticut senate, Montana senate, New Mexico senate and Pennsylvania senate.  They were the worst only in the Ohio senate race (showing a 10 point Sherrod Brown win when it was only 5.2 points).

Conversely, Rasmussen was the worst national firm in 10 races, Colorado, Iowa, Virginia, Wisconsin, Wisconsin senate, Arizona senate, Massachusetts senate, New Mexico senate, North Dakota senate and Pennsylvania senate.  They were the best in only three races (New Hampshire, Pennsylvania and Indiana senate) (interesting that they were both the best and the worst in Pennsylvania: Rasmussen's 5-point Obama win was almost dead on, but the 1-point Casey win was 8 points lower than reality).

____________________________________________
 
So you can pick your own best and worst from this info.  It's pretty clear to me that the best pollsters were PPP and Marist, and the worst were Rasmussen and Mason-Dixon. 

I look forward to seeing Nate Silver's pollster ratings.

*my database excluded a few polls for reasons of methodology that in hindsight I wish I had included, including Mellman polls.