State Poll Results: Mason-Dixon, Rasmussen Duke it Out for Last Place

So which pollsters were the best and which were the worst? That depends on how you want to analyze things. 

I'm not a statistician or anything, so my analysis can only be very rudimentary. But here it is.

I took only the final state polls from every pollster I had in my database*.  The most recent final poll was from the day before election, the oldest from September 13 (Quinnipiac poll of New York), but most were from October and November.  I used the pollster's spread as the metric by which to measure it against the actual election's percentage spread as of Friday morning.  I calculated a house effect for each poll (predicted minus actual) and an error for each poll (absolute value of predicted minus actual); averages were the sum of each poll's house effect / error divided by the number of final polls that pollster did.

I analyzed every pollster's final poll, but I focus on the eleven pollsters with five or more states in this analysis. 

By the lowest average error metric, Marist won, with PPP second.  Mason-Dixon was by far the worst, with Rasmussen being the second worst.














The most centrist national pollster ended up being Pharos.  Pharos was also the only pollster with a left-leaning house effect on average over all presidential and senate polls.  Everyone else tilted right of actual results, even PPP.  The most partisan pollster on the right?  ARG (4 points).

______________________________________

You could also say a quality pollster is the one which has the most number of guesses within some given number of points. 





















The spread in every single final Marist poll was within 5 points of the actual result in both the presidential race and the senate race, which is a pretty amazing accomplishment.  By contrast, fewer than half of Mason-Dixon's polls were within five, also an amazing accomplishment really.

Quinnipiac had the best senate poll on-the-nose mark (3 of 8 within a point), Gravis (amazingly) had the best presidential on-the-nose mark (3 of 11 within a point), but PPP had the best combined on-the-nose mark (9 of 35).  Mason-Dixon was not within a point in any of its 15 final polls (president or senate), though Rasmussen's 2-for-38 mark deserves some special scorn as well just because they had so many more chances than the rest. 

Pharos was within three points of the final margin 9 out of 14 times.  For comparison, Mason-Dixon was within three points only 2 out of 15 times.

_________________________________________

Another way you can take the measure of the best and the worst pollsters is by the number of battleground races that they call better than anyone else, because really it's the battleground races that most people focus on.  So I looked at the presidential races in the battleground 9 states plus the expanded map 3 states of Minnesota, Michigan and Pennsylvania.  I also looked at the 14 closest senate races plus Nebraska. 

The overwhelming winner by this standard: PPP.

(1 * indicates a pollster was the best or tied for the best national pollster in the given race, but not the best pollster. 2 ** indicate the best or tied for the best pollster. 1 ? indicates a pollster was the worst or tied for the worst national pollster in a given race, but not the worst pollster. 2 ?? indicate the worst or tied for the worst pollster.)

PPP was tops among national firms in 15 races: Colorado, Florida, Michigan, Minnesota, New Hampshire, Nevada, Nevada senate, Virginia, Virginia senate, Wisconsin senate, Arizona senate, Connecticut senate, Montana senate, New Mexico senate and Pennsylvania senate.  They were the worst only in the Ohio senate race (showing a 10 point Sherrod Brown win when it was only 5.2 points).

Conversely, Rasmussen was the worst national firm in 10 races, Colorado, Iowa, Virginia, Wisconsin, Wisconsin senate, Arizona senate, Massachusetts senate, New Mexico senate, North Dakota senate and Pennsylvania senate.  They were the best in only three races (New Hampshire, Pennsylvania and Indiana senate) (interesting that they were both the best and the worst in Pennsylvania: Rasmussen's 5-point Obama win was almost dead on, but the 1-point Casey win was 8 points lower than reality).

____________________________________________
 
So you can pick your own best and worst from this info.  It's pretty clear to me that the best pollsters were PPP and Marist, and the worst were Rasmussen and Mason-Dixon. 

I look forward to seeing Nate Silver's pollster ratings.

*my database excluded a few polls for reasons of methodology that in hindsight I wish I had included, including Mellman polls. 



 

Pollster House Effects

I've talked about house effects against my median poll spread in brief before, but I haven't revealed my calculated list in full until now.  But first, some methodology.

The median poll spread is the difference between the median Obama number and the median Romney number.  These numbers may come from two separate polls; it is not the most central spread of any single poll.  When I first started keeping track of polls in September, I didn't really think much about how I'd choose the polls to put into my database.  I just used whatever polls were in the RealClearPolitics database, though I did not include polls marked with the partisan tag (except for PPP polls).  As I got into it more, I learned that RCP didn't always include PPP polls if they weren't independent; that is, if they were produced for left-leaning organizations.  But aside from maybe the wording of some of the questions that occurred after the presidential survey, these polls were no different than the usual ones PPP included, and I decided it wouldn't be fair to state that PPP's latest guess was a poll produced two weeks ago when really they had just done one yesterday.

And then I discovered all the polls in Huffpost Pollster, and then Pharos Research Group, and then Steve Singiser's daily roundup of polls that I had never heard of or considered before, and so I realized I needed a better methodology to selecting the polls.  Here's what I've got so far:

Include all polls except:

1. Polls produced for political parties or individual campaigns
2. Polls produced for advocacy organizations (except PPP) (so, no Grove, Lake or Mellman polls)
3. Internet polls (no real reason, just that RCP doesn't use them)
4. Class assignment academic polls (with long period of time in the field and/or small number of responses) (the Old Dominion poll and the U. of Iowa poll are the only two to fall under this exception)
5. Polls that only report registered voters instead of likely voters after September (one High Point poll in North Carolina was excluded).
6. Any poll with the word "Newsmax" in it.

I wish I could say there was more science behind the methodology, but it's been more ad hoc and hunch than scientific.  More sophisticated poll averagers use weighting, emphasizing the quality polls and minimizing partisan polls, but since I use a median instead of a mean, each poll has to count the same.  What this method does provide is a quick and easy way to gauge the current margin in any given state using the knowledge base of every pollster's latest guess in that state.
We can't know until after the election which polls are nearest to the final result, but we can know which polls are nearest to the median poll spread and calculate house effects based on this.  I have calculated the median spread for every day since September 1 in 19 state presidential contests and 14 senate contests.  The average distance each poll's spread is from the calculated median spread provides the house effect.

The following are my calculated house effects for each pollster with three or more presidential polls in my limited database.

Pollster	No. of pres. polls	No. of senate polls	House effect (D-R)
Susquehanna	4	4	-7.1
Gravis Marketing	25	13	-3.5
Baydoun/Foster	3	3	-2.8
ARG	15	0	-2.2
Rasmussen	63	46	-2.1
U. of Cincinnati	3	3	-1.8
Muhlenberg Coll.	4	4	-1.6
Mason-Dixon	10	8	-1.3
Civitas	3	0	-1.2
WeAskAmerica	15	9	-1.0
Survey USA	16	10	-0.8
Opinion Research	3	0	+0.0
MassINC	4	4	+0.1
Purple Strategies	5	0	+0.4
Fox News	6	5	+0.9
Suffolk U.	7	6	+1.2
PPP	57	32	+1.2
Marist	22	14	+1.3
Pharos Research Group	16	16	+1.4
Quinnipiac	19	17	+1.7
Marquette U.	5	5	+2.6
Detroit News	3	3	+3.8
Washington Post	4	4	+4.1
Epic-MRA	4	3	+4.4
UNH	4	0	+4.9

So if you read this morning that Rasmussen found the race in Wisconsin tied, you can guess that given Rasmussen's 2.1 point slant towards Republicans, Obama is actually up by 2.1 points there. 

I have found similar house effects to those published elsewhere, like these from Simon Jackman, which gives me confidence that I'm not just finding a bunch of hooey.

_________________

As of yesterday's polls, the median poll spread in the 9 battleground states plus Romney's Extended Map States (MI, PA, MN) are as follows:

State	Obama	Romney	Spread	EV
Minnesota	51	44	O+7	201
Wisconsin	51	46	O+5	211
Pennsylvania	49	45	O+4	231
Ohio	49	45	O+4	249
Michigan	48	45	O+3	265
Iowa	49	46	O+3	271
Nevada	50	47	O+3	277
New Hampshire	49	47	O+2	281
Colorado	47	47.5	R+0.5	257
Virginia	47	47.5	R+0.5	248
Florida	47	48	R+1	235
North Carolina	46	49	R+3	206

•  The North Carolina number has grown more and more Romneyish this week, and a big part of that was a poll from the centrist Survey USA that showed Romney up 5. 
•  Romney's margin in Virginia changed with the Roanoke College poll that was released yesterday. Roanoke College showed an 8 point Obama lead right before the first debate, and now shows a 5 point Romney lead. More impressively, Roanoke College went from a 10-point Tim Kaine lead over George Allen (highly implausible) to a 5-point George Allen lead over Tim Kaine (also highly implausible).  They swung from being a blue outlier to a red outlier in one of the largest poll-to-poll shifts I've seen in my database.
• It almost doesn't matter about any other state besides Ohio at this point, and Obama's Ohio numbers constantly show him in the lead by a few percentage points
• This Michigan number (Obama +3) is why my method of calculating a median spread may not be the best.  It's calculated from seven polls that show a margin of 0, 3, 4, 6, 6, 7, 8, but since the Obama numbers are 47, 48, 48, 48, 50, 52, 53 and the Romney numbers are 42, 42, 45, 45, 45, 46, 47, the spread is 3 instead of 6.