Thursday, July 07, 2011

PolitiFact Truth Index, Part 5: That Other PolitiFact Truth Index

This week PolitiFact announced a new feature on their mobile phone app. They developed a chart of general truthiness against time so that trends could be seen more easily. In order to do this, they had to create some way of quantifying their own Truth-o-meter® ratings, so they assigned values to their ratings with positive numbers representing truths and negative numbers representing lies. Then they took the average of the last seven days and charted the results.

The feature is almost exactly like my PolitiFact Truth Index.

What did PolitiFact call this new feature? The PolitiFact Truth Index.

Coincidence? Who knows.

So now when you search for "politifact truth index" in Google, PolitiFact's site comes up first rather than mine. On the other hand, now that the PolitiFact Truth Index is actually a thing, maybe auxiliary search traffic to this site will increase.

Not that any of that actually matters.

There are differences between our methods, PolitiFact and me, but most of them are arcane, with one big exception: PolitiFact's PolitiFact Truth Index doesn't break it down by political party. This is probably in an attempt to avoid any overt semblance of partisanship, because otherwise PolitiFact would have to defend the reason why Democrats have a higher Truth Index value than Republicans, and then they get into the whole debate about selection bias versus actual lying, and then they lose their non-partisan aura which is essential to their reputation as a source for the mainstream media.

I obviously don't have any of those qualms. I just charts 'em as I sees 'em, and I see Republicans lying (although recently the numbers have been evening out, as shown below).




Other differences between the Quibbling Potatoes PolitiFact Truth Index and the PolitiFact PolitiFact Truth Index have more to do with the calculation of points. My points for the six categories of statements in descending truth order are as follows: 1, 1, 0, 0, -1, -1. PolitiFact's are essentially as follows: 1, 0.5, 0, -0.5, -1, -1.5. They also multiply the result by 100 because it just looks cooler that way. And their average is calculated by date that they analyzed the statement, not by the date that the statement was made. So even though PolitiFact claims to be tracking the "ups and downs of political discourse" with its Truth Index, they are really tracking the ups and downs of PolitiFact's choice of statements for the week. There are differences between those two methods as you can see by comparing the chart below to the one above.



PolitiFact's method is so similar to mine that it was really easy to calculate and make a chart based on one of the other methods of quantification that I tried which was based on a 5-4-3-2-1-0 point system. So the chart below represents what PolitiFact's PolitiFact Truth Index would be if they broke their numbers down by political party.



For the week ending July 1, 2011, Republicans had an Index value of -30.43 and Democrats had an Index value of -25.00.

PolitiFact's method makes everyone look a little worse, which is to be expected when three out of the six categories essentially are downgraded by half a point so that Pants-on-fire ratings are 50% worse on the bad side than True ratings are on the good side. But it still shows that, but for the week ending on May 20, 2011, Democrats have scored better than Republicans in every week I've looked at so far.

9 comments:

Bryan said...

Hi, Steve. Still hoping you'll get around to continuing that earlier discussion. But as for the here & now:

Given that the tendency of Republicans to lie accounts for what appears to be your relatively consistent results that have Republicans doing more lying in the PolitiFact data, what do you think is the best explanation for the recent poor performance of Democrats? Did Wonder Woman loosen the Lariat of Truth from around their collective middles? ;-)

Steve said...

Hello again, Bryan. I wish I could say that the recent relatively poor performance in truth-telling from Democrats could be tied to some concrete outrageously false media buy or policy statement, something like if the DNC made a series of ads claiming that Republicans were going to start beheading the elderly in order to save on Medicare or something.

But I think really the answer lies probably in the selection bias of PolitiFact. But who knows? Maybe it's the start of an artificial neutralization policy, whereby PolitiFact will now select an even percentage of true and false statements from both parties. Will we all be better served by a selection policy that's even less random than before?

____

As to the earlier discussion, I'm done with it. You refuse to acknowledge that your determination of what is and what is not a PolitiFact error is necessarily dependent on your subjective perception, which is incidentally one of the biggest fallacies of Ayn Randian objectivist thought. I mean, we're not talking mathematical errors. We're talking about words and their meanings.

Bryan said...

But I think really the answer lies probably in the selection bias of PolitiFact.

Excellent! That's what I was fishing for. I think you're probably right.

You refuse to acknowledge that your determination of what is and what is not a PolitiFact error is necessarily dependent on your subjective perception, which is incidentally one of the biggest fallacies of Ayn Randian objectivist thought.

I asked you for evidence in support of your above claim, and you haven't provided it. On what basis do I assume that your judgment is not a product of your own subjectivity?

The fact is that there are accepted ways of doing literary interpretation, and those methods do lead to the possibility of an objective determination of error (if that were not the case then what's the point of PolitiFact in the first place?).

Talk facts and I'll prove you wrong. Your baseless insistence on my subjectivity is a retreat in the worst sense.

Steve said...

Hey, I think we're in perfect agreement about PolitiFact's selection bias. And even in some instances, I think we can agree that individual truth-o-meter ratings are suspiciously one-sided, like our earlier discussion of a Grover Norquist entry, or your recent astute observation at politifactbias of the vast disparity between ratings of essentially the same facts (how the slowest recovery since Hoover was false when it came out of Mitt Romney's mouth but mostly true when it came out of Adam Hasner's tweeting thumbs).

________

There's plenty of evidence of your subjectivity guiding your determination. The evidence lies in the fact that there are people who disagree with you. This can mean one of only two things: either we're all denying reality (your facts are facts, but our facts are wrong), or the focus of the argument deals with things of a subjective nature. The first option leads to dogmatic thought, where self-examination is pointless and stupid. The second leads to uncertainty and doubt. I don't know much of anything, but I know arguing with someone who refuses to examine his own beliefs is exhausting and a waste of my time.

Bryan said...

There's plenty of evidence of your subjectivity guiding your determination. The evidence lies in the fact that there are people who disagree with you.

That's not a good start for your argument.

This can mean one of only two things: either we're all denying reality (your facts are facts, but our facts are wrong), or the focus of the argument deals with things of a subjective nature. The first option leads to dogmatic thought, where self-examination is pointless and stupid. The second leads to uncertainty and doubt. I don't know much of anything, but I know arguing with someone who refuses to examine his own beliefs is exhausting and a waste of my time.

And that's not a good finish, either.

You've misidentified the consequences of the first option you mentioned.

In the first instance if we both agree that there are objective facts then we have the option of discussing the reasons for our differences. There is no reason to suppose that an impasse inevitably follows.

Additionally (and as I pointed out earlier), your complaint applies just as well to PolitiFact. So how come you don't automatically count PolitiFact as biased and discount their analysis of the facts?

Why not opt for consistency, Steve? It's hard to wear self-contradiction well.

Steve said...

The bedrock of your conviction that PolitiFact errs a lot stems from your belief in your own infallibility, that you can detect errors perfectly, that what you say is objectively true and that others who disagree with your views are either irrational or will eventually come around to your way of thinking if you just leave enough comments on their blogs.

What I think about PolitiFact is irrelevant to this discussion of your subjectivity, but as we both think PolitiFact is imperfect, I don't think there's much more to say about that.

Bryan said...

The bedrock of your conviction that PolitiFact errs a lot stems from your belief in your own infallibility, that you can detect errors perfectly, that what you say is objectively true and that others who disagree with your views are either irrational or will eventually come around to your way of thinking if you just leave enough comments on their blogs.

Your assertion is blissfully free of supporting evidence. I'm quite aware that most folks are closed-minded, so I reserve persuasive speech for those I think might engage in dialog. I find your reluctance to engage amusing. It just doesn't fit with the way you try to blog. You write a bunch of stuff that appears to accept the existence of objective fact yet in dealing with me you offer a dilemma between hopeless subjectivity and intransigent stubbornness. Your blog approach contradicts your dilemma and it's as though you're not aware of it. Why waste your time if it's that hopeless? Or is it just hopeless with the likes of me (leaving us to wonder how you made that determination without yourself succumbing to subjectivity)?

What I think about PolitiFact is irrelevant to this discussion of your subjectivity, but as we both think PolitiFact is imperfect, I don't think there's much more to say about that.

lol
Discussion is pointless if we agree and pointless if we disagree.

Perfect!

Seriously, are you scared or something? I don't get it. You really can't be serious, can you?

Are you familiar with the Socratic method? Answer my questions and we'll have a productive discussion if you're not one of those closed-minded people you despise.

Karen S. said...

I've just done a post on your along with some links providing some detail about Mr. White's reputation:
http://politi-psychotics.blogspot.com/2011/08/politi-score-quibbling-potatoes-very.html

Steve said...

Karen, I feel honored that you took the time to write such nice things about me and this site! You make me wish I had more time to blog about stuff, because as we both know there's a lot of stuff to blog about out there. I look forward to reading more of your blog in the future.