Tuesday 21 October 2014

Statistical Ties

This post is essentially just me being angry, though if you don't have a foundation in statistics, I might have inadvertently put some content in as well.

Not too many years ago the media got wind of something called "error" in polling. It turns out polls don't necessarily accurately reflect the population! But that's okay, the media got wise, and now they understand and responsibly report on margins of error in polls.

Something you will hear in the news often is the phrase "statistical tie." By that, the reporter will mean that two people are within the poll's margin of error of one another. So, if the poll gives an accurate result within 3%, 19 times out of 20, then a "statistical tie" might be 37 to 40 or 33 to 36, or something like that. If the poll is only accurate within 3%, then 3% apart is basically the same thing.

This doesn't have the properties we normally look for in an equivalence relation, though. If 33 is basically the same as 36 and 36 is basically the same as 39 then isn't 33 basically the same as 39? That's called transitivity, and it's a property we expect to find in any relationship that tells us when things are the same. Obviously it doesn't work here, so we might suspect something more is going on.

The idea that 36 is basically the same as 39 is fantastically dumb. Calling it a tie is nonsense. If the margin of error is plus or minus 3%, 19 times out of 20, that doesn't mean that within that range all bets are off. We could also calculate a 90% confidence interval, and a 50% one. The 50% one would be a lot tighter.

But even if everything within the range was equally probable, that would still mean that there was only a one-in-eight chance that the candidate that polled at 36 actually had more support than the candidate who polled at 39. That's hardly a tie.

Let's also remember that the numbers we are seeing are rounded. If "within three" makes two numbers the same, it's a bit problematic that one of them might actually be 35.9 and the other 39.1.
And if overlap of a margins of error meant equality then we should actually be doubling the margin. That 33 might be 36 and that 39 might be 36. That's a tie too!

Two numbers that are half of a 95% confidence interval apart from one another are actually pretty far apart. The one that polled lower is unlikely to be higher than the one that polled higher. There is no "statistical tie," statistics allows us to calculate the actual chances that each is higher and the answer is not anything like 50-50.

The media presentation of margins of error is one of those we-learned-something-and-know-we-are-worse-informed scenarios. For people who don't actually understand statistics and math, a far more useful understanding is to see the numbers in the polls, and assume that the difference between them is the actual difference, but keep in mind that polls get things wrong sometimes.

No comments:

Post a Comment