The blog I wrote a few months ago on political polls and election results (here) has generated a lot of debate and several media appearances. The most recent of these screened last night - it was Maori Television's new show Media Take (hosted by Russell Brown and Toi Iti). I'll start by thanking Russell for inviting me on the show - it's always fun to debate these issues and I think Media Take in general is going to be a really interesting show.
One of the things Russell asked me to comment on was Paddy Gower's report on a TV3 poll, which had New Zealand First on 4.9%. TV3 based their analysis on New Zealand First being below the threshold. Quite apart from the fact that New Zealand First always does better on election day than they do on the final polls (which could be as much due to votes shifting during the campaign as to polls being accurate), to suggest that a poll result of 4.9% means that any party is definitely below the 5% threshold is statistical nonsense.
As I said to Russell, shifting from 4.9% to 5.0% in a poll of n=1000 requires just one person to change their minds. Alternatively, it requires one of the people who were supporting another party to be out at the time the phone rang, and someone who supports New Zealand First to be home at the right time. That's why we talk about probability and margins of error.
One question Russell asked me which I couldn't quite answer at the time was, if a party polls 4.9%, what's the probability that they're really over 5%? I couldn't answer off the top of my head because the statistical formula is pretty complicated (oddly enough I can't do square roots in my head ;) ), but I guessed it would be pretty high (something like 40%). It's worth bearing in mind that margins of error work as a bell curve - if a survey says a particular party will get say 50%, then just under half the time the actual result will be less than 50%, and just under half the time the actual result will be more than 50% (allowing for the fact that sometimes the result will be exactly right). 95% of the time the actual percentage will be between 'survey percentage less the margin of error' and 'survey percentage plus the margin of error'.
I asked one of our number crunchers to work it out properly - if a survey of n=1000 people gives a party 4.9%, what is the probability that the actual vote for that party is 5.0% or more? The answer is 0.496.
In other words, when TV3 based their report on New Zealand First being below the threshold because they polled 4.9%, there was a 50.4% chance that they were right, and a 49.6% change that they were wrong. That's leaving aside any potential issues with the sampling. They'd have been far better to report two different scenarios - state that the poll showed New Zealand First on 4.9% and present the seats in parliament that way, but also present an alternative assuming that NZ First did get increase their vote by the necessary one person in a thousand.
There are a few more points from the show I'd like to discuss further, but don't have time to do them justice at the moment. In case you're interested though, here are the three main interviews covering the political polling blog.
Media Take (panel discussion with Colmar Brunton's Andrew Robertson): here
Re-Think (panel discussion with Winston Peters & Thomas Lumley): here
Radio NZ Media Watch (one-on-one interview): here