This story annoyed me, I guess because I unfairly to expect better of ABC news than everyone else. And, no, it wasn’t anything to do with the politics or politicians involved. My highlighting should give away the problem:
Labor gains ground, Abbott loses sheen
Dissatisfaction with Opposition Leader Tony Abbott hit a record high as Federal Labor’s primary vote staged a comeback in the latest opinion poll.
The Newspoll published in The Australian newspaper shows Labor’s primary vote is up three points to 32 per cent and has now climbed seven points from its record low of 26 per cent seven weeks ago.
The Coalition’s primary vote is down one point to 44 per cent, with the Greens down three points to 12 per cent.
On a two-party preferred basis, the Coalition lost one point, but is still ahead of Labor 53 to 47 per cent.
Dissatisfaction with the job performance of Mr Abbott is up two points to 57 per cent, with 34 per cent satisfied.
Prime Minister Julia Gillard has improved but dissatisfaction for her performance stands at 60 per cent and satisfaction at 30 per cent.
Mr Abbott remains preferred prime minister, just ahead of Ms Gillard 40 to 39 per cent.
I normally avoid stories about polls because, well, they’re pretty pointless and uninteresting filler for the hamster wheel. And also because they often annoy me. Why?
All polls have margins of error, which means that small changes in percentage points may not be real and may just be due to sampling error (you guys already know about this, and anyone else reading who doesn’t can go here for an explanation – full version). I have no idea what the Newspoll error margin is off the top of my head, but I can immediately guess that statements like “Mr Abbott … just ahead of Ms Gillard 40 to 39 per cent” will be completely meaningless.
I looked up Newspoll’s margin of error:
The latest survey is based on 1,146 interviews among voters. The maximum margin of sampling error is plus or minus 3 percentage points.
Any difference of 3% or less, either up or down, could be due to random variation. So, simply speaking, only results with more than a 3% difference should be presented or interpreted as likely to be real differences.
The red and blue highlighting indicates this problem. The red highlighting shows the parts of the report where differences are meaningless – they present an impression of changes or differences that we can’t be sure are anything more than error and draw conclusions based on wishful thinking. (Of course there’s nothing intrinsically wrong with stating that Mr Abbott is on 40% and Ms Gillard 39%, but to say that based on this poll Abbott is ahead is wrong. Or even tell readers why they’re not statistically different: but the difference is inside Newspoll’s 3% margin of error.)
The bits that show a difference greater than the margin of error are highlighted in blue.
Warning, statistics: One thing to consider with the error of a proportion (which is really annoying if you’re graphing data close to 50%), is that error bars get larger the closer the proportion is to 0.5 (ie. 50%) and the error bars get smaller the further away the proportion is from 0.5 (eg. 1% or 99%). So the maximum margin of error would be for a poll result where 50% of people approved of Ghengis Khan.
But if I now put in 12% (p=0.12, n=1146) then the error is only 1.9% (because 0.12 is further away from 0.5). Why have I bothered to determine this? Because it means that the “Greens down three points to 12 per cent” is actually greater than the margin of error.
[Note: saying Labor's primary vote staged a comeback up three points to 32 per cent is still wrong. I checked and the margin of error for 32% (p=0.32, n=1146) is 2.7%, which rounds off to 3% (all the poll numbers are given rounded off). So if it was, presumably, 29% last poll and 32% this poll, then the errors totally overlap.]
What could be a good idea is if Newspoll put a little * or something in its reports that indicated that the difference between eg Liberal/Labour approvals is greater than the margin of error. That way understandably harried journos wouldn’t have to think about it, they’d just see the star and know the difference was ‘real’.