Friday, 6 December 2019

Driving Us Up The Poll.

Rubbish In, Rubbish Out: Put all this together, and it’s difficult to avoid the conclusion that anyone who responds positively to a pollster’s request to “answer a few questions” is just ever-so-slightly weird. Desperately lonely? Some sort of psephological train-spotter? Political party member primed to skew the poll for or against her opponents? All of the above?

THE INCONVENIENT TRUTH about opinion polls is that the people who participate in them are not really typical Kiwis. Agreeing to participate in anything remotely public-spirited is something fewer and fewer New Zealanders are willing to do. Charities struggle to attract volunteers. Sports teams can’t get enough players. Political parties have long since ceased to be mass organisations. Just finding enough people to satisfy the statistical requirements for an accurate public opinion survey gets harder and harder with every passing year.

It wasn’t always this way. Back in the era when nearly every New Zealand household had an old-fashioned land-line telephone; and the easiest way to locate somebody was to simply ‘look them up’ in the phonebook; polling was a breeze.

It was a time when community action and political debate was engaged in by an extremely broad cross-section of the population. Indeed, New Zealanders were gently chided by Austin Mitchell, the best-selling author of The Half-Gallon, Quarter-Acre, Pavlova Paradise, for being inveterate committee formers. “Pressure groups” were studied forensically by political scientists. Overseas visitors marvelled at a nation of joiners.

The late Professor Keith Jackson, in his book New Zealand: Politics of Change, confirms the strongly participatory character of our democracy by citing the research of R.S. Milne:

“Membership of the New Zealand Labour Party which had peaked in the year 1939-40 at 235,605 remained high after the war at 201,765. By 1960, however, this figure was down to 180,000 distributed through more than 600 branches.”

National’s engagement with New Zealanders was no less impressive: “Much the same pattern appears to have developed within the National Party. Speaking in 1956 the President of the National Party claimed that membership varied from 143,000 in a non-election year to 250,000 in an election year.”

In a nation this politicised, the opinion polling companies of the 1960s and 70s easily assembled the requisite number of participants.

The contrast between those times and our own could hardly be sharper. Who uses the land-line-generated phone book anymore? Asked to do so, most younger Kiwis would probably look at you blankly. The ubiquitous cell-phone presents the pollsters with endless difficulties. There’s no “phone book” for a start, and caller ID allows us all to screen our incoming calls. Many people simply don’t answer unidentified callers – justifiably fearing tele-marketers and scammers.

These latter miscreants have become the bane of land-line subscribers’ lives. For many citizens – especially the elderly – it is considered foolhardy to converse with anyone whose voice isn’t instantly recognisable. Someone can say they’re calling from Colmar Brunton or Reid Research – but how do you know? Better to politely decline and hang-up the receiver.

Put all this together, and it’s difficult to avoid the conclusion that anyone who responds positively to a pollster’s request to “answer a few questions” is just ever-so-slightly weird. Desperately lonely? Some sort of psephological train-spotter? Political party member primed to skew the poll for or against her opponents? All of the above?

These distorting possibilities are only increased when the fact that landlines tend to be attached to owner-occupied dwellings is factored into the polling equation. Just ask any Gen-Xer or Millennial what sort of person is likely to pick up the phone in their own home and they will hiss “Baby Boomer!” Quite correctly. Which way, do you suppose, a voter sitting on a million dollars-plus of tax-free capital gain is more likely to vote – Left or Right? No wonder, really, that about 45 percent of the Party Vote appears to be welded-on to the National Party!

So, what do the pollsters do? Basically, they innovate. They try to assemble a representative number of cell-phone-using voters to offset the encrusted biases of land-liners. Or, like the new kid on the New Zealand polling block – YouGov – they step away from phones altogether in favour of a “panel” of potential online participants many thousands strong.

Trouble is, these innovations require the pollsters to run the raw data through all manner of algorithms to make sure their samples remain representative. They then have to make some, frankly, subjective assumptions about voter behaviour. That’s when things can turn very seriously pear-shaped.

The highly-experienced pollster advising the campaigners for “Remain” in 2016 assumed those who didn’t vote in the 2015 UK General Election would also sit out the Brexit referendum.

That worked out well.

This essay was originally published in The Otago Daily Times of Friday, 6 December 2019.

10 comments:

kiwidave said...

The Colmar Brunton poll has a very good record.

Here’s the last four pre-election CB poll and the actual election results for Labour:

2017: 37% poll vs 36.9% election
2014: 25.2% poll vs 25.1% election
2011: 28.0% poll vs 27.5% election
2008: 35.0% poll vs 34.0% election

Of course these are very close the election date but do suggest that there's not a lot wrong with their methods - or that the respondance are not "typical" Kiwis.

Guerilla Surgeon said...

My wife wants to keep the landline in case of earthquakes or something. But to be honest, nobody except one equally ancient friend of mine rings us up on it anymore. Except of course for the occasional Indian guy apparently called "Colin" who says there's something wrong with my computer. On the other hand, only half a dozen people know my cell phone number, and it's remarkably free of people called Colin.
That aside, how accurate were the polls at the last election? Some of them in the States were pretty damned accurate, just seem to forget about the electoral college on occasion. So somebody somewhere is doing something right.

Unknown said...

It strikes me Chris that in describing the sort of people who answer political pollsters on theirlandlines you are also describing the sort of people - chardonnay in hand - who read Bowalley Road. Maybe it is more balanced than you think - the gap now is not left right but young (unhoused) / old (landlined).

swordfish said...

(1) First Nationwide Polling here was conducted by a group called Gallup NZ (albeit with no ties to the famous US Gallup company) ... it carried out a single Poll during each of the Election Campaigns of 1960 / 63 / 66 ... & I believe it did so entirely on spec. I think I'm right in saying they successfully sold the 60 & 63 Polls to the Herald ... but had no takers in 66. (The NZ Media were still fairly suspicious of sampling accuracy given sharp memories of the 1948 US Presidential Campaign when nearly all Pollsters predicted a Dewey victory).

As it happens, Gallup NZ's polls turned out to be remarkably accurate (absolutely spot on with 2 of the 3 parties - Nat, Lab, SC - in their inaugural one in 1960 for instance).

National Research Bureau began regular political polling in the run-up to the 1969 GE (largely for the Herald) ... they were the only Pollster in town until Heylen joined them (commissioned mainly by TVNZ) in September 1974 (just after Kirk's death).


(2) Contrasting Sampling Methodologies of Major Pollsters:

Colmar Brunton: 50% Landline / 50% Mobile

Reid Research: 75% Landline / 25% Online Panel

UMR: 100% Landline (certainly the case a few years ago ... not entirely sure of current methodolgy but suspect it's still the same)

Stuff / YouGov: 100% Online Panel



(3) @ kiwidave

Actually Colmar Brunton had Labour far too high in the final weeks of the 2017 campaign ... 43 / 43/ 44 ... (contrasting with most other established Pollsters) ... & were forced (through a certain amount of ridicule in the Media) to beat a hasty retreat & do a complete flip-flop with their final Poll (if their last 2 pre-Election polls were taken literally it would've meant an extraordinarily unlikely 6 to 7 point swing from Labour to National in the final week).

The other problem is that Colmar Brunton was no more accurate with its final Poll than other Pollsters (indeed, arguably mildly less so):


2017

Labour:

Election Result: 36.9
UMR: 37
Colmar Brunton: 37
Reid Research: 37.3


National:

Election Result: 44.4
UMR: 43
Reid Research: 45.8
Colmar Brunton: 46


NZF:

Election Result: 7.2
Reid Research: 7.1
UMR: 8
Colmar Brunton: 4.9


Green:

Election Result: 6.3
UMR: 7
Reid Research: 7.1
Colmar Brunton: 8


In other words you simply can't cite greater pre-Election accuracy to argue that CB's results are more robust than those of other Pollsters currently producing decidedly less Nat-friendly results.


And bear in mind that every Pollster puts in much greater effort & resources to get their final pre-Election Polls right, knowing that their professional reputation rests on its accuracy. You can't necessarily judge the precision of current Polls by how the particular Company did in its final Poll before the last Election.


Nick J said...

The old fashioned polling technique is passe not in its accuracy but in its usefulness to those wanting to win an election.

For example a lesser known part of Trumps triumph was the strategy to win the key votes in the Electoral College. Then fining the messages to a very narrow section of swing voters at crucial locations. This was done using smart algorithms on data from social media to collect data, send out messages specifically targeted and refining results. Check out the role of Cambridge Analytical and their use of Google and Facebook data. Trumps triumph was a victory using new methods that now will become mandatory for all seeking election.

Guerilla Surgeon said...

For example a lesser known part of Trumps triumph was the strategy to win the key votes in the Electoral College. Then fining the messages to a very narrow section of swing voters at crucial locations. This was done using smart algorithms on data from social media to collect data, send out messages specifically targeted and refining results."

It's the weakness of the electoral college system, that presidents and parties simply concentrate on those marginal states that will get them over the hump with the electoral college. It was Clinton's mistake that she didn't campaign in these key states at all AFAIK. There are those who claim that if she had done, she would have won the election.

Nick J said...

Indeed GS, my take is that Clinton's hubris that it was her turn, that she couldn't lose against an idiot and that his supporters were deplorable blinded her to the threat. History is great. Did you know tanks can't get through the Ardennes and that Japanese soldiers can't see in the dark?

Shane McDowall said...

What pisses me off is TV One presenting the results of polls as "news".

Worse, it is the lead item on the 6 pm news, and they drag it out over 10 to 15 minutes.

Polls, especially ones taken a year out from the election, are not news.

Guerilla Surgeon said...

Nick. Yes there was a certain amount of hubris, but she only describes some of his supporters as deplorables – although obviously Fox News made a meal of this. Some certainly are.
Fun fact the French knew that tanks could, because they done an exercise on it a year before the war. In this exercise the French army collapsed and the Germans were across the river Meuse in a few days. When the Germans actually did it – they took three hours less than the wargame estimation.

Nick J said...

Bloody hell GS, that really is amusing.