Bob CollinsWe don’t do polls at this news organization anymore and with good reason — you can’t trust them and news organizations are risking a lot when they put that trust in the hands of third-party pollsters to issue polls under their brand.
MPR pulled out of the polling business in 2012 to spend the money on other endeavors. The move came two years after its pollster showed an out-of-whack gubernatorial race just before the final election in 2010.
Today, the New York Times’ Upshot column casts further doubt on the value of polls, examining the secrecy surrounding their methodology and tactics.
For example, they often are so afraid that they’re wrong, that they release results at the same time as other polls showing similar results, known as “herding.”
There is strong evidence that some firms have engaged in herding. Ahead of the 2012 presidential election, surveys by Public Policy Polling (PPP), a Democratic firm, showed an unusual pattern. As voters’ preferences shifted, the estimated racial composition of the electorate shifted as well. When President Obama lost support among white voters, for instance, the poll would include more nonwhite voters than a previous survey had. As a result, the top line results remained fairly stable.
Tom Jensen, Public Policy’s director, said that these shifts were a result of randomly deleting respondents until the sample matched PPP’s expectation for how respondents said they voted in 2008. But the practice was not enumerated in the firm’s public methodology statement, and the public releases of the survey did not include the statement.
There is no way to be sure if other pollsters engage in similar practices or defensible decisions with obvious consequences for the result. If they did, we probably wouldn’t know.
But there’s at least a suspicion that the polls are cooked.
It’s possible to figure it out when the polling firms release methodology and microdata, but that doesn’t often happen until months after the poll is released and is widely quoted.
New standards have been proposed, but polling firms have been reluctant to climb aboard the movement.
That’s worth remembering as we get into the insufferable political horse race season.
