Gallup Red, Pew Blue - "House Effects" or Polling Firm Bias

June 22, 2012, 8:52 AM

Calculating ‘House Effects’ of Polling Firms

By NATE SILVER

http://fivethirtyeight.blogs.nytimes.com/2012/06/22/calculating-house-effects-of-polling-firms/?partner=rss&emc=rss

One of the ways that our forecast model seeks to make polling data more robust is by calculating and adjusting for “house effects,” which are systematic tendencies for polling firms to favor either the Democratic or Republican candidate.

Fairly often, what is perceived as a change in the polls is really just a reflection of these tendencies. A polling firm that tends to show favorable results for Barack Obama might survey a state one week, showing him three percentage points ahead. A different polling firm that usually has favorable results for Mitt Romney comes out with a survey in the same state the next week, showing him up by two points. Rather than indicating a change in voter preferences, these results may just reflect the different techniques and assumptions that the polling firms have applied.

The chart below reflects our model’s current estimate of the house effects for some of the more prolific polling firms. (A polling firm with a Democratic house effect tends to show better numbers for Mr. Obama than in comparable surveys, while one with a Republican house effect tends to have good numbers for Mr. Romney.)

As you can see, there is a fairly wide spread in the polls this year. For instance, the firm Public Policy Polling shows results that are about three percentage points more favorable to Mr. Obama than the consensus of surveys. (How is the consensus determined? More about that in a moment.) Conversely, Gallup’s polls have been leaning Republican this year, by about 2.5 percentage points. (See the Huffington Post’s Mark Blumenthal for a good explanation as to why.)

Rasmussen Reports, which has had Republican-leaning results in the past, does so again this year. However, the tendency is not very strong – a Republican lean of about 1.3 points.

If you are used to looking at Rasmussen Reports polls, your impression may be that their partisan lean is stronger than that. This is not wrong, actually. However, some of the difference results from the fact that Rasmussen is polling likely voters, while many other polling firms are polling registered voters.

Our methodology applies a “likely voter adjustment” in addition to the house effects adjustment. In fact, the likely voter adjustment is calculated first.

We’ll explain the likely voter adjustment more completely in a separate post, but it works by comparing the results when the same polling firm issues results for both registered and likely voters in the same survey.

Historically in presidential years, polls of likely voters have been about 1.5 points more favorable to the Republican candidate than registered voter polls. The model uses this 1.5-point adjustment as a default, but updates it as new polls become available that provide for the comparison. So far this year, very few pollsters have released results like these, and they’ve been fairly close to the historical average, so the estimate hasn’t strayed very far from the default – it is now 1.6 points rather than 1.5.

Thus, all polls of registered voters are shifted by 1.6 points toward Mitt Romney so that they are comparable to likely-voter polls. (A similar adjustment is applied to polls of all adults, rather than registered voters, although these cases come up only rarely in presidential surveys.)

Since Rasmussen Reports is one of the few polling firms to be surveying likely voters, adjusting other polls to a likely-voter basis tends to bring the other polling firms closer in line with it. Without that likely-voter adjustment, Rasmussen Reports would have roughly a three point Republican-leaning house effect.

The philosophy of the model is simply to strip most of the house effect out of the poll. So a Public Policy Polling survey that showed Barack Obama ahead by seven points in Colorado would be treated as more like a four point lead for Mr. Obama once its house effect is accounted for.

(As a more technical matter, the model does allow the polling firm to retainsome of its house effect. Since calculating the house effect is subject to error – anomalous results may reflect sampling error rather than anything systemic – it reverts them to the mean somewhat. But the adjustment is fairly aggressive, especially if the house effect is large or if the polling firm has released a large amount of data.)

The house effect adjustment is calculated by applying a regression analysis that compares the results of different polling firms’ surveys in the same states. For instance, if Marist comes out with a survey that shows Barack Obama ahead by four points in Ohio, and Quinnipiac has one that shows him ahead by one point instead, that is evidence that Marist’s polls are 3 points more Democratic-leaning than Quinnipiac’s.

The regression analysis makes these comparisons across all combinations of polling firms and states, and comes up with an overall estimate of the house effect as a result. National polls are treated as a ‘state’ and are used in the calculation. The calculation accounts for changes in the national polling trendline over time, and so ideally will reflect true differences in methodology rather than just accidents of timing.

The simpler method of taking an average of the polls is each state will certainly go some way toward reducing house effects. However, the mixture of the polling firms that are active in each state is not always the same, and sometimes the Republican-leaning and Democratic-leaning polls do not cancel out. For instance, most of the polls released in Michigan recently are by firms like Rasmussen Reports and We Ask America with Republican-leaning house effects, while most of those released in Washington State are by firms like Public Policy Polling and SurveyUSA that have had Democratic-leaning ones.

I have left one question unanswered, however. We might say that Public Policy Polling has a three percentage point Democratic house effect. But that reflects 3 points as compared to what exactly?

The basis for the comparison is a weighted average of the polls, with heavy emphasis given to a firm’s pollster rating in calculating its say in the consensus. Polling firms that we believe to produce more reliable results – based on their past performance and their methodological standards – have a lot more say in the calculation.

In addition, in a new wrinkle in the model this year, the consensus is estimated solely from polling firms that include cellphones in their sample. More and more polling firms are including cellphones, including almost all of the major news organizations. But some still do not, and this is an increasingly unacceptable practice. About one-third of American households do not have landlines at all, while another one-sixth have landlines but rarely or never accept calls on them.

What this means is that polling firms that are not including cellphones are missing somewhere between one-third and one-half of the American population. That really stretches the definition of a scientific survey. There is reasonably persuasive evidence that this can bias results. Polling firms can try to compensate for the problem by applying demographic weights, but this entails making a lot of assumptions that may introduce other types of bias and error.

There has been a modest tendency this year for polling firms that do notinclude cellphones to show more favorable results for Mr. Romney than those that do include cellphones. The more powerful pattern, however, may simply be that polling firms that do not include cellphones are producing more erratic results in one direction or another as their weighting algorithms try and sometimes struggle to compensate for their failure to collect a random sample. We are not quite at the point of completely throwing these results out (although that may be a defensible position). But we do calculate the consensus based solely on the polls that do include cellphones. Therefore the house effect adjustment includes an implicit cellphone adjustment.