How Accurate Have Senate Polls Been, And What Could That Mean For November?

How Accurate Have Senate Polls Been, And What Could That Mean For November?

The 2016 election was an off year for state-level polls of the presidential race. But for polls of U.S. Senate races, it was a perfectly ordinary year — better than average, even. And Senate polls in the 2018 election were even more accurate.

In the 2017-18 cycle, U.S. Senate polls conducted in the three weeks leading up to the general election had a weighted average error of 4.3 points.4 (As always, we define error as the absolute difference between a poll’s margin and the actual election’s margin between the top two candidates. For example, if a poll gave the Democratic candidate a lead of 2 percentage points but the Republican won the election by 4 points, that would be a 6-point error.) In the 2015-16 cycle, it was 4.9 points. And the long-term average, for all U.S. Senate elections since 1998, is 5.2 points.

In the grand scheme of things, that’s pretty accurate — not as true as polls of presidential general elections have historically been, but better than polls of presidential primaries or U.S. House races.

What’s more, Senate polling has been remarkably consistent in its accuracy from year to year. Since the 2001-02 cycle, the weighted average error of U.S. Senate polls conducted in the final 21 days before the election has never been lower than 4.2 points (its score in 2005-06) or higher than 5.4 points (2013-14). But that doesn’t mean there haven’t been any surprises in Senate races. The battle for the U.S. Senate, of course, is waged on a state-by-state basis, and while polling errors may average out to around 5 points across these races, they are higher in some states and lower in others.

You can see this clearly in the table below, which shows the weighted average error of U.S. Senate polls by cycle for states that had at least 10 polls conducted during the last 21 days of the campaign. (This threshold prevents us from showing states where the sample size of polls is too small to draw meaningful conclusions; it also focuses our attention on the states with the most competitive Senate races each cycle, as close races tend to attract polling interest.) The table also shows the weighted average statistical bias of Senate polls in each state, which tells us whether that polling error went in favor of Democrats or Republicans.

Senate polls have been pretty accurate in most states

Weighted average error and statistical bias of U.S. Senate polls in the final 21 days before general elections, for states with at least 10 such polls, among polls in FiveThirtyEight’s pollster ratings database

2017-18 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
Florida 34 3.1 D+1.8
Arizona 19 3.2 R+1.8
Alabama 16 6.0 R+3.7
Michigan 14 4.2 D+3.1
Missouri 12 4.3 D+4.0
2015-16 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
Florida 30 4.3 D+2.4
Pennsylvania 23 4.5 D+4.2
North Carolina 22 4.5 D+4.4
New Hampshire 21 3.6 D+0.3
Nevada 16 3.0 R+1.2
Georgia 13 4.2 D+2.5
Arizona 12 4.2 D+3.6
Missouri 12 2.8 D+2.5
Ohio 12 5.8 D+5.8
Colorado 10 3.2 D+1.8
Louisiana 10 3.9 D+2.5
Wisconsin 10 6.0 D+6.0
2013-14 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
Colorado 21 2.5 D+0.3
Iowa 21 6.6 D+6.6
North Carolina 21 2.8 D+2.0
Georgia 20 6.1 D+6.1
Massachusetts 19 4.1 R+2.0
New Hampshire 16 2.6 R+1.0
Louisiana 15 5.1 R+5.0
New Jersey 13 3.1 D+2.7
Michigan 11 2.2 R+0.8
Kansas 10 11.5
2011-12 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
Ohio 28 3.0 R+1.5
Florida 21 5.8 R+4.8
Virginia 19 4.1 R+3.9
Wisconsin 15 4.4 R+4.4
Pennsylvania 12 3.4 R+2.7
Massachusetts 10 3.9 R+3.6
2009-10 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
California 23 5.2 R+5.2
New York 22 6.1 R+5.0
Pennsylvania 18 3.0 R+2.0
Ohio 15 2.8 D+0.4
Florida 13 6.2
Illinois 13 2.7 D+1.0
Massachusetts 13 7.7 D+3.5
Colorado 12 3.0 R+3.0
Kentucky 11 4.8 D+4.0
West Virginia 11 6.8 R+6.8
Connecticut 10 3.1 R+1.3
Washington 10 3.4 R+3.0
2007-08 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
Georgia 18 5.0 D+1.4
North Carolina 16 4.7 R+4.6
Virginia 16 3.9 R+2.9
Minnesota 13 4.2 D+0.2
New Hampshire 13 2.9 D+2.2
Colorado 11 3.5 D+0.7
2005-06 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
New Jersey 22 3.4 R+3.0
Tennessee 18 3.7 R+0.3
Missouri 16 2.4 R+1.3
Virginia 15 3.0 D+0.6
Ohio 13 4.9 R+0.8
Maryland 12 4.8 R+4.2
Pennsylvania 12 6.0 R+6.0
Michigan 11 3.3 R+1.9
Minnesota 11 5.0 R+3.4
Montana 10 3.2 D+3.0
2003-04 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
Florida 20 2.5 D+1.9
Colorado 13 3.5 R+1.6
Pennsylvania 13 9.1 R+8.7
Ohio 12 3.4 R+0.6
Oklahoma 10 9.6 D+9.6
2001-02 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
Colorado 13 4.5 D+3.6
New Jersey 13 2.0 R+0.4
Texas 13 6.3 D+6.3
1999-2000 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
New York 18 7.6 R+7.6
Michigan 16 5.8 R+4.9
Pennsylvania 15 10.6 R+10.6
Florida 13 2.5 R+0.4
Ohio 12 2.9 R+1.3
New Jersey 11 3.8 D+0.8
1998 Elections
State No. of Polls Weighted
Avg. Error
Weighted Avg. Statistical Bias
New York 19 8.1 R+8.1
Illinois 10 10.5 R+9.2

Averages are weighted by the square root of the number of polls that a particular pollster conducted in that particular cycle. Polls that are banned by FiveThirtyEight because we know or suspect they faked data are excluded from the analysis. Bias is calculated only for elections where the top two finishers were a Republican and a Democrat. Therefore, it is not calculated for Florida in 2010 or Kansas in 2014, where the second-place finishers were independents.

For instance, in the 2015-16 cycle, polls of most Senate races were pretty good, as 10 of the 12 states with enough polling to analyze had weighted average errors of 4.5 points or lower. In Missouri (2.8) and Nevada (3.0), the polls were downright excellent. But two states were trouble spots: Ohio (5.8) and Wisconsin (6.0). In both states, the polls overestimated Democrats, delivering the party a nasty surprise on election night.

In the 2017-18 cycle, polls of the U.S. Senate races in Missouri, Michigan and especially Arizona and Florida were quite accurate — all had weighted average errors of 4.3 or below. But the race in Alabama proved difficult to poll, with a weighted average error of 6.0. (The fact that it was an irregularly scheduled special election — and the Republican candidate was abandoned by his party after allegations of sexual misconduct emerged — probably contributed to the challenges.)

You can do the same exercise for other years, too. In 2013-14, Kansas, Iowa and Georgia saw the largest weighted average errors, each 6 points or greater. In 2005-06, Pennsylvania polls were particularly inaccurate (their weighted average statistical bias was 6 points toward Republicans), while Missouri polls were exceptionally good (their weighted average error was 2.4 points). In 2003-04, Pennsylvania and Oklahoma polls missed, while three other states’ were very reliable.

But much like the accuracy of swing-state polls in presidential elections, it’s difficult to predict where these trouble spots will be. Some states have very accurate Senate polls one year but particularly inaccurate ones in later cycles, or vice versa. Take Florida: In 2003-04, polls of the Sunshine State had a very low weighted average error of 2.5 points. In 2009-10, their error was a shaky 6.2 points.

That said, some states have an impressive track record of polling accuracy. Colorado, for example, appears six times in the table above, and only once has its polls’ weighted average error exceeded 3.5 points (it was 4.5 points in the 2001-02 elections — still pretty good!). We don’t want to make too much of this — it could just be that Colorado is on a polling hot streak, so to speak, and hasn’t yet hit a rough patch this century.

On the other hand, it could also be a vote of confidence in polls of Colorado’s 2020 U.S. Senate race, which will once again be closely watched. Other 2020 Senate battleground states that have historically produced pretty accurate Senate polls include Arizona (a weighted average error of 4.2 points in 2015-16 and 3.2 points in 2017-18) and North Carolina (4.7 in 2007-08, 2.8 in 2013-14 and 4.5 in 2015-16). On the other hand, polls of Georgia (6.1 in 2013-14) and Iowa (6.6 in 2013-14) have missed the mark significantly in recent elections — which doesn’t necessarily mean they will again in 2020, but we can’t rule it out.

We’ve already seen some pretty unbelievable — perhaps literally so — Senate polls this cycle. Those polls were obviously conducted several months before Election Day, so it’s not fair to compare them to the polls in this analysis. But if polls within the final 21 days of the campaign are still showing the same thing, it might be time to believe them. If the historical pattern holds, Senate polls should be pretty accurate once again in 2020 — at least in most states. Just remember there will probably be a state or two where they misfire.

'I totally buy … Democrats have a chance at winning (Senate)': Nate Silver

Powered by WPeMatico

Comments are closed.
%d bloggers like this: