Telling the truth
TELLING THE TRUTH
TELLING THE TRUTH:
RESPONDENT ACCURACY IN MASS MEDIA POLLING
Public opinion polls fulfill several functions for the mass media. They
can serve as the first measure of citizen opinion on an emerging
1 allow reporters to explore the context in which events take place;
and enable news organizations to make news as well as report it.3
Journalists and social scientists use polls in different manners, with
the former emphasizing their immediate news value and the latter
focusing on their value in explaining a general social process.4 Media
coverage of polls has been criticized for serving as a form of
"checkbook journalism"; 5 for encouraging politicians and government
policy makers to make decisions based on short-term, parochial
interests;6 and for failing to fully inform readers about a poll's
methodology7 or the significance of the findings.8
Journalists and social scientists share a desire to ensure that poll
data accurately reflect the views of the public. Attempts to ensure
accuracy can be affected by a concern that goes to the heart of the
issue of validity in survey research: whether the respondents are
telling the truth about the issues on which they comment. Survey
researchers "rarely go beyond self-report"9 in ascertaining whether
respondents are telling the truth during an interview. More than 50
years ago, Herbert Hyman summarized the concern of survey researchers
the title of his paper, "Do They Tell the Truth".10 That concern has
changed and has been underscored by the transformation of survey
research from a tool for academic researchers to an instrument that
helps shape political campaigns11 and public policy.12
Despite the reliance on self-reporting, relatively little is known
about the accuracy of the self-reporting method of data collection.13
More emphasis has been given to the twin issue of the reliability of
self-reports than to their validity.14 Validation studies have been
on such sensitive issues as drug use, bankruptcies and arrests for
drunken driving,15 as well as voting.16 There is evidence that socially
desirable activities like voter registration, voting and charitable
contributions tend to be over-reported, while undesirable experiences
like bankruptcy tend to be under-reported. 17 In voting validation
projects, the process of double checking turnout levels is difficult,
expensive and time-consuming and can generate information on only a
limited number of variables in a survey (e.g., voting turnout) that are
verifiable.18 The National Election Study, which validated
turnout in past elections, chose not to do so in 1992. In
media-sponsored polls, the need to make deadlines makes record
validation checks on poll data even less likely. 19
The tendency of respondents to inflate their self-reports on voting has
been a chronic problem in public opinion polling. Wolfinger and
Rosenstone reported that since 1948, "reported turnout in national
post-election surveys has never been less than 5% higher than the best
estimate of the true turnout figure, and the gap between reported
actual turnout has sometimes approached 20%".20 Four efforts by the
National Election Study to validate turnout records found between 20%
and 30% of respondents over-reported voting.21 Other studies
that in terms of self-reported voting behavior, from 12% to 15% of
respondents provided inaccurate responses after their answers were
compared to official voting records.22
There is less agreement regarding which respondent characteristics and
attitudes might be significant predictors of misreporting of voting.
Some studies have found respondents who were inaccurate in describing
their voting behavior matched the profile of non-voters in general:
young, lower income, non-white citizens with lower political
Other researchers suggest that inaccurate reporters of votes were
different from actual voters in terms of characteristics and
24 For example, Tittle and Hill concluded that "in every instance,
incorrectly claiming that they had voted resembled more closely the
actual voters than nonvoters." 25 A recent study found education had an
independent effect on voter misreporting: the better educated felt
pressure to misreport their vote because of social desirability.26
Little work has been done on two other potential explanations for the
variance in accuracy of self-reports: the respondents' level of
involvement with the news media and their attitudes toward the accuracy
of polling in general. Respondents who attend to the news media on a
regular basis and those who generally believe polls to be accurate
be expected to be more likely to be accurate in their responses than
their counterparts due to a desire to avoid distorting the news
and the polls they find useful and reliable.
This study attempts to clarify the conflicting evidence on whether
inaccurate respondents are significantly different from accurate
respondents. It offers an initial exploration of whether media exposure
and attitudes toward polling significantly affect the accuracy of
responses. The study, based on self-reported turnout levels among
Arizona voters in the 1992 presidential election, also addresses a
broader question: given the high cost of validation checks to ensure
accuracy of poll results, does the level of misreporting justify the
effort required to validate poll results.
The study was based on a two-stage research project. The first stage
involved conducting a statewide telephone survey of 639 registered
voters in Arizona between Nov. 4-11, the week after the 1992
presidential election. The survey was conducted as part of an ongoing
polling program in the Media Research Program of the Department of
Journalism and Telecommunication at Arizona State University. It was
conducted in conjunction with public television station KAET-TV, whose
volunteers were trained to make the telephone calls under the
and direct supervision of the Media Research Program.
Respondents were asked to address a variety of issues, including
whether they had voted in the preceding week's election. They
represented a systematic random sample of voters generated from a list
of all registered voters in the state. A comparison of the
characteristics of the sample statistics to known population
indicated the sample was representative
of the electorate. The overall confidence interval for the sample,
based on a 95% confidence level and dichotomous answers estimated at
.50, is plus or minus 3.9% on responses from the entire sample and
higher for responses from subgroups.
The second stage of the study was based on a validation check to
determine whether respondents actually voted. The authors validated
self-reported information on voting by examining voting records at
of the state's county voter registration offices. Of the 639
respondents, 592 (92%) were successfully validated; the remainder
removed from the study. The major cause of the 47 voters not being
validated was differences between the name of the person as offered
during the initial telephone survey and the formal registration name on
the records; when the names did not precisely match, the respondents
were dropped from the survey.
Accurate and inaccrate respondents were then analyzed based on their
demographic characteristics, media use patterns, political
characteristics and attitudes toward polling in general. Due to the
nominal nature of the dependent variable, Cramer's V, a Chi-Square
measure, was used to measure the level of association. It can
the value of 1 for tables of any dimension, thereby facilitating
comparisons across studies.
The high turnout in the 1992 presidential contest, which reduced the
number of non-voters who might overstate their vote, served as a
limiting factor for this study. Turnout in Arizona reached 75.7% of
registered voters, up from 68.1% in 1988.27 A lower turnout level
have resulted in more non-voters who might have been tempted to tell
interviewers they had voted so as to be associated with a socially
desirable activity. Similarly, the use of only registered voters as the
subjects for the study also may have limited the variance in
responses; if non-registered voters also had been included, the
desirability associated with voting may have prompted a greater
percentage of non-registered voters to say they cast a ballot. Finally,
the study also may have been affected by the accuracy of the master
of voters from which the sample was drawn. The list was generated in
early October and did not include all final registrants.
The results of the self-reported voting and the validated voting levels
are presented in Table 1. The validation check showed that 7.6% of
registered voters in Arizona misrepresented their voting activity after
the 1992 presidential election, while 92.4% accurately reported
they had cast a ballot. Of the 41 inaccurate respondents, 39 claimed
have voted while the validation check showed they had not done so.
Interestingly, two respondents said they had not voted while the
validation check indicated they had done so.
The analysis then turned to independent variables that may help explain
the variance in the accuracy of the self-reports. The initial
of five demographic variables shown in Table 2 found two -- race and
-- were significantly related to the accuracy of responses (although
the weak levels of V=.12, p < .01 for race and V=.11, p < .05 for
Non-white respondents and younger respondents appeared less likely
accurate in reporting whether they voted.
The initial analysis then looked at the media use patterns of
and the accuracy of responses on voting to see if higher media exposure
levels were associated with higher levels of accuracy. The survey
respondents how many days per week they "watch the evening news on
and read a newspaper; it also asked how much attention they paid to
task. None of the relationships were statistically significant.
Seven political characteristics were examined, ranging from
respondents' voting history to their ideology and candidate choice.
Three of the variables were found to be significantly related in the
initial analysis to voting accuracy, again at relatively weak levels:
party affiliation (V=.12, p < .05), whether the person registered to
vote this year (V=.15, p < .01), and choice of presidential candidate
(V=.12, p < .05). Political independents, newly registered voters and
those supporting Clinton appeared to be less likely to be accurate in
reporting whether they voted.
Finally, four additional questions were examined to see if a person's
overall view toward public opinion polling would be related to the
accuracy of their self-reported vote. Respondents were asked to estimate
whether most people lied to pollsters, if polls were inherently
inaccurate, if polls have little influence and if they follow polls
much. None of the indicators were significantly correlated with whether
the respondent accurately reported their vote.
In summary, the initial data analysis suggested that five of the
characteristics and attitudes most closely associated in the literature
with high turnout -- non-minority group members, older voters,
registered Republicans, not newly registered voters and those supporting
a Republican candidate -- also were associated with accurate
of their activities on Election Day.
These five variables were then reexamined while controlling for any
intervening impact of education, as shown in Table 3. This secondary
analysis showed none of the relationships remained significant across
the four education levels examined. Ironically, education by itself
not found to be significantly related to accuracy of responses on
turnout in the initial analysis. But as an intervening variable,
education explained much of the variance in accuracy, especially due to
the tendency of those respondents with a high school education or
to vote at much lower levels than those with more schooling.
the initial significant relationships between characteristics and
attitudes and a voter's accuracy proved to be spurious in the secondary
This study suggests that several potential problems related to
over-reporting of turnout may be less significant than expected.
Differences between accurate and inaccurate respondents in terms of
demographic characteristics, media use patterns and political
characteristics were found to be non-significant or spurious. Even a
respondent's feeling on whether people generally lie to pollsters did
not have a significant relationship as to whether they then lied to
pollster in the context of this study.
Some earlier studies of vote over-reporting did find several variables
were significant predictors of overstated turnout, including age28,
29, party affiliation and participation in earlier elections.30
the results here are similar to other studies31 that found no
significant differences between accurate and non-accurate respondents in
terms of self-reported voting behavior. The similarity between the
groups persisted even after this study examined two additional sets
independent variables: respondents' media exposure levels and
The present results echoed the conclusion of Sigelman, who suggested
"it seems safe to say that researchers who fit models of voting using
self-reported rather than validated data would not be led very far
astray in terms of what they conclude about the overall extent to which
voting is related to demographic and
political characteristics." 32
While validated results would always be the preferred method of data
analysis, the results of this study suggest that the high cost of the
process and the quality of the validation itself should be weighed by
the researcher. For this small Arizona study, it took about 100 hours
and about $1,000 to contact all county boards of elections and
the records of 639 registered voters. This effort yielded only 41
individuals who inaccurately reported their vote, and their
characteristics and attitudes were not significantly different than
those respondents who did accurately report their vote. The high cost
the validation process, coupled with the need for news organizations
report poll results quickly, makes record validation checks on poll
less attractive.33 In this study, the high cost of validation and
lack of significant differences between accurate and non-accurate
respondents -- even on such issues as their media use or views toward
polling -- offers additional evidence that the process may not be
Finally, the accuracy of the validated records themselves may not
always be valid, further limiting the value of the checking process.
Presser and Traugott warn "the standard against which responses are
validated may itself contain errors. ... At least some of the record
entries are probably in error, and mistakes may have been made in
matching survey cases to record cases." 34
So while the journalist and the academic continue to share a desire to
maximize accuracy of polls, this study suggests one traditional
validating whether a respondent actually voted in an election -- does
not yield enough significant information as to justify the effort.
lack of significant differences between respondents who told the
and those who did not reduces any distortion caused by inclusion of
in a survey. The time and effort consumed by such validation checks
might be better spent on the many other issues associated with
maximizing the reliability and validity of polling.
1. Richard Morin, "Polls: When Are They The News?" Washington Post
syndicated story, published in The Record of Stockton, CA (Dec. 21,
2. David H. Weaver and Maxwell E. McCombs, "Journalism and Social
Science: A New Relationship?" Public Opinion Quarterly 44 (1980):
3. Morin, "Polls: When," A13. Burns W. Roper, "Evaluating the Polls
with Poll Data," Public Opinion Quarterly 50 (1986): 10-16.
4. J. Ronald Milavsky, "Improving the Public's Opinion of Public
Opinion," Public Opinion Quarterly 51 (1987): 436-447. Everett C. Ladd,
"Polling and the Press: The Clash of the Institutional Imperatives,"
Public Opinion Quarterly 44 (1980): 574-584.
5. Nicholas Von Hoffman, "Public Opinion Polls: Newspapers Making Their
Own News?" Public Opinion Quarterly 44 (Winter 1980): 572-583.
6. Albert E. Gollin, "Polling and the News Media," Public Opinion
Quarterly 51 (1987): S86-S94. David L. Paletz, Jonathan Y. Short, Helen
Baker, Barbara Cookman Campbell, Richard J. Cooper, and Rochelle M.
Oeslander, "Polls in the Media: Content, Credibility, and
Public Opinion Quarterly 44 (Winter 1980): 495-513. James Fallows,
User's Perspective: Round Table on the Impact of Polls" in Polling
the Issues, ed. Albert W. Cantril (Washington, DC: Seven Locks,
7. Ladd, "Polling and," 574-584. M. Mark Miller and Robert Hurd,
"Conformity to AAPOR Standards in Newspaper Reporting of Public Opinion
Polls," Public Opinion Quarterly 46 (Summer 1982): 243-249.
8. Milavsky, "Improving the," 436-447. Gerhart D. Wiebe, "The New York
Times and Public Opinion Research: A Criticism," Journalism
9. Jean M. Converse, Survey Research in the United States: Roots and
Emergence 1890-1960. (Berkeley, CA: University of California Press,
10. Herbert H. Hyman, "Do They Tell the Truth?" Public Opinion
Quarterly 8 (1944): 557-559.
11. Golin, "Polling and," 86-94.
12. Ladd, "Polling and," 574-584.
13. Stanley Presser & Michael Traugott, "Little White Lies and Social
Science Models," Public Opinion Quarterly 56 (1992): 77-86.
14. Donald G. Granberg and Soren Holmberg, "Self-Reported Turnout and
Voter Validation," American Journal of Political Science 35 (1991):
15. Norman M. Bradburn, Seymour Sudman & Associates, Improving
Interview Method and Questionnaire Design: Response Effects to
Threatening Questions in Survey Research. (San Francisco: Jossey-Bass,
16. Presser and Traugott, "Little White Lies," 77-86. Brian D. Silver,
Paul R. Abramson and Barbara. A. Anderson, "The Presence of Others
Over-Reporting in American National Elections. Public Opinion
50 (1986): 228-239. Michael W. Traugott and John P. Katosh,
Validity in Surveys on Voting Behavior," Public Opinion Quarterly 43
(1979): 359-377. Aage R. Clausen, "Response Validity: Vote Report."
Public Opinion Quarterly 32 (1968): 588-606.
17. Stanley Presser, "Can Changes in Context Reduce Vote Overreporting
in Surveys?" Public Opinion Quarterly 54 (1990): 586-593. Peter V.
Miller and Robert M. Groves, "Matching Survey Responses to Official
Records: An Exploration of Validity in Victimization Reporting." Public
Opinion Quarterly 49 (1985): 366-380.
18. Presser and Traugott, "Little White Lies," 77-86.
19. Arnold H. Ismach, "Polling as a News-Gathering Tool." The Annals
472 (March 1984): 106-118. John N. Rippey, "Use of Polls as a
Tool," Journalism Quarterly 57 (Winter 1980): 642-646, 721.
20. Raymond E. Wolfinger & Steven J. Rosenstone, Who Votes? (New Haven,
CT: Yale University Press, 1980), 115.
21. Presser, "Can Changes," 586-593.
22. Traugott and Katosh, "Response Validity," 359-377. Presser and
Traugott, "Little White Lies," 77-86. Silver, Anderson and Abramson,
"The Presence," 228-239.
23. Traugott and Katosh, "Response Validity," 359-377. John P. Katosh
and Michael W. Traugott, "The Consequences of Validated and
Self-Reported Voting Measures," Public Opinion Quarterly 45 (1981):
24. Silver, Anderson and Abramson, "The Presence," 228-239. C. R.
Tittle & R. J. Hill, "The Accuracy of Self-Reported Data and Prediction
of Political Activity, Public Opinion Quarterly 31 (1967): 103-106.
Sigelman, "The Nonvoting Voter in Voting Research," American Journal
Political Science 26 (1982): 47-56.
25. Tittle and Hill, "The Accuracy," 104.
26. Presser and Traugott, "Little White Lies," 77-86.
27. Royce Crocker, "Voter Registration and Turnout, 1948-1992,"
Congressional Research Service, Library of Congress (1993): CRS-22.
28. Traugott and Katosh, "Response Validity," 359-377.
29. Sigelman, "The Non-Voting Voter," 47-56.
30. Traugott and Katosh, "Response Validity," 359-377.
31. Silver, Anderson and Abramson, "The Presence," 228-239. Tittle and
Hill, "The Accuracy," 103-106.
32. Sigelman, "The Non-Voting Voter," 53.
33. Ismach, "Polling as," 106-118. Rippey, "Use of," 642-646.
34. Presser and Traugott, "Little White Lies," 78.
Results of Validation Check of Voting Records
% of all n
Reported did vote, 90.0% 487
Validated did vote
Reported did not vote, 2.4% 13
Validated did not vote
Reported did vote, 7.2% 39
Validated did not vote
Reported did not vote, 0.4% 2
Validated did vote
Characteristics of Accurate vs. Non-Accurate Respondents
Accurate Accurate Total n V*
White 94% 6% 100% 490 .12##
Non-white 84% 16% 100% 70
18-45 90% 10% 100% 460 .11#
45 + 95% 5% 100% 282
< High school 90% 10% 100% 124 .05
Some college 94% 6% 100% 203
College graduate 93% 7% 100% 139
Post-graduate 91% 9% 100% 68
Male 91% 9% 100% 214 .04
Female 93% 7% 100% 325
Protestant 96% 4% 100% 183 .11
Catholic 90% 10% 100% 126
Mormon 97% 3% 100% 37
Other 90% 10% 100% 89
No relig pref. 90% 10% 100% 90
2. Media use characteristics
evening news on TV
0 days 92% 8% 100% 25 .09
1 93% 7% 100% 27
2 85% 15% 100% 33
3 93% 7% 100% 58
4 90% 10% 100% 52
5 91% 9% 100% 86
6 94% 6% 100% 49
7 94% 6% 100% 206
Attention paid to TV
cvg. of pres. campaign
A great deal 92% 8% 100% 201 .05
Quite a bit 92% 8% 100% 147
Some 94% 6% 100% 116
Very little 91% 9% 100% .43
None 90% 10% 100% .20
Accurate Accurate Total n V*
0 days 95% 5% 100% 42 .12
1 88% 12% 100% 41
2 90% 10% 100% 38
3 86% 14% 100% 42
4 100% 0% 100% 23
5 91% 9% 100% 44
6 90% 10% 100% 29
7 94% 6% 100% 271
Attention paid to newsp.
cvg. of pres. campaign
A great deal 97% 3% 100% 141 .11
Quite a bit 90% 10% 100% 130
Some 90% 10% 100% 134
Very little 92% 8% 100% 67
None 93% 7% 100% 43
3. Political characteristics
Republican 96% 4% 100% 254 .12#
Democrat 90% 10% 100% 231
Other 85% 15% 100% 48
vote this year
Yes 86% 14% 100% 154 .15##
No 95% 5% 100% 376
Bush 96% 4% 100% 199 .12#
Clinton 89% 11% 100% 194
Perot 93% 7% 100% 99
in past 10 years
All elections 95% 5% 100% 302 .12
Half of election 90% 10% 100% 158
Less than half 89% 11% 100% 45
First time voted 84% 16% 100% 19
Whether it was
person's first time
Yes 90% 10% 100% 29 .03
No 93% 7% 100% 487
Accurate Accurate Total n V*
Extremely closely 94% 6% 100% 116 .13
Very closely 95% 6% 100% 206
Somewhat closely 90% 10% 100% 174
Not very close 93% 7% 100% 28
Not closely at all 78% 22% 100% 11
Conservative 94% 6% 100% 201 .09
Moderate 92% 8% 100% 254
Liberal 87% 13% 100% 68
4. Attitudes toward public opinion polling
Influence of polls
no influence 93% 7% 100% 94 .06
some influence 94% 6% 100% 181
a lot of influence 91% 9% 100% 165
a great deal of infl. 96% 4% 100% 68
Are people truthful
in talking to pollsters
never 83% 7% 100% 12 .10
seldom 90% 10% 100% 92
most time 95% 5% 100% 313
all time 89% 11% 100% 56
How accurate are polls
in the media
extremely accurate 93% 7% 100% 14 .05
very accurate 96% 4% 100% 66
generally accurate 92% 8% 100% 271
not very accurate 93% 7% 100% 122
not accurate at all 93% 7% 100% 15
How much do you
follow media polls
frequently 96% 4% 100% 74 .05
often 92% 8% 100% 113
seldom 92% 8% 100% 214
never 93% 7% 100% 118
Measure of association expressed by Cramer V
# p < .05
## p < .01
Strength of Association, Respondent Characteristics Vs. Turnout Accuracy,
Controlling for Education
< High Some College Post
School College Grad Graduate
V p V p V p V p
Race .37 <.001 .03 >.05 .09 >.05 .16 <.001
Age .22 <.05 .08 >.05 .07 >.05 .11 >.05
Party .28 <.01 .09 >.05 .11 >.05 .10 >.05
Register .09 >.05 .19 <.01 .16 >.05 .19 >.05
Presidential .15 >.05 .10 >.05 .12 >.05 .10 >.05
Measure of association expressed by Cramer V.