|
Telling the truth
TELLING THE TRUTH
TELLING THE TRUTH:
RESPONDENT ACCURACY IN MASS MEDIA POLLING
Public opinion polls fulfill several functions for the mass media. They
can serve as the first measure of citizen opinion on an emerging
issue;
1 allow reporters to explore the context in which events take place;
2
and enable news organizations to make news as well as report it.3
Journalists and social scientists use polls in different manners, with
the former emphasizing their immediate news value and the latter
focusing on their value in explaining a general social process.4 Media
coverage of polls has been criticized for serving as a form of
"checkbook journalism"; 5 for encouraging politicians and government
policy makers to make decisions based on short-term, parochial
interests;6 and for failing to fully inform readers about a poll's
methodology7 or the significance of the findings.8
Journalists and social scientists share a desire to ensure that poll
data accurately reflect the views of the public. Attempts to ensure
accuracy can be affected by a concern that goes to the heart of the
issue of validity in survey research: whether the respondents are
telling the truth about the issues on which they comment. Survey
researchers "rarely go beyond self-report"9 in ascertaining whether
respondents are telling the truth during an interview. More than 50
years ago, Herbert Hyman summarized the concern of survey researchers
in
the title of his paper, "Do They Tell the Truth".10 That concern has
not
changed and has been underscored by the transformation of survey
research from a tool for academic researchers to an instrument that
helps shape political campaigns11 and public policy.12
Despite the reliance on self-reporting, relatively little is known
about the accuracy of the self-reporting method of data collection.13
More emphasis has been given to the twin issue of the reliability of
self-reports than to their validity.14 Validation studies have been
done
on such sensitive issues as drug use, bankruptcies and arrests for
drunken driving,15 as well as voting.16 There is evidence that socially
desirable activities like voter registration, voting and charitable
contributions tend to be over-reported, while undesirable experiences
like bankruptcy tend to be under-reported. 17 In voting validation
projects, the process of double checking turnout levels is difficult,
expensive and time-consuming and can generate information on only a
limited number of variables in a survey (e.g., voting turnout) that are
verifiable.18 The National Election Study, which validated
self-reported
turnout in past elections, chose not to do so in 1992. In
media-sponsored polls, the need to make deadlines makes record
validation checks on poll data even less likely. 19
The tendency of respondents to inflate their self-reports on voting has
been a chronic problem in public opinion polling. Wolfinger and
Rosenstone reported that since 1948, "reported turnout in national
post-election surveys has never been less than 5% higher than the best
estimate of the true turnout figure, and the gap between reported
and
actual turnout has sometimes approached 20%".20 Four efforts by the
National Election Study to validate turnout records found between 20%
and 30% of respondents over-reported voting.21 Other studies
suggested
that in terms of self-reported voting behavior, from 12% to 15% of
respondents provided inaccurate responses after their answers were
compared to official voting records.22
There is less agreement regarding which respondent characteristics and
attitudes might be significant predictors of misreporting of voting.
Some studies have found respondents who were inaccurate in describing
their voting behavior matched the profile of non-voters in general:
young, lower income, non-white citizens with lower political
interest.23
Other researchers suggest that inaccurate reporters of votes were
not
different from actual voters in terms of characteristics and
attitudes.
24 For example, Tittle and Hill concluded that "in every instance,
those
incorrectly claiming that they had voted resembled more closely the
actual voters than nonvoters." 25 A recent study found education had an
independent effect on voter misreporting: the better educated felt
more
pressure to misreport their vote because of social desirability.26
Little work has been done on two other potential explanations for the
variance in accuracy of self-reports: the respondents' level of
involvement with the news media and their attitudes toward the accuracy
of polling in general. Respondents who attend to the news media on a
regular basis and those who generally believe polls to be accurate
would
be expected to be more likely to be accurate in their responses than
their counterparts due to a desire to avoid distorting the news
reports
and the polls they find useful and reliable.
This study attempts to clarify the conflicting evidence on whether
inaccurate respondents are significantly different from accurate
respondents. It offers an initial exploration of whether media exposure
and attitudes toward polling significantly affect the accuracy of
responses. The study, based on self-reported turnout levels among
Arizona voters in the 1992 presidential election, also addresses a
broader question: given the high cost of validation checks to ensure
accuracy of poll results, does the level of misreporting justify the
effort required to validate poll results.
Method
The study was based on a two-stage research project. The first stage
involved conducting a statewide telephone survey of 639 registered
voters in Arizona between Nov. 4-11, the week after the 1992
presidential election. The survey was conducted as part of an ongoing
polling program in the Media Research Program of the Department of
Journalism and Telecommunication at Arizona State University. It was
conducted in conjunction with public television station KAET-TV, whose
volunteers were trained to make the telephone calls under the
training
and direct supervision of the Media Research Program.
Respondents were asked to address a variety of issues, including
whether they had voted in the preceding week's election. They
represented a systematic random sample of voters generated from a list
of all registered voters in the state. A comparison of the
demographic
characteristics of the sample statistics to known population
parameters
indicated the sample was representative
of the electorate. The overall confidence interval for the sample,
based on a 95% confidence level and dichotomous answers estimated at
.50, is plus or minus 3.9% on responses from the entire sample and
higher for responses from subgroups.
The second stage of the study was based on a validation check to
determine whether respondents actually voted. The authors validated
self-reported information on voting by examining voting records at
each
of the state's county voter registration offices. Of the 639
original
respondents, 592 (92%) were successfully validated; the remainder
were
removed from the study. The major cause of the 47 voters not being
validated was differences between the name of the person as offered
during the initial telephone survey and the formal registration name on
the records; when the names did not precisely match, the respondents
were dropped from the survey.
Accurate and inaccrate respondents were then analyzed based on their
demographic characteristics, media use patterns, political
characteristics and attitudes toward polling in general. Due to the
nominal nature of the dependent variable, Cramer's V, a Chi-Square
based
measure, was used to measure the level of association. It can
attain
the value of 1 for tables of any dimension, thereby facilitating
comparisons across studies.
The high turnout in the 1992 presidential contest, which reduced the
number of non-voters who might overstate their vote, served as a
limiting factor for this study. Turnout in Arizona reached 75.7% of
registered voters, up from 68.1% in 1988.27 A lower turnout level
would
have resulted in more non-voters who might have been tempted to tell
interviewers they had voted so as to be associated with a socially
desirable activity. Similarly, the use of only registered voters as the
subjects for the study also may have limited the variance in
accuracy of
responses; if non-registered voters also had been included, the
social
desirability associated with voting may have prompted a greater
percentage of non-registered voters to say they cast a ballot. Finally,
the study also may have been affected by the accuracy of the master
list
of voters from which the sample was drawn. The list was generated in
early October and did not include all final registrants.
Results
The results of the self-reported voting and the validated voting levels
are presented in Table 1. The validation check showed that 7.6% of
registered voters in Arizona misrepresented their voting activity after
the 1992 presidential election, while 92.4% accurately reported
whether
they had cast a ballot. Of the 41 inaccurate respondents, 39 claimed
to
have voted while the validation check showed they had not done so.
Interestingly, two respondents said they had not voted while the
validation check indicated they had done so.
The analysis then turned to independent variables that may help explain
the variance in the accuracy of the self-reports. The initial
analysis
of five demographic variables shown in Table 2 found two -- race and
age
-- were significantly related to the accuracy of responses (although
at
the weak levels of V=.12, p < .01 for race and V=.11, p < .05 for
age).
Non-white respondents and younger respondents appeared less likely
to be
accurate in reporting whether they voted.
The initial analysis then looked at the media use patterns of
respondents
and the accuracy of responses on voting to see if higher media exposure
levels were associated with higher levels of accuracy. The survey
asked
respondents how many days per week they "watch the evening news on
TV"
and read a newspaper; it also asked how much attention they paid to
each
task. None of the relationships were statistically significant.
Seven political characteristics were examined, ranging from
respondents' voting history to their ideology and candidate choice.
Three of the variables were found to be significantly related in the
initial analysis to voting accuracy, again at relatively weak levels:
party affiliation (V=.12, p < .05), whether the person registered to
vote this year (V=.15, p < .01), and choice of presidential candidate
(V=.12, p < .05). Political independents, newly registered voters and
those supporting Clinton appeared to be less likely to be accurate in
reporting whether they voted.
Finally, four additional questions were examined to see if a person's
overall view toward public opinion polling would be related to the
accuracy of their self-reported vote. Respondents were asked to estimate
whether most people lied to pollsters, if polls were inherently
inaccurate, if polls have little influence and if they follow polls
much. None of the indicators were significantly correlated with whether
the respondent accurately reported their vote.
In summary, the initial data analysis suggested that five of the
characteristics and attitudes most closely associated in the literature
with high turnout -- non-minority group members, older voters,
registered Republicans, not newly registered voters and those supporting
a Republican candidate -- also were associated with accurate
reporting
of their activities on Election Day.
These five variables were then reexamined while controlling for any
intervening impact of education, as shown in Table 3. This secondary
analysis showed none of the relationships remained significant across
the four education levels examined. Ironically, education by itself
was
not found to be significantly related to accuracy of responses on
turnout in the initial analysis. But as an intervening variable,
education explained much of the variance in accuracy, especially due to
the tendency of those respondents with a high school education or
less
to vote at much lower levels than those with more schooling.
Therefore,
the initial significant relationships between characteristics and
attitudes and a voter's accuracy proved to be spurious in the secondary
analysis.
Discussion
This study suggests that several potential problems related to
over-reporting of turnout may be less significant than expected.
Differences between accurate and inaccurate respondents in terms of
demographic characteristics, media use patterns and political
characteristics were found to be non-significant or spurious. Even a
respondent's feeling on whether people generally lie to pollsters did
not have a significant relationship as to whether they then lied to
the
pollster in the context of this study.
Some earlier studies of vote over-reporting did find several variables
were significant predictors of overstated turnout, including age28,
race
29, party affiliation and participation in earlier elections.30
However,
the results here are similar to other studies31 that found no
significant differences between accurate and non-accurate respondents in
terms of self-reported voting behavior. The similarity between the
two
groups persisted even after this study examined two additional sets
of
independent variables: respondents' media exposure levels and
attitudes
toward polling.
The present results echoed the conclusion of Sigelman, who suggested
"it seems safe to say that researchers who fit models of voting using
self-reported rather than validated data would not be led very far
astray in terms of what they conclude about the overall extent to which
voting is related to demographic and
political characteristics." 32
While validated results would always be the preferred method of data
analysis, the results of this study suggest that the high cost of the
process and the quality of the validation itself should be weighed by
the researcher. For this small Arizona study, it took about 100 hours
and about $1,000 to contact all county boards of elections and
validate
the records of 639 registered voters. This effort yielded only 41
individuals who inaccurately reported their vote, and their
characteristics and attitudes were not significantly different than
those respondents who did accurately report their vote. The high cost
of
the validation process, coupled with the need for news organizations
to
report poll results quickly, makes record validation checks on poll
data
less attractive.33 In this study, the high cost of validation and
the
lack of significant differences between accurate and non-accurate
respondents -- even on such issues as their media use or views toward
polling -- offers additional evidence that the process may not be
worth
the effort.
Finally, the accuracy of the validated records themselves may not
always be valid, further limiting the value of the checking process.
Presser and Traugott warn "the standard against which responses are
validated may itself contain errors. ... At least some of the record
entries are probably in error, and mistakes may have been made in
matching survey cases to record cases." 34
So while the journalist and the academic continue to share a desire to
maximize accuracy of polls, this study suggests one traditional
method
--
validating whether a respondent actually voted in an election -- does
not yield enough significant information as to justify the effort.
The
lack of significant differences between respondents who told the
truth
and those who did not reduces any distortion caused by inclusion of
both
in a survey. The time and effort consumed by such validation checks
might be better spent on the many other issues associated with
maximizing the reliability and validity of polling.
Notes
1. Richard Morin, "Polls: When Are They The News?" Washington Post
syndicated story, published in The Record of Stockton, CA (Dec. 21,
1994): A13.
2. David H. Weaver and Maxwell E. McCombs, "Journalism and Social
Science: A New Relationship?" Public Opinion Quarterly 44 (1980):
477-494.
3. Morin, "Polls: When," A13. Burns W. Roper, "Evaluating the Polls
with Poll Data," Public Opinion Quarterly 50 (1986): 10-16.
4. J. Ronald Milavsky, "Improving the Public's Opinion of Public
Opinion," Public Opinion Quarterly 51 (1987): 436-447. Everett C. Ladd,
"Polling and the Press: The Clash of the Institutional Imperatives,"
Public Opinion Quarterly 44 (1980): 574-584.
5. Nicholas Von Hoffman, "Public Opinion Polls: Newspapers Making Their
Own News?" Public Opinion Quarterly 44 (Winter 1980): 572-583.
6. Albert E. Gollin, "Polling and the News Media," Public Opinion
Quarterly 51 (1987): S86-S94. David L. Paletz, Jonathan Y. Short, Helen
Baker, Barbara Cookman Campbell, Richard J. Cooper, and Rochelle M.
Oeslander, "Polls in the Media: Content, Credibility, and
Consequences,"
Public Opinion Quarterly 44 (Winter 1980): 495-513. James Fallows,
"The
User's Perspective: Round Table on the Impact of Polls" in Polling
on
the Issues, ed. Albert W. Cantril (Washington, DC: Seven Locks,
1980):
514-529.
7. Ladd, "Polling and," 574-584. M. Mark Miller and Robert Hurd,
"Conformity to AAPOR Standards in Newspaper Reporting of Public Opinion
Polls," Public Opinion Quarterly 46 (Summer 1982): 243-249.
8. Milavsky, "Improving the," 436-447. Gerhart D. Wiebe, "The New York
Times and Public Opinion Research: A Criticism," Journalism
Quarterly 44
(1967): 654-658.
9. Jean M. Converse, Survey Research in the United States: Roots and
Emergence 1890-1960. (Berkeley, CA: University of California Press,
1987), 415.
10. Herbert H. Hyman, "Do They Tell the Truth?" Public Opinion
Quarterly 8 (1944): 557-559.
11. Golin, "Polling and," 86-94.
12. Ladd, "Polling and," 574-584.
13. Stanley Presser & Michael Traugott, "Little White Lies and Social
Science Models," Public Opinion Quarterly 56 (1992): 77-86.
14. Donald G. Granberg and Soren Holmberg, "Self-Reported Turnout and
Voter Validation," American Journal of Political Science 35 (1991):
448-459.
15. Norman M. Bradburn, Seymour Sudman & Associates, Improving
Interview Method and Questionnaire Design: Response Effects to
Threatening Questions in Survey Research. (San Francisco: Jossey-Bass,
1979).
16. Presser and Traugott, "Little White Lies," 77-86. Brian D. Silver,
Paul R. Abramson and Barbara. A. Anderson, "The Presence of Others
and
Over-Reporting in American National Elections. Public Opinion
Quarterly
50 (1986): 228-239. Michael W. Traugott and John P. Katosh,
"Response
Validity in Surveys on Voting Behavior," Public Opinion Quarterly 43
(1979): 359-377. Aage R. Clausen, "Response Validity: Vote Report."
Public Opinion Quarterly 32 (1968): 588-606.
17. Stanley Presser, "Can Changes in Context Reduce Vote Overreporting
in Surveys?" Public Opinion Quarterly 54 (1990): 586-593. Peter V.
Miller and Robert M. Groves, "Matching Survey Responses to Official
Records: An Exploration of Validity in Victimization Reporting." Public
Opinion Quarterly 49 (1985): 366-380.
18. Presser and Traugott, "Little White Lies," 77-86.
19. Arnold H. Ismach, "Polling as a News-Gathering Tool." The Annals
472 (March 1984): 106-118. John N. Rippey, "Use of Polls as a
Reporting
Tool," Journalism Quarterly 57 (Winter 1980): 642-646, 721.
20. Raymond E. Wolfinger & Steven J. Rosenstone, Who Votes? (New Haven,
CT: Yale University Press, 1980), 115.
21. Presser, "Can Changes," 586-593.
22. Traugott and Katosh, "Response Validity," 359-377. Presser and
Traugott, "Little White Lies," 77-86. Silver, Anderson and Abramson,
"The Presence," 228-239.
23. Traugott and Katosh, "Response Validity," 359-377. John P. Katosh
and Michael W. Traugott, "The Consequences of Validated and
Self-Reported Voting Measures," Public Opinion Quarterly 45 (1981):
519-535.
24. Silver, Anderson and Abramson, "The Presence," 228-239. C. R.
Tittle & R. J. Hill, "The Accuracy of Self-Reported Data and Prediction
of Political Activity, Public Opinion Quarterly 31 (1967): 103-106.
Lee
Sigelman, "The Nonvoting Voter in Voting Research," American Journal
of
Political Science 26 (1982): 47-56.
25. Tittle and Hill, "The Accuracy," 104.
26. Presser and Traugott, "Little White Lies," 77-86.
27. Royce Crocker, "Voter Registration and Turnout, 1948-1992,"
Congressional Research Service, Library of Congress (1993): CRS-22.
28. Traugott and Katosh, "Response Validity," 359-377.
29. Sigelman, "The Non-Voting Voter," 47-56.
30. Traugott and Katosh, "Response Validity," 359-377.
31. Silver, Anderson and Abramson, "The Presence," 228-239. Tittle and
Hill, "The Accuracy," 103-106.
32. Sigelman, "The Non-Voting Voter," 53.
33. Ismach, "Polling as," 106-118. Rippey, "Use of," 642-646.
34. Presser and Traugott, "Little White Lies," 78.
TABLE 1
Results of Validation Check of Voting Records
% of all n
respondents
______________________________________________________________________________
Accurate Respondents
Reported did vote, 90.0% 487
Validated did vote
Reported did not vote, 2.4% 13
Validated did not vote
Inaccurate Respondents
Reported did vote, 7.2% 39
Validated did not vote
Reported did not vote, 0.4% 2
Validated did vote
n=541
TABLE 2
Characteristics of Accurate vs. Non-Accurate Respondents
Non-
Accurate Accurate Total n V*
1. Demographic
Characteristic
White 94% 6% 100% 490 .12##
Non-white 84% 16% 100% 70
Age
18-45 90% 10% 100% 460 .11#
45 + 95% 5% 100% 282
Education
< High school 90% 10% 100% 124 .05
Some college 94% 6% 100% 203
College graduate 93% 7% 100% 139
Post-graduate 91% 9% 100% 68
Male 91% 9% 100% 214 .04
Female 93% 7% 100% 325
Protestant 96% 4% 100% 183 .11
Catholic 90% 10% 100% 126
Mormon 97% 3% 100% 37
Other 90% 10% 100% 89
No relig pref. 90% 10% 100% 90
2. Media use characteristics
Days/week watch
evening news on TV
0 days 92% 8% 100% 25 .09
1 93% 7% 100% 27
2 85% 15% 100% 33
3 93% 7% 100% 58
4 90% 10% 100% 52
5 91% 9% 100% 86
6 94% 6% 100% 49
7 94% 6% 100% 206
Attention paid to TV
cvg. of pres. campaign
A great deal 92% 8% 100% 201 .05
Quite a bit 92% 8% 100% 147
Some 94% 6% 100% 116
Very little 91% 9% 100% .43
None 90% 10% 100% .20
Non-
Accurate Accurate Total n V*
Days/week
read newspaper
0 days 95% 5% 100% 42 .12
1 88% 12% 100% 41
2 90% 10% 100% 38
3 86% 14% 100% 42
4 100% 0% 100% 23
5 91% 9% 100% 44
6 90% 10% 100% 29
7 94% 6% 100% 271
Attention paid to newsp.
cvg. of pres. campaign
A great deal 97% 3% 100% 141 .11
Quite a bit 90% 10% 100% 130
Some 90% 10% 100% 134
Very little 92% 8% 100% 67
None 93% 7% 100% 43
3. Political characteristics
Republican 96% 4% 100% 254 .12#
Democrat 90% 10% 100% 231
Other 85% 15% 100% 48
Registered to
vote this year
Yes 86% 14% 100% 154 .15##
No 95% 5% 100% 376
Presidential choice
Bush 96% 4% 100% 199 .12#
Clinton 89% 11% 100% 194
Perot 93% 7% 100% 99
Voting frequency
in past 10 years
All elections 95% 5% 100% 302 .12
Half of election 90% 10% 100% 158
Less than half 89% 11% 100% 45
First time voted 84% 16% 100% 19
Whether it was
person's first time
voting
Yes 90% 10% 100% 29 .03
No 93% 7% 100% 487
Non-
Accurate Accurate Total n V*
How closely
respondent
follows politics
Extremely closely 94% 6% 100% 116 .13
Very closely 95% 6% 100% 206
Somewhat closely 90% 10% 100% 174
Not very close 93% 7% 100% 28
Not closely at all 78% 22% 100% 11
Conservative 94% 6% 100% 201 .09
Moderate 92% 8% 100% 254
Liberal 87% 13% 100% 68
4. Attitudes toward public opinion polling
Influence of polls
no influence 93% 7% 100% 94 .06
some influence 94% 6% 100% 181
a lot of influence 91% 9% 100% 165
a great deal of infl. 96% 4% 100% 68
Are people truthful
in talking to pollsters
never 83% 7% 100% 12 .10
seldom 90% 10% 100% 92
most time 95% 5% 100% 313
all time 89% 11% 100% 56
How accurate are polls
in the media
extremely accurate 93% 7% 100% 14 .05
very accurate 96% 4% 100% 66
generally accurate 92% 8% 100% 271
not very accurate 93% 7% 100% 122
not accurate at all 93% 7% 100% 15
How much do you
follow media polls
frequently 96% 4% 100% 74 .05
often 92% 8% 100% 113
seldom 92% 8% 100% 214
never 93% 7% 100% 118
_______________________
Measure of association expressed by Cramer V
# p < .05
## p < .01
TABLE 3
Strength of Association, Respondent Characteristics Vs. Turnout Accuracy,
Controlling for Education
< High Some College Post
School College Grad Graduate
V p V p V p V p
Race .37 <.001 .03 >.05 .09 >.05 .16 <.001
Age .22 <.05 .08 >.05 .07 >.05 .11 >.05
Party .28 <.01 .09 >.05 .11 >.05 .10 >.05
Register .09 >.05 .19 <.01 .16 >.05 .19 >.05
this year
Presidential .15 >.05 .10 >.05 .12 >.05 .10 >.05
preference
________________________________________
Measure of association expressed by Cramer V.
|
