Content-Type: text/html Poll reporting Reporting of public opinion polls in American newspapers: The case of the 1998 US senate race Young Jun Son Doctoral student School of Journalism, Indiana University Ernie Pyle Hall Bloomington, IN 47405 [log in to unmask] (812) 857-7754 Rasha Kamhawi Doctoral student School of Journalism, Indiana University Ernie Pyle Hall Bloomington, IN 47405 [log in to unmask] (812) 857-8229 Submitted to Newspaper division AEJMC 2000 Phoenix, Arizona Reporting of public opinion polls in American newspapers: The case of the 1998 US senate race ABSTRACT Published poll results can be misleading if they are not accompanied by methodological information that explains how the results were obtained. This study investigates whether metropolitan daily newspapers provide their readers with sufficient information to evaluate poll stories. Using the guidelines of the American Association for Public Opinion Research (AAPOR) for reporting surveys and polls, a content analysis was conducted. The amount of information disclosed is still far from accurate. National newspapers did worse than state newspapers. Public opinion polls are regarded by scholars, decision-makers and the public as accurate gauges of public opinion. No other method can more concisely represent the basic values and sympathies embraced by a majority of the people at one time. Accuracy in reporting polls is thus very important. Election time is probably the most important time for accurate political information, information which the public may rely on as a basis for their voting decision. The American electorate is bombarded, during these periods, by pre-election polls. But they have not been provided with a mechanism to sift through all these polls and evaluate which ones are accurate and which are deceptive. Mass communication texts are consistently encouraging journalists to report poll results as clearly and accurately as possible. The documentation of certain details about the poll is integral to responsible journalism and essential to a valid interpretation of the meaning of poll results. There have been several criticisms that the reporting of polls is not accurate (Crossen, 1994). A study on the quality of poll reporting is timely now, since journalism schools have increased their emphasis on accurate poll reporting. It would be interesting to see if newspapers have responded to criticism and applied new standards. Election periods have usually been the time frame during which accuracy of poll reporting is investigated. If opinion polls are not reported accurately during these periods, it is very likely that they will not be reported any better during a non-election period. Presidential campaign polls have received considerable attention in academic studies. Political campaign polls for other offices have largely been overlooked (Conway, 1984)[1]. This study aims to look at how other non-presidential political polls are reported in American newspapers. Polling and the Mass Media Polling information has become a newsworthy staple, particularly during election campaigns where poll results become one of the leading categories of news. Pollsters have traditionally treated their methods as trade secrets to keep their competitors from finding out how they produced their results. But there are now plenty of books that explain what pollsters do and how they do their work. Secrecy about the polling method has helped to hide reliable research practices from critics and consumers (Meyer, 1968). Readers were left to judge the validity of polls, without a mechanism to make accurate assessments (Wu and Weaver, 1997). According to Salwen (1985) readers need methodological data to evaluate the accuracy of a poll. Since 1969 the American Association for Public Opinion Research (AAPOR) has provided guidelines that reporters should include in their news reports. By requiring that the polls they report have these standards, news organizations are protecting themselves from deception by interest groups who conduct polls. The guidelines essentially provide a mechanism of evaluating polls into grades. By reporting them, deception in the results, intended or not, could be detected. But so far, providing these standards is not compulsory; they continue to be a do-it-yourself proposition (Meyers, 1968). The eight AAPOR standards are: 1 The sponsor of the poll: The organization that paid for the poll. 2 The sample size: The total number of respondents actually questioned. 3 The population: The exact definition of the population sampled. 4 The polling time: The specific dates when the poll was conducted. 5 The sampling error: The percentage of error at some statistical level of significance. 6 Method of polling: The method employed to question respondents (telephone, mail, in person etc.). 7 Question wording: The wording of the questions on which results were reported 8 Response rate: The actual number of people contacted who accepted to be interviewed. Past Studies Show Different Results The increasing interest in poll reporting, has made news organizations, especially large ones, train research staff to cope with source claims, and raise red flags when suspicious material is encountered (Ismach, 1984). These developments suggest that poll reporting is improving. While earlier studies showed the overall level of conformity to the guidelines set by AAPOR was low (Auh and Willhoit, 1973, Greenberg and Lissance, 1955, Weibe, 1967), recent studies reveal mixed results. Paletz et al. (1980) examined every poll in the New York Times and in two television networks during 1973, 1975, and 1977. They found 67% of polls conformed to AAPOR guidelines. Miller and Hurd (1982) studied three prestigious newspapers over a six-year period. They found conformity was improving for election polls. The overall level was 60%. Salwen (1985) conducted a longitudinal study on opinion polls during the presidential years from 1968-1984. He found that overall there was an improvement in the reporting of methodological information over five elections . The overall disclosure level was 53%. Rollberg et al. (1990) studied poll reporting of the 1988 presidential election in 6 newspapers. They wanted to compare results with Miller & Hurd (1982) and Salwen (1985). The overall conformity rate was 36%, less than both Miller and Hurd (60%) and Salwen (53%). Conformity levels appear to have fluctuated, sometimes being very poor and sometimes approaching satisfactory. It has to be pointed out that the selection criteria of poll stories could be the reason behind these fluctuations[2]. German newspapers fared better in poll reporting quality than American newspapers. German scholar Brettschneider (1997) examined 443 federal election poll articles published in Germany's four leading dailies from 1980-1994. He used the eight AAPOR standards and found that poll reports provided an average of 55% of methodical information. In light of the lack of consistent empirical evidence to explain the precision of American newspapers in reporting election polls, the first research question is asked: RQ1: What is the overall level of disclosure of AAPOR guidelines in American newspapers today? The importance of specific guidelines There are eight guidelines that most studies of accurate poll reporting searched for in poll reports. There has been an imbalance in the frequency of reporting these variables; some are routinely mentioned while others are often neglected, others still have an abysmal reporting rate. Sample size, population, and sponsor were the ones most frequently mentioned. Sampling time and method were next (Rollberg et. al, 1990; Salwen, 1985). Salwen attributed this to the who (population and sponsor), When (date) and how (method) elements of journalism. Technical information such as sampling error was less likely to be included presumably not to confuse readers (Meyer, 1968; Rollberg et al. 1990; Salwen, 1985). Similarly response rate was seldom included because it was thought that readers would not understand it. Question wording was also infrequently mentioned, presumably because it takes up valuable space (Salwen, 1985, Rollberg et al. 1990). This tendency to favor some methodological guidelines over others in reporting polls leads to the second research question. RQ2: Which of the eight AAPOR guidelines are most likely to be mentioned in American state and national newspapers? Schramm (1947) measuring how fast readers drop off throughout a newspaper article found that the story loses readers rapidly in the first few paragraphs. On average a story loses 5% of its readers after the first paragraph, and another 5% after the second paragraph. Then the curve of loss flattens out. This finding is used to measure the importance that American newspapers assign to different disclosure standards in poll reporting. Salwen (1985) reports that population, sponsor, polling time and method were most likely to be reported in the lead. This leads to the second research question. RQ3: What guidelines are more likely to be mentioned in the upper part of the article? The Sponsor effect The proliferation of poll reports since the 1970's is partly due to the increased interest of large news organizations in sponsoring or conducting their own polls. It is no longer the limited domain of a few independent polling organizations. In the 1970s a number of books on how journalists can conduct polls were published (McCombs et al., 1976; Meyer, 1973; Stevens, 1974; Wilhoit and Weaver, 1980). This was the era of precision journalism. The term stems from the title of Philip Meyer's book emphasizing social science research methods for journalists. It was a blueprint for in-house news research (Ismach, 1984). By the late 1970's major news organizations developed their own in-house polling capabilities. In-house polls gave the organization a new found independence (Matusow, 1988) [3]. Editors believe that stories about newspaper polling are well read thus becoming a valuable journalistic newsgathering tool (Rippey, 1980). Those working in the field expect it to grow in use and sophistication (Ismach, 1984). Miller and Hurd (1982) found that newspapers conform more to the guidelines when they reported their own in-house polls, than when they reported polls from the wire service or from syndicated sources. This may be because the newspaper gives space and prominence to its own polls and is less likely to discard the methodological information (Atkin & Gaudino, 1984). Critics however warn that newspapers do not always contract professional polling organizations to conduct polls, but rather carry out the poll themselves making these media research efforts conceptually and methodologically poor. Relying heavily on news staff for the essential elements of a survey may in fact reduce its professional level (Rippey, 1980). From this follows the fourth research question. RQ4: Do newspapers that sponsor their own polls have higher levels of conformity to AAPOR guidelines than those that do not sponsor polls? State Versus National Newspapers and Poll Reports National newspapers generally have more funds and well-trained staff who are aware and presumably more capable of accurate reporting than smaller state newspapers. Moreover, research shows that newspapers with higher circulation are more likely to hire help from professional polling organizations than smaller newspapers (Rippey, 1980). There is thus reason to investigate the performance of state newspapers compared to national newspapers. RQ5: Do national newspapers have higher levels of conformity to AAPOR guidelines than state newspapers? RQ6: Do national newspapers give more prominence to the poll reports than state newspapers? As we enter the new millennium, 30 years after the AAPOR standards have been established, it is important to once again assess the quality of reporting of opinion polls, and compare the results with earlier studies. Method This paper investigates whether large metropolitan dailies provide their readers with sufficient information to evaluate poll stories. The authors wanted to examine the current status of poll reporting and also compare their findings with earlier studies. The authors also wanted to find out if newspapers reported their own in-house polls more thoroughly than other polls and if there is a difference between state and national newspapers in the prominence they give to polls. Two coders content analyzed senate race polls published in the major newspapers in all of the 34 states and two national newspapers. The period under investigation is the last two months before the elections (September 1 - November 2 of 1998). AAPOR standards are used for several reasons. First it makes comparison with previous studies possible. Second, journalism texts regard these standards as important ingredients in precision journalism (Rollberg et al., 1990). Sample A decision had to be made about which newspaper to study from each state. Ideally the newspaper of the state capital should be selected because it is located in the seat of government and would presumably emphasize election polls more than the other newspapers in the state. But in some states like New York, Florida, California, and North Carolina the circulation of the capital's newspaper was far exceeded by other newspapers in the same state. Therefore it was decided to select the newspaper with the highest circulation in every state[4]. If the most prominent newspapers do not adequately report poll results, serious questions about how other papers are reporting polls should be raised. In the case of New York, the New York Daily News was selected. Two other newspapers exceed the Daily News' circulation but they are special cases. The New York Times is considered more a national newspaper rather than a state one[5]. The Wall Street Journal is concerned more with business and economic issues than with local politics. ________________ Insert Tables 1& 2 about here ________________ The Lexis-Nexis database was used to find poll reports. The main circulation newspapers from 19 states were listed under a library and a file[6]. The search term in each of these newspapers was "senate and poll and date (aft 9/1/1998 and bef 11/2/1998)"[7]. This general search term produced many articles. The coders selected from these articles ones that conformed to the definition of poll report used in this study. Only news articles were selected. Columns or editorials and features were excluded. The selection of poll reports to be coded was based on clear standards. There were four criteria: the poll had to be mentioned at least three times, or it was specifically mentioned in the headline or it was central to the main theme of the article or its results were visualized in a graphic. For the poll to be included in the sample it had to meet at least one of these criteria[8]. The number of polls coded from 16 local newspapers and the two national newspapers was 103. Coding Instrument The coding sheet composed of two main sections. The first dealt with poll prominence. It coded the page the poll appeared in, whether the poll was depicted in a graphic, whether the article was accompanied by a photo or photos and whether the poll was specifically mentioned in the headline of the article. The second section coded the poll for the eight variables that made up the construct of 'quality'. These variables are based on AAPOR guidelines. Categories measured (1) the presence of each (yes/no) and (2) where the guideline was mentioned (in the upper part of the article/ the middle part/ the end part/in a graphic). Data collection The two authors of this study carried out the data collection. Being already familiar with the topic at hand and what was being measured, not much coder training was required. Evaluating the coding sheet and writing a codebook was completed after three sessions training. During this period a pretest of the questionnaire was conducted. Reliability using Krippendorf's alpha was 0.90. This resulted in a few changes to the coding sheet. 15 poll articles or 15% of the sample that has been studied were randomly selected and recoded to check for intercoder reliability. Overall coder reliability was .96 (Krippendorf's Alpha). The inter-coder reliability varied somewhat for some of the eight variables: for sponsor, polling time, sampling error, question wording, response rate, and method of contacting respondents, it was 100%, sample size, 88%; and exact definition of population, 87%. Data was collected in one week (not including the newspapers not found on the Lexis-Nexis). Findings There were 103 poll articles, which provided 323 disclosure standards out of a possible 824 standards (eight standards multiplied by 103 poll reports) for a 39 percent overall rate of conformity to the standards. Each poll report provided on average 3.01 of 8 standards. Comparing earlier studies (Paletz et al., 1980, 67 percent; Miller and Hurd, 1982, 60 percent; Salwen, 1985, 53 percent; Rollberg, 1990, 36 percent), we found newspapers' poll reports have not improved over 20 years. The national and state newspapers reported population, sampling error, sponsor, sample size and polling time more often than the other disclosure standards - population, 71.8 percent of the time; sampling error, 58.3 percent; sponsor, 57.3 percent; sample size, 50.5 percent; polling time, 47.6 percent. Other standards were reported in less than 20 percent of the items. Polling method was reported in 16.5 percent of the items, question wording was reported in 12.6 percent, and no poll item included response rate. Forty-six poll reports were analyzed in two national newspapers the New York Times and USA Today. In these two newspapers, sampling error (76.1 percent), population (76.1 percent), and sample size (45.7 percent) appeared more than the other disclosure standards. The lowest disclosure rates were for polling method, 8.7 percent; question wording, 4.3 percent; response rate, 0 percent. On average, 2.8 disclosure standards were contained in each poll reporting. Fifty-seven poll reports were analyzed in 16 state newspapers during 2 months before the 1998 U.S. Senate campaign. Both sponsor (71.9 percent) and population (68.4 percent) were the most highly reported standards. The lowest rates were for polling method, 22.8 percent; question wording, 19.3 percent; response rate, 0 percent. In each poll item, 3.4 disclosure standards appeared on average per poll. The results of the chi-square analyses indicate that the state newspapers exposed more often the poll sponsor (p<.001), polling time (p<.020), question wording (p<.023), and polling method (p<.055) than national newspapers. But national newspapers did a better job reporting the poll sponsor (p<.001). Table 3 about here A crosstabulation also shows that 33.3 percent of state newspapers' poll reporting disclosed more than 5 AAPOR standards, while 15.2 percent of national newspapers' poll involved more than 5 standards (p<.057). Table 4 about here State newspapers sponsored polls more often than national newspaper. The state newspapers sponsored 59% of campaign polls that were reported in their newspapers, while national newspaper paid for 28% of the polls they reported. In state newspapers and in national newspapers, disclosure standards were more frequently mentioned in self-sponsored campaign polls than in non-sponsored polls. They were certainly applying 'double rules' in poll reporting. When newspapers quoted poll results sponsored by other organizations, they generally disclosed fewer guidelines. In self- sponsored polls, state newspapers' disclosure rates of polling method (chi-square test, p<.000), sample size (chi-square test, p<.013), and sampling error (chi-square test, p<.047) were increased at least more than 20 percentage points respectively, and the other standards' rate were also enhanced around 10 percentage points. Even though the sample size is small, the national newspapers showed a similar pattern. Table 5 about here The type(s) of each AAPOR poll standard was also examined. The types of poll sponsors were highly different. State newspapers sponsored more polls than national newspapers, while national newspapers generally quoted other media's poll results (T-test, p<.013). The reported sampling error was crowded between � 3.01 and � 4.00 category (T-test, p<.056). Both national and state newspapers also quoted poll results of which sample sizes were under 500 likely voters. Between national and state newspapers, there was no significant difference in sample size (T-test, p<.830). The mean of sample size of national papers was 787 persons, while that of state newspapers was 725 persons. Table 6 about here We also studied the prominence of poll reporting. First, state newspapers paid more attention to poll reporting than national newspaper. State newspapers' poll items were located more often in the first page of each section than those of national newspapers. Weight was put to each location of poll reporting. For example, 1 was allotted to the cover page of the main section, 2 was allotted to the cover page of the main section and completed in inside page, 3 was allotted to the cover page of other section, 4 was allotted to the cover page of other sections and completed in an inside page, and 5 was allotted to items just appearing in inside page. The mean for state papers (N=57) was 3.28, while the mean for national papers (N=46) was 4.80 (T-test analysis, p<.001). Additionally, state newspapers' polls were mentioned more often in the headline (47 percent), while 10 percent of national newspapers' polls were referred in the headline (chi-square analysis, p<.001). Second, the national paper's poll items were more often accompanied by graphics and/or photos than state newspaper articles. Each graphic or photo, accompanied with the senate race poll, was counted and allotted numbers: no graphic or no photography 1, one graphic or photo 2, two graphics or photos 3, and more than three graphic or photos. The mean for national papers was 2.15 ; the mean for state papers was 1.63 ( T- test analysis, p<.016). Table 7 about here Placement of exposure standards was different. Sponsor, sample size and population appeared more in the upper part of each poll article, while polling time, sampling error and polling method were mentioned more frequently in the middle or end part. About 71 percent of poll sponsors appeared in the upper 1/3 part of each article, while that of polling method was 23. To see whether national and state newspapers differ in how they reported the disclosure standards, we also employed t-test analyses. Results show that the placement differences between the two kinds of papers are statistically insignificant, while the population and sponsor variables were approaching statistical significance. Table 8 about here Discussion The robustness of democracy depends on far more than public opinion polls. Nevertheless, a creative use of polling that conforms to what is methodologically valid can make a positive contribution to our democratic way of life. It's because no other method, in a democratic society, can more concisely represent the basic values and sympathies embraced by a majority of the people than polls. Our findings show that Turner and Martin's (1984) conclusion that "existing standards for the conduct and reporting of surveys appear to have limited impact on survey practice" still appears to be valid. The overall poor disclosure of guidelines might be explained by journalistic considerations; (1) statistics, (2) budgetary, space, and time restriction, and (3) the need to be timely (Crespi, 1989). To compensate for the limited information available from the typical news reports on polls, some polls (such as CBS/ New York Times, ABC/ Washington Post, and the Los Angeles Times) distribute tabulations of poll results and written commentary to selected mailing lists (Crespi, 1989). Informative as these sources may be, they are accessible only to a limited audience. The general public has little opportunity to become aware of, let alone have access to, these sources. One of our findings, that state and national newspapers are adopting 'double rules'; the exposures of the AAPOR standards differ remarkably between self-sponsored and non-self-sponsored polls, suggests that Crespi's explanations are not enough. We think two more explanations are needed. First is reliability. When the poll is funded or conducted by the news organization itself, it can put more reliability on the poll results because it can obtain all the information about the poll. Second is economic consideration. While poll conducting is costly, campaign poll reporting may attract more public attention. Because the paper itself funded for the poll, the news organization may want to recover its cost with more space and time to poll reporting. By attracting more public attention, the news organization may survive in the competitive market situation. Another finding is that national papers did not do very well in reporting polls. They did worse than state papers. Leading state newspapers were more honest to readers in mentioning disclosure standards. Even though the national newspapers, by virtue of their being 'national' in focus and scope, could not pay much attention to each senatorial campaign, they should have been more accurate when they reported the poll results of senatorial races. It's because any newspaper' poll reports, regardless of its scope, may be equally accepted to concerned voters. We found good examples in one national paper. When the New York Times reported its in-house polls, it detailed how the sample was chosen, what sampling error meant, and even warned that question wording could make biased results. It made comparatively higher levels of disclosure than USA Today. However, when the New York Times reported other media sponsored polls, the level of disclosure became much lower than its in-house polls. Comparing with the latest research of Rollberg and his colleagues' (1990), this study shows that the disclosed rate of sampling error in newspapers' poll reporting has considerably risen from 10 percent to 58 percent in 10 years. This could be explained by the influence of 'precision journalism.' The total absence of response rate (none in 103 poll items) may be the result of a belief by journalists that readers do not care or do not understand this methodological information (Crespi, 1989). Yet, educated readers might be frustrated by the lack of specific methodological information in news stories. More importantly, it is misleading to withhold the response rate from readers. Perhaps pollsters want to avoid mistrust of poll findings due to low response rates. But if the response rate was worse than expected, they should have paid more attention to response rate.[9] To enhance the level of response rate, some polling organizations, such as Harris, are adopting a three-time call policy. If family members are absent, they try to call at least three times in three days until connected. It's because lower response rate means the continuous addition of samples in the process of polling, even if they choose the method of random sampling again. This practice is certainly questionable in guaranteeing higher level of internal validity. In some cases, sample size was under 300 respondents, and sampling error was over � 7.00. Small sampled or inaccurate polls might not only receive little value as a pre-election campaign poll but also mislead newspaper readers. For better or for worse, polls have become an integral part of contemporary politics and it is highly unlikely that the prominence of polls in the media will decline in the foreseeable future. To the contrary, polls are expanding rather than contracting in political life. But poor poll reporting may mislead readers, and incomplete reporting may let voters have some suspicion of the media. That's the reason why newspapers should have a responsibility to inform their readers better, not to mislead people. References Atkin, C. K. and Gaudino, J. (1984). The impact of polling on the mass media. In L. John Martin (Ed.), The Annals of the American Academy of Political and Social Sciences, 462: 119-128. Auh, T. S. and Wilhoit, C. G. (1973). "Metropolitan daily newspaper treatment of political polls during the 1970 elections" paper presented to the Association for Education in Journalism, Fort Collins, Colorado. From Salwen, M. B. (1985). The reporting of public opinion polls during presidential years, 1968-1984. Journalism Quarterly, 62, 272-277. Bretschneider, Frank. (1997). The Press and the Poll in Germany. International Journal of Public Opinion Research, 9, 248-265. Crespi, Irving. (1989). Public Opinion, Polls, and Democracy. San Francisco: Westview Press. Conway, M. M.(1984). The use of polls in congressional, state and local elections. In L. John Martin (Ed.), The Annals of the American Academy of Political and Social Sciences, 462: 97-105. Crossen, C. (1994). Tainted truth: The manipulation of fact in America. New York: Simon and Schuster. Greenberg, A. and Lissance, D. (1955). The accuracy of a journalistic poll. Public Opinion Quarterly, 19, 45-52. Ismach, A. H. (1984). Polling as a news-gathering tool. In L. John Martin (Ed.), The Annals of the American Academy of Political and Social Science, 472, 106-118. Matusow, B. (1988). Are the polls out of control? Washington Journalism Review, October, 17-20. McCombs, M. E., Shaw, D. L. and Grey, David. (1976). Handbook of reporting methods. Boston: Houghton-Mifflin. Meyer, P. (1968). Truth in polling. Columbia Journalism Review, summer, 20-23. Meyer, P. (1973). Precision journalism: A reporter's introduction to social science methods. Bloomington: Indian University Press. Miller, M. M., and Hurd, R.(1982). Conformity to AAPOR standards in newspaper reporting of public opinion polls. Public Opinion Quarterly, 46, 243-249. Paletz, David. L, Jonathan Y. Short, Helen Baker, Barbara C. Campell, Richard J. Cooper and Rochelle M. Oeslander. (1985). Polls in the Media: Content, Credibillty, and Consequences. Public Opinion Quarterly, 44, 495-505. Rippey, J. N. (1980) Use of polls as a reporting tool. Journalism Quarterly, 57, 642-656, 721. Rollberg, Jeanne N., Luther W. Sanders and M.D.Buffalo. (1990). Down to the Wire. Newspaper Research Journal, 11, 80-93. Salwen, M. B. (1985). The reporting of public opinion polls during presidential years, 1968-1984. Journalism Quarterly, 62, 272-277. Schramm, W. (1947). Measuring another dimension of newspaper reading. Journalism Quarterly, 24(4), 293-306. Turner, C. F. and Martin, E. (Eds.) (1984). Surveying Subjective Phenomena (Vol. 1). New York: Russell Sage Foundation. Wiebe,G.D. (1967). The New York Times and public opinion research. Journalism Quarterly, 44: 654-658. Willhoit, G.C. and Weaver, D. H. (1980). Newsroom guide to polls and surveys. Bloomington, IN: Indiana University Press. Wu, W. and Weaver, D. (1997). On-line democracy. Harvard International Journal of Press/Politics, 2(4), 71-86. Appendix 1 Tables Table 1: State Newspapers in the study STATE NEWSPAPER CIRCULATION* Arizona The Arizona Republic 437,118 Arkansas Arkansas Democrat Gazette 170,766 California Los Angeles Times 1,050,176 Colorado The Denver Post 337,372 Connecticut The Hartford Courant 210,800 Florida St. Petersburg Times 342,189 Idaho The Idaho Statesman 63,908 Indiana The Indianapolis Star 224,372 Iowa The Des Moines Register 164,912 Louisiana The Times Picayune 260,552 Maryland The Sun 312,826 New Hampshire The Union Leader 65,758 New York New York Daily News 721,256 Ohio The Plain Dealer 383,586 Utah The Salt Lake Tribune 129,836 Washington The Seattle Times 227,162 Wisconsin Milwaukee Journal Sentinel 288,173 * The circulation was taken from Editors & International Yearbook 1998. Table 2: National newspapers in the study NATIONAL NEWSPAPERS CIRCULATION New York Times 1,074,741 USA Today 1,653,428 Table 3: Percentage of newspaper stories which contains each AAPOR exposure standards* Standard Overall National paper State paper N=103 N=46 N=57 p<** Sample size 50.5 % 45.7 % 54.4 .378 Sponsor 57.3 39.1 71.9 .001 Question wording 12.6 4.3 19.3 .023 Sampling error 58.3 76.1 43.9 .001 Population 71.8 76.1 68.4 .390 Polling method 16.5 8.7 22.8 .055 Polling time 47.6 34.8 57.9 .020 Response rate 0 0 0 - Average 39.2 35.6 42.3 1 *Whitney includes polling organization as one of disclosure standards. In this study, 67.0 percent of all items of mentioned polling organization. (national newspaper 69.6 percent; state newspaper 64.9 percent) ** Chi-square analysis Table 4: Percentage of stories conforming to AAPOR standard between national and state newspapers Number of disclosed standards 0 1 2 3 4 5 6 7 8 National paper 10.9 % 15.2 21.7 8.7 28.3 8.7 2.2 4.3 0 State paper 5.3 % 19.3 19.3 14.0 8.8 7.0 15.8 10.5 0 *Chi-square analysis, p<.057 Table 5: Percentage of poll stories that contain each exposure standard when newspaper itself sponsored the poll (*T-test analysis significance level) National newspaper State newspaper Standard N= 5 N=24 p<* Sample size 80.0 % 79.2 % .013 Question wording 40.0 29.2 na Sampling error 100.0 66.7 .047 Population 100.0 83.3 .081 Polling method 60.0 54.2 .000 Polling time 80.0 66.7 .375 Table 6: Standard type in poll reports (% of reports in each standard mentioned poll reports). *T-test analysis significance level National paper State paper National paper State paper Sponsor(N=55, p<.013*) Sample size (N=52, p<830*) Newspaper itself 9.1% 43.6% Under 500 3.8% 11.5 % Other media 23.6 21.8 501 - 750 9.6 11.5 News agency - 1.8 751- 1,000 15.4 19.2 1,001- 1,250 3.8 7.7 1,251- 1,500 3.8 7.7 Over 1,501 3.8 1.9 Table 7: Location of each exposure standards in each poll reporting (cumulative percentage) Sample size Method Time Population Sampling error Sponsor Upper 1/3 part 55.8% 23.5% 36.7% 54.2 33.3 % 71.2 Upper 2/3 par 80.8 44.1 57.1 77.0 63.3 84.6 Upper 3/3 part 100.0 100.0 100.0 100.0 100.0 100.0 Table 8: The average placement of disclosure standards in each item. Sample size Method Time Population Sampling error Sponsor National paper 1.81 2.00 1.50 1.92 1.94 1.77 State paper 1.51 2.36 1.97 1.57 2.12 1.41 p<* .193 .501 .110 .086 .418 .089 *T-test analysis significance level Appendix II Coding Sheet Coding sheet : The Poll Coder sheet number _ _ _ Coder _ _ Date _ _ _ _ Newspaper Number _ _ Item number _ _ Poll number_ 1. Page that senate race opinion poll appeared in 1.Front page only 4.Cover page inside and completed inside 7.Other Specify ________ 2.Front page and completed inside 5.Inside page 3.Cover page inside section only 6.Unknown or undetermined 2. What's the style of senate race opinion poll reporting? 1.Article only 4. Article with two graphic 7.Other Specify __________________ 2.Graphic only 5.Article with three or more graphic 3.Article with one graphic 6.Unknown 3. How many photos were included? 1.None 3.Two 5.Unknown or undetermined 2.One 4.Three or more 6.Other Specify __________ 4. Was the poll mentioned in the headline? 1.Yes 2.No 5. How many senate race polls were mentioned in the article? 1.One 4.Four 7.Seven and more 2.Two 5.Five 8.Other Specify _________________ 3.Three 6.Six 6.Was the polling organization mentioned? 1.yes 2.no 7.If yes, what was the polling organization? 1.commercial 3.university 2.newspaper itself 4.other specify ____________ 8. Was the sponsor mentioned? 1-yes 2-no 9. Who was the sponsor? 0- Does not apply 1-By the newspaper itself 2-By other newspaper(s) or broadcasting company(s) 3-By syndication (news agency wire service or other syndication) 4-by cooperation (the newspaper and another org.) 5-University 6-Other specify __________ 10.Where did the sponsor appear in the article? 0. It does not apply 3.Middle part of the Article 6.Other Specify _______________ 1.Headline 4.End part of the Article 2.Upper part of the Article 5.Graphic or Photo 11. Was the sample size mentioned in detail? 1-yes 2-no 12. How was the sample size? 0-Does not apply 3.751-1,000 6.Over 1,500 1.Under 500 4.1,001-1,250 2. 500-750 5.1,251-1,500 13. Where did the sample size appear in the article? 0. It does not apply 3.Middle part of the Article 6.Other Specify _______________ 1.Headline 4.End part of the Article 2.Upper part of the Article 5.Graphic or Photo 14.Was the population mentioned? 1-yes 2-no 15. What was the population? 0 Does not apply 2.Likely voters 4.Newspaper readers 1.Registered voters 3.Over 18 year old adults 5.Other Specify ___________ 16.Where did the definition of the population appear in the article? 0. It does not apply 3.Middle part of the Article 6.Other Specify _______________ 1.Headline 4.End part of the Article 2.Upper part of the Article 5.Graphic or Photo 17.Was the polling time mentioned? 1-mentioned in detail 2-mentioned partly 3-was not mentioned 18. Where was the polling time mentioned in the article? 0. It does not apply 3.Middle part of the Article 6.Other Specify _______________ 1.Headline 4.End part of the Article 2.Upper part of the Article 5.Graphic or Photo 19. Was the sampling error mentioned? 1-yes 2-no 20. How was the sampling error? 0 Does not apply 3.�2.01% - �3.00% 6.Over �5.01% 1.Under �1.0% 4.�3.01% - �4.00% 2.�1.01% - �2.00% 5.�4.01% - �5.00% 21. Where did the sampling error appear in the article? 0. It does not apply 3.Middle part of the Article 6.Other Specify _______________ 1.Headline 4.End part of the Article 2.Upper part of the Article 5.Graphic or Photo 22. Was the Response rate mentioned? 1-yes 2-no 23. How was the overall response rate? 0 Does not apply 3..31%-40% 6.61%-70% 1.Under 20% 4.41%-50% 7.Over 71% 2.21%-30% 5.51%-60% 24. Where was the response rate mentioned in the article? 0. It does not apply 3.Middle part of the Article 6.Other Specify _______________ 1.Headline 4.End part of the Article 2.Upper part of the Article 5.Graphic or Photo 25.Was the method of polling mentioned? 1-yes 2-no 26.What was the method of polling? 0 Does not apply 2.Mail 4.On-line web 1.Telephone 3.Face-to-face interview 5 Other Specify ______________ 27.Where was the method of polling mentioned in the article? 0. It does not apply 3.Middle part of the Article 6.Other Specify _______________ 1.Headline 4.End part of the Article 2.Upper part of the Article 5.Graphic or Photo 28.Was the wording of the main question included in the article? 1.Yes 2.No 29.How was the wording? 0 Does not apply 2.Only keywords of the question appeared 1.Complete wording appeared 3.Other Specify ___________ Endnotes [1] Auh and Wilhoit studied congressional elections (1973). [2] Miller and Hurd examined prestigious newspapers like Los Angeles Times, Chicago Tribune and Atlanta Constitution. The reporting of methodological information was expected to be considerably better than in smaller newspapers. Salwen's study also has two drawbacks. First of all he studied only newspapers from one state, so generalizing from these papers to all newspapers in the United States is questioned. The author himself says that the newspapers studied may be atypical of newspapers in the Unites States (274). Second, his selection criteria were very strict that it could have tilted the balance in favor of accurate reporting. Only poll stories that appeared on the front section of newspapers and that referred to the poll in their headlines were included. [3] At first newspapers were limited in the information they got from campaign politicians. They had to accept the information as is, even though it could be wrong. It also allowed journalists to expand their horizons beyond the confines of available information. Questions that went unanswered because no one had collected the information can now become answered. [4] The circulation of newspapers was taken from Editor & Publisher International Yearbook 1998. [5] The New York Times for example covers out of state election campaigns more than other state newspapers. [6] The Lexis-Nexis version employed in this study was at the Indiana University at Bloomington library [7] Lexis Nexis does not show graphics. But at the end of the article there is information on what the graphic was. If it was a chart the information on that chart is also given. Half of the newspapers were not on Indiana University's Lexis Nexis account. They were searched on microfilm [8] These standards may not be as strict as the ones used by other researchers to select poll reports. We believe that if a poll report meets any of these criteria it should apply AAPOR guidelines. [9] Generally, the response rate in telephone survey is under 50 percent. This means that at least half of most pre-assigned sample residents are not contacted. Some households, which are selected as samples, may absent or do not want to be interviewed. If response rate is lower than expected, more samples are added with carrying out interviewing.