Content-Type: text/html
Talk the Talk, Walk the Walk: Advancing Measurement in Public Relations
Talk the Talk, Walk the Walk: Advancing Measurement in Public Relations
By
Yuhmiin Chang
Doctoral Student
Fritz Cropp, Ph. D.
Assistant Professor
Glen T. Cameron, Ph. D.
Professor
Contact first author at
1632 Towne Dr. Apt. 3A
Columbia, MO 65202
(573) 886-0756
[log in to unmask]
A manuscript presented to the AEJMC to be considered for possible presentation at the annual convention to be held in Phoenix, AZ, August 9-12, 2000.
Abstract
Lindenmann found that public relations research is talked about much more than it is done. To help assess why this might be hold true, we track the steps in an extensive evaluation of a public relations campaign. Following Grunig's exemplification of the use of focus group research in public relations (1992), this case study offers a guide to the use of a quasi-experimental design with control group for evaluating public relations efforts. Attention is given to challenges to validity and some of the realities the field faces in conducting significant research in public relations practice.
Introduction
Over the past several years, the topic of public relations evaluation has become "hot." A series of reference guides has been produced, scholars have developed conferences and sessions to address evaluative and methodological issues, and a task force responsible for evaluating public relations education has devoted a chapter to the topic of evaluation.
In their Manager's Guide to Excellence and Communication Management, Dozier, L. Grunig and J. Grunig (1995) stress the importance of strategic research to professional, managerial approach. Strategic research, which includes evaluation research, involves "the ability to systematically collect reliable information, organize that information into a manageable form, and share that information with the dominant coalition to make improve strategic decisions (p. 42)." Cameron, Lariscy and Sweep (1992) found that characteristics of individuals, not structural or environmental factors, were predictors of systematic research within organizations. Among the predictors were encouragement by administration, higher education levels, and public relations education. Sweep, Cameron and Lariscy (1994) revealed two constraints on how public relations is practiced. For managers, participation in the decision making process -- which includes information collection and dissemination -- is constrained.
Yet while researchers and practitioners have called for more research, as recently as 1998, Walter Lindenmann, the Director of Research for Ketchum Public Relations Worldwide, noted that "PR measurement and evaluation had been treated as something of a novelty (p. 73)." This is consistent with Lindenmann's earlier position (1990) that practitioners tend to talk about evaluation much more than they do it.
Indeed, although academics and practitioners trumpet the importance of evaluative research, only a few scholars have put forth prospective methods or experimented with them. This paper will examine this limited body of literature. Then, it will provide an exemplar for evaluative research from an extensive two-year study with a complex data design. Finally, it will discuss the techniques and tactics employed in communicating the findings as part of a public relations and integrated strategic communication plan.
A Paucity of Methods Pieces
Scriven (1967) defined evaluation as summative evaluation or determining whether program goals and objectives have been met. Using this definition -- as opposed to smaller measures inherent in any program -- academics have long extolled systematic evaluation to demonstrate public relations effectiveness (Hon, 1998). Key topic areas include the importance of research (Gronstedt, 1997; Hauss, 1993; Lindenmann, 1998); models for conducting research (Baldinger, 1996, Freitag, 1998, Katz & Lendrevie, 1996; Newlin, 1991); and analyses of or information about using measurement techniques (Broom & Dozier, 1983, 1990; Dozier, 1984; Grunig, 1977) and software to facilitate research in public relations (Cameron & Curtin, 1992; Curtin & Cameron, 1992).
However, despite anecdotal emphasis on the importance of evaluative research, only a few methodological studies have been published that suggest, test or experiment with actual techniques. Notable exceptions include L. Grunig's (1992) discussion of using focus group research to help mental health service providers better understand their mentally ill consumers and Fischer's (1995) discussion of control construct design, which combined "a single-group time-series design with a novel way to infer causality (p. 45)." In both cases, the primary advantages were the simplicity of design and the ease with which both the evaluative method and results can be understood by management.
In addition, the Institute for Public Relations Research and Education (IPRRE) produced "Guidelines and Standards for Measuring and Evaluation PR Effectiveness (1997), the International Committee of Public Relations Consultancies Associations (ICO) produced "How to Get Real Value from Public Relations: A Client Guide to Designing Measurable Communications Objectives (1997); and the Association of Media Evaluation Companies (AMEC)(1997) produced "The Power of the Media and How to Measure It: A Client Guide to Media Evaluation." All three of these stem from Nager and Allen's (1984) emphasis on measurable objectives in public relations.
Greater emphasis on developing better measurement instruments can be found in the health communication literature. Chen and Rossi (1983) argued for theoretical models in connection with impact assessment; Dunt, Crowley and Day (1995) called for economic appraisal techniques to assess programs; Cole et al (1995) proposed a systematic planning and evaluation model (SPEM); and Roe (1997) reported a case study for evaluating programs when working with a limited budget. Holman et al (1996) described the use of evaluation forms with more than 1,500 subjects to measure the effectiveness of promotional efforts in communicating health messages.
Lines of Inquiry
This paper involves an analysis of techniques and tactics used to measure the effectiveness of an extensive two-year strategic communications campaign for Kansas Action for Children. Our emphasis is not on the results of survey data collected from 1,637 participants, nor is it an outline of the communications strategies employed. Instead, we hope to provide an exemplar of a model/method for public relations evaluation.
Toward that end, this paper will pursue the following lines of inquiry:
ù Can research be used to measure effectiveness?
ù What are the threats to validity and reliability?
ù Are the costs in time and money too great?
The Campaign for Kansas Action for Children (KAC)
The Kansas Children's Health Report Card proposed by Kansas Action for Children aims to (1) increase the knowledge of Kansas Action for Children (KAC) and its founder, Kansas Health Foundation (KHF), (2) produce accurate perceptions about issues related to Kansas children and teens' well-being, (3) increase people's attention to those issues, and (4) promote actions. Besides Health Report Card, a TV commercial called, "The Number One Question: Is It Good for the Children?" was aired.
Study design
To evaluate the effectiveness of the campaign, we conducted two waves of telephone surveys with an identical questionnaire before and after the campaign respectively. The telephone survey that was conducted before the campaign began was called the Baseline study and the one that was conducted after the campaign was called the Post-campaign Combined study. Each of the studies was composed of two sub-groups. The two sub-groups in the Baseline study were the Panel and Pre-campaign Non-panel groups. The two sub-groups in the Post-campaign Combined study were Post-panel and Post-campaign Non-panel groups. People in the Panel and Post-panel groups were identical. That is, the same people were interviewed twice with the same questionnaire.
Because of the complexity of the study design, we developed a color scheme that facilitated the client's understanding about the structure of this study (See Figure 1). In the color scheme, we identified the two telephone surveys and their associations to time. The boxes with dashed borders represent the pre and post groups in general, and the boxes with solid borders represent the subgroups. The double-arrowhead lines indicate which two groups were involved in either the testing of the test effects or the campaign effects. Darker and lighter colors of the same hue were used to further identify the groups in comparison.
Therefore, the scheme helps visualize (1) how the sub-groups related to the pre and post-campaign surveys (e.g., The Baseline survey included Panel and Pre-campaign Non-panel groups), (2) how the groups related to others (e.g., dark purple for Panel versus light purple for Post-panel groups), and (3) which groups were involved in the later analytical procedures (e.g., Post-panel and Post-campaign Non-panel were compared in the evaluation of the test effect.).
Two advantages were found by using this color scheme. First, the client could get a quick overview about the structure and analytical procedures involved in the study. Second, by putting the color box(es) in the header of each section, the client could refer back to the color scheme and know exactly which study/sub-study or which analysis that section is about. For example, by putting a dark purple box in the header, the client will know that the section talks about the preliminary analysis of the data from the Panel group. By putting a dark and a light purple box in the header of the section that tests the first campaign effect, the client would not only know which step in the color scheme that we were doing but also know that the test of the first campaign effect involved fewer people than the test of the second campaign effect. For a study this complicated, a color scheme lessens the effort to comprehend.
Figure - The color scheme for the Kansas Action for Children study
Testing for Test Effects
Before the campaign effect was assessed, the differences between Post-panel and Post-campaign Non-panel were estimated. If significant differences between the two groups were found, the campaign effects may be contaminated by the first telephone survey, since the people in the Post-panel groups were interviewed twice.
Two steps were involved in the test of test effects. First, it is very important to make sure that people in the Post-panel and Post-campaign Non-panel share similar characteristics because people from different demographic backgrounds are very likely to have different levels of knowledge and hold different attitudes. We compared demographics (i.e., age, employment, years in community, education, race, income and gender) and number of children in the household (i.e., children under 18, number of children under 5, number of children aged 5-9, 10-13 and 14-17 respectively) between the two groups. No significant differences were found. That is, discrepancies of people's responses between these two groups could be viewed as a function of previous exposure to the same questionnaire but not the characteristics per se.
The second step in testing the test effect is to compare people's responses, including attitudes, knowledge, beliefs, and behaviors. Comparing to total number of questions asked, not many test effects were found. If campaign effects found later were confounded with the test effects, we suggested to the client that the particular campaign effects were not conclusive.
Campaign Effects
Two tests for campaign effects were performed for comparison to increase the reliability our conclusions. To maintain our conservative stance, we excluded all of the cases that falsely reported exposures to the state report card or the TV commercial before the campaign began (i.e., increase validity). The first campaign effects were assessed by testing the differences between the Panel and Post-panel groups. By interviewing the same person before and after the campaign, we control for individual differences, affording a relatively more pure measure of a campaign effect. The second campaign effect was tested by comparing the responses between the Baseline and the Post-campaign Combined groups. Even though the individual differences between these two groups existed, the large sample size increases the power of the testing.
Two specific challenges were encountered:
1. How to interpret results that were found in one comparison but not found in the other?
2. How to interpret results that were inconsistent between the two comparisons?
For example, the second test of the campaign effect (i.e., Baseline vs. Post-campaign Combined) showed that people's knowledge about the Kansas Health Foundation had increased significantly after the campaign, but the first test of the campaign effect (i.e., Panel vs. Post-panel) found no significant difference. Also, the first test of campaign effect showed that people believed their neighborhoods had become a worse place to raise children after the campaign, whereas the second test of the campaign effect found the opposite result. Because the first test of the campaign effect involved the same individuals, we suggested to the client that the results from the first analysis were relatively more trustworthy than those from the second one. Nonetheless, to be more conservative, all of the conflicting results should be treated as undermining.
Statistical techniques
Because the first test of campaign effect involved the same group of people, paired-samples T test was used to compare the responses with either interval or ratio scales and marginal homogeneity test was used to compare those with categorical scales. The paired-samples T test procedure compares the means of two variables for a single group. It computes the differences between values of the two variables for each case and tests whether the average differs from 0. The marginal homogeneity test is a nonparametric test for two related ordinal variables and tests for changes in response using the chi-square distribution.
ANOVA and chi-square techniques were used in the tests of the second campaign effect. The analysis of variance test (i.e., ANOVA) compares the means of variables with either interval or ratio scales for two or more groups and chi-square compares the means of variables with either nominal or ordinal scales for two or more groups.
Data Reduction and Inferential Analysis
After evaluating the campaign effects, we aimed to identify the variables that contributed most to predict the effectiveness of our client's five objectives: (1) increase the knowledge of Kansas Action for Children (KAC) and Kansas Health Foundation (KHF), (2) produce accurate perceptions about issues related to Kansas children and teens' well-being, (3) increase people's attention to those issues, and (4) promote actions. Two steps were taken: question reduction and regression analyses.
Question reduction. To reduce the number of questions and to group associated variables into more efficient indicators, several factor analyses with orthogonal rotations were used to construct new indices: an issue attention index, a news attention index, and a news consumption index. The accuracy index was constructed by comparing the survey respondents' reported perceptions with the facts reported in the KAC report card released as part of the campaign. After recording all of the accurate perceptions as "1" and inaccurate perceptions as "0", we averaged the scores and made it the accuracy index. Cronbach's alpha was used to test scale reliabilities if the scale was composed of more than two questions. If the scale was composed of two questions (e.g., media consumption), a Pearson's correlation was used. For our study, reliabilities range from .55 to .88.
Regression Analyses. Two stepwise regression analyses were performed for each of the objectives (i.e., knowledge about KAC, knowledge about KHF, the accuracy index, the issue attention index, and an action question). In the first regression analysis, (1) whether people have children under the age of 18, (2) the news attention index, and (3) the news consumption index were treated as independent variables. In the second regression analysis, we added demographic variables (i.e., age, years in community, income, education, race, and gender) into consideration. We ran five sets of regressions with samples from Post-panel responses, and ran another five sets of regressions with samples from Post-campaign Combined for verification.
Findings. The results were consistent across the two groups. We found that younger people tended to have more knowledge about KHF (the funding agency); people who paid less attention to the news and had higher education held more accurate perceptions; people who had higher news attentions and consumption and who are females tend to spend more time thinking about those issues; and people who paid more attention to news, who had a higher degree, and who are younger spend more time talking about those issues. No single variable significantly contributed to the knowledge of KAC (the campaign group).
Separating the impact of KHF and KAC
Finally, we aimed to distinguish the impact between KAC and KHF. KHF and KAC sometimes distribute very similar messages across the state. It is, however, impossible to use statistical tests to separate and test the distinct effects of KAC and KHF efforts. Thus, we divided people into three groups: exposed to state report card only, exposed to TV commercial only, and exposed to both stimuli. People who did not meet any of these three criteria were excluded. After that, we recoded the issue attention index, the only scale that did not range from zero to one. Originally, the issue attention index was a categorical scale with one equaled to "Hardly at all", two "Sometimes", and three "A lot." We divided the scale by three to make the index ranged from .33 to one to facilitate our comparisons. We then created graphs for each of the groups to examine changes over time. For the group who were exposed to both stimuli, the graph clearly shows that whether people talked about health was directly related to their knowledge about KAC, but not KHF. Similar patterns were found in both the overall samples and panels (see Figure 2).
Even though this is not the most scientific way to separate effects from two related sources, this is undoubtedly a reasonable and plausible alternative to solve the problem. Besides, a graph helped the client to visualize the patterns without digging into the complexity of statistics and involving too many statistical jargons.
Figure - A way to separate the effects of two related institutions
Conclusions
This case study shows the potential to confirm, with qualifications, favorable campaign effects using a rather complex design and a large sample size. The evaluation research here is offered as an opportunity to display a number of features of rigorous research, not to suggest that most campaigns can necessarily justify the time and expense involved. Nevertheless, the fairly complex design with both its strengths and its own set of threats to validity should be useful as the field of public relations finally realizes its managerial potential with an empirical foundation. The challenges in conveying research to the communication professionals conducting the campaign should be particularly useful for those who provide research services to communicators. The relationship management with the party paying the bills may prove useful for faculty teaching students about the research process "in real life" and for professionals emulating the research program offered here.
Bibliography
The Association of Media Evaluation Companies (AMEC). (1997). The Power of the Media and How to Measure It: A Client Guide to Media Evaluation.
Baldinger, A. L. (1996). Integrated Communication and Measurement: The Case for Multiple Measures. In Esther Thorson & Jeri Moore (Eds.), Integrated communication: Synergy of persuasive voices (pp. 271-282). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.
Broom, G. M., & Dozier, D. M. (1983). An overview: Evaluation research in public relations. Public Relations Quarterly, 5-8.
Broom, G. M., & Dozier, D. M. (1990). Using research in public relations: Applications to program management. Prentice-Hall, Englewood Cliffs, NJ.
Cameron, G. T, Weaver-Lariscy, R., & Sweep, D. (1992). Predictors of Systematic Public Relations Research in Higher Education. Journalism Quarterly, 466-470.
Cameron, G. T. & Curtin, P. A. (1992). Reception and use of a prototype expert system in public relations campaigns classes. Journalism Educator, 47(2), 13-18.
Curtin, P. A., & Cameron G. T. (1992). Expert systems and the public relations classroom. Teaching Public Relations, 25, 1-4
Chen, H. T., & Rossi, P. H. (1983). Evaluating with Sense: The Theory-Drive Approach. Evaluation Review, 283-302.
Cole, G. E., Pogostin, C. L., Westover, B. J., Rios, N. M., & Cllier, C. B. (1995). Addressing Problems in Evaluating Health-Relevant Programs Through Systematic Planning and Evaluation. Risk: Health, Safety, & Environment, 38-57.
Dozier, D. M., Grunig, L. A., & Grunig, J. E. (1995). Manager's guide to excellence in public relations and communication management. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Dozier, D. M. (1984). Program evaluation and the roles of practitioners. Public Relations Review, 13-21.
Dunt, D. R., Crowley, S., & Day, N. A. (1995). Is prevention really better than cure: Parameters of the debate and implications for program evaluation design. Health Promotion International, 325-334.
Fischer, R. (1995). Control Construct Design in Evaluating Campaigns. Public Relations Review, 45-58.
Freitag, A. R. (1998). How to measure what we do. Public Relations Quarterly, 42-47.
Grunig, L. A. (1977). Using Focus Group Research in Public Relations. Public Relations Review, 36-49.
Grunig, L. A. (1992). Matching Public Relations Research to the Problem: Conducting a Special Focus Group. Journal of Public Relations Research, 21-43.
Gronstedt, A. (1997). The role of research in public relations strategy and planning. The handbook of strategic public relations and integrated communications (pp. 34-59). New York: McGraw-Hill
Hauss, D. (1993). Measuring The Impact of Public Relations: Electronic Techniques Improve Campaign Evaluation. The Public relations journal, 14.
Holman, C. J., Donovan, R. J., Corti, B., Jalleh, G., Frizzell, S. K., & Carroll, A. M. (1996). Evaluating Projects Funded by the Western Australian Health Promotion Foundation: First Results. Health Promotion International, 75-88.
Hon, L. C. (1998). Demonstrating Effectiveness in Public Relations: Goals, Objectives, and Evaluation. Journal of Public Relations Research, 103-135.
The Institute for Public Relations Research and Education (IPRRE). (1997). Guidelines and Standards for Measuring and Evaluation PR Effectiveness.
The International Committee of Public Relations Consultancies Associations (ICO). (1997). How to Get Real Value from Public Relations: A Client Guide to Designing Measurable Communications Objectives.
Katz, H., & Lendrevie, J. (1996). In Search of the Holy Grail: First Steps in Measuring Total Exposures of an Integrated Communications Program. In Esther Thorson & Jeri Moore (Eds.), Integrated communication: Synergy of persuasive voices (pp.259-270). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.
Lindenmann, W. K. (1998). Only PR outcomes count - That is the real bottom line. Journal of Communication Management, 66-73.
Lindenmann, W. K. (1990). Research, Evaluation and Measurement: A National Perspective. Public Relations Review (pp. 3-16). Silver Spring, Maryland.
Newlin, P. R. (1996). A PR Measurement and Evaluation Model. Public Relations Quarterly, 40.
Roe, K. M. (1997). Evaluation Methodologies with Limited Resources. Paper prepared of Academy for Educational Development.
Sweep, D., Cameron, G.T., & Weaver-Lariscy, R. (1994). Rethinking Constraints on Public Relations Practice. Public Relations Review, 319-331.