Demonstrating Effectiveness in Public Relations:
Goals, Objectives, and Evaluation
Abstract
Public relations planning and evaluation were explored among 32 practitioners
and 10 top executives. Practitioners said their goals reflect the priorities of
their institution. The CEOs believed public relations' ultimate aim is
communicating the image of the organization. Responses showed many
practitioners conduct informal evaluation while only a few conduct formal
evaluation. This research suggests public relations planning and evaluation are
becoming more systematic but are still constrained by lack of resources and
difficulty.
Demonstrating Effectiveness in Public Relations:
Goals, Objectives, and Evaluation
Linda Childers Hon
Assistant Professor
Department of Public Relations
College of Journalism and Communications
University of Florida
Gainesville, FL 32611
352-392-6522
[log in to unmask]
Paper submitted to the Public Relations Division
Association for Education in Journalism and Mass Communication
April 1997
Demonstrating Effectiveness in Public Relations:
Goals, Objectives, and Evaluation
Evaluating the effectiveness of public relations continues to be a topic of
critical importance to practitioners and scholars. As pressures for
accountability mount, practitioners increasingly must demonstrate that public
relations activities help achieve meaningful goals for their organization or
clients (Johnson, 1994; Kirban, 1983; "Measurement driving more PR programs,"
1996).
Academics have long extolled the link between demonstrating public relations
effectiveness and evaluation.[1] And, as a result, much attention has been
devoted to analyzing public relations measurement techniques (see Broom &
Center, 1983; Broom & Dozier, 1983; Broom & Dozier, 1990; Cline, 1984; Dozier,
1984, 1985; J. Grunig, 1977a, 1983; J. Grunig & Hickson, 1983; Suchman, 1967;
Wright, 1979).
Although definitions of effectiveness in public relations abound (Hon, 1997),
the most predominant theme in scholarly literature and the trade press is that
effective public relations occurs when communication activities achieve
communication goals (in a cost-efficient manner). Implicit in this model is
that public relations goals are derived from the overall organizational mission,
goals, and objectives. Thus, through public relations evaluation, practitioners
can demonstrate either directly or indirectly public relations' role in
organizational goal achievement.
Given the pervasiveness of this model, it is ironic that so little research has
been conducted on goal setting and evaluation from practitioners' points of
view. The scant bit of research about goal setting in public relations is
almost entirely prescriptive--emphasizing what practitioners should be doing
rather than illuminating what they really are doing and why. Research on public
relations evaluation is much more voluminous than that on goal setting, and it
is both descriptive and prescriptive. Yet, this body of work too could be
enriched by showcasing practitioners' views of the complex realities they face
when trying to evaluate the effectiveness of their public relations efforts.
Demonstrating Effectiveness
Purpose and Significance of the Study
The purpose of this research then is to explore issues surrounding goals,
objectives, and evaluation in public relations. The ultimate intent is to
develop a better understanding of the challenges and opportunities practitioners
face in demonstrating the value of public relations.
The significance of this increased understanding is great. Few scholars or
practitioners would dispute that the long-term viability of the public relations
function is directly linked to a better understanding of public relations'
value-added contribution (see Campbell, 1993; Geduldig, 1986; J. Grunig, L.
Grunig, Dozier, Ehling, Repper, & White, 1991; J. Grunig, Dozier, Ehling, L.
Grunig, Repper, & White, 1992; Macnamara, 1992; "Measurement popular in U.S.,"
1995; Pritchitt, 1994; "Setting benchmarks leads," 1993; Strenski, 1981, 1983;
Weiner, 1995; Winokur & Kinkead, 1993). Effectiveness measures that have
sufficed in the past (e.g., volume of clippings) often are no longer adequate
(Cramp, 1994; Hause, 1993; Houlder, 1994; Sutton, 1988; Wylie & Slovacek, 1984).
In other words, public relations output increasingly must be tied to meaningful
outcomes for organizations and clients (Holmes, 1996). A critical step in
making this connection is setting appropriate goals and measurable objectives.
Research on Public Relations Goals and Objectives
The assumption found in public relations literature is that goals are general
guidelines or a framework for the public relations department while objectives
are the specific outcomes desired from programs (Grunig and Hunt, 1984).
Explaining the distinction further, Kendall (1996) offered the following
definitions of goals and objectives, derived from the Accreditation Board of the
Public Relations Society of America: A goal is "often related to one specific
aspect of the mission or purpose, and commonly described as the desired outcome
of a plan of action designed to solve a specific problem over the life of a
campaign" (p. 248). An objective, on the other hand, "addresses a specific
aspect of the problem with each of several objectives contributing toward
achieving the goal" (p. 248). Objectives specify the communication outcome
desired, the targeted public, the expected level of attainment, and the time
frame in which the outcome is expected to occur.
Other scholars have offered detailed definitions of objectives as well. For
example, Swinehart (1979) emphasized that public relations objectives should
specify content, target population, when the intended change should occur,
whether the changes are unitary or multiple, and how much of an effect is
desired (see also Broom and Dozier, 1990; Dozier and Ehling, 1992).
Research on public relations planning shows the impact that MBO (management by
objectives) has had on the communication function (Cutlip, Center, & Broom,
1985; Dozier & Ehling, 1992; Finn, 1984; Koestler, 1977; Nager, 1984; VanLeuven,
1989; VanLeuven, O'Keefe, & Salmon, 1988). This scholarship typically has
stressed the importance of public relations goals and objectives that relate to
organizational goals. As Heitpas (1984) argued, public relations objectives
should stem from organizational objectives, and they should be oriented toward
improvement and clearly defined.
Few studies, though, have explored how well these ideal models for setting
public relations goals and objectives hold up in everyday practice. One
exception is Barlow's (1993) analysis based on interviews with 31 corporate
executives and 16 academicians, a project conducted for the Institute of Public
Relations Research and Education. He found that most interviewees thought goal
and objective setting in public relations is actually pretty uncomplicated--much
more so than evaluation.
However, several respondents disagreed. "I don't worry so much about
evaluation," one executive said. "I worry more about setting [public relations]
goals that are unrealistic, or that are not directly related to our market-place
objectives" (p. 15).
Another executive echoed this sentiment:
The culture of the company heavily influences this (setting goals and
objectives).
Ours is a results-oriented management, not yet customer-driven. Defining
the results
they regard as meaningful is extremely difficult, when communications is
only a factor.
What hits management directly is what counts with them, and trying to
fathom that in
terms of realistic goal setting poses real problems. (p. 15)
Whatever their frustrations, though, most practitioners in this study
acknowledged their companies' process for setting public relations goals and
objectives has become more formalized as the demand for measurable results has
increased. Obviously, the systemization of public relations planning is a
necessary precursor to meaningful public relations evaluation.
Research on Public Relations Evaluation
Research on public relations evaluation can be divided roughly into several
main categories. Of course, some studies have had multiple objectives and fall
into than one category. But, for the purpose of explication, multipurpose
studies can be grouped here according to what seems to be their main purpose.
The following is by no means an exhaustive review of the literature relevant to
public relations evaluation. The goal of this synthesis merely is to provide
some examples of the major topics that have been explored (see also Burnstead,
1983; Chapman, 1990; Jacobson, 1980; Lorimer, 1994; Marzano, 1988; Ruff, 1986;
Tucker & Shortridge, 1994; Wiesendanger, 1994; Williamson, 1995).
Analyses of Communication Effects
One category of research relevant to public relations evaluation explains how
communication effects happen and how threats to internal and external validity
limit causal inferences (Campbell and Stanley, 1963; Flay & Cook, 1981; McGuire,
1973; Mendelsohn, 1962, 1973; O'Keefe & Reid, 1990; Rogers & Storey, 1987; Webb
& Campbell, 1973). In other words, linking the cause--public relations
activities--to some effect--generally assumed to be cognitive, attitudinal, or
behavioral--is neither easy nor straightforward.
Yet, Reeves (1983) argued that meaningful evaluation of communication effects
goes beyond issues of research design. Evaluation must include evidence along
several dimensions. That is, effective evaluation must answer all of the
following questions: "What's having an effect (content)?; Who's affected
(exposure)?; What changes are desired (effect)?; And, when and how does the
effect occur (conditional processes)?
Dozier and Ehling (1992) provided an extensive review of the literature about
communication effects and its implications for public relations evaluation.
They discussed how many communication programs have assumed the domino model,
which implies that communicated messages necessarily will cause changes in
knowledge, attitude, and behavior (in that order) among target publics.
Pointing out fallacies in this model, they determined the likelihood of
achieving behavior change with any member of a target public is only about .04
percent. They explained that, as program objectives move from message exposure
to behavioral change, the likelihood of achieving those objectives decreases
dramatically.
Prescriptive Research
The next category entails scholarship whose main goal is to outline the
evaluation process and point out the importance of evaluation for public
relations.[2] In Caro (1977), for example, various types of evaluation were
described and several case studies were offered. The authors in this collection
of readings argued strongly for the inclusion of evaluation in all social
programming.
Weiss' (1972) work also made a strong case for evaluation. But, she discussed
numerous constraints such as organizational resistance to change and the
tendency of evaluation to show little or no effect. She also mentioned that
practitioners too often attempt to evaluate programs for which there is no clear
statement of purpose. Or, if objectives are concrete, they are stated as
communication activities rather than program effects.
The importance of public relations evaluation that measures impact rather than
process was reiterated by Broom and Dozier (1983). They asserted that achieving
specific outcomes detailed in public relations program objectives is the
yardstick by which success must be measured. For them, focusing on output or
viewing public relations outcomes as unmeasurable inhibits the development of
public relations as a management function.
These themes were emphasized again by Dozier (1985), who contrasted pseudo
planning and pseudo evaluation with true planning and evaluation in public
relations. He discussed the tendency to confuse communication products with
desired ends. In other words, organizational resources are devoted to measuring
output such as number of news release placements and publications rather than
specifying desired effects and measuring whether these ends were achieved.
Another example of prescriptive research was provided by O'Neill (1984), who
explained the importance of measuring public opinion as part of public relations
evaluation. He argued that survey research on corporate reputation is vital
given that organizations increasingly are expected to be more responsive to
publics. Thus, more public relations research is needed to provide
practitioners with information on how publics perceive an organization and how
perceptions change.
Similarly, Stamm (1977) warned against collecting evaluation data on the wrong
variables or collecting data at the wrong time. For him, meaningful evaluation
is contingent upon practitioners' situating evaluation in the proper cognitive
context. Practitioners, in conjunction with clients or management, must first
determine the when, where, how, and what of evaluation if the process is to be
useful (see also Atkin, 1981).
Broom and Dozier's (1990) textbook, Using Research in Public Relations:
Applications to Program Management, is the most comprehensive prescription for
public relations research and evaluation. In this text, Broom and Dozier
discussed how research fits into public relations management, arguing that
evaluation is a necessary component of sophisticated public relations practice.
That is, without evaluation data, practitioners cannot plan and manage
communication programs most effectively.
Case Studies of Public Relations Evaluation
Another category of evaluation research was exemplified by a 1977 special issue
of Public Relations Review, which featured examples of public relations research
in a variety of institutional settings. For example, Grass (1977) discussed the
evaluation of DuPont's corporate advertising. Starting in the 1930s, DuPont
began tracking changes in public attitudes toward DuPont and other companies.
DuPont found that over time, its corporate image advertising was having the
desired effect, especially its television advertising. People who had been
exposed to the campaign demonstrated an overall shift toward favorable
attitudes, especially in areas addressed by the campaign. Yet, despite this
success, Grass concluded that DuPont was unable to determine the bottom-line
value of DuPont's significant investment in the campaign.
J. Grunig (1977b) studied the effectiveness of internal communication at the
National Bureau of Standards (now National Institute of Standards and
Technology). The conceptual framework used was his situational theory of
publics (see J. Grunig and Hunt, 1984). This theory predicts communication
behavior based on communicators' recognition of an issue or problem, their
involvement in the issue or problem, and their constraint recognition, or the
degree to which they feel unable to do something about the issue or problem.
J. Grunig's study suggested that communication effectiveness is maximized by
directing communication resources to those publics most likely to need and want
information. He uncovered four different employee publics and two publications
used most frequently among them. All publics expressed a desire for more
information about administration, a topic not covered in any publication.
Reviewing J. Grunig's work, Franzen (1977) argued that Grunig's situational
theory provides a meaningful framework for evaluating communication
effectiveness.
Another case study of public relations evaluation was offered by Tirone (1977),
who discussed the Bell System's attempt to develop measures that would assess
the overall performance of public relations. To measure the efficiency of
employee communications, Bell gauged how effectively a publication was
distributed; the company also measured readability, and awareness and
understanding among readers. Assessing the effectiveness of media relations,
Bell conducted a national study to determine the media's evaluation of the
company's materials. The company also content analyzed news stories. However,
measuring the effectiveness of Bell System's advertising about long distance
services proved to be difficult. So did linking the company's community and
educational relations to effectiveness in public relations. With community
relations, it was hard to isolate public relations' contribution since so many
people were involved (see also Tichenor, Donohue, & Olien, 1977). The problem
with educational relations was a lack of clear objectives against which to
measure performance. Tirone concluded that, although some component evaluation
measures had been developed at Bell, a total system for capturing the value of
public relations was elusive.
In the same issue of the Review, Marker (1977) discussed how Armstrong Cork
Company developed a systematic program for evaluating the value of its product
publicity. He began by explaining that the success of his public relations
efforts used to be judged soley by volume of press generated. But, when a
marketing executive asked how much all the publicity was really worth, Marker
was forced to develop more meaningful measures. One he developed was return on
investment, which he calculated by taking the dollar value of print space and
broadcast time generated and dividing it by actual project expenditures. The
department also began linking customer inquiries to specific placements.
Summing up the success of the evaluation program, Marker concluded: "No one asks
us what PR is 'worth' any more--they know what it's worth. And they can see
much more clearly how it ties into the corporate marketing effort" (p. 59,
emphasis in original).
A later case study of public relations evaluation was provided by Larson and
Massetti-Miller (1984). They examined the effectiveness of a public education
campaign to promote recycling. The post-campaign public opinion surveys showed
little change in attitudes or recycling behavior. Larson and Massetti-Miller
concluded that the campaign may have failed because it relied too heavily on
mass media at the expense of strategies encouraging interpersonal communication
and individual participation. They also suggested that the finding of few
effects may be due to the campaign's success at merely reinforcing existing
behaviors that otherwise might have discontinued (see also Files, 1984; Hyman &
Sheatsley, 1947).
A focus on research in public relations was revisited by the Review in a 1990
special issue. One article profiled a readership survey conducted by Pavlik,
Vastyan, and Maher (1990). These researchers found moderate support for their
hypotheses that level of organizational integration (functional and
psychological) and importance of organizational surveillance (system functions
and social network) among employees was related positively to level of
newsletter readership. This research helped evaluate who is likely to read the
newsletter and what content readers are looking for.
Another case study was a field experiment designed to test learning from a
health campaign. Rosser, Flora, Chaffee, and Farquhar (1990) evaluated the
effects of a six-year campaign designed to provide information about heart
disease. Not surprisingly, they found that a number of variables--such as age,
risk, gender, level of prior knowledge, and educational level--affected
subjects' learning from the campaign.
More recently Watson (1995) profile four case studies of public relations
evaluation in the United Kingdom. He chose this methodology because he thought
the models of evaluation promoted in the U.S. literature "have not been tested
against the normal constraints of practice" (p. 4). Among the case studies,
Watson found both a short-term and a long-term model of public relations
evaluation. In one short-term case, a three-month lobbying campaign, evaluation
of effectiveness was "quickly visible" and "could be expressed as a Yes or No
result" (p. 4). The government found money for a project and the organization's
lobbying efforts ceased.
Two of the case studies--a major industrial redevelopment and a proposal for a
new community--involved long-term evaluation efforts. As Watson explained:
The planning and development issues embodied in the two projects were so
sensitive
that no pre-testing of attitudes could be undertaken. In one of them, it
may still be
too sensitive to poll some publics. At the industrial redevelopment,
research has been
used to validate the community relations programme and modify it for the
future. An
iterative loop has been used and continues to be used to sustain a
continuing public
relations process. (p. 4)
Two more case studies of public relations evaluation were provided by Knobloch
(1996). She used time series analysis to track media articles before and after
a manufacturer responded to charges of selling products detrimental to
consumers' health. She also profiled a campaign where seven press mailings
induced 190 neutral or positive articles. Knobloch explained that the time
series method projects data before the public relations intervention to the
period after the public relations activity. Thus, one can infer what would have
happened without the intervention as well as the effect of the intervention.
Although Knobloch's study is limited to publicity tracking, sophisticated
methods such as time series analysis may provide a framework for modeling other
public relations functions and outcomes (see also Ehling, 1992).
Some case studies of evaluation have attempted to measure relationship
indicators rather than informational, attitudinal, and behavioral effects on
publics. Most often, the theoretical framework used is coorientation, which
posits communication understanding, accuracy, and agreement as measures of a
relationship (see Broom, 1977; Broom and Dozier, 1990; Dozier & Ehling, 1992).
One example comes from Bowes and Stamm (1975), who determined in their study of
a public agency that the agency's information sources did not increase accuracy
among publics about the agency's stands even though the public believed that the
communication did. These researchers also discovered that there was more
agreement among groups studied than the groups thought. Bowes and Stamm
concluded that relationship measures, such as accuracy and agreement, can
provide meaningful evaluation data that might be missed by self reports of
knowledge and attitude (see also J. Grunig & Stamm, 1973; Hesse, 1976; Knodell,
1976; Stegall & Sanders, 1986).
In 1984, Ferguson suggested that relationships between organizations and
publics become a primary focus of theory building in public relations. She
posited many variables related to public relationships that could be evaluated:
the degree to which the relationship is dynamic versus static and open versus
closed; the degree of satisfaction one or both parties derives from the
relationship; how much control parties believe they have; the distribution of
power in the relationship; whether parties believe they share goals; and whether
there is understanding and agreement.
Despite Ferguson's call, though, little evaluation of public relations has
focused on relationships. Making this point, Broom, Casey, and Ritchey (1997)
said:
Many scholars and practitioners say that public relations is all about
building and
maintaining an organization's relationships with its publics. However,
anyone reading
the literature of the field would have difficulty finding a useful
definition of such
relationships in public relations. Instead, it appears that authors assume
that readers
know and agree on the meaning and measurement of the important concept of
relationship.
Unfortunately, the assumption is not supported by evidence. (p. 83)
Broom et al. went on to explain that public relations evaluation typically only
measures impact on one or both sides of the relationship and then makes
inferences about the relationship. In other words, no direct measures of the
relationship itself are made.
Also exploring evaluation of relationships in public relations is a task force
established in 1996 at an Evaluation Summit sponsored by Ketchum Public
Relations and the Institute for Public Relations Research and Education. As
Kitty Ward, president of K. Ladd Ward and Company and member of the task force,
explained: "My perspective is that most, if not all, of marketing and public
relations has to do with changing or building relationships. I wonder if we
shouldn't go at this (public relations measurement) from the point of view of
evaluating these relationships" (quoted in "Seeking minimum standards," 1996, p.
16-17).
Another task force member, Professor James Grunig of the University of
Maryland, agreed, arguing that most of the outcomes practitioners try to measure
tend to occur "too far down the road" (if at all) to be helpful in demonstrating
public relations effectiveness (p. 16). Given this dilemma, J. Grunig argued
that public relations needs to evaluate maintenance strategies (e.g., building
coalitions of management and publics) that are predictors of relationship
outcomes such as commitment and trust.
Status and Types of Public Relations Evaluation
The final category of research about public relations evaluation includes
Lindenmann's 1988 survey of research activity among public relations
professionals in the United States (published in Lindenmann, 1990; see also
Booth, 1986; Broom & Dozier, 1983; Chapman 1982; Grunig, 1977a; Leffingwell,
1975). His results suggested a mix picture. A clear majority of respondents
believed that public relations evaluation is and can be an integral part of
programming. And, more than half agreed that it is possible to measure public
relations outcomes, impact, and effectiveness precisely. Moreover, most
respondents said they allocate funds for public relations research and that the
volume of research projects they are involved in has increased.
On the other hand, Lindenmann found that more than 90 percent of survey
respondents believed that public relations research is still more talked about
than actually done. And, about 70 percent thought that most public relations
research is still "casual and informal rather than scientific and precise" (p.
15). Lindenmann concluded that, although the status of evaluation in public
relations had improved since the 1960s and 1970s, there is "still a considerable
distance to travel" (p. 15).
Bissland (1990) also addressed the status of evaluation in public relations by
reviewing the 60 winning case summaries from the 1980-1981 Silver Anvil
competition. He compared these to the 72 winners from 1988--1989. In 1982,
PRSA had specified that all entries should include a section on evaluation.
Bissland found three main categories of evaluation among the cases: (1)
measures of communication output (e.g., quantity of output, number of media
contacts, quality and quantity of media placements), (2) measures of
intermediate effect (effects on key audiences such as number who attended to the
message or participated in an event; audience feedback; and behavior science
measurements--awareness, attitude, understanding, behavior), and (3) measures of
organizational goal achievement (e.g., numbers of products sold, new members
recruited, funds raised, legislation passed or defeated).
The most frequently used evaluation measures were output and intermediate
effects--36 percent, and 39.6 percent, respectively. Measures of organizational
achievement accounted for only 24.6 percent and some of these were inferred
rather than substantiated.
Yet, Bissland was able to show meaningful progress in evaluation during the
decade examined. The number of different evaluation methods rose from a mean of
3.6 to 4.57, which was a statistically significant difference (p <.01). In
1988-1989, more practitioners were using behavioral science measures--44.4
percent, up from 25 percent in 1980-1981 (significant at p < .05). Also, the
increase in measures of organizational goal achievement was significant--28.6
percent, up from 18.5 percent (p < .001).
Bissland concluded that the status of evaluation in public relations did
improve in the 1980s. Yet, in 1988-1989, still over half of the entries lacked
rigorous methods. And, twice as many contained unsubstantiated claims of goal
achievement as substantiated ones.
Barlow's (1993, cited earlier) assessment of public relations evaluation
revealed that many scholars and practitioners he interviewed believe meaningful
measurement is hampered by widespread lack of research sophistication among
communication professionals. However, a majority said that the tools for
measuring "just about anything" do exist (p. 10). The respondents acknowledged,
though, that a macro-level measurement of public relations' value is baffling.
And, some made the point that developing this macro measure--such as a
comprehensive cost-benefit analysis--may not be worth it, given the time and
money needed to do so.
Barlow also examined the status of expenditures for communication research.
Nine of the 31 corporate executives said that their research budget has gone up
but there was no clear pattern as to why. And, among those whose budget was
about the same, there was a tendency to protect evaluation. "We're real proud
of the fact that we didn't have to cut here," said one executive (quoted in
Barlow, p. 44). For those facing cuts, the reasons were obvious--the recent
recession and downsizing. Overall, though, the majority of practitioners
thought that top management is "generally supportive" about spending money on
research (p. 45).
Watson (1994) examined the status of public relations evaluation in the United
Kingdom by surveying a random sample of members of the Institute of Public
Relations. His results were similar to U.S. survey findings: Public relations
evaluation is hindered in the United Kingdom by lack of research knowledge,
clients' and employers' unwillingness to devote resources to evaluation, and
practitioners' reluctance to tie objective performance measures to their
programs.
Watson (1995) later cross-checked his U.K. survey with a sample of 30
practitioners from 25 countries. He found that the barriers to evaluation
uncovered in the UK survey were mirrored worldwide: lack of time, lack of
personnel, lack of budget, cost of evaluation, doubts about usefulness, lack of
knowledge, and aversion to scientific methodology [see also Baerns' (1993) study
of evaluation in Germany].
Other researchers have focused specifically on identifying types of public
relations evaluation. For example, Lerbinger's 1977 analysis of corporate use
of research identified four main categories--environmental monitoring, public
relations audits, communication audits, and social audits. Environmental
monitoring was used by organizations to gauge public opinion and observe social
and political events that may impact the organization. The most common research
technique Lerbinger found was the public relations audit, the most comprehensive
of which involves assessing relevant publics, an organization's standing with
publics, issues of concern to publics, and power of publics. A communications
audit evaluates the effectiveness of an organization's communication products
among internal and external audiences. Typically included here were readership
and readability studies and content analysis of publications. And, last,
Lerbinger found that organizations were using social audits to asses how they
are performing as corporate citizens although there was little agreement about
what this evaluation method entails (see also Reeves & Ferguson-DeThorne, 1980).
Also addressing types of evaluation, Swinehart (1979) outlined four levels of
evaluation--process, quality, intermediate objectives, and ultimate objectives.
Process refers to measuring the preparation and dissemination of communication
materials. Evaluation of quality has to do with features of communication
materials or programs--their accuracy, clarity, production values, and
suitability for the audience. Evaluating intermediate objectives typically
involves looking at precursors, such as media placement, to ultimate objectives.
This last category--ultimate objectives--has to do with assessing effects on the
target audience's knowledge, attitudes, or behavior.
Similarly, Lindenmann (1993) presented a yardstick for measuring public
relations effectiveness, which included basic, intermediate, and advanced
evaluation. The basic level involves measuring outputs or media placements,
impressions, and targeted audiences. The intermediate level, or public
relations outgrowths, has to do with evaluating reception, awareness,
comprehension, and retention among target publics (p. 8). Last, advanced
evaluation measures public relations outcomes such as opinion, attitude, and
behavior change (see also Lindenmann, 1988, 1995).
Perhaps the most comprehensive scholarship about types of evaluation
comes from Dozier. Using data from a 1981 survey of practitioners, Dozier
(1984) uncovered three types of evaluation. Scientific impact evaluation is the
most formal of the three and involves evaluating public reaction to the
organization and public relations programs before, during, and after
implementation. Seat-of-the-pants evaluation uses informal techniques (e.g.,
feedback from media contacts) to gauge program impact. And, last, scientific
dissemination, involves systematically measuring the distribution of messages.
Dozier then correlated types of evaluation and practitioner roles. He
determined that technicians are unlikely to use any type of evaluation.
Managers predominately use seat-of-the-pants evaluation to augment measurement
of scientific impact (see also Dozier, 1985, 1990).
Lesly (1986) addressed "levels of measurability" in public relations but dealt
with intangible as well as tangible effects (p. 6; see also Lesly, 1991). He
argued that many of the most valuable contributions public relations makes go
beyond the outcomes of specific activities that can be measured easily with
numbers. In other words, some indicators of public relations effectiveness defy
numerical measurement.
Lesly called the first level of evaluating intangibles semi-specific
measurement. As an example, he mentioned practitioners' assessing "the reaction
to a presentation before a Senate committee, or to an officer's speech" (p. 6).
No numerical measurement is implied yet a judgment can be made about the
effectiveness of public relations.
The next level was acceptance on the basis of judgment. This occurs when
management believes on faith that public relations efforts have been effective
in areas such as working with minority, community, and church groups.
Lesly also mentioned recognizing the value of input from public relations
people as an intangible evaluation measure. Here, public relations might be
judged on how effective the department is at monitoring public opinion and
keeping management abreast.
Last, Lesly posited prevention and guidance as key measures of public relations
effectiveness. He argued that preventing negative outcomes, such as bad press,
may be the best indicator of public relations success. Similarly, the ongoing
guidance public relations provides, "like a car--operating so quietly that no
one is concerned with it," is a strong testimonial to effectiveness (p. 7).
Geduldig (1986), however, took issue with the argument that intangible public
relations outcomes demonstrate value. He maintained that public relations
departments must establish concrete measurement systems to justify their
existence to others:
A hard-nosed manager would have a tough job evaluating a function that
cannot be
defined and can do well when it does nothing....Don't expect others to buy
public
relations on faith. If public relations doesn't set standards of
measurement that are
both objective and meaningful, management will apply its own, and the value
of public
relations will ultimately be measured against the bottom line. (p. 6)
Research Questions
Geduldig's assertion underscores the need for a better understanding of the
intricacies involved in setting public relations goals and objectives and
conducting meaningful evaluation. This study explored the challenges and
opportunities involved in these tasks among a selected group of public relations
practitioners and top executives. The following research questions were posed:
--What do practitioners and other managers consider the main goals and
objectives for public relations at their organization?
--What do they think is the link between goals and objectives for public
relations and goals and objectives for their organization?
--How do they evaluate public relations?
Methodology
These research questions were investigated through in-depth, non-directed
interviews with 32 public relations practitioners and 10 other executives.[3] A
qualitative method was chosen to provide the opportunity for respondents to
discuss research topics from their point of view rather than responding to
structured categories. Qualitative research seemed especially fitting given the
multifaceted issues and perspectives uncovered in the literature review.
Sampling
The interviewees were selected following Broom and Dozier's (1990) guidelines
for nonprobability sampling. The type of sampling used was a combination of
snowball and dimensional. Snowballing, or asking initial interviewees to
suggest other interviewees, served to expand the list of participants from the
original handful (drawn from the researcher's professional contacts) to the
final 32.
When recruiting participants, two dimensions also were considered to ensure
diversity among the group--type of organization and level of experience. A wide
array of organizations throughout the United States was assembled--corporations,
government, nonprofits, associations, and public relations agencies. And,
although most of the practitioners (and all of the other managers) are seasoned
executives, some junior-level practitioners deliberately were chosen. The
rationale here was that those relatively new to the field might have a different
perspective given that evaluation increasingly is stressed in public relations
curricula.
Interview Procedure
With the exception of three interviews, which were held face to face, all of
the interviews were conducted over the phone. The three research questions
served as grand tour questions (Spradley, 1979, 1980), which provided the
over-arching framework for the interviews. When appropriate, probes were used
to elicit explanatory information and examples. And, as typical with
qualitative inquiry, interviewees brought up related issues unprompted.
After conducting the 42 interviews and an initial review of findings, the
researcher determined closure was appropriate. At this point, the emerging data
seemed sufficient to provide meaningful insight into the areas addressed by the
research questions.
Snow (1980) referred to this aspect of qualitative research as "informational
sufficiency" and described its three dimensions (p. 103). Taken-for grantedness
occurs when very little respondents say surprises the researcher and the
interviews begin to seem routine. Theoretical saturation describes diminishing
returns--the additional data add very little novel or useful information. And,
finally, a heightened confidence occurs when the researcher feels the "findings
are faithful to the empirical world under study and shed light on preexisting or
emergent questions and propositions" (p. 104).
Data Analysis
Interview data next were transcribed, and responses were arranged under the
research question they answered. The findings then were synthesized with the
dual purpose of revealing common themes or trends and uncovering the range of
response, or to what extent practitioners' and other managers' experiences and
examples diverge.
Successfully achieving this purpose is the measure of any qualitative study's
merit. As Lindlof (1995) pointed out, the conventional canons of research
reliability and validity do not fit the qualitative paradigm very well. Based
on interpretive assumptions, qualitative research recognizes "the constantly
changing character of social reality" (p. 238). Thus, "little is gained from
trying to achieve reliability," or consistency of results (p. 238).
Assessing validity is also difficult because qualitative inquiry assumes the
world consists of "multiple, constructed realities" (p. 238). Therefore no
single representation can be identified as the criterion for valid measurement.
Lindlof pointed out that concerns about external validity also are suspect since
"the qualitative researcher studies social action and cultural sensibility
situated in time and place" (p. 238). For this reason, generalizations are
"neither warranted nor particularly desirable" (p. 238).
Lindlof's arguments, however, do not imply that qualitative researchers lack
credible and dependable data. As he said: "Basically we want to inspire
confidence in our readers (and ourselves) that we have achieved right
interpretations. Notice that I do not say the right interpretation. There are
many possible interpretations of a case" (p. 238, emphasis in original).
One method for ensuring meaningful interpretations is member checks, whereby
investigators elicit critiques of their data analysis (Lincoln and Guba, 1985).
The member check conducted for this study involved 10 of the research
participants, who were asked to review the major topics and themes that the
researcher had identified in the data.[4] These participants validated the
efficacy of the findings by confirming that these topics and themes were
accurate interpretations of their answers to the research questions.
Findings
What do practitioners and other managers consider the main goals and objectives
for public relations at their organization?
There was tremendous variability in the answers to this question, perhaps
reflecting the diversity of organizations represented. Each department divides
its responsibilities differently and, not surprisingly, priorities tend to
reflect the mission of the organization. For example, practitioners employed by
corporations tended to mention communication goals that support the bottom
line--increasing sales and revenue and bringing in new business. Other
practitioners talked more about public relations' role in enhancing the image of
the organization and disseminating positive messages.
Most of the interviewees said that their public relations department does have
written, quantifiable goals and objectives although several acknowledged that
this systemization happened somewhat recently. Only one practitioner--a public
affairs adviser for an oil company--said that his department has no written
goals and objectives. He described a situation of great instability where the
business units of his organization are driving public affairs goals, while
corporate public affairs is being downsized drastically.
A few others said that their organization is moving toward a more goal-oriented
approach but it is not there yet. For example, a communication manager for a
trade association said that setting public relations goals and objectives is a
"process that we're still working out." She explained that public relations
traditionally has been geared toward "really nebulous concepts such as good will
and outreach." One reason is that the organization has suffered from a lack of
data on which to base more concrete objectives. She also mentioned people's
tendency to keep "doing things the way they did them in the past."
A similar response came from a marketing communications manager at a private
university. She said that public relations goals typically have been vague and
more "intuitive" than explicit. She said that increased competition among
universities has brought the realization that public relations goals have to be
more precise.
The most recurring theme in practitioners' responses was that public relations
goals and objectives have to be "strategic," meaning that public relations goals
and objectives must be tied directly to organizational goals and objectives.
Many said that this strategic goal setting takes place every year, although some
described a more long-term approach such as a two- or three-year plan.
A community liaison for a national health institute said that her department
sets goals and objectives that are "in line with institutional goals and overall
objectives for each year." Each person's personal goals have to support the
communication programs that help the department fulfill the broader goals of the
organization--such as increasing patient volume.
A director of corporate communications for a government contracting firm also
stressed that public relations has to be "tied to goals of the organization."
Her department looks to the CEO's published goals--such as diversification--and
then writes a plan based on those.
The importance of this link was emphasized by a CEO of a public relations
agency. She said that too often public relations goals become separate from the
organizational goals and then the function seems expendable. She warned against
creating "isolated, separate" goals such as increased publicity that are merely
a means to an end. For her, public relations goals should match clients'
broader goals, such as increasing market share or sales.
Several other agency executives also emphasized that public relations goals
must focus on the "end result" expected. As one senior vice president said:
The first thing you ask the client is, "What do you want to happen as a
result of PR
activity?" You try to get as specific as possible. It can be a change in
awareness,
buying behavior, but not "supporting" something. The client has an outcome
in mind. At
the end of your contract, they are going to make a decision about whether
you did what
they want.
A similar process was described by a marketing communication associate for a
management consulting firm. As she explained, her agency first uncovers what
the client ultimately wants. The group then determines the target audience. At
that point, goals and objectives can be specified. One practitioner in
particular--a national vice president of public affairs for a health
association--emphasized that public relations goals should lead to quantifiable
objectives. As he explained:
As companies become more and more fixated on the bottom line, maintaining a
healthy
cash flow, cutting the fat, on downsizing or rightsizing, it becomes more
and more
important for the public relations function or communications function,
like every other
function, to be measurable, to show measurable results, and that of course
opens the door
to really good, solid, quantifiable objectives.
Among the other managers who were interviewed, the most common response was
that public relations' ultimate aim is communicating the image of the
organization. An executive vice president of a health association was emphatic
on this point: "Our success depends on our ability to communicate with the
public--being believable as an organization, giving them (the public) credible
information. Raising funds, being supported by the public, depends on our
public image. PR is part of that image."
Similarly, a president of a state university said that the goal of public
relations is "to tell the story of the institution, to inform the public about
what's going on, to let people know the value they earn for the investment they
make in the institution." He also noted, "Obviously, there is a 'best foot
forward' side of PR, which I think is their ongoing basic responsibility."
A CEO of a food products company mentioned that public relations should be
carrying the organization's "proper message" to internal and external audiences.
A senior vice president for a health benefits provider also referred to public
relations' responsibility as "any involvement that gets [name of company]
communicated externally and internally."
Only one top executive--a manager of federal relations for an oil company--was
indefinite about public relations' mission. He said that communication goals
are "nebulous" and "hard to pin down." Direction for public relations is in
flux, depending upon the whim of Congress and federal and state regulatory
agencies. Yet, the overall scheme is to push for legislative initiatives that
do not "negatively affect" the organization's refining and marketing efforts.
What is the link between goals and objectives for public relations and goals
and objectives for the organization?
Most interviewees stated that the connection is very clear to them and, for the
most part, others in the organization. For example, a vice president of
membership for a trade association said that "everything we do" relates to the
organizational mission:
It's very simple--from our magazine, to the way we market and put on our
education
programs, through our development of membership and our service to
membership--everything
we do is promoting and enhancing responsible and effective philanthropy.
We don't do it
unless there is a direct link.
Similarly, a director of public relations for a health benefits provider said
that public relations is part of "every thread of the fabric" at her
institution--"so intertwined, you can't separate" public relations goals from
the organization's mission. She explained that since the organization's
activities are controversial, the agency "inherently needs PR to explain
[itself] to constituents and provide a channel for constituents to give
feedback."
The link also was clear to a director of public affairs for a highway safety
coalition. She talked about how crucial the organization's media relations is
to keeping issues before the public and Congress. She also thought that the
president of the organization recognizes effective public relations is the
organization's lifeline.
"The president truly values PR; she is media savvy," she said. "If anything,
maybe she relies on it too much in the 11th hour. But, that's a nice problem
for a PR person to have."
A national vice president of public affairs for a health association also had
no trouble associating public relations with organizational goals:
By constantly putting our messages out there and associating them with our
name, the
public has a heightened sense of our mission and is more pliant when it
comes to
supporting it. Our role is that controversies are headed off, successes
are featured, inf
ormation flows constantly, and that, in turn, creates very high name
recognition. It's
not by accident that these things happen. That's our role in making sure
that the
organization is successful.
However, some practitioners thought the link between public relations and
organizational goal achievement is obscure. For example, a vice president of
corporate communications for a mortgage company thought that when it comes to
bringing in new business, public relations' contribution is not so obvious.
"It's difficult to link getting loans to something we have done," she said.
"It's more in the relationship building than bringing loans in the door. It's a
lag effect."
A marketing specialist for a management consulting firm also alluded to the
difficulty of definitively relating some public relations outcomes to
organizational goals. Convincing others in her organization of the value of
effective communication therefore is not easy. As she said:
They are looking for a return on investment. They are looking at how much
we put in
versus how much we get out. I have tried to explain that it's difficult to
measure. PR
has not been able to provide quantitative, tangible results, but it's good
at providing
qualitative results.
A CEO of a public relations agency also talked about the difficulty of
convincing others of the relationship between public relations goals and
organizational success:
Sometimes one of our biggest challenges is proving our value. The more we
can align
PR goals with corporate goals, the better--to show that PR is not just in
support of
those goals, but very much a part of achieving those goals.
The need for public relations to prove its worth also was emphasized by a
public relations director for a private school. She too believed that public
relations needs to show how its goals fit into the organization's.
"You have to look at the criterion of whether or not it (a public relations
goal) is contributing to the bottom line of the organization or its goals," she
said. "If it doesn't directly contribute, then I would say it's not necessary."
Perhaps the most grim situation was described by a communication manager for an
animal rights and conservation organization. She described the link between
public relations and the organization's mission as "not very strong." She
thought there was a "huge gap between what public affairs does and what
organizational leaders think."
"The ideal model works only on paper," she said. "I don't see communications
incorporated as much as it could be--except in a crisis. Then it's, 'Man the
torpedoes; you guys do something.'"
This practitioner thought part of the problem is that others in the
organization do not understand public relations since the outcomes tend to be
less tangible than other functions. Making this point, she recalled how, after
one of her particularly successful efforts, a scientist at her organization
grudgingly stated during a meeting, "My [new] advice to everyone here is don't
run away from [name of practitioner] when she comes down the hall."
CEOs and other managers offered a variety of perspectives about how public
relations goals relate to the organization. A CEO of a food products company
described a "firm link" between public relations and the organizational mission,
noting that the vice president of corporate communications is very involved in
strategic planning for the organization.
However, a development director for a private university acknowledged that
public relations has not fed into her activities as much as it could. She said,
though, that public relations and fundraising increasingly are mapping out
mutual goals.
In two relatively new organizations, competing viewpoints were offered. A CEO
of a national health institute thought that proactive public relations is
fundamental to achieving the organization's goal of establishing itself as a
premier institution.
On the other hand, a senior vice president of a health benefits provider
thought that because his organization is so new, the link between public
relations and organizational goals is not "as strong as it is for established
organizations with specific strategies for public-corporate stewardship." He
described his company's public relations activities to date as reactive, stating
that proactive public relations happens only as companies evolve.
"Most of the people here aren't that aware of professional PR," he said. "It's
not totally respected because of the difficulty in developing value proofs that
say, 'If you put the resources here, this is what you get back.'"
Still others provided more perspectives on the connection between public
relations and organizational goals. A president of a state university said that
"PR is the window on the external world about the institution and its mission
and how well it is progressing toward realizing its mission."
Similarly, the president of a government contracting firm believed that public
relations supports the organization's strategic plans: "Every company has a
vision of where it wants to be. If we want to change the image, public
relations would assist in providing that support." But, he said that there is
not "much connectivity on a day-to-day basis" between public relations goals and
the organization's mission. Instead, public relations' contribution is more
long term.
How do practitioners and other managers evaluate public relations?
Responses here ranged from virtually no evaluation to formal and on-going.
However, despite this range, none of the interviewees disregarded the need for
public relations evaluation. Some practitioners said that, although they do not
feel pressure to show how the needle moved for every public relations activity,
they still must demonstrate "value." Others were emphatic about showing
measurable results.
Not surprisingly, many respondents commented on how difficult public relations
evaluation is. They bemoaned inadequate resources--time, staff, and money--as
well as the intangible nature of some public relations goals.
A director of university relations for a state university referred to
evaluation as the "Achilles heel" of public relations. As he said:
It's real easy to measure things that don't matter and very difficult to
measure
things that do matter. The end result of our efforts here should be
changes,
enhancements in attitudes of carefully defined publics....It doesn't matter
how many
stories get placed; attitudes count. We need statistically valid attitude
research.
This practitioner went on to characterize the university as lagging behind
others in market research. Just this year, the Office of Institutional Studies
will do its first comprehensive stakeholder survey, which will provide him with
baseline data. He mentioned, though, that public relations' needs are
considered "perimeter" rather than "center" to the study.
A vice president of membership for a trade association was skeptical in general
about public relations evaluation. She believed that effectiveness is just too
slippery:
I'm not sure you ever really know with public relations. Yes, you can
evaluate an
event, a mood, an opinion poll, but from the view of 20 odd years, that can
change
overnight....Any company that says their public relations is absolutely
effective is kiddi
ng itself.
A similar theme was heard from a public relations director at a trade
association:
Effectiveness in communications is always very difficult to measure. You
spend an
awful lot of energy and time and resources and you probably don't even know
the effect
until maybe years later....You just work on going after your goal. And,
it's not that
you are hit or miss; you know certain things from experience. But, you
don't sit down
and analyze that.
A vice president of corporate communications for a food products company also
was doubtful about public relations evaluation: "We are not turning in sales
reports on a weekly or monthly basis by which we can be judged. A lot of this
is intangible."
He acknowledged, though, that "the demand for measurable results is increasing"
and that research is "vital" and often "the bread and butter of many successful
programs." However, at his company, the budget is just not there. As he
explained:
The difficulty really comes in the budgetary area. When you are having to
watch
costs very closely, there is little support for going outside to embark
upon a major and
costly program to help you determine just how well you are doing.
Despite constraints, though, most practitioners were doing some sort of
evaluation. The measures used typically varied by type of program and target
audience.
Several practitioners referred to their evaluation as informal, mentioning
feedback from audiences or others in the organization about public relations'
products or services. For example, a publications editor for a trade
association said that her department relies on "oral communication from other
departments, members calling in, letters from members."
However, a senior vice president of a national agency disparaged this approach,
arguing that "informal evaluation is about as good as kissing your sister." She
described her agency's evaluation as formal and contingent upon the goals of the
campaign.
"The days of companies spending lots to get their name in the paper are over,"
she said. "Our clients are specific."
Other agency executives disagreed. A communication associate for a management
consulting firm thought that clients often want nothing more from public
relations than "get my name in the paper."
A CEO of another agency thought selling clients on evaluation can be difficult
because clients think that public relations is free advertising and therefore
should not cost a lot. Still another agency vice president thought clients
often do not understand research or place little faith in the measure of public
relations activities. Thus, evaluation at her agency amounts to little more
than tallying media impressions or counting heads at special events.
Some practitioners were doing little or no evaluation but planned to do more.
A director of public relations for a health benefits provider explained that,
before she joined her company, the CEO's yardstick for evaluating public
relations was "whether or not a good time was had by all." She wants, however,
to begin conducting survey research that would provide benchmark data about
levels of brand awareness. The survey then would be repeated every six months
or year. She believes other managers will support the project because "they
would like to know they have measurable results; TQM (Total Quality Management)
has brought this mentality."
Evaluation techniques described by other practitioners were those one would
expect to find in public relations--conducting focus groups and readership and
customer surveys; monitoring government research; and tracking media placement,
attendance at events, store traffic, calls to 1-800 numbers, and "hits" on the
organization's World Wide Web homepage.
Some practitioners evaluated behavioral objectives almost exclusively, which
they feel makes measurement straightforward. An example was a director of
alumni relations for a state university who said that evaluation is "kind of
easy." Her department gauges its ultimate success by tracking donations.
Similarly, an executive director of a trade association explained that at his
organization "measurement is built in," and the measure is "specific regulatory
action." Referring to the organization's members, he said: "They don't care if
we increased awareness or opinions if their ultimate goal was not achieved.
They don't care about winning the battles if you lose the war. They don't want
to spend money on the battles."
A management associate for a passenger railroad also thought that the acid test
for public relations is effects on behavior. As the company cuts services in
some states, the measure of effective public relations has been whether those
states buy the lines back. This practitioner went on to explain that the
pressure to downsize and rightsize has been so intense recently that the only
outcome management really cares about is whether or not the organization
survives.
Perhaps the most debatable issue among these practitioners had to do with
evaluating media placement. Almost all of the interviewees mentioned using
clipping services. However, opinions about the value of doing so varied
greatly.
A senior vice president of a national agency summarily dismissed clippings:
"You might as well count press releases....You still have to tie it (placement)
to some impact."
However, a director of corporate communications for a government contracting
firm said that her organization has "found very elaborate ways to quantify how
much press we have gotten and what kind." She said that the company has become
more "sophisticated in quantifying that (media placement) in terms of
advertising dollars." She also is tracking "inroads into trade publications and
newspapers important to us."
Most other practitioners were ambivalent about the worth of evaluating media
placement. They acknowledged that placement may indicate message exposure.
But, as an agency CEO said, placement is not the "panacea" of public relations
evaluation.
One example of systematic, on-going evaluation was provided by a director of
external relations for an international exchange organization. She explained
that before the association's yearly conference, her department does a "needs
assessment survey." The results are used to plan the conference. A follow-up
survey and focus groups then are conducted.
"We tend to evaluate projects like this," she said. "The board and members are
open to evaluation, research, getting feedback from members, putting it into
action."
Only one practitioner--a communication manager for a media and technology
company--explicitly pointed out the connection between public relations
evaluation and organizational goal achievement:
One way that we measure [public relations goals and objectives] is that
performance
plans are supposed to be linked back to goals for the organization. We
haven't done a
good job of communicating the vision of the company, so people have a hard
time seeing
the link.
When the CEOs and other managers were asked about measures of effectiveness in
public relations, their answers were similar to those provided by the
practitioners--successful media placement and desired changes in attitudes and
behaviors. And, every single top executive stressed that effective public
relations successfully builds the organization's image.
Making this point, a senior vice president at a health benefits provider
suggested the following indicators for evaluating public relations:
Every opportunity that we can identify that [name of organization] should
have been
in external media, we were there. Every time a major issue arose that
required experts
in our area, we were contacted. We were known. When a significant issue
arose that
could have been a PR disaster, it was diffused and diffused rapidly. The
marketing
materials occurred in such a way that nothing was disruptive; the proper
message was out;
it was timely. Supporting materials had the look, the feel; there was
synergy.
Similarly, a president of a state university said there is "one way" to measure
the effectiveness of public relations--"the success of the organization in
having stories about the organization in the media, the frequency and quality of
communication with the public, the attitude of alumni, parents, and students
about the institution." He also mentioned growth in private giving as an
organizational goal related to successful public relations.
Some of the executives also related public relations evaluation to performance
indicators for the organization. A market analyst for a passenger railroad was
very clear about the ultimate measure of effective communication: "In terms of
our performance, [we ask], 'Do our revenues go up?.' If we are really a good,
competitive form of transportation and that word is out, then we should be able
to make a little more money."
When probed about whether others in the organization understand the link
between effective public relations and increased revenue, he said that the
connection has become clear only within the last six months: "Before that, it
(public relations) was pretty much isolated, and when it came time to make cuts,
it was the first. There wasn't the understanding."
The company's new found appreciation for public relations came from sheer
"desperation," he said. "What we had done before just wasn't working." He said
that a new management team with a new philosophy then was recruited: "These new
individuals understand communication better--internally and externally."
Other executives recognized the connection between measures of effective public
relations and organizational goal achievement but acknowledged that the link is
not always explicit. For example, a CEO of a national health institute said
that public relations at his organization is evaluated through physician and
consumer attitude surveys. Yet, he said the line is "fuzzy" between attitude
data and performance indicators such as "success with the legislature, success
with getting national grants, [and] numbers of national news pieces."
A senior associate at an animal rights and conservation organization echoed
this theme. She explained that at her agency, evaluating a direct marketing
program is easy--just tally responses. Yet, the only concrete measure for
public relations the agency uses is media placement.
"We were never able to quantify whether public relations support enhanced our
direct marketing responses," she said about one membership campaign. "But, we
have come to accept that it does."
This notion of accepting public relations on faith seemed to run through most
of the CEOs' and other managers' responses.[5] An executive vice president of a
national health association clearly believed that the awareness and trust public
relations creates are directly related to his organization's ultimate measure of
success--reducing disease and risk factors for disease.
He made this point with an example about women and breast cancer:
Getting women to change their behavior is in part dependent upon our public
relations image because when we go to the public, there are other messages
that they are
getting. Our image has to be consistent, good, [and] accurate so women
will believe us
and get mammograms. The public relations part of that has to do with
credibility.
Discussion
This example illustrates the main point made in much of the literature about
public relations evaluation: Measuring the outcomes of public relations
programs provides data needed to demonstrate that public relations helps
organizations and clients meet their performance goals.
However, as this study underscores, setting goals and objectives and conducting
meaningful evaluation in public relations continue to be complicated tasks.
Part of the problem is that public relations itself encompasses so many
different activities that goals and objectives will vary, depending on
organizational priorities. It goes without saying then that no single
evaluation formula will be appropriate for all programs and organizations.
Yet, despite the variable responses among these practitioners and CEOs, several
trends are discernible. The first is that these practitioners seem to be moving
toward more systematic public relations programming. All but one practitioner
said that his or her organization has written goals and objectives for public
relations. It is perhaps not surprising that the one organization lacking
written goals and objectives is downsizing its public relations department.
Another striking theme is the long-range plans several interviewees had for
building more formal evaluation into their public relations efforts. Most
often, the junior practitioners were the ones who described such schemes. So,
although some of these practitioners may be overly optimistic,[6] their outlook
suggests that evaluation may be becoming a more integral part of public
relations programming.
A related point is that several practitioners seemed to be educating others in
the organization about measurement issues in public relations. An example is
the practitioner whose CEO used to think that "a good time was had by all" is
the ultimate gauge of public relations effectiveness.
Perhaps the most encouraging finding is the understanding among these
practitioners and CEOs of how public relations feeds into the strategic goals of
their organization. Although most of the time this connection was inferred
only, obviously most interviewees do not see public relations as an isolated
function. Instead, they can articulate clearly how the ultimate success of the
organization is inextricably linked to effective communication. As one
practitioner said, public relations is part of "every thread of the fabric" at
her organization. Similarly, the president of a state university thought that
public relations is no less than the university's "window" to the external world
about the university's mission and progress toward that mission.
Of course, some exceptions were revealed. Several interviewees described
situations where public relations continues to be devalued. One reason given
was that others in the organization do not understand the function. Making this
point, one of the senior executives added at the end of his interview that
professional associations in public relations need to make sure business
students are exposed to public relations courses in college. He mentioned that,
in his own marketing curriculum, public relations was only a "very small piece."
He went on to describe what he sees as a real research opportunity for public
relations practitioners:
We are becoming more and more of an information society. Public relations
[practitioners] know where information is, where it can be obtained. They
can become
brokers of information for organizations very easily. [There will be] more
and more data
and power to crunch. Proactive public relations people of the future are
going to
anticipate issues and convince organizations to position themselves and
have the right
numbers [to do so].
Other interviewees expressed frustration that the effects of public relations
are hard to isolate and too long term to connect back to practitioners' efforts.
As one executive mentioned, coming up with "value-proofs" for public relations
programs is very difficult. And, one practitioner argued that even if you
could, people's perceptions can change overnight.
Practitioners working in public affairs and development were the only ones who
seem to think public relations evaluation is direct. Their measures are
strictly behavioral: Did the legislation go our way? Did donations increase?
Of course, such ease in evaluation is double edged. Failure can be measured as
effortlessly as success.
Not surprisingly, many interviewees said that public relations evaluation
continues to be hampered by strained resources. Given this recurring theme, it
is distressing that not one practitioner mentioned turning to public relations
academicians for research help. Obviously, more collaboration between
practitioners and academicians would be mutually beneficial.[7]
Despite constraints, though, almost all of the practitioners are doing some
sort of evaluation. They mentioned virtually all of the different types of
evaluation that had been identified in earlier research.
Many of these practitioners still rely heavily on informal methods such as
gathering feedback from audiences. And, several feel that assessing media
placement remains a valuable indication of public relations success. A couple
of the CEOs certainly seem fixated on media exposure.
Other interviewees thought this approach is anachronistic. These practitioners
are using or moving toward more sophisticated evaluation methods. Several
described plans for gathering benchmark survey data that they hope is the first
step toward systematic, on-going research.
Only two practitioners described evaluation scenarios that seem pretty close to
ideal. One mentioned that, at her organization, public relations research is
conducted both to plan and evaluate. The information gleaned from evaluation
then is fed back into planning. Another practitioner said that, because
clients demand accountability, formal research is built into every one of the
agency's programs. However, not even these two practitioners described
measuring any relationship indicators. Another point is that all of the
evaluation schemes mentioned by practitioners and top executives seem
characteristic of an asymmetrical model of public relations (see J. Grunig &
Hunt, 1984). No one offered any examples of measuring management's position or
monitoring the organization's responsiveness to publics. Although some
mentioned getting "feedback," doing so involved gleaning information from
publics alone.
And last, no one described experimental research designs whereby a strong case
could be made for causal inferences between public relations activities and
specific outcomes. Most often the link just was assumed.
In many organizations, though, this understanding may be what is needed.
Almost all the interviewees--either implicitly or explicitly--stressed that
effectiveness in public relations has more to do with demonstrating value than
documenting communication effects per se. Fortunately, demonstrating value may
be easier to do (L. Grunig, Dozier, & J. Grunig, 1994; Hon, 1997). Goals,
objectives, and evaluation can provide part of the answer. The task that lies
ahead is finding ways to capture the rest. References
Atkin, C.K. (1981). Mass media information campaign effectiveness. In R. Rice
and W. Paisley (Eds.), Public communication campaigns (pp. 265-279). Beverly
Hills, CA: Sage.
Baerns, B. (1993, May). Understanding the development of public relations in
German, East and West, Proceedings of the Second European Seminar for Teachers,
Practitioners and Researchers. Prague, Germany.
Barlow, W. (1993). Establishing public relations objectives and assessing public
relations results. Paper prepared for the Institute for Public Relations
Research and Education. Princeton, NJ: Research Strategies Corporation, 217
Wall Street, Princeton, NJ, 08540.
Bissland, J.H. (1990). Accountability gap: Evaluation practices show
improvement. Public Relations Review, 16(2), 25-35.
Booth, A.L. (1986). Strength in numbers. Public Relations Journal, 42, 26-30,
43.
Bowes, J.E., & Stamm, K.R. (1975). Evaluating communication with public
agencies. Public Relations Review, 1, 23-37.
Broom, G.M. (1977). Coorientational measurement of public issues. Public
Relations Review, 3(4), 110-112.
Broom, G.M., Casey, S., & Ritchey, J. (1997). Toward a concept and theory of
organization-public relationships. Journal of Public Relations Research, 9(2),
83-98.
Broom, G.M. & Center, A.H. (1983). Evaluation research. Public Relations
Quarterly, 28(3), 2-3.
Broom, G.M. & Dozier, D. M. (1983). An overview: Evaluation research in public
relations. Public Relations Quarterly, 28(3), 5-8.
Broom, G.M. & Dozier, D.M. (1990). Using research in public relations.
Applications to program management. Englewood Cliffs, NJ: Prentice-Hall.
Burnstead, R.R. (1983, March 18). How to measure the effectiveness of a PR
campaign. Marketing News.
Campbell, C. (1993, October). Does public relations affect the bottom line.
Study shows CEOs think so. Public Relations Journal, 49(10), 14-17.
Campbell, D.T., Stanley, J.C. (1963). Experimental and quasi-experimental
designs for research. Chicago: Rand McNally.
Caro, F. (Ed.). (1977). Readings in evaluation research (2nd ed.). New York, NY:
Russell Sage Foundation.
Chapman, R. (1982). Measurement: Alive and Well. Public Relations Journal, 38,
28-29.
Cline, C. (1984). Evaluation and measurement in public relations and
organization communication: A literature review. San Francisco: International
Association of Business Communicators Foundation.
Cramp, B. (1994, July 28). Ahead of the count: Counting the press mentions is
no longer a good measurement of PR. Marketing Magazine (London).
Cutlip, S.M, Center, A.H., & Broom, G.M. (1985). Effective public relations (6th
ed.). Englewood Cliffs, NJ: Prentice-Hall.
Dozier, D.M. (1984). Program evaluation and roles of practitioners. Public
Relations Review, 10, 13-21.
Dozier, D.M. (1985). Planning and evaluation in public relations practice.
Public Relations Review, 11, 17-25.
Dozier, D.M. (1990). The innovation of research in public relations practice:
Review of a program of studies. Public Relations Research Annual, 2, 3-28.
Dozier, D.M., & Ehling, W.P. (1992). Evaluation of public relations programs:
What the literature tells us about their effects. In J. Grunig (Ed.),
Excellence in public relations and communications management (pp. 159-184). New
York: Erlbaum.
Ehling, W.P. (1992). Estimating the value of public relations and communication
to an organization. In J. Grunig (Ed.), Excellence in public relations and
communications management (pp. 617-638). New York: Erlbaum.
Ferguson, M.A. (1984, August). Building theory in public relations:
Interorganizational relationships as a public relations paradigm. Paper
presented to the Public Relations Division, Association for Education in
Journalism and Mass Communication.
Flay, B.R., & Cook, T.D. (1981). Evaluation of mass media prevention campaigns.
In R.E. Rice & W. Paisley (Eds.), pp. 239-264. Beverly Hills, CA: Sage.
Files, J.A. (1984). Study shows that public information campaigns work. Public
Relations Journal, January, 8-9.
Finn, D. (1984). The elements of sound public relations planning. New York, NY:
The Forum.
Franzen, R.S. (1977). An NBS internal communications study: A comment. Public
Relations Review, 3(4), 83-88.
Geduldig, A. (1986). Measure for measure. Public Relations Journal, 42(4), 6-7.
Grass, R. C. (1977). Measuring the effects of corporate advertising. Public
Relations Review, 3(4), 39-50.
Grunig, J.E. (1977a). Measurement in public relations. An overview. Public
Relations Review, 3(4), 5-10.
Grunig, J.E. (1977b). Evaluating employee communication in a research operation.
Public Relations Review, 3(4), 61-62.
Grunig, J.E. (1983). Basic research provides knowledge that makes evaluation
possible. Public Relations Quarterly, Fall, 28-32.
Grunig, J.E. & Hickson, R.H. (1983). An evaluation of academic research in
public relations, Public Relations Review, Spring, 31-43.
Grunig, J.E. & Hunt, T. (1984). Managing public relations. New York: Holt,
Reinhart and Winston.
Grunig, J.E., & Stamm, K.R. (1973). Communication and coorientation of
collectivities. American Behavioral Scientist, March/April, 567-591.
Grunig, J.E., Grunig, L.A., Dozier, D.M., Ehling, W.P., Repper, F.C., & White,
J. (1991). Initial results of survey confirm value of communication and the
components of excellent communication. Report to the International Association
of Business Communicators Research Foundation. San Francisco: IABC.
Grunig, J.E., Dozier, D.M., Ehling, W.P., Grunig, L.A., Repper, F.C., & White,
J. (1992). Excellence in public relations and communication management. New
York: Erlbaum.
Grunig, L.A., Dozier, D.M., Grunig, J.E. (1994). IABC excellence in public
relations and communication management. Phase 2: Qualitative study. San
Francisco: IABC Research Foundation.
Hause, D. (1993). Measuring the impact of public relations. Electronic
techniques improve campaign evaluation. Public Relations Journal, 49,(2), 14-21.
Hesse, M.B. (1976). A coorientation study of Wisconsin state senators.
Journalism Quarterly, 53, 626-633, 660.
Heitpas, Q. (1984). Planning. In B. Cantor (Ed.), Experts in action: Inside
public relations (pp. 291--301). New York, NY: Longman.
Holmes, P.A. (1996, April 18). Evaluation: What is more important that who.
Inside PR, 2.
Hon, L.C. (1997). What have you done for me lately? Exploring effectiveness in
public relations. Journal of Public Relations Research, 9, 1-30.
Houlder, V. (1994, September 8). Getting the measure of PR. Financial Times of
London.
Hyman, H., & Sheatsley, P. (1947). Some reasons why information campaigns fail.
Public Opinion Quarterly, 11, 413-423.
Jacobson, H.K. (1980). Guidelines for evaluating public relations programs.
Public Relations Quarterly, Summer, 7-10.
Johnson, B. (1994). Prove public relations affects the bottom line. Public
Relations Journal, 50(4), 40,31.
Kendall, R. (1996). Public relations campaigns strategies. Planning for
implementation. New York: HarperCollins.
Kirban, L. (1983). Showing what we do makes a difference. Public Relations
Quarterly, 28(3), 22-27.
Knobloch, S. (1996, June). What would have happened without public relations?
Time series analysis as a tool of evaluating public relations. Paper presented
to the Public Relations Division, Association for Education in Journalism and
Mass Communication, Anaheim, CA.
Knodell, J.E. (1976). Matching perceptions of food editors, writers, and
readers. Public Relations Review, 2, 37-56.
Koestler, F. (1977). Planning and setting objectives: Guidelines for non-profit
organizations. New York, NY: Foundation for Public Relations Research and
Education.
Larson, M.A., & Massetti-Miller, K.L. (1984). Measuring change after a public
education campaign. Public Relations Review, 10, 23-32.
Leffingwell, R.J. (1975). Breakthrough in application of social science
knowledge by public relations counselors. Public Relations Quarterly, 20, Fall,
8-9.
Lerbinger, O. (1977). Corporate uses of research in public relations. Public
Relations Review, 3(4), 11-19.
Lesly, P. (1986). Multiple measurements of public relations. Public Relations
Review, 12(2), 3-8.
Lesly, P. (1991). Lesly's handbook of public relations and communications.
Chicago: Probus.
Lincoln, Y.S., & Guba, E.G. (1985). Naturalistic inquiry. Beverly Hill, CA:
Sage.
Lindenmann, W.K (1988). Beyond the clipbook. Public Relations Journal, 44(12),
22-26.
Lindenmann, W.K (1990). Research, evaluation and measurement: A national
perspective. Public Relations Review, 16(2), 3-16.
Lindenmann, W.K. (1993). An effectiveness yardstick to measure public relations
success. Public Relations Quarterly, 38(1), 7-9.
Lindenmann, W.K. (1995, April 25). Measurement: It's the hottest thing these
days in PR. Speech given to the PRSA Counselors Academy, Key West. Available
from Ketchum Public Relations, 220 East 42nd Street, New York, NY, 10017.
Lindlof, T.R. (1995). Qualitative communication research methods. Thousand Oaks,
CA: Sage.
Lorimer, J.E. (1994, November 14). Measuring the effectiveness of your exhibit.
Advertising Age.
Macnamara, J.R. (1992). Evaluation of public relations: The Achilles Heel of the
PR profession. International Public Relations Review, 15(4), 17-31.
Marker, R.K. (1977). The Armstrong/pr data measurement system. Public Relations
Review, 3(4), 51-60.
Marzano, R. (1988). Evaluating public relations. In R.E. Heibert (Ed.),
Precision public relations (pp. 267-290). New York: Longman.
McElreath, M.P. (1977). Public relations evaluative research: Summary
statement. Public Relations Review, 3(4), 129-136.
McGuire, W.J. (1973). Persuasion, resistance and attitude change. In I. de Sola
Pool & W. Schramm (Eds.), Handbook of communication (pp. 216-252). Chicago:
Rand-McNally.
Measurement driving more PR programs (1996, March 18). PR News, 52, 1-2.
Measurement popular in the U.S.--and for right reasons. (1995, January 23). PR
News, 51(4).
Mendelsohn, H. (1962). Measuring the process of communication effects. Public
Opinion Quarterly 26(3), 411-416.
Mendelsohn, H. (1973). Some reasons why information campaigns can succeed.
Public Opinion Quarterly, 37(1), 50-61.
Nager, N.R., & Allen, T.H. (1984). Public relations goals and objectives. New
York: Longman.
O'Keefe, G.J., & Reid, K. (1990). The uses and effects of public service
advertising. Public Relations Research Annual, 2, 67-91.
O'Neill, H.W. (1984). How opinion surveys can help public relations strategy.
Public Relations Review, 10, 3-12.
Pavlik, J., Vastyan, J., & Maher, M.F. (1990). Using readership research to
study employee views. Public Relations Review, 16(2), 50-60.
Pritchitt, J. (1994). Public relations evaluation: Professional accountability.
Geneva, Switzerland: The International Public Relations Association, Gold Paper
No. 11.
Reeves, B. (1983). Now you see them, now you don't: Demonstrating effects of
communication programs. Public Relations Quarterly, 28(3), 17-21, 27.
Reeves, B., & Ferguson-DeThorne, M.A. (1980). Measuring the effect of messages
about social responsibility. Public Relations Review, 6, 40-55.
Rogers, E.M. & Storey, J.D. (1987). Communication campaigns. In C. Berger & S.
Chaffee (Eds.), Handbook of communication science (pp. 871--886), Newbury Park.
CA: Sage.
Rosser, C., Flora, J.A., Chaffee, S. H., Farquhar, J.W. (1990). Using research
to predict learning from a PR campaign. Public Relations Review, 16(2), 61-77.
Ruff, C. (1968). Measurement of publicity effectiveness by inquiry analysis.
American Management Association Bulletin No. 110, Measuring and evaluating
public relations activities. New York: AMA.
Scriven, M. (1967). The methodology of evaluation. In R.W. Typer, R.M. Gagne, &
M. Scriven (Eds.), Perspectives of curriculum evaluation (pp. 39-83). AERA
Monograph Series on Curriculum Evaluation 1. Chicago: Rand McNally.
Seeking minimum standards for measuring public relations effectiveness. (1996,
November). Summary of Evaluation Summit, available from Ketchum Public Relations
Research and Measurement Department, New York, NY.
Setting benchmarks leads to effective programs. (1993, February). Public
Relations Journal, 16.
Snow, D.A. (1980). The disengagement process: A neglected problem in
participant-observation research. Qualitative Sociology, 3, 100-122.
Spradley, J.P. (1979). The ethnographic interview. New York: Holt, Rinehart &
Winston.
Spradley, J.P. (1980). Participant observation. New York: Holt, Rinehart &
Winston.
Stamm, K.R. (1977). Strategies for evaluating public relations. Public Relations
Review, 3(4), 120-128.
Stegall, S.K., & Sanders, K. (1986). Coorientation of PR practitioners and news
personnel in education. Journalism Quarterly, 63, 341-347.
Strenski, J.B. (1981). New concerns for public relations measurement. Public
Relations Journal, May, 16-17.
Strenski, J.B. (1983). Techniques for measuring public relations effectiveness.
Public Relations Quarterly, Spring, 21-24.
Suchman, E.A. (1967). Evaluation Research. New York: Russell Sage Foundation.
Sutton, S. (1988, March 29-April 3). Are we measuring the rights things? PR
Week, 9.
Swinehart, J.W. (1979). Evaluating public relations. Public Relations Journal,
35, 13-16.
Tichenor, P.J., Donohue, G.A., & Olien, C.N. (1977). Community research and
evaluating community relations. Public Relations Review, 3(4), 96-109.
Tirone, J.F. (1977). Measuring the Bell System's public relations. Public
Relations Review, 3(4), 21-38.
Tucker, K. & Shortridge, R. (1994, October/November). Behavioral principles in
public relations. Public Relations Journal, 50(8), 56-57.
VanLeuven, J.K. (1989). Theoretical models for public relations campaigns. In C.
Botan & V. Hazleton (Eds.), Communication theory in public relations. New York,
NY: Erlbaum.
VanLeuven, J.K., O'Keefe, G., & Salmon, C. (1988, July). Effects-based planning
for public relations campaigns. Paper presented to the Public Relations
Division, Association for Education in Journalism and Mass Communication,
Portland, OR.
Watson, T. (1994). Public relations evaluation: Nationwide survey of practice
in the United Kingdom. Paper presented to the International Public Relations
Research Symposium, Lake Bled, Slovenia.
Watson, T. (1995). Evaluating public relations: Models of measurement for
public relations practice. Paper presented to the International Public
Relations Research Symposium, Lake Bled, Slovenia.
Webb, E.J., & Campbell, T. (1973). Experiments on communication effects. In I.
de Sola Pool (Ed.), Handbook of communication. Chicago: Rand McNally.
Weiner, M. (1995, March). Put client 'values' ahead of newsclips when measuring
PR. PR Services, p. 48.
Weiss, C.H. (1972). Evaluation research: Methods of assessing program
effectiveness. Englewood Cliffs, NJ: Prentice-Hall.
Wiesendanger, B. (1994). A research roundup. Public Relations Journal, 50(5),
23-27.
Williamson, D.A. (1995, October 16). Inching toward web measurement. Advertising
Age, 48.
Winokur, D., & Kinkead, R.W. (1993, May). How public relations fits into
corporate strategy. CEOs assess the importance of PR today and in the future.
Public Relations Journal, 49(5), 16-23.
Wright, D.K. (1979). Some ways to measure public relations. Public Relations
Journal, July, 17-18.
Wylie, F.W. & Slovacek, S.P. (1984). Public relations evaluation: Myth, option,
or
necessity? Public Relations Review, 10(2), 22-27.
[1] Evaluation as used here refers mostly to summative evaluation, or
determining whether program goals and objectives have been met (Scriven, 1967).
[2] Related to prescriptive research about public relations evaluation is
McElreath's (1977) discussion of theoretical models of evaluation research. He
posited two frameworks for understanding public relations evaluation.
The first of these, a contingency model, is concerned with conditions that
predict whether public relations evaluation will occur or not. This model
implies that degree of environmental uncertainty and problem recognition within
the organization are two predictors of when and how public relations evaluation
will occur.
By contrast, an open systems model emphasizes the dynamics between an
organization and its environment. Public relations research and evaluation
contribute to organizational effectiveness by supplying information
organizations need to successfully adapt to their environment.
[3] These other managers were recruited by asking practitioners if they thought
the CEO or another top manager would be willing to be interviewed about the
research questions.
[4] Member checks also may involve people who are not actually involved in the
research project but are members of the "culture" being investigated.
[5] Of course, this may be a function of selection bias. No doubt the CEOs and
other managers willing to be interviewed about public relations probably value
public relations more than others might.
[6] And, this question may have produced some normative responses. Knowing
that the study was about evaluation, practitioners inadvertently could have
exaggerated their plans. However, most gave rather detailed descriptions of
efforts already underway.
[7] A related issue is that few mentioned exploiting the availability of
secondary research, which is often inexpensive or free. Perhaps more emphasis
should be given in research methods courses on finding and using this kind of
information.
|
