Fourth Generation Evaluation: Implications for Public Relations
Education
Debra A. Miller, Ed. D., APR
Associate Professor
Advertising and Public Relations
Florida International University
Academic 2, Room 328A
3000 NE 145th Street
North Miami, Florida 33181
(305) 919-5629
(305) 919-5215
[log in to unmask]
A paper submitted to the
Public Relations Division. AEJMC Call for Papers
Topic: Teaching Public Relations
April 1, 1997
Abstract
The topic of evaluating student learning outcomes continues to
receive attention from public relations educators. Although
quantitative approaches are still widely used, what has not been
addressed is an effective way of qualitatively assessing the achie
vement of instructional objectives, student attitudes about course
content and teaching effectiveness. This paper discusses results of a
study which tests the use of a fourth generation evaluation method
used during a semester length course entitled "Multicultural
Communications" and suggests implications for public relations
educators.
Introduction
For many years quantitative methods have been the mainstay of
evaluation by
academicians and training professionals. Even today, both groups
continue to use quantitative
techniques to assess the achievement of instructional objectives,
student attitudes about course
content, and teaching effectiveness. Noted instructional design and
evaluation scholars such
Briggs and Wager (1980),Dick and Carey (1985), Gangue, Briggs and
Wager (1988) and
Guba and Lincoln (1989) support the use of quantitative as well as
qualitative approaches
however quantitative methods remain the preferred choice. The
reluctance of the academy and
the profession to adopt qualitative evaluation methods has resulted
in the lost of a plethora
valuable information which could prove useful to educators and
students.
In 1993, a study was conducted to investigate the feasibility of
using fourth generation
evaluation during the process of instruction. A semester length
course entitled "Multicultural
Communications (PUR 5406/PUR 4934)," designed in response to the need
for
communications profession, specifically public relations to produce
well-trained culturally
sensitive practitioners for the workforce and the market place. A
revised pause model
consisting of three one-on-one in-depth interviews conducted outside
of the class, three
reflections periods during the class and a self-reflective essay
prepared one week before the end
of the course was analyzed. Narrative and graphic summaries of
participant responses
produced significant results, however only the narrative results are
discussed.
The revised pause model was found to be an effective evaluation
method for use during
multicultural education in public relations under certain conditions
as perceived by the
participants in the study. Participant self-perceived behavior
change and knowledge acquisition
was identified through use of the revised pause model. Study
results suggest that by using the
revised pause model of evaluation, instructors teaching multicultural
education, i.e. public
relations courses in schools of journalism and mass communication is
yet another way of
enhancing their ability to become both the researcher and the
research subject. In addition, the
introduction of this qualitative model was found to be a more
effective way of generating
participant involvement and introspection. Several implications
surface from the results of this
study that are particularly relevant to public relations educators.
Literature Review
Fourth generation evaluation as defined by Guba and Lincoln (1989),
insists that as
evaluation involves humans as clients, as stakeholders and as
information sources. This
process of evaluation requires the evaluator to interact with those
humans in a manner
respecting their dignity, integrity, and privacy. Of course
conventional evaluators have been
careful about ethics; rules respecting fully informed consent, harm,
deception, and
privacy/confidentiality are well understood and mostly practiced.
The issue here is full participative involvement which takes place
when the
stakeholders and others who may be drawn into the evaluation are
welcomed as equal partners
in every aspect of design, implementation, interpretation, and
resulting action of an evaluation;
that is they are accorded a full measure of political parity and
control. It means that human
participants are accorded the privilege of sharing their
constructions and working toward
common, consensual, more fully informed and sophisticated, joint
construction. They are
accorded a full measure of conceptual parity. It means that the
participants continue to be
treated as humans, not as subjects of experimentation or objects of
study. This new form of
evaluation empowers, enfranchise and fuses the act of evaluation and
its follow-up into one indistinguishable whole. The mode of fourth
generation evaluation meets
the challenge, at least to a first level of approximation. Before
this emergent new form of
evaluation can be given adequate consideration it must be tested
utilizing assorted subject
matter and instructional design. There is an increasing need for an
alternative to the quantitative
paradigm. The belief exists that the quantitative paradigm has
reached the limits of its
effectiveness and there is a need for a more holistic approach. An
emerging qualitative
paradigm must be included as a legitimate alternative in evaluation
if the holistic approach is to
be achieved. "To approach evaluation scientifically is to miss
completely its fundamental
social, political, and value-oriented character" (Guba and Lincoln,
1989, p. 31). The first three
generations of evaluation processes as defined by Guba and Lincoln
(1989), were indeed
scientific. Collection of data from individuals was not
systematically possible until the
development of appropriate quantitative instruments of the sort
characterized
in the first generation. Evaluation would have stagnated at that
level had not the second
generation shown the way to evaluate the many non-human evaluands
(the entity to be
evaluated) as well--the programs, materials, teaching strategies,
organizational patterns, and
treatments in general. The third generation required that evaluation
lead to judgment, both
about an evaluand (the entity to be evaluated) and the methodDits
inner or intrinsic value--and
about its worth--its extrinsic or contextual value. "All three
generations, as a group, have
suffered and continue to suffer from certain flaws or defects
sufficiently serious to warrant
raising whether additional refinements--or even a complete
reconstruction--may well be
needed" (Guba and Lincoln, 1989, p. 31).
In 1991, Liebowitz, introduced a new form of fourth generation
evaluation as a way
for training and development professionals to pause in the midst of
their training to critically
reflect upon how their immediate actions relate to their personal and
professional aspirations.
The major components of the Liebowitz Pause Model (1991) are
reflection periods; interviews;
split sheet journals; and member checks. To capture and articulate
actual learning outcomes as
presented in the pause model Liebowitz (1991) used four techniques:
1. Learner self-reports, split-sheet journals, writing tasks, self
reflective essay;
2. Interviews, reflective conversations between the trainer and the
learners;
3. Reflection periods, open class discussions governed by reflective
contracts, virtual worlds; and
4 Observations by either learners or the trainer, which are noted and discussed
inform all one-to-one (not in an interview format) within two or
three hours of the observation (p. 151).
Liebowitz (1991) did not actually test the model he designed. He
did offer potential
strategies for testing the model. The revised pause model tested in
the 1993 study was
developed after a comprehensive review of his strategies. A true test
of the model is its
usefulness to adult education and training professionals, as well as
communications educators.
Usefulness as defined by Liebowitz is the model's ability to enable
practitioners to deal with
complexity, uncertainty, instability, uniqueness and values education
and training. Although
the Liebowitz pause model in its entirety is certainly useful, a
revised pause model seemed
more appropriate for the subject matter at hand, the participants and
the time constraints of
using the model during actual instruction. The revised pause model
was designed to make use
of those elements of qualitative research that are deemed feasible
for replication by
communications educators.
A complete replication of the Liebowitz pause model is perceived as
time consuming
and difficult studying large populations. It was determined that
rather than use all of the
aspects of the Liebowitz (1991) pause model the following components
were selected as the
major components of the revised pause model: Reflection periods,
interviews and self-reflective
essays are more manageable and useful for the subject matter being
presented. The
components were chosen because they could be used to gather a wealth
of in-depth
information can easily be incorporated into a course in which
students are asked to participate
in activities inside and outside of the classroom.
Multicultural Communications (PUR 5406/PUR 4934) - The Course.
Initially begun as a
graduate-level seminar, then later cross-listed with senior-level
undergraduate seminar. the
course Multicultural Communications (PUR 5406/PUR 4934) examines the
nuances
of cultural characteristics and behavior that enables public
relations practitioners to more
effectively communicate various messages. The impact of culture on
communications is also
studied, as well as the ways in which perception, prejudice, myths
and stereotypes shape
attitudes. The course includes a series of structured exercises
requiring students to form dyads
and triads. At the end of each exercise and module, the larger group
reconvenes for discussion
and summary.
To achieve the objective of creating practitioners trained in
Multicultural
Communications (PUR 5406/PUR 4934), the course was divided into three
segments: Cultural
Awareness and Sensitivity, South Florida: America's Boiling Pot, and
Intercultural
/International Communication. A host of textbooks were reviewed
before one required and
two recommended texts were selected. The required text is Race and
Ethnic Relations:
American and Global Perspectives, 4th Edition (1993) by Martin N.
Marger. The
recommended texts are Intercultural Communication: A Reader, 5th
Edition (1988) by Larry
A. Samovar and Richard E. Porter, and Majority and Minority: The
Dynamic of Race and
Ethnicity in American Life, 4th Edition (1985) by Norman R. Yetman.
Segment One, Cultural Awareness and Sensitivity includes three
training modules:
y Module I, Culture and Cultural Context, developed to help students
examine cultural and interpersonal relationships as they function in
the work world. Particular emphasis was placed on understanding
cultural characteristics and behavior. At the end of this module,
students should be able to define culture and define
multiculturalism, understand culture, race, and ethnicity from
different perspectives, be aware of the value of culture, recognize
the negative attitudes about cultures which foster inappropriate
behavior, and show and experience the commonalties of participants'
different cultures. Module I also provides the students with an
opportunity to experience each other's culture. Includes three
structured exercises requiring students to form pairs, dyads and
triads: Paired Cultural Interviews and Introductions
(self-identified cultural heritage); Circle of Culture (comparison of
participant culture/gender to dominant culture); and, Where is this
from? (culture/country artifact identification).
y Module II, Intercultural Communication, examines the impact of
culture on communication. Particular emphasis is placed on
presenting the impact of language, verbal and non-verbal, on
intercultural interaction. At the end of this module, students
should be able to define communication, intercultural communication,
dialect, prejudice and perception. Students should also be able to
explain the differences between perception and reality, and describe
the comfort and discomfort level created among and between cultural
groups when some of the students are communicating in a language they
do not understand. Includes one structured exercise requiring
students to form dyads to list all the words phrases or
communications styles which hinder effective communication with
African-Americans, Latinos, Asians and Native Americans (ALANA) group
members; to identify at least three personal communication factors
which obstruct effective cultural communication; to recognize that
people solve problems and manifest behaviors according to cultural
learning. How these behavior are exhibited (verbal/nonverbal)
affects the process; and to developing strategies to overcome trigger
words, phrases and actions that negatively impact our own
intercultural communication and factors which may impact on
intercultural communications (environment, purpose, time,
interpersonal factors, personality, situational factors,
communications style, language/dialect/accent, world view, filters
and culture.
y Module III, Myths and Stereotypes, helps students become aware of
how perceptions, prejudice, myths, and stereotypes shape attitudes
and to explore participants' attitudes about African-Americans,
Latinos, Asians, and Native Americans (ALANA). At the end of this
module, students will be able to define prejudice, myths, and
stereotypes. They will be able to give at least three examples of
myth about ALANA group members, identify behaviors that may evoke
negative stereotypes about ALANA group members, discuss at least one
strategy to use in approaching someone who has accepted a negative
stereotype about members of ALANA groups, identify one type of
behavior that conflicts with the students' values, and identify at
least two possible outcomes for persons who are stereotyped by the
institutions of which they are a part. Includes one structured
exercise requiring students to list and identify at least three
examples of myths and stereotypes as they related to ALANA and
non-ALANA group members, preparation and performance in a three
minute skit in which each member must portray stereotypical behavior
of a cultural group other than his/her own situations which occur in
the workplace and or in everyday life. A collection of videotapes
are aired during this module including, "Beyond Hate" with Bill
Moyers, "The Color of Your Skin," and "Cultural Communication: A
Video on Media Stereotypes."
Segment two, South Florida: America's boiling pot. A cadre of
culture-specific subject matter
experts are invited to make presentations. Speakers are asked to
cover the origin of the
ethnic/cultural/racial group in South Florida and issues of concern
to this group locally and
nationally for an hour including time for questions. Identification
of the communication
channels appropriate in reaching these groups, issues of salience to
these groups are discussed,
appropriate organizations which represent these groups locally,
nationally and internationally
are discussed.
Segment Three Intercultural/International communications. A series
of quest speakers,
assigned readings, and in-class exercises are used in this segment of
the course.
Representatives from corporations, agencies, and nonprofit
organizations that are responsible
for communicating effectively with audiences in Central and Latin
America, South America,
the Caribbean, Europe, and Japan. The speakers discuss public
relations case studies
conducted outside of the United States.
Assignments. The first assignment requires students to complete an
annotated bibliography of
readings from research journals, periodicals, and major newspapers on
the topic areas
discussed in class. Assignment two requires students to prepare a
20-page briefing paper on the
top three issues of concern locally and nationally to the
multicultural groups discussed in class.
The final project asks students to develop a campaign designed to
increase awareness among
ALANA group members on one of four issues: organ donations,
recycling, aids prevention,
and U. S. immigrations policies. Students were given a choice of two
other assignments.
Choice Two asked students to develop a multicultural exhibit for a
diverse ethnic community
and Choice Three asked the students to design a week-long training
session for corporate
employees to increase their level of cultural awareness and
sensitivity.
The Fall 1993 Study
Data Collection Methods. A pilot study was conducted during the
Spring 1993 offering of
Multicultural Communications . The pilot was a dress-rehearsal for
a component of the
revised pause model tested in the Fall 1993. The participants were
interviewed between class
meetings for 30 minutes or less. With the permission of the
participants, all of the interviews
were audio-taped. All of the participants were asked the eight
questions. The coding process
for this study was determined based on a set of key concepts included
in multicultural
education and instruction, course objectives, and intended/unintended
learning outcomes. Six
participants interviewed were asked predetermined questions in the
same order.
Pilot study results indicated that the one-on-one in-depth interview
component of the
revised pause model as a qualitative method of evaluation did not
suggest that the quantitative
evaluation models should be abandoned. The results did, however
provide support for
qualitative models being considered a full-fledged comprehensive
alternative to quantitative
models. When the qualitative model is used specifically in
evaluating multicultural instruction
utilizes the most important aspect of this type of
education---critical thinking.
The pilot study accomplished three objectives. First, it has
showed that
communication educators, specifically public relations educators in
need to pause and
critically reflect upon the relationship of their actions and
aspirations during instruction. Both
the educators and their students can benefit from such reflection in
action. Second, although
this pilot study tested only one component of the revised pause model
that can be used when
communication educators/educators can use when they do pause to
critically reflect. And,
third this pilot study suggested a viable method for teaching
communication students to
become more culturally aware and sensitive and how to communicate
effectively in our
culturally diverse world.
The pilot study results also indicated that the course
"Multicultural Communications
(PUR5406/PUR 4934) created an environment which encouraged
interaction between the
learners and the teacher as well as with other learners. Students
appeared to like the diversity
of working in dyads and triads in structured in-class assignments.
Participants welcomed the
opportunity for discussion and active participation. The reality
based assignments were viewed
by participants as an attractive aspect of a course. The inclusion
of guest speakers who are
knowledgeable and willing to be candid in their remarks were also
rated as a high point in the
course.
The Actual Study. Multicultural Communications (PUR 5406/PUR 4934)
was offered once
again during the Fall 1993 semester, with the inclusion of a revised
pause evaluation model. A
group of 22 participants registered in the Fall 1993 course. All of
the students voluntarily
agreed to participate in the study. At the beginning of the first
class meeting each participant
randomly selected a number from 1-23. Throughout the remainder of
the study each
participant was identified by the number selected. Each participant
was interviewed twice
during a ten-week period for 30-45 minutes during the instructors
regularly scheduled office
hours. Thirty to forty-five days following the completion of the
course, participants were
interviewed once again. Two female graduate students/teaching
assistants not enrolled in the
course coded the data. With the permission the students in the course
each designation
reflection period and one-on-one in-depth interview was audio-taped,
transcribed and coded
within 48 hours. The results from the course with inclusion of the
revised pause
model into the course was analyzed and summarized in tabular and
narrative form, however
only the narrative information is presented in this paper. The
interviews were conducted by the
instructor for the following reasons:
1. the revised pause model is designed to allow for the maximum
interaction between the instructor and the students. It is only
through instructor involvement that evaluation can be conducted
during instruction;
2. the level of instructor involvement in administering the
components of the model is dictated by the model itself;
3. the instructor is provided with immediate information that can be
used to asses student attitudes, the achievement of instructional
objectives, intended and unintended learning outcomes and teaching
effectiveness; and,
4. the instructor involvement creates the opportunity for the teacher
to become the researcher and the research subject.
To qualitatively evaluate the effectiveness of the revised pause
model as well as the
course itself, students were asked to participate in reflection
periods after each exercise in the
course. All of the participants were asked to prepare self-reflective
essays for submission at the
end of the class. Each student was interviewed two times during the
course and once 30-45
days after the completion of the course. The three reflection periods
were audio-taped with the
permission of all of the participants at the end of each module. The
audio-tapes of the session
were transcribed and reviewed after each class. The tapes from each
reflection period were
transcribed and reviewed to uncover similarities, shifts in thinking
or direction and to identify
the use of key concepts. If at anytime there is an indication that
intended learning outcomes are
not being met, modifications were made to the module. Additional
structured exercises were
conducted and additional outside readings were assigned. Unintended
learning outcomes were
also identified.
The participants interviewed twice during the course for thirty
minutes and after the
completion of the course. The first interview was conducted after
the completion of Modules I
and II. Each of the first two interviews took place at a different
point in the course for each
participant. The second interview took place after the "South
Florida: America's Boiling Pot"
segment of the course. The third interview took place 30-45 days
after the completion of the
course to determine not only if learning outcomes were achieved, but
also to learn whether the
participants began to use what they have learned in the workplace, at
home and in the market
place. Although an Interview Guide was provided, participants were
allowed to ramble and/or
discuss areas not included in the guide.
[Table One Interview Guides Here]
Self Reflective Essay. All the participants were asked to write an
essay of 100 words or less
that included, but was not limited to the following: a description of
their opinions about the
course; a critique of the information learned; if and how they plan
to implement what was
learned; and, how they think the course could be improved or
revised. Copies of four
randomly selected essays are presented in Table Two.
[Table Two Selected Self-Reflective Essays]
Description of the Coding System. The coding process for this study
was determined based
on a set of key concepts included in multicultural education and
instruction, the objectives
Multicultural Communications (PUR 5406/4934), and the intended and
unintended learning
outcomes. A letter coding scheme was used because it is anticipated
that the responses during
the reflection periods, interviews and the self-reflective essays
would be detailed and simple
concepts would easily be identifiable.
[Table Three Coding System Here]
Actual coding was done on the transcribed interview copy. Each of
the graduate
research assistants was instructed circle the terms or phrases which
most closely answer the
questions asked and those key concepts identified on the coding
system sheet. An asterisk (*)
was used in responses in which the same concept was identified more
than once. Each
interview guide/contact sheet transcript. The name of the coder and
the date coded was
indicated at the bottom of the interview guide/contact sheet
transcript.
Results and Analysis
Qualitative data are thought to be exceedingly complex, and not
readily convertible
into standard measurable units of objects seen and heard. They vary
in level of abstraction, in
frequency of occurrence and in relevance to central questions in the
research. The analytic
procedures used in this research included organizing the data,
generating categories, themes,
and patterns and searching for alternative explanations of the data
collected.
One-on-one Interviews. The results of the three interviews in
tabular and narrative form
produced an extensive amount of valuable information. On all three
interviews the responses
collectively advance the idea that individual cultural identity is a
dynamic process which
operates in transactions among people of different cultures. All of
the responses support the
notion that culture and ethnicity, although distinct, are closely
related. That behavior manifest
itself in their homes, workplaces and communities over time.
Participants expressed self-
identified behavior change as a result of the course. They indicated
that they had become more
culturally aware and sensitive to diversity as a result of being
enrolled in the course.
The process of cultural awareness then continues as the participants
develop the
behavior of dealing with everyone as individuals rather than
stereotypes of their groups. In the
process of developing awareness and sensitivity to the variety of
people in the world, the
participants appeared to have gained the added benefit of enhancing
self-knowledge and
increasing their own objectivity. There was consensus among
participants regarding the
existence of racism in America. Even though all of the participants
agreed that racism will
always exist in America, they felt that racism is taught and is not
an innate human
characteristic. The inequities of society such as poverty,
discrimination, politics, and lack of
education were seen as contributors.
The format, speakers, exercises and assignments in the course were
viewed as
important by the participants. Their responses support the use of a
variety of instructional
methods and materials, a seminar format, reality based assignments,
an assortment of small
group exercises and activities, and use dynamic speakers,
knowledgeable of their subject
matter. Participants were also found to support the inclusion of
specified learning objectives
and the presence of an instructor who serves as a facilitator rather
than a teacher. All of the
participants indicated they would recommend the course to others
including students and the
public. They stressed that the course should be required for all
students entering the university. The course was seen as way of
preparing oneself to survive and prosper in our
multicultural society. The success of a course of this type requires
involvement by the
participants. Participant responses indicated that the course
provided an environment that
supports open discussion, questions and comments. In addition
participant responses also
indicated that the tenets of multicultural education and instruction
were imparted in a manner
which made them feel they could make comments and ask questions.
Participants reported that
they expected to walk away from the course with increased cultural
awareness and sensitivity
as well as an ability to effectively communicate with diverse
multicultural groups. They
indicated that the course had exceeded their expectations and had
armed them with knowledge
and skills to help them respond constructively to the changes in
their lives, workplaces and
communities.
Participants' responses supported the use of the revised pause
model. The quantitative
evaluation method used provided feedback on learning outcomes,
teaching performance and
the reflective viewpoints from the students. The components of the
revised pause model was
seen by participants as a way of measuring what they learned without
being tested. Participants
felt they were empowered through this evaluation method and could
have an impact on the
direction of instruction in the course.
Reflection Periods. Three reflection periods were conducted during
the course at the end of
specified modules. Each reflection period lasted one hour. The
first reflection took place
following specific structured exercises. The objectives of the module
were summarized and a
dialogue took place between the instructor and all of the
participants. The instructor began the
discussion by asking participants a series of questions related to
the objectives of the course
and each module.
Throughout the three reflection periods participants made comments
which indicated a
recognition of cultural influences on behavior. They stated that
their behavior was most
influenced by culture at home, school and work. Participants
conveyed a sense of increased
cultural awareness and sensitivity to diversity throughout the
reflection periods. Participants
alluded to the fact that the course had a positive impact on their
behavior and once again that
self-perceived change in behavior was seen most at work, school and
home.
As a result of the discussions that took place in the reflection
periods the a section on
"American" culture was added to the course as well as group
exercises which focus on the
impact of language on culture. Module III, Myths and Stereotypes was
expanded to include
more small group exercises and discussions focusing on how each
negatively impacts behavior
toward cultures other than our own. Group consensus emerged during
the reflection periods
that racism exists and will continue to exist in America.
Participants felt that racist attitudes are
learned at home as children and perpetuated through stereotypes
presented in the media.
Throughout the three reflection periods participants mentioned the
speakers themselves, as
well as comments made by the speakers. They also made reference to
specific assignments,
exercises and the seminar format. On numerous occasions during the
reflection periods the
participants indicated that they would recommend this course not only
to other students but to
the general public. They were impressed with the information learned
and wished that the
course could have been offered earlier in their academic life. There
was group consensus that
the environment exists in the course that fosters group discussion,
questions and comments.
The reflection periods provided them with an opportunity to share, to
criticize and to learn
about themselves and the other members of the class.
Participants indicated during the three reflection periods that
their learning expectations
were met. They also stated that they learned more than they expected
especially from the
speakers, assigned readings, assignments and from other class
members. All of the participants
indicated that they liked the qualitative evaluation method used in
the study better than the
quantitative method they were required to complete at the end of the
course. Participants
referred to the interviews and reflection periods on numerous
occasions. Participants also
stated that the video tapes, textbook and exercises improved their
knowledge about the
concepts presented in the course. The discussion ended with
participants expressing pleasure \
with the fact that they could interact and share experiences with
their classmates without being
criticized or ridiculed for their opinions.
The reflection periods were seen as an opportunity to discuss
related issues and to
reflect on the objectives of the course, and the just completed
module. Participants stated that
it helped them determine what they learned and what they intended to
do with the information
outside of the classroom. Group consensus indicated that
self-identified behavior change and
knowledge acquisition had taken place as a result of the course. On
only one occasion the
course was revised to address concerns raised in the reflection
periods. Participants expressed
a preference for the seminar format, exercises, speakers, assignments
and instructional methods
used in the course. They also expressed a willingness to recommend
the course to other
students as well as a preference for use of qualitative evaluation
method to evaluate the teacher
and the course.
Participants indicated that their learning expectations were met and
exceeded,
and that they had become more culturally aware and sensitive to
diversity as a result of the
course. The participants indicated that the course had succeeded in
preparing them to respond
constructively to the changes in their lives, workplaces and
communities. The discussion
emphasized the idea that individual cultural identity is a dynamic
process which
operates in transactions among people of different cultures.
Participants responded by
providing definitions of the culture and multiculturalism. They
expressed an understanding of
culture, race and ethnicity from different perspectives and an
awareness of the value of culture.
Throughout the discussion participants shared their recognition of
negative attitudes about
cultures which foster inappropriate behavior. Participants also
recognized that people
problem-solve and manifest behaviors according to cultural
learning.
Self-Reflective Essays. A self-reflective essay consisting of 100
words or less was requested
and completed by the participants one week prior to the completion of
the course. They were
asked to complete the following statement. "This class will be over
in one week. Now that the
course is ending, I feel....." An essay was submitted by all of the
participants. All of the
participants communicated self- expressed increased awareness and
sensitivity to diversity.
Participants also stated that they felt the course positively
impacted their behavior. A majority
of the participants included statements about the exercises,
speakers, assignments and the
format. A few of the participants alluded to racism and what this
course could do to improve
racial relations. Several of the respondents indicated that their
learning expectations were not
only met but exceeded. While only four participants mentioned that
they would recommend
the course to others in their essay, five participants stated that
they felt comfortable making
comments and answering questions during the course.
Participant responses included comments about what they learned as a
result of their
participation in the class, descriptions about what they intended to
do with the information they
learned, recommendations about how the course should be revised or
improved, format,
exercises and assignments, personal reflections about their feelings
while writing the essay, how
the course had affected their behavior and comments about the
instructor. Participants did not
make reference to their specific cultural identity in the essays.
Participants used the essays as an opportunity to address concerns
not raised in
the interviews or the reflection periods. They related thoughts,
feelings and behaviors from
outside of the classroom to events which had occurred during the
course. In most cases,
participants revealed their feelings and attitudes, pausing to
critically reflect on the information
learned from the course. The essays provide valuable secondary
information which adds
another dimension to the data obtained from the other components of
the revised pause model.
Conclusions
Analysis of the data collected and presented in tabular and
narrative form confirm that
fourth generation evaluation is viable for use in a public relations
such as course, Multicultural
Communications (PUR 5406/PUR 4934). The analysis determined that the
revised pause
model can be used in a course such as Multicultural Communications
(PUR 5406/PUR 4934)
under certain conditions. The most desirable conditions are as
follows:
1. The participants are culturally, ethnically and demographically
diverse. The more diversity that is present among the participants,
the greater the opportunity for divergent opinions, attitudes,
experiences and perspectives.
2. The class size does not exceed twenty-five students. Larger
enrollments may detract from the instructor's ability to successfully
implement the components of the model.
3. The course is taught for a minimum of twelve weeks. Multicultural
Communications (PUR 5406/PUR 4934) (PUR 5406/4934) was designed to be
a semester-length course to allow for comprehensive instruction in
the tenets of multicultural education as well as the principles of
communications with ethnically and culturally diverse audiences.
4. Class meeting time of two or more hours. The content of the
course includes an assortment of small group exercises, speakers and
participant interaction and discussion as well as reflection periods
and cannot be done successfully in less that two hours.
5. A variety of instructional methods and materials are used. This
is necessary to address the different learning styles of participants
in the course and to create an environment that promotes knowledge
acquisition for all of the learners.
6. The course is taught in a seminar format is with participants
seated facing each other in a semi-circle. The seminar format is
preferable because students know in advance they will be expected to
participate in the class in a variety of ways. The semi-circular
seating encourages interpersonal communication. Participants are
forced to talk to one another, rather than at one another.
7. An environment is created that supports open discussion, questions
and comments. This type of environment facilitates the instruction
of multicultural education and communication, and the implementation
of the revised pause model.
8. Reality based assignments. Assignments should be given that
require participants to use newly acquired skills as well as those
they already possess. The assignments should resemble those the
participants will be expected to complete in the work place and the
marketplace.
9. Dynamic speakers, knowledgeable of their subject matter are
presented. It is important that participants are exposed to subject
matter experts who are good speakers and who are comfortable
addressing issues of salience to their ethnic and/cultural group.
10. An assortment of small group exercises and activities. People
learn differently. The inclusion of small group exercises and
activities encourages interaction, interpersonal communication and
creates an opportunity for everyone in the class to learn from
another.
11. Specified learning objectives. The inclusion of specified
learning objectives serves as a check and balance system for the
instructor and the student. It also provides a basis for negotiation
between the student and the instructor if specified learning o
bjectives are not met.
12. The instructor serves as a facilitator. The instructor becomes a
part of the instructional process. That individual is there to
facilitate learning among adults who bring their past and present to
the learning situation. In the role of facilitator, the instructor
and the students learn from one another, and learning becomes a
collaborative process.
The analysis revealed that the revised pause model is an effective
evaluation method for
use in multicultural education as perceived by the participants.
Participant responses in each
component of the substantiate this finding. The information obtained
through the use of the
revised pause model represent meaningful constructions that the
participants formed to make
sense of the situations they find themselves in as inhabitants of a
multicultural society. It is
important to note that participant involvement in the study was
voluntary. All 22 students
participated in the three interview sessions and the reflection
periods, and all prepared a self-
reflective essay. Student commitment to the process was demonstrated
by the fact that all
three of the interview sessions were conducted in addition to the
two-hour class meetings,
preparation of the self-reflective essay without any incentives and
the completion
of three written assignments required as part of the course.
Further review of the tables indicate that, with respect to the
effectiveness of the
model, each of the three components of the model accomplished what
they were designed to
accomplish. The interviews indicated what participants said they
learned as a result of their
participation in the course, and established the connection between
learning outcome and
classroom events. These responses obtained during the interviews
imply that people manifest
behaviors according to cultural learning. How this behavior is
exhibited affects the process.
The reflection periods confirmed that the participants thoughts,
feelings and behaviors
can be related to the world outside the classroom to events inside
the classroom. The candid
discussions provided the instructor with an opportunity for
spontaneous, unstructured
dialogue, allowing the instructor the opportunity to reveal her
thoughts and feelings and the
meanings she gives to connections made by the participants. It also
gave the instructor the
opportunity to negotiate and agree on changes that should be made to
the course and supplied
feedback on the progress of the agreed changes. The self-reflective
essay affirmed that
participants can indeed pause and critically reflect upon what they
experienced and learned as a
result of the course at home, work and school. The information
obtained verified the
relationship between predetermined learning outcomes and their
connections
with classroom activities. The concept of cultural awareness and
sensitivity is a reaffirmation
of the truth which many people already know. In interdependent
communities, different people
must learn to get along with each other to solve common problems and
satisfy common needs
inherent to community life.
Multicultural Communications (PUR 5406/PUR 4934) was predicated on
the
assumption that cultural awareness and sensitivity reduces prejudice
and induces respect,
through the encouragement of non-stressful interaction between
members of different cultures.
The process requires that participants take an active role in
developing a state of sensitivity to
the variations in the way people think, feel and act. This process
begins as they define their
own cultural heritage, examine their own values and assumptions, and
learn how these affect
their relationships with others, particularly with cultural groups
other than their own.
Participant responses indicated that an active role was taken in the
process.
An analysis the information obtained from the interviews,
reflection periods and self-
reflective essays reveal that the revised pause model is useful in
identifying self-perceived
change in the participants. The model was successful in capturing the
participants' feelings and
attitudes about the model and the course. Each element of the model
extracted information
from the participants at various levels and indicates that
participant feelings, attitudes and self-
perceived change are consistent throughout the three stages of the
model.
Implications For Public Relations Educators
The first and perhaps most significant implication is that fourth
generation, qualitative
evaluation methods, especially the revised pause model, is a
legitimate alternative to the
quantitative model. Although it requires more of the educator and
the student, the information
garnered from such a model is invaluable in enabling evaluation
during the process of
instruction. It also allows the public relations educator to
identify areas where modification and
revision is needed during the instruction.
It was found that the revised pause model can be used during the
process of
instruction, a second implication is that communications educators
can begin to use this model
as a way of engaging themselves and their students in a continuing
process of self-education.
By serving as both the researcher and the research subject, public
relations educators can move
beyond the traditional first, second and third generation evaluation
methods of the past to one
that offers a more holistic approach discussed by Guba and Lincoln
(1989). Fourth generation
evaluation seeks to empower the student and the instructor while both
are involved in the
process. Thus, the information obtained takes into consideration the
different contexts
(physical, psychological, social, and cultural) which influence the
learning process.
A third implication is that the value of the revised pause model
lies in its ability to solicit
self-perceptions and knowledge acquisition from the participants
while instruction is being
given. The model furnishes regular opportunities for feedback and
modification of instruction,
and allows the instructor to reveal and discuss his or her thoughts
and feelings and the
meanings given to connections made by the participants. The
information obtained allows the
instructor to understand the students' world and to determine with
what criteria they judge it as
well and the instruction given. Educators may find that certain
instructional methods are not
effective because of the type of learners in the class. People learn
differently when they are
learning to perform skills and when they are learning information to
change their attitudes.
The revised pause model encourages reflection which enables the
educator to correct
distortions in participant's beliefs and errors in problem solving.
Public relations educators
should find this an attractive feature of qualitative evaluation
particularly since it provides an
immediate assessment of whether the stated objectives of instruction
are being met. In fact, the
model supplies the educator with valuable information that can aid in
making revisions or
improvements to the instructional design while the instruction is in
process.
As the debate concerning teaching evaluations and their importance
continues in the
academy, a fourth implication of this study is that it offers the
educator an additional
opportunity to evaluate teaching performance. There is widespread
skepticism about students
as evaluators of teachers. Students have been criticized for not
understanding the importance of
teaching evaluations. Their estimations of teachers vary wildly,
their evaluations are not
reliable measures, they just go on feelings, what they like, what's
fun or entertaining, and, they
can be influenced by a good show and easy grades. There seems to be
agreement that there
should be some type of evaluation of teachers by students. And, even
the most cynical of
teachers knows that colleges operate in a competitive marketplace
where students are
consumers and without them the academy would not exist. Students can
learn without
teachers, but teachers cannot teach without students.
Good evaluation is more work, but there is merit in an evaluation
that provides useful
feedback on learning outcomes, teaching performance and reflective
viewpoints from students.
Qualitative evaluation, specifically the revised pause model
provides the teacher with evidence
that the student understood his or her goals and intentions in the
course, as well as the results
of those intentions and goals. Students possess more data because
they see many other
teachers in just as much detail. They are in an ideal position to
make informed comparisons
about the effectiveness of teaching methods. Students know more about
the success of
different styles of and approaches to teaching than the teacher in
many cases. Teachers need
discriminating feedback about particular practices and strengths and
weaknesses. This
information can be captured in vivid detail continuously by using the
revised pause model.
Collegiate administrators are accustomed to looking for the bottom
line and making
oversimplified verdicts about teaching effectiveness based on
quantitative data. While we
cannot underestimate the link between good teaching and good
learning, our current
preoccupation with a Neilsen rating approach to classroom success
encourages student
passivity and disregards the importance of what the learner must do
in order to learn
(Cholakian 1994). More useful results can be obtained from
qualitative evaluation using the
components of the revised pause model. The data obtained by using
the revised pause model
can do more to improve teaching than quantitative methods. And
finally, the revised pause
model offers an alternative which dignifies student evaluations of
teachers and makes the
process thoughtful and reflective rather than mechanical. Thus, the
teacher and the student
benefits.
A fifth implication is that the model encourages educators to
become facilitators rather
than teachers. Today's college classrooms are filled with adult
learners of varying ages,
experiences, learning styles and motives for learning. Therefore, in
order to meet the adult
education needs of those enrolling in universities today the educator
must become a facilitator.
This model encourages facilitation rather than teaching. It is
designed to help educators assist
adults making sense of and act upon the personal, social, occupation,
and political environment
in which they live. Facilitation is collaborative and the
qualitative evaluation tested in this study
is designed to facilitate cooperation and negotiation. When
confronting new learning
situations either in their personal lives or at work adults can have
difficulty adapting to change.
They may lack the ability to see new alternatives because of past
experiences or inhibiting
values, prejudices or assumptions.
A sixth implication is that the revised pause model helps adults
learn how to transform
their rich life experience from a potential barrier to change into a
basis for growth and lifelong
learning. When used the model becomes a method of helping adults
engage in the kind of
critical reflection that will enable them to respond constructively
to changes in their
lives, workplaces and communities. The components of the model
fosters critical
reflection and can facilitate learning by providing them with an
unencumbered opportunity to
recognize and reexamine deeply ingrained values. The model offers
participants the chance to
pause and critically reflect alone or with one other person or with a
group. What results from
that reflection is exploration, integration, self-scrutiny,
self-reflection and eventually learning.
Finally, the seventh implication is that the instructional design
used in this study is one
that communications educators can use to more adequately teach their
students to be more
culturally aware and sensitive to diversity. The tenets of
multicultural education can be
incorporated into a semester-length course targeted to reach
communications students at the
graduate and undergraduate level. If curriculum constraints prevent
the addition of another
course, portions of the course can be used in a pre-existing course.
Another important point must be stressed. This course does not
require an ALANA
(African-American, Latino, Asian, Native American) group member to
teach it, so the absence
of such a person should not be a deterrent for providing this sorely
need information to aspiring
communicators who must be able to communicate effectively in our
multicultural society. One
need only read the daily newspapers or look at the evening news to
see that this country has
regressed in terms of its ability to comfortably and effectively deal
with its multiculturalism.
Racism continues to threaten our lives and our futures. Discussions
and debates are taking
place, but we have not been able to identify viable solutions to the
problems that cause a
division among the races. The academy is in a unique position to
turn the rhetoric of
acceptance and tolerance into action.
Recommendations for Further Research
As the emphasis on improved teaching continues to permeate the
academy, alternative
forms of evaluation besides the quantitative methods of the past,
should be examined. The
findings of the study discussed in this paper do not suggest that
quantitative methods be
abandoned. It does, however present strong evidence in support of
fourth generation
evaluation and the use of a revised pause model during instruction
of a multicultural education
course in public relations.
The revised pause model should be tested using a variety of public
relations courses.
The subject matter used in this study covered a small portion of the
public relations body of
knowledge. It is likely that the results of testing model could be
radically different with subject
matter that does not require so much introspection and attitude
change. The fact remains that
as we look to the future, fourth generation evaluation should not be
overlooked or taken
lightly. If evaluation is truly a teaching and learning process as
it relates to public relations, the
academy and the profession has no choice but to build on what has
been learned from this
research.
References
Bogdan, R. C. & Biklen, S. K. (1992). Qualitative research: an
introduction to theory and methods.
Boston: Allyn and Bacon.
Briggs, L. J. & Wager, W. W. (1981). Handbook of procedures for the
design of instruction
(2nd ed.). Englewood Cliffs, New Jersey: Educational Technology Publications.
Brookfield, S. D. (1987). Developing critical thinkers. San
Francisco: Jossey- Bass.
Brookfield, S. D. (1986). Understanding and facilitating adult
learning. San Francisco: Jossey- Bass.
Carr, W. & Kemmis, S. (1983). Becoming critical: knowing through
action research. Victoria, Australia: Deakin University Press.
Cholakian, Rouben (1994). The value of evaluating. Academe,
September-October, pp. 24-26.
Dick, W. & Carey, L. (1978). The systematic design of instruction.
Glenview, IL: Scott, Foresman and Company.
Dick, W. & Carey, L. (1978). The systematic design of instruction
(3rd ed). Glenview, IL: Scott, Foresman and Company.
Gagne, R. M., Briggs, L. J. and Wager, W.W. (1988). Principles of
instructional
design (3rd ed.). New York: Holt, Rinehart and Winston, Inc.
Gagne, R. M., Briggs, L. J. and Wager, W. W. (1985). The Conditions
of learning and theory of instruction (4th ed.). New York: Holt,
Rinehart and Winston, Inc.
Gollnick, D. M. & Chinn, P. C. (1990). Multicultural education in a
pluralistic society. New York: Merrill.
Guba, E. G. & Lincoln, Y. S. (1989). Fourth generation evaluation.
Belmont, CA: Sage
Liebowitz, S. E. (1991). Developing a model that training and
development professionals may use to study their own practices.
(Unpublished Doctoral dissertation, Florida International University).
Marger, M. N. (1993). Race and ethnic relations: american and
global perspectives (5th ed.). Belmont, CA: Wadsworth Publishing Co.
Mezirow, J. (1991). Transformative dimensions of adult learning. San
Franciso: Jossey-Bass.
Mezirow, J. (1990). Fostering critical reflection in adulthood. San
Francisco: Jossey-Bass.
Miller, D. A. (1994). Fourth generation evaluation revisited: testing
a revised pause model. (Unpublished Doctoral dissertation, Florida
International University).
Miller, D. A. (1993). Multicultural communications: an annotated
bibliography Public Relation Society of America Foundation: New York.
Miller, D. A. (1993). Multicultural communications: sensitizing
public relations students to a multicultural society. Diversity in
Public Relations Education: Issues, Implications and Opportunities, A
Collection of Essays. Public Relations Society of America Educators
Section: New York.
Miller, D. A. & Foxworth, M. K. (1993). Multicultural journalism
education revisited, 1982-91. Journalism Educator, Summer.
Miller, D. A. (1991). Multicultural diversity: the communications
challenge of the 1990s. Public Relations Society of America: New
York..
Schon, D. A. (1983). The reflective practitioner: how professionals
think in action. New York: Basic Books.
Shor, I. and Freire, P. (1987). A pedagogy for liberation:
dialogues on transforming education. South Hadley, MA: Bergin and
Garvey.
Starr, J. (1989). The great textbook war. In H. Holtz, I. Marcus, J.
Soughtery, J. Michaels, and R. Peduzzi (Eds.), Education and the
american dream: conservatives, liberals and radicals debate the
future of education. Granby, MA: Bergin and Garvey.
Yetman, N. R. (1985). Majority and minority: the dynamic of race
and ethnicity in american life, (4th ed.). Boston, MA: Allyn and
Bacon, Inc.
Table One: Fall 1993 Study Interview Guide
1. I describe my cultural identify/gender/race/religion as..........
2. Cultural influences shape my behavior at work, at home and/or at school in
the
following ways.....................
3. You've been enrolled in PUR 5406/4934 for at least two or three weeks as a
result of this course I am or am not more culturally aware and sensitive to
diversity because........
4. What I have learned in this course has/has not affected my behavior in
dealing
with racial/ethnic groups other than my own at work, school or etc.,
because now
I.............
5. I strongly believe that racism will/will not always exist in America
because......................
6. Review the syllabus. Did you enjoy the course format, exercises, speakers
and
assignments? If so why or why not?
7. I would/would not recommend PUR 5406/4934 to other students
because............................
8. I do or do not ask questions or make comments in the class because I
feel...................................
9. I expect to walk away from this course having
learned..........................
10. In my opinion the qualitative evaluation used in this course
are..............................
Note: Only question three changed in Interviews Two ( six or eight weeks) and
Three
(now that the class is over)
Table Two: Fall 1993 Study Coding System
Question 1:
CI = Cultural Identity
CI-R = Race
AA/African-American
C/Caucasian
H/Hispanic
JA/Jamaican
WI/West Indian
IA/Irish American
A/Asian
PR/ Puerto Rican
CI-RE = Religion
C/Christian
CA/Catholic
BA/Baptist
P/Protestant
Question 2:
IB = Cultural Influences on Behavior
IB-H = At Home
IB-S = At School
IB-W = At Work
IB-sig = How? In what way?
Question 3
CA = Cultural Awareness and Sensitivity to Diversity
CA+ = Am more culturally aware and sensitive to diversity
CA- = Am not more culturally aware and sensitive to diversity
CA-sig = How? In what way?
Question 4
LO = Self-perceived Impact of Learning from Course on
Behavior
LO+ = Positive Impact
LO- = Negative Impact
LO-sig = Where? Location (home, work, school)
How?
Table Two: Fall 1993 Study Coding System (con't)
Question 5
RA = Feeling About Racism in America
RA+ = Positive Feelings/Racism will continue to exist
RA- = Negative Feelings/Racism will not continue to exist
RA-sig = Why do you think so?
Question 6
CD = Course Descriptors
CD-A = Assignments
CD-S = Speakers
CD-F = Format
CD-E = Exercises
CD-sig = Why do you think so?
Question 7
RC = Recommendations Re-grading the Course
RC+ = Would recommend the course to others
RC- = Would not recommend the course to others
RC-sig = Why? Why not?
Question 8
CP = Class Participation Descriptors
CP+ = Positive descriptors/Feel comfortable making comments and asking
questions
CP- = Negative descriptors/Do not feel comfortable making comments and asking
questions
CP-sig = Why? Why not?
Question 9
LE = Learning Expectations
LE+ = Expectations met
LE- = Expectations not met
LE-sig = Why? How?
Question 10
EM = Evaluation Method
EM+ = Liked the qualitative evaluation methods used in the course
EM- = Did not like the qualitative evaluation methods used in the course
EM-sig = Why? Why not
Table Three: Selected Self Reflective Essays
"This is the last class meeting for PUR 5406/4934. Now that the class is ending
I
feel .............".
Participant Number 17: "That I have learned a lot about other people's culture
and
heritage. The class had a
good mixture of students from different backgrounds and cultures. The
assignments
were challenging.
Professor Miller is a very enthusiastic person and takes great interest in the
development of her students."
Participant Number 8: "I have a better understanding of those ethnic groups that
were highlighted in the book.
What was more interesting was the members of the class. They were multicultural
and
able t complement
Professor Miller's presentations and plan for interaction. The overall
experience
has made me more keenly
aware of other cultural differences that impact my life. I have always dealt
with
different cultures but now I'm
more interested in interaction."
Participant Number 18: "An example of how I feel is the task of completing the
final project. It was nice to be
able to conjure up ideas on a subject matter I knew little about. I feel that
if
there actually was a "Facing the
Challenges of Diversity" training program, I could contribute. The class has
been
a real learning experience. It
seem dependent on the students and thus is predictable. The class should be a
requirement for both graduate and
undergraduate communication students."
Participant Number 3: "Good about taking this course. I was hesitant to take a
course prefixed PUR, but after
learning all the good things from this class, I am glad to hear how different
cultures do things and it was also fun
learning how similar we all are. The activities were fun and educational. The
way
the activities were planned
and conducted added to their education value. The discussions were frank and
this
added to the value of the
class. Due to the frankness and the open mindedness of class members we all
able
to contribute and collect
information.
|
