|
A Survey of Faculty Evaluation Practices
in Journalism and Mass Communication
Presented to:
Association for Education in Journalism
and
Mass Communication
Advertising Teaching Division
c/o Nancy Mitchell
College of Journalism and Mass Communication
45 Avery, University of Nebraska
Lincoln, Nebraska 8588-0138
Presented by:
Robert C Sitz PhD
&
Frank Thayer PhD
Dept of Journalism and Mass Communication
New Mexico State University
PO Box 300001, Dept 3J
Las Cruces, NM 88003
Ph: (505) 646-1539
Student Evaluation
Abstract
Although it was presumed that most schools formally and systematically
evaluate faculty for purposes of salary, tenure and promotion, the
literature revealed little current evidence regarding the extent or nature
of faculty participation. A lengthy survey questionnaire was mailed
to
members of the Association for Education in Journalism and Mass
Communication to learn about faculty evaluation practices.
This paper discusses the evolution of faculty evaluation, reviews several
survey questions, and reports that despite diverse course formats,
formal
classroom evaluation is widely practiced; the preferred method is
student
evaluation; and rating instruments are popular for both formative and
summative decisions.
Student Evaluation
INTRODUCTION
There was a time when faculty evaluation was conducted informally and
unsystematically--a period when the criterion of teacher effectiveness
was
objective and direct. Those were the days when Socrates could be seen
talking in the marketplace to whoever would stop to listen. Those were
the
times when Aristotle walked in the Lyceum Gardens and gathered around him
anyone who wished to engage in dialogue. The origins of student rating
of
faculty members can be traced to this era when to be an effective
teacher
was to be a person who attracted students (Travers, 1981).
By the Middle Ages, the ability to attract students as a criterion of
effectiveness took on an additional dimension--a remunerative measure of
a
teacher's worth. In addition to attracting students, the professor
had to
be able to extract fees from those students who were being taught.
The
University of Paris, founded in the 10th century, required professors
to
generate their own source of income through fees paid directly by the
students. To survive, a professor would have to receive an adequate
evaluation in the form of coins pressed into an outstretched palm (or
dropped into the mortarboard held in the hand) as the students filed out
of
the classroom at the end of a lecture (Travers, 1981). At the University
of Padua in Italy, students even hired and fired their own teachers
(Seldin, 1980).
BACKGROUND
In modern times the evaluation of faculty performance has become less
direct and much more complex. When school attendance became law, the
idea
of educational effectiveness became a public issue (Travers, 1981).
In
America, the system of education is generally regarded as a "mass"
approach
with open access. It is interesting to note that this "mass" perspective
of education parallels the mass consumption characterization of the
public
by marketing
Student Evaluation
practitioners. In the early 1930s, business began to study "these masses"
by conducting marketing research surveys that utilized consumer
questionnaires. Although student evaluation was officially born at Harvard
in 1924 (Seldon, 1980), the 1930s is cited as the period that inaugurated
the use of similar survey questionnaires to evaluate faculty. In both
marketing and educational contexts, this "mass" perspective has raised
issues and spawned debate as part of the momentum of the accountability
movement. The modern use of student evaluation is an outgrowth of the
accountability movement of the 1960s and 1970s, a period which teacher
performance was examined as a measurable element of modern education.
As
Robert Lovett said in 1970, "accountability defined as the right to
insure
a good education and to sever from the school system those who do not
contribute to that end. In a very true sense, the teacher is first
accountable to his pupils (Lovett, 1970 p. 130)".
The problem is that in modern times there is a very diverse profile of
educational institutions. Colleges and universities differ greatly in
quality, character, purpose, size, complexity, and economic stability.
The
role of the professor in the educational process varies, and the job
description of teaching is elusive and value-laden. Establishment of
performance criteria and techniques for evaluation of faculty performance
has become commensurately elusive. Therefore, throughout the evolving
history of behavior evaluation, students, faculty and administrators
have
all claimed at one time or another that criteria are both reliable,
valid
useful and unreliable, or invalid and useless (Aleamoni, 1981).
Despite the debate, the popularity of faculty performance evaluation
gained momentum as America entered the turbulent sixties. Then, students
began to reject autocratic administrators and put pressure on leaders
for
greater involvement in educational institution policies and activities
that
pertained to them. Students asked, and demanded, that administrations
give them greater voice in the instruction they were experiencing
(Seldin,
1980). Evaluation of faculty gave students the voice they were
seeking.
Student Evaluation
The 1960s also defined a period when college enrollment was in a rapid
expansion mode and many institutions had been largely concerned with
recruiting and retaining faculty members. Tenure and promotion were
almost
automatic, and most institutions had long gotten by with poorly defined
evaluation procedures (Centra, 1979). Eventually, enrollments began
to
taper off, and institutions of higher education no longer needed to add
faculty and staff members. Institution budgets became tight, and many
were
at or near their prescribed limits in the percentage of faculty on tenure.
Nationwide, 62 percent had tenure in 1977 (AAUP, 1978 in Centra, 1979).
Teachers began having to prove that they deserved tenure. This forced
institutions to make fine distinctions among generally competent
instructors. Evaluation systems helped administrators to make these
"distinctions."
The confluence of student demands for greater involvement in
administrative and instructional decisions, a historically loose
tenure and
promotion policy, and new challenges pertaining to the business of running
institutions that were beginning to emerge in the 1970s resulted in a very
hurried movement by institutions to embrace a wide variety of new
procedures for evaluating instructional performance. From about 1976,
education standards and costs have been continuously and increasingly
under
scrunity. Institutions of higher education, like other non-profit making
organizations, have been increasingly asked to justify their
activities and
to account for their use of resources, as well as their performance, to
external funding bodies such as state legislatures (Sizer, 1979). The
case
for development of some type of performance indicator (PI's) has been
building up and gaining momentum throughout the 1980s.
LITERATURE REVIEW
In the aftermath of the numerous educational reform studies of the early
and mid-1980s, educators and policy makers across the United States
established procedures for assessing and
Student Evaluation
refining educational programs. By 1990, educators in 41 states were
required to evaluate teachers. Of those states, educators in 34 were
required to do so because of state legislation, 30 because of state
department of education policy, and 5 because of litigation. Of 41 states,
24 "mandated that specific evaluative procedures and/or forms be used, 7
"suggested" specific procedures, and 10 neither required nor suggested
specific procedures and forms" (Valentine, 1992).
Pressure on administrations in regard to the design, and implementation of
instruction now comes from students, parents, and legislators. Students
are much less timid in expressing dissatisfaction than in previous
times,
and many parents express uncertainty that the effectiveness of
instruction
warrants the high costs of a college education. Since higher
education is
a labor-intensive industry with 70 to 80 percent of typical budgets
going
for salaries, (Seldin, 1980), it has been natural for institutions to
look
at instructional staffs as fertile areas to reduce costs. Granting
tenure,
for example, can have serious financial implications. It can commit an
institution to a salary that can be cumulatively substantial over a
30-year
period. So, for economic reasons alone, standards for tenure have become
more rigorous. It isn't surprising that a diminishing number of
professors
are selected for promotion and tenure.
In considering a professor for promotion in rank, tenure, or retention,
there is a wide variety of procedures and factors that an institution
can
select and utilize. Scholarly research and publication, creative
activities, service, peer reviews, classroom visitations by administrators,
teacher self-assessments, testing, and learning outcomes are examples of
potential criteria for the evaluation of faculty. However, without
reverting to the methods of the Middle Ages, the harvesting of student
"opinion" about a professor is theoretically limited to exit interviews,
suggestion boxes, small discussion groups, questionnaires to alumni,
outcomes assessment techniques, face-to-face discussions, student
testimonials and "consumer" questionnaires. In reality, a written
questionnaire or rating scale appears to serve as the predominant method of
Student Evaluation
gaining information about professors. In a 1983 survey that went to 770
academic deans listed in the U.S. Department of Education's
"Directory",
all listed as accredited, four-year, undergraduate, liberal arts
colleges,
classroom teaching was regarded, of 13 criteria, as the most important
index of overall faculty performance with the frequency of use reported
as
98.7 percent. Of the 13 sources of information for faculty evaluation
enumerated in the study, 67.5 percent of deans surveyed stated that they
a
lways used student rating information. This compares to 54.8 percent
of
all of the deans who were studied in 1978 (Seldin in Gabbin, 1990).
It is apparent that both administrators and faculty committees are relying
on student ratings to help shape their judgements about faculty
performance. Today, the use of student evaluations of faculty may be by
far the most common method of rating the effectiveness of instruction
(Wright et al in Gabbin, 1990), with classroom visits by peers a distant
second (Miller, 1987). The technique developed by market researchers
in
the U.S.A. in the 1930s, the survey questionnaire, appears to be the
most
favored of all methods of obtaining student judgements about their
courses
(Winter Hebron, 1984).
RESEARCH PROBLEM
The evidence is compelling that administrative judgements concerning
faculty tenure, promotion, and retention rely heavily, if not
exclusively,
upon some type of feedback regarding teacher performance in the
classroom.
It has been reported that when evaluating faculty for promotion and
tenure, classroom teaching is a more important factor than publication,
research, service or any other criteria. Although there are a number
of
methods of gauging teacher performance, student evaluation is
generally
agreed to have the most influence on administrative decisions
concerning a
professor's future (Miller, 1987). In this context, it appears that
rating
instruments are increasingly becoming the evaluative method of favor for
both formative and
Student Evaluation
summative decisions about instructional effectiveness (Aleamoni, 1990).
However, the literature yields very little current empirical data
concerning the extent of the use of student rating systems.
Earlier it was suggested that the American higher education system is
really a "non-system" consisting of an open patchwork of institutions
with
a variety of missions. Within the framework of higher education,
Journalism
and Mass Communications programs generally present an eclectic course of
studies that epitomize the idea of diversity. Approaches in academic
philosophy within the Mass Communications classroom run the gamut from
the
theoretical to the hands-on. Classroom formats represent a range of
pos
sibilities. It is likely that student evaluation of faculty systems
and
procedures might vary to the same extent as the diverse approach to
curriculum design and content that is inherent in the subject matter of
Journalism and Mass Communications courses. It would therefore not be
surprising to learn that such programs of study walk to a unusual
drumbeat
regarding faculty evaluation.
This study was designed in part to determine how many faculty in the
membership of the Association for Education in Journalism and Mass
Communication (AEJMC) use feedback in the form of systematic student
evaluation of faculty. Verification of the nature of measuring student
feedback was also sought. There is no evidence that programs in
Journalism
and Mass Communications have embraced faculty evaluation methodologies to
the extent that has been attributed to other disciplines. And for
good
reason it would be presumptuous to assume that Journalism and Mass
Communications programs have followed the national trend. Therefore, an
additional purpose of this research study was to determine the extent
to
which faculty have adopted the use of feedback instruments to evaluate
the
quality of teaching performance. Empirical insight was sought
concerning
how evaluation is conducted at the undergraduate level, how often
evaluation measures are taken, and how the instruments when used are
administered.
Student Evaluation
METHODOLOGY
A survey questionnaire was sent to faculty members of the Association for
Education in Journalism and Mass Communication. As an organization,
the
AEJMC has 16 divisions that represent subject matter in all of the
relevant
disciplines. A cover letter introducing the research investigators and the
general purpose of the survey was sent to AEJMC members in each of the
divisions. As a result, schools in every state and several foreign
countries were represented in the survey. More than one response was rece
ived from each state with the greatest representation from Texas,
California, Wisconsin, and Florida.
The survey instrument was lengthy, consisting of 60 questions; however,
the survey questions were straight forward and bi-polar requiring a
simple
yes/no response. Although the entire questionnaire addressed the
subject
of evaluation of faculty by students, this particular report focuses
on
only those particular questions that provide a quantitative picture
concerning the numbers of faculty that are involved in some form of
systematic evaluation by students.
The survey questionnaire was sent to a total of 2,811 current members of
AEJMC. Each mailing included a pre-addressed stamped return envelope
with
a request to respond to the survey by April 29, 1994. It was presumed
that
a 30-day-plus turnaround for responders to the mailing would be adequate.
However, a sizable number of survey questionnaires floated in long after
the deadline. Those responses to the survey were not included in the
response rate. The total number of usable responses to the survey q
uestionnaire was 961 out of 2,811, for a return rate of 34.19%.
Of those who responded to the survey, 66% were male and 38% had taught for
fifteen years or more. Over half of the survey respondents had been at
their respective institutions for at least seven years. Approximately
one
half of the participants were over the age of 46. Fifty four percent
reported that they are tenured professors.
Student Evaluation
SURVEY RESULTS
More than 90 percent of the AEJMC survey respondents stated that they
currently participate in a formal system of faculty classroom
performance
evaluation at the undergraduate level. In practically all of the
cases
(95%) a rating instrument is used and the most popular format is a
questionnaire. Fifty seven percent of the respondents agreed that the
measurement instrument used could be "most clearly" described as a
semantic
differential type instrument (e.g. a Likert format that generates
responses along a continuum from fair to good, etc.). However, in more
than 86% of the cases, open-ended responses to questions were provided
for.
A very small percentage of survey participants suggested that a
measurement instrument different than a questionnaire or a semantic
differential format was used.
Table 1: Response Frequency to Selected Survey Questions
Student Evaluation
Nearly all of the classroom evaluations were student generated, although
it is apparent that some faculty also participate in peer evaluations
and
observation by the department chair. Approximately 14% of the faculty
members responding to the survey indicated that they are observed by the
department chair. Twenty percent also suggest that they are peer
evaluated. The data do not enable a determination of which faculty only
use one of the methods and which participate in more than one, or all
three
of the procedures. A small percent of the respondents said they did not
appear to participate in any of the three types of evaluation
specifically
cited in the questions (student, peer, chair evaluations), yet when
asked
to give examples of different types of classroom evaluation, no
examples
were forthcoming.
In nearly all cases the faculty evaluation procedure is implemented in the
classroom setting and student responses are collected at the time of
administration. In other words, the instrument does not appear to be of
the take-home variety. Implementation of the evaluation procedure
occurred
within 2-4 weeks of the end of the academic session in over 90 percent of
all situations. AEJMC members were split in terms of when during the
class
session the evaluation procedure was implemented. About one half of the
faculty state that implementation of the evaluation process occurred
at the
beginning of the class session. The other half state that the procedure
began toward the end of a classroom session.
Interesting were the responses to a series of questions concerning who is
actually responsible for implementing the evaluation procedure during
the
classroom session. All of the responses were predominantly "no" to
each
query that was intended to yield specific information
as to the identity of the administrator. Eighty-two percent of the faculty
stated that it was not
"me" who implemented the evaluation procedure, 95% stated that it was not
the department chair, 87 percent said that it was not a peer member of
the
faculty, 85% agreed that it was not an administrator such as a
secretary,
and 65% said that it was not a student assigned to the department
(graduate
assistant, work-study student). One can only infer that the evaluation
administrator may be the percentage differential indicated. For
example,
looking at the latter
Student Evaluation
percentage of 65% of the respondents who stated that it was not a student
assigned to the department who administrated the evaluation process,
it can
be inferred that 45% of the evaluations were administered by a graduate
assistant or work-study student.
DISCUSSION
This study confirms the common perception that evaluation of journalism
and mass communications faculty is almost universal in American
colleges
and universities, and that the evaluators are overwhelmingly the
students
in the college classroom. There are few who would argue with the
appropriateness of this evolutionary process.
While peers may have a better grasp of the level of knowledge the
instructor possesses, and supervisors might understand best the protocol of
the classroom environment, the student is the only person who has daily
contact with the instructor and it is the student who can best
evaluate the
totality of the learning experience. No evaluation system is without
shortcomings, but student evaluation of faculty is more in tune with
allowing market forces to determine product success.
The problem with faculty evaluation does not lie with the overall methods
of evaluation, but with their ultimate design and uses. It would seem
that
academic researchers would be fully
aware of the elements of survey research that can affect the results of
their efforts. The
atmosphere of the evaluation, the pre-survey preparation done by the
faculty member, the person who administers the survey, the period of the
semester in which the evaluation takes place, and the grade the
student
believes he or she is earning at the time of the evaluation, and the
nature
of the evaluation questions, are all partial explainers of evaluation
results. If all this is true, it is not critical if evaluation results
are
used as aids to improving instruction. When the instruments become the
performance indicators that determine the professional fate of the
faculty
member, then the process is taken to an entirely different level.
Student Evaluation
Four years ago, a study at New Mexico State University revealed at least
21 different evaluation questionnaires being used at the same time.
The
College of Arts and Science allowed each department to use a different
teaching evaluation form, but still apportioned salary increase monies
in
accord with a ranking based on research, teaching, and service. To
compare
a department using a form of six questions, all on a four-point scale,
with a department using a 12 question instrument employing a 7-point
scale
is just not very good science, yet such unequal faculty evaluation
comparisons are still commonly used among university departments whose
survey standards in other academic research projects are rigorously
controlled to minimize errors.
Unless tenure, promotion, and salary questions are to be determined 90% on
the basis of research achievements, teaching evaluations will be used for
summative judgements of faculty performance. Because journalism and
mass
communications faculty interviewed normally see teaching as their
first
priority, most are interested in having this competence evaluated and
used
as a major component of their professional success. To fairly and
effectively achieve this goal, it is necessary to fully understand the eva
luation process and to be able to rely on the instruments being used.
Because it is clear that student evaluations dominate the field of faculty
evaluations, it is
then necessary to focus upon what certain evaluations mean. Most would
agree that, on a 5-point
scale for bad teacher-good teacher, a cumulative rating of 4.65 indicates
that good teaching is occurring in the classroom; however, using the
same
scale for easy grader-hard grader, a 4.65 could have different
implications
to different observers. Faculty and administrators generally respect
"hard" graders, but it is not clear whether students have a similar
attitude. It may be that students would see a 2.5 as the best rating on
the easy grader-hard grader scale, but this would not contribute to
the
faculty member's overall rating when the department chair is
determining
salary increases.
The argument over validity of student evaluation of faculty is obviated by
the concern over Student Evaluation
the reliability of the instruments being used. This means that it is
unnecessary to have a nationwide standard evaluation instrument but that
it
is important for universities, or colleges within universities, to use
carefully constructed evaluation instruments for the purpose of
comparison.
If the current survey shows the dominance of the numerical scale question
over the open-ended question, it suggests that more attention be paid to
the scales being used and to what the evaluators mean by the answers
they
give to the questions.
Many faculty look forward to student evaluations and see them as both
necessary and constructive, but administrators may look on the process
as a
means for ranking the faculty for purposes of promotion, tenure, and
salary determination. Whether evaluations can serve both purposes is a
moot question: They can, and they do serve both purposes.
Faculty evaluations by students are a permanent fixture of higher
education, and their reliability can be improved if more attention is paid
to the process and to the instruments being used. For this to be
achieved,
more attention must be paid to the student evaluators and the meanings
they attach to evaluation terms. Further, the process used to
administer
evaluations should be consistent semester to semester and instructor
to
instructor.
CONCLUSION
The nationwide patterns outlined in this study are indicators of
evaluation trends that will
continue in institutions of higher learning into the next century. Perhaps
the instruments that are
used to measure performance and provide feedback to faculty and
administration will provide
objective insights that everyone can be comfortable with by that time. And
since the evaluation instruments are used in part for purposes of making
summative judgements about faculty, maybe faculty should be given
equal
opportunity in terms of evaluating administrators.
It is interesting to note that there is a trend among organizations
outside of academia to implement a performance evaluation tactic known as
"upward feedback appraisal." Simply put, employees get to review
their
bosses. Similar to student evaluations of faculty, a list of questions
is
devised and workers are asked to rate the boss as very good, good,
average,
poor, or very poor in key areas. The questions usually center on such
things as good listening skills, how well the
Student Evaluation
boss communicates, motivates, and displays trust and confidence in
subordinates. At many companies, lack of improvement by managers from year
to year will jeopardize their raises and
promotion potential. Although these performance appraisals have been
around for about ten years, according to Michael Seitchik, director of
program development at the Wharton School's Division of Executive
Education
at the University of Pennsylvania, they've become "extremely popular in
the last two or three years" (Vaughan, 1994).
If turnabout is fair play, and given the fact that students and faculty
already extensively evaluate each other, how about including annual
evaluations of the administrators? Further study should also be conducted
to discover whether faculty are currently satisfied with the
administration, the form, the results, or the use made of teaching
evaluation as conducted in college classrooms today.
Primary Questions
Yes Yes No No
Question
Frequency Percent
Frequency Percent
Does your institution currently use a formal system of faculty 782 88
108 12
classroom performance evaluation at the undergraduate level?
Does your department (college or school) participate in a formal 801
90 88 10
system of faculty classroom performance evaluation at the
undergraduate level?
Does the formal faculty evaluation system that you use include the 796
95 40 5
use of some type of rating instrument?
If a rating instrument is used, could it most clearly be described 437
57 324 43
as a semantic differential type instrument?
If a rating instrument is used, could it most clearly be described 673
83 134 17
as a survey or questionaire?
Are there open ended responses to atleast part of your instrument? 716
86 115 14
References:
AAUP (1978) in Centra, John A (1979) Determining Faculty Effectiveness, San
Francisco: Jossey-Bass Publishers.
Aleamoni, L.M. (1981). Student ratings of instructor. In Handbook of
Teacher Evaluation. Millman ed. Beverly Hills, CA, p110.
Centra, John A (1979) Determining Faculty Effectiveness, San Francisco:
Jossey-Bass Publishers p2
Gabbin, Alexander L, Cairns, Scott N, Benke, Ralph L. Editors (1990),
Faculty Performance Appraisal - Center For Research in Accounting
Education, James Madison University.
Lovett, Robert (1970) "Professional Accountability in the Schools"
p129-p134 in Sciara, F. J. (1970) Accountability in American Education
Allyn & Bacon, Inc. : Boston
Millman, Jason, Editor (1981) Handbook of Teacher Evaluation, Beverly
Hills: Sage Publications
Miller, Richard I (1987) Evaluating Faculty for Promotion and Tenure, San
Francisco: Jossey-Bass
Seldin, Peter (1980) Successful Faculty Evaluation Programs, New York:
Coventry Press.
Seldin (1990) Faculty Evaluations: Surveying Policy and Practices pp 3-10
Gabbin, Alexander L, Cairns, Scott.
Travers, Robert (1981). Criteria of Good Teachers in Millman, Jason,
editors, Handbook of Teacher Evaluation, Beverly Hills: Sage
publications,
p39.
Sizer, J (1979) Assessing Institutional Performance: An Overview
International Journal of Institutional Management in Higher Education Vol 3
No 1, pp 49-77
Vaughan, Viki (1994). Workers bet to review boss The Arizona Republic page
E1 July 25
Valentine, Jerry W (1992) Principles and Practices for Effective Teacher
Evaluation, Boston, Allyn & Bacon
Winter Hebron, C. de (1984) An Aid for Evaluating Teaching in Higher
Education - Assessment and Evaluation in Higher Education Vol (9) No (2)
Summer, pp 145-163
Wright, Penny; Whittington, Ray; Whittenburg, G.E in Gabbin (Editor) (1990)
Faculty Performance Appraisal - Center for Research in Accounting
Education James Madison University, p55
|
