This paper was presented at the Association for Education in Journalism and
Mass Communication in San Antonio, Texas August 2005.
If you have questions about this paper, please contact the author
directly. If you have questions about the archives, email
rakyat [ at ] eparker.org. For an explanation of the subject line,
send email to
[log in to unmask] with just the four words, "get help info aejmc," in the
body (drop the "").
(Jan 2006)
Thank you.
Elliott Parker
====================================================================
JUDGING JOURNALISM
What criteria affect the determination of excellence
by judges in journalism awards programs?
(A study of Canada's two leading national programs.)
Ivor Shapiro, Patrizia Albanese, Leigh Doyle
Submitted to the Civic Journalism Interest Group of the Association
of Educators in Journalism and Mass Communication for possible
presentation at the Association's convention in San Antonio, Texas,
August 10-13, 2005.
Ivor Shapiro
Assistant Professor
School of Journalism
Ryerson University
350 Victoria Street
Toronto, ON M4S 2P4
Canada
Phone: 416-979-5000 ext 7195
Email: [log in to unmask]
Patrizia Albanese
Assistant Professor
Department of Sociology
Ryerson University
Leigh Doyle
Student/Research Assistant
School of Journalism
Ryerson University
JUDGING JOURNALISM
What criteria affect the determination of excellence
by judges in journalism awards programs?
(A study of Canada's two leading national programs.)
Ivor Shapiro, Patrizia Albanese, Leigh Doyle
ABSTRACT
What does "excellence" mean in journalism? The literature reveals no
universally agreed set of standards, and awards guidelines are often
unclear. When interviewed about how they assess submissions, judges
in Canada's two leading journalism awards programs emphasized their
intuition and experience rather than specific criteria, but placed
special weight on writing style and on the amount and depth of
reporting. Other values included originality, relevance and public
impact, integrity, and analysis.
JUDGING JOURNALISM
What criteria affect the determination of excellence
by judges in journalism awards programs?
(A study of Canada's two leading national programs.)
Ivor Shapiro, Patrizia Albanese, Leigh Doyle[1]
Introduction
When Seymour Topping was asked about the concentration of winners of
The Pulitzer Prizes shortly before his retirement as the prizes'
administrator in 2002, he replied:
When the Pulitzer Board reviews an entry, it doesn't discuss the
circulation of the newspaper - except possibly in Public Service -
the location of the newspaper, the ownership of the newspaper, or
whether or not the newspaper has won any previous Pulitzer prizes
.
The judging is based solely on excellence, comparative excellence.[2]
But what exactly is excellence, and how is it measured? The year 2001
saw the publication of the first book-length scholarly work to look
specifically at the nature of excellence in journalism and the
prospects for its achievement. Its authors compared the way
journalists talk about their field to the way geneticists talk about
theirs, and painted a bleak picture of the news business. After
extensive interviews with reporters, editors, their audiences,
scholars, and shareholders of media corporations, Gardner,
Csikszentmihalyi and Damon reported that the field of journalism was
"wracked with tension". The stakeholding groups "differ sharply in
their aspirations," the authors wrote. Under these circumstances, the
idea of achieving excellence was "but a distant dream."[3]
Those who grant, judge, and win journalism awards would, presumably,
disagree. So might those in news organizations who devote
considerable energy and spend thousands of dollars near the beginning
of each year to enter their journalists' work for awards in order to
be included where excellence is showcased.[4] "American journalism,"
according to Shepard, "is locked in the iron grip of prize frenzy."[5]
As Bogart points out, awards "are, surely, an indicator of how
quality is assessed by colleagues," even if "the subject of awards
often carries in its train the epithet of elitism." [6] "Prizes are
the only way we have to keep score," one leading journalism educator
has been quoted as saying. "Every journalist you ever talk with will
say our obsession with prizes is criminal
. But it's the only
quantifiable way of the industry recognizing you as a player."[7]
It's true that even the most venerable awards come under attack from
time to time[8], especially after a scandalous result, such as the
return of the 1981 Pulitzer Prize awarded to Janet Cooke for what
turned out to be a fabricated story.[9] But in the main, the
attention that continues to be given by journalists and news
organizations to leading national awards is ample evidence of their
prestige in the industry.
Certainly, the gravitas attached to leading national awards makes it
highly likely that those selected for the juries would take their
responsibility with a high degree of seriousness. That seriousness,
in turn, suggests that these jurors comprise a suitable cohort to
whom to put questions about how excellence may be defined in
journalism. How important, for example, is the subject matter of a
story in determining its excellence? How important is its public
impact or benefit the degree to which (to use the language of
civic journalism) a story may "address people as citizens, potential
participants in public affairs, rather than victims or spectators"
and "help the political community act upon, rather than just learn
about, its problems"?[10] How much weight should be placed on the
amount, depth or difficulty of reporting, or on writing style, or on
the originality of the theme? To what degree do issues of fairness,
balance, and independence from sources come into play?
To gain some initial insight into how excellence is measured in
journalism, we surveyed judges in the two leading Canadian print
journalism awards programs. In this paper, we present preliminary
findings about the criteria by which those judges measure excellence.
Literature Review
Only recently has the study of excellence in journalism emerged as a
field of interest outside mass-communications content research, and
nothing close to an agreed list of consensus standards has yet
emerged in the literature. Kovach and Rosenstiel, reporting on the
work of the Committee of Concerned Journalists, made a start by
listing, explaining and illustrating ten "principles that journalists
agree on." The first is a statement of the purpose of journalism ("to
provide people with the information they need to be free and
self-governing") and the other nine are statements about what is
needed for journalists "to fulfill this task." Those nine statements
address journalists' obligations of truthfulness and verification,
their duties of loyalty to citizens and independence from those they
cover, their role as monitors of power and providers of "a forum for
public criticism and compromise," the need for journalism to be
interesting, relevant, comprehensive and proportional, and the need
for journalists to exercise freedom of conscience.[11] While these
"elements of journalism" do not masquerade as agreed standards of
excellence, they have started a widespread discussion on these
standards and, generally, been well received by practising
journalists and scholars alike.
Clearly, criteria for excellence within a field would be different
from the "standards and practices" against which all quality work is
measured. Gardner, Csikszentmihalyi and Damon clearly enunciate the
latter (they spotlight truthfulness and fairness as particular
standards) but do not provide criteria for excellence. They suggest
that excellence is most likely to be achieved, and to be perceived as
being achieved, when there's a fair degree of alignment among three
areas: the values of the culture, the knowledge and values of the
profession, and the everyday structures and roles among the
practitioners. They say this kind of alignment is more evident in
some professions, such as genetics, than in others, including journalism.[12]
Meanwhile, a substantial literature has existed for some time on the
measurement of "quality" in journalism[13], using criteria such as
accuracy, impartiality in reporting, and investigative
enterprise[14], and quantifiable measurements including the sources
and contents of stories (wire copy versus local reporting[15]). This
field (which has often been tied to a study of the link between
quality journalism and business success[16]) seems highly similar to
the idea of research into standards of excellence, but the
relationship between "quality" and "excellence" is murky. Gladney,
for example, uses "excellence" to describe his research into how
editors and readers rank journalistic standards,[17] but those
standards are very similar to the "quality" criteria employed, for
example, by Bogart (who himself appears to use the two terms
interchangeably).[18] In any case, Gladney's research produced
seminal rankings of nine "content standards" for excellence in
newspapers (including news interpretation; lack of sensationalism;
strong local coverage; visual appeal; accuracy; strong editorial
page; comprehensive coverage and good writing) and nine
"organizational standards" (including integrity, staff enterprise,
editorial independence and courage, and decency).[19]
It's possible to see excellence as, essentially, quality in
abundance. That is, "excellence" would refer, in effect, to high
scores for "quality." This notion, which seems to underlie Gladney's
work, seems entirely appropriate to his and others' studies of how
excellence is measured within a publication or organization. But
excellence in individual works of journalism is another matter both
because many of the criteria do not apply (e.g. the amount of wire
copy) and because some of the others (e.g. accuracy) would be seen by
most journalists not as criteria of excellence for particular works
but as bare minimum requirements.
An Australian project last year set out to list "The Best Australian
Journalism Of The 20th Century." The jury's criteria do seem to
define a list of characteristics of excellence with which many
journalists might agree. They are:
Excellence of writing or production;
Bravery and perseverance in gathering the information or image;
Intelligence and initiative shown in finding the story;
The impact the story or broadcast or photograph had on the public mind;
Originality and innovation in gathering and telling the news." [20]
But the most visible measure of excellence in journalism remains the
major national awards programs. In a 1974 study of U.S. managing
editors, Allen and Blankenburg reported that 68.4% of respondents
held a "favorable" or "very favorable" view of journalism contests,
and that 91.2% of their newspapers or staff had entered news or
editorial contests in the past year.[21] In a follow-up study 15
years later, Coulson reported that 90% of editors considered awards valuable.
Valuable in what way? For 80%, awards bolstered journalistic
prestige; only 50% said they provided a measure of achievement, while
37% believed winning created a false standard of excellence.
[22] Another study, in 1986, found that winners of journalism awards
are more likely than their colleagues to enjoy organizational and
occupational prestige.[23] We are unaware of comparable studies in
the years since, but the unabated energy devoted to pursuing prizes
and celebrating victories suggests little reason to suspect a
substantial change.
There's an obvious paradox in the fact that, while awards bring
prestige, many journalists are skeptical about their usefulness as a
measure of excellence. To some degree, the paradox is resolved by
remembering that there are awards and then there are Awards. The
authors of the above studies, among others, acknowledge differences
between two broad types of awards programs. The first type is judged
by leading journalists and by people chosen and respected by
journalists. The second group consists of dozens of less well-known
awards including many that are awarded by interest groups for
stories that cover those groups' fields.[24] (In December, 2003,
Editor & Publisher magazine listed 256 U.S. and international
journalism competitions, and 61 regional competitions.[25]) Attitudes
to the lesser awards will naturally skew journalists' responses when
asked about journalism awards in general. Some hold the view,
regarding interest-group awards, that many journalists "enter
contests they know are jokes".[26]
In this paper, the awards under discussion are those national
programs in which journalists in general hold a high degree of
interest, rather than those given out by groups seeking to influence
the media. This does not deny the fact that journalists remain
skeptical, to some degree, about even major prizes, but there's no
longer a serious question about the prestige associated which such
awards. At least one pair of researchers considered the Pulitzer
Prize so synonymous with quality that they used prize-winning as an
established measure of quality against which to test the possibility
that competition among newspapers leads to improvement in the quality
of journalism[27]. If, therefore, one is looking for a list of
characteristics that reflect a de facto consensus on standards of
excellence, an obvious place to look would be the guidelines
developed by journalism's most respected awards programs.
Unfortunately (as journalists everywhere know) obvious sources are
not necessarily the most productive ones. We searched for and
requested lists of criteria applied by judges in several leading
awards programs, including the Pulitzer Prizes, the George Polk
awards, and the national awards programs in Canada, Britain, and
Australia; the results were often less than helpful. Many programs
simply don't have judging guidelines, while others consist mostly of
lists of terms, without explanation or illustration.
Some of the Pulitzer Prizes' one-sentence category definitions do
provide terse clues as to what jurors are expected to look for. ("For
a distinguished example of explanatory reporting that illuminates a
significant and complex subject, demonstrating mastery of the
subject, lucid writing and clear presentation, ten thousand dollars."
"For a distinguished example of beat reporting characterized by
sustained and knowledgeable coverage of a particular subject or
activity, ten thousand dollars." "For a distinguished example of
feature writing giving prime consideration to high literary quality
and originality, ten thousand dollars." ) Others do not. ("For a
distinguished example of reporting on national affairs, ten thousand
dollars." "For a distinguished example of reporting on international
affairs, including United Nations correspondence, ten thousand
dollars.") The Pulitzer web site includes a list of frequently asked
questions, of which Number 19 is: "What are the criteria for the
judging of The Pulitzer Prizes?" The answer: "There are no set
criteria for the judging of the Prizes. The definitions of each
category
are the only guidelines. It is left up to the Nominating
Juries and The Pulitzer Prize Board to determine exactly what makes a
work `distinguished.'"[28]
Isolated first-person accounts by members of Pulitzer juries, and
published interviews with jurors, do shed some light on the jurors'
de facto criteria. In one, a juror - the Managing Editor of the Wall
Street Journal says: "I found myself using the same standards I
use in evaluating Journal job prospects and whether to print stories:
Are the stories factual? Are there good explanations? Is there an
absence of hype? Is the writing lively? Is there evidence of
enterprise and initiative?"[29] One story quoted 1999 Pulitzer
jurors as saying that they had asked questions such as: "Is the
subject significant? Is this entry really original and breaking new
ground? Did the story make a difference? " One juror said: "I found
myself looking for results, as a way to separate competing entries
If an entry showed good work but had no consequences, I would find
myself gravitating to another entry that had impact." Another said
that he and his fellow jurors were drawn to "good, old fashioned
reporting - finding something and pursuing it," instead of
pre-planning a major project.[30] And a content study has suggested
that Pulitzer juries favour "information richness" that is, they
reward stories that used more, and more diverse, sources.[31]
Some awards do publish criteria. In the U.S., the National Magazine
Awards provides judges with a brief general definition of what's
expected in each category. The Reporting category, for instance,
honours "enterprise, exclusive reporting and intelligent analysis."
The Feature Writing category honours "stylishness and originality."[32]
Australia's Walkley Awards provides judges with a brief statement
about judging criteria. This statement says: "The emphasis should be
on creative and courageous entries the different rather than the
predictable. We are looking to recognize research and dedicated
journalism that seeks out the truth." The statement also includes a
list of 11 aspects to be considered by judges. They are:
newsworthiness; research; writing; production; incisiveness; impact;
public benefit; ethics; originality; innovation; and creative flair.[33]
Likewise, Canada's National Magazine Awards (NMA) offers judges
brief explanations for each category, but these are supplemented by a
general list of four criteria (each worth twenty-five percent of
judges' final score), without explanation or illustration. The four
criteria are: style, content, fairness, originality, and "how well
the article engages the reader for whom it was intended."[34]
The Canadian National Newspaper Awards (NNA) issues exceptionally
detailed criteria (described as "guidelines") for each category of
the awards. For example, the guidelines for the Investigation
category emphasize "enterprise and depth" and include a list of
thirteen questions to consider, including:? "Was this work a
worthwhile allocation of this newspaper's resources does the
subject involve a matter of reasonable importance to the public? Is
this a significant exposι? Is the public interest or the rights of
individuals at stake? Does this work emanate primarily from the
initiative of the reporter/newspaper? Does this work expose secrets
and/or wrongdoing? Does fact-gathering go beyond routine, drawing on
computer databases, analysis, public records and authoritative
(perhaps reluctant?) sources for its information?" [35]
The NNA's judges are also provided with a set of "general notes"
listing the following "elements" for evaluation:
Idea: Significance (Was it worth reporter and reader spending time
on?); Newsworthiness; Timeliness; Originality and creativity;
Humorous; Initiative.
Reporting: Depth and breadth; Context and background. Accuracy;
Fairness and balance; Comprehensive, relevant sources (officials and
real people); Detail that engages reader; Answers readers questions;
Enterprise and effort.
Writing: Language (precision of usage, elegance); Style, tone, mood
(appropriate to content); Credibility/authority; Compelling
lead/opening; Clarity; Strong focus/theme what is this story
about?; Structure and organization; Effective anecdotes quotes and
examples; Narrative and description; Accuracy and fairness;
Creativity/Risk-taking; Reader interest.
Overall impression: Excellent; Good; Indifferent.[36]
Drawing together the criteria of quality or excellence cited in all
the sources mentioned above, it's possible to identify twelve
discrete types of criteria. In alphabetical order, they are:
Analysis: thoughtful and logical reflection on the subject matter.
Benefit to society: the story's public impact, relevance, or service.
Breaking news.
Clarity brought to a complicated subject.
Complexity or other difficulty concerning the subject matter.
Context: thoroughness in exploring the story's background, and the
"how" and "why" questions behind the news.
Integrity: fairness and independence from the story's sources.
Originality or uniqueness of the subject matter.
Reporting: depth, rigour and volume of research and investigative
enterprise.
Technique: innovative and engaging method of presenting the information.
Transparency of method: the extent to which the audience will
understand where the information comes from.
Writing: superior style and story-telling technique. [37]
Research Questions
In light of the above assessment of the literature, we set out to
answer three primary questions:
RQ1: Against what de facto criteria do awards judges measure
excellence in journalism, and how may these criteria be ranked?
RQ2: To what extent do the judges' criteria for excellence adhere to
or deviate from established benchmarks in literature about quality in
journalism?
RQ3: What is the process by which judges go about defining
excellence? What impact might the process have on findings? What
impacts do factors such as selection of judges, time spent
considering entries, and established processes have on the outcome?
This paper will report on, and discuss, only the portion of our
research that is relevant to RQ1 and RQ2, above that is, our
attempt to shed light on the de facto criteria against which awards
judges determine excellence.
Method
To investigate the above questions, we chose to pursue a qualitative
method involving in-depth interviewing of a relatively small number
of judges, rather than a mail-out quantitative survey. We made this
choice because we were breaking new ground in this research: not only
had journalism awards judges never been surveyed about their
criteria, but the literature failed to provide a clear starting point
for a list of standards that could be readily applied to individual
works of journalism. We felt that we should allow judges to tell us
about their considerations in an open-ended way, rather than having
to fit their responses into a predetermined framework that might or
might not correlate well to actual practice. We therefore set out,
not to arrive at definitive answers on judges' criteria, but to help
establish a foundation for future research.
Accordingly, we interviewed fifty judges in Canadian national awards
programs between 2001 and 2004, of whom twenty-five judged various
categories of the National Newspaper Awards (NNA) and the remaining
twenty-five judged various categories of the National Magazine Awards
(NMA). Judges' names were selected randomly from lists provided by
the awards programs' organizers; we then set out to contact the
selected judges and interviewed (by telephone) the first twenty-five
reached in each program. To achieve this number, we contacted 61 NNA
judges (of whom two declined to be interviewed and 34 did not
respond), and 67 NMA judges (of whom four declined and 38 did not
respond). All judged text entries, as opposed to visuals;
overwhelmingly, they judged categories involving features or
explanatory writing, or local or beat or spot-news reporting, or
reporting on politics, business, science, medicine, or international affairs.
The interview questionnaire was a mix of closed- and open-ended
questions; it took an average of 26 minutes to administer. The
complete questionnaire began with questions about the judge's
background and the experience of being a judge (such as how many
times the judge had participated in the awards program and why s/he
believed s/he was selected as a judge). Subsequent questions probed
aspects of the work done by the judge, including the number of pieces
read, the method used to identify winners, the time it took and the
quality of the entries. These process questions were followed by four
questions about criteria (see below). Other questions explored
subject matter favoured by judges and the effect of knowing or not
knowing the authorship of articles. Finally, the respondents were
offered an opportunity to provide additional comments.
The questions about standards of excellence started some ten minutes
into the interview. The first of these asked about the judge's
criteria in an open-ended way. It was followed by two specific
questions prompting the respondents to evaluate the twelve predefined
criteria of excellence that we had drawn from the literature and
awards materials (see Literature Review, above). A later, open-ended
question concerned the best piece the judge had ever read. The
questions are listed together with a summary of responses under
"Results," below.
After looking at the responses to open-ended questions, we identified
common responses and coded the results accordingly. The codes, which
indicated de facto criteria[38], were then grouped under seven
headings. In alphabetical order, these seven criteria groupings were:
Balance or fairness.
Insight, analysis, or examination of the context of the story.
(This grouping included references to the journalist having probed
the questions "why" and "how" behind a story).
Mention of criteria supplied to judges by the awards program.
Originality. (Included originality of idea and/or its importance,
originality of subject, originality of execution, initiative,
surprising angle, interesting subject.)
Relevance and public impact or benefit of the story. (Included
relevance to community, public service impact, effect on society or
reader, "leads to action on part of readers", "challenges the
reader", value/utility for reader.)
Reporting: Amount or depth. (Included references to information
gathering, depth of research or number of sources, thoroughness of
reporting, imaginative research, new information, and factual detail.)
Writing and/or story-telling techniques. (Included writing quality,
engagement and emotional impact, vivid presentation,
compelling/captured attention, structure/architecture, creativity,
style or flair, voice, and narrative technique.)
If a judge mentioned one or more criteria within a particular
grouping, we recorded a single citation of that grouping, regardless
of how many criteria within that grouping had been mentioned.[39]
We then tabulated and assessed our findings, as follows.
Results
When asked about their criteria in an open-ended way, many judges
tended to avoid naming specific standards. Instead, they would refer
to their professional experience or tastes, or the overall experience
itself. "I can't really describe it. We don't deconstruct it. I don't
look at it that way. Do they compel me? Do they draw me in?" said one
judge. Another said: "If it gets my attention, keeps me reading and I
forget that I'm judging, then it's worthwhile. It comes down to how
the writer speaks to me." However, when pressed for specifics, all
the respondents were able to name criteria. Table 1 shows the number
of judges who mentioned criteria in each grouping in response to the
initial, open-ended question on judging standards. The groupings are
then ranked according to the number of judges who mentioned each.
Results are presented for the newspaper and magazine awards
separately, and then for the total sample.
TABLE 1
Judges' Preliminary Responses On Criteria Of Excellence[40] (Ranked)
Newspaper
Magazine
Total
Grouping
Mentions
Rank
Mentions
Rank
Mentions
Rank
Writing and
storytelling
21
1
24
1
45
1
Amount/depth
of reporting
14
2
12
2
26
2
Originality
11
3
11
3
22
3
Relevance,
public impact
9
4
7
4
16
4
Insight/analysis/
context
4
5
4
5
8
5
Awards
criteria
3
6
1
6*
4
6
Balance/
fairness
1
7
1
6*
2
7
* Tie
It seems fair to characterize judges' responses to this preliminary
and open-ended question about criteria as top-of-mind responses. By
this we mean that judges, before being prompted to consider a range
of standards that have been suggested as criteria for excellence by
awards programs and in the literature, may have a relatively small
set of values uppermost in their minds. Above all, these top-of-mind
criteria for excellence seem to emphasize writing and story-telling,
followed (some way behind) by reporting rigour and then by
originality in subject or approach, with relevance or public impact
the only other theme to win mentions in the double digits. The two
groups of respondents (magazine and newspaper judges) answered this
first open-ended question in very similar ways.
A rather different picture emerges from responses to the next
question about criteria. Table 2 shows how judges scored the twelve
predefined criteria of excellence that we had drawn from the
literature. The scores were on a scale of one to five.
TABLE 2
Average Scores[41] For Twelve Predefined Criteria Of Excellence[42]
(Alphabetical)
Newspaper
Magazine
Total
Criterion
Score
Rank
Score
Rank
Score
Rank
Analysis
4.0
6*
4.6
2
4.3
3*
Benefit to society
3.6
8*
3.3
10
3.5
9*
Breaking news
2.8
12
2.7
12
2.7
12
Clarity to com-
plicated subject
4.5
1*
4.5
3
4.5
1
Context
4.5
1*
4.1
4*
4.3
3*
Difficult subject
3.4
11
3.7
9
3.5
9*
Independence,
fairness
4.2
4
3.8
7*
4.0
6
Originality
4.0
6*
3.8
7*
3.9
7
Reporting
4.3
3
4.1
4*
4.2
5
Technique
3.6
8*
4.0
6
3.8
8
Transparency
3.5
10
3.2
11
3.3
11
Writing
4.1
5
4.7
1
4.4
2
*Tie
Note: Scores are too close to one another for the rankings
to be statistically significant.
These results were very close: rather than homing in on a few
criteria, judges were ready to give ratings averaging 3.5 or higher
not just in the areas of interest that reflect their own top-of-mind
values, but to as many as ten different criteria (with at least six
criteria garnering ratings of 4.0 or higher). These high scores are
too close together for us to conclude anything significant from
ranking them. However, the two that stand out somewhat among the
newspaper judges' scores are context and clarity to complicated
subject, while magazine judges seem to gravitate toward writing
(reflecting their top-of-mind leaning) as well as analysis and, as
with the newspaper judges, clarity to complicated subject.
Table 3 shows how, after scoring the twelve predefined criteria, the
judges named the three most important of the twelve.
TABLE 3
"Most Important" Criteria[43] (Alphabetical)
Newspaper
Magazine
Total
Criterion
Mentions
Rank
Mentions
Rank
Mentions
Rank
Analysis
8
4*
8
3*
16
4
Benefit to society
4
8
2
8*
6
9
Breaking news
0
12
1
11
1
12
Clarity to com-
plicated subject
6
6*
7
5
13
5*
Context
6
6*
2
8*
8
8
Difficult subject
2
9*
0
12
2
11
Independence,
fairness
11
1*
6
6
17
3
Originality
8
4*
5
7
13
5*
Reporting
11
1*
10
2
21
2
Technique
1
11
8
3*
9
7
Transparency
2
9*
2
8*
4
10
Writing
9
3
19
1
28
1
*Tie
The responses to this question narrow the field considerably as
compared with Table 2. It seems fair to characterize these forced
top-three responses as trade-off rankings that is, when push came
to shove, these top-three-ranked standards would likely influence
judges to prefer one contender over another. Seen this way, judges'
trade-off criteria for excellence seem to emphasize writing style and
reporting rigour. Magazine judges chose writing most often (by far),
while newspaper judges gave a slight edge to reporting, along with
independence and fairness. For magazine judges, analysis and
technique were other frequent mentions.
Table 4 shows the judges' responses to the fourth and final question
open-ended, like the first about judging criteria.
TABLE 4
Qualities Of The Best Story Ever Judged[44]
Newspaper
Magazine
Total
Grouping
Mentions
Rank
Mentions
Rank
Mentions
Rank
Writing and
storytelling
16
1
18
1
34
1
Amount/depth
of reporting
12
2*
10
2
22
2
Originality
9
4
7
3
16
4
Relevance,
public impact
12
2*
6
4
18
3
Insight/analysis/
context
1
5*
3
5
4
5
Awards
criteria
0
7
0
6*
0
7
Balance/
fairness
1
5*
0
6*
1
6
*Tie
Responses to this final (open-ended) question somewhat confirm the
earlier responses reported in Table 1. Describing the best story they
had ever judged, both newspaper and magazine judges continued to
emphasize writing followed by reporting, though the gap between the
top two criteria has narrowed.
Discussion
When first asked about their criteria, judges tend to emphasize their
intuition and experience, rather than specific standards. But when
pressed, they find they can identify criteria, though the weight
placed on individual criteria appears to vary depending on the how
the question is asked. How may these varying responses be understood?
We began to see a clue toward answering this question when we
juxtaposed the respondents' answers to all four criteria-related
questions. In order to do this, we needed to find a way to compare
responses to the two open-ended questions with responses to the two
questions that presented predefined criteria. We decided to use the
system of equivalencies that is presented in Table 5. In the first
column of this Table, we list themes found in open-ended responses
(as cited in Tables 1 and 4) and juxtapose them against the scored,
predefined criteria drawn from the literature (cited in Tables 2 and
3).[45] The juxtaposed pairings were then assigned general headings,
which we list in the final column.
TABLE 5
Equivalencies For Comparison Of Responses
Open-ended
mentions
of these themes:
were juxtaposed
with values given
to these criteria:
under these
general
headings:
Writing and
storytelling
Superior
writing style
Writing
Writing and
storytelling
Innovative and engaging
technique
Technique
Amount/depth
of reporting
Depth and amount of reporting or investigative rigour
Reporting
Balance/
fairness
Integrity, fairness
and independence
Integrity
Originality
Originality or uniqueness
of the subject matter
Originality
Insight/analysis/
context
Thoughtful
and logical analysis
Analysis
Insight/analysis/
context
Thoroughness in exploring
the context
Context
Insight/analysis/
context
Bringing clarity
to a complicated subject
Clarity
Relevance,
public impact
Benefit to society
(public service)
Relevance
The juxtaposed responses are collated in Table 6. Here, the "Top Of
Mind" (TOM) columns show how each theme was initially ranked (as
reported in Table 1 above). The "Grade" (GR) columns reflect the
judges' average scores for the juxtaposed predefined criteria (as
reported in Table 2). For the purposes of this table, we
converted[46] the original one-to-five scores to a crude grading
scale from C to A+. The "Trade-Off" (TO) columns rank judges'
responses (as reported in Table 3) when asked to identify the three
most important predefined criteria. The Best Ever (BE) columns rank
judges' responses (as reported in Table 4) when describing the best
story ever judged.
TABLE 6
Juxtaposed Responses On Criteria
Newspaper
Magazine
Total
General
heading
TOM
GR
TO
BE
TOM
GR
TO
BE
TOM
GR
TO
BE
Writing
1
A
3
1
1
A+
1
1
1
A
1
1
Technique
1
B
11
1
1
A
3
1
1
B
7
1
Reporting
2
A
1
2
2
A
2
2
2
A
2
2
Integrity
7
A
1
5
6
B
6
6
7
A
3
6
Originality
3
A
4
4
3
B
7
3
3
B
5
4
Analysis
5
A
4
5
5
A+
3
5
5
A
4
5
Context
5
A+
6
5
5
A
8
5
5
A
8
5
Clarity
5
A+
6
5
5
A+
5
5
5
A+
5
5
Relevance
4
B
8
2
4
C
10
4
4
B
9
3
We have used shading on Table 6 to highlight what we see as the
significance of these various juxtapositions. Dark shading highlights
a theme that scored an A or A+ grade, and that was one of the three
most commonly mentioned themes in response to all three of the other
criteria questions. In other words, these themes held fast in judges'
responses on criteria, no matter what questions they were asked.
Light shading highlights an A or A+ grade or a top-three theme. That
is, these themes did not hold fast when questions about criteria were
asked in different ways.
If these shading patterns are reviewed in the context of the order in
which the various questions were asked, it's possible to hear the
data telling a story. We suggest that, when prompted with predefined
criteria, the respondents had an opportunity to rethink their
"top-of-mind" thoughts in light of accepted notions about
journalistic excellence. The prompted list may have reminded
individual judges of criteria of which they were aware (or perhaps,
at least, of which they now felt they should have been aware),
whether explicitly or not, at the time of judging but which were
not currently "top of mind" at the time of our survey. Then, when
asked to name their trade-off (top-three) criteria, they were obliged
to restate their understandings of the chief elements of excellence
in terms of established values. And after that, when we asked them to
identify the qualities of the best story they had ever judged, they
received an opportunity to do so in a way that relied not only on
their own top-of-mind standards but on the full spectrum of received
standards.
At this point, we yield to a temptation to speculate. We feel that
the above narrative might offer a glimpse of how judges perceive
their own judging standards at various points during the time of
judging. There is, of course, no way to take a live snapshot of the
minds of the judges at any time during their review of the nominated
materials. But we suggest that a variety of influences could play a
role in the various stages of the judging process. As we learned from
responses to our questions about the judging process (which we can
do no more than mention here), most judges follow a two- or
three-step process to arrive at their scoring decisions for nominated
pieces. Though details vary, most judges begin by either reading
quickly through the entire pile of nominated pieces, or scanning the
various pieces in whole or part. (For example, eleven judges said
they would read an entry until they lost interest, and fourteen said
they could tell relatively quickly if an article was a potential
winner.) After that first dip into the pile, most would go back to
re-read or otherwise re-examine the pieces with most potential, and
there would then be a time to discuss findings with fellow judges and
review their own initial judgments.[47]
We suggest that in the first stage of this iterative process, judges'
top-of-mind inclinations would be likely to play a strong role; in
other words, the initial cut would be made more or less intuitively.
Then, when re-examining the high-potential pieces in a more
analytical way, the judges would likely be aware of a range of
received values that should be considered in determining excellence
elements drawn from journalistic tradition, culture and practice (and
therefore represented in the literature from which our twelve
predefined criteria were drawn). Trade-offs would have to be made
along the way as judges weigh the merits of pieces that excel in
different ways (for example, in originality versus relevance).
Finally, a judgment might have to be made between two or more
competing pieces that both or all seem worthy of the award: at this
point, intuition and personal preference would likely play strong
roles in determining the best story.
As the dark shading in Table 6 shows clearly, two themes writing
and reporting stood fast among judges' top-rated elements, no
matter what question about criteria was asked.[48] It seems clear
that these two elements are the dominant characteristics of
excellence in journalism in the interviewed judges' minds, throughout
the judging process. But the light shading in Table 6 may help paint
a more dynamic picture of judges' evaluations. Apart from the quality
of writing and the rigour of reporting, judges (in both newspaper and
magazine awards) clearly have both originality and relevance on their
minds when they think about journalistic excellence in an open-ended
way. It seems likely that these additional two criteria would be
influential during the initial cut. Then, in what we might call the
sober-second-thought phase (when the judges are reconsidering the
high-potential stories and thinking ahead to, or in the midst of,
consultations with their fellow panelists), individual judges'
top-of-mind inclinations may sometimes be overshadowed by the
standards of the journalistic community (especially, it seems,
integrity for the newspaper judges, and analysis for the magazine
judges, since these themes were rated highly as "trade-off" values).
In the end, though, if personal preference returns to the fore in
determining which story shines out as "best" amongst those that have
made the final cut, judges will focus once again on writing and
reporting, though relevance (in newspapers) or originality (in
magazines) also come into play.
How does this hierarchy compare to the themes that emerge from the
literature about quality in journalism? Writing is mentioned by
Gladney as one of his nine "content standards" of editorial
excellence, and is an important aspect of one of the ten "elements"
of Kovach and Rosenstiel: the striving to "make the significant
interesting and relevant." On the other hand, "literary style"
finished last among seven attributes of editorial quality rated in
Bogart's 1977 survey of U.S. newspaper editors and managing
editors[49], and none of the annual awards guidelines studied by us
gives particular prominence to writing style. The amount and rigour
of reporting, however, is clearly and pre-eminently emphasized
throughout the literature (though not necessarily in those words[50])
and, unsurprisingly, underlie important elements in all the surveyed
awards guidelines. Relevance or public impact does not tend to be
emphasized in the literature or awards guidelines to anywhere near
the extent that it apparently weighs on our newspaper judges' minds,
but originality is explicitly emphasized in all the reviewed awards
guidelines. Almost uncannily, the content and order of the criteria
defining the Best Australian Journalism of the 20th Century exactly
echo our deduced hierarchy of judges' values: writing is at the top
of the Australian list, followed by rigour in information-gathering
and "finding the story," and then public impact and originality![51]
Finally, it seems significant that both newspaper and magazine judges
mention writing much more often than reporting when they describe
qualities of excellence in an open-ended way (as shown by the number
of mentions for these criteria in Tables 1 and 4). We suggest,
therefore, that writing is likely to carry the most weight at the end
of the judging process when winners get determined. In other words,
where more than one story demonstrates excellence in reporting and in
other aspects, judges will probably favour the one that they consider
better written.
Several areas for further investigation and analysis have suggested
themselves during this initial phase of our project, including the
differences between newspaper and magazine judging and the
differences among judges in various awards categories. (The latter
difference could, in principle, prove an important limitation on the
validity of the tentative conclusions drawn in this paper, because
the small number of interviewees leaves us unable to control for the
influence of the categories on judges' criteria.)
We are also intrigued by the relatively low priority given to the
social-benefit or public-service value of a story in the minds of
most judges, given the widespread view that journalism's primary
social purpose is to help citizens participate in democracy. Indeed,
the content of journalism, in general, seems clearly secondary to
more ideologically neutral and perhaps more technical aspects,
such as the amount of reporting and the writing style. It would be
interesting to see whether this apparent neutrality to the social
purpose of journalism is echoed among journalism judges in the
United States (though we have no particular reason to suspect a
difference between the two nations in this respect).
All of that must await further study. The next logical step would be
a larger, quantifiable study of jurors in North America's leading
print journalism awards applying and testing the standards isolated
in this paper. Meanwhile, we hope we have made a modest contribution
toward a more developed understanding of the criteria of
excellence in journalism.
[1] The authors acknowledge with gratitude the assistance of Bryan
Cantley and staff of the National Newspaper Association and of Pat
Kendall and Terry Sellwood of the National Magazine Awards
Foundation; of Murray Pomerance, chair of the Department of
Sociology, Ryerson University; and of funding provided by the
Research Assistants Fund of Ryerson University and by the research
projects fund of the university's Faculty of Communication and
Design. Selected highlights of this study were informally presented
at a conference on "The Best Australian Journalism of the 20th
Century" hosted by the Royal Melbourne Institute of Technology in
Melbourne. Australia, on November 26, 2004.
[2] Brent Cunningham, "Excellence is the only guidepost," Columbia
Journalism Review 41, no. 1 May/Jun (2002): 40.
[3] Howard Gardner, Mihaly Csikszentmihalyi, and William Damon,
Good Work: When Excellence and Ethics Meet (New York: Basic Books, 2001), 35.
[4] Alicia C. Shepard, "Journalism's Prize Culture," American
Journalism Review, April 2000, 22-31. "By January," Shepard writes,
"life becomes insane [for those responsible for getting entries
submitted to the contests]. Twelve-hour days, working weekends,
hiring temporary employees to help with staggering amounts of paperwork."
[5] Shepard, Journalism's Prize Culture, 22.
[6] Leo Bogart, "Reflections on Content Quality in Newspapers,"
Newspaper Research Journal 25, no. 1 Winter (2004): 45.
[7] Thomas Kunkel, dean of the College of Journalism at the
University of Maryland, quoted by Shepard, Journalism's Prize Culture, 24.
[8] In 1994, Seymour Topping, the Pulitzer Prizes' administrator,
found himself in the position of denying, for the record, that the
competition was a "crapshoot." ("Discussing the Pulitzer prizes,"
Editor And Publisher 127, no. 27 (1994): 58-60.)
[9] See William Green, "Janet's World: The Story of a Child Who
Never Existed -- How and Why It Came to be Published," The Washington
Post, April 19 1981, sec. A, p. 1-14-15. Laurence G. O'Donnell, "The
Reflections of a Pulitzer Prize Juror," The Wall Street Journal,
April 13 (1982): 34. O'Donnell calls the Cooke incident "the ultimate
insult" to the Pulitzer Prize, which had already become, he says,
"controversial and flawed, hurt by whispers and suspicions that some
major newspapers organize their coverage to try to win Pulitzers and
that some years the judges play politics in picking the winners."
[10] Jay Rosen, "The Action of the Idea: Public Journalism in Built
Form," in The Idea of Public Journalism, ed. Theodore L. Glasser.
(New York: The Guilford Press, 1999), 22.
[11] Bill Kovach and Tom Rosenstiel, The Elements of Journalism:
What Newspeople Should Know and the Public Should Expect (New York:
Three Rivers Press, 2001), 12.
[12] Gardner, Csikszentmihalyi, and Damon, Good Work: When
Excellence and Ethics Meet, 291.
[13] See, for examples: Bogart, Reflections on Content Quality in
Newspapers, 40. Stephen Lacy and Frederick Fico, "The Link Between
Newspaper Content Quality and Circulation " Newspaper Research
Journal 12, no. 2 Spring 1991 (1991): 46-57. Stephen Lacy and
Frederick Fico, "Newspaper Quality and Ownership: Rating the Groups,"
Newspaper Research Journal 11, no. 2 Spring (1990): 42-57. John V.
Bodle, "Assessing news quality: A comparison between community and
student daily newspapers," Journalism and Mass Communication
Quarterly 73, no. 3 Autumn (1996): 672. On the relationship between
quality and profitability, see, for example: Philip Meyer, "Saving
Journalism: How to nurse the good stuff until it pays." Columbia
Journalism Review 43, no. 4 Nov/Dec (2004): 55-57. Geneva Overholser,
"Good Journalism and Business: An Industry Perspective," Newspaper
Research Journal 25, no. 1 Winter (2004): 8.
[14] Bogart, Reflections on Content Quality in Newspapers, 45.
[15] See Leo Bogart, Press and public : who reads what, when, where,
and why in American newspapers, 2nd ed. (Hillsdale, N.J.: L. Erlbaum
Associates, 1989), 260. Also, Lacy and Fico, The Link Between
Newspaper Content Quality and Circulation , 50.
[16] For a comprehensive survey, see Philip Meyer, The vanishing
newspaper: Saving journalism in the information age (Columbia:
University of Missouri Press, 2004), 269.
[17] George Albert Gladney, "Newspaper Excellence: How Editors of
Small & Large Papers Judge Quality," Newspaper Research Journal 11,
no. 2 Spring (1990): 58-72. George Albert Gladney, "How Editors and
Readers Rank and Rate the Importance of Eighteen Traditional
Standards of Newspaper Excellence " Journalism and Mass Communication
Quarterly 73, no. 2 (1996): 319.
[18] See, for example, Bogart, Reflections on Content Quality in
Newspapers, 44.
[19] For full lists, see Gladney, Newspaper Excellence: How Editors
of Small & Large Papers Judge Quality, 66-67.
[20] "The Best Australian journalism of the 20th century." (The
Fifth Estate , Royal Melbourne Institute of Technology)
<http://fifth.estate.rmit.edu.au/Febo4/106.html> (October 1, 2004).
[21] William B. Blankenburg and Richard L. Allen, "The journalism
contest thicket: is it time for some guidelines?" Associated Press
Managing Editors News , no. 76 (1974): 8.
[22] David C. Coulson, "Editors' Attitudes and Behavior Toward
Journalism Awards," Journalism Quarterly Spring (1989): 143-147.
[23] Randal A. Beam, Sharon Dunwoody, and Gerald M. Kosicki, "The
Relationship of Prize-winning to Prestige and Job Satisfaction,"
Journalism Quarterly Winter (1986): 693-699.
[24] The ethical conflict that may be involved when journalists
accept awards of this kind is explored by David Zinman, "Should
Newsmen Accept PR Prizes?" Columbia Journalism Review, Spring 1970, 37-43.
[25] Anonymous, "2003 Journalism Awards & Fellowships Directory,"
Editor & Publisher, Dec. 16 2002, J9-J30. These numbers exclude
honorary awards and citations, and fellowships, grants and
scholarships. A more limited list of 43 popular awards, posted at
Journalismjobs.com, runs from the American Association for the
Advancement of Science Awards to the Women's Economic Round Table,
Inc. prize for entrepreneurship journalism. "Journalism
Awards/Contests" (Journalismjobs.com)
<http://www.journalismjobs.com/awards.cfm> (March 20, 2005).
[26] Bob Ingle, "We're Drowning in a Sea of Media Awards," The
Masthead Fall-Winter (1982): 18.
[27] H. Allen White and Julie L. Andsager, "Winning Newspaper
Pulitzer Prizes: The (Possible) Advantage of Being a Competitive
Paper," Journalism Quarterly 67, no. Winter (1990): 912-919.
[28] "The Pulitzer Prizes," <http://pulitzer.org/> (12 March, 2005).
[29] O'Donnell, The Reflections of a Pulitzer Prize Juror, 34. Most
of these accounts tend to focus on the judging process, and
especially its intensity and pace. See, for example: John McCormally,
"Who Cares About the Pulitzer Prize?" More (Columbia Journalism
Review), May 1972, 9-11. Bill Dedman, "Picking the Pulitizers,"
Columbia Journalism Review 30, no. 1 May (1991): 41. McCormally also
alludes to the juror's search for "cutting-edge" journalism,
reportorial initiative, and "beautiful writing."
[30] "Inside the Pulitzers," Columbia Journalism Review 38, no. 1
May/Jun (1999): 26. For more comments from Pulitzer jurors focused
on the tendency for prizes to go to a handful of large metropolitan
papers see Cunningham, Excellence is the only guidepost, 40.
[31] Kathleen A. Hansen, "Information Richness and Newspaper
Pulitzer Prizes," Journalism Quarterly 67 Winter (1990): 930-935.
[32] American Society of Magazine Editors, "National Magazine
Awards; Category Definitions,"
<http://www.magazine.org/Editorial/National_Magazine_Awards/Category_Definitions/>
(30 November 2004).
[33] With regard to ethics, an ethics code is attached to the list
of criteria; the other ten terms are merely listed, not explained.
[34] "National Magazine Awards 2003/2004: Instructions for
first-tier text juries." (National Magazine Awards Foundation, Toronto.)
[35] Canadian Newspaper Association
<http://www.cna-acj.ca/client/cna/cna.nsf/web/NNARulesGuidelines>
(October 15, 2004).
[36] Canadian Newspaper Association web site (see above).
[37] David E. Sumner, a journalism professor who says he has been a
judge in many journalism contests, emphasizes the importance of
writing and story-telling technique. His six "tips" for winning
writing contests are narrative engagement, a positive meaning to the
story, a clear central theme, an element of suspense or uncertainty,
and a theme that "goes against the grain or contradicts conventional
wisdom." (David E. Sumner, "A Few Tips on How to do Well in Writing
Contests," Editor & Publisher 129, no. 27 (1996): 48.)
[38] No limit was set on the number of criteria coded in this way;
most judges mentioned more than one.
[39] Two student coders independently verified the classification of
responses and coding discrepancies were resolved case by case.
[40] Responses to Question 4: "There are many ways to measure
excellence in journalism and you probably had some particular
criteria in mind as you tried to identify the best of the nominated
pieces. What were those criteria? [Or, if the respondent could not
remember or was unable to say: Generally, what do you think are the
most important criteria of excellence in a piece of journalism?]"
Criteria mentioned were coded and then grouped.
[41] Judges' average scores, on a scale of zero to five, are rounded
to one decimal point.
[42] Responses to Question 6 (a): "How important would you say each
of the following criteria of excellence are on a scale of one to
five, with one being somewhat important and five being extremely
important? (1) The originality or uniqueness of the subject matter.
(2) The depth and amount of reporting or investigative rigour. (3)
Transparency of method (that is, the extent to which the audience
will understand where the information comes from). (4) Integrity,
fairness, and independence from the story's sources. (5) Thoroughness
in exploring the context of the story (that is, asking the "how" and
"why" questions). (6) Thoughtful and logical analysis of the subject
matter. (7) Superior writing style. (8) Innovative and engaging
technique. (9) Bringing clarity to a complicated subject. (10)
Taking on difficult subject matter. (11) Breaking news. (12) The
story is a benefit to society (public service)."
[43] Responses to Question 6 (b): "I'm going to read the list [from
Question 6 (a)] again. Can you tell me which two or three are the
most important overall?"
[44] Responses to Question 8: "Thinking just about the best piece
you ever judged, what did you especially like about it?"
[45] Not juxtaposed because juxtapositions were not intuitively
suggested were the "Awards criteria" group of open-ended responses
and the following scored criteria: "Transparency of method"; "Taking
on difficult subject matter"; "Breaking news." In any case, none of
these items was a big "winner" in the judges' responses to any question.
[46] We used the following conversions: C=3.0-3.4; B=3.5-3.9;
A=4.0-4.4; A+=4.5-4.9. We decided to use these "grades" rather than
precise numbers because the average scores were so close together
that we judged their relative rankings statistically insignificant,
especially given the small sample size.
[47] Some judging panels elected to split the nominated pieces
amongst themselves in order to identify potential winners: the whole
panel read only the high-potential nominees.
[48] For magazine judges, innovative technique an element that is
closely connected to writing style was also a consistent theme.
[49] Bogart, Reflections on Content Quality in Newspapers, 40.
[50] For instance, Kovach and Rosenstiel refer to the
comprehensiveness of the news and to reporting as a "discipline of
verification." (Kovach and Rosenstiel, The Elements of Journalism:
What Newspeople Should Know and the Public Should Expect,
207.) Gladney's "content standards" include "strong local coverage"
and "comprehensive coverage." (Gladney, How Editors and Readers Rank
and Rate the Importance of Eighteen Traditional Standards of
Newspaper Excellence , 319.).
[51] "The Best Australian journalism of the 20th century" (2004).
|
