|
Open, Closed, or Both:
Expanding the Flexibility of Question Formats in Survey Research
by
Paul S. Voakes
200Q Ernie Pyle Hall
Indiana University School of Journalism
Bloomington, IN 47405
(812) 855-1708
(e-mail) [log in to unmask]
submitted to
The Communication Theory and Methodology Division
of the Association for Education in Journalism
and Mass Communication
April 1, 1995
Open, Closed, or Both:
Expanding the Flexibility of Question Formats in Survey Research
One of the early, fundamental challenges in designing a survey instrument
is
deciding whether to use open-ended (free-response) or closed-ended
(fixed-choice)
questions. The advantages and disadvantages of each format have been debated
at
length. Rossi, Wright and Anderson (1983), for example, note that open-ended
questions provoke answers in the respondents' own words, uninfluenced by any
specific
alternatives suggested by the interviewer. Singleton, Straits, Straits and
McAllister
(1988) would add that open-ended questions are especially valuable at the
beginning of a
questionnaire, to induce respondents to "warm up" to the interview by speaking
freely on
a topic without being channeled into any specific line of thinking.
There are major disadvantages as well to open-ended questions. De Vaus
(1990)
points out that this format demands more effort from respondents, who may be
unwilling
to reveal their thoughts in their own words -- especially if they feel their
responses may
be perceived as socially undesirable. This format's greatest drawback, which
Rossi,
Wright and Anderson (1983) emphasize, is in the reliability of its measurement.
No
matter how well trained the coders are, the process of translating respondents'
spontaneous musings to numerical summaries invariably results in some degree of
error.
With closed-ended questions, the reliability of the measurement is
generally much
higher: The respondent has numerically "coded" her response at the time she
gives it.
De Vaus adds that closed-ended questions are not only less demanding (which is
important in settings such as a lengthy telephone interview) but they do not
arbitrarily
discriminate between the glib and verbally reticent respondent. More
importantly, the
closed-ended format enables the questions to follow the structure of the
theoretical
model that is driving the study.
But closed-ended formats too have their drawbacks. If there is an
unintended
nuance in the wording of a question, that nuance may threaten the validity of
the item.
And because of its lack of spontaneity, it may force respondents to choose among
alternatives that do not correspond to their true feelings, or it may suggest
responses that
they never would have considered otherwise (Rossi, Wright & Anderson, 1983).
Several contemporary researchers still cite Kahn and Cannell (1957) and
their
criteria for using open-ended questions. They recommend this format when the
research
seeks a basis upon which opinions are founded; when a spontaneous, uninfluenced
response is sought; and when respondents are unlikely to have given previous
thought to
the matter in question.
But in virtually all of the methodological discussion, the assumption seems
to be
that the researcher should choose one format or the other, depending on the
needs of the
particular project. The purpose of this paper is to question that assumption
and to
explore, by use of a case study, the possibility of combining open-ended and
closed-
ended questions in the same survey instrument, using the same methods of
quantitative
analysis, in pursuit of answers to the same research question.
A Case in Point: Journalistic Ethics
The case study begins with a survey of 118 journalists regarding their
decision-
making process as they consider an ethical dilemma. A theoretical framework was
constructed through the adaptation of a more comprehensive model of influences
on
media content, by Shoemaker and Reese (1991). The overall theoretical goal for
the
present model is to learn the relative importance of each determinant in
influencing a
particular journalistic behavior and thus influencing the content of the media.
The
present model proposes seven social determinants of ethical decision-making:
Individual -- the personal background and style of moral reasoning that
each
journalist brings to her work;
Small Group -- the influence of peers and informal social constraints;
Organization -- company policies and other formal or structural
constraints;
Competition -- reaction to (or anticipation of) the work of competing
media;
Occupation -- the influence of training, norms, professional codes of
ethics
and other notions of professionalism;
Extramedia -- consideration of news subjects, news sources and
audiences;
Legal -- the influence of law, in both formal and informal
constructions.
Shoemaker and Reese suggest that there exists a "hierarchy of influences"
on
media content, and one of the key research questions here is whether such a
hierarchy
exists in influencing ethical decision-making (which in itself can be viewed as
a subset of
antecedents of media content).
This conception of influences on ethical decision-making would seem to
challenge the more traditional assumption among media ethicists (see for example
Lambeth, 1986, and Klaidman & Beauchamp, 1986) that ethics is done through a
process of moral reasoning that uses, either inductively or deductively, moral
principles.
The implication is that the process is internal and highly individual. Even
much of the
empirical work in this area suggests the notion of individual, autonomous moral
agency
(see for example Meyer, 1987; Wright, 1989; Shamir, Reed & Connell, 1990;
Singletary,
Caudill, Caudill & White, 1990; White & Pearce, 1991; White & Singletary, 1993).
The principal research instrument was a telephone survey of 118
probability-
sampled journalists in southern Wisconsin, designed to address the fundamental
research
questions: Under what conditions and influences is a certain decision made in an
ethical
situation, and is there a hierarchy of these influences?
The telephone interview schedule began with the presentation of three
hypothetical scenarios. In the first, a news reporter and photographer sneak
into a
nursing home (on which they are doing an investigative piece, and from which
they had
been formally denied access) by posing as janitors. The second scenario
concerns
whether to publish (or broadcast) the name of a teen-age suspect in a
highly-publicized
murder in the community. In the third situation, a mayoral candidate is the
target of
allegations, lodged by disgruntled former employees, of improprieties. A news
outlet
decides to report the allegations without verifying them or presenting the
candidate's side
of the story.
The first question after the telling of each scenario is whether the
journalist's
behavior is acceptable to the respondent, on a numerical rating scale of one to
ten, where
one is utterly unacceptable and ten is perfectly acceptable. This rating of
acceptability,
especially when it is combined across all three scenarios, becomes the key
dependent
variable in the study.
Of greater concern in this study than the perceptions of rightness or
wrongness,
however, are the primary independent variables that comprise the hierarchy of
influences
upon that rating of acceptability. The relative strength of each variable
concept was
measured in two ways: in an open-ended and a closed-ended format of questioning.
First, after rating each story for its journalistic acceptability, the
respondent was asked to
discuss, in an open-ended format, what factors or considerations led to the
assessment of
acceptability. (Later, the first three idea elements mentioned by each
respondent were
categorized according to the social determinant each represented.) After thus
reacting to
each scenario in her own words, the respondent then reacted to closed-ended
items, each
intended to represent one of the seven social determinants. Respondents were
asked to
rate the salience of each statement on a scale of one to ten, where one meant
that the
statement was irrelevant to her thinking about the scenario, and ten meant that
the
statement was highly important to her thinking. The set of closed-ended items
was read
three times, once after each scenario.
The central hypothesis of the study was this:
Among the seven independent variable concepts, the Individual influence
will enjoy the strongest predictive value across all three scenarios. This
reflects the
general findings from the literature that personal, intrinsic values, largely
unfiltered by
social influences, have great predictive power. The hypothesis was tested with
a
comparison of the means of the relevant items, and with multiple regression to
determine
the relative power of influence enjoyed by each of the key independent
variables. As
tables 1 through 4 demonstrate, the hypothesis lies in disgrance. The overall
salience of
Individual items was lower than the mean for all determinants, and the
relationship
between the items and the dependent variable remained nonsignificant throughout
the
regression analysis. Individual moral autonomy seems not to prevail in ethical
decision-
making; the data support the notion of a far more dynamic and complex pattern of
social
influences.
But the results of each statistical analysis are stated here twice: once
each time
for open-ended responses and closed-ended responses. Inferring from the
discussion of
the literature above on open-ended methods, it seemed possible that through a
free-
response format, social influences might be identified that were not included
among the
closed-ended indicators. These conditions seem to argue for the use of
open-ended
questions, especially considering the context of journalism ethics. Most of the
existing
empirical work tells us whether journalists accept certain behavior but does not
attempt
to tell us why or how. Even if the previous surveys presented journalists with
a list of
possible reasons for their ethical decisions, the bias of social desirability
(DeMaio, 1984)
might nudge the respondent toward the most virtuous-sounding of the choices.
Respondents naturally want to make a good impression on the researchers and the
Table 1
Means of the Open-Ended Measures
of Social Determinants
Nursing Home Juvenile's Name Candidate Overall
Individual .32 (.52) .36 (.59) .13** (.34)
.81**
Small Group .03** (.18) .00** (.00)
.02** (.13) .05**
Organizational .44 (.61) .59** (.52)
.11** (.76) 1.10
Occupational .95** (.75) .25 (.47) 1.47**
(.76) 2.66**
Extramedia .36 (.55) .30 (.60) .23*
(.25) .89
Legal .11** (.34) .38 (.62) .11**
(.48) .60**
Column mean .37 .31 .34 1.02
n = 118
Notes:
1) Values expressed are the means (and standard deviations) of the number
of mentions
coded into each variable category for each scenario.
2) Asterisks denote significant difference, either higher or lower, from
the mean of mentions
for that particular column, as determined by a two-tailed t-test (* p<.05,
** p<.01).
research audience by choosing the most ethically sensitive response.
However, the need to test a specifically-structured theoretical model
equally
strongly suggests the use of a corresponding structure of closed-ended
questions. The
approach therefore has been to employ both methods. It is hoped here that
the indirect
wording of the closed-ended indicators and the scattered ordering of the
items has reduced
the possibility of desirability bias or pattern bias.
Table 2
Means of the Closed-ended Measures
of Social Determinants
Nursing Home Juvenile's Name
Candidate Overall
Individual 6.97 (2.43) 6.63* (2.59) 6.96 (2.38) 6.87 (2.17)
Personal Values 7.01 (2.87) 6.69 (2.80) 6.91 (2.68) 6.88
(2.44)
Own Logic 6.92 (2.68) 6.56* (2.69) 6.95 (2.49) 6.82 (2.31)
Small Group 7.00 (2.24) 7.38 (2.10) 7.47** (1.95) 7.27
(1.81)
Observe others 6.01** (2.75) 6.47* (2.77) 6.47 (2.59) 6.31*
(2.35)
Talk to others 7.99** (2.70) 8.31** (2.40) 8.47** (1.93) 8.26** (2.04)
Organizational 8.10** (2.77) 9.15** (1.68) 7.63** (2.56) 8.32**
(1.92)
Competition 6.11** (2.67) 5.53** (2.65) 6.17** (2.71)
5.97** (2.35)
Occupational 7.27 (2.11) 7.46 (2.07) 7.41** (1.80) 7.37*
(1.69)
'Professionalism' 7.86** (2.59) 7.73* (2.26) 8.07** (1.95)
7.90** (1.88)
Code of Ethics 6.64 (2.61) 7.19 (2.68) 6.77 (2.38) 6.87
(2.14)
Extramedia 5.09** (2.47) 5.90** (2.40) 5.58** (2.27) 5.53**
(1.95)
Audience 5.44** (2.87) 5.43** (2.71) 5.98** (2.67) 5.61** (2.32)
Subjects 4.70** (3.21) 6.34* (2.76) 5.17** (2.62) 5.42** (2.21)
Legal 7.83** (2.69) 8.16** (1.67) 7.21 (2.23) 7.73**
(1.72)
The Law Itself 8.28** (2.44) 9.26** (1.35) 7.68** (2.53) 8.40**
(1.65)
Threat of Suit 7.38 (2.69) 7.07 (2.76) 6.73 (2.93) 7.07 (2.42)
Column mean 6.91 7.17 6.92 7.01
n = 118
Notes:
1) Values expressed are the means (and standard deviations) of the
responses (on a
scale from 1 to 10) to closed-ended items.
2) The indented phrases represent the individual items that comprise the
boldfaced,
composite variables immediately above them..
3) Asterisks denote significant difference, either higher or lower, from
the mean of the
responses for that column, as determined by a two-tailed t-test (* p<.05, **
p<.01).
Table 3
Predicting Acceptability of Journalists'
Actions
in Ethical Situations
Multiple Regression
OPEN-ENDED RESPONSES
Simple r Step 1 Step 2
Step 3 Incr. R2
Demographics .04
Gender (male) .00 .04 .04 .07
Age -.16 -.15 -.04 -.04
Education .12 .13 .13 .09
Other Key Characteristics .03
Ever threatened
with a Lawsuit .11 .19 .19*
Yrs. in Journalism -.13 -.16 -.19
Social Determinants .13*
Individual .06 .02
Small Group .16 .24*
Occupational .08 .09
Organizational -.18* -.13
Extramedia .23* .24*
Legal -.08 -.05
Equation F-ratio 2.39 (.01)
n = 118, ^p<.10, *p<.05, **p<.01
Notes:
1) Dependent variable is the acceptability of journalists' actions, summed
across all three
scenarios.
2) "Other Key Characteristics" are objective variables whose strong
bivariate associations
had suggested possible predictive power in a regression equation.
3) "Equation F-ratio" refers to the ANOVA test as to whether the blocks of
variables
entered in the equation explained a significant amount of the variance in
the dependent variable.
Table 4
Predicting Acceptability of Journalists' Actions
in Ethical Situations
Multiple Regression
CLOSED-ENDED RESPONSES
Simple r Step 1 Step 2
Step 3 Incr. R2
Demographics .04
Gender (male) .00 .04 .04 -.02
Age -.16 -.15 -.04 -.11
Education .12 .13 .13 .04
Other Key Characteristics .03
Ever Threatened
with a Lawsuit .11 .19 .11
Yrs./ Journalism -.13 -.16 -.12
Social Determinants .15**
Individual -.04 .00
Small Group -.07 .21^
Competition .05 .20^
Organizational -.21* -.06
Occupational -.23* -.18
Extramedia -.21* -.22^
Legal -.26** -.26*
Equation F-ratio 2.40 (.009)
n = 118, ^p<.10, *p<.05, **p<.01
Notes:
1) Dependent variable is the acceptability of journalists' actions, summed
across all three
scenarios.
2) "Other Key Characteristics" are objective variables whose strong
bivariate associations
had suggested possible predictive power in a regression equation.
3) In Step 3, the coefficients of Small Group, Competition and Extramedia
achieved alpha
levels of significance, respectively, of .065, .062, and .053.
4) "Equation F-ratio" refers to the ANOVA test as to whether the blocks of
variables
entered in the equation explained a significant amount of the variance in
the dependent variable.
Some Unsettling Discrepancies
The rationale of multi-method triangulation is that dissimilar methods will
suffer
different methodological weaknesses, and thus exhibit different patterns of
error. If two
methods independently produce the same general findings, our confidence in
the findings
increases. Those hopes were tested statistically in this study, but with
results that are
not encouraging for direct triangulation. Table 5 shows simple correlations
between
open-ended response categories and closed-ended response categories, across
all three
scenarios. The underlined values are the correlations between measures of
the same
concept -- the measures that we might expect to be highly correlated.
Obviously, they are not. The correlations are nonsignificant, and with two
of the
concepts, Occupational and Extramedia, they seem to drift toward a negative
relationship. There are two possible interpretations of the
non-correspondence: One of
the two methods has utterly failed to capture the meaning of the social
determinants, and
therefore it should be rejected as inaccurate and invalid. Or, the two
methods are simply
capturing different dimensions of the same concepts and should both be
reported.
The discrepancy is not unique to this project. To revisit the literature
more
specifically, Schuman and Presser (1979) systematically explored the
discrepancy in
results by designing part of a survey instrument with parallel open and
closed questions.
Half the respondents were randomly assigned to an open-ended question, and
half to a
closed-ended version of the same question (asking what benefits of a job the
respondent
would most prefer). The two versions produced different results: For
example, nearly 20
Table 5
Relationships Between Open- and Closed-Ended
Measures
Of the Same Independent Variable Concepts
Open-Ended Measures
Indiv. Sm Grp. Org. Occup.
Extram. Legal
Closed-Ended
Individual .10 -.06 -.00 -.09 -.01 -.03
Small Group .00 .03 .11 .02 -.08 -.16
Organizational -.09 -.01 -.01 .15 -.24* .02
Competitive -.03 -.04 -.01 .05 .10 .04
Occupational .12 -.04 -.01 -.06 -.12 .01
Extramedia -.11 -.07 .12 .04 -.04 -.06
Legal -.05 .16 .24* -.02 -.17 -.01
n = 118, *p<.05, **p<.01
Notes:
1) Values shown are Pearson correlation coefficients.
2) Underlined coefficients indicate theoretically similar concepts.
3) Values shown summarize results across all three scenarios.
percent of the closed-ended respondents cited advancement as the most
important job
value, whereas only 2 percent preferred those opportunities in the
open-ended responses.
Inexplicably, however, the authors, like many others on this subject (see
Kahn &
Cannell, 1957; Rossi et al., 1983; De Vaus, 1990; and Singleton et al.,
1988) conclude
that properly constructed closed-ended items can provide more valid
information than
open-ended responses can, and they recommend the use of open-ended questions
only in
certain exploratory or qualitative situations.
But is it really just a matter of superior validity in one method and
inferior
validity in another? The validity of each method can be tested, one against
the other,
more directly than in Schuman's and Presser's experiment. One strategy is
to ask which
Table 6
Comparing the Predictive Power
of Open- and Closed-ended Measures
of the Same Independent Variable Concepts
Multiple Regression
Order of Entering Blocks
Incr. R2
Closed-ended .17**
Open-ended .08
Open-ended .13*
Closed-ended .13*
n = 118, *p<.05, **p<.01
Notes:
1) Dependent variable is the acceptability of journalists' actions, summed
across all three
scenarios.
2) Each block consisted of the aggregate scores, across all three
scenarios, of the six
(open-ended) or seven (closed-ended) independent variables being measured.
set of measures, open-ended or closed-ended, seems to have greater power to
predict the
dependent variable (in our case, the acceptability of the journalist's
actions in each
scenario). If one method was failing to capture the meaning of the
concepts, its results
would be scattered and it would probably fail to explain much of the
variance of the
dependent variable. Thus a multiple regression was computed from this
study's data that
pitted the block of closed-ended variables against the block of open-ended
variables.
The results are shown in Table 6.
When the closed-ended block of variables was entered first in the
regression
equation, its incremental R-square (the standard measure for the degree of
explanation of
total variance) was substantially greater than that of the open-ended
variables. However,
when the order of entry was reversed, the incremental R-squares were almost
identical --
and both significant.
The test seems to be saying that both open- and closed-ended responses,
while at
times unrelated to each other, both contribute significantly to explanations
for why the
respondents answered as they did on the matter of acceptability. Could it be
that each
method is tapping a different set of (equally valid) considerations? This
survey
instrument included only two closed-ended items to represent each social
determinant
No matter how accurately the two represented that concept, there are
certainly many
other facets (dimensions) of the same concept that have meaning for the
respondent but
simply are not tapped in the closed-ended instrument. For example, the
items measuring
Occupational influence covered a general conception of professionalism, and
a reference
to a professional code of ethics. In the open-ended responses, a large
number of
respondents cited specific norms of "professional" journalistic behavior --
not necessarily
thinking about the origins of those norms or about professionalism per se.
It is entirely
possible that codes of conduct could be one dimension of the Occupational
influence,
training could be another dimension, and the situation-specific norms of
journalism --
the "do's and don't's" -- could be another dimension. The open- and
closed-ended
measures together did tap all three, but no single item or method managed to
tap the
three at once.
To cite a more intriguing example, the actual directions of the Extramedia
influence, in open- and closed-ended formats, are different (see tables 3
and 4). In the
open-ended responses, journalists who cited Extramedia factors on their own
tended to
support the ethically and legally risky decisions that were made in the
scenarios. In the
closed-ended format, with an equally strong degree of influence, those who
assigned
importance to Extramedia factors tended to disapprove of the hypothetical
actions.
How can we explain this apparent contradiction?
As with the other social determinants, Extramedia was represented in the
survey
by two closed-ended indicators. "I wonder what the (subject or source) will
think when
they see the story" represents the perception of the personal influence of
news sources or
news subjects. The other indicator, "I wonder what our readers (viewers)
will think
when they see this story" represents the more general anticipation of
audience reaction.
It is hardly surprising that statements of concern over the reactions of
sources, subjects
or audience would induce a reluctance to proceed with the risky story.
The open-ended responses were not tapping the same dimension of the
concept.
There were references to audience and subjects, to be sure, but a
substantial number of
responses mentioned a different, unanticipated aspect of audience influence:
the broader
impact of a story on the general community. Whether the "public benefit"
(like other
aspects of audience influence) is ever measured by news organizations is a
matter of
some doubt. The suggestion here is that its presence does nonetheless play
a part -- at
least as documented by several open-ended responses -- in the journalist's
decision. The
other unanticipated piece of reasoning in the open-ended responses concerned
another
dimension of source influence -- a notion we might summarize as "just
deserts." The
feeling among several respondents is that the source or subject involved
deserves
whatever consequences might arise from the ethically risky story; therefore
the decision
to proceed is justifiable.
Differences in Cognition?
There is also very likely a difference in the cognitive processing and
response
strategies in the two different methodsof questioning. There has been
considerable
work in the field of social cognition on questions rather close to this one.
Social
psychologists, working primarily in the domain of personality or candidate
evaluation,
have discovered many cases in which people's open-ended thoughts are
uncorrelated
with responses to closed-ended questions about the same subject (Zaller,
1992). Studies
have distinguished two processes of evaluative judgment: memory-based and
"on-line"
(Hastie & Park, 1986). Memory-based judgment relies on the retrieval of
facts,
reasoning and attitudes from long-term memory; "on-line" processing involves
the
frequent and immediate updating and adjustment of evaluations as new pieces
of
information are acquired. Thus an on-line evaluation is not the same
independent
judgment that a memory-based judgment is, because it has been colored by
recent cues
as to the evaluation. Lodge, McGraw and Stroh (1989) found that on-line
processing is
a stronger determinant than memory-based judgment of citizens' evaluations
of political
candidates. They concluded, however, that the human mind does not rely
exclusively on
either mode of decision-making. Independent, long-term recall memory still
seems to
have an influence on one's evaluative decision. But on-line processing
seems to have a
stronger relationship to the decision, they found, under many conditions.
The research on cognition has obvious implications for the present
methodological discussion. The closed-ended response is a response to
suggestion. It
relies on recognition memory and would seem to use the mode of "on-line"
processing,
as respondents are given cues that immediately update their evaluation of
the forces at
work. Even if a social determinant -- Small Group influence, for example
-- never
entered the mind of the respondent during open-ended questioning, it was
always
presented as a closed-ended option, and almost always the respondent gave
those items
some level of numerical score. The item may remind the respondent of an
influence that
is honestly important, and the score reflects that importance.
Alternatively, it could
simply invoke social desirability, wherein the respondent feels she ought to
have
mentioned that influence (personal values or the law, for example) in her
open-ended
response and now has an opportunity to declare that determinant's
importance. The
open-ended response, on the other hand, sought at the beginning of the
interview
schedule, is spontaneous and unprompted. It relies on recall memory. It
may capture
the most "honest" thoughts off the top of the respondent's head, often in
the form of
schematics or heuristic solutions that are applied to everyday problems.
But this format
is also vulnerable to the respondent's inability to summarize
comprehensively her view of
the situation quickly over the telephone. She could simply forget to mention
a
consideration that is extremely important to her.
The most instructive example of this difference from the present study is
in the
measurement of the Legal influence on an ethical decision. In Table 2 we
see a
significantly high mean response for the closed-ended Legal items, and in
Table 4 the
Legal influence held its significant Beta coefficient throughout the
regression of closed-
ended variables. And the association was negative. When asked to assess
the salience of
Legal considerations as closed-ended items, the journalists were usually
quick to accord
them high salience, and that appreciation led in turn to a disapproval of
the ethically risky
actions of the journalists in the hypothetical scenarios.
But the recognition memory on display in the closed-ended format seems not
to
tell the whole story: When left to their own recall memory, the respondents
did not often
think to mention Legal considerations. The open-ended format seems to put
Legal
influence near the bottom of the hierarchy. According to Table 1, a Legal
"mention" was
made on the average of only once every five respondents. In terms of its
influence on
the dependent variable, the Legal variable in the open-ended format never
approaches
significance in one direction or the other. The discrepancy suggests that
perhaps the
influence of the law is often mediated by the other social influences. For
example, the
open-ended response "You just don't use the name of a juvenile suspect" was
coded as
an Occupational consideration -- a behavioral norm common to most
journalists. It
could be, however, that the pronouncement was also a reflection of some
legal result that
preceded the understanding of that norm, such as a lawsuit or
contempt-of-court citation
against a colleague, and is now incorporated into that journalist's
conception of
professionalism. While a detailed discussion of the ethical and legal
implications of the
findings is beyond the purview of this particular paper, it is worth noting
that journalists
do hold a fairly high degree of respect for the legal rules of journalism,
but only when
reminded of those rules. Left to their own devices, they more likely fall
under the
influence of any of the other determinants in the model.
Computational Differences
Not only can conceptualization and cognition explain differences between
the
two methods' results, but the statistical organization of the results also
suggests different
results as well. The closed-ended items were given scores between one and
ten, and
computed on the same scale. With open-ended responses, the first three idea
elements
(see Gamson & Modigliani, 1987) expressed were coded, and these "mentions"
were
given equal weight as they were counted. Thus a Small Group influence could
be
important to a respondent, but the first three mentions of her response may
have been
bundled in a monologue about occupational norms and company policy. While
she may
have given the Small Group items the exact same score as she gave
Occupational and
Organizational items in the closed-ended segment, her open-ended response
necessarily
gives Small Group a score of zero, simply because it was not among her first
three
mentions. Indeed, Small Group may not have been even among the first ten
mentions if
extended questioning were conducted, but the determinant of Small Groups may
still be
important to her.
Conclusion
Thus it seems unwise to jettison either method of measurement. Just as
Lodge,
McGraw and Stroh made room for both cognitive processes in their conceptual
model of
political evaluations, we can utilize both methods of question formats to
shed light on the
present research questions. The open-ended responses were spontaneous,
without a
theoretical superstructure imposed on the respondents. The closed-ended
items
represented a structure that does seem to carry a great deal of theoretical
validity.
In most of the reporting of the results of this study, therefore, the
closed-ended and
open-ended findings appear together, with their similarities or contrasts
open for
discussion.
One of the basic tasks of empirical work in the behavioral sciences, it
seems, is to
measure the indicators of variable concepts. It is highly unlikely,
however, that any one
set of indicators can capture and represent every dimension that is imbedded
in the
meaning of a concept. Therefore it is unlikely that open-ended and
closed-ended
strategies to address the same research question will yield highly
correlated results every
time. Part of the reason for the lack of correlation could be that each
format draws upon
a different cognitive process -- each with an influence upon
decision-making, but one
with a more direct influence than the other. Further research, perhaps in a
more
controlled (experimental) setting, may provide some elaboration of this
cognitive
relationship.
It seems clear from the results presented here that researchers who choose
one
question format at the expense of the other are choosing not to gather
information that
may be pertinent to their research objectives. When both formats are used
the results are
certainly sloppier, and require far more qualifiers in their reporting. But
the results are
also richer -- and closer to the complex behavioral "realities" that this
research is meant
to discover.
References
De Maio, T. J. (1984). Social Desirability and Survey Measurement: A
Review. In C.
F. Turner & E. Martin (Eds.), Surveying Subjective Phenomena (Vol. 2).
Beverly Hills: Sage Publications.
De Vaus, D. A. (1990). Surveys in Social Research (2nd ed.). London: Unwin
Hyman.
Gamson, W. A., & Modigliani, A. (1987). The Changing Culture of Affirmative
Action.
In R. G. Braungart & M. Braungart (Eds.), Research in Political Sociology
(pp.
137-177). Greenwich, CN: JAI Press.
Hastie, R., & Park, B. (1986). The Relationship Between Memory and
Judgment
Depends on Whether the Judgment Task is Memory-based or On-Line.
Psychological Review, 93, 258-268.
Kahn, R. L., & Cannell, C. F. (1957). The Dynamics of Interviewing:
Theory,
Technique, and Cases. New York: Wiley.
Klaidman, S., & Beauchamp, T. L. (1992). The Virtuous Journalist: Morality
in
Journalism. In E. D. Cohen (Ed.), Philosophical Issues in Journalism.
New
York: Oxford University Press.
Lambeth, E. B. (1986). Committed Journalism: An Ethic for the Profession.
Bloomington, Ind.,: Indiana University Press.
Lodge, M., McGraw, K., & Stroh, P. (1989). An Impression-based Model of
Candidate
Formation. American Political Science Reivew, 83, 399-420.
Meyer, P. (1987). Ethical Journalism: A Guide for Students, Practitioners,
and
Consumers. New York: Longman.
Rossi, P. H., Wright, J. D., & Anderson, A. B. (1983). Handbook of Survey
Research.
San Diego: Academic Press, Inc.
Schuman, H., & Presser, S. (1979). The Open and Closed Question. American
Sociological Review, 44, 692-712.
Shamir, J., Reed, B. S., & Connell, S. (1990). Individual Differences in
Ethical Values
of Public Relations Practitioners. Journalism Quarterly, 67 (4),
956-965.
Shoemaker, P. J., & Reese, S. D. (1991). Mediating the Message: Theories
of
Influences on Mass Media Content. New York: Longman Publishing Group.
Singletary, M. W., Caudill, S., Caudill, E., & White, A. (1990). Motives
for Ethical
Decision-Making. Journalism Quarterly, 67 (4), 964-971.
Singleton, R., Straits, B. C., Straits, M. M., & McAllister, R. J. (1988).
Approaches to
Social Research. New York: Oxford University Press.
White, H. A., & Pearce, R. C. (1991). Validating an Ethical Motivations
Scale:
Convergence and Predictive Ability. Journalism Quarterly, 68,
455-464.
___________ & Singletary, M. W. (1993). Internal Work Motivation: Predictor
of
Using Ethical Heuristics and Motivation. Journalism Quarterly, 70 (2),
381-
392.
Wright, D. K. (1989). Examining Ethical and Moral Values of Public
Relations People.
Public Relations Review, 15, 19-33.
Zaller, J. (1992). The Nature and Origins of Mass Opinion. New York:
Cambridge
University Press.
Notes
|
