Content-Type: text/html

Two Schools -





Two Schools, Two Time Zones, One Set of PR Class Objectives: On Asynchronous Learning Networks an In-Class Discussions

Introduction
In terms of both information delivery and content, public relations educators and their colleagues in journalism and mass communication are facing an enormous wave of technology-related innovation in pedagogy.
Pavlik and Powell (2000) note recent trends in how new media are incorporated into journalism and mass communication curricula. They advise educators to employ an integrated model of education with new media, in which new technologies are part of all classes, rather than a stand-alone model, in which only one or two courses in a program focus on technology. Curtin and Witherspoon (1999) found that PR educators are heeding this advice, integrating a wide range of computer skills into their curricula. This integration of technology into the classroom also responds to the demand in professional circles for tech-savvy graduates (e.g., Commission on Public Relations Education, 1999; Gower & Cho, 2000).
While both students and employers are looking for technology-based skills, PR educators should not attempt to apply technology simply for technology's sake. Content must not be overlooked. This paper focuses on how specific curriculum goals can be achieved employing new technologies. The first step is to define the domain of our instruction.
Web-supported Instruction
Although distance learning and entirely Web-based instruction are becoming more common in PR education, this paper looks to new media only as a support mechanism for traditional classroom instruction. Blake noted the "false dilemma" between traditional courses and computer-mediated instruction and found, not surprisingly, that many journalism students prefer a combination of face-to-face instruction and computer-mediated instruction (2000, p. 12).
        Nonetheless, computer-mediated instruction requires different pedagogical thinking. Traditional models are dominated by the idea of a teacher as the single source of information. But with new communication technologies that are just as new to teachers as they are to students, the idea of an "omnipotent teacher" or "sage on the stage" seems less realistic.  Rather, emerging technologies may be better taught by an experienced and critical "guide on the side" (Pavlik and Powell, 2000, p. 10; Rossman, 1999).
        The current study considers the utility of asynchronous learning networks (ALNs) to support pedagogical objectives in advanced public relations courses.  The asynchronous learning network discussed in this study is basically an online discussion forum put to use by advanced undergraduate public relations students and their respective teachers at two distant state universities. Although the two classes had many distinct educational goals, the courses were designed to use the ALNs to support a single subset of common objectives synchronized within the two syllabi.

Potential Pedagogical Benefits of the Online Discussion Forum
Using ALNs to overcome the constraints of time and geographical distance allows students from far-apart cities and time zones participate in mutual learning activities. This aspect of ALNs increases the feasibility of cross-cultural input. As PR teachers work to prepare their students for the increasingly global and intercultural demands of the profession, they seek to broaden students' perspectives on issues and the publics involved with those issues. Can ALNs help them reach this goal?
Aside from the obvious conveniences of anytime, anywhere instruction, ALNs have been reported to support a wide range of pedagogical goals. They allow students to participate in conversations at their own pace, digesting information, seeking additional information, and reading others' responses before making thoughtful responses to forum items (Brace-Govan and Clulow, 2000).
Wheeler, Valacich, Alavi and Vogul (1995) discussed the potential benefits of sharing heterogeneous student and faculty perspectives in collaborative learning environments. Likewise, several researchers have taken a constructivist approach to identify the benefits of collaborative online learning (Hiltz, Coppola, Rotter, Turoff, & Benbunan-Finch, 2000; McAlpine, 2000). The constructivist approach emphasizes the different experiences that each learner brings to the discussions and the possible benefits of sharing these perspectives.
However, the shift from "sage on the stage" to "guide on the side" models of teaching with technology has met fair criticism. Lauzon (2000), for example, warns of letting technology define content in an age when pedagogical decisions are especially influenced by financial factors. The appeal of cheap, convenient instructional delivery via new technology should not preclude sound educational objectives. Online discussions - just 1ike face-to-face discussions in the classroom - should carefully be guided by such objectives. Although increased access to information offered by new technologies is important, content is the key to a productive educational experience (Thompson and McGrath, 1999).
Studies published regarding the outcomes of ALNs and similar online teaching tools commonly gauge the value of these tools by the level of student or teacher satisfaction.  In a notable exception, Hiltz et al. (2000) combined qualitative data with the results of a quasi-experimental design to test the hypothesis that learning via an ALN can improve measurable learning outcomes. Hiltz et al. presented students with an ethical case in a Computers and Society course and found that students working in ALNs submitted reports of higher quality regarding the case than students in in-class conditions, as rated by independent judges.
The current study takes a similar approach to measure learning outcomes with PR-specific content. Both ALNs and in-class discussions were used as forums for discussing public relations cases and issues.

Objectives and Research Questions
Four specific teaching objectives related to central areas of PR curriculum, including issues management and segmentation of publics, were written prior to the discussions. In terms of Bloom's (1956) taxonomy, these objectives range from knowledge and comprehension (Objectives 1 and 2) to synthesis and evaluation (Objectives 3 and 4):
ù       Objective 1: Students will identify a broad range of issues relevant to the cases discussed.
ù       Objective 2: Students will identify a broad range of publics involved with the cases discussed.
ù       Objective 3: Students will articulate the central issue of each case discussed.
ù       Objective 4: Students will choose a position on the cases discussed and offer compelling support for their position.

Given these objectives, a study was designed to compare the outcomes of online ALNs shared by two distant PR classes to traditional in-class discussions that took place within each class that participated.
ù       RQ1: Will students who participate in an online discussion forum with students from distant geographical areas demonstrate greater success in achieving course objectives than their counterparts who participate in an in-class forum?
ù       RQ2: Will student satisfaction with the forums correlate with achievement of learning outcomes?
ù       RQ3: What are the qualitative differences between online discussions of PR cases among two distant university PR classes and in-class discussions within each class?

Procedure
Twenty students enrolled in Advanced Public Relations at the University of Hawaii at Manoa and 18 students enrolled in Public Relations Techniques at Kansas State University participated in this study during a three-week period in the spring semester of 2001. All 38 students were introduced to the online forum with in-class demonstrations during the week prior to the experiment. They were then asked to log on to the forum and post a brief self-introduction for their own classmates and students at the other university to read.
Week 1. On Tuesday, students were assigned reading from a recent "Face Off" section of The Public Relations Strategist. The reading, "Companies in Schools," presented two sides to the issue of corporate sponsorship in schools. On Thursday, the teachers randomly assigned each student to one of two groups. The first group was asked to log on to the ALN and participate in an online discussion about the case with other students from their own school and the other university. To start the threads, the teachers posted three questions related to the case. The questions asked students to discuss related PR issues, target publics and possible solutions to the PR problems. Each student in the online/ALN condition was asked to contribute to each of the three threads at least once before the following Monday.
The second group was asked to stay and discuss the case study in class. Of course, the face-to-face, in-class discussions only included classmates within each university, with no interaction between universities. The teachers prompted in-class discussion with the same questions presented online. To hold teacher contributions relatively constant across experimental conditions and universities, each teacher only passively participated in the discussions after introducing the discussion prompts.
        Week 2. On Tuesday, all students from both groups were asked to write responses to four open-ended questions based on a hypothetical scenario related to the previous week's case: "Suppose you are hired as a PR consultant for the superintendent of schools_ She e-mails you_ 'Microsoft approached our school board last week with a proposal to donate computers_'"
        After completing the open-ended questions, students were asked to complete a confidential questionnaire including Likert-type items adapted from previous studies that measured student satisfaction with discussion forums (Gunawardena & Duphorne, 2000; Hiltz et al., 2000; McAlpine, 2000).
At the end of class on Tuesday, the teachers assigned a second reading, also from the "Face Off" section of a recent edition of The Public Relations Strategist. The second reading assignment concerned the World Trade Organization (WTO).
        On Thursday, students who participated in online discussions the previous week were asked to participate in in-class discussions and vice-versa. Forums were managed in the same manner as the previous week.
        Week 3. Again, all students from both groups were asked to answer open-ended questions related to a scenario based on the previous week's case: "Suppose you are hired as a PR consultant for a U.S. congressman_ He e-mails you_ 'the WTO will meet next year in my district_'" The WTO questions paralleled prior questions from the Companies in Schools case. And again, students were asked to complete a questionnaire reporting their satisfaction with the forum. Students were then debriefed about the study and asked for any further oral feedback.

Measurement of Outcomes
        Each student's questionnaires were coded to identify their gender, condition (online or in-class), school (UH or KSU) and topic (Companies in Schools or WTO).  Anonymous photocopies of the open-ended responses were then given to two independent reviewers (one PR professor at UH, and one at KSU) to be evaluated based on the educational objectives stated above. Reviewers were blind to condition and school, as well as any other potentially identifying information.

Results
RQ1: Will students who participate in an online discussion forum with students from distant geographical areas demonstrate greater success in achieving course objectives than their counterparts who participate in an in-class forum?

No significant differences were found between online and in-class conditions in terms of learning outcomes as rated by the independent judges. Since the identification of key publics and key issues related to controversial issues is a rather subjective task, each judge's ratings were analyzed independently.[1] Although the two judges may have differed in opinion about key publics and key issues for each case, both sets of ratings, as well as the average ratings of the two judges, indicated no significant differences between experimental conditions. Possible ratings for each item ranged from 1 = strongly disagree to 5 = strongly agree. Means are reported in Tables 1-3.

Table 1. Mean Scores for Learning Outcome Items as Rated by Judge 1
In-Class
ALNs
N
Min
Max
Mean
SD
N
Min
Max
Mean
SD
Identifies a broad range of issues
33
2
5
3.76
.97
30
2
5
3.70
.84
Identifies relevant issues
33
2
5
3.52
.87
30
2
5
3.47
1.01
Articulates central issue well
33
2
5
3.58
.83
30
2
5
3.47
.90
Identifies a broad range of publics
33
2
4
3.82
.58
30
4
5
4.07
.25
Identifies relevant publics
33
2
4
3.55
.71
30
2
5
3.80
.66
Offers compelling support
33
2
5
3.91
.68
30
2
5
3.90
.61

 Table 2. Mean Scores for Learning Outcome Items as Rated by Judge 2
In-Class Forums
ALNs
N
Min
Max
Mean
SD
N
Min
Max
Mean
SD
Identifies a broad range of issues
38
1
4
3.00
.93
31
1
5
2.97
1.02
Identifies relevant issues
38
2
5
3.66
.67
31
2
5
3.52
.81
Articulates central issue well
38
2
5
3.42
.83
31
2
4
3.16
.69
Identifies a broad range of publics
38
1
5
2.97
.88
31
1
4
2.84
.86
Identifies relevant publics
38
2
4
3.74
.55
31
3
4
3.77
.43
Offers compelling support
38
1
4
2.66
.71
31
1
4
2.74
.73

Table 3. Mean Scores for Learning Outcome Items Averaging Among Both Judges
In-Class
ALNs
N
Min
Max
Mean
SD
N
Min
Max
Mean
SD
Identifies a broad range of issues
33
1.5
4.0
3.40
.80
30
1.5
4.50
3.32
.78
Identifies relevant issues
33
2.0
4.0
3.60
.67
30
2.0
5.00
3.48
.73
Articulates central issue well
33
2.0
4.5
3.50
.61
30
2.0
4.50
3.30
.62
Identifies a broad range of publics
33
2.0
4.0
3.38
.47
30
2.5
4.50
3.43
.47
Identifies relevant publics
33
2.0
4.0
3.64
.58
30
2.5
4.50
3.78
.49
Offers compelling support
33
1.5
4.5
3.27
.56
30
2.0
4.00
3.30
.55

        To ensure that potential differences were not being masked by differences in gender, university, or topic of discussion, mean scores for learning outcomes were compared based on each of these variables (e.g., outcome scores for males v. females). No significant differences were found. Furthermore, chi-square tests indicated no significant interactions between each of these variables and experimental conditions in terms of learning outcomes.

RQ2: Will student satisfaction with the forums correlate with achievement of learning outcomes?

        A "shopping list" of 21 items from previous studies was created to measure student satisfaction with discussion forums. Each item was then adapted for the present study and presented to students in a Likert-type format ranging from 1 = strongly disagree to 5 = strongly agree. An exploratory factor analysis of these 21 items was calculated, yielding a simpler five-item factor including the following:
1.      This discussion forum was a good learning tool for me.
2.      The material discussed was valuable to me.
3.      I enjoyed the discussion.
4.      The discussion was interesting.
5.      The discussion was useful.

Based on the logical fit among these five items, responses to each item were combined to create an index, which proved a reliable indicator of student satisfaction (( = .90).
        However, this index of student satisfaction with the discussion forums did not correlate significantly with any of the measures of learning outcomes.

RQ3: What are the qualitative differences between online discussions of PR cases among two distant university PR classes and in-class discussions within each class?

Linearity. Online discussion forums were clearly more linear than in-class discussions in this study. Three questions for each topic were posted online and students were asked to respond at least once to each question. The typical pattern of participation in the online forum varied almost none ( a student would log on to the ALN, respond to each of the three questions and log off.  For the first few respondents, this meant they were not exposed to the majority of others' opinions.
Later respondents normally acknowledged the contributions of their peers who participated before them, added an agreeable comment and then logged off. The following thread excerpts from the online discussion of the WTO scenario illustrate this pattern. Student numbers are inserted here in place of student names for anonymity and to illustrate the order of conversation. As with most the threads in this study, no student participated more than once.

Teacher - What are some of the PR issues for your Representative (Congressperson) and why?

Student 2 - It seems like some of the issues that need to be addressed here are_

Student 3 - I have to agree with [Student 2]_

Student 5 - I agree with most from above_

Student 6 - I also agree with most of the answers above_

Student 7 - I tend to agree with [Student 2]_

Student 8 - I am going to have to agree with [Student 3] and [Student 2]_

Student 10 - I agree with [Student 8] (who also agrees with [Student 3] and [Student 2])_

Student 11 - I tend to agree with [Student 2] and co. _

Student 12 - Everything that has been said above is what I would address as some of the public relations issues that I would inform to the representative. A lot that was said by [Student 4], [Student2] and others has an underlying meaning that all goes back to the bottom line of communication_

Student 13 - I agree with most of what everyone said_

Student 15 - Sorry I'm late on this, but as I was reading everyone else's, I pretty much agree with you all_

Student 16 - Like many people stated above, communication is the key_

         In debriefing sessions, students were quick to note the lack of dialog in the online forums compared to the in-class forums. Regarding the online forum, one student (Student 2 above) seemed to speak for her peers when she said, "we didn't get to hear others' opinions." Another suggested that the online forum was "not as interactive."
Openness to Divergent Ideas.    The in-class discussions seemed to encourage the expression of a greater variety of original ideas and experiences. Whereas "I agree with_" was the most common introductory phrase in online discussions, more opposing views were aired in class at both UH and KSU. As one UH student mentioned in the debriefing session, the online forum seemed "more formal," which led to "a fear of being wrong."  Students at KSU made similar comments: "I was afraid to say anything different from what was being presented because I didn't know who had access to this forum." This reluctance to risk being too frank also may have been related to uncertainty about how messages online would be interpreted without the communication cues available in face-to-face conversation. "The in-class discussion was better because you could tell if people understood you or not."
In class, students commonly talked about their personal experiences and opinions. For example, UH students discussing the issues surrounding the Companies in Schools case compared their personal experiences with vending machines in schools on Oahu, the Big Island and Maui, as well as mainland communities. Likewise, students at KSU discussed personal opinions about how companies gave to the more affluent school in their area, rather than schools in rural Kansas or inner city schools because the more affluent communities would be more likely to purchase whatever the company was donating.
These in-class discussions created some debate, whereas the online discussions had no disagreement. As one KSU student succinctly stated, there was "a lack of originality" in the online discussions.
Time to Think. Although not all students wholeheartedly agreed, many students reported that the online forum gave them more time to develop their own responses than the in-class discussions. For example, when asked if there were any advantages to the online discussion format, a UH student responded that "it gives you time to think," and KSU students mentioned that the online forum made them "more organized" in their input.

Limitations
Measurement and Design. Statistically speaking, this experiment was limited by the small number of participants and the reliability of dependent measures.
Reporting "no differences" begs the question of whether actual differences would have become evident with more students participating. But even a superficial look at the data (i.e., Tables 1-3) suggests that increasing the number of students who participated would have made little difference in conclusions regarding learning outcomes. Increasing class size was neither a feasible nor desirable option in this particular experiment since both classes involved were upper-level, writing-intensive PR courses.
In other words, increasing the number of participants would require a different experimental design altogether ( perhaps one involving a few lower level PR classes with more students or several upper-level PR classes with several smaller discussion forums. In any case, the number of students participating in each forum should be carefully considered in future research, as small-group discussions will certainly differ qualitatively from large-group discussions in either format.
        The subjective measures of learning outcomes introduce another limitation. Who, exactly, are the "key" publics for the cases discussed? What, precisely, is the "central" issue for each case?  And what qualifies as "compelling" support for a position? If these questions had clear, objective answers, then the cases would not have encouraged the type of dialog that teachers often seek in attempts to achieve the higher-order teaching objectives listed above. Although different dependent measures can be designed that will yield higher inter-coder reliability statistics (e.g., How many publics does the student identify?), this increase in statistical reliability will come at the expense of qualitative interpretations of results. Although future research in this area should focus more sharply on either qualitative or quantitative issues in design, balancing the demands of both is central to classroom teaching and evaluation.
        Pedagogical Limitations. Indeed, the pursuit of experimental control in this study affected the way the courses were taught during the three weeks of the study. In order to minimize differences due to teaching style and to control for biases related to discussions guided by the experimenters/teachers, teacher participation in both the online and in-class discussions was limited to that of a passive moderator. In debriefing sessions, students noted this lack of teacher guidance. Students especially missed teacher guidance in the WTO discussions, in which students reported feeling less informed. Future studies, quantitative or qualitative, should allow for more natural teacher participation.

Discussion
        This study was experimental in two senses. In terms of method, students were randomly assigned to one of two conditions ( ALN or in-class discussion forum ( to discuss PR topics. Then learning outcomes for each condition were measured and compared. In terms of pedagogy, this study was experimental in the sense that the teachers and students were trying something entirely new. This activity, requiring asynchronous communication about complex PR issues among peers in entirely different geographical and cultural contexts, was an "experiment" in PR education for everyone involved.
        Although a body of literature exists suggesting that ALNs foster learning via thoughtful exchange of diverse viewpoints, this study's quantitative data indicates that ALNs have no inherent advantage over in-class forums. But learning from trial and error is what makes experimentation, in both senses of the word, worthwhile. Several lessons were learned from this experiment.
        The "Guide on the Side" must be active. Whether online or in class, the teacher's role is just as important in discussion forums as it is in lecture-style, sage-on-the-stage formats. Conducting this study yielded no evidence to suggest that successful online discussion forums require any less planning and participation than in-class activities. Learning outcomes in both forums would have likely benefited from more active teacher participation, but active teacher participation in the ALNs would require time and effort above and beyond the teachers' workload for traditional classroom planning. This investment of teacher costs is what makes it so important to realistically gauge benefits in terms of learning outcomes.
        It takes time. Discussing important issues in public relations means being frank and considering different viewpoints. The ALN exercises for this study were originally designed to use technology to bridge physical distances and increase the breadth of perspectives shared by students. At best, the results showed no advantage for the ALNs. Students reported, and teachers' observations supported the conclusion, that the ALNs were more intimidating than in-class discussions. Although students initially introduced themselves with brief biographical information, there was no evidence of "breaking the ice."
        Getting students to engage in more productive dialog about sensitive PR issues requires that they feel comfortable in the forum. In traditional classrooms, many students are reluctant to go out on a limb in discussions during the first few class periods. Then the environment often warms up in later weeks as students get to know each other.  If the same holds true for online discussion forums, students in this study simply may have needed more time.
        Consider student motivations. Teachers in this study asked students in online conditions to log on the ALN and contribute "at least once" to each posted question. The overwhelming majority of students then made exactly one contribution to each of the forum questions. Getting students to log on and make their presence known in an ALN is the easy part. The challenge lies in moving them beyond purely extrinsic-driven behavior and getting them intrinsically interested in the learning exercise. For ALNs this will require new ideas for activity design. Perhaps using a grading rubric focusing on both quality and quantity of participation would be a more effective way to get students engaged.

Conclusions
Past research has shown that online discussion forums can be a success if student satisfaction with the forums is the teacher's main goal. Of course, student satisfaction is an extremely important ingredient of well-rounded education. Student satisfaction also provides an important measure by which most college faculty are evaluated when it comes to teaching performance. But beyond student satisfaction, and beyond consumer models of education, is the question of what students actually learn. The content of public relations curriculum is such that diverse perspectives voiced in class forums can be a valuable resource for understanding why and how publics interact with organizations in the context of a wide range of issues.
In this experiment, using ALN technology to help achieve PR-related teaching objectives resulted in no apparent advantage over traditional classroom discussions. Furthermore, student satisfaction with the forums did not correlate with learning outcomes.
This study suggests several ways to improve the teaching tactics it examined. By playing an active role in discussions, allowing students time to get comfortable with discussion forums and carefully considering student motivations when designing activities and grading schemes, PR teachers in future classes may have more success with online forums.
And just as content must not be overlooked, the value of getting PR students comfortable with new media should also not be overlooked. Online discussion forums ( asynchronous and real-time ( should be considered part of the content of PR curriculum as well as tools for teaching PR. In addition to teaching tips, this paper offers a foundation for future research on the costs and benefits of integrating technology into PR classes.


REFERENCES

Blake, K.R.  (2000).  Using the World Wide Web to teach news writing online.
Journalism & Mass Communication Educator, 55 (1), 4-13.

Bloom, B.S. (1956). Taxonomy of educational objectives: The classification of
educational goals: Handbook I, cognitive domain. New York: D. McKay Co., Inc.

Brace-Govan, J., & Clulow, V.  (2000).  Varying expectations of online students and the
implications for teachers: Findings from a journal study.  Distance Education, 21 (1), 118-135.

Commission on Public Relations Education.  (1999, October).  A port of entry: Public
relations education for the 21st Century.

Curtin, P.A., & Witherspoon, E.M.  (1999).  Computer skills integration in public
relations curricula.  Journalism & Mass Communication Educator, 54 (1), 23-34.

Gower, K.K., & Cho, J. (2000, August).  The Internet and public relations curricula:
Fitting a "jet engine to a horse-drawn carriage."  A paper presented to the public relations teaching session at the Association for Education in Journalism and Mass Communication annual conference, Phoenix, AZ.

Geimann, S.  (2000).  AEJMC task force on the professions in the new millennium.  In
Kopenhaver, L.L. (Ed.) AEJMC and ASJMC: Remembering our past_ Anticipating our future: Journalism and mass communication education moving into the new millennium.  Columbia, SC: AEJMC.

Gunawardena, C.N., & Duphorne, P.L. (2000).  Predictors of learner satisfaction in an
academic computer conference.  Distance Learning, 21 (1), 101-117.

Hiltz, S.R., Coppola, N., Rotter, N., Turoff, M., & Benbunan-Fich, R.  (2000).
Measuring the importance of collaborative learning for the effectiveness of ALN: A multi-measure, multi-method approach. Journal of Asynchronous Learning Networks, 4 (2).  Available online at http://www.aln.org/alnweb/journal/jaln-vol4issue2.htm

Hines, B.B. (2000).  Task force on teaching and learning in the new millennium:
Subcommittee on inclusivity. In Kopenhaver, L.L. (Ed.) AEJMC and ASJMC: Remembering our past_ Anticipating our future: Journalism and mass communication education moving into the new millennium.  Columbia, SC: AEJMC.

Lauzon, A.C. (2000).  Distance education and diversity: Are they compatible?  The
American Journal of Distance Education, 14 (2), 61-70.

McAlpine, I.  (2000).  Collaborative learning online.  Distance Education, 21 (1), 66-80.

Pavlik, J.V., & Powell, A.C. III. (2000).  New media and journalism and mass
communication education.  In Kopenhaver, L.L. (Ed.) AEJMC and ASJMC: Remembering our past_ Anticipating our future: Journalism and mass communication education moving into the new millennium.  Columbia, SC: AEJMC.

Rossman, M.H. (1999).  Successful online teaching using an asynchronous learner
discussion forum. Journal of Asynchronous Learning Networks, 3 (2).  Available online at http://www.aln.org/alnweb/journal/jaln-vol3issue2.htm

Thompson, M.M., & McGrath., J.W. (1999).  Using ALNs to support a complete
educational experience. Journal of Asynchronous Learning Networks, 3 (2).  Available online at http://www.aln.org/alnweb/journal/jaln-vol3issue2.htm

Wheeler, B.C., Valacich, J.S., Alavi, M., & Vogel, D. (1995).  A framework for
technology-mediated inter-institutional telelearning relationships.  Journal of Computer Mediated Communication, 1 (1).  Available online at http://www.ascusc.org/jcmc/vol1/issue1/index.html

[1]  Correlations between judges were marginal at best. The item on which the judges agreed most was, "The student identifies a broad range of issues" for the companies in schools case (r = .78). Averaging the two judges' ratings to measure learning outcomes also yielded no significant differences between experimental conditions.