College Composition Weekly: Summaries of research for college writing professionals

Read, Comment On, and Share News of the Latest from the Rhetoric and Composition Journals


Leave a comment

Patchan and Shunn. Effects of Author and Reviewer Ability in Peer Feedback. JoWR 2016. Posted 11/25/2016.

Patchan, Melissa M., and Christian D. Shunn. “Understanding the Effects of Receiving Peer Feedback for Text Revision: Relations between Author and Reviewer Ability.” Journal of Writing Research 8.2 (2016): 227-65. Web. 18 Nov. 2016. doi: 10.17239/jowr-2016.08.02.03

Melissa M. Patchan and Christian D. Shunn describe a study of the relationship between the abilities of writers and peer reviewers in peer assessment. The study asks how the relative ability of writers and reviewers influences the effectiveness of peer review as a learning process.

The authors note that in many content courses, the time required to provide meaningful feedback encourages many instructors to turn to peer assessment (228). They cite studies suggesting that in such cases, peer response can be more effective than teacher response because, for example, students may actually receive more feedback, the feedback may be couched in more accessible terms, and students may benefit from seeing models and new strategies (228-29). Still, studies find, teachers and students both question the efficacy of peer assessment, with students stating that the quality of review depends largely on the abilities of the reviewer (229).

Patchan and Shunn distinguish between the kind of peer review characteristic of writing classrooms, which they describe as “pair or group-based face-to-face conversations” emphasizing “qualitative feedback” and the type more often practiced in large content classes, which they see as more like “professional journal reviewing” that is “asynchronous, and written-based” (228). Their study addresses the latter format and is part of a larger study examining peer feedback in a widely required psychology class at a “large, public research university in the southeast” (234).

A random selection of 189 students wrote initial drafts in response to an assignment assessing media handling of a psychological study using criteria from the course textbook (236, 238). Students then received four drafts to review and were given a week to revise their own drafts in response to feedback. Participants used the “web-based peer assessment functions of turnitin.com” (237).

The researchers rated participants as high-ability writers using SAT scores and grades in their two first-year writing courses (236). Graduate rhetoric students also rated the first drafts. The protocol then included a “median split” to designate writers in binary fashion as either high- or low-ability. “High” authors were categorized as “high” reviewers. Patchan and Shunn note that there was a wide range in writer abilities but argue that, even though the “design decreases the power of this study,” such determinations were needed because of the large sample size, which in turn made the detection of “important patterns” likely (236-37). They feel that “a lower powered study was a reasonable tradeoff for higher external validity (i.e., how reviewer ability would typically be detected)” (237).

The authors describe their coding process in detail. In addition to coding initial drafts for quality, coders examined each reviewer’s feedback for its attention to higher-order problems and lower-order corrections (239-40). Coders also tabulated which comments resulted in revision as well as the “quality of the revision” (241). This coding was intended to “determine how the amount and type of comments varied as a function of author ability and reviewer ability” (239). A goal of the study was to determine what kinds of feedback triggered the most effective responses in “low” authors (240).

The study was based on a cognitive model of writing derived from the updated work of Linda Flower and John R. Hayes, in which three aspects of writing/revision follow a writer’s review of a text: problem detection, problem diagnosis, and strategy selection for solving the diagnosed problems (230-31). In general, “high” authors were expected to produce drafts with fewer initial problems and to have stronger reading skills that allowed them to detect and diagnose more problems in others’ drafts, especially “high-level” problems having to do with global issues as opposed to issues of surface correctness (230). High ability authors/reviewers were also assumed to have a wider repertoire of solution strategies to suggest for peers and to apply to their own revisions (233). All participants received a rubric intended to guide their feedback toward higher-order issues (239).

Some of the researchers’ expectations were confirmed, but others were only partially supported or not supported (251). Writers whose test scores and grades categorized them as “high” authors did produce better initial drafts, but only by a slight margin. The researchers posit that factors other than ability may affect draft quality, such as interest or time constraints (243). “High” and “low” authors received the same number of comments despite differences in the quality of the drafts (245), but “high” authors made more higher-order comments even though they didn’t provide more solutions (246). “High” reviewers indicated more higher-order issues to “low” authors than to “high,” while “low” reviewers suggested the same number of higher-order changes to both “high” and “low” authors (246).

Patchan and Shunn considered the “implementation rate,” or number of comments on which students chose to act, and “revision quality” (246). They analyzed only comments that were specific enough to indicate action. In contrast to findings in previous studies, the expectation that better writers would make more and better revisions was not supported. Overall, writers acted on only 32% of the comments received and only a quarter of the comments resulted in improved drafts (248). Author ability did not factor into these results. Moreover, the ability of the reviewer had no effect on how many revisions were made or how effective they were (248).

It was expected that low-ability authors would implement more suggestions from higher-ability reviewers, but in fact, “low authors implemented more high-level criticism comments . . . from low reviewers than from high reviewers” (249). The quality of the revisions also improved for low-ability writers when the comments came from low-ability reviewers. The researchers conclude that “low authors benefit the most from feedback provided by low reviewers” (249).

Students acted on 41% of the low-level criticisms, but these changes seldom resulted in better papers (249).

The authors posit that rates of commenting and implementation may both be impacted by limits or “thresholds” on how much feedback a given reviewer is willing to provide and how many comments a writer is able or willing to act on (252, 253). They suggest that low-ability reviewers may explain problems in language that is more accessible to writers with less ability. Patchan and Shunn suggest that feedback may be most effective when it occurs within the student’s zone of proximal development, so that weaker writers may be helped most by peers just beyond them in ability rather than by peers with much more sophisticated skills (253).

In the authors’ view, that “neither author ability nor reviewer ability per se directly affected the amount and quality of revisions” (253) suggests that the focus in designing effective peer review processes should shift from how to group students to improving students’ ability to respond to comments (254). They recommend further research using more “direct” measures of writing and reviewing ability (254). A major conclusion from this study is that “[h]igher-ability students will likely revise their texts successfully regardless of who [they are] partnered with, but the lower-ability students may need feedback at their own level” (255).


1 Comment

Moxley and Eubanks. Comparing Peer Review and Instructor Ratings. WPA, Spring 2016. Posted 08/13/2016.

Moxley, Joseph M., and David Eubanks. “On Keeping Score: Instructors’ vs. Students’ Rubric Ratings of 46,689 Essays.” Journal of the Council of Writing Program Administrators 39.2 (2016): 53-80. Print.

Joseph M. Moxley and David Eubanks report on a study of their peer-review process in their two-course first-year-writing sequence. The study, involving 16,312 instructor evaluations and 30,377 student reviews of “intermediate drafts,” compared instructor responses to student rankings on a “numeric version” of a “community rubric” using a software package, My Reviewers, that allowed for discursive comments but also, in the numeric version, required rubric traits to be assessed on a five-point scale (59-61).

Exploring the literature on peer review, Moxley and Eubanks note that most such studies are hindered by small sample sizes (54). They note a dearth of “quantitative, replicable, aggregated data-driven (RAD) research” (53), finding only five such studies that examine more than 200 students (56-57), with most empirical work on peer review occurring outside of the writing-studies community (55-56).

Questions investigated in this large-scale empirical study involved determining whether peer review was a “worthwhile” practice for writing instruction (53). More specific questions addressed whether or not student rankings correlated with those of instructors, whether these correlations improved over time, and whether the research would suggest productive changes to the process currently in place (55).

The study took place at a large research university where the composition faculty, consisting primarily of graduate students, practiced a range of options in their use of the My Reviewers program. For example, although all commented on intermediate drafts, some graded the peer reviews, some discussed peer reviews in class despite the anonymity of the online process, and some included training in the peer-review process in their curriculum, while others did not.

Similarly, the My Reviewers package offered options including comments, endnotes, and links to a bank of outside sources, exercises, and videos; some instructors and students used these resources while others did not (59). Although the writing program administration does not impose specific practices, the program provides multiple resources as well as a required practicum and annual orientation to assist instructors in designing their use of peer review (58-59).

The rubric studied covered five categories: Focus, Evidence, Organization, Style, and Format. Focus, Organization, and Style were broken down into the subcategories of Basics—”language conventions”—and Critical Thinking—”global rhetorical concerns.” The Evidence category also included the subcategory Critical Thinking, while Format encompassed Basics (59). For the first year and a half of the three-year study, instructors could opt for the “discuss” version of the rubric, though the numeric version tended to be preferred (61).

The authors note that students and instructors provided many comments and other “lexical” items, but that their study did not address these components. In addition, the study did not compare students based on demographic features, and, due to its “observational” nature, did not posit causal relationships (61).

A major finding was that. while there was some “low to modest” correlation between the two sets of scores (64), students generally scored the essays more positively than instructors; this difference was statistically significant when the researchers looked at individual traits (61, 67). Differences between the two sets of scores were especially evident on the first project in the first course; correlation did increase over time. The researchers propose that students learned “to better conform to rating norms” after their first peer-review experience (64).

The authors discovered that peer reviewers were easily able to distinguish between very high-scoring papers and very weak ones, but struggled to make distinctions between papers in the B/C range. Moxley and Eubanks suggest that the ability to distinguish levels of performance is a marker for “metacognitive skill” and note that struggles in making such distinctions for higher-quality papers may be commensurate with the students’ overall developmental levels (66).

These results lead the authors to consider whether “using the rubric as a teaching tool” and focusing on specific sections of the rubric might help students more closely conform to the ratings of instructors. They express concern that the inability of weaker students to distinguish between higher scoring papers might “do more harm than good” when they attempt to assess more proficient work (66).

Analysis of scores for specific rubric traits indicated to the authors that students’ ratings differed more from those of instructors on complex traits (67). Closer examination of the large sample also revealed that students whose teachers gave their own work high scores produced scores that more closely correlated with the instructors’ scores. These students also demonstrated more variance than did weaker students in the scores they assigned (68).

Examination of the correlations led to the observation that all of the scores for both groups were positively correlated with each other: papers with higher scores on one trait, for example, had higher scores across all traits (69). Thus, the traits were not being assessed independently (69-70). The authors propose that reviewers “are influenced by a holistic or average sense of the quality of the work and assign the eight individual ratings informed by that impression” (70).

If so, the authors suggest, isolating individual traits may not necessarily provide more information than a single holistic score. They posit that holistic scoring might not only facilitate assessment of inter-rater reliability but also free raters to address a wider range of features than are usually included in a rubric (70).

Moxley and Eubanks conclude that the study produced “mixed results” on the efficacy of their peer-review process (71). Students’ improvement with practice and the correlation between instructor scores and those of stronger students suggested that the process had some benefit, especially for stronger students. Students’ difficulty with the B/C distinction and the low variance in weaker students’ scoring raised concerns (71). The authors argue, however, that there is no indication that weaker students do not benefit from the process (72).

The authors detail changes to their rubric resulting from their findings, such as creating separate rubrics for each project and allowing instructors to “customize” their instruments (73). They plan to examine the comments and other discursive components in their large sample, and urge that future research create a “richer picture of peer review processes” by considering not only comments but also the effects of demographics across many settings, including in fields other than English (73, 75). They acknowledge the degree to which assigning scores to student writing “reifies grading” and opens the door to many other criticisms, but contend that because “society keeps score,” the optimal response is to continue to improve peer-review so that it benefits the widest range of students (73-74).


1 Comment

T. Bourelle et al. Using Instructional Assistants in Online Classes. C&C, Sept. 2015. Posted 10/13/2015.

Bourelle, Tiffany, Andrew Bourelle, and Sherry Rankins-Robertson. “Teaching with Instructional Assistants: Enhancing Student Learning in Online Classes.” Computers and Composition 37 (2015): 90-103. Web. 6 Oct. 2015.

Tiffany Bourelle, Andrew Bourelle, and Sherry Rankins-Robertson discuss the “Writers’ Studio,” a pilot program at Arizona State University that utilized upper-level English and education majors as “instructional assistants” (IAs) in online first-year writing classes. The program was initiated in response to a request from the provost to cut budgets without affecting student learning or increasing faculty workload (90).

A solution was an “increased student-to-teacher ratio” (90). To ensure that the creation of larger sections met the goal of maintaining teacher workloads and respected the guiding principles put forward by the Conference on College Composition and Communication Committee for Best Practices in Online Writing Instruction in its March 2013 Position Statement, the team of faculty charged with developing the cost-saving measures supplemented “existing pedagogical strategies” with several innovations (91).

The writers note that one available cost-saving step was to avoid staffing underenrolled sections. To meet this goal, the team created “mega-sections” in which one teacher was assigned per each 96 students, the equivalent of a full-time load. Once the enrollment reached 96, a second teacher was assigned to the section, and the two teachers team-taught. T. Bourelle et al. give the example of a section of the second semester of the first-year sequence that enrolled at 120 students and was taught by two instructors. These 120 students were assigned to 15-student subsections (91).

T. Bourelle et al. note several reasons why the new structure potentially increased faculty workload. They cite research by David Reinheimer to the effect that teaching writing online is inherently more time-intensive than instructors may expect (91). Second, the planned curriculum included more drafts of each paper, requiring more feedback. In addition, the course design required multimodal projects. Finally, students also composed “metacognitive reflections” to gauge their own learning on each project (92).

These factors prompted the inclusion of the IAs. One IA was assigned to each 15-student group. These upper-level students contributed to the feedback process. First-year students wrote four drafts of each paper: a rough draft that received peer feedback, a revised draft that received comments from the IAs, an “editing” draft students could complete using the writing center or online resources, and finally a submission to the instructor, who would respond by either accepting the draft for a portfolio or returning it with directions to “revise and resubmit” (92). Assigning portfolio grades fell to the instructor. The authors contend that “in online classes where students write multiple drafts for each project, instructor feedback on every draft is simply not possible with the number of students assigned to any teacher, no matter how she manages her time” (93).

T. Bourelle et al. provide extensive discussion of the ways the IAs prepared for their roles in the Writers’ Studio. A first component was an eight-hour orientation in which the assistants were introduced to important teaching practices and concepts, in particular the process of providing feedback. Various interactive exercises and discussions allowed the IAs to develop their abilities to respond to the multimodal projects required by the Studio, such as blogs, websites, or “sound portraits” (94). The instruction for IAs also covered the distinction between “directive” and “facilitative” feedback, with the latter designed to encourage “an author to make decisions and [give] the writer freedom to make choices” (94).

Continuing support throughout the semester included a “portfolio workshop” that enabled the IAs to guide students in their production of the culminating eportfolio requirement, which required methods of assessment unique to electronic texts (95). Bi-weekly meetings with the instructors of the larger sections to which their cohorts belonged also provided the IAs with the support needed to manage their own coursework while facilitating first-year students’ writing (95).

In addition, IAs enrolled in an online internship that functioned as a practicum comparable to practica taken by graduate teaching assistants at many institutions (95-97). The practicum for the Writers’ Studio internship reinforced work on providing facilitative feedback but especially incorporated the theory and practice of online instruction (96). T. Bourelle et al. argue that the effectiveness of the practicum experience was enhanced by the degree to which it “mirror[ed]” much of what the undergraduate students were experiencing in their first-year classes: “[B]oth groups of beginners are working within initially uncomfortable but ultimately developmentally positive levels of ambiguity, multiplicity, and open-endedness” (Barb Blakely Duffelmeyer, qtd. in T. Bourelle et al. 96). Still quoting Duffelmeyer, the authors contend that adding computers “both enriched and problematized” the pedagogical experience of the coursework for both groups (96), imposing the need for special attention to online environments.

Internship assignments also gave the IAs a sense of what their own students would be experiencing by requiring an eportfolio featuring what they considered their best examples of feedback to student writing as well as reflective papers documenting their learning (98).

The IAs in the practicum critiqued the first-year curriculum, for example suggesting stronger scaffolding for peer review and better timing of assignments. They wrote various instructional materials to support the first-year course activities (97).

Their contributions to the first-year course included “[f]aciliting discussion groups” (98) and “[d]eveloping supportive relationships with first-year writers” (100), but especially “[r]esponding to revised drafts” (99). T. Bourelle et al. note that the IAs’ feedback differed from that of peer reviewers in that the IAs had acquired background in composition and rhetorical theory; unlike writing-center tutors, the IAs were more versed in the philosophy and expectations embedded in the course itself (99). IAs were particularly helpful to students who had misread the assignments, and they were able to identify and mentor students who were falling behind (98, 99).

The authors respond to the critique that the IAs represented uncompensated labor by arguing that the Writers’ Studio offered a pedagogically valuable opportunity that would serve the students well if they pursued graduate or professional careers as educators, emphasizing the importance of designing such programs to benefit the students as well as the university (101). They present student and faculty testimony on the effectiveness of the IAs as a means of “supplement[ing] teacher interaction” rather than replacing it (102). While they characterize the “monetary benefit” to the university as “small” (101), they consider the project “successful” and urge other “teacher-scholars to build on what we have tried to do” (102).