College Composition Weekly: Summaries of research for college writing professionals

Read, Comment On, and Share News of the Latest from the Rhetoric and Composition Journals

Category Archives: Error


by 1 Comment

Moore & MacArthur. Automated Essay Evaluation. JoWR, June 2016. Posted 10/04/2016.

Moore, Noreen S., and Charles A. MacArthur. “Student Use of Automated Essay Evaluation Technology During Revision.” Journal of Writing Research 8.1 (2016): 149-75. Web. 23 Sept. 2016.

Noreen S. Moore and Charles A. MacArthur report on a study of 7th- and 8th-graders’ use of Automated Essay Evaluation technology (AEE) and its effects on their writing.

Moore and MacArthur define AEE as “the process of evaluating and scoring written prose via computer programs” (M. D. Shermis and J. Burstein, qtd. in Moore and MacArthur 150). The current study was part of a larger investigation of the use of AEE in K-12 classrooms (150, 153-54). Moore and MacArthur focus on students’ revision practices (154).

The authors argue that such studies are necessary because “AEE has the potential to offer more feedback and revision opportunities for students than may otherwise be available” (150). Teacher feedback, they posit, may not be “immediate” and may be “ineffective” and “inconsistent” as well as “time consuming,” while the alternative of peer feedback “requires proper training” (151). The authors also posit that AEE will increasingly become part of the writing education landscape and that teachers will benefit from “participat[ing]” in explorations of its effects (150). They argue that AEE should “complement” rather than replace teacher feedback and scoring (151).

Moore and MacArthur review extant research on two kinds of AEE, one that uses “Latent Semantic Analysis” (LSA) and one that has been “developed through model training” (152). Studies of an LSA program owned by Pearson and designed to evaluate summaries compared the program with “word-processing feedback” and showed enhanced improvement across many traits, including “quality, organization, content, use of detail, and style” as well as time spent on revision (152). Other studies also showed improvement. Moore and MacArthur note that some of these studies relied on scores from the program itself as indices of improvement and did not demonstrate any transfer of skills to contexts outside of the program (153).

Moore and MacArthur contend that their study differs from previous research in that it does not rely on “data collected by the system” but rather uses “real time” information from think-aloud protocols and semi-structured interviews to investigate students’ use of the technology. Moreover, their study reveals the kinds of revision students actually do (153). They ask:

  • How do students use AEE feedback to make revisions?
  • Are students motivated to make revisions while using AEE technology?
  • How well do students understand the feedback from AEE, both the substantive feedback and the conventions feedback? (154)

The researchers studied six students selected to be representative of a 12-student 7th- and 8th-grade “literacy class” at a private northeastern school whose students exhibited traits “that may interfere with school success” (154). The students were in their second year of AEE use and the teacher in the third year of use. Students “supplement[ed]” their literacy work with in-class work using the “web-based MY Access!” program (154).

Moore and MacArthur report that “intellimetric” scoring used by MY Access! correlates highly with scoring by human raters (155). The software is intended to analyze “focus/coherence, organization, elaboration/development, sentence structure, and mechanics/conventions” (155).

MY Access provides feedback through MY Tutor, which responds to “non-surface” issues, and MY Editor, which addresses spelling, punctuation, and other conventions. MY Tutor provides a “one sentence revision goal”; “strategies for achieving the goal”; and “a before and after example of a student revising based on the revision goal and strategy” (156). The authors further note that “[a]lthough the MY Tutor feedback is different for each score point and genre, the same feedback is given for the same score in the same genre” (156). MY Editor responds to specific errors in each text individually.

Each student submitted a first and revised draft of a narrative and an argumentative paper, for a total of 24 drafts (156). The researchers analyzed only revisions made during the think-aloud; any revision work prior to the initial submission did not count as data (157).

Moore and MacArthur found that students used MY Tutor for non-surface feedback only when their submitted essays earned low scores (158). Two of the three students who used the feature appeared to understand the feedback and used it successfully (163). The authors report that for the students who used it successfully, MY Tutor feedback inspired a larger range of changes and more effective changes in the papers than feedback from the teacher or from self-evaluation (159). These students’ changes addressed “audience engagement, focusing, adding argumentative elements, and transitioning” (159), whereas teacher feedback primarily addressed increasing detail.

One student who scored high made substantive changes rated as “minor successes” but did not use the MY Tutor tool. This student used MY Editor and appeared to misunderstand the feedback, concentrating on changes that eliminated the “error flag” (166).

Moore and MacArthur note that all students made non-surface revisions (160), and 71% of these efforts were suggested by AEE (161). However, 54.3% of the total changes did not succeed, and MY Editor suggested 68% of these (161). The authors report that the students lacked the “technical vocabulary” to make full use of the suggestions (165); moreover, they state that “[i]n many of the instances when students disagreed with MY Editor or were confused by the feedback, the feedback seemed to be incorrect” (166). The authors report other research that corroborates their concern that grammar checkers in general may often be incorrect (166).

As limitations, the researchers point to the small sample, which, however, allowed access to “rich data” and “detailed description” of actual use (167). They note also that other AEE program might yield different results. Lack of data on revisions students made before submitting their drafts also may have affected the results (167). The authors supply appendices detailing their research methods.

Moore and MacArthur propose that because the AEE scores prompt revision, such programs can effectively augment writing instruction, but recommend that scores need to track student development so that as students score near the maximum at a given level, new criteria and scores encourage more advanced work (167-68). Teachers should model the use of the program and provide vocabulary so students better understand the feedback. Moore and MacArthur argue that effective use of such programs can help students understand criteria for writing assessment and refine their own self-evaluation processes (168).

Research recommendations include asking whether scores from AEE continue to encourage revision and investigating how AEE programs differ in procedures and effectiveness. The study did not examine teachers’ approaches to the program. Moore and MacArthur urge that stakeholders, including “the people developing the technology and the teachers, coaches, and leaders using the technology . . . collaborate” so that AEE “aligns with classroom instruction” (168-69).


by 1 Comment

Arnold, Lisa. International Response to Rhet/Comp Theory. CS, Spring 2016. Posted 06/14/2016.

Arnold, Lisa R. “‘This is a Field that’s Open, not Closed’: Multilingual and International Writing Faculty Respond to Composition Theory.” Composition Studies 44.1 (2016): 72-88. Web. o2 June 2016.

Lisa R. Arnold discusses the responses of teachers at the American University of Beirut (AUB) to canonical texts of rhetoric and composition theory, in particular “Language Difference in Writing: A Translingual Approach,” by Bruce Horner, Min-Zhan Lu, Jacqueline Jones Royster, and John Trimbur. Arnold notes that in Lebanon, where translingualism is an “everyday reality” (80), the question of how to accommodate and value multiple language practices can resonate very differently than it does in the presumably monolingual North-American context in which the theory was proposed.

As the first director of the AUB writing program, Arnold hoped to provide faculty with professional development opportunities (75), at the same time responding to questions from scholars like Mary N. Muchiri and her colleagues and Christiane Donahue that ask composition professionals in North America to recognize “the diverse pedagogical traditions, methods of research, and values attached to literacy in non-U.S. contexts” (72).

As an “American-style university that is a leader in the Middle-East North-Africa (MENA) region,” AUB presents an opportunity for the study of such issues because it is “unique” among institutions outside of North America in having four “full-time, professorial-rank” lines for rhetoric and composition PhDs; the university also plans to implement an M.A. in rhetoric and composition (74).

In order to further faculty engagement with composition theory, the university offered a ten-session seminar during the 2013-2014 academic year. These seminars, attended by seventeen AUB faculty with varied levels of experience teaching in the program, explored a range of topics addressing writing theory and instruction (75). The final sessions each semester addressed teaching writing in the particular context of Lebanon/AUB.

Arnold attended all seminar sessions as a participant-observer and subsequently conducted interviews with fifteen participants, asking them to focus on what seemed “most relevant” to teaching and to the specific environment of AUB (77). Five faculty who had audited a previous graduate course on writing theory and pedagogy facilitated the sessions. Participants also completed an anonymous survey (76).

General responses indicated that faculty found rhetoric and composition theory to be “open,” “tolerant,” and “concrete,” engaged with students as individual writers (77-78). The issue of translingualism was among the discussions that inspired a range of responses (78), especially in regard to the question of how rhetoric and composition theory applied to teaching in Lebanon (78).

The Horner et al. article, which attendees read during the final fall-semester session, addressed the monolingual audience that presumably characterizes North American contexts. To this audience, according to Arnold, Horner et al. argue that rather than being treated as “an obstacle to be overcome,” difference in language should be viewed through a lens that “takes advantage of and appreciates students’ different strengths in English as well as in other languages and . . . reflects the heterogeneity of communicative practices worldwide” (79).

AUB faculty expressed interest in the theory but also voiced concerns about what it might mean in their context when implemented in the classroom. Many seminar attendees brought backgrounds in EFL or ESL to the sessions; Arnold reports general agreement that a “more flexible approach toward language difference” would be worth considering (79).

Concern, however, seemed to center around the degree to which a more tolerant attitude toward error might impact the need for students to learn formal English in order to succeed in the non-U.S. context (80). Arnold writes that in Lebanon, as in the African contexts discussed by Muchiri et al., universities like AUB are “highly selective” and “English carries a different value for its users” (80). She notes the concerns of “Rania,” who posits that British universities expect less expertise in English from students from “developing nations” who will presumably return home after graduation than from native speakers. Rania fears that allowing students flexibility in their use of English will become a process of withholding “correct English” in order to impose “a new form of colonialism” (81). However, according to Arnold, Rania subsequently appreciated the opportunities for learning offered by a translingual approach (81).

The response of “Rasha” similarly indicates ambivalence toward translingualism. Students either liked the opportunity to use Arabic or, in her words, “just hated it” (qtd. in Arnold 82), but she found that discussions of whether or not such multilingual practice was appropriate increased student engagement with issues of language use itself (82). Other examples demonstrate that students do translingual work regardless of the teacher’s goals, for example, using Arabic for group work (82). A number of the teachers drew on their own experiences as learners of multiple languages to encourage students to embrace the challenges involved in a multilingual context. Arnold reports that these teachers felt empowered by translingual theory to draw on language difference as a resource (84-85).

Teachers like “Malik,” however, highlighted the importance of providing students with the kinds of English skills that would serve them in their culture (83), while “Jenna” expressed concerns that the increased tolerance urged by Horner et al. would lead students to become “too confident” that audiences would understand translingually inflected communication: “[Students] get this false perception of abilities and skills which are not there” (qtd. in Arnold 85).

For Arnold, her experience working with writing instructors charged with teaching English outside of an English-speaking environment gives presence to the theoretical precepts of translingualism. She notes that graduates of rhetoric and composition programs may often find themselves taking jobs or providing resources to colleagues outside of the North-American context, and she urges these graduates to attend to the degree to which their multilingual colleagues are often already unacknowledged “experts in their own right” with regard to working with language difference (87):

[T]here is a complexity to literacy practices and pedagogies that practitioners outside of North America understand deeply, and from which those of us trained in a presumably monolingual context can learn. (87)

 


by

Lu and Horner. Introduction: Translingual Work. CE, Jan. 2016. Posted 02/28/2016.

Lu, Min-Zhan, and Bruce Horner. “Introduction: Translingual Work.” College English 78.3 (2016): 207-18. Print.

In their Introduction to the symposium on translingualism in the January, 2016, issue of College English, Min-Zhan Lu and Bruce Horner address the complexities of defining and implementing a translingual pedagogy. The Introduction previews the contributions of the participants, who were among those invited after a “conversation among four of us—Bruce Horner, Min-Zhan Lu, Anis Bawarshi, and Juan Guerra . . . .” that pointed to the need to explore translingualism as “one possible entry point for work contesting the monolingualism that continues to dominate the teaching and study of college writing and reading in the United States and elsewhere” (207).

Participants received a list of “tenets for a ‘translingual approach'” developed from a list compiled by Lu. Among the concerns noted were attention to

  • “language . . . as performative: not something we have but something we do”;
  • “users of language as actively forming and transforming the very conventions we use. . . .”;
  • “communicative practices as not neutral or innocent but informed by and informing . . . cultural relations of asymmetrical power”; [and]
  • “all communicative practices as mesopolitical acts, actively negotiating and constituting complex relations of power. . . .” (208)

These tenets also posit “difference as the norm of all utterances”; translingualism, in this view, does not address solely “deviations from the norm” (208).

Participants were further invited to consider the question of which tenets were important in their own work and how further work on these issues might “enhance the work of composition in each of the areas” (209).

Following their discussion of the contributions, Lu and Horner address a number of broad questions that they feel shape and emerge from the symposium. They describe translingualism not as it is sometimes construed—as a focus on L2 learning or other apparent deviations from standard usage; rather, they see it as addressing the use of language by “ordinary people” in daily lived experience, naming as its true “other” the claim that there is, indeed, a monolinguistic norm that sets universal standards (212).

They particularly address what they describe as a “conundrum” addressed in a number of the symposium articles: whether a translingual approach can be understood as an extension of approaches already in use in writing studies and classrooms, or whether it ought to be seen as calling for a more active engagement to “combat” the “deleterious sociocultural effects of the monolingual ideology” (213). They endorse the idea of translingualism as a forceful “rejection” of this ideology and its effects, arguing that the work of Mina P. Shaughnessy and others in exploring the language uses of students in 1960s and 1970s constituted “a sociopolitical movement” that, in Shaughnessy’s words, “pedagogically radicalized” writing instruction (qtd. in Lu and Horner 213).

To further address the conundrum, they explore the claims of Louis-Jean Calvet, who contends that, in fact, “languages do not exist” (qtd. in Lu and Horner 213; emphasis original); what does exist are “representations—what people think about languages and the way they are spoken” (qtd. in Lu and Horner 213; emphasis original). Understanding language this way leads Lu and Horner to the view that this process of representation itself makes users active in creating the languages they use. Recognizing the agency of users through their practice, the authors believe, can enable action on the power relations that govern these representations and can, in the words of two contributors, “open up” possibilities within language, genres, and modes that had previously been closed off (214).

Understanding language as “always emergent” rather than “time-less” (214) or fixed, Lu and Horner argue, requires as its corollary understanding that a translingual approach does not point to a particular set of language practices that can be identified in usage or texts. Rather, translingualism itself, as a representation, will be subject to “inevitable reworking,” not just by scholars attempting to apply it but also by “students at the pedagogical site” (215). Translingual pedagogy requires a “shift” from a transmission model of language instruction “to a more dialogical course of study” (215).

Lu and Horner caution that a focus on dialogic classrooms as sites of translingual teaching will not, in itself, upend the dominant monolingual ideology or ensure increased social justice or reorientations of power; instead, it is “an occasion for labor, the labor of revision,” a set of practices that “can be reworked” toward desirable ends (216). They call for readers of the symposium to see the contributions as a call to join in that labor of ongoing reworking and as examples of the kinds of work that can be done (216).


by

Trimbur, John. Translingualism and Close Reading. CE, Jan. 2016. Posted 01/30/2016.

Trimbur, John. “Translingualism and Close Reading.” College English 78.3 (2016): 219-27. Print.
The January 2016 issue of College English addresses the question of “translingualism,” a term that Min-Zhan Lu and Bruce Horner, in their Introduction to the issue, see as “one possible entry point” for overcoming the perception that there is only a single form of English that is universally standard and acceptable (207). They discuss at length the challenges of defining translingualism, presenting it in part as the recognition that difference in language use is not just a phenomenon of L2 learning but rather is a feature of “the normal transactions of daily communicative practice of ordinary people” (212).
In this issue, John Trimbur “traces a branch of translingualism to its source” (220). He focuses on texts by Mina Shaughnessy, David Bartholomae, Bruce Horner, and Min-Zhan Lu. He locates the origin of this translingual impulse in the evolution of open admissions at the City University of New York (CUNY) in the 1960s and 1970s, as writing teachers confronted evidence that the edifice of “monolingualism” in English was an ideology of exclusion rather than a fact.
Trimbur argues that, far from being an accurate description of United States English prior to the turmoil of the 1960s, “monolingualism is not a possible linguistic condition at all” (220). He contends that all speakers move among various dialects and registers; the heterogeneous voices that are now becoming more audible demonstrate the existence of “a plurilingual periphery within the Anglophone centers” such as London and New York (219; emphasis original).
Trimbur recounts the history of CUNY from its birth in 1847 as the Free Academy, documenting that despite initiatives such as Search for Excellence, Elevation, and Knowledge (SEEK), the system remained largely White (220). In the late 1960s, demands from groups like the Black and Puerto Rican Student Community (BPRSC), in concert with growing civil-rights activism, pressured CUNY administrators to establish a true open-admissions policy (221). This shift introduced writing teachers to student writing that many considered worthy only of “eradicat[ion]” (221). In this new environment, Trimbur writes, “literature MAs and PhDs,” among them Mina Shaughnessy, began to draw on their expertise in New Critical close reading “to find order . . . in the language differences of students formerly excluded by selective admissions” (221).
Trimbur contrasts Shaughnessy’s work to understand the logic behind apparently anomalous usage with the approach of Bartholomae, one of the scholars Trimbur designates as members of the “Pitt school” (222). These scholars, Trimbur writes, recognized that literary theorists routinely constructed meaning from arcane texts by literary authors such as Donald Barthelme or e. e. cummings; the Pitt school critics “placed an extraordinary pressure on themselves” to apply these same approaches to student writing in order to understand “evidence of intention” (222).
To clarify this contrast, Trimbur hypothesizes Bartholomae’s response to an example of student writing addressed by Shaughnessy in her 1977 Errors and Expectations: A Guide to the Teacher of Basic Writing. Whereas Shaughnessy argued for “a logic of nonstandard English” in the essay by noting its use of the conventions of an “evangelical sermon,” Trimbur posits that Bartholomae would see the student practicing rhetorical strategies that positioned him as an applicant to academic authority, such as “moves up and down the ladder of abstraction” from concepts to examples and a gesture toward academic citation (223). In Trimbur’s view, Bartholomae would interpret this student’s effort as a sign not of a writer unable to abandon his “home language” bur rather as a writer “activated by his intention to ‘invent the university'” (223).
Trimbur then compares Bartholomae’s contribution to the approaches of Horner and Lu. Though he contends that both Horner and Bartholomae viewed language difference as socially and historically constructed (220), he contends that Bartholomae maintained in some part a view of standard English as a destination toward which students evolved, somewhat as an L2 learner might move toward a “target language” (224). In Trimbur’s contrast, Horner develops a “dialectical and resolutely social sense of error” in which editing becomes a “negotiation in situations of unequal power and authority”; in this view, teachers might look at student writing “not just for its errors but for the possible rhetorical effects of its language differences” (224).
Trimbur argues that Min-Zhan Lu further complicates the idea of a monolingual center for English by challenging the integrity of standard usage itself. In Lu’s view, Trimbur states, supposedly standard language is inherently “unstable, fluctuating, and hybrid” (225). The resistance of monolingual ideologies to the unconventional and different is the product of a “struggle among conflicting discourses with unequal sociopolitical power” (224-25). In this view, linguistic hierarchies become “momentary hegemon[ies}” (225), within which close reading can locate the value of elided difference.
Trimbur sees an important benefit in such approaches to student writing in their power to bring basic and second-language writing in from “the margins,” where they have been “orbiting around the mainstream English at the center in first-year composition” (226). He calls on composition to cease seeing difference as a reason to isolate the unacceptable but rather to recognize the degree to which difference actually inhabits all language use, thus “dismant[ing] these divisions and the pernicious judgments about language differences and about the differences between people that they have rested on” (226).