College Composition Weekly: Summaries of research for college writing professionals

Read, Comment On, and Share News of the Latest from the Rhetoric and Composition Journals

Category Archives: empirical research


by 4 Comments

Grouling and Grutsch McKinney. Multimodality in Writing Center Texts. C&C, in press, 2016. Posted 08/21/2016.

Grouling, Jennifer, and Grutsch McKinney, Jackie. “Taking Stock: Multimodality in Writing Center Users’ Texts.” (In press.) Computers and Composition (2016). http://dx.doi.org/10.1016/j.compcom.2016.04.003 Web. 12 Aug. 2016.

Jennifer Grouling and Jackie Grutsch McKinney note that the need for multimodal instruction has been accepted for more than a decade by composition scholars (1). But they argue that the scholarship supporting multimodality as “necessary and appropriate” in classrooms and writing centers has tended to be “of the evangelical vein” consisting of “think pieces” rather than actual studies of how multimodality figures in classroom practice (2).

They present a study of multimodality in their own program at Ball State University as a step toward research that explores what kinds of multimodal writing takes place in composition classrooms (2). Ball State, they report, can shed light on this question because “there has been programmatic and curricular support here [at Ball State] for multimodal composition for nearly a decade now” (2).

The researchers focus on texts presented to the writing center for feedback. They ask three specific questions:

Are collected texts from writing center users multimodal?

What modes do students use in creation of their texts?

Do students call their texts multimodal? (2)

For two weeks in the spring semester, 2014, writing center tutors asked students visiting the center to allow their papers to be included in the study. Eighty-one of 214 students agreed. Identifying information was removed and the papers stored in a digital folder (3).

During those two weeks as well as the next five weeks, all student visitors to the center were asked directly if their projects were multimodal. Students could respond “yes,” “no,” or “not sure” (3). The purpose of this extended inquiry was to ensure that responses to the question during the first two “collection” weeks were not in some way unrepresentative. Grouling and Grutsch McKinney note that the question could be answered online or in person; students were not provided with a definition of “multimodal” even if they expressed confusion but only told to “answer as best they could” (3).

The authors decided against basing their study on the argument advanced by scholars like Jody Shipka and Paul Prior that “all communication practices have multimodal components” because such a definition did not allow them to see the distinctions they were investigating (3). Definitions like those presented by Tracey Bowen and Carl Whithaus that emphasize the “conscious” use of certain components also proved less helpful because students were not interviewed and their conscious intent could not be accessed (3). However, Bowen and Whithaus also offered a “more succinct definition” that proved useful: “multimodality is the ‘designing and composing beyond written words'” (qtd. in Grouling and Grutsch McKinney 3).

Examination of the papers led the researchers to code for a “continuum” of multimodality rather than a present/not-present binary (3-4). Fifty-seven, or 74%, of the papers were composed only in words and were coded as zero or “monomodal” (4). Some papers occupied a “grey area” because of elements like bulleted lists and tables. The researchers coded texts using bullets as “1” and those using lists and tables “2.” These categories shared the designation “elements of graphic design”; 19.8%, or 16, papers met this designation. Codes “3” and “4” indicated one or more modes beyond text and thus indicated “multimodal” work. No paper received a “4”; only eight, or 9.9%, received a “3,” indicating inclusion of one mode beyond words (4). Thus, the study materials exhibited little use of multimodal elements (4).

In answer to the second question, findings indicated that modes used even by papers coded “3” included only charts, graphs, and images. None used audio, video, or animation (4). Grouling and Grutsch McKinney posit that the multimodal elements were possibly not “created by the student” and that the instructor or template may have prompted the inclusion of such materials (5).

They further report that they could not tell whether any student had “consciously manipulated” elements of the text to make it multimodal (5). They observe that in two cases, students used visual elements apparently intended to aid in development of a paper in progress (5).

The “short answer” to the third research question, whether students saw their papers as multimodal, was “not usually” (5; emphasis original). Only 6% of 637 appointments and 6% of writers of the 81 collected texts answered yes. In only one case in which the student identified the paper as multimodal did the coders agree. Two of the five texts called multimodal by students received a code of 0 from the raters (5). Students were more able to recognize when their work was not multimodal; 51 of 70 texts coded by the raters as monomodal were also recognized as such by their authors (5).

Grouling and Grutsch McKinney express concern that students seem unable to identify multimodality given that such work is required in both first-year courses, and even taking transfer students into account, the authors note that “the vast majority” of undergraduates will have taken a relevant course (6). They state that they would be less concerned that students do not use the term if the work produced exhibited multimodal features, but this was not the case (6).

University system data indicated that a plurality of writing center attendees came from writing classes, but students from other courses produced some of the few multimodal pieces, though they did not use the term (7).

Examining program practices, Grouling and Grutsch McKinney determined that often only one assignment was designated “multimodal”—most commonly, presentations using PowerPoint (8). The authors advocate for “more open” assignments that present multimodality “as a rhetorical choice, and not as a requirement for an assignment” (8). Such emphasis should be accompanied by “programmatic assessment” to determine what students are actually learning (8-9).

The authors also urge more communication across the curriculum about the use of multiple modes in discipline-specific writing. While noting that advanced coursework in a discipline may have its own vocabulary and favored modes, Grouling and Grutsch McKinney argue that sharing the vocabulary from composition studies with faculty across disciplines will help students see how concepts from first-year writing apply in their coursework and professional careers (9).

The authors contend that instructors and tutors should attend to “graphic design elements” like “readability and layout” (10). In all cases, they argue, students should move beyond simply inserting illustrations into text to a better “integration” of modes to enhance communication (10). Further, incorporating multimodal concepts in invention and composing can enrich students’ understanding of the writing process (10). Such developments, the authors propose, can move the commitment to multimodality beyond the “evangelical phase” (11).

 


by 1 Comment

Moxley and Eubanks. Comparing Peer Review and Instructor Ratings. WPA, Spring 2016. Posted 08/13/2016.

Moxley, Joseph M., and David Eubanks. “On Keeping Score: Instructors’ vs. Students’ Rubric Ratings of 46,689 Essays.” Journal of the Council of Writing Program Administrators 39.2 (2016): 53-80. Print.

Joseph M. Moxley and David Eubanks report on a study of their peer-review process in their two-course first-year-writing sequence. The study, involving 16,312 instructor evaluations and 30,377 student reviews of “intermediate drafts,” compared instructor responses to student rankings on a “numeric version” of a “community rubric” using a software package, My Reviewers, that allowed for discursive comments but also, in the numeric version, required rubric traits to be assessed on a five-point scale (59-61).

Exploring the literature on peer review, Moxley and Eubanks note that most such studies are hindered by small sample sizes (54). They note a dearth of “quantitative, replicable, aggregated data-driven (RAD) research” (53), finding only five such studies that examine more than 200 students (56-57), with most empirical work on peer review occurring outside of the writing-studies community (55-56).

Questions investigated in this large-scale empirical study involved determining whether peer review was a “worthwhile” practice for writing instruction (53). More specific questions addressed whether or not student rankings correlated with those of instructors, whether these correlations improved over time, and whether the research would suggest productive changes to the process currently in place (55).

The study took place at a large research university where the composition faculty, consisting primarily of graduate students, practiced a range of options in their use of the My Reviewers program. For example, although all commented on intermediate drafts, some graded the peer reviews, some discussed peer reviews in class despite the anonymity of the online process, and some included training in the peer-review process in their curriculum, while others did not.

Similarly, the My Reviewers package offered options including comments, endnotes, and links to a bank of outside sources, exercises, and videos; some instructors and students used these resources while others did not (59). Although the writing program administration does not impose specific practices, the program provides multiple resources as well as a required practicum and annual orientation to assist instructors in designing their use of peer review (58-59).

The rubric studied covered five categories: Focus, Evidence, Organization, Style, and Format. Focus, Organization, and Style were broken down into the subcategories of Basics—”language conventions”—and Critical Thinking—”global rhetorical concerns.” The Evidence category also included the subcategory Critical Thinking, while Format encompassed Basics (59). For the first year and a half of the three-year study, instructors could opt for the “discuss” version of the rubric, though the numeric version tended to be preferred (61).

The authors note that students and instructors provided many comments and other “lexical” items, but that their study did not address these components. In addition, the study did not compare students based on demographic features, and, due to its “observational” nature, did not posit causal relationships (61).

A major finding was that. while there was some “low to modest” correlation between the two sets of scores (64), students generally scored the essays more positively than instructors; this difference was statistically significant when the researchers looked at individual traits (61, 67). Differences between the two sets of scores were especially evident on the first project in the first course; correlation did increase over time. The researchers propose that students learned “to better conform to rating norms” after their first peer-review experience (64).

The authors discovered that peer reviewers were easily able to distinguish between very high-scoring papers and very weak ones, but struggled to make distinctions between papers in the B/C range. Moxley and Eubanks suggest that the ability to distinguish levels of performance is a marker for “metacognitive skill” and note that struggles in making such distinctions for higher-quality papers may be commensurate with the students’ overall developmental levels (66).

These results lead the authors to consider whether “using the rubric as a teaching tool” and focusing on specific sections of the rubric might help students more closely conform to the ratings of instructors. They express concern that the inability of weaker students to distinguish between higher scoring papers might “do more harm than good” when they attempt to assess more proficient work (66).

Analysis of scores for specific rubric traits indicated to the authors that students’ ratings differed more from those of instructors on complex traits (67). Closer examination of the large sample also revealed that students whose teachers gave their own work high scores produced scores that more closely correlated with the instructors’ scores. These students also demonstrated more variance than did weaker students in the scores they assigned (68).

Examination of the correlations led to the observation that all of the scores for both groups were positively correlated with each other: papers with higher scores on one trait, for example, had higher scores across all traits (69). Thus, the traits were not being assessed independently (69-70). The authors propose that reviewers “are influenced by a holistic or average sense of the quality of the work and assign the eight individual ratings informed by that impression” (70).

If so, the authors suggest, isolating individual traits may not necessarily provide more information than a single holistic score. They posit that holistic scoring might not only facilitate assessment of inter-rater reliability but also free raters to address a wider range of features than are usually included in a rubric (70).

Moxley and Eubanks conclude that the study produced “mixed results” on the efficacy of their peer-review process (71). Students’ improvement with practice and the correlation between instructor scores and those of stronger students suggested that the process had some benefit, especially for stronger students. Students’ difficulty with the B/C distinction and the low variance in weaker students’ scoring raised concerns (71). The authors argue, however, that there is no indication that weaker students do not benefit from the process (72).

The authors detail changes to their rubric resulting from their findings, such as creating separate rubrics for each project and allowing instructors to “customize” their instruments (73). They plan to examine the comments and other discursive components in their large sample, and urge that future research create a “richer picture of peer review processes” by considering not only comments but also the effects of demographics across many settings, including in fields other than English (73, 75). They acknowledge the degree to which assigning scores to student writing “reifies grading” and opens the door to many other criticisms, but contend that because “society keeps score,” the optimal response is to continue to improve peer-review so that it benefits the widest range of students (73-74).


by

Kellogg et al. Verbal Working Memory and Sentence Composition. J of Writing Research, Feb. 2016. Posted 04/24/2016.

Kellogg, Ronald T., Casey E. Turner, Alison P. Whiteford, and Andrew Mertens. “The Role of Working Memory in Planning and Generating Written Sentences.” The Journal of Writing Research 7.3 (2016): 397-416. Web. 04 Apr. 2016.

Kellogg et al. conduct an experiment designed to shed light on how working memory (WM) relates to the sequence of the processes that go into generating written sentences. They draw on a body of research like that of Linda Flower and John Hayes that posits a conceptual “planning” phase that is largely nonverbal and a phase of grammatical and orthographical translation that leads to writing or typing (398).

The models propose that the planning stage draws on a visual component when the information to be translated involves concrete nouns that evoke images (399). The researchers hypothesized that a visual working-memory load (that is, a non-verbal load) during the writing task would interact with the planning, or non-verbal, stage, while imposing a verbal working-memory load would impact transcription of the written sentences during the phase when grammatical processing was taking place.

To test these hypotheses, Kellogg et al. provided participants with two different kinds of prompts and asked for two different kinds of sentences using these prompts. The prompts were paired words to be used in sentences. One set of pairs involved words that were semantically related, such as “door” and “knob.” The other set provided words that were not related, such as “ice” and “jail.” All word pairs were concrete and taken from a scale tested to calculate how easily they would be recognized (401).

The rationale for these choices was that more planning was needed to link unrelated word pairs in a sentence, so that the load on visual memory would also increase (399). But because the models specified that planning takes place before “grammatical encoding,” the kind of word pair required should have no effect on this latter process (400).

To investigate the relationship of verbal memory to the sentence-generating process, the researchers imposed another manipulation by asking half of the subjects to compose active-voice sentences, while the other half composed passive-voice sentences. In this case, the rationale was research showing that passive-voice constructions are more complex than active and thus should burden “verbal working memory” but not visual (400). Subjects were screened for their ability to produce the kinds of sentences required as instructed in the prompts (403).

Thus the protocol required subjects to produce sentences, either in the passive or active voice as instructed, in three conditions: while being asked to perform a “concurrent” visual working-memory task, a concurrent verbal working-memory task, or no concurrent task (401). The visual concurrent task involved studying and learning a set of images chosen from “the SPSS Marker Set in Microsoft Word” that were “designed to be not readily named” (401). In contrast, the verbal concurrent task was composed of nine digits that “could be coded verbally” (401). In both cases, after completing the required sentence, the participants were asked to determine whether a set of symbols or digits matched the ones they had previously seen (402-03).

The level of accuracy in this ability to match the symbols or digits displayed after the writing task with those seen previously constituted the main data for the study (409). The researchers wanted to test the hypothesis that impeding visual working memory would not interact with the verbal process of grammatical encoding, but that impeding verbal working memory would do so (399).

Additional data came from measuring factors such as the length of the sentences produced, the number of words produced per second in each trial, the time involved in producing the sentences, and the time before typing began. These factors were controlled for in the interpretation of the major findings.

The authors’ predictions, therefore, were that factors that made planning harder, such as efforts to work with unrelated nouns, would show up in less accurate results for the visual working-memory task, the symbols, while factors that impeded grammatical encoding, such as writing passive-voice sentences, would manifest themselves in less accuracy in recalling the verbal working-memory task components, the digits.

Unexpectedly, they found that even though, as predicted, passive voice sentences took longer to write, required more words, and resulted in fewer words per second, the type of sentence made “no reliable difference” on the verbal concurrent task (the digits): “if anything, actives produced more interference than passives” (410). They found also that, contrary to expectations, related word pairs “most disrupted the verbal WM task” (410). Thus, the operations assumed to be simplest required the most verbal working-memory effort, and factors that were expected to affect visual working memory because they were presumably involved in planning did not produce the hypothesized interference with the visual task (409-10).

That the presumably simpler effort involved in producing an active-voice sentence using a related pair of words demanded more verbal working memory led the researchers to consult the work of M. Fayol, who proposed that the “Kellogg (1996) model” may fail to take into account “an understanding of the temporal dynamics” involved in writing sentences (411). To explain their findings in the current study, Kellogg et al. posited that the conceptual work of planning for the simpler active-voice/related-pair resulted in “a single chunk that was then immediately cascaded forward to grammatical encoding” (411). In contrast, they suggest, the more difficult planning for a sentence using the unrelated pair occurred incrementally, possibly in parallel with grammatical encoding or during pauses, for example, after typing the first of the words. Thus, the grammatical encoding process that would have shown up as a demand on verbal working memory was broken up into “piecemeal” activities by planning phases rather than implemented all at once (411-12). Such intermittent planning/encoding has been demonstrated in studies of speech and typing (412). In short, easier planning can result in more pressure on verbal working memory.

The authors conclude that “the role of WM in written sentence production is markedly more complex than previously postulated” (414).


by

Comer and White. MOOC Assessment. CCC, Feb. 2016. Posted 04/18/2016.

Comer, Denise K., and Edward M. White. “Adventuring into MOOC Writing Assessment: Challenges, Results, and Possibilities.” College Composition and Communication 67.3 (2016): 318-59. Print.

Denise K. Comer and Edward M. White explore assessment in the “first-ever first-year-writing MOOC,” English Composition I: Achieving Expertise, developed under the auspices of the Bill & Melinda Gates Foundation, Duke University, and Coursera (320). Working with “a team of more than twenty people” with expertise in many areas of literacy and online education, Comer taught the course (321), which enrolled more than 82,000 students, 1,289 of whom received a Statement of Accomplishment indicating a grade of 70% or higher. Nearly 80% of the students “lived outside the United States” and for a majority, English was not the first language, although 59% of these said they were “proficient or fluent in written English” (320). Sixty-six percent had bachelor’s or master’s degrees.

White designed and conducted the assessment, which addressed concerns about MOOCs as educational options. The authors recognize MOOCs as “antithetical” (319) to many accepted principles in writing theory and pedagogy, such as the importance of interpersonal instructor/student interaction (319), the imperative to meet the needs of a “local context” (Brian Huot, qtd. in Comer and White 325) and a foundation in disciplinary principles (325). Yet the authors contend that as “MOOCs are persisting,” refusing to address their implications will undermine the ability of writing studies specialists to influence practices such as Automated Essay Scoring, which has already been attempted in four MOOCs (319). Designing a valid assessment, the authors state, will allow composition scholars to determine how MOOCs affect pedagogy and learning (320) and from those findings to understand more fully what MOOCs can accomplish across diverse populations and settings (321).

Comer and White stress that assessment processes extant in traditional composition contexts can contribute to a “hybrid form” applicable to the characteristics of a MOOC (324) such as the “scale” of the project and the “wide heterogeneity of learners” (324). Models for assessment in traditional environments as well as online contexts had to be combined with new approaches that addressed the “lack of direct teacher feedback and evaluation and limited accountability for peer feedback” (324).

For Comer and White, this hybrid approach must accommodate the degree to which the course combined the features of an “xMOOC” governed by a traditional academic course design with those of a “cMOOC,” in which learning occurs across “network[s]” through “connections” largely of the learners’ creation (322-23).

Learning objectives and assignments mirrored those familiar to compositionists, such as the ability to “[a]rgue and support a position” and “[i]dentify and use the stages of the writing process” (323). Students completed four major projects, the first three incorporating drafting, feedback, and revision (324). Instructional videos and optional workshops in Google Hangouts supported assignments like discussion forum participation, informal contributions, self-reflection, and peer feedback (323).

The assessment itself, designed to shed light on how best to assess such contexts, consisted of “peer feedback and evaluation,” “Self-reflection,” three surveys, and “Intensive Portfolio Rating” (325-26).

The course supported both formative and evaluative peer feedback through “highly structured rubrics” and extensive modeling (326). Students who had submitted drafts each received responses from three other students, and those who submitted final drafts received evaluations from four peers on a 1-6 scale (327). The authors argue that despite the level of support peer review requires, it is preferable to more expert-driven or automated responses because they believe that

what student writers need and desire above all else is a respectful reader who will attend to their writing with care and respond to it with understanding of its aims. (327)

They found that the formative review, although taken seriously by many students, was “uneven,” and students varied in their appreciation of the process (327-29). Meanwhile, the authors interpret the evaluative peer review as indicating that “student writing overall was successful” (330). Peer grades closely matched those of the expert graders, and, while marginally higher, were not inappropriately high (330).

The MOOC provided many opportunities for self-reflection, which the authors denote as “one of the richest growth areas” (332). They provide examples of student responses to these opportunities as evidence of committed engagement with the course; a strong desire for improvement; an appreciation of the value of both receiving and giving feedback; and awareness of opportunities for growth (332-35). More than 1400 students turned in “final reflective essays” (335).

Self-efficacy measures revealed that students exhibited an unexpectedly high level of confidence in many areas, such as “their abilities to draft, revise, edit, read critically, and summarize” (337). Somewhat lower confidence levels in their ability to give and receive feedback persuade the authors that a MOOC emphasizing peer interaction served as an “occasion to hone these skills” (337). The greatest gain occurred in this domain.

Nine “professional writing instructors” (339) assessed portfolios for 247 students who had both completed the course and opted into the IRB component (340). This assessment confirmed that while students might not be able to “rely consistently” on formative peer review, peer evaluation could effectively supplement expert grading (344).

Comer and White stress the importance of further research in a range of areas, including how best to support effective peer response; how ESL writers interact with MOOCs; what kinds of people choose MOOCs and why; and how MOOCs might function in WAC/WID situations (344-45).

The authors stress the importance of avoiding “extreme concluding statements” about the effectiveness of MOOCs based on findings such as theirs (346). Their study suggests that different learners valued the experience differently; those who found it useful did so for varied reasons. Repeating that writing studies must take responsibility for assessment in such contexts, they emphasize that “MOOCs cannot and should not replace face-to-face instruction” (346; emphasis original). However, they contend that even enrollees who interacted briefly with the MOOC left with an exposure to writing practices they would not have gained otherwise and that the students who completed the MOOC satisfactorily amounted to more students than Comer would have reached in 53 years teaching her regular FY sessions (346).

In designing assessments, the authors urge, compositionists should resist the impulse to focus solely on the “Big Data” produced by assessments at such scales (347-48). Such a focus can obscure the importance of individual learners who, they note, “bring their own priorities, objectives, and interests to the writing MOOC” (348). They advocate making assessment an activity for the learners as much as possible through self-reflection and through peer interaction, which, when effectively supported, “is almost as useful to students as expert response and is crucial to student learning” (349). Ultimately, while the MOOC did not succeed universally, it offered many students valuable writing experiences (346).


by

Bourelle et al. Multimodal in f2f vs. online classes. C&C, Mar. 2016. Posted 01/24/2016.

Bourelle, Andrew, Tiffany Bourelle, Anna V. Knutson, and Stephanie Spong. “Sites of Multimodal Literacy: Comparing Student Learning in Online and Face-to-Face Environments.” Computers and Composition 39 (2015): 55-70. Web. 14 Jan. 2016.

Andrew Bourelle, Tiffany Bourelle, Anna V. Knutson, and Stephanie Spong report on a “small pilot study” at the University of New Mexico that compares how “multimodal liteacies” are taught in online and face-to-face (f2f) composition classes (55-56). Rather than arguing for the superiority of a particular environment, the writers contend, they hope to “understand the differences” and “generate a conversation regarding what instructors of a f2f classroom can learn from the online environment, especially when adopting a multimodal curriculum” (55). The authors find that while differences in overall learning measures were slight, with a small advantage to the online classes, online students demonstrated considerably more success in the multimodal component featured in both kinds of classes (60).

They examined student learning in two online sections and one f2f section teaching a “functionally parallel” multimodal curriculum (58). The online courses were part of eComp, an online initiative at the University of New Mexico based on the Writers’ Studio program at Arizona State University, which two of the current authors had helped to develop (57). Features derived from the Writers’ Studio included the assignment of three projects to be submitted in an electronic portfolio as well as a reflective component in which the students explicated their own learning. Additionally, the eComp classes “embedded” instructional assistants (IAs): graduate teaching assistants and undergraduate tutors (57-58). Students received formative peer review and feedback from both the instructor and the IAs. (57-58).

Students created multimodal responses to the three assignments—a review, a commentary, and a proposal. The multimodal components “often supplemented, rather than replaced, the written portion of the assignment” (58). Students analyzed examples from other classes and from public media through online discussions, focusing on such issues as “the unique features of each medium” and “the design features that either enhanced or stymied” a project’s rhetorical intent (58). Bourelle et al. emphasize the importance of foregrounding “rhetorical concepts” rather than the mechanics of electronic presentation (57).

The f2f class, taught by one of the authors who was also teaching one of the eComp classes, used the same materials, but the online discussion and analysis were replaced by in-class instruction and interaction, and the students received instructor and peer feedback (58). Students could consult the IAs in the campus writing center and seek other feedback via the center’s online tutorials (58).

The authors present their assessment as both quantitative, through holistic scores using a rubric that they present in an Appendix, and qualitative, through consideration of the students’ reflection on their experiences (57). The importance of including a number of different genres in the eportfolios created by both kinds of classes required specific norming on portfolio assessment for the five assessment readers (58-59). Four of the readers were instructors or tutors in the pilot, with the fifth assigned so that instructors would not be assessing their own students’ work (58). Third reads reconciled disparate scores. The readers examined all of the f2f portfolios and 21, or 50%, of the online submissions. Bourelle et al. provide statistical data to argue that this 50% sample adequately supports their conclusions at a “confidence level of 80%” (59).

The rubric assessed features such as

organization of contents (a logical progression), the overall focus (thesis), development (the unique features of the medium and how well the modes worked together), format and design (overall design aesthetics . . . ), and mechanics. . . . (60)

Students’ learning about multimodal production was assessed through the reflective component (60). The substantial difference in this score led to a considerable difference in the total scores (61).

The authors provide specific examples of work done by an f2f student and by an online student to illustrate the distinctions they felt characterized the two groups. They argue that students in the f2f classes as a group had difficulties “mak[ing] choices in design according to the needs of the audience” (61). Similarly, in the reflective component, f2f students had more trouble explaining “their choice of medium and how the choice would best communicate their message to the chosen audience” (61).

In contrast, the researchers state that the student representing the online cohort exhibits “audience awareness with the choice of her medium and the content included within” (62). Such awareness, the authors write, carried through all three projects, growing in sophistication (62-63). Based on both her work and her reflection, this student seemed to recognize what each medium offered and to make reasoned choices for effect. The authors present one student from the f2f class who demonstrated similar learning, but argue that, on the whole, the f2f work and reflections revealed less efficacy with multimodal projects (63).

Bourelle et al. do not feel that self-selection for more comfort with technology affected the results because survey data indicated that “life circumstances” rather than attitudes toward technology governed students’ choice of online sections (64). They indicate, in contrast, that the presence of the IAs may have had a substantive effect (64).

They also discuss the “archival” nature of an online environment, in which prior discussion and drafts remained available for students to “revisit,” with the result that the reflections were more extensive. Such reflective depth, Claire Lauer suggests, leads to “more rhetorically effective multimodal projects” (cited in Bourelle et al. 65).

Finally, they posit an interaction between what Rich Halverson and R. Benjamin Shapiro designate “technologies for learners” and “technologies for education.” The latter refer to the tools used to structure classrooms, while the former include specific tools and activities “designed to support the needs, goals, and styles of individuals” (qtd. in Bourelle et al. 65). The authors posit that when the individual tools students use are in fact the same as the “technologies for education,” students engage more fully with multimodality in such an immersive multimodal environment.

This interaction, the authors suggest, is especially important because of the need to address the caveat from research and the document CCCC Online Writing Instruction, 2013, that online courses should prioritize writing and rhetorical concepts, not the technology itself (65). The authors note that online students appeared to spontaneously select more advanced technology than the f2f students, choices that Daniel Anderson argues inherently lead to more “enhanced critical thinking” and higher motivation (66).

The authors argue that their research supports two recommendations: first, the inclusion of IAs for multimodal learning; and second, the adoption by f2f instructors of multimodal activities and presentations, such as online discussion, videoed instruction, tutorials, and multiple examples. Face-to-face instructors, in this view, should try to emulate more nearly the “archival and nonlinear nature of the online course” (66). The authors call for further exploration of their contention that “student learning is indeed different within online and f2f multimodal courses,” based on their findings at the University of New Mexico (67).


by

Anderson et al. Contributions of Writing to Learning. RTE, Nov. 2015. Posted 12/17/2015.

Anderson, Paul, Chris M. Anson, Robert M. Gonyea, and Charles Paine. “The Contributions of Writing to Learning and Development: Results from a Large-Scale, Multi-institutional Study.” Research in the Teaching of English 50.2 (2015): 199-235. Print

Note: The study referenced by this summary was reported in Inside Higher Ed on Dec. 4, 2015. My summary may add some specific details to the earlier article and may clarify some issues raised in the comments on that piece. I invite the authors and others to correct and elaborate on my report.

Paul Anderson, Chris M. Anson, Robert M. Gonyea, and Charles Paine discuss a large-scale study designed to reveal whether writing instruction in college enhances student learning. They note widespread belief both among writing professionals and other stakeholders that including writing in curricula leads to more extensive and deeper learning (200), but contend that the evidence for this improvement is not consistent (201-02).

In their literature review, they report on three large-scale studies that show increased student learning in contexts rich in writing instruction. These studies concluded that the amount of writing in the curriculum improved learning outcomes (201). However, these studies contrast with the varied results from many “small-scale, quasi-experimental studies that examine the impact of specific writing interventions” (200).

Anderson et al. examine attempts to perform meta-analyses across such smaller studies to distill evidence regarding the effects of writing instruction (202). They postulate that these smaller studies often explore such varied practices in so many diverse environments that it is hard to find “comparable studies” from which to draw conclusions; the specificity of the interventions and the student populations to which they are applied make generalization difficult (203).

The researchers designed their investigation to address the disparity among these studies by searching for positive associations between clearly designated best practices in writing instruction and validated measures of student learning. In addition, they wanted to know whether the effects of writing instruction that used these best practices differed from the effects of simply assigning more writing (210). The interventions and practices they tested were developed by the Council of Writing Program Administrators (CWPA), while the learning measures were those used in the National Survey of Student Engagement (NSSE). This collaboration resulted from a feature of the NSSE in which institutions may form consortia to “append questions of specific interest to the group” (206).

Anderson et al. note that an important limitation of the NSSE is its reliance on self-report data, but they contend that “[t]he validity and reliability of the instrument have been extensively tested” (205). Although the institutions sampled were self-selected and women, large institutions, research institutions, and public schools were over-represented, the authors believe that the overall diversity and breadth of the population sampled by the NSSE/CWPA collaboration, encompassing more than 70,000 first-year and senior students, permits generalization that has not been possible with more narrowly targeted studies (204).

The NSSE queries students on how often they have participated in pedagogic activities that can be linked to enhanced learning. These include a wide range of practices such as service-learning, interactive learning, “institutionally challenging work” such as extensive reading and writing; in addition, the survey inquires about campus features such as support services and relationships with faculty as well as students’ perceptions of the degree to which their college experience led to enhanced personal development. The survey also captures demographic information (205-06).

Chosen as dependent variables for the joint CWPA/NSSE study were two NSSE scales:

  • Deep Approaches to Learning, which encompassed three subscales, Higher-Order Learning, Integrative Learning, and Reflective Learning. This scale focused on activities related to analysis, synthesis, evaluation, combination of diverse sources and perspectives, and awareness of one’s own understanding of information (211).
  • Perceived Gains in Learning and Development, which involved subscales of Practical Competence such as enhanced job skills, including the ability to work with others and address “complex real-world problems”; Personal and Social Development, which inquired about students’ growth as independent learners with “a personal code of values and ethics” able to “contribut[e] to the community”; and General Education Learning, which includes the ability to “write and speak clearly and effectively, and to think critically and analytically” (211).

The NSSE also asked students for a quantitative estimate of how much writing they actually did in their coursework (210). These data allowed the researchers to separate the effects of simply assigning more writing from those of employing different kinds of writing instruction.

To test for correlations between pedagogical choices in writing instruction and practices related to enhanced learning as measured by the NSSE scales, the research team developed a “consensus model for effective practices in writing” (206). Eighty CWPA members generated questions that were distilled to 27 divided into “three categories based on related constructs” (206). Twenty-two of these ultimately became part of a module appended to the NSSE that, like the NSSE “Deep Approaches to Learning” scale, asked students how often their coursework had included the specific activities and behaviors in the consensus model. The “three hypothesized constructs for effective writing” (206) were

  • Interactive Writing Processes, such as discussing ideas and drafts with others, including friends and faculty;
  • Meaning-Making Writing Tasks, such as using evidence, applying concepts across domains, or evaluating information and processes; and
  • Clear Writing Expectations, which refers to teacher practices in making clear to students what kind of learning an activity promotes and how student responses will be assessed. (206-07)

They note that no direct measures of student learning is included in the NSSE, nor are such measures included in their study (204). Rather, in both the writing module and the NSSE scale addressing Deep Approaches to Learning, students are asked to report on kinds of assignments, instructor behaviors and practices, and features of their interaction with their institutions, such as whether they used on-campus support services (205-06). The scale on Perceived Gains in Learning and Development asks students to self-assess (211-12).

Despite the lack of specific measures of learning, Anderson et al. argue that the curricular content included in the Deep Approaches to Learning scale does accord with content that has been shown to result in enhanced student learning (211, 231). The researchers argue that comparisons between the NSSE scales and the three writing constructs allow them to detect an association between the effective writing practices and the attitudes toward learning measured by the NSSE.

Anderson et al. provide detailed accounts of their statistical methods. In addition to analysis for goodness-of-fit, they performed “blocked hierarchical regressions” to determine how much of the variance in responses was explained by the kind of writing instruction reported versus other factors, such as demographic differences, participation in various “other engagement variables” such as service-learning and internships, and the actual amount of writing assigned (212). Separate regressions were performed on first-year students and on seniors (221).

Results “suggest[ed] that writing assignments and instructional practices represented by each of our three writing scales were associated with increased participation in Deep Approaches to Learning, although some of that relationship was shared by other forms of engagement” (222). Similarly, the results indicate that “effective writing instruction is associated with more favorable perceptions of learning and development, although other forms of engagement share some of that relationship” (224). In both cases, the amount of writing assigned had “no additional influence” on the variables (222, 223-24).

The researchers provide details of the specific associations among the three writing constructs and the components of the two NSSE scales. Overall, they contend, their data strongly suggest that the three constructs for effective writing instruction can serve “as heuristics that instructors can use when designing writing assignments” (230), both in writing courses and courses in other disciplines. They urge faculty to describe and research other practices that may have similar effects, and they advocate additional forms of research helpful in “refuting, qualifying, supporting, or refining the constructs” (229). They note that, as a result of this study, institutions can now elect to include the module “Experiences with Writing,” which is based on the three constructs, when students take the NSSE (231).

 


by 2 Comments

Hansen et al. Effectiveness of Dual Credit Courses. WPA Journal, Spring 2015. Posted 08/12/15.

Hansen, Kristine, Brian Jackson, Brett C. McInelly, and Dennis Eggett. “How Do Dual Credit Students Perform on College Writing Tasks After They Arrive on Campus? Empirical Data from a Large-Scale Study.” Journal of the Council of Writing Program Administrators 38.2 (2015): 56-92). Print.

Kristine Hansen, Brian Jackson, Brett C. McInelly, and Dennis Eggett conducted a study at Brigham Young University (BYU) to determine whether students who took a dual-credit/concurrent-enrollment writing course (DC/CE) fared as well on the writing assigned in a subsequent required general-education course as students who took or were taking the university’s first-year-writing course. With few exceptions, Hansen et al. concluded that the students who had taken the earlier courses for their college credit performed similarly to students who had not. However, the study raised questions about the degree to which taking college writing in high school, or for that matter, in any single class, adequately meets the needs of maturing student writers (79).

The exigence for the study was the proliferation of efforts to move college work into high schools, presumably to allow students to graduate faster and thus lower the cost of college, with some jurisdictions allowing students as young as fourteen to earn college credit in high school (58). Local, state, and federal policy makers all support and even “mandate” such opportunities (57), with rhetorical and financial backing from organizations and non-profits promoting college credit as a boon to the overall economy (81). Hansen et al. express concern that no uniform standards or qualifications govern these initiatives (58).

The study examined writing in BYU’s “American Heritage” (AH) course. In this course, which in September 2012 enrolled approximately half of the first-year class, students wrote two 900-word papers involving argument and research. They wrote the first paper in stages with grades and TA feedback throughout, while they relied on peer feedback and their understanding of an effective writing process, which they had presumably learned in the first assignment, for the second paper (64). Hansen et al. provide the prompts for both assignments (84-87).

The study consisted of several components. Students in the AH course were asked to sign a consent form; those who did so were emailed a survey about their prior writing instruction. Of these, 713 took the survey. From these 713 students,189 were selected (60-61). Trained raters using a holistic rubric with a 6-point scale read both essays submitted by these 189 students. The rubric pinpointed seven traits: “thesis, critical awareness, evidence, counter-arguments, organization, grammar and style, sources and citations” (65). A follow-up survey assessed students’ experiences writing the second paper, while focus groups provided additional qualitative information. Hansen et al. note that although only eleven students participated in the focus groups, the discussion provided “valuable insights into students’ motivations for taking pre-college credit options and the learning experiences they had” (65).

The 189 participants fell into five groups: those whose “Path to FYW Credit” consisted of AP scores; those who received credit for a DC/CE option; those planning to take FYW in the future; those taking it concurrently with AH; and those who had taken BYU’s course, many of them in the preceding summer (61, 63). Analysis reveals that the students studied were a good match in such categories as high-school GPA and ACT scores for the full BYU first-year population (62). However, strong high-school GPAs and ACT scores and evidence of regular one-on-one interaction with instructors (71), coupled with the description of BYU as a “private institution” with “very selective admission standards” (63) indicate that the students studied, while coming from many geographic regions, were especially strong students whose experiences could not be generalized to different populations (63, 82).

Qualitative results indicated that, for the small sample of students who participated in the focus group, the need to “get FYW out of the way” was not the main reason for choosing AP or DC/CE options. Rather, the students wanted “a more challenging curriculum” (69). These students reported good teaching practices; in contrast to the larger group taking the earlier survey, who reported writing a variety of papers, the students in the focus group reported a “literature[-]based” curriculum with an emphasis on timed essays and fewer research papers (69). Quotes from the focus-group students who took the FYW course from BYU reveal that they found it “repetitive” and “a good refresher,” not substantially different despite their having reported an emphasis on literary analysis in the high-school courses (72). The students attested that the earlier courses had prepared them well, although some expressed concerns about their comfort coping with various aspects of the first-year experience (71-72).

Three findings invited particular discussion (73):

  • Regardless of the writing instruction they had received, the students differed very little in their performance in the American Heritage class;
  • In general, although their GPAs and test scores indicated that they should be superior writers, the students scored in the center of the 6-point rubric scale, below expectations;
  • Scores were generally higher for the first essay than for the second.

The researchers argue that the first finding does not provide definitive evidence as to whether “FYW even matters” (73). They cite research by numerous scholars that indicates that the immediate effects of a writing experience are difficult to measure because the learning of growing writers does not exhibit a “tidy linear trajectory” (74). The FYW experience may trigger “steps backward” (Nancy Sommers, qtd. in Hansen et al. 72). The accumulation of new knowledge, they posit, can interfere with performance. Therefore, students taking FYW concurrently with AH might have been affected by taking in so much new material (74), while those who had taken the course in the summer had significantly lower GPAs and ACT scores (63). The authors suggest that these factors may have skewed the performance of students with FYW experience.

The second finding, the authors posit, similarly indicates students in the early-to-middle stages of becoming versatile, effective writers across a range of genres. Hansen et al. cite research on the need for a “significant apprenticeship period” in writing maturation (76). Students in their first year of college are only beginning to negotiate this developmental stage.

The third finding may indicate a difference in the demands of the two prompts, a difference in the time and energy students could devote to later assignments, or, the authors suggest, the difference in the feedback built into the two papers (76-77).

Hansen et al. recommend support for the NCTE position that taking a single course, especially at an early developmental stage, does not provide students an adequate opportunity for the kind of sustained practice across multiple genres required for meaningful growth in writing (77-80). Decisions about DC/CE options should be based on individual students’ qualifications (78); programs should work to include additional writing courses in the overall curriculum, designing these courses to allow students to build on skills initiated in AP, DC/CE, and FYW courses (79).

They further recommend that writing programs shift from promising something “new” and “different” to an emphasis on the recursive, nonlinear nature of writing, clarifying to students and other stakeholders the value of ongoing practice (80). Additionally, they recommend attention to the motives and forces of the “growth industry” encouraging the transfer of more and more college credit to high schools (80). The organizations sustaining this industry, they write, hope to foster a more literate, capable workforce. But the authors contend that speeding up and truncating the learning process, particularly with regard to a complex cognitive task like writing, undercut this aim (81-82) and do not, in fact, guarantee faster graduation (79). Finally, citing Richard Haswell, they call for more empirical, replicable studies of phenomena like the effects of DC/CE courses in order to document their impact across broad demographics (82).