Methods
Participants
Participants in this study
were 278 eighth-grade language arts/English students enrolled in classes
during the spring of 2002 in a middle school in Pocatello, Idaho. All
students and their parents or guardians gave written, informed consent
prior to being considered for this study. Of the students who participated,
51% were female and 49% were male. No students withdrew from the study
after the study began; however, due to absences from school because
of illness or other activities, complete data was obtained for only
226 students.
Instrumentation
Two assessment instruments
were used in this study: the State of Idaho’s Direct Writing Assessment
rubric (DWA) and a concept map rubric developed by the Center of Electronic
Studying at the University of Oregon (Ditson et al., 2001). DWA is a
holistic scoring procedure using a five-point scoring standard. A rating
of five, four, or three indicate that a student is writing appropriately
at the grade level and a score of two, one, or zero indicates the writing
sample is below grade level (Idaho Department of Education, 1999).
Students’ concept
maps were assessed quantitatively and qualitatively. First, the number
of ideas (e.g., reasons, examples) in each student’s map
was counted. Second, the students’ maps were scored qualitatively
using a concept-map rubric developed by Ditson et al., 2001. Three dimensions
of the quality of the students’ concept maps were evaluated: (a)
the quality of the thesis statement, (b) the quality of the attention
getters, and (c) the total argument quality. The quality of the thesis
statements and attention getters for the topic were rated using a five-point
scale (0 = not valid, 1 = low, to 5 = high). Argument quality was determined
by rating the relationship between reasons and examples for each argument
node in the students’ concept maps using a 5-point scoring scale
(0 = not valid, 1 = low, to 5 = high). Adding the argument quality scores,
the thesis statement score, and the attention getter score generated
a total quality score for each concept map.
Design
The design for this study
was a 2 (computer-based versus paper-and-pencil concept mapping) by
2 (teacher) by 7 (class) nested factorial design, with teacher and class
serving as nested factors. Four teachers were randomly assigned to either
experimental or control groups together with their intact classes of
students. Under each method, the first teacher taught two classes, and
the second teacher taught 5 classes. Because both teachers and classes
were considered to be interchangeable with similar teachers and classes,
the effects of teacher and class were treated as random effects in this
design, whereas the effect for concept mapping methods was treated as
a fixed effect. Pre- and postassessments on persuasive writing were
administered to all participants. Preassessment scores were used as
a covariate in the statistical analyses of the postassessment scores.
According to McMillan (2000), this design was appropriate for this type
of study and best suited when participants are in intact classes.
The quality of the concept
maps generated by the students in the two concept-mapping prewriting
conditions was investigated using the same nested factorial design (with
no covariate). Separate analyses were conducted for the two concept-mapping
measures. In addition, stepwise linear regression analysis was used
to determine whether the mapping scores predicted the students’
persuasive writing scores on the postassessment.
Scoring
All the raters of the students’
persuasive essays were middle school language arts/English teachers
working in the targeted school. Each received training in the use of
the DWA to score student writing assignments. They also received specific
training in scoring persuasive writing samples. Each language arts teacher
scored his/her own students’ persuasive writing essays for the
pre- and postassessments. In addition to the teachers’ scores,
a second rater, a retired English teacher from the target school, was
asked to score randomly selected samples to assess inter-rater reliability.
The reliability coefficient for scoring of the persuasive writing preassessment
was r = .70 (n = 78), p < .001, and the reliability coefficient for
scoring of the persuasive writing postassessment was r = .80 (n = 76),
p < .001. Both coefficients indicate sufficient interrater agreement
for the purposes of this study. Thus, the four teachers’ scores
were used in all analyses.
The students’ concept
maps were scored independently by two English-major graduate students.
Both of these raters had teaching experience in college level writing
classes. The two raters received three hours of training from the researcher.
Training consisted of two sessions. In the first session, the researcher
introduced the components of concept maps and the rubric used for the
study. During the second session, the raters were trained to score concept
maps using the rubric. Six concept maps were randomly selected from
both control and experimental groups. The raters scored the selected
maps independently. First, ideas in a given map were identified and
counted. Second, the quality of thesis statement and attention getter
was validated based on 5-point scoring scale. Third, the quality of
relationship between reasons and examples for each argument node were
scored. After they finished scoring each map, the two raters compared
their results and discussed their viewpoints on the ideas, thesis statement,
attention getter, and argument quality scores to maximize their agreement
in scoring.
The interrater agreement
for the concept-map scores was computed for the two raters using Pearson
correlations. The interrater reliability coefficients were: r = .98
for the number of ideas, and r = .87 for the total quality scores. These
reliability coefficients were higher than those obtained by Kessler,
Ditson, Anderson-Inman, and Morris (1996) in their two previous investigations
of the interrater reliability of this concept map scoring system. Hence,
the interrater reliability was sufficient for the purposes of this study.
The second rater’s scores were selected for all the analyses.
Procedures
This study took one and
one-half months. The four steps that established the sequence and delivery
of this study were: preinstruction, preassessment, writing instruction,
and postassessment.
Preinstruction.
Training in the use of computer-based concept mapping was conducted
for both the teachers and students in the experimental group to maximize
the treatment fidelity of this study. Teachers in the experimental group
received one hour of training in the use of the concept mapping software,
Inspiration, from the researcher. In addition to learning the
technical skills of constructing computer-based concept maps, teachers
were trained to integrate a persuasive concept map template for their
prewriting instruction. After receiving training in the use of concept
mapping software, teachers then trained their students to use the computer-based
concept mapping software during two periods of class time (90 minutes).
During the time of training the researcher was in the classroom in case
the teachers needed technical assistance in the use of the software.
Preassessment.
All students wrote their persuasive essays in 90 minutes, which spanned
two class periods. The essays were based on a prompt that was mutually
agreed upon by all four of the language arts teachers. Students in both
groups responded to the writing task using paper-and-pencil. The writing
prompt for this task was: Are school uniforms required? Should all middle
school students be required to wear uniforms? Why or why not? Support
your position with examples, explanations, and details.
Writing Instruction.
This third step provided time for students to build familiarity and
skills with the concept mapping prewriting strategy used in the classroom
(Jonassen, Reeves, Hong, Harvey, & Peters, 1997). The teachers in
the experimental group used computer-based concept mapping as a prewriting
activity in writing instruction, whereas the teachers in the control
group used hand-drawn concept mapping as a prewriting strategy for persuasive
writing instruction.
After receiving instruction
in using the concept mapping prewriting strategy, the students in the
treatment group then constructed their own concept maps on computers
using Inspiration software. The control group did not receive
the computer-based concept mapping prewriting instruction, but rather
produced their concept maps using paper-and-pencil. Students in both
groups were asked to construct and elaborate their concept maps based
on a persuasive concept map template (see Figure 1). Once the concept
maps were created, they were used to complete individual persuasive
writing tasks. All students in both groups were afforded the opportunity
to practice this concept mapping prewriting strategy for two or three
persuasive writing tasks.
Figure 1. Persuasive concept
map format.
The writing instruction
in both treatment and control groups was identical. In other words,
the process-oriented writing (i.e., the skills needed to write a persuasive
essay) and the instructional materials were the same, except for the
difference in the tool used for the prewriting activities (computer-based
versus paper-and-pencil concept mapping).
Postassessment.
The administration of the post assessment was similar to that of preassessment.
The writing prompt, which was created by all the four language arts
teachers, was given to all the students. Again, students were given
90 minutes to complete the task. Students in the experimental group
constructed their concept maps using Inspiration for the writing
task and wrote their essays using Microsoft Word on computers.
On average, the students in the computer-based mapping condition took
M = 25.7 (SD = 9.3) minutes to complete their concept maps. Students
in the control group constructed their concept maps for the writing
task and wrote their essays by paper-and-pencil. On average, the students
in the control group took M = 21.5 (SD = 13.2) minutes to complete their
concept maps. The writing prompt was as follows: Argue for one change
in your school/community/or country. For example: School-open campus,
moving 6th grade up to middle school, activity period, electives; community-levy,
taxes, curfews, activities for teens; country-war on Iraq, NASA, cloning,
sports utility vehicles.
