In proceedings ASEE Annual Conf., St. Louis, June 2000.
Comparing the reliability of two peer evaluation instruments
Matthew W. Ohland and Richard A. Layton

This paper presents an analysis of student peer evaluations in project teams to compare the reliability of two different evaluation procedures. The project teams consist of junior-level students in a mechanical engineering design course taught by Layton for five semesters in 1997, 1998, and 1999.

The peer-evaluation instruments were used by students to evaluate their teammatesí contributions to the teamís deliverables-oral and written presentations of their solution to a technical design problem. The first instrument is an adaptation of the one advocated by Brown, in which students use a prescribed list of terms such as "excellent," "very good," "satisfactory," and so forth. The second form, by Layton, asked students to assign a numerical rating (from 0 to 5) to 10 different aspects of contribution to the team.

Analysis of variance was used to study the reliability of each of the instruments, using a special form by Crocker and Algina to study inter-rater reliability. The similarity of the reliability coefficients of the two instruments (p=0.34 for Brownís instrument and p=0.41 for Laytonís instrument) strengthens the assumption made in the first study---that data from the two instruments are similar enough to be normalized for comparison. At the same time, the higher reliability of Laytonís instrument lends credence to Layton and Ohlandís conclusion that focusing on identified behavioral characteristics of good teamwork (as Laytonís instrument does) can improve peer evaluation. Laytonís instrument accomplishes this to an extent, yielding a modest improvement in reliability. More focused attempts to define teamwork success behaviorally, such as the modification of Brownís instrument by Kaufman et al., may yield further improvements in reliability. The overall reliability of the two instruments validates such instruments as repeated measures of a consistent trait.

©2000 American Society for Engineering Education.

Return to list of publications

Richard A. Layton
Last modified: 14 Jul 02