Chapter 1 the study of collocations

i) Holistic Rating 
 
 
The free compositions were scored on a holistic scale of 1-100 which is a 
standardised and widely used scale compiled by Jacobs, Zinkgraf, Wormuth, 
Hartfiel and Hughey (1981) (see Appendix H for a list of the criteria for 
scoring).  Each composition was assessed by two raters.  The raters were native 
speakers of English and experienced English teachers. 
 
Each essay received two scores, one from each rater.  In cases where the 
two raters had more than ten points difference in their evaluation of a 
particular essay, the essay in question was scored by a third rater.  If the third 
rater gave a score that was half way between the previous two scores, e.g. Rater 
1 gave a 50, Rater 2 gave a 30, and Rater 3 gave a 40, then the third rater's score 
was counted, and the previous two ignored.  If the third rater gave a score that 
was the same as or closer to one of the previous two scores, then the third score 
and the closest other score were averaged and the estimated score was given to 
the essay, e.g. Rater 1 gave a 50, Rater 2 gave a 30, Rater 3 gave also a 30, the 
scores from Rater 2 and Rater 3 were averaged, while Rater 1 was ignored.  If in 
 
220

---

the previous case Rater 3 gave a 60, then Rater 1 and Rater 3 would be 
averaged, and Rater 2 would be ignored.  In 32 out of 275 essays (11.6%) where 
a third rater was needed, the score from the third rater had no more than nine 
points difference with at least one of the previous two scores.  Finally, each 
essay received a score on a scale 1-100 based on the average of the two ratings.