Automated Essay Scoring

Studies in Social Sciences have revealed that when people evaluate someone else, their eval- uations often reflect their biases. As a re- sult, rater bias may introduce highly subjec- tive factors that make their evaluations inaccu- rate. This may affect automated essay scoring models in many ways, as these models are typ- ically designed to model (potentially biased) essay raters. While there is sizeable literature on rater effects in general settings, it remains unknown how rater bias affects automated es- say scoring. To this end, we present a new annotated corpus containing essays and their respective scores. Different from existing cor- pora, our corpus also contains comments pro- vided by the raters in order to ground their scores. We present features to quantify rater bias based on their comments, and we found that rater bias plays an important role in auto- mated essay scoring. We investigated the ex- tent to which rater bias affects models based on hand-crafted features. Finally, we propose to rectify the training set by removing essays associated with potentially biased scores while learning the scoring model.