This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Chang, L. Articles by Vos, H. J. Search for Related Content Social Bookmarking                         What's this?

Setting Standards and Detecting Intrajudge Inconsistency Using Interdependent Evaluation of Response Alternatives

Chinese University of Hong Kong, leichang{at}cuhk.edu.hkor

Wim J. Van Der Linden

University of Twente, w.j.vanderlinden{at}edte.utwente.nl

Hans J. Vos

University of Twente

This article introduces a new test-centered standard-setting method as well as a procedure to detect intrajudge inconsistency of the method. The standard-setting method that is based on interdependent evaluations of alternative responses has judges closely evaluate the process that examinees use to solve multiple-choice items. The new method is analyzed against existing methods, particularly the Nedelsky and Angoff methods. Empirical results from three different experiments confirm the hypothesis that standards set by the new method are higher than those of the Nedelsky but lower than those of the Angoff method. The procedure for detecting intrajudge inconsistency is based on residual diagnosis of the judgments, which makes it possible to identify the sources of inconsistencies in the items, response alternatives, and/or judges. An empirical application of the procedure in an experiment with the new standard-setting method suggests that the method is internally consistent and has also revealed an interesting difference between residuals for the correct and incorrect alternatives.

Key Words: standard setting • Angoff method • Nedelsky method • intrajudge inconsistency • judgmental item analysis • multiple-choice test • polytomous response models

Educational and Psychological Measurement, Vol. 64, No. 5, 781-801 (2004)
DOI: 10.1177/0013164404264847


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				F. J. Abad, J. Olea, and V. Ponsoda The Multiple-Choice Model: Some Solutions for Estimation of Parameters in the Presence of Omitted Responses Applied Psychological Measurement, May 1, 2009; 33(3): 200 - 221. [Abstract] [PDF]