This Article Full Text (PDF) All Versions of this Article: 0013164407301546v1 68/1/25    most recent References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Ping Yin Articles by Sconing, J. Search for Related Content Social Bookmarking                         What's this?

Estimating Standard Errors of Cut Scores for Item Rating and Mapmark Procedures

A Generalizability Theory Approach

ACT, Inc., ping.yin{at}act.org

James Sconing

ACT, Inc.

Standard-setting methods are widely used to determine cut scores on a test that examinees must meet for a certain performance standard. Because standard setting is a measurement procedure, it is important to evaluate variability of cut scores resulting from the standard-setting process. Generalizability theory is used in this study to estimate standard errors of cut scores resulting from two standard-setting methods: item rating (Angoff-based) and mapmark (bookmark-based) methods. In this study, two different generalizability (G) study designs and four different decision (D) study designs were examined, and the impact of varying different aspects of the study design and universes of generalization was examined. Results suggest that cut scores were generally consistent for both methods. The first round standard setting contributed the most to the overall variability for the mapmark method. Also, it is clear that there is no one standard error associated with a certain cut score.

Key Words: standard setting • item rating method • mapmark method • generalizability theory

This version was published on February 1, 2008

Educational and Psychological Measurement, Vol. 68, No. 1, 25-41 (2008)
DOI: 10.1177/0013164407301546


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?