This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Schuster, C. Articles by Smith, D. A. Search for Related Content Social Bookmarking                         What's this?

Estimating with a Latent Class Model the Reliability of Nominal Judgments Upon Which Two Raters Agree

Justus-Liebig-Universität Giessen

David A. Smith

University of Notre Dame

Because nominal-scale judgments cannot directly be aggregated into meaningful composites, the addition of a second rater is usually motivated by a desire to estimate the quality of a single rater's classifications rather than to improve reliability. When raters agree, the aggregation problem does not arise. Nevertheless, a proportion of this agreement is likely to have occurred by chance, so reliability issues remain even if the focus is on nominal judgments upon which raters concur. In this article, the reliability of nominal judgments upon which raters agree is addressed in the framework of a latent class model that includes a systematic agreement parameter. It is shown that if attention is limited to agreement cases, the value of this parameter increases as expected. Circumstances under which agreement case reliability is important are discussed.

Key Words: rater agreement • rater reliability • nominal judgments • latent class analysis

Educational and Psychological Measurement, Vol. 66, No. 5, 739-747 (2006)
DOI: 10.1177/0013164405285905


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?