This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Janson, H. Articles by Olsson, U. Search for Related Content Social Bookmarking                         What's this?

A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges

University of Oslo, harald.janson{at}psykologi.uio.no

Ulf Olsson

Swedish University of Agricultural Sciences

This article addresses the problem of accounting overall multivariate chance-corrected interobserver agreement when targets have been rated by different sets of judges (not necessarily equal in number). The proposed approach builds on Janson and Olsson’s multivariate generalization of Cohen’s kappa but incorporates weighting for number of judges and applies an expression for expected disagreement suitable for the case with different judges. The authors suggest that the attractiveness of this approach to multivariate agreement measurement lies in the interpretability of the terms of expected and observed disagreement as average distances between observations, and that addressing agreement without regard to the covariance structure among variables has advantages in simplicity and interpretability. Correspondences to earlier approaches are noted, and application of the proposed measure is exemplified using hypothetical data sets.

Key Words: interrater reliability • multivariate analysis • rating • measurement

Educational and Psychological Measurement, Vol. 64, No. 1, 62-70 (2004)
DOI: 10.1177/0013164403260195


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				R. D. Brown and N. M. A. Hauenstein Interrater Agreement Reconsidered: An Alternative to the rwg Indices Organizational Research Methods, April 1, 2005; 8(2): 165 - 184. [Abstract] [PDF]