|
Sign In to gain access to subscriptions and/or personal tools.
|
Reliability Generalization: Exploring Variance in Measurement Error Affecting Score Reliability Across Studies
Tammi Vacha-Haase
Western Michigan University, vacha-haase{at}wmich.edu
Because tests are not reliable, it is important to explore score reliability in virtually all studies. The present article proposes and illustrates a new method-reliability generalization-that can be used in a meta-analysis application similar to validity generalization. Reliability generalization characterizes (a) the typical reliability of scores for a given test across studies, (b) the amount of variability in reliability coefficients for given measures, and (c) the sources of variability in reliability coefficients across studies. The use of reliability generalization is illustrated here by analyzing 87 reliability coefficients reported for the two scales of the Bem Sex Role Inventory (BSRI).
Educational and Psychological Measurement, Vol. 58, No. 1,
6-20 (1998)
DOI: 10.1177/0013164498058001002
 CiteULike  Complore  Connotea  Del.icio.us  Digg  Reddit  Technorati  Twitter What's this?
This article has been cited by other articles:
|
|
|
|
 
J. L. Romano and J. D. Kromrey
What Are the Consequences If the Assumption of Independent Observations Is Violated in Reliability Generalization Meta-Analysis Studies?
Educational and Psychological Measurement,
June 1, 2009;
69(3):
404 - 428.
[Abstract]
[PDF]
|
|
|
|
|
|
|
K. M. Kieffer and R. J. Reese
Measurement of Test and Study Worry and Emotionality in College Students: A Psychometric Evaluation of the Test and Study Attitudes Inventory
Educational and Psychological Measurement,
April 1, 2009;
69(2):
303 - 321.
[Abstract]
[PDF]
|
|
|
|
|
|
|
Q.-L. Huynh, R. T. Howell, and V. Benet-Martinez
Reliability of Bidimensional Acculturation Scores: A Meta-Analysis
Journal of Cross-Cultural Psychology,
March 1, 2009;
40(2):
256 - 274.
[Abstract]
[PDF]
|
|
|
|
|
|
|
K. R. Rexrode, S. Petersen, and S. O'Toole
The Ways of Coping Scale: A Reliability Generalization Study
Educational and Psychological Measurement,
April 1, 2008;
68(2):
262 - 280.
[Abstract]
[PDF]
|
|
|
|
|
|
|
S. N. Beretvas, M.-A. Suizzo, J. A. Durham, and L. M. Yarnell
A Reliability Generalization Study of Scores on Rotter's and Nowicki-Strickland's Locus of Control Scales
Educational and Psychological Measurement,
February 1, 2008;
68(1):
97 - 119.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. T. Howell and A. L. Shields
The File Drawer Problem in Reliability Generalization: A Strategy to Compute a Fail-Safe N With Reliability Coefficients
Educational and Psychological Measurement,
February 1, 2008;
68(1):
120 - 128.
[Abstract]
[PDF]
|
|
|
|
|
|
|
C. S. Miller, A. L. Shields, D. Campfield, K. A. Wallace, and R. D. Weiss
Substance Use Scales of the Minnesota Multiphasic Personality Inventory: An Exploration of Score Reliability Via Meta-Analysis
Educational and Psychological Measurement,
December 1, 2007;
67(6):
1052 - 1065.
[Abstract]
[PDF]
|
|
|
|
|
|
|
C. Mason, R. Allam, and M. T. Brannick
How to Meta-Analyze Coefficient-of-Stability Estimates: Some Recommendations Based on Monte Carlo Studies
Educational and Psychological Measurement,
October 1, 2007;
67(5):
765 - 783.
[Abstract]
[PDF]
|
|
|
|
|
|
|
A. Li and J. Bagger
The Balanced Inventory of Desirable Responding (BIDR): A Reliability Generalization Study
Educational and Psychological Measurement,
June 1, 2007;
67(3):
525 - 544.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. E. Helms, K. T. Henze, T. L. Sass, and V. A. Mifsud
Treating Cronbach's Alpha Reliability Coefficients as Data in Counseling Research
The Counseling Psychologist,
September 1, 2006;
34(5):
630 - 660.
[Abstract]
[PDF]
|
|
|
|
|
|
|
C. M. Hellman, D. R. Fuqua, and J. Worley
A Reliability Generalization Study on the Survey of Perceived Organizational Support: The Effects of Mean Age and Number of Items on Score Reliability
Educational and Psychological Measurement,
August 1, 2006;
66(4):
631 - 642.
[Abstract]
[PDF]
|
|
|
|
|
|
|
L. F. Leach, R. K. Henson, L. R. Odom, and L. S. Cagle
A Reliability Generalization Study of the Self-Description Questionnaire
Educational and Psychological Measurement,
April 1, 2006;
66(2):
285 - 304.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. J. De Ayala, D. J. Vonderharr-Carlson, and D. Kim
Assessing the Reliability of the Beck Anxiety Inventory Scores
Educational and Psychological Measurement,
October 1, 2005;
65(5):
742 - 756.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. S. Campbell, S. Pulos, M. Hogan, and F. Murry
Reliability Generalization of the Psychopathy Checklist Applied in Youthful Samples
Educational and Psychological Measurement,
August 1, 2005;
65(4):
639 - 656.
[Abstract]
[PDF]
|
|
|
|
|
|
|
M. E. Ross, M. Blackburn, and S. Forbes
Reliability Generalization of the Patterns of Adaptive Learning Survey Goal Orientation Scales
Educational and Psychological Measurement,
June 1, 2005;
65(3):
451 - 464.
[Abstract]
[PDF]
|
|
|
|
|
|
|
K. Dunderdale, D. R. Thompson, J. N.V. Miles, S. F. Beer, and G. Furze
Quality-of-life measurement in chronic heart failure: do we take account of the patient perspective?
Eur J Heart Fail,
June 1, 2005;
7(4):
572 - 582.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
N. O'rourke
Reliability Generalization of Responses by Care Providers to the Center for Epidemiologic Studies-Depression Scale
Educational and Psychological Measurement,
December 1, 2004;
64(6):
973 - 990.
[Abstract]
[PDF]
|
|
|
|
|
|
|
C. K. Enders
The Impact of Missing Data on Sample Reliability Estimates: Implications for Reliability Reporting Practices
Educational and Psychological Measurement,
June 1, 2004;
64(3):
419 - 436.
[Abstract]
[PDF]
|
|
|
|
|
|
|
A. L. Shields and J. C. Caruso
A Reliability Induction and Reliability Generalization Study of the Cage Questionnaire
Educational and Psychological Measurement,
April 1, 2004;
64(2):
254 - 270.
[Abstract]
[PDF]
|
|
|
|
|
|
|
W. Van Den Noortgate and P. Onghena
Multilevel Meta-Analysis: A Comparison with Traditional Meta-Analytical Procedures
Educational and Psychological Measurement,
October 1, 2003;
63(5):
765 - 790.
[Abstract]
[PDF]
|
|
|
|
|
|
|
A. L. Shields and J. C. Caruso
Reliability Generalization of the Alcohol Use Disorders Identification Test
Educational and Psychological Measurement,
June 1, 2003;
63(3):
404 - 413.
[Abstract]
[PDF]
|
|
|
|
|
|
|
S. N. Beretvas and D. A. Pastor
Using Mixed-Effects Models In Reliability Generalization Studies
Educational and Psychological Measurement,
February 1, 2003;
63(1):
75 - 95.
[Abstract]
[PDF]
|
|
|
|
|
|
|
K. M. Kieffer and R. J. Reese
A Reliability Generalization Study of the Geriatric Depression Scale
Educational and Psychological Measurement,
December 1, 2002;
62(6):
969 - 994.
[Abstract]
[PDF]
|
|
|
|
|
|
|
D. M. Dimitrov
Reliability: Arguments for Multiple Perspectives and Potential Problems with Generalization across Studies
Educational and Psychological Measurement,
October 1, 2002;
62(5):
783 - 801.
[Abstract]
[PDF]
|
|
|
|
|
|
|
A. Cooper-Hakim and C. Viswesvaran
A Meta-Analytic Review of the MaCandrew Alcoholism Scale
Educational and Psychological Measurement,
October 1, 2002;
62(5):
818 - 829.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. Vacha-Haase, R. K. Henson, and J. C. Caruso
Reliability Generalization: Moving toward Improved Understanding and Use of Score Reliability
Educational and Psychological Measurement,
August 1, 2002;
62(4):
562 - 569.
[Abstract]
[PDF]
|
|
|
|
|
|
|
S. N. Beretvas, J. L. Meyers, and W. L. Leite
A Reliability Generalization Study of the Marlowe-Crowne Social Desirability Scale
Educational and Psychological Measurement,
August 1, 2002;
62(4):
570 - 589.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. M. Capraro and M. M. Capraro
Myers-Briggs Type Indicator Score Reliability Across: Studies a Meta-Analytic Reliability Generalization Study
Educational and Psychological Measurement,
August 1, 2002;
62(4):
590 - 602.
[Abstract]
[PDF]
|
|
|
|
|
|
|
L. L. B. Barnes, D. Harp, and W. S. Jung
Reliability Generalization of Scores on the Spielberger State-Trait Anxiety Inventory
Educational and Psychological Measurement,
August 1, 2002;
62(4):
603 - 618.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. J. Reese, K. M. Kieffer, and B. K. Briggs
A Reliability Generalization Study of Select Measures of Adult Attachment Style
Educational and Psychological Measurement,
August 1, 2002;
62(4):
619 - 646.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. E. Nilsson, C. K. Schmidt, and W. D. Meek
Reliability Generalization: An Examination of the Career Decision-Making Self-Efficacy Scale
Educational and Psychological Measurement,
August 1, 2002;
62(4):
647 - 658.
[Abstract]
[PDF]
|
|
|
|
|
|
|
W. E. Hanson, K. T. Curry, and D. L. Bandalos
Reliability Generalization of Working Alliance Inventory Scale Scores
Educational and Psychological Measurement,
August 1, 2002;
62(4):
659 - 673.
[Abstract]
[PDF]
|
|
|
|
|
|
|
K. A. Wallace and A. J. Wheeler
Reliability Generalization of the Life Satisfaction Index
Educational and Psychological Measurement,
August 1, 2002;
62(4):
674 - 684.
[Abstract]
[PDF]
|
|
|
|
|
|
|
G. G. Lane, A. E. White, and R. K. Henson
Expanding Reliability Generalization Methods with KR-21 Estimates an RG Study of the Coopersmith Self-Esteem Inventory
Educational and Psychological Measurement,
August 1, 2002;
62(4):
685 - 711.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. K. Henson and D.-Y. Hwang
Variability and Prediction of Measurement Error in Kolb's Learning Style Inventory Scores a Reliability Generalization Study
Educational and Psychological Measurement,
August 1, 2002;
62(4):
712 - 727.
[Abstract]
[PDF]
|
|
|
|
|
|
|
H. K. Deditius-Island and J. C. Caruso
An Examination of the Reliability of Scores from Zuckerman's Sensation Seeking Scales, Form V
Educational and Psychological Measurement,
August 1, 2002;
62(4):
728 - 734.
[Abstract]
[PDF]
|
|
|
|
|
|
|
B. Thompson and C. Cook
Stability of the Reliability of LibQual+TM Scores a Reliability Generalization Meta-Analysis Study
Educational and Psychological Measurement,
August 1, 2002;
62(4):
735 - 743.
[Abstract]
[PDF]
|
|
|
|
|
|
|
F. Baugh
Correcting Effect Sizes for Score Reliability: A Reminder that Measurement and Substantive Issues are Linked Inextricably
Educational and Psychological Measurement,
April 1, 2002;
62(2):
254 - 263.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. Vacha-Haase, C. R. Tani, L. R. Kogan, R. A. Woodall, and B. Thompson
Reliability Generalization: Exploring Reliability Variations on MMPI/MMPI-2 Validity Scale Scores
Assessment,
December 1, 2001;
8(4):
391 - 401.
[Abstract]
[PDF]
|
|
|
|
|
|
|
X. Fan and B. Thompson
Confidence Intervals for Effect Sizes: Confidence Intervals about Score Reliability Coefficients, Please: An EPM Guidelines Editorial
Educational and Psychological Measurement,
August 1, 2001;
61(4):
517 - 531.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. C. Caruso, K. Witkiewitz, A. Belcourt-Dittloff, and J. D. Gottlieb
Reliability of Scores from the Eysenck Personality Questionnaire: A Reliability Generalization Study
Educational and Psychological Measurement,
August 1, 2001;
61(4):
675 - 689.
[Abstract]
[PDF]
|
|
|
|
|
|
|
M. M. Capraro, R. M. Capraro, and R. K. Henson
Measurement Error of Scores on the Mathematics Anxiety Rating Scale across Studies
Educational and Psychological Measurement,
June 1, 2001;
61(3):
373 - 386.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. K. Henson, L. R. Kogan, and T. Vacha-Haase
A Reliability Generalization Study of the Teacher Efficacy Scale and Related Instruments
Educational and Psychological Measurement,
June 1, 2001;
61(3):
404 - 420.
[Abstract]
[PDF]
|
|
|
|
|
|
|
B. Thompson
Editor's Note on the "Colloquium on Effect Sizes: the Roles of Editors, Textbook Authors, and the Publication Manual"
Educational and Psychological Measurement,
April 1, 2001;
61(2):
211 - 212.
[PDF]
|
|
|
|
|
|
|
J. Sanchez-Meca and F. Marin-Martinez
Meta-Analysis of 2 x 2 Tables: Estimating a Common Risk Difference
Educational and Psychological Measurement,
April 1, 2001;
61(2):
249 - 276.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. Vacha-Haase, L. R. Kogan, C. R. Tani, and R. A. Woodall
Reliability Generalization: Exploring Variation of Reliability Coefficients of MMPI Clinical Scales Scores
Educational and Psychological Measurement,
February 1, 2001;
61(1):
45 - 59.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. Vacha-Haase, L. R. Kogan, and B. Thompson
Sample Compositions and Variabilities in Published Studies versus Those in Test Manuals: Validity of Score Reliability Inductions
Educational and Psychological Measurement,
August 1, 2000;
60(4):
509 - 522.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. J. Barnette
Effects of Stem and Likert Response Option Reversals on Survey Internal Consistency: If You Feel the Need, There is a Better Alternative to Using those Negatively Worded Stems
Educational and Psychological Measurement,
June 1, 2000;
60(3):
361 - 370.
[Abstract]
[PDF]
|
|
|
|
|
|
|
K. C. Mittag and B. Thompson
Research news and Comment: A National Survey of AERA Members' Perceptions of Statistical Significance Tests and Other Statistical Issues
Educational Researcher,
May 1, 2000;
29(4):
14 - 20.
[PDF]
|
|
|
|
|
|
|
S. S. Sawilowsky
Psychometrics versus Datametrics: Comment on Vacha-Haase's "Reliability Generalization" Method and Some Epm Editorial Policies
Educational and Psychological Measurement,
April 1, 2000;
60(2):
157 - 173.
[Abstract]
[PDF]
|
|
|
|
|
|
|
B. Thompson and T. Vacha-Haase
Psychometrics is Datametrics: the Test is not Reliable
Educational and Psychological Measurement,
April 1, 2000;
60(2):
174 - 195.
[Abstract]
[PDF]
|
|
|
|
|
|
|
P. Yin and X. Fan
Assessing the Reliability of Beck Depression Inventory Scores: Reliability Generalization across Studies
Educational and Psychological Measurement,
April 1, 2000;
60(2):
201 - 223.
[Abstract]
[PDF]
|
|
|
|
|
|
|
C. Viswesvaran and D. S. Ones
Measurement Error in "Big Five Factors" Personality Assessment: Reliability Generalization across Studies and Measures
Educational and Psychological Measurement,
April 1, 2000;
60(2):
224 - 235.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. C. Caruso
Reliability Generalization of the Neo Personality Scales
Educational and Psychological Measurement,
April 1, 2000;
60(2):
236 - 254.
[Abstract]
[PDF]
|
|
|
|
|
|
|
M. Martinussen and J. F. Bjornstad
Meta-Analysis Calculations Based on Independent and Nonindependent Cases
Educational and Psychological Measurement,
December 1, 1999;
59(6):
928 - 950.
[Abstract]
[PDF]
|
|
|
|
|
|
|
K. C. Mittag
Measuring the Jungian Personality Types of Hispanic High School Students
Assessment,
September 1, 1999;
6(3):
293 - 300.
[Abstract]
[PDF]
|
|
|
|
|
|
|
F. J. Kier and A. R. Buras
Perceived Affiliation with Family Member Roles: Validity and Reliability of Scores on the Children's Role Inventory
Educational and Psychological Measurement,
August 1, 1999;
59(4):
640 - 650.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. J. Barnette
Nonattending Respondent Effects on Internal Consistency of Self-Administered Surveys: A Monte Carlo Simulation Study
Educational and Psychological Measurement,
February 1, 1999;
59(1):
38 - 46.
[Abstract]
[PDF]
|
|
|
|
|
