|
Sign In to gain access to subscriptions and/or personal tools.
|
Psychometrics is Datametrics: the Test is not Reliable
Bruce Thompson
Texas A&M University and Baylor College of Medicine
Tammi Vacha-Haase
Colorado State Universitytvh{at}lamar.colostate.edu.
The present article responds to selected criticisms of some EPM editorial policies and Vacha-Haase’s "reliability generalization" meta-analytic methods. However, the treatment is more broadly a manifesto regarding the nature of score reliability and what are reasonable expectations for psychometric reporting practices in substantive inquiries. The consequences of misunderstandings of score reliability are explored. It is suggested that paradigmatic misconceptions regarding psychometric issues feed into a spiral of presumptions that measurement training is unnecessary for doctoral students, which then in turn further reinforces misunderstandings of score integrity issues.
Educational and Psychological Measurement, Vol. 60, No. 2,
174-195 (2000)
DOI: 10.1177/0013164400602002
 CiteULike  Complore  Connotea  Del.icio.us  Digg  Reddit  Technorati  Twitter What's this?
This article has been cited by other articles:
|
|
|
|
 
J. L. Romano and J. D. Kromrey
What Are the Consequences If the Assumption of Independent Observations Is Violated in Reliability Generalization Meta-Analysis Studies?
Educational and Psychological Measurement,
June 1, 2009;
69(3):
404 - 428.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. F. Dedrick, J. M. Ferron, M. R. Hess, K. Y. Hogarty, J. D. Kromrey, T. R. Lang, J. D. Niles, and R. S. Lee
Multilevel Modeling: A Review of Methodological Issues and Applications
Review of Educational Research,
March 1, 2009;
79(1):
69 - 102.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
M. Vassar and W. Hale
Reliability Reporting Across Studies Using the Buss Durkee Hostility Inventory
J Interpers Violence,
January 1, 2009;
24(1):
20 - 37.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. X. Tan, R. F. Dedrick, and K. Marfo
Factor Structure and Clinical Implications of Child Behavior Checklist/1.5-5 Ratings in a Sample of Girls Adopted from China
J. Pediatr. Psychol.,
August 1, 2007;
32(7):
807 - 818.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
A. Li and J. Bagger
The Balanced Inventory of Desirable Responding (BIDR): A Reliability Generalization Study
Educational and Psychological Measurement,
June 1, 2007;
67(3):
525 - 544.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. M. Graham
Congeneric and (Essentially) Tau-Equivalent Estimates of Score Reliability: What They Are and How to Use Them
Educational and Psychological Measurement,
December 1, 2006;
66(6):
930 - 944.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. E. Helms, K. T. Henze, T. L. Sass, and V. A. Mifsud
Treating Cronbach's Alpha Reliability Coefficients as Data in Counseling Research
The Counseling Psychologist,
September 1, 2006;
34(5):
630 - 660.
[Abstract]
[PDF]
|
|
|
|
|
|
|
L. F. Leach, R. K. Henson, L. R. Odom, and L. S. Cagle
A Reliability Generalization Study of the Self-Description Questionnaire
Educational and Psychological Measurement,
April 1, 2006;
66(2):
285 - 304.
[Abstract]
[PDF]
|
|
|
|
|
|
|
C. T. Bruckner, P. J. Yoder, and R.A. McWilliam
Generalizability and Decision Studies: An Example Using Conversational Language Samples
Journal of Early Intervention,
January 1, 2006;
28(2):
139 - 153.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. J. De Ayala, D. J. Vonderharr-Carlson, and D. Kim
Assessing the Reliability of the Beck Anxiety Inventory Scores
Educational and Psychological Measurement,
October 1, 2005;
65(5):
742 - 756.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. S. Campbell, S. Pulos, M. Hogan, and F. Murry
Reliability Generalization of the Psychopathy Checklist Applied in Youthful Samples
Educational and Psychological Measurement,
August 1, 2005;
65(4):
639 - 656.
[Abstract]
[PDF]
|
|
|
|
|
|
|
M. E. Ross, M. Blackburn, and S. Forbes
Reliability Generalization of the Patterns of Adaptive Learning Survey Goal Orientation Scales
Educational and Psychological Measurement,
June 1, 2005;
65(3):
451 - 464.
[Abstract]
[PDF]
|
|
|
|
|
|
|
W.-C. Wang
Direct Estimation of Correlation as a Measure of Association Strength Using Multidimensional Item Response Models
Educational and Psychological Measurement,
December 1, 2004;
64(6):
937 - 955.
[Abstract]
[PDF]
|
|
|
|
|
|
|
W.-C. Wang and W. Chyi-In
Gain Score in Item Response Theory as an Effect Size Measure
Educational and Psychological Measurement,
October 1, 2004;
64(5):
758 - 780.
[Abstract]
[PDF]
|
|
|
|
|
|
|
C. K. Enders
The Impact of Missing Data on Sample Reliability Estimates: Implications for Reliability Reporting Practices
Educational and Psychological Measurement,
June 1, 2004;
64(3):
419 - 436.
[Abstract]
[PDF]
|
|
|
|
|
|
|
W.-C. Wang and H.-C. Chen
The Standardized Mean Difference within the Framework of Item Response Theory
Educational and Psychological Measurement,
April 1, 2004;
64(2):
201 - 223.
[Abstract]
[PDF]
|
|
|
|
|
|
|
B. S. Morgan, C. P. Buscemi, and V. P. Fajardo
Assessing Instruments in a Cuban American Population With Type 2 Diabetes Mellitus
J Transcult Nurs,
April 1, 2004;
15(2):
139 - 146.
[Abstract]
[PDF]
|
|
|
|
|
|
|
X. Fan and P. Yin
Examinee Characteristics and Score Reliability: An Empirical Investigation
Educational and Psychological Measurement,
June 1, 2003;
63(3):
357 - 368.
[Abstract]
[PDF]
|
|
|
|
|
|
|
S. N. Beretvas and D. A. Pastor
Using Mixed-Effects Models In Reliability Generalization Studies
Educational and Psychological Measurement,
February 1, 2003;
63(1):
75 - 95.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. G. Melancon
Reliability, Structure, and Correlates of Learning and Study Strategies Inventory Scores
Educational and Psychological Measurement,
December 1, 2002;
62(6):
1020 - 1027.
[Abstract]
[PDF]
|
|
|
|
|
|
|
D. M. Dimitrov
Reliability: Arguments for Multiple Perspectives and Potential Problems with Generalization across Studies
Educational and Psychological Measurement,
October 1, 2002;
62(5):
783 - 801.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. Vacha-Haase, R. K. Henson, and J. C. Caruso
Reliability Generalization: Moving toward Improved Understanding and Use of Score Reliability
Educational and Psychological Measurement,
August 1, 2002;
62(4):
562 - 569.
[Abstract]
[PDF]
|
|
|
|
|
|
|
S. N. Beretvas, J. L. Meyers, and W. L. Leite
A Reliability Generalization Study of the Marlowe-Crowne Social Desirability Scale
Educational and Psychological Measurement,
August 1, 2002;
62(4):
570 - 589.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. M. Capraro and M. M. Capraro
Myers-Briggs Type Indicator Score Reliability Across: Studies a Meta-Analytic Reliability Generalization Study
Educational and Psychological Measurement,
August 1, 2002;
62(4):
590 - 602.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. E. Nilsson, C. K. Schmidt, and W. D. Meek
Reliability Generalization: An Examination of the Career Decision-Making Self-Efficacy Scale
Educational and Psychological Measurement,
August 1, 2002;
62(4):
647 - 658.
[Abstract]
[PDF]
|
|
|
|
|
|
|
W. E. Hanson, K. T. Curry, and D. L. Bandalos
Reliability Generalization of Working Alliance Inventory Scale Scores
Educational and Psychological Measurement,
August 1, 2002;
62(4):
659 - 673.
[Abstract]
[PDF]
|
|
|
|
|
|
|
G. G. Lane, A. E. White, and R. K. Henson
Expanding Reliability Generalization Methods with KR-21 Estimates an RG Study of the Coopersmith Self-Esteem Inventory
Educational and Psychological Measurement,
August 1, 2002;
62(4):
685 - 711.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. K. Henson and D.-Y. Hwang
Variability and Prediction of Measurement Error in Kolb's Learning Style Inventory Scores a Reliability Generalization Study
Educational and Psychological Measurement,
August 1, 2002;
62(4):
712 - 727.
[Abstract]
[PDF]
|
|
|
|
|
|
|
H. K. Deditius-Island and J. C. Caruso
An Examination of the Reliability of Scores from Zuckerman's Sensation Seeking Scales, Form V
Educational and Psychological Measurement,
August 1, 2002;
62(4):
728 - 734.
[Abstract]
[PDF]
|
|
|
|
|
|
|
B. Thompson and C. Cook
Stability of the Reliability of LibQual+TM Scores a Reliability Generalization Meta-Analysis Study
Educational and Psychological Measurement,
August 1, 2002;
62(4):
735 - 743.
[Abstract]
[PDF]
|
|
|
|
|
|
|
F. Baugh
Correcting Effect Sizes for Score Reliability: A Reminder that Measurement and Substantive Issues are Linked Inextricably
Educational and Psychological Measurement,
April 1, 2002;
62(2):
254 - 263.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. Vacha-Haase, C. R. Tani, L. R. Kogan, R. A. Woodall, and B. Thompson
Reliability Generalization: Exploring Reliability Variations on MMPI/MMPI-2 Validity Scale Scores
Assessment,
December 1, 2001;
8(4):
391 - 401.
[Abstract]
[PDF]
|
|
|
|
|
|
|
X. Fan and B. Thompson
Confidence Intervals for Effect Sizes: Confidence Intervals about Score Reliability Coefficients, Please: An EPM Guidelines Editorial
Educational and Psychological Measurement,
August 1, 2001;
61(4):
517 - 531.
[Abstract]
[PDF]
|
|
|
|
|
|
|
J. C. Caruso, K. Witkiewitz, A. Belcourt-Dittloff, and J. D. Gottlieb
Reliability of Scores from the Eysenck Personality Questionnaire: A Reliability Generalization Study
Educational and Psychological Measurement,
August 1, 2001;
61(4):
675 - 689.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. A. Charter
It is Time to Bury the Spearman-Brown "Prophecy" Formula for Some Common Applications
Educational and Psychological Measurement,
August 1, 2001;
61(4):
690 - 696.
[Abstract]
[PDF]
|
|
|
|
|
|
|
C. Cook, F. Heath, R. L. Thompson, and B. Thompson
Score Reliability in Webor Internet-Based Surveys: Unnumbered Graphic Rating Scales versus Likert-Type Scales
Educational and Psychological Measurement,
August 1, 2001;
61(4):
697 - 706.
[Abstract]
[PDF]
|
|
|
|
|
|
|
M. M. Capraro, R. M. Capraro, and R. K. Henson
Measurement Error of Scores on the Mathematics Anxiety Rating Scale across Studies
Educational and Psychological Measurement,
June 1, 2001;
61(3):
373 - 386.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. K. Henson, L. R. Kogan, and T. Vacha-Haase
A Reliability Generalization Study of the Teacher Efficacy Scale and Related Instruments
Educational and Psychological Measurement,
June 1, 2001;
61(3):
404 - 420.
[Abstract]
[PDF]
|
|
|
|
|
|
|
D. H. Atkins, K. T. Kelly, and G. S. Morrison
Development of the Child Evaluation Measure: An Assessment of Children's Learning across Disciplines and in Multiple Contexts
Educational and Psychological Measurement,
June 1, 2001;
61(3):
505 - 511.
[Abstract]
[PDF]
|
|
|
|
|
|
|
S. A. Vidal-Brown and B. Thompson
The Career Assessment Diagnostic Inventory: A New Career Indecision Assessment Tool
Journal of Career Assessment,
May 1, 2001;
9(2):
185 - 202.
[Abstract]
[PDF]
|
|
|
|
|
|
|
R. C. Arnau, R. L. Thompson, and C. Cook
Do Different Response Formats Change the Latent Structure of Responses? an Empirical Investigation Using Taxometric Analysis
Educational and Psychological Measurement,
February 1, 2001;
61(1):
23 - 44.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. Vacha-Haase, L. R. Kogan, C. R. Tani, and R. A. Woodall
Reliability Generalization: Exploring Variation of Reliability Coefficients of MMPI Clinical Scales Scores
Educational and Psychological Measurement,
February 1, 2001;
61(1):
45 - 59.
[Abstract]
[PDF]
|
|
|
|
|
|
|
B. T. Gray
A Factor Analytic Study of the Substance Abuse Subtle Screening Inventory (SASSI)
Educational and Psychological Measurement,
February 1, 2001;
61(1):
102 - 118.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. Vacha-Haase, L. R. Kogan, and B. Thompson
Sample Compositions and Variabilities in Published Studies versus Those in Test Manuals: Validity of Score Reliability Inductions
Educational and Psychological Measurement,
August 1, 2000;
60(4):
509 - 522.
[Abstract]
[PDF]
|
|
|
|
|
|
|
T. P. Hogan, A. Benjamin, and K. L. Brezinski
Reliability Methods: A Note on the Frequency of Use of Various Types
Educational and Psychological Measurement,
August 1, 2000;
60(4):
523 - 531.
[Abstract]
[PDF]
|
|
|
|
|
|
|
K. C. Mittag and B. Thompson
Research news and Comment: A National Survey of AERA Members' Perceptions of Statistical Significance Tests and Other Statistical Issues
Educational Researcher,
May 1, 2000;
29(4):
14 - 20.
[PDF]
|
|
|
|
|
|
|
S. S. Sawilowsky
Reliability: Rejoinder to Thompson and Vacha-Haase
Educational and Psychological Measurement,
April 1, 2000;
60(2):
196 - 200.
[Abstract]
[PDF]
|
|
|
|
|
|
|
P. Snyder
Guidelines for Reporting Results of Group Quantitative Investigations
Journal of Early Intervention,
January 1, 2000;
23(3):
145 - 150.
[PDF]
|
|
|
|
|
