Educational and Psychological Measurement current issue

Examining the Relationship Between Race-Based Differential Item Functioning and Item Difficulty

Scherbaum, C. A., Goldstein, H. W. — 2008-07-25

Recent research examining racial differences on standardized cognitive tests has focused on the impact of test item difficulty. Studies using data from the SAT and GRE have reported a correlation between item difficulty and differential item functioning (DIF) such that minority test takers are less likely than majority test takers to respond correctly to easy test items. The statistical techniques used and the effect sizes reported in these studies have been heavily criticized. This study addresses these criticisms by examining the relationship between item difficulty and DIF by using alternative statistical techniques based on item response theory and a different standardized test. The results replicate previous research and provide support for the generalizability of the findings.

Computer-Based and Paper-and-Pencil Administration Mode Effects on a Statewide End-of-Course English Test

Kim, D.-H., Huynh, H. — 2008-07-25

The current study compared student performance between paper-and-pencil testing (PPT) and computer-based testing (CBT) on a large-scale statewide end-of-course English examination. Analyses were conducted at both the item and test levels. The overall results suggest that scores obtained from PPT and CBT were comparable. However, at the content domain level, a rather large difference in the reading comprehension section suggests that reading comprehension test may be more affected by the test administration mode. Results from the confirmatory factor analysis suggest that the administration mode did not alter the construct of the test.

IRT-LR-DIF With Estimation of the Focal-Group Density as an Empirical Histogram

Woods, C. M. — 2008-07-25

Item response theory–likelihood ratio–differential item functioning (IRT-LR-DIF) is used to evaluate the degree to which items on a test or questionnaire have different measurement properties for one group of people versus another, irrespective of group-mean differences on the construct. Usually, the latent distribution is presumed normal for both groups, but previous research shows that results are biased if the true distribution is not approximately normal. This article introduces a variation of IRT-LR-DIF, called empirical histogram–differential item functioning (EH-DIF), in which the focal-group latent density is estimated simultaneously with the item parameters as an empirical histogram (EH). A simulation study shows that if the focal-group density is nonnormal, Type I error rates and focal-group estimates of the item parameters, mean, and SD are more accurate using EH-DIF than standard IRT-LR-DIF methods that presume normality. A pseudoempirical example is analyzed to illustrate EH-DIF.

Comparison of Multiple-Indicators, Multiple-Causes- and Item Response Theory-Based Analyses of Subgroup Differences

Willse, J. T., Goodman, J. T. — 2008-07-25

This research provides a direct comparison of effect size estimates based on structural equation modeling (SEM), item response theory (IRT), and raw scores. Differences between the SEM, IRT, and raw score approaches are examined under a variety of data conditions (IRT models underlying the data, test lengths, magnitude of group differences, and relative size of reference and focal groups). Results show that all studied methods perform similarly. All methods tend to underestimate effects as effect sizes become larger. SEM-based approaches to effect size estimation perform somewhat better at shorter test lengths, whereas approaches based on IRT and raw score perform somewhat better at longer test lengths. Although these differences between methods are detectable, they are small in magnitude.

A Generalizability Theory Approach to Standard Error Estimates for Bookmark Standard Settings

Lee, G., Lewis, D. M. — 2008-07-25

The bookmark standard-setting procedure is an item response theory–based method that is widely implemented in state testing programs. This study estimates standard errors for cut scores resulting from bookmark standard settings under a generalizability theory model and investigates the effects of different universes of generalization and error sources on standard errors. This study produced several notable results. First, different patterns of variance component estimates are found for different cut scores; therefore, researchers should estimate separate variance components for each cut score and use them to estimate corresponding standard errors. Second, different universes of generalization produce different standard error estimates; thus, policy makers should consider which universe is appropriate for the proposed use of cut scores. Third, participants and groups have nonnegligible effects on several error sources. To decrease the standard errors for cut scores, increasing the number of small groups seems more efficient than increasing the number of participants.

Latent Mean and Covariance Differences With Measurement Equivalence in College Students With Developmental Difficulties Versus the Wechsler Adult Intelligence Scale-III/Wechsler Memory Scale-III Normative Sample

2008-07-25

Intelligence tests are usually part of the assessment battery for the diagnosis of adults with learning disabilities (LD) and attention deficit hyperactivity disorder (ADHD). Professionals must ensure that inferences drawn from such test scores are equivalent across populations with and without disabilities. Examination of measurement equivalence provides a direct test of the hypothesis that the same set of latent variables underlies a set of test scores in different groups and metric relationships between observed scores and the corresponding latent variables are the same. The hypothesis of measurement equivalence was examined in two samples of college students: one sample with LD and one sample with ADHD. Scores on the third editions of the Wechsler Adult Intelligence and Memory Scales were compared with an age-matched subset of the conorming sample. Results supported the assumption of measurement equivalence but revealed marked differences across samples in latent variable variances and covariances and latent variable means.

A Psychometric Evaluation of Two Achievement Goal Inventories

Donnellan, M. B. — 2008-07-25

The properties of the achievement goal inventories developed by Grant and Dweck (2003) and Elliot and McGregor (2001) were evaluated in two studies with a total of 780 participants. A four-factor specification for the Grant and Dweck inventory did not closely replicate results published in their original report. In contrast, the structure of the Elliot and McGregor inventory was more or less replicable. In addition, the associations between the scales on both inventories were examined, and both inventories have scales that tap approach-related forms of performance and learning goals. However, a notable advantage of the shorter Elliot and McGregor inventory is that it also includes measures of the avoidance forms of these goals. In sum, the author concluded that the Elliot and McGregor inventory was a superior measure of achievement goal constructs on the basis of practical, psychometric, and theoretical considerations.

Parent Ratings Using the Chinese Version of the Parent Gifted Rating Scales-School Form Reliability and Validity for Chinese Students

Huijun Li, , Lee, D., Pfeiffer, S. I., Petscher, Y. — 2008-07-25

This study examined the reliability and validity of the scores of a Chinese-translated version of the Gifted Rating Scales–School Form (GRS-S) using parents as raters and explored the effects of gender and grade on the ratings. A total of 222 parents participated in the study and rated their child independently using the Chinese version of the Parent GRS-S (CVPGRS-S). Results indicate high reliability for parent rating scores and statistically significant correlations between CVPGRS-S scale scores and students' classroom academic achievement scores. The effect sizes of the relationships were moderate. Results revealed no statistically significant grade effect on any of the six CVPGRS-S scales; small but statistically significant differences in favor of females on the six CVPGRS-S scales were found. However, the effect size was small. Research results suggest that the Chinese-translated parent version of the GRS-S holds promise for use with Chinese parents.

Internalizing and Externalizing Behavior Problem Scores: Cross-Ethnic and Longitudinal Measurement Invariance of the Behavior Problem Index

Guttmannova, K., Szanyi, J. M., Cali, P. W. — 2008-07-25

Accurate measurement of behavioral functioning is a cornerstone of research on disparities in child development. This study used the National Longitudinal Survey of Youth 1979 (NLSY79) data to test measurement invariance of the Behavior Problem Index (BPI) during middle childhood across three ethnic groups. Using the internalizing and externalizing behavior problem division derived by Parcel and Menaghan (1988) and suggested for use with NLSY79 data, the configural invariance hypothesis was not supported. The BPI factor structure model was revised based on theoretical considerations using the division of items from the Child Behavior Checklist. This model demonstrated configural invariance across ethnic groups and over time. Moreover, measurement invariance of factor loadings and thresholds across ethnic groups at each time point and within each ethnic group over time was also supported. The implications of these findings for educational and cross-cultural research are outlined.

Dimensionality Assessment Using the Full-Information Item Bifactor Analysis for Graded Response Data: An Illustration With the State Metacognitive Inventory

Immekus, J. C., Imbrie, P. K. — 2008-07-25

Dimensionality assessment using the full-information item bifactor model for graded response data is provided. The model applies to data in which each item relates to a general factor and one group factor. Specifically, alternative model specification within item response theory (IRT) is shown to test a scale's factor structure. For illustrative purposes, the bifactor model and competing IRT models were fit to the data of separate cohorts of incoming college students (Cohort 1, n = 1,490; Cohort 2, n = 1,533) to test the dimensionality of an adapted version of the State Metacognitive Inventory. Overall, the bifactor analysis did not strongly support distinct group factors after accounting for the general factor. Instead, results suggested conceptualizing the scale as unidimensional, indicating that scores should be based on the total scale, not subscales. Considerations related to the use of the bifactor IRT model are discussed.