Educational and Psychological Measurement current issue

Guest Reviewers in 2009

Wed, 18 Nov 2009 00:43:24 PST

Magnitude of Task-Sampling Variability in Performance Assessment: A Meta-Analysis

Huang, C. — Wed, 18 Nov 2009 00:43:24 PST

This study examined the percentage of task-sampling variability in performance assessment via a meta-analysis. In total, 50 studies containing 130 independent data sets were analyzed. Overall results indicate that the percentage of variance for (a) differential difficulty of task was roughly 12% and (b) examinee’s differential performance of the same task was approximately 26%. Based on moderator analysis, research design and subject area were significant predictors of variance component for tasks (²_t) and variance component for person—task interaction (²_pt). Mathematics and listening were associated with relatively high proportion of task-sampling variability whereas foreign language was least affected by this variability. To reduce task variation, some techniques are recommended: (a) incorporating more facets when possible, (b) using a crossed rather than a nested design, and (c) incorporating occasion as a facet.

A Model Fit Statistic for Generalized Partial Credit Model

Liang, T., Wells, C. S. — Wed, 18 Nov 2009 00:43:24 PST

Investigating the fit of a parametric model is an important part of the measurement process when implementing item response theory (IRT), but research examining it is limited. A general nonparametric approach for detecting model misfit, introduced by J. Douglas and A. S. Cohen (2001), has exhibited promising results for the two-parameter logistic model and Samejima s graded response model. This study extends this approach to test the fit of generalized partial credit model (GPCM). The empirical Type I error rate and power of the proposed method are assessed for various test lengths, sample sizes, and type of assessment. Overall, the proposed fit statistic performed well under the studied conditions in that the Type I error rate was not inflated and the power was acceptable, especially for moderate to large sample sizes. A further advantage of the nonparametric approach is that it provides a convenient graphical display of possible misfit.

The Effect of Auxiliary Variables and Multiple Imputation on Parameter Estimation in Confirmatory Factor Analysis

Yoo, J. E. — Wed, 18 Nov 2009 00:43:24 PST

This Monte Carlo study investigates the beneficiary effect of including auxiliary variables during estimation of confirmatory factor analysis models with multiple imputation. Specifically, it examines the influence of sample size, missing rates, missingness mechanism combinations, missingness types (linear or convex), and the absence or presence of the auxiliary variables on convergence failure, bias, standard error, and confidence interval coverage of parameters. Including auxiliary variables in the imputation model is found to improve parameter estimation in most cases, particularly with the convex type of missingness and the nonignorable cases caused by MAR and absence of auxiliary variables in the imputation model. The results of this study can be applied to test validity studies where item selection is needed because of the presence of many alternative items (e.g., instrument development from an item bank). Implications and recommendations for proper imputation are discussed.

Comparison of Methods for Adjusting Incorrect Assignments of Items to Subtests: Oblique Multiple Group Method Versus Confirmatory Common Factor Method

Stuive, I., Kiers, H. A.L., Timmerman, M. E. — Wed, 18 Nov 2009 00:43:24 PST

A common question in test evaluation is whether an a priori assignment of items to subtests is supported by empirical data. If the analysis results indicate the assignment of items to subtests under study is not supported by data, the assignment is often adjusted. In this study the authors compare two methods on the quality of their suggestions to adjust incorrect assignments of items to subtests. The confirmatory common factor (CCF) method is often used in practice. However, previous research reported rather poor quality of the suggested adjustments. Therefore, the CCF method is compared with a less often used but promising method, the oblique multiple group (OMG) method. The authors compared both methods with a simulation study taken under various conditions. For each method, several adjustment procedures were studied. The best adjustment procedure within the OMG method performed better than or highly comparable to the procedures within the CCF method.

Do Student Growth Scores Measure Academic Growth?

Pomplun, M. R. — Wed, 18 Nov 2009 00:43:24 PST

This study investigated convergent validity evidence for student growth scores with high school course grades. The Measures of Academic Progress and Educational Planning and Assessment System growth scores for approximately 1,800 ninth-grade students over 2 years were related to language, arts, and mathematics course grades for developmental, standard, and honors courses. For mathematics, the results indicated that students in honors courses demonstrated more test score growth than students in standard courses, and students with "As" and "‘Bs" demonstrated more test score growth than students with "Ds" and "Fs." These relationships were not shown between growth scores and grades in English courses. In addition, the relatively high percentage of negative growth scores for honors students in both mathematics and English courses suggest that more investigation is needed about the use and validity of these growth scores.

Validity of Scores for a Developmental Writing Scale Based on Automated Scoring

Attali, Y., Powers, D. — Wed, 18 Nov 2009 00:43:24 PST

A developmental writing scale for timed essay-writing performance was created on the basis of automatically computed indicators of writing fluency, word choice, and conventions of standard written English. In a large-scale data collection effort that involved a national sample of more than 12,000 students from 4th, 6th, 8th, 10th, and 12th grade, students wrote (in 30-min sessions) up to four essays in two modes of writing on topics selected from a pool of 20 topics. Scale scores were created by combining essay indicators in a standard way to compute essay scores that shared the same scoring standards across essay prompts and student grade levels. A series of ancillary analyses and studies were conducted to examine the validity of scale scores. Crossclassified random effects modeling of scores confirmed that the particular prompts on which essays are written have little effect on scores. The reliability of scores was found to be higher compared to previous reliability estimates of human essay scores. A human scoring experiment confirmed that the developmental sensitivity of scale scores and human scores was similar. A longitudinal study confirmed the expected gains in scores over a 1-year period.

Item Parameter Invariance of the Kaufman Adolescent and Adult Intelligence Test Across Male and Female Samples

Immekus, J. C., Maller, S. J. — Wed, 18 Nov 2009 00:43:24 PST

The Kaufman Adolescent and Adult Intelligence Test (KAIT^TM) is an individually administered test of intelligence for individuals ranging in age from 11 to 85+ years. The item response theory—likelihood ratio procedure, based on the two-parameter logistic model, was used to detect differential item functioning (DIF) in the KAIT across males and females in the standardization sample. Root mean squared differences and item parameter differences were used to indicate the magnitude of DIF and identify which group the item parameter favored. A z test of proportion differences was conducted to determine if the number of parameters exhibiting gender DIF exceeded the number expected by chance, estimated by randomly dividing the sample in half and repeating the analyses. Of the 176 item parameters examined, 42 (24%) lacked invariance, with most items reporting uniform DIF. Implications for test score interpretation and future research are discussed.

Development and Validation of Scores on the Distributed Leadership Inventory

Hulpia, H., Devos, G., Rosseel, Y. — Wed, 18 Nov 2009 00:43:24 PST

Systematic quantitative research on measuring distributed leadership is scarce. In this study, the Distributed Leadership Inventory (DLI) was developed and evaluated to investigate leadership team characteristics and distribution of leadership functions between formally designed leadership positions in large secondary schools. The DLI was presented to a sample of 2,198 respondents in 46 secondary schools. The input from a first subsample was used to perform exploratory factor analyses; the second subsample was used to verify the factor structure via confirmatory factor analysis. A one-factor structure for the leadership team characteristics (coherent leadership team) and a two-factor structure for the leadership functions (support and supervision) were confirmed. The results of the DLI underpin that leading schools involve multiple individuals, which differs by the type of function.

Developing and Validating Field Measurement Scales for Absorptive Capacity and Experienced Community of Practice

Cadiz, D., Sawyer, J. E., Griffith, T. L. — Wed, 18 Nov 2009 00:43:24 PST

Research on knowledge transfer in organizations has been hampered by the lack of tools yielding valid scores for studying critical constructs in concert. The authors developed survey measures of absorptive capacity (the ability to transform new knowledge into usable knowledge) and experienced community of practice (the extent to which a person is engaged with the given practice community) to provide tools appropriate for field research. A holdout sample of 1,971 engineers in a Fortune 100 science/technology company yielded 583 responses. Confirmatory factor analysis was used to assess internal structure, and convergent and discriminant evidence of validity. Path analysis was used to assess criterion-related validity. Results demonstrate that the new measures are internally consistent, are related in meaningful ways to other organizational variables, and provide distinct explanatory power. An additional 231 responses from a second Fortune 100 science/technology company provides cross-validation.

Testing the Second-Order Factor Structure and Measurement Equivalence of the Wong and Law Emotional Intelligence Scale Across Gender and Ethnicity

Whitman, D. S., Van Rooy, D. L., Viswesvaran, C., Kraus, E. — Wed, 18 Nov 2009 00:43:24 PST

The present study examined the measurement equivalence of a second-order factor model of emotional intelligence (EI). Using scores for 921 job applicants obtained during a personnel selection process, measurement equivalence of the Wong and Law Emotional Intelligence Scale (WLEIS) was tested across ethnic (Whites, Blacks, and Hispanics) and gender groups. Results (a) supported the four-dimension, second-order factor structure of EI and (b) indicated that scores on the WLEIS are comparable across gender and ethnic groups. Findings are discussed in the context of applied and research-based relevance.