Educational and Psychological Measurement recent issues

Guest Reviewers in 2009

Wed, 18 Nov 2009 00:43:24 PST

Magnitude of Task-Sampling Variability in Performance Assessment: A Meta-Analysis

Huang, C. — Wed, 18 Nov 2009 00:43:24 PST

This study examined the percentage of task-sampling variability in performance assessment via a meta-analysis. In total, 50 studies containing 130 independent data sets were analyzed. Overall results indicate that the percentage of variance for (a) differential difficulty of task was roughly 12% and (b) examinee’s differential performance of the same task was approximately 26%. Based on moderator analysis, research design and subject area were significant predictors of variance component for tasks (²_t) and variance component for person—task interaction (²_pt). Mathematics and listening were associated with relatively high proportion of task-sampling variability whereas foreign language was least affected by this variability. To reduce task variation, some techniques are recommended: (a) incorporating more facets when possible, (b) using a crossed rather than a nested design, and (c) incorporating occasion as a facet.

A Model Fit Statistic for Generalized Partial Credit Model

Liang, T., Wells, C. S. — Wed, 18 Nov 2009 00:43:24 PST

Investigating the fit of a parametric model is an important part of the measurement process when implementing item response theory (IRT), but research examining it is limited. A general nonparametric approach for detecting model misfit, introduced by J. Douglas and A. S. Cohen (2001), has exhibited promising results for the two-parameter logistic model and Samejima s graded response model. This study extends this approach to test the fit of generalized partial credit model (GPCM). The empirical Type I error rate and power of the proposed method are assessed for various test lengths, sample sizes, and type of assessment. Overall, the proposed fit statistic performed well under the studied conditions in that the Type I error rate was not inflated and the power was acceptable, especially for moderate to large sample sizes. A further advantage of the nonparametric approach is that it provides a convenient graphical display of possible misfit.

The Effect of Auxiliary Variables and Multiple Imputation on Parameter Estimation in Confirmatory Factor Analysis

Yoo, J. E. — Wed, 18 Nov 2009 00:43:24 PST

This Monte Carlo study investigates the beneficiary effect of including auxiliary variables during estimation of confirmatory factor analysis models with multiple imputation. Specifically, it examines the influence of sample size, missing rates, missingness mechanism combinations, missingness types (linear or convex), and the absence or presence of the auxiliary variables on convergence failure, bias, standard error, and confidence interval coverage of parameters. Including auxiliary variables in the imputation model is found to improve parameter estimation in most cases, particularly with the convex type of missingness and the nonignorable cases caused by MAR and absence of auxiliary variables in the imputation model. The results of this study can be applied to test validity studies where item selection is needed because of the presence of many alternative items (e.g., instrument development from an item bank). Implications and recommendations for proper imputation are discussed.

Comparison of Methods for Adjusting Incorrect Assignments of Items to Subtests: Oblique Multiple Group Method Versus Confirmatory Common Factor Method

Stuive, I., Kiers, H. A.L., Timmerman, M. E. — Wed, 18 Nov 2009 00:43:24 PST

A common question in test evaluation is whether an a priori assignment of items to subtests is supported by empirical data. If the analysis results indicate the assignment of items to subtests under study is not supported by data, the assignment is often adjusted. In this study the authors compare two methods on the quality of their suggestions to adjust incorrect assignments of items to subtests. The confirmatory common factor (CCF) method is often used in practice. However, previous research reported rather poor quality of the suggested adjustments. Therefore, the CCF method is compared with a less often used but promising method, the oblique multiple group (OMG) method. The authors compared both methods with a simulation study taken under various conditions. For each method, several adjustment procedures were studied. The best adjustment procedure within the OMG method performed better than or highly comparable to the procedures within the CCF method.

Do Student Growth Scores Measure Academic Growth?

Pomplun, M. R. — Wed, 18 Nov 2009 00:43:24 PST

This study investigated convergent validity evidence for student growth scores with high school course grades. The Measures of Academic Progress and Educational Planning and Assessment System growth scores for approximately 1,800 ninth-grade students over 2 years were related to language, arts, and mathematics course grades for developmental, standard, and honors courses. For mathematics, the results indicated that students in honors courses demonstrated more test score growth than students in standard courses, and students with "As" and "‘Bs" demonstrated more test score growth than students with "Ds" and "Fs." These relationships were not shown between growth scores and grades in English courses. In addition, the relatively high percentage of negative growth scores for honors students in both mathematics and English courses suggest that more investigation is needed about the use and validity of these growth scores.

Validity of Scores for a Developmental Writing Scale Based on Automated Scoring

Attali, Y., Powers, D. — Wed, 18 Nov 2009 00:43:24 PST

A developmental writing scale for timed essay-writing performance was created on the basis of automatically computed indicators of writing fluency, word choice, and conventions of standard written English. In a large-scale data collection effort that involved a national sample of more than 12,000 students from 4th, 6th, 8th, 10th, and 12th grade, students wrote (in 30-min sessions) up to four essays in two modes of writing on topics selected from a pool of 20 topics. Scale scores were created by combining essay indicators in a standard way to compute essay scores that shared the same scoring standards across essay prompts and student grade levels. A series of ancillary analyses and studies were conducted to examine the validity of scale scores. Crossclassified random effects modeling of scores confirmed that the particular prompts on which essays are written have little effect on scores. The reliability of scores was found to be higher compared to previous reliability estimates of human essay scores. A human scoring experiment confirmed that the developmental sensitivity of scale scores and human scores was similar. A longitudinal study confirmed the expected gains in scores over a 1-year period.

Item Parameter Invariance of the Kaufman Adolescent and Adult Intelligence Test Across Male and Female Samples

Immekus, J. C., Maller, S. J. — Wed, 18 Nov 2009 00:43:24 PST

The Kaufman Adolescent and Adult Intelligence Test (KAIT^TM) is an individually administered test of intelligence for individuals ranging in age from 11 to 85+ years. The item response theory—likelihood ratio procedure, based on the two-parameter logistic model, was used to detect differential item functioning (DIF) in the KAIT across males and females in the standardization sample. Root mean squared differences and item parameter differences were used to indicate the magnitude of DIF and identify which group the item parameter favored. A z test of proportion differences was conducted to determine if the number of parameters exhibiting gender DIF exceeded the number expected by chance, estimated by randomly dividing the sample in half and repeating the analyses. Of the 176 item parameters examined, 42 (24%) lacked invariance, with most items reporting uniform DIF. Implications for test score interpretation and future research are discussed.

Development and Validation of Scores on the Distributed Leadership Inventory

Hulpia, H., Devos, G., Rosseel, Y. — Wed, 18 Nov 2009 00:43:24 PST

Systematic quantitative research on measuring distributed leadership is scarce. In this study, the Distributed Leadership Inventory (DLI) was developed and evaluated to investigate leadership team characteristics and distribution of leadership functions between formally designed leadership positions in large secondary schools. The DLI was presented to a sample of 2,198 respondents in 46 secondary schools. The input from a first subsample was used to perform exploratory factor analyses; the second subsample was used to verify the factor structure via confirmatory factor analysis. A one-factor structure for the leadership team characteristics (coherent leadership team) and a two-factor structure for the leadership functions (support and supervision) were confirmed. The results of the DLI underpin that leading schools involve multiple individuals, which differs by the type of function.

Developing and Validating Field Measurement Scales for Absorptive Capacity and Experienced Community of Practice

Cadiz, D., Sawyer, J. E., Griffith, T. L. — Wed, 18 Nov 2009 00:43:24 PST

Research on knowledge transfer in organizations has been hampered by the lack of tools yielding valid scores for studying critical constructs in concert. The authors developed survey measures of absorptive capacity (the ability to transform new knowledge into usable knowledge) and experienced community of practice (the extent to which a person is engaged with the given practice community) to provide tools appropriate for field research. A holdout sample of 1,971 engineers in a Fortune 100 science/technology company yielded 583 responses. Confirmatory factor analysis was used to assess internal structure, and convergent and discriminant evidence of validity. Path analysis was used to assess criterion-related validity. Results demonstrate that the new measures are internally consistent, are related in meaningful ways to other organizational variables, and provide distinct explanatory power. An additional 231 responses from a second Fortune 100 science/technology company provides cross-validation.

Testing the Second-Order Factor Structure and Measurement Equivalence of the Wong and Law Emotional Intelligence Scale Across Gender and Ethnicity

Whitman, D. S., Van Rooy, D. L., Viswesvaran, C., Kraus, E. — Wed, 18 Nov 2009 00:43:24 PST

The present study examined the measurement equivalence of a second-order factor model of emotional intelligence (EI). Using scores for 921 job applicants obtained during a personnel selection process, measurement equivalence of the Wong and Law Emotional Intelligence Scale (WLEIS) was tested across ethnic (Whites, Blacks, and Hispanics) and gender groups. Results (a) supported the four-dimension, second-order factor structure of EI and (b) indicated that scores on the WLEIS are comparable across gender and ethnic groups. Findings are discussed in the context of applied and research-based relevance.

The MIMIC Method With Scale Purification for Detecting Differential Item Functioning

Wang, W.-C., Shih, C.-L., Yang, C.-C. — Sun, 27 Sep 2009 23:16:08 PDT

This study implements a scale purification procedure onto the standard MIMIC method for differential item functioning (DIF) detection and assesses its performance through a series of simulations. It is found that the MIMIC method with scale purification (denoted as M-SP) outperforms the standard MIMIC method (denoted as M-ST) in controlling false-positive rates and yielding higher true-positive rates. Only when the DIF pattern is balanced between groups or when there is a small percentage of DIF items in the test does M-ST perform as appropriately as M-SP. Moreover, both methods yield a higher true-positive rate under the two-parameter logistic model than under the three-parameter model. M-SP is preferable to M-ST, because DIF patterns in real tests are unlikely to be perfectly balanced and the percentages of DIF items may not be small.

A Comparison of Unidimensional and Three-Dimensional Differential Item Functioning Analysis Using Two-Dimensional Data

Snow, T. K., Oshima, T.C. — Sun, 27 Sep 2009 23:16:08 PDT

Oshima, Raju, and Flowers demonstrated the use of an item response theory—based technique for analyzing differential item function (DIF) and differential test function for dichotomously scored data that are intended to be multidimensional. Their study assumed that the number of intended-to-be measured dimensions was correctly identified. In practice, however, the number of dimensions may be misidentified. Therefore, the purpose of this study was to demonstrate the effects of both underestimation and overestimation of the number of intended-to-be measured dimensions on the multidimensional DIF analysis using simulated two-dimensional data with known DIF items. Results show that overestimation of the number of y traits had a consequence of decreased power. Underestimation resulted in missing a certain type of nonuniform DIF, as well as confounding the impact with DIF. Recommendations are made on how to conduct a DIF investigation with a multidimensional within-item test.

Accuracy of the Parallel Analysis Procedure With Polychoric Correlations

Cho, S.-J., Li, F., Bandalos, D. — Sun, 27 Sep 2009 23:16:08 PDT

The purpose of this study was to investigate the application of the parallel analysis (PA) method for choosing the number of factors in component analysis for situations in which data are dichotomous or ordinal. Although polychoric correlations are sometimes used as input for component analyses, the random data matrices generated for use in PA typically consist of Pearson correlations. In this study, the authors matched the type of random data matrix to the type of input matrix. Analyses were conducted on both polychoric and Pearson correlation matrices, and random matrices of the same type (polychoric or Pearson) were generated for the PA procedure. PA based on random Pearson correlations was found to perform at least as well as PA based on random polychoric correlations, for nearly all of the conditions studied.

Construct Equivalence Across Grades in a Vertical Scale for a K-12 Large-Scale Reading Assessment

Wang, S., Jiao, H. — Sun, 27 Sep 2009 23:16:08 PDT

In practice, vertical scales have been continually used to measure students’ achievement progress across several grade levels and have been considered very challenging psychometric procedures. Recently, such practices have been drawing many criticisms. The major criticisms focus on dimensionality and construct equivalence of the latent trait or construct of a vertical scale. Using data from a large-scale reading comprehension assessment program with a vertical scale, this study investigated the dimensionality or the factorial structure for each grade and the equivalence of the factorial structure across grades. The empirical results from this study support the unidimensionality assumption of vertical scale across grades. The results provide empirical evidence regarding construct invariance of vertical scale across wide span of grades.

Item Selection in Computerized Classification Testing

Thompson, N. A. — Sun, 27 Sep 2009 23:16:08 PDT

Several alternatives for item selection algorithms based on item response theory in computerized classification testing (CCT) have been suggested, with no conclusive evidence on the substantial superiority of a single method. It is argued that the lack of sizable effect is because some of the methods actually assess items very similarly through different calculations and will usually select the same item. Consideration of methods that assess information across a wider range is often unnecessary under realistic conditions, although it might be advantageous to utilize them only early in a test. In addition, the efficiency of item selection approaches depend on the termination criteria that are used, which is demonstrated through didactic example and Monte Carlo simulation. Item selection at the cut score, which seems conceptually appropriate for CCT, is not always the most efficient option. A broad framework for item selection in CCT is presented that incorporates these points.

Motivation and Engagement Across the Academic Life Span: A Developmental Construct Validity Study of Elementary School, High School, and University/College Students

Martin, A. J. — Sun, 27 Sep 2009 23:16:08 PDT

From a developmental construct validity perspective, this study examines motivation and engagement across elementary school, high school, and university/college, with particular focus on the Motivation and Engagement Scale (comprising adaptive, impeding/maladaptive, and maladaptive factors). Findings demonstrated developmental construct validity across the three distinct educational stages in terms of goodfitting first- and higher order factors, invariance of factor structure across gender and age, and a pattern of correlations with cognate constructs (e.g., homework completion, academic buoyancy, class participation) consistent with predictions. Notwithstanding the predominantly parallel findings, there was also notable distinctiveness, primarily in terms of mean-level effects, such that elementary school students were generally more motivated and engaged than university/college students who in turn were more motivated and engaged than high school students. Implications for motivation and engagement measurement and theory, research in the psychoeducational domain, and the subsequent potential for performance profiling across the academic life span are discussed.

Developing Short Forms of the EARLI Numeracy Measures: Comparison of Item Selection Methods

Lei, P.-W., Wu, Q., DiPerna, J. C., Morgan, P. L. — Sun, 27 Sep 2009 23:16:08 PDT

Currently, few measures are available to monitor young children’s progress in acquiring key early academic skills. In response to this need, the authors have begun developing measures (i.e., the Early Arithmetic, Reading and Learning Indicators, or EARLI) of preschoolers’ numeracy skills. To accurately and efficiently monitor acquisition of early skills, users require multiple short forms that are appropriate in difficulty level for young children at different points in time. In the current study, the authors used three item selection methods to create multiple short forms for two EARLI numeracy measures. The authors then compared these item selection methods on projected internal consistency and concurrent validity estimates for the resulting forms. The short forms selected by these methods did not differ significantly on either criterion and appeared to be sufficiently sensitive to measure initial and acquisition of numeracy skills over time by preschool children enrolled in Head Start.

Measurement Invariance of the Pay Satisfaction Questionnaire Across Gender: Explaining the ''Contented Female Worker Paradox'' More Thoroughly

De Gieter, S., Hofmans, J., De Cooman, R., Pepermans, R. — Sun, 27 Sep 2009 23:16:08 PDT

Although women tend to earn less for doing similar jobs, research indicates that they are at least as satisfied with their pay as their male colleagues. In examining explanations for this paradox of the contented female worker, the Pay Satisfaction Questionnaire (PSQ) is often used to measure and compare the pay satisfaction of both gender groups. However, using the PSQ without testing its measurement invariance across gender can lead to rash conclusions. Therefore, this study submitted PSQ data of 699 female and 448 male employees from different organizational sectors to a series of measurement invariance tests (i.e., configural invariance, metric invariance, scalar invariance, and error variance invariance). All tests’ results support the measurement invariance of the PSQ across gender, indicating it as an appropriate instrument to examine the contented female worker paradox. The article concludes with some suggestions for future research.

Studying the Equivalence of Computer-Delivered and Paper-Based Administrations of the Raven Standard Progressive Matrices Test

Arce-Ferrer, A. J., Martinez Guzman, E. — Sun, 27 Sep 2009 23:16:08 PDT

This study investigates the effect of mode of administration of the Raven Standard Progressive Matrices test on distribution, accuracy, and meaning of raw scores. A random sample of high school students take counterbalanced paper-and-pencil and computer-based administrations of the test and answer a questionnaire surveying preferences for computer-delivered test administrations. Administration mode effect is studied with repeated measures multivariate analysis of variance, internal consistency reliability estimates, and confirmatory factor analysis approaches. Results show a lack of test mode effect on distribution, accuracy, and meaning of raw scores. Participants indicate their preferences for the computer-delivered administration of the test. The article discusses findings in light of previous studies of the Raven Standard Progressive Matrices test.

Development and Psychometric Evaluation of the High School Chemistry Self-Efficacy Scale

Aydin, Y. C., Uzuntiryaki, E. — Sun, 27 Sep 2009 23:16:08 PDT

The aim of this study was to develop a scale assessing high school students’ selfefficacy beliefs in chemistry-related tasks and to assess psychometric properties of scores on this scale. A pilot study with a sample of 150 high school students provided initial evidence for two-factor structure of 16-item scale, named High School Chemistry Self-Efficacy Scale (HCSS). The final form of the HCSS was administered to 362 high school students in Turkey. Confirmatory factor analysis fit indices and factor pattern coefficients supported the proposed structure, with the two factors directly corresponding to the hypothesized dimensions. These dimensions were chemistry self-efficacy for cognitive skills (10 items, a reliability = .90) and self-efficacy for chemistry laboratory (6 items, a reliability = .92). In addition, the two-factor model for the HCSS was invariant across school types. Analyses of latent mean structure indicated that private school students were statistically more efficacious than public school students in chemistry laboratory skills. The HCSS is a promising tool to identify chemistry self-efficacy beliefs in high school students.

Direct and Inverse Problems of Item Pool Design for Computerized Adaptive Testing

Belov, D. I., Armstrong, R. D. — Wed, 15 Jul 2009 10:35:22 PDT

The recent literature on computerized adaptive testing (CAT) has developed methods for creating CAT item pools from a large master pool. Each CAT pool is designed as a set of nonoverlapping forms reflecting the skill levels of an assumed population of test takers. This article presents a Monte Carlo method to obtain these CAT pools and discusses its advantages over existing methods. Also, a new problem is considered that finds a population ability density function best matching the master pool. An analysis of the solution to this new problem provides testing organizations with effective guidance for maintaining their master pools. Computer experiments with a pool of Law School Admission Test items and its assembly constraints are presented.

Modeling Socially Desirable Responding and Its Effects

Ziegler, M., Buehner, M. — Wed, 15 Jul 2009 10:35:22 PDT

The impact of socially desirable responding or faking on noncognitive assessments remains an issue of strong debate. One of the main reasons for the controversy is the lack of a statistical method to model such response sets. This article introduces a new way to model faking based on the assumption that faking occurs due to an interaction between person and situation. The technique combines a control group design with structural equation modeling and allows a separation of trait and faking variance. The model is introduced and tested in an example. The results confirm a causal influence of faking on means and covariance structure of a Big 5 questionnaire. Both effects can be reversed by the proposed model. Finally, a real-life criterion was implemented and predicted by both variance sources. In this example, it was the trait but not the faking variance that was predictive. Implications for research and practice are discussed.

Bootstrap Standard Error and Confidence Intervals for the Difference Between Two Squared Multiple Correlation Coefficients

Wai Chan, — Wed, 15 Jul 2009 10:35:22 PDT

A typical question in multiple regression analysis is to determine if a set of predictors gives the same degree of predictor power in two different populations. Olkin and Finn (1995) proposed two asymptotic-based methods for testing the equality of two population squared multiple correlations, ²₁ and ²₂. Simulation results indicated that these methods failed to perform accurately under certain model conditions (Algina & Keselman, 1999). In the present study, a unified bootstrap procedure is proposed for estimating the standard error of R²₁ – R²₂ and constructing the confidence interval for ²₁ – ²₂. A simulation study was conducted to examine the empirical performance of the proposed procedure under different levels of ², sample sizes, numbers of predictors, and types of data distribution. Results indicated that the asymptotic method, in general, can only work well with normal data. The bootstrap procedure, on the other hand, performs satisfactorily with both normal and nonnormal data. However, both methods fail when ²₁ and ²₂ are zero.

Using Item Response Theory and Model--Data Fit to Conceptualize Differential Item and Person Functioning for Students With Disabilities

Engelhard, G. — Wed, 15 Jul 2009 10:35:22 PDT

The major purpose of this study is to describe a conceptual framework for examining differential item functioning (DIF) and differential person functioning (DPF) as types of model—data misfit within the context of assessing students with disabilities. Specifically, DIF and DPF can be viewed through the lens of residual analyses. Residual analyses can be used to explore DIF (item fit) as well as extended to explore DPF (person fit). One of the advantages of this conceptual framework is that the size of the subgroups can be quite small with interpretable results produced even for individuals. To illustrate this conceptual framework, Rasch measurement theory is used as the item response theory model. Methodological and theoretical issues are discussed. Data from a high-stakes assessment in mathematics in Georgia (Grade 7, geometry items) are used to illustrate the conceptual framework for students with disabilities. The substantive research questions used to illustrate the conceptual framework addresses whether modifications in test administration (use of resource guides or calculators) influence item functioning for students with disabilities (DIF analyses) and whether students with various disabilities differentially benefit from these modifications (DPF analyses).

The Potential for Differential Findings Among Invariance Testing Strategies for Multisample Measured Variable Path Models

Mann, H. M., Rutstein, D. W., Hancock, G. R. — Wed, 15 Jul 2009 10:35:22 PDT

Multisample measured variable path analysis is used to test whether causal/structural relations among measured variables differ across populations. Several invariance testing approaches are available for assessing cross-group equality of such relations, but the associated test statistics may vary considerably across methods. This study is a population analysis, examining five different strategies for invariance testing using an illustrative measured variable path model. The results demonstrate how inferences about parameters across populations can depend greatly upon the invariance testing approach used, thereby potentially leading to improper inference regarding the true invariance status of relevant parameters.

Gender and Ethnicity Differences on the Abridged Big Five Circumplex (AB5C) of Personality Traits: A Differential Item Functioning Analysis

Mitchelson, J. K., Wicher, E. W., LeBreton, J. M., Craig, S. B. — Wed, 15 Jul 2009 10:35:23 PDT

The current study evaluates the measurement precision of the Abridged Big Five Circumplex (AB5C) of personality traits by identifying those items that demonstrate differential item functioning by gender and ethnicity. Differential item functioning is found in 33 of 45 (73%) of the AB5C scales, across gender and ethnic groups (Caucasian vs. African American). More differential item functioning is found by ethnic groups compared with gender groups. The amount of differential item functioning found in these personality measures suggests the need to consider differential item functioning in the construction of measures. The authors encourage those using the AB5C measure to discard or reword these problematic items.

Psychometric Properties of the Scores on the Behavioral Inhibition and Activation Scales in a Sample of Norwegian Children

Bjornebekk, G. — Wed, 15 Jul 2009 10:35:23 PDT

The primary aim of this study was to examine the psychometric properties of the scores on a version for children of the Carver and White Behavioral Inhibition and Activation scales (the BIS—BAS scales). This involved administering the BIS—BAS scales, the Positive and Negative Affect Schedule, the Junior Eysenck Personality Questionnaire Revised—Abbreviated, and the Achievement Motives Scale to a population of 661 Norwegian sixth graders. The findings reveal that the scores on the BIS—BAS scales for children have a theoretically meaningful factor structure as well as satisfactory convergent validity and reliability. However, the results indicate that the version for children of the BAS scale actually may consist of two subscales: one related to the experience of pleasurable affect and one to persistence in goal pursuit. To a large extent, the relationship with the other scales was as expected.

Development of the Tempe Sorting Task: A Principled Approach to Assessment of Children's Executive Functioning

Marshall, S. J., Wodrich, D. L., Gorin, J. S. — Wed, 15 Jul 2009 10:35:23 PDT

This study examined psychometric properties of the Tempe Sorting Task (TST), a new measure of executive function (EF) for children. To increase the meaningfulness of test score interpretations, an age-appropriate construct was employed to incorporate Denckla's description of EF. Multiple measures of EF, including the TST, were collected for children (N = 81). Interrater reliability estimates of the TST scores across trials were high. Split-half reliability estimates were slightly lower and varied across trials. Multiple significant correlations were observed between the TST, Expressive Attention task, and teacher Attention Deficit Hyperactivity Disorder (ADHD) rating scales. Furthermore, statistically significant differences in TST scores between children with elevated versus normal-range ADHD ratings (d_Trial1 = 1:49, d_Trial2 = 0:86) provided some evidence of the discriminatory power of the TST. Overall, results provided some evidence supporting the interpretation of TST measures as indicators of children's EF. Advantages of the TST over existing children's EF measures are considered.

Evaluation of an Observer Form of the Coping Inventory for Stressful Situations

Han, K., Burns, G. N., Weed, N. C., Hatchett, G. T., Kurokawa, N. K. S. — Wed, 15 Jul 2009 10:35:23 PDT

The present study evaluates a prospective observer form of the Coping Inventory for Stressful Situations (CISS) by comparing the two forms in terms of factor structure, mean differences, reliability, and examining correlations between self-report and peer ratings. A total of 163 pairs of friends complete the CISS and an observer form of the CISS. Confirmatory factor analysis results indicate that for both rating forms, the four-factor solution fits better. Although self-rating data fit the theoretical model better, the peer ratings show higher reliability. The correlation between self and peer latent factors is moderate for Avoidance-oriented coping and for its subscales, but low for Task- and Emotion-oriented coping. Internal consistency coefficients for the CISS scales are high across type of rating, and a significant cross-form mean difference is found on the Task latent factor. Overall, the results provide evidence of substantial measurement equivalence between the self-rating form and the observer form, and the authors propose its use in dispositional coping research.

Exploratory and Confirmatory Studies of the Structure of the Bem Sex Role Inventory Short Form With Two Divergent Samples

Choi, N., Fuqua, D. R., Newman, J. L. — Wed, 15 Jul 2009 10:35:23 PDT

The short form of the Bem Sex Role Inventory (BSRI) contains half as many items as the long form and yet has often demonstrated better reliability and validity. This study uses exploratory and confirmatory factor analytic methods to examine the structure of the short form of the BSRI. A structure noted elsewhere also emerged here, consisting of two masculine factors and a single feminine factor. The three-factor model was found to be invariant across gender groups and also across two divergent samples, the first sample of college students and the second sample of accountants. As expected, women were found to score higher on the feminine factor. On a masculine factor that seemed to represent social control, men scored significantly higher than women did. However, no differences were found between men and women on a second masculine factor that seemed to represent a more internal, self-control dimension.

Expected Equating Error Resulting From Incorrect Handling of Item Parameter Drift Among the Common Items

Miller, G. E., Fitzpatrick, S. J. — Thu, 07 May 2009 11:33:14 PDT

Incorrect handling of item parameter drift during the equating process can result in equating error. If the item parameter drift is due to construct-irrelevant factors, then inclusion of these items in the estimation of the equating constants can be expected to result in equating error. On the other hand, if the item parameter drift is related to the construct being measured, then removal of those items from the estimation of the equating constants can be expected to result in equating error. The effect of incorrect handling of item parameter drift on equating has been examined via simulation studies and other methods by a number of authors with mixed results. In this article, this effect is assessed by estimating the expected equating error under the assumptions that the three-parameter item response theory model is used for calibration—equating.

Multidimensional Rasch Analysis of a Psychological Test With Multiple Subtests: A Statistical Solution for the Bandwidth--Fidelity Dilemma

Cheng, Y.-Y., Wang, W.-C., Ho, Y.-H. — Thu, 07 May 2009 11:33:14 PDT

Educational and psychological tests are often composed of multiple short subtests, each measuring a distinct latent trait. Unfortunately, short subtests suffer from low measurement precision, which makes the bandwidth—fidelity dilemma inevitable. In this study, the authors demonstrate how a multidimensional Rasch analysis can be employed to take into account the information about the correlation between latent traits such that the precision of each subtest measure can be improved and the correlation between latent traits can be accurately estimated. A real data set of the 13-scale Thinking Styles Inventory was analyzed with the traditional unidimensional approach and the multidimensional approach. The results demonstrate that in contrast to the unidimensional approach, the multidimensional approach yields a much higher level of measurement precision and a more appropriate estimate for the correlation between thinking styles. In conclusion, even short subtests can yield highly precise measures such that the bandwidth—fidelity dilemma is resolved.

Survey Response Rates and Survey Administration in Counseling and Clinical Psychology: A Meta-Analysis

Van Horn, P. S., Green, K. E., Martinussen, M. — Thu, 07 May 2009 11:33:14 PDT

This article reports results of a meta-analysis of survey response rates in published research in counseling and clinical psychology over a 20-year span and describes reported survey administration procedures in those fields. Results of 308 survey administrations showed a weighted average response rate of 49.6%. Among possible moderators, response rates differed only by population sampled, journal in which articles were published, sampling source and method, and use of follow-up. Researchers whose studies were included in this meta-analysis used follow-up but rarely used incentives, prenotification, or other response-facilitation methods to maximize response rates. Although the future of survey research in general may rely more heavily on Internet data collection, mail surveys dominate in this field.

What Are the Consequences If the Assumption of Independent Observations Is Violated in Reliability Generalization Meta-Analysis Studies?

Romano, J. L., Kromrey, J. D. — Thu, 07 May 2009 11:33:14 PDT

This study was conducted to evaluate alternative analysis strategies for the meta-analysis method of reliability generalization when the reliability estimates are not statistically independent. Five approaches to dealing with the violation of independence were implemented: ignoring the violation and treating each observation as independent, calculating one mean or median from each study, selecting only one observation per study, and using a mixed-effects model. Monte Carlo methods were used to simulate samples under known and controlled population conditions. The results suggest that the type of approach does not have a noticeable impact on the accuracy of the reliability results but that researchers should be cautious when the intraclass correlation is relatively large. The simulations in this study also resulted in very poor confidence band coverage.

Tetrachoric Correlation: A Permutation Alternative

Long, M. A., Berry, K. J., Mielke, P. W. — Thu, 07 May 2009 11:33:14 PDT

An exact permutation test is provided for the tetrachoric correlation coefficient. Comparisons with the conventional test employing Student's t distribution demonstrate the necessity of using the permutation approach for small sample sizes and/or disproportionate marginal frequency totals.

Rasch Modeling of the Self-Deception Scale of the Balanced Inventory of Desirable Responding

Cervellione, K. L., Lee, Y.-S., Bonanno, G. A. — Thu, 07 May 2009 11:33:14 PDT

Self-deception has become a construct of great interest in individual differences research because it has been associated with levels of resilience and mental health. The Balanced Inventory of Desirable Responding (BIDR) is a self-report measure used for quantifying self-deception. In this study we used Rasch modeling to examine the properties of the self-deception scale of the BIDR in terms of dimensionality, use of response category, sample appropriateness, and reliability. A total of 315 university students (ages 18-21) were administered the self-deception scale of the BIDR. Seven-category and 2-category scoring methods were compared, as approved by the developers of the scale. Overall, the 7-category model was the best fit for the data and the sample. We concluded that the scale has the best reliability using a 7-category model with Item 13 deleted. Because of low person measure separation and reliability, the appropriateness of use of this instrument in undergraduate populations is questioned; the use of the measure in populations with larger ranges of self-deception is not recommended.

The Chinese Intelligence Scale for Young Children: Testing Factor Structure and Measurement Invariance Using the Framework of the Wechsler Intelligence Tests

Guo, B., Aveyard, P., Dai, X. — Thu, 07 May 2009 11:33:14 PDT

The Wechsler intelligence test has four factors representing four components of intellectual function. In China, there are marked cultural, educational, and economic disparities between rural and urban dwellers, which could lead to cultural bias. The aim of this study was to apply the four-factor structure to responses to the Chinese Intelligence Scale for Young Children (CISYC) of 820 rural and 664 urban children aged 3 to 7 years. Measurement invariance testing using confirmatory factor analysis showed that the same four factors nested under a higher-order factor held for both rural and urban children. The general intelligence factor mean and variance were invariant and the means and range of scores for rural and urban children were similar. The results show no evidence of cultural bias and that the four factors can be interpreted similarly to those in the Wechsler intelligence test.

Using the Standardized Letters of Recommendation in Selection: Results From a Multidimensional Rasch Model

Liu, O. L., Minsky, J., Ling, G., Kyllonen, P. — Thu, 07 May 2009 11:33:14 PDT

In an effort to standardize academic application procedures, the authors developed the Standardized Letters of Recommendation (SLR) to capture important cognitive and noncognitive qualities of graduate school candidates. The SLR, which consists of seven scales, is applied to an intern-selection scenario. Both professor ratings (n = 414) during the application process and mentor ratings of the selected students (n = 51) are collected using the SLR. A multidimensional Rasch investigation suggests that the SLR displays satisfactory internal consistency, model fit, and item fit. The two cognitive scales, knowledge and analytical skills, are found to be the best predictors for intern selection. The professor ratings are systematically higher than the mentor ratings. Possible reasons for the rating discrepancies are discussed. Also, implications for how the SLR can be used and improved in other selection situations are suggested.

A Motivational Perspective on Engagement and Disaffection: Conceptualization and Assessment of Children's Behavioral and Emotional Participation in Academic Activities in the Classroom

Skinner, E. A., Kindermann, T. A., Furrer, C. J. — Thu, 07 May 2009 11:33:14 PDT

This article presents a motivational conceptualization of engagement and disaffection: First, it emphasizes children's constructive, focused, enthusiastic participation in the activities of classroom learning; second, it distinguishes engagement from disaffection, as well as behavioral features from emotional features. Psychometric properties of scores from teacher and student reports of behavioral engagement, emotional engagement, behavioral disaffection, and emotional disaffection were examined using data from 1,018 third through sixth graders. Structural analyses of the four indicators confirm that a multidimensional structure fits the data better than do bipolar or unidimensional models. Validity of scores is supported by findings that teacher reports are correlated with student reports, with in vivo observations in the classroom, and with markers of self-system and social contextual processes. As such, these measures capture important features of engagement and disaffection in the classroom, and any comprehensive assessment should include markers of each. Additional dimensions are identified, pointing the way to future research.