Educational and Psychological Measurement RSS feed -- OnlineFirst Articles

Athletes' Perceptions of Coaching Competency Scale II-High School Teams

Myers, N. D., Chase, M. A., Beauchamp, M. R., Jackson, B. — Fri, 16 Oct 2009 17:02:11 PDT

The purpose of this validity study was to improve measurement of athletes’ evaluations of their head coach’s coaching competency, an important multidimensional construct in models of coaching effectiveness.A revised version of the Coaching Competency Scale (CCS) was developed for athletes of high school teams (APCCS II-HST). Data were collected from athletes (N = 748) of seven relevant sports. Athlete observations were clustered within teams (G = 74). Multigroup confirmatory factor analyses of the asymptotic within-teams covariance matrix provided evidence for factorial invariance, except for one residual variance, by athlete gender (n_male = 427, n_female= 321). An exploratory multilevel confirmatory factor analysis provided evidence for close fit of an oblique five-factor within-teams structure and a one-factor between-teams structure.

Examining Dual Meanings of Items in 2 x 2 Achievement Goal Questionnaires Through MTMM Modeling and MDS Approach

Wu, C.-H., Chen, L. H. — Fri, 16 Oct 2009 17:02:11 PDT

In 2001, Elliot and McGregor proposed a 2 x 2 (mastery–performance x approach–avoidance) achievement goal frameworks and developed a questionnaire to measure four goals (mastery–approach, mastery–avoidance, performance–approach, and performance–avoidance goals).This study examines the dual meanings of items in 2 x 2 achievement goal questionnaires by confirmatory factor analysis with an uncorrelated-trait and uncorrelated-method model and multidimensional scaling technique. A total of 2,111 freshmen completed the Achievement Goal Questionnaire for General Academics, and a total of 1,014 university students in Taiwan provided their ratings on the Chinese Achievement Goal Questionnaire for Physical Education. Confirmatory factor analysis results show that the uncorrelated-trait and uncorrelated-method model is better than the 2 x 2 four-factor model, supporting the dual meanings of items. In addition, multidimensional scaling results show that items can be plotted on a two-dimensional (mastery–performance and approach–avoidance) map, in which four quadrants represent one type of goal. The evidence indicates that the items in the Chinese Achievement Goal Questionnaire for Physical Education have adequate construct validity in their dual meanings derived from the combinations of mastery–performance and approach–avoidance dimensions.

Weighting by Inverse Variance or by Sample Size in Random-Effects Meta-Analysis

Marin-Martinez, F., Sanchez-Meca, J. — Tue, 13 Oct 2009 09:23:39 PDT

Most of the statistical procedures in meta-analysis are based on the estimation of average effect sizes from a set of primary studies. The optimal weight for averaging a set of independent effect sizes is the inverse variance of each effect size, but in practice these weights have to be estimated, being affected by sampling error. When assuming a random-effects model, there are two alternative procedures for averaging independent effect sizes: Hunter and Schmidt’s estimator, which consists of weighting by sample size as an approximation to the optimal weights; and Hedges and Vevea’s estimator, which consists of weighting by an estimation of the inverse variance of each effect size. In this article, the bias and mean squared error of the two estimators were assessed via Monte Carlo simulation of meta-analyses with the standardized mean difference as the effect-size index. Hedges and Vevea’s estimator, although slightly biased, achieved the best performance in terms of the mean squared error. As the differences between the values of both estimators could be of practical relevance, Hedges and Vevea’s estimator should be selected rather than that of Hunter and Schmidt when the effect-size index is the standardized mean difference.

Students' Group and Member Attachment to Their University: A Construct Validity Study of the University Attachment Scale

France, M. K., Finney, S. J., Swerdzewski, P. — Tue, 13 Oct 2009 09:23:38 PDT

This study examined the psychometric properties of scores from the University Attachment Scale, a measure that operationalizes group and member attachment as two separate dimensions of attachment to a university. A two-factor model was championed over a one-factor model providing evidence of a distinction between university attachment and member attachment. Relationships with external criteria provided further support for this distinction and construct validity evidence. As predicted, "involved" students had practically and statistically significantly higher group attachment than "noninvolved" students. Furthermore, transfer students had practically and statistically significantly lower member attachment than nontransfer students. Additionally, there was a statistically significant positive relationship between students’ perceived cohesion to the university and both group and member attachment. Overall, the authors believe that this is a promising new measure of university attachment.

The Effects of Initially Misclassified Data on the Effectiveness of Discriminant Function Analysis and Finite Mixture Modeling

Holden, J. E., Kelley, K. — Tue, 08 Sep 2009 17:37:44 PDT

Classification procedures are common and useful in behavioral, educational, social, and managerial research. Supervised classification techniques such as discriminant function analysis assume training data are perfectly classified when estimating parameters or classifying. In contrast, unsupervised classification techniques such as finite mixture models (FMM) do not require, or even use if available, knowledge of group status to estimate parameters or classifying. This study investigates the impact of two types of misclassification errors on the classification accuracy of discriminant function analysis (both linear [LDA] and quadratic [QDA]) and FMM for two groups with a single predictor. Analytic and Monte Carlo results are provided for a variety of misclassification scenarios to investigate the performance of the two methods. Discriminant function techniques recovered the highest overall percentages of correctly classified data, whereas FMM captured higher percentages of the smaller group when group sizes are unequal. LDA marginally outperformed QDA under misclassified conditions.

The Validity of the Graduate Record Examination for Master's and Doctoral Programs: A Meta-analytic Investigation

Kuncel, N. R., Wee, S., Serafin, L., Hezlett, S. A. — Fri, 04 Sep 2009 18:07:25 PDT

Extensive research has examined the effectiveness of admissions tests for use in higher education. What has gone unexamined is the extent to which tests are similarly effective for predicting performance at both the master’s and doctoral levels. This study empirically synthesizes previous studies to investigate whether or not the Graduate Record Examination (GRE) predicts the performance of students in master’s programs as well as the performance of doctoral students. Across nearly 100 studies and 10,000 students, this study found that GRE scores predict first year grade point average (GPA), graduate GPA, and faculty ratings well for both master’s and doctoral students, with differences that ranged from small to zero.

A Generalized Model With Internal Restrictions on Item Difficulty for Polytomous Items

Wang, W.-C., Jin, K.-Y. — Fri, 04 Sep 2009 18:07:25 PDT

In this study, the authors extend the standard item response model with internal restrictions on item difficulty (MIRID) to fit polytomous items using cumulative logits and adjacent-category logits. Moreover, the new model incorporates discrimination parameters and is rooted in a multilevel framework. It is a nonlinear mixed model so that existing parameter estimation procedures and computer packages for nonlinear mixed models can be directly adopted to estimate the parameters. Through simulations, it was found that the SAS NLMIXED procedure could recover the parameters fairly well and produce appropriate standard errors, except when the two-parameter adjacent-category logits MIRID was fit to data with a small sample size and a short test length; overall, cumulative logits yielded a better parameter recovery than adjacent-category logits. A real data set about guilt was analyzed with gender as a Level 2 predictor to illustrate applications and applications of the new model. Further model generalization is discussed.

Modeling the Effects of Person Group Factors on Discrimination

Humphry, S. M. — Fri, 04 Sep 2009 18:07:25 PDT

Discrimination has traditionally been parameterized for items but not other empirical factors. Consequently, if person factors affect discrimination they cause misfit. However, by explicitly formulating the relationship between discrimination and the unit of a metric, it is possible to parameterize discrimination for person groups. This article applies the Rasch model with a person group discrimination parameter to demonstrate the empirical effect of a person group factor on the degree of discrimination. The model is applied to the responses of students in different grades of schooling to a reading test, resulting in improved equating and fit of the data to the model. A simulation study confirms the efficacy of the model and tests of fit. Applied implications are discussed.

Assessing Fit and Dimensionality in Least Squares Metric Multidimensional Scaling Using Akaike's Information Criterion

Ding, C. S., Davison, M. L. — Fri, 04 Sep 2009 18:07:24 PDT

Akaike’s information criterion is suggested as a tool for evaluating fit and dimensionality in metric multidimensional scaling that uses least squares methods of estimation. This criterion combines the least squares loss function with the number of estimated parameters. Numerical examples are presented. The results from analyses of both simulation data and real data demonstrate the utility of the Akaike’s information criterion in identifying the best approximating models in multidimensional scaling analyses.

The Mathematics Value Inventory for General Education Students: Development and Initial Validation

Wed, 02 Sep 2009 19:17:24 PDT

The goal of this study was to develop a self-report inventory that measures individual differences in the perceived value of mathematical literacy for general education students. The Mathematics Value Inventory (MVI) is grounded in the Eccles et al. model of achievement-related choices and surveys students’ beliefs in four areas: interest, general utility, need for high achievement, and personal cost. This study describes the development and initial score validation of the MVI. As hypothesized, it was found that (a) MVI scores for students who were not majoring in math did not differ by gender, (b) students who had higher MVI scores had completed more college course work in math than did students with lower scores, and (c) MVI scores were not related to scores on a measure of social desirability.

An Algorithm for Converting Ordinal Scale Measurement Data to Interval/Ratio Scale

Granberg-Rademacker, J. S. — Wed, 02 Sep 2009 19:17:23 PDT

The extensive use of survey instruments in the social sciences has long created debate and concern about validity of outcomes,especially among instruments that gather ordinal-level data. Ordinal-level survey measurement of concepts that could be measured at the interval or ratio level produce errors because respondents are forced to truncate or round off their responses to fit a given ordinal scale. This article presents a Markov chain Monte Carlo modeling technique that converts ordinal measurements to interval/ ratio. Simulated data demonstrate the robustness of this technique,and implications of this technique are discussed.

The Impact of Outliers on Cronbach's Coefficient Alpha Estimate of Reliability: Ordinal/Rating Scale Item Responses

Liu, Y., Wu, A. D., Zumbo, B. D. — Wed, 02 Sep 2009 19:17:23 PDT

In a recent Monte Carlo simulation study, Liu and Zumbo showed that outliers can severely inflate the estimates of Cronbach’s coefficient alpha for continuous item response data—visual analogue response format. Little, however, is known about the effect of outliers for ordinal item response data—also commonly referred to as Likert, Likert-type, ordered categorical, or ordinal/rating scale item responses. Building on the work of Liu and Zumbo, the authors investigated the effects of outlier contamination for binary and ordinal response scales. Their results showed that coefficient alpha estimates were severely inflated with the presence of outliers, and like the earlier findings, the effects of outliers were reduced with increasing theoretical reliability. The efficiency of coefficient alpha estimates (i. e. , sample-to-sample variation) was inflated as well and affected by the number of scale points. It is worth noting that when there were no outliers, the alpha estimates were downward biased because of the ordinal scaling. However, the alpha estimates were, in general, inflated in the presence of outliers leading to positive bias.

Attempted Validation of the Scores of the VARK: Learning Styles Inventory With Multitrait-Multimethod Confirmatory Factor Analysis Models

Leite, W. L., Svinicki, M., Shi, Y. — Mon, 31 Aug 2009 14:00:11 PDT

The authors examined the dimensionality of the VARK learning styles inventory. The VARK measures four perceptual preferences: visual (V), aural (A), read/write (R), and kinesthetic (K). VARK questions can be viewed as testlets because respondents can select multiple items within a question. The correlations between items within testlets are a type of method effect. Four multitrait–multimethod confirmatory factor analysis models were compared to evaluate the dimensionality of the VARK. The correlated trait–correlated method model had the best fit to the VARK scores. The estimated reliability coefficients were adequate. The study found preliminary support for the validity of the VARK scores. Potential problems related to item wording and the scale’s scoring algorithm were identified, and cautions with respect to using the VARK with research were raised.

Homework Purpose Scale for High School Students: A Validation Study

Xu, J. — Mon, 31 Aug 2009 14:00:11 PDT

The purpose of this study is to test the validity of scores on the Homework Purpose Scale using 681 rural and 306 urban high school students. First, confirmatory factor analysis was conducted on the rural sample. The results reveal that the Homework Purpose Scale comprises three separate yet related factors, including Learning-Oriented Reasons, Adult-Oriented Reasons, and Peer-Oriented Reasons. This factor structure is tested with the data from the urban sample. Given an adequate level of configural, factor loading, common error covariance, and intercept invariance, the difference between the group means is further tested. The results reveal that urban high school students, as compared with their rural counterparts, are more likely to do homework for adult-oriented reasons.

The Internal Structure of Positive and Negative Affect: A Confirmatory Factor Analysis of the PANAS

Tuccitto, D. E., Giacobbi, P. R., Leite, W. L. — Mon, 31 Aug 2009 14:00:10 PDT

This study tested five confirmatory factor analytic (CFA) models of the Positive Affect Negative Affect Schedule (PANAS) to provide validity evidence based on its internal structure. A sample of 223 club sport athletes indicated their emotions during the past week. Results revealed that an orthogonal two-factor CFA model,specifying error correlations according to Zevon and Tellegen’s mood content categories, provided the best fit to our data. In addition, parameter estimates for this model suggest that PANAS scores are reliable and explain large proportions of item variance. Taken together with previous research, the findings further suggest that the PANAS may be a higher-order measure of affect and includes several consistently problematic items. The authors recommend that affect researchers attempt to improve the PANAS by (a) revising consistently problematic items, (b) adding new items to better capture mood content categories, and (c) providing additional internal structure validity evidence through a diagonally weighted least squares estimation of a second-order PANAS CFA model.

The Core Self-Evaluation Scale: Further Construct Validation Evidence

Gardner, D. G., Pierce, J. L. — Mon, 31 Aug 2009 14:00:10 PDT

The authors empirically examined two operationalizations of the core self-evaluation construct: (a) the Judge, Erez, Bono, and Thoresen 12-item scale and (b) a composite measure of self-esteem, self-efficacy, locus of control, and neuroticism. The study found that the composite scale relates more strongly than the shorter scale to performance, perceived job complexity, positive affectivity, personal trust, and belief in a just world. However, the short scale performed well and may be more practical in organizational research. The authors conclude that the 12-item measure is better used in research when participant time is limited and that a composite index is better when time is not a constraining factor in the data-collection process.

Factor Invariance Assessment of the Dean-Woodcock Sensory-Motor Battery for Patients With ADHD Versus Nonclinical Subjects

Finch, H., Davis, A., Dean, R. S. — Mon, 31 Aug 2009 14:00:09 PDT

The current study examined the measurement invariance of the Dean–Woodcock Sensory–Motor Battery (DWSMB) for children diagnosed with attention deficit hyperactivity disorder (ADHD) and an age- and gender-matched nonclinical sample. The DWSMB is a promising new instrument for assessing a wide range of cortical and subcortical sensory and motor skills and has been shown to conform to a three-factor latent structure for the nonclinical population. Because this instrument is used with a broad spectrum of individuals, including those with ADHD, it is important to verify that it does indeed measure the same constructs in the same way. The results of the invariance analyses showed that there were specific indicators on the DWSMB that were not invariant between the two groups, though the majority of the measurement model was found to be invariant. Specifically, several tasks that required sustained attention were not as effective for individuals with ADHD as they were for nonclinical subjects. In addition, certain measures of motor coordination were not as strongly associated with the latent variables for those with ADHD as compared with the nonclinical individuals. A number of clinical implications from these findings are discussed in detail.

Development of the Competitive Work Environment Scale: A Multidimensional Climate Construct

Fletcher, T. D., Nusbaum, D. N. — Mon, 31 Aug 2009 14:00:10 PDT

Recent research suggests that competitive work environments may influence individual’s attitudes, behaviors, stress, and performance. Unfortunately, adequate measures of competitive environments are lacking. This article traces the development of a new multidimensional competitive work environment scale. An initial 59-item pool covering five dimensions of a competitive environment was generated and scrutinized. Then, using two independent data samples, the authors psychometrically analyzed, organized, and reduced the items using a combination of exploratory factor analysis, confirmatory factor analysis,and item response theory.The final scale consists of 20 items across five distinct dimensions: competition for tangible rewards, nontangible rewards,recognition,status,and competition influenced by coworkers.Results indicate that the final multidimensional scale maintains strong psychometric properties and provides preliminary evidence of convergent and discriminant validity. Implications and directions for future research are discussed.

Factor Structure Invariance of the Kaufman Adolescent and Adult Intelligence Test Across Male and Female Samples

Immekus, J. C., Maller, S. J. — Thu, 27 Aug 2009 13:57:30 PDT

Multisample confirmatory factor analysis (MCFA) and latent mean structures analysis (LMS) were used to test measurement invariance and latent mean differences on the Kaufman Adolescent and Adult Intelligence Scale^TM(KAIT) across males and females in the standardization sample. MCFA found that the parameters of the KAIT two-factor model were invariant across groups. A follow-up LMS found intercept differences on the Memory for Block Designs, Famous Faces, Auditory Comprehension, and Logical Steps subtests, indicating low to moderately higher expected scores for males.Thus, latent means were not tested for invariance.Although the KAIT two-factor model met partial measurement invariance, it did not demonstrate strong factorial invariance. Implications for test score interpretation are discussed.

Immediate Feedback and Opportunity to Revise Answers to Open-Ended Questions

Attali, Y., Powers, D. — Mon, 02 Mar 2009 20:17:31 PST

Two experiments examine the psychometric effects of providing immediate feedback on the correctness of answers to open-ended questions, and allowing participants to revise their answers following feedback. Participants answering verbal and math questions are able to correct many of their initial incorrect answers, resulting in higher revised scores. In addition, the reliability of these scores is significantly higher than the reliability of scores based on no feedback. Finally, anxiety is significantly lower following a test section with feedback and revision.

Configural, Metric, and Scalar Invariance of the Modified Achievement Goal Questionnaire Across African American and White University Students

Campbell, H. L., Barry, C. L., Joe, J. N., Finney, S. J. — Fri, 23 May 2008 09:01:38 PDT

There has been growing interest in comparing achievement goal orientations across ethnic groups. Such comparisons, however, cannot be made until validity evidence has been collected to support the use of an achievement goal orientation instrument for that purpose. Therefore, this study investigated the measurement invariance of a particular measure of achievement goal orientation, the modified Achievement Goal Questionnaire (AGQ-M), across African American and White university students. Confirmatory factor analyses supported measurement invariance across the two groups. These findings provide additional validity evidence for the newly conceptualized 2 x 2 framework of achievement goal orientation and for the equivalence of functioning of the AGQ-M across these distinct groups. Because this level of invariance was established, researchers can make more valid inferences about differences in the AGQ-M scores across African American and White students.