Detecting Rater Centrality Effect Using Simulation Methods and Rasch Measurement Analysis
| dc.contributor.author | Yue, Xiaohui | en |
| dc.contributor.committeecochair | Wolfe, Edward W. | en |
| dc.contributor.committeecochair | Skaggs, Gary E. | en |
| dc.contributor.committeemember | Creamer, Elizabeth G. | en |
| dc.contributor.committeemember | Miyazaki, Yasuo | en |
| dc.contributor.department | Educational Leadership and Policy Studies | en |
| dc.date.accessioned | 2014-03-14T20:14:23Z | en |
| dc.date.adate | 2011-09-01 | en |
| dc.date.available | 2014-03-14T20:14:23Z | en |
| dc.date.issued | 2011-07-14 | en |
| dc.date.rdate | 2011-09-01 | en |
| dc.date.sdate | 2011-07-27 | en |
| dc.description.abstract | This dissertation illustrates how to detect the rater centrality effect in a simulation study that approximates data collected in large scale performance assessment settings. It addresses three research questions that: (1) which of several centrality-detection indices are most sensitive to the difference between effect raters and non-effect raters; (2) how accurate (and inaccurate), in terms of Type I error rate and statistical power, each centrality-detection index is in flagging effect raters; and (3) how the features of the data collection design (i.e., the independent variables including the level of centrality strength, the double-scoring rate, and the number of raters and ratees) influence the accuracy of rater classifications by these centrality-detection indices. The results reveal that the measure-residual correlation, the expected-residual correlation, and the standardized deviation of assigned scores perform better than the point-measure correlation. The mean-square fit statistics, traditionally viewed as potential indicators of rater centrality, perform poorly in terms of differentiating central raters from normal raters. Along with the rater slope index, the mean-square fit statistics did not appear to be sensitive to the rater centrality effect. All of these indices provided reasonable protection against Type I errors when all responses were double scored, and that higher statistical power was achieved when responses were 100% double scored in comparison to only 10% being double scored. With a consideration on balancing both Type I error and statistical power, I recommend the measure-residual correlation and the expected-residual correlation for detecting the centrality effect. I suggest using the point-measure correlation only when responses are 100% double scored. The four parameters evaluated in the experimental simulations had different impact on the accuracy of rater classification. The results show that improving the classification accuracy for non-effect raters may come at a cost of reducing the classification accuracy for effect raters. Some simple guidelines for the expected impact of classification accuracy when a higher-order interaction exists summarized from the analyses offer a glimpse of the "pros" and "cons" in adjusting the magnitude of the parameters when we evaluate the impact of the four experimental parameters on the outcomes of rater classification. | en |
| dc.description.degree | Ph. D. | en |
| dc.identifier.other | etd-07272011-104720 | en |
| dc.identifier.sourceurl | http://scholar.lib.vt.edu/theses/available/etd-07272011-104720/ | en |
| dc.identifier.uri | http://hdl.handle.net/10919/28423 | en |
| dc.publisher | Virginia Tech | en |
| dc.relation.haspart | Yue_X_D_2011.pdf | en |
| dc.rights | In Copyright | en |
| dc.rights.uri | http://rightsstatements.org/vocab/InC/1.0/ | en |
| dc.subject | ANOVA | en |
| dc.subject | Rasch measurement | en |
| dc.subject | centrality | en |
| dc.subject | rater effects | en |
| dc.subject | Type I and Type II errors | en |
| dc.subject | performance assessment | en |
| dc.subject | statistical power | en |
| dc.subject | logistic regression | en |
| dc.title | Detecting Rater Centrality Effect Using Simulation Methods and Rasch Measurement Analysis | en |
| dc.type | Dissertation | en |
| thesis.degree.discipline | Educational Leadership and Policy Studies | en |
| thesis.degree.grantor | Virginia Polytechnic Institute and State University | en |
| thesis.degree.level | doctoral | en |
| thesis.degree.name | Ph. D. | en |
Files
Original bundle
1 - 1 of 1