Diagnostic Modeling of Intra-Organizational Mechanisms for Supporting Policy Implementation

The Virginia Guidelines for Uniform Performance Standards and Evaluation Criteria for Teachers represented a significant overhaul of conventional teacher evaluation criteria in Virginia. The policy outlined seven performance standards by which all Virginia teachers would be evaluated. This study explored the application of cognitive diagnostic modeling to measure teachers' perceptions of intra-organizational mechanisms available to support educational professionals in implementing this policy.

It was found that a coarse-grained, four-attribute compensatory, re-parameterized unified model (C-RUM) fit teacher perception data better and had lower standard errors than the competing finer-grained models. The Q-matrix accounted for the complex loadings of items to the four theoretically and empirically driven mechanisms of implementation support including characteristics of the policy, teachers, leadership, and the organization. The mechanisms were positively, significantly, and moderately correlated which suggested that each mechanism captured a different, yet related, component of policy implementation support. The diagnostic profile estimates indicated that the majority of teachers perceived support on items relating to "characteristics of teachers." Moreover, almost 60% of teachers were estimated to belong to profiles with perceived support on "characteristics of the policy." Finally, multiple group multinomial log-linear models (Xu and Von Davier, 2008) were used to analyze the data across subjects, grade levels, and career status. There was lower perceived support by STEM teachers than non-STEM teachers who have the same profile, suggesting that STEM teachers required differential support than non-STEM teachers.

The precise diagnostic feedback on the implementation process provided by this application of diagnostic models will be beneficial to policy makers and educational leaders. Specifically, they will be better prepared to identify strengths and weaknesses and target resources for a more efficient, and potentially more effective, policy implementation process. It is assumed that when equipped with more precise diagnostic feedback, policy makers and school leaders may be able to more confidently engage in empirical decision making, especially in regards to targeting resources for short-term and long-term organizational goals subsumed within the policy implementation initiative.