A multi-dimensional index of evaluating systems thinking skills from textual data

Abstract

Systems thinking (ST) includes a set of critical skills and approaches for addressing today's complex societal problems. Therefore, it has been introduced into the curricula of many educational programmes around the world. Despite all the attention to ST, there is less consensus when it comes to evaluating and assessing ST skills. Particularly, a quantitative assessment approach that captures ST's multi-dimensionality is crucial when evaluating the degree to which one has learned and utilizes ST. This paper proposes a systematic approach to create such a multi-dimensional Index of ST from textual data. Initially, we provide an overview of the theoretical background as it pertains to different measurement approaches of ST skills. Then we provide a conceptual framework based on ST skill measures and transform this framework into a quantifiable model. Finally, using student data, we provide an illustration of an integrated index of ST skills. We compute this index by using a mixed methods approach, including robust principal component analysis, data envelopment analysis and two-staged bootstrapping approach. The results show that (i) our model serves as a systematic multi-dimensional ST approach by including multiple measures of ST skills and (ii) international students and self-reported math skills are found as significant predictors of one's level of ST in the graduate student dataset (N = 30), however no significant factors are found in the first-year engineering student dataset (N = 144).