Failure to Reject the p-value is Not the Same as Accepting it: The Development, Validation, and Administration of the KPVMI Instrument
Files
TR Number
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The purpose of this study was to investigate on a national scale the baseline level of p-value fluency of future researchers (i.e., doctoral students). To that end, two research questions were investigated. The first research question, Can a sufficiently reliable and valid measure of p-value misinterpretations (in a research context) be constructed?, was addressed via the development and validation of the Keller P-value Misinterpretation Inventory instrument (KPVMI). An iterative process of expert review, pilot testing, and field testing resulted in an adequately reliable measure (Alpha = .8030) of p-value fluency as assessed across 18 misinterpretations and 2 process levels as well as an independently validated sub-measure of p-value fluency in context as assessed across 18 misinterpretations (Alpha = .8298). The second research question, What do the results of the KPVMI administration tell us about the current level of p-value fluency among doctoral students nationally?, was addressed via analysis of a subset of the field test data (n = 147) with respect to performance on the subset of items considered sufficiently validated as developed in Phases I-III (KPVMI-1). The median score was 10/18 items answered correctly indicating that future researchers on the aggregate struggle to properly interpret and report p-values in context; furthermore, there was insufficient evidence to indicate training and experience are positively correlated with performance. These results aligned with the extant literature regarding the p-value misinterpretations of practicing researchers.