Much Ado About Immersion: Power, Reported Results, and the Validity of Research on the Psychology of Virtual Reality and Immersive Simulations

Abstract

Virtual reality (VR) technology has permeated consumer culture in recent years, consequentially inspiring a hotbed of interdisciplinary academic VR research to better understand its effects as a medium. It has become a popular subject of study in fields as varied as engineering, computer science, communication, and psychology. The present study evaluates methodological trends in behavioral research on VR in terms of best practices regarding data collection, reporting, and availability. A meta-scientific content analysis of 61 articles focused on power, p-values, reporting errors, and transparency of data, all of which respectively represent four stages of research: data collection, analysis, reporting, and sharing. The findings from 1,122 statistical tests show that there is room for improvement in much behavioral research on VR in terms of methodological trends regarding number of participants, reporting of results, and data availability. Although no firm conclusions can be drawn about the presence of p-hacking or other questionable research practices (QRPs), the present study demonstrates that chronically small sample sizes, instances of errors in reporting, and a lack of transparent supplemental data are evident. The trends observed are broad, yet informative, and further research in this area is crucial. Methodological recommendations are made for future research dealing with VR applications, particularly given the potential social and cultural impact of the technology.