Comparison of Traditional and Activity Theory Based Analysis Methods for Verbal Protocol Data

Abstract

The think aloud method has been used in this research to generate data that reveals the thoughts of participants of a study while they are performing tasks. The pioneers of this method, Simon and Ericsson, have provided a method to analyze the data so as to obtain meaningful results. However, this analysis method is complicated and time consuming. Most researchers use some form of categorization to perform their analysis. Critical incidents were used to categorize the data gathered in the tests conducted as part of this research. This research proposed the use of tenets of Activity Theory while performing data analysis so that the cultural and environmental aspects that influence task performance are identified and addressed as part of the analysis. A data analysis template was created that directs the analyst to follow activity theory while performing the analysis. Sample data was gathered using the Think Aloud Method. The results obtained after analyzing this data using the proposed Activity Theory Based method were compared with those obtained when the same data was analyzed using a representative traditional method of analysis. The research included positive critical incidents, negative critical incidents and level of severity of negative critical incidents as the dependent measures. No significant differences were found between the two methods based on these dependent measures. Task type had a significant effect on the number of positive and negative critical incidents identified.