Characterizing Student Attention in Technology-Infused Classrooms Using Real-time Active Window Data

As computers become more prevalent (and required) in engineering classrooms, it becomes increasingly important to address the dichotomy in our current understanding of their impact on student attention and learning. While some researchers report increased student learning, others report computers as a distraction to learning. To address this conflict, the research community must gain a fundamental understanding of how students use their computers in-class and how student attention is connected to learning and pedagogical practice. By gaining such an understanding, instructors' design of classroom interventions aimed at increasing positive computer usage will be better informed. The purpose of this quantitative research study is to answer the overarching question "How do students use computers in technology-infused classrooms?" through an investigation of student attention. Based on the premise that one's senses must be oriented towards a stimulus to receive the stimulus, it is hypothesized that attention in a technology-infused classroom can be measured by monitoring a students' top-most, active window (the Active Window Method). This novel approach mitigates issues with prior data collection methods, and allows researchers the opportunity to capture real-time student computer usage. This research serves the dual purpose of validating the Active Window Method and investigating applications of the method. The Active Window Method is validated by comparing real-time active window data with in-class observations of attention in engineering courses with large enrollments. The bootstrap resampling technique is used to estimate mean error rate. Post-tests are used to establish convergent validity by relating learning to active window data. Polytomous logistic regression is used to examine the probability of post-test score (response) over the range of attention levels (factor). Subsequent to validation, two applications of the Active Window Method were pursued. First, student computer use is characterized in multiple large lecture sections. Second, in answering calls to link student computer usage to pedagogical practices, an investigation into the relationship between pedagogy and attention is conducted by aligning time stamps of the active window record with technology-infused pedagogical activities identified in video recordings of lectures. An intervention time series analysis is employed to quantify the change in average attention due to pedagogical activities. Results demonstrate strong construct validity when directly comparing active window and attention. Convergent validity was weak when relating active window to learning. Results from the two applications illustrate that instructors' use of technology and their pedagogical practices impact student computer use. Specifically, collecting student-generated content and polling question activities encourage on-task behavior. However, activities that include a website link encourage off-task behavior.