An Exploration of Generative AI in Engineering Education and Research

Abstract

Generative artificial intelligence (GAI) in engineering education and research is explored in this dissertation, specifically focusing on performing thematic analysis with the aid of large language models (LLMs) and natural language processing (NLP) tools, detecting figurative language in faculty discourse about assessment, and engineering faculty members' thinking on reshaping assessment practices due to the increased availability of GAI tools. Through three interrelated manuscripts, my research demonstrates the application of LLMs and natural NLP techniques to traditional qualitative methodologies. The first manuscript provides a roadmap for researchers to adapt a traditional thematic analysis method, incorporating LLMs and NLP tools for the identification of key themes with a human-in-the-loop approach. The second manuscript investigates engineering faculty members' use of figurative language when discussing assessment, exploring the identification and evaluation of figurative language through a GAI-supported analytical process. The third manuscript examines faculty members' evolving perspectives on assessment due to the increasing availability of GAI tools. The dissertation illustrated that LLMs and NLP can be used to perform common steps in some forms of qualitative data analysis, maintaining ethical standards and producing meaningful findings while keeping human researchers at the center of this analysis. Limitations such as variable detection accuracy, potential biases, and insufficient cultural context understanding highlight the necessity of human-in-the-loop validation processes. This dissertation underscores the importance of integrating human oversight with the current capabilities of GAI technologies to enhance qualitative research methodologies and advance understanding of faculty perspectives on assessment in engineering education. Moreover, I explore how faculty members have been reshaping their assessment with the availability of GAI technologies in education. Common concerns included issues related to academic integrity, the authenticity of student work, and GAI's ability to engage meaningfully with disciplinary content. Although faculty perspectives varied, there was an overall cautious approach to integrating GAI into assessment practices, with only a small percentage of faculty member participants reporting significant changes in their views over time.