Towards LLM-Based Automatic Playtest

Abstract

Playtest is the process in which people play a video game for testing. It is critical for the quality assurance of gaming software. Manual playtest is time-consuming and expensive. However, automating this process is challenging, as playtest typically requires for the domain knowledge and problem-solving skills that most conventional testing tools lack. Recent advancements in artificial intelligence (AI) have opened up new possibilities of applying Large Language Models (LLMs) to playtest. However, significant challenges remain: current LLMs cannot visually perceive game environments; most existing research focuses on text-based games or games with robust API; While many non-text games lack APIs to provide textual descriptions of game states, making it almost impossible to naïvely apply LLMs for playtest. This paper introduces Lap, our novel approach of LLM-based Automatic Playtest, which uses ChatGPT to test match-3 games—a category of games where players match three or more identical tiles in a row or column to earn points. Lap encompasses three key phrases: processing of game environments, prompting-based action generation, and action execution. Given a match-3 game, Lap takes a snapshot of the game board and converts it to a numeric matrix; it then prompts ChatGPT-O1-mini API to suggest moves based on that matrix; finally, Lap tentatively applies the suggested moves to earn points and trigger changes in the game board. It repeats the above-mentioned three steps iteratively until timeout. For evaluation, we conducted a case study by applying Lap to an open-source match-3 game—CasseBonbons, and empirically compared Lap with three existing tools. Our results are promising: Lap outperformed existing tools by achieving higher code coverage and triggering more program crashes. Our research will shed light on future research of automatic testing and LLM application.