Scholarly Works, Computer Science

Permanent URI for this collection

Research articles, presentations, and other scholarship

Browse

Recent Submissions

Now showing 1 - 20 of 707
  • Optimizing Effectiveness and Defense of Drone Surveillance Missions via Honey Drones
    Wan, Zelin; Cho, Jin-Hee; Zhu, Mu; Anwar, Ahmed; Kamhoua, Charles; Singh, Munindar (ACM, 2024)
    This work aims to develop a surveillance mission system using unmanned aerial vehicles (UAVs) or drones when Denial-of-Service (DoS) attacks are present to disrupt normal operations for mission systems. In particular, we introduce the concept of cyber deception using honey drones (HDs) to protect the mission system from DoS attacks. HDs exhibit fake vulnerabilities and employ stronger signal strengths to lure DoS attacks, unlike the legitimate drones called mission drones (MDs) deployed for mission execution. This research formulates an optimization problem to identify an optimal set of signal strengths of HDs and MDs to best prevent the system from DoS attacks while maximizing mission performance under the resource constraints of UAVs. To solve this optimization problem, we leverage deep reinforcement learning (DRL) to achieve these multiple objectives of the mission system concerning system security and performance. Particularly, for efficient and effective parallel processing in DRL, we utilize a DRL algorithm called the Asynchronous Advantage Actor-Critic (A3C) algorithm to model attack-defense interactions. We employ a physical engine-based simulation testbed to consider realistic scenarios and demonstrate valid findings from the realistic testbed. The extensive experiments proved that our HD-based approach could achieve up to a 32% increase in mission completion, a 20% reduction in energy consumption, and a 62% decrease in attack success rates compared to existing defense strategies.
  • XplainScreen: Unveiling the Black Box of Graph Neural Network Drug Screening Models with a Unified XAI Framework
    Ahn, Geonhee; Haque, Md Mahim Anjum; Hazarika, Subhashis; Kim, Soo Kyung (ACM, 2024-10-21)
    Despite the powerful capabilities of GNN-based drug screening model in predicting target drug properties, the black-box nature of these models poses a challenge for practical application, particularly in a field as critical as drug development where understanding and trust in AI-driven decisions are important. To address the interpretability issues associated with GNN-based virtual drug screening, we introduce XplainScreen: a unified explanation framework designed to evaluate various explanation methods for GNN-based models. XplainScreen offers a user-friendly, web-based interactive platform that allows for the selection of specific GNN-based drug screening models and multiple cutting-edge explainable AI methods. It supports both qualitative assessments (through visualization and generative text descriptions) and quantitative evaluations of these methods, utilizing drug molecules in SMILES format. This demonstration showcases the utility of XplainScreen through a user study with pharmacological researchers focused on virtual screening tasks based on toxicity, highlighting the framework’s potential to enhance the integrity and trustworthiness of AI-driven virtual drug screening. A video demo of XplainScreen is available at https://youtu.be/Q4yobrTLKec, and the source code can be accessed at https://github.com/GeonHeeAhn/XplainScreen.
  • Hermes: Boosting the Performance of Machine-Learning-Based Intrusion Detection System through Geometric Feature Learning
    Zhang, Chaoyu; Shi, Shanghao; Wang, Ning; Xu, Xiangxiang; Li, Shaoyu; Zheng, Lizhong; Marchany, Randy; Gardner, Mark; Hou, Y. Thomas; Lou, Wenjing (ACM, 2024-10-14)
    Anomaly-Based Intrusion Detection Systems (IDSs) have been extensively researched for their ability to detect zero-day attacks. These systems establish a baseline of normal behavior using benign traffic data and flag deviations from this norm as potential threats. They generally experience higher false alarm rates than signature-based IDSs. Unlike image data, where the observed features provide immediate utility, raw network traffic necessitates additional processing for effective detection. It is challenging to learn useful patterns directly from raw traffic data or simple traffic statistics (e.g., connection duration, package inter-arrival time) as the complex relationships are difficult to distinguish. Therefore, some feature engineering becomes imperative to extract and transform raw data into new feature representations that can directly improve the detection capability and reduce the false positive rate. We propose a geometric feature learning method to optimize the feature extraction process. We employ contrastive feature learning to learn a feature space where normal traffic instances reside in a compact cluster. We further utilize H-Score feature learning to maximize the compactness of the cluster representing the normal behavior, enhancing the subsequent anomaly detection performance. Our evaluations using the NSL-KDD and N-BaloT datasets demonstrate that the proposed IDS powered by feature learning can consistently outperform state-of-the-art anomaly-based IDS methods by significantly lowering the false positive rate. Furthermore, we deploy the proposed IDS on a Raspberry Pi 4 and demonstrate its applicability on resource-constrained Internet of Things (IoT) devices, highlighting its versatility for diverse application scenarios.
  • VizGroup: An AI-assisted Event-driven System for Collaborative Programming Learning Analytics
    Tang, Xiaohang; Wong, Sam; Pu, Kevin; Chen, Xi; Yang, Yalong; Chen, Yan (ACM, 2024-10-13)
    Programming instructors often conduct collaborative learning activities, like Peer Instruction, to foster a deeper understanding in students and enhance their engagement with learning. These activities, however, may not always yield productive outcomes due to the diversity of student mental models and their ineffective collaboration. In this work, we introduce VizGroup, an AI-assisted system that enables programming instructors to easily oversee students’ real-time collaborative learning behaviors during large programming courses. VizGroup leverages Large Language Models (LLMs) to recommend event specifications for instructors so that they can simultaneously track and receive alerts about key correlation patterns between various collaboration metrics and ongoing coding tasks. We evaluated VizGroup with 12 instructors in a comparison study using a dataset collected from a Peer Instruction activity that was conducted in a large programming lecture. The results showed that VizGroup helped instructors effectively overview, narrow down, and track nuances throughout students’ behaviors.
  • Evaluating Layout Dimensionalities in PC+VR Asymmetric Collaborative Decision Making
    Enriquez, Daniel; Tong, Wai; North, Christopher L.; Qu, Huamin; Yang, Yalong (ACM, 2024-10-20)
    With the commercialization of virtual/augmented reality (VR/AR) devices, there is an increasing interest in combining immersive and non-immersive devices (e.g., desktop computers) for asymmetric collaborations. While such asymmetric settings have been examined in social platforms, significant questions around layout dimensionality in data-driven decision-making remain underexplored. A crucial inquiry arises: although presenting a consistent 3D virtual world on both immersive and non-immersive platforms has been a common practice in social applications, does the same guideline apply to lay out data? Or should data placement be optimized locally according to each device's display capacity? This study aims to provide empirical insights into the user experience of asymmetric collaboration in data-driven decision-making. We tested practical dimensionality combinations between PC and VR, resulting in three conditions: PC2D+VR2D, PC2D+VR3D, and PC3D+VR3D. The results revealed a preference for PC2D+VR3D, and PC2D+VR2D led to the quickest task completion. Our investigation facilitates an in-depth discussion of the trade-offs associated with different layout dimensionalities in asymmetric collaborations.
  • Causality-Aware Spatiotemporal Graph Neural Networks for Spatiotemporal Time Series Imputation
    Jing, Baoyu; Zhou, Dawei; Ren, Kan; Yang, Carl (ACM, 2024-10-21)
    Spatiotemporal time series are usually collected via monitoring sensors placed at different locations, which usually contain missing values due to various failures, such as mechanical damages and Internet outages. Imputing the missing values is crucial for analyzing time series. When recovering a specific data point, most existing methods consider all the information relevant to that point regardless of the cause-and-effect relationship. During data collection, it is inevitable that some unknown confounders are included, e.g., background noise in time series and non-causal shortcut edges in the constructed sensor network. These confounders could open backdoor paths and establish non-causal correlations between the input and output. Over-exploiting these non-causal correlations could cause overfitting. In this paper, we first revisit spatiotemporal time series imputation from a causal perspective and show how to block the confounders via the frontdoor adjustment. Based on the results of frontdoor adjustment, we introduce a novel Causality- Aware Spatiotemporal Graph Neural Network (Casper), which contains a novel Prompt Based Decoder (PBD) and a Spatiotemporal Causal Attention (SCA). PBD could reduce the impact of confounders and SCA could discover the sparse causal relationships among embeddings. Theoretical analysis reveals that SCA discovers causal relationships based on the values of gradients. We evaluate Casper on three real-world datasets, and the experimental results show that Casper could outperform the baselines and could effectively discover the causal relationships.
  • An Exploratory Mixed-methods Study on General Data Protection Regulation (GDPR) Compliance in Open-Source Software
    Franke, Lucas; Liang, Huayu; Farzanehpour, Sahar; Brantly, Aaron F.; Davis, James C.; Brown, Chris (ACM, 2024-10-24)
    Background: Governments worldwide are considering data privacy regulations. These laws, such as the European Union’s General Data Protection Regulation (GDPR), require software developers to meet privacy-related requirements when interacting with users’ data. Prior research describes the impact of such laws on software development, but only for commercial software. Although opensource software is commonly integrated into regulated software, and thus must be engineered or adapted for compliance, we do not know how such laws impact open-source software development. Aims: To understand how data privacy laws affect open-source software (OSS) development, we focus on the European Union’s GDPR, as it is the most prominent such law. We investigated how GDPR compliance activities influence OSS developer activity (RQ1), how OSS developers perceive fulfilling GDPR requirements (RQ2), the most challenging GDPR requirements to implement (RQ3), and how OSS developers assess GDPR compliance (RQ4). Method:We distributed an online survey to explore perceptions of GDPR implementations from open-source developers (N=56). To augment this analysis, we further conducted a repository mining study to analyze development metrics on pull requests (N=31,462) submitted to open-source GitHub repositories. Results: Our results suggest GDPR policies complicate OSS development and introduce challenges, primarily regarding the management of users’ data, implementation costs and time, and assessments of compliance. Moreover, we observed negative perceptions of the GDPR from OSS developers and significant increases in development activity, in particular metrics related to coding and reviewing, on GitHub pull requests related to GDPR compliance. Conclusions: Our findings provide future research directions and implications for improving data privacy policies, motivating the need for relevant resources and automated tools to support data privacy regulation implementation and compliance efforts in OSS.
  • Goldilocks Zoning: Evaluating a Gaze-Aware Approach to Task-Agnostic VR Notification Placement
    Ilo, Cory; DiVerdi, Stephen; Bowman, Douglas A. (ACM, 2024-10-07)
    While virtual reality (VR) offers immersive experiences, users need to remain aware of notifications from outside VR. However, inserting notifications into a VR experience can result in distraction or breaks in presence, since existing notification systems in VR use static placement and lack situational awareness. We address this challenge by introducing a novel notification placement technique, Goldilocks Zoning (GZ), which leverages a 360-degree heatmap generated using gaze data to place notifications near salient areas of the environment without obstructing the primary task. To investigate the effectiveness of this technique, we conducted a dualtask experiment comparing GZ to common notification placement techniques. We found that GZ had similar performance to state-ofthe- art techniques in a variety of primary task scenarios. Our study reveals that no single technique is universally optimal in dynamic settings, underscoring the potential for adaptive approaches to notification management. As a step in this direction, we explored the potential to use machine learning to predict the task based on the gaze heatmap.
  • Breaking Privacy in Model-Heterogeneous Federated Learning
    Haldankar, Atharva; Riasi, Arman; Nguyen, Hoang-Dung; Phuong, Tran; Hoang, Thang (ACM, 2024-09-30)
    Federated learning (FL) allows multiple distrustful clients to collaboratively train a machine learning model. In FL, data never leaves client devices; instead, clients only share locally computed gradients with a central server. As individual gradients may leak information about a given client’s dataset, secure aggregation was proposed. With secure aggregation, the server only receives the aggregate gradient update from the set of all sampled clients without being able to access any individual gradient. One challenge in FL is the systemslevel heterogeneity that is quite often present among client devices. Specifically, clients in the FL protocol may have varying levels of compute power, on-device memory, and communication bandwidth. These limitations are addressed by model-heterogeneous FL schemes, where clients are able to train on subsets of the global model. Despite the benefits of model-heterogeneous schemes in addressing systems-level challenges, the implications of these schemes on client privacy have not been thoroughly investigated. In this paper, we investigate whether the nature of model distribution and the computational heterogeneity among client devices in model-heterogeneous FL schemes may result in the server being able to recover sensitive data from target clients. To this end, we propose two attacks in the model-heterogeneous FL setting, even with secure aggregation in place. We call these attacks the Convergence Rate Attack and the Rolling Model Attack. The Convergence Rate Attack targets schemes where clients train on the same subset of the global model, while the Rolling Model Attack targets schemes where model parameters are dynamically updated each round. We show that a malicious adversary can compromise the model and data confidentiality of a target group of clients. We evaluate our attacks on the MNIST and CIFAR-10 datasets and show that using our techniques, an adversary can reconstruct data samples with near perfect accuracy for batch sizes of up to 20 samples.
  • An empirical study to understand how students use ChatGPT for writing essays and how it affects their ownership
    Jelson, Andrew; Lee, Sang Won (ACM, 2024-05-11)
    As large language models (LLMs) become more powerful and ubiquitous, systems like ChatGPT are increasingly used by students to help them with writing tasks. To better understand how these tools are used, we investigate how students might use an LLM for essay writing, for example, to study the queries asked to ChatGPT and the responses that ChatGPT gives. To that end, we plan to conduct a user study that will record the user writing process and present them with the opportunity to use ChatGPT as an AI assistant. This study’s findings will help us understand how these tools are used and how practitioners — such as educators and essay readers — should consider writing education and evaluation based on essay writing.
  • An Empirical Study on Current Practices and Challenges of Core AR/VR Developers
    Bose, Dibyendu Brinto; Brown, Chris (ACM, 2024-10-27)
    Augmented reality (AR) and virtual reality (VR) applications are increasingly integral to modern society. Core AR/VR developers, pivotal in crafting these advanced technologies, face significant challenges throughout the software development lifecycle. In this context, ‘core AR/VR developers’ refers to professionals who actively engage in developing AR/VR technologies, including researchers and developers. We surveyed such professionals to directly understand these challenges and received 48 responses. Our findings categorize the unique challenges into three major stages of SDLC - Design, Implementation, Testing that core AR/VR developers pointed out. These challenges include creating immersive experiences, complexity in 3D interaction, cross-platform compatibility, and reproducing bugs. This study highlights significant AR/VR development obstacles and provides foundational insights for future research to improve development practices and tools in this rapidly evolving field.
  • How Do Developers Reuse StackOverflow Answers in Their GitHub Projects?
    Chen, Juntong; Zhao, Yan; Meng, Na (ACM, 2024-10-27)
    StackOverflow (SO) is a widely used question-and-answer (Q&A) website for software developers and computer scientists. GitHub is an online development platform used for storing, tracking, and collaborating on software projects. Prior work relates the information mined from both platforms without carefully inspecting the answer-reuse practices. For this paper, we did an empirical study by mining the SO answers reused by Java projects available on GitHub. We created a hybrid approach of clone detection, keyword-based search, and manual inspection, to identify the answer(s) actually used by developers. Based on those answers, we studied topics of the discussion threads, answer characteristics (e.g., scores, ages, code lengths, and text lengths), and developers’ reuse practices. We observed that most reused answers offer programs to implement specific coding tasks. Among all analyzed SO discussion threads, the reused answers often have higher scores, older ages, longer code, and longer text than unused answers. In only 9% of scenarios (40/430), developers fully copied answer code for reuse. In the remaining scenarios, they reused partial code or created brand new code from scratch. Our study characterized 130 SO discussion threads referred to by Java developers in 357 GitHub projects. Our observations can guide SO answerers to provide better answers, and shed lights on future human-centric research that creates better tools to help with code reuse.
  • SparseAuto: An Auto-scheduler for Sparse Tensor Computations using Recursive Loop Nest Restructuring
    Dias, Adhitha; Anderson, Logan; Sundararajah, Kirshanthan; Pelenitsyn, Artem; Kulkarni, Milind (ACM, 2024-10-08)
    Automated code generation and performance enhancements for sparse tensor algebra have become essential in many real-world applications, such as quantum computing, physical simulations, computational chemistry, and machine learning. General sparse tensor algebra compilers are not always versatile enough to generate asymptotically optimal code for sparse tensor contractions. This paper shows how to generate asymptotically better schedules for complex sparse tensor expressions using kernel fission and fusion. We present generalized loop restructuring transformations to reduce asymptotic time complexity and memory footprint. Furthermore, we present an auto-scheduler that uses a partially ordered set (poset)-based cost model that uses both time and auxiliary memory complexities to prune the search space of schedules. In addition, we highlight the use of Satisfiability Module Theory (SMT) solvers in sparse auto-schedulers to approximate the Pareto frontier of better schedules to the smallest number of possible schedules, with user-defined constraints available at compile-time. Finally, we show that our auto-scheduler can select better-performing schedules and generate code for them. Our results show that the auto-scheduler provided schedules achieve orders-of-magnitude speedup compared to the code generated by the Tensor Algebra Compiler (TACO) for several computations on different real-world tensors.
  • Seed Grant Programs to Promote Community Transformation in Higher Education Institutions
    Fleming, Gabriella Coloyan; Cobb, Sydni Alexa; Watson, Del; Boklage, Audrey; Borrego, Maura; Contreras, Lydia; Julien, Christine (MDPI, 2024-10-16)
    Used in higher education for many decades, seed grants are now beginning to be applied as a strategy to advance diversity, equity and inclusion goals, including rebuilding community post-pandemic. There is little research on the effectiveness of seed grants for such communal goals. This work is innovative in two key ways. First, these seed grants focus on promoting a strong sense of community at the institution rather than promoting individual investigators and research projects. Second, engaging students and staff as principal investigators (PIs) disrupts power structures in the academy. We present a systematic analysis of seed grant project reports (n = 45) and survey data (n = 56) from two seed grant programs implemented at the same institution. A diverse set of projects was proposed and funded. Projects had a positive impact on awardees and their departments and colleges. Seed grant program activities were successful at building community among awardees and recognizing individual efforts. Most noteworthy are the career development opportunities for graduate students, postdocs and staff, which are afforded by changes to PI eligibility. We conclude that seed grant programs have the potential for organizational learning and change around community building in higher education.
  • Ajna: A Wearable Shared Perception System for Extreme Sensemaking
    Wilchek, Matthew; Luther, Kurt; Batarseh, Feras A. (ACM, 2024)
    This paper introduces the design and prototype of Ajna, a wearable shared perception system for supporting extreme sensemaking in emergency scenarios. Ajna addresses technical challenges in Augmented Reality (AR) devices, specifically the limitations of depth sensors and cameras. These limitations confine object detection to close proximity and hinder perception beyond immediate surroundings, through obstructions, or across different structural levels, impacting collaborative use. It harnesses the Inertial Measurement Unit (IMU) in AR devices to measure users? relative distances from a set physical point, enabling object detection sharing among multiple users across obstacles like walls and over distances. We tested Ajna's effectiveness in a controlled study with 15 participants simulating emergency situations in a multi-story building. We found that Ajna improved object detection, location awareness, and situational awareness, and reduced search times by 15%. Ajna's performance in simulated environments highlights the potential of artificial intelligence (AI) to enhance sensemaking in critical situations, offering insights for law enforcement, search and rescue, and infrastructure management.
  • Toward an Edge-Friendly Distributed Object Store for Serverless Functions
    Chen, Xin; Paidiparthy, Manoj Prabhakar; Hu, Liting (ACM, 2024-09-04)
    Serverless computing is changing the way in which we structure and deploy computations in Internet-scale edge systems. This paper presents Capybara, a new scalable and programmable distributed object store for storing and sharing serverless function data objects (state) on edge infrastructures. The key innovations here are (1) achieving scalability and avoiding the significant DRAM cost through a consistent DHT-based P2P architecture; and (2) providing a programmable handler abstraction to customize state management policies (e.g., data caching policies, container “keepalive” times, access control methods, and data replication policies). We implement Capybara prototype on the Pastry DHT, deploy it on 150 Amazon EC2 nodes, and evaluate it by building several use cases to conduct real-world experiments, demonstrating its significant gains in data locality, state management customization, and scalability compared to the state-of-the-art.
  • ShouldAR: Detecting Shoulder Surfing Attacks Using Multimodal Eye Tracking and Augmented Reality
    Corbett, Matthew; David-John, Brendan; Shang, Jiacheng; Ji, Bo (ACM, 2024-09-09)
    Shoulder surfing attacks (SSAs) are a type of observation attack designed to illicitly gather sensitive data from "over the shoulder' of victims. This attack can be directed at mobile devices, desktop screens, Personal Identification Number (PIN) pads at an Automated Teller Machine (ATM), or written text. Existing solutions are generally focused on authentication techniques (e.g., logins) and are limited to specific attack scenarios (e.g., mobile devices or PIN Pads). We present ShouldAR, a mobile and usable system to detect SSAs using multimodal eye gaze information (i.e., from both the potential attacker and victim). ShouldAR uses an augmented reality headset as a platform to incorporate user eye gaze tracking, rear-facing image collection and eye gaze analysis, and user notification of potential attacks. In a 24-participant study, we show that the prototype is capable of detecting 87.28% of SSAs against both physical and digital targets, a two-fold improvement on the baseline solution using a rear-facing mirror, a widely used solution to the SSA problem. The ShouldAR approach provides an AR-based, active SSA defense that applies to both digital and physical information entry in sensitive environments.
  • Toward Declarative Auditing of Java Software for Graceful Exception Handling
    St. Amour, Leo; Tilevich, Eli (ACM, 2024-09-13)
    Despite their language-integrated design, Java exceptions can be difficult to use effectively. Although Java exceptions are syntactically straightforward, negligent practices often result in code logic that is not only inelegant but also unsafe. This paper explores the challenge of auditing Java software to enhance the effectiveness and safety of its exception logic. We revisit common anti-patterns associated with Java exception usage and argue that, for auditing, their detection requires a more nuanced approach than mere identification. Specifically, we investigate whether reporting such anti-patterns can be prioritized for subsequent examination. We prototype our approach as Händel, in which antipatterns and their priority, or weight, are expressed declaratively using probabilistic logic programming. Evaluation with representative open-source code bases suggests Händel’s promise in detecting, reporting, and ranking the antipatterns, thus helping streamline Java software auditing to ensure the safety and quality of exception-handling logic.
  • Epigenomic tomography for probing spatially defined chromatin state in the brain
    Liu, Zhengzhi; Deng, Chengyu; Zhou, Zirui; Ya, Xiao; Jiang, Shan; Zhu, Bohan; Naler, Lynette B.; Jia, Xiaoting; Yao, Danfeng (Daphne); Lu, Chang (Cell Press, 2024-03-25)
    Spatially resolved epigenomic profiling is critical for understanding biology in the mammalian brain. Singlecell spatial epigenomic assays were developed recently for this purpose, but they remain costly and labor intensive for examining brain tissues across substantial dimensions and surveying a collection of brain samples. Here, we demonstrate an approach, epigenomic tomography, that maps spatial epigenomes of mouse brain at the scale of centimeters. We individually profiled neuronal and glial fractions of mouse neocortex slices with 0.5 mm thickness. Tri-methylation of histone 3 at lysine 27 (H3K27me3) or acetylation of histone 3 at lysine 27 (H3K27ac) features across these slices were grouped into clusters based on their spatial variation patterns to form epigenomic brain maps. As a proof of principle, our approach reveals striking dynamics in the frontal cortex due to kainic-acid-induced seizure, linked with transmembrane ion transporters, exocytosis of synaptic vesicles, and secretion of neurotransmitters. Epigenomic tomography provides a powerful and cost-effective tool for characterizing brain disorders based on the spatial epigenome.
  • OntoType: Ontology-Guided and Pre-Trained Language Model Assisted Fine-Grained Entity Typing
    Komarlu, Tanay; Jiang, Minhao; Wang, Xuan; Han, Jiawei (ACM, 2024-08-25)
    Fine-grained entity typing (FET), which assigns entities in text with context-sensitive, fine-grained semantic types, is a basic but important task for knowledge extraction from unstructured text. FET has been studied extensively in natural language processing and typically relies on human-annotated corpora for training, which is costly and difficult to scale. Recent studies explore the utilization of pre-trained language models (PLMs) as a knowledge base to generate rich and context-aware weak supervision for FET. However, a PLM still requires direction and guidance to serve as a knowledge base as they often generate a mixture of rough and fine-grained types, or tokens unsuitable for typing. In this study, we vision that an ontology provides a semantics-rich, hierarchical structure, which will help select the best results generated by multiple PLM models and head words. Specifically, we propose a novel annotation-free, ontology-guided FET method, OntoType, which follows a type ontological structure, from coarse to fine, ensembles multiple PLM prompting results to generate a set of type candidates, and refines its type resolution, under the local context with a natural language inference model. Our experiments on the Ontonotes, FIGER, and NYT datasets using their associated ontological structures demonstrate that our method outperforms the state-of-the-art zero-shot fine-grained entity typing methods as well as a typical LLM method, ChatGPT. Our error analysis shows that refinement of the existing ontology structures will further improve fine-grained entity typing.