ImageSI: Interactive Deep Learning for Image Semantic Interaction

TR Number



Journal Title

Journal ISSN

Volume Title


Virginia Tech


Interactive deep learning frameworks are crucial for effectively exploring and analyzing complex image datasets in visual analytics. However, existing approaches often face challenges related to inference accuracy and adaptability. To address these issues, we propose ImageSI, a framework integrating deep learning models with semantic interaction techniques for interactive image data analysis. Unlike traditional methods, ImageSI directly incorporates user feedback into the image model, updating underlying embeddings through customized loss functions, thereby enhancing the performance of dimension reduction tasks. We introduce three variations of ImageSI, ImageSItextMDS−1, prioritizing explicit pairwise relationships from user interaction, and ImageSItextDRTriplet and ImageSItextPHTriplet, emphasizing clustering by defining groups of images based on user input. Through usage scenarios and quantitative analyses centered on algorithms, we demonstrate the superior performance of ImageSItextDRTriplet and ImageSItextMDS−1 in terms of inference accuracy and interaction efficiency. Moreover, ImageSItextPHTriplet shows competitive results. The baseline model, WMDS−1, generally exhibits lower performance metrics.



Semantic Interaction, Deep Learning, Dimension Reduction, Images