Audio augmented reality using sonification to enhance visual art experiences: Lessons learned

Abstract

Augmenting visual art in art galleries can be an effective Audio Augmented Reality (AAR) application for indoor exploration. In the current study, eight paintings from four genres were augmented with audio through their sonification. Basic Audio was generated using a sonification algorithm by identifying the major colors of the paintings, and Enhanced Audio was generated by a musician enhancing the Basic Audio; these were presented with the paintings to compare against No Audio. Twenty-six participants viewed each painting in all three conditions; eye gaze metrics, and qualitative data were collected. Results showed that Enhanced Audio led to significantly greater engagement and positive sentiments, compared to Basic Audio. Thematic analysis showed semantic and syntactic relationships of the audio with the paintings, and a tendency to guide users’ gaze over time. Findings from this study can guide future AAR developments to improve auditory display designs to enhance visual experiences.