Effects of Sentiment-Based Sonification on Fairy Tale Listening Experience

Abstract

Sonification can translate emotional data into sound, allowing us to interpret emotional information. However, the effectiveness of sonification varies, depending on its application. The main purpose of the current study is to investigate how the emotion-reflecting sonification vs. the emotion-mitigating sonification influences the experience of listening to fairy tales. We analyzed three fairy tales using sentiment analysis and created two types of sonification: one that reflects the representative emotion of each paragraph (emotion-reflecting sonification), and another that reflects the opposite emotion of the paragraph (emotion-mitigating sonification). For the story-telling agent, we used a humanoid robot, NAO and designed three voices with Azure. We conducted five focus groups with 22 participants. The results showed that participants' ratings of the “pleasing”, “empathy”, and “immersiveness” categories were higher in emotion-mitigating sonification than in emotion-reflecting sonification. Moreover, participants showed a preference for male voice than female or child voices. The results are discussed with implications and future directions.