%402x.svg){kind=icon}
Authors: Raja Kushalnagar, Si Chen, Haocong Cheng, Suzy Su, Stephanie Patterson, Yun Huang, Qi Wang
Venue: Proceedings of the ACM on Human-Computer Interaction (PACMHCI) (2025). doi:10.1145/3710953
This study investigates innovative interaction designs for communication and collaborative learning between learners of mixed hearing and signing abilities, leveraging advancements in mixed reality technologies like Apple Vision Pro and generative AI for animated avatars. Adopting a participatory design approach, we engaged 15 d/Deaf and hard of hearing (DHH) students to brainstorm ideas for an AI avatar with interpreting ability (sign language to English and English to sign language) that would facilitate their face-to-face communication with hearing peers. Participants envisioned the AI avatars to address some issues with human interpreters, such as lack of availability, and provide affordable options to expensive personalized interpreting services. Our findings indicate a range of preferences for integrating the AI avatars with actual human figures of both DHH and hearing communication partners. The participants highlighted the importance of having control over customizing the AI avatar, such as AI-generated signs, voices, facial expressions, and their synchronization for enhanced emotional display in communication. Based on our findings, we propose a suite of design recommendations that balance respecting sign language norms with adherence to hearing social norms. Our study offers insights into improving the authenticity of generative AI in scenarios involving specific and sometimes unfamiliar social norms.
The findings from this study contribute to ongoing efforts to create more inclusive communication and learning technologies. Using participatory approaches, we explored the innovative design of MR (Mixed-Reality) empowered by AI that can interpret to enhance face-to-face communication between DHH and hearing people, featuring a signing avatar for hearing individuals and a voicing avatar for DHH individuals. Throughout the study, DHH participants expressed enthusiasm about the envisioned technology to improve communication with hearing individuals in various aspects of their lives, including in work and family settings. The concept of instant, personalized, and emotion-rich communication resonated positively with the participants, who recognized its potential to make DHH-Hearing face-to-face communication more seamless. While acknowledging the benefits, DHH participants also expressed concerns about how such AI-generated voices might behave and adapt to auditory social norms. Exploring and evaluating more fine-grained controls for customizing the AI-generated avatar could be a focus of future research. We discuss design recommendations for future MR-supported communication and learning for individuals with mixed abilities, extending beyond just the hearing and DHH communities.
