%402x.svg){kind=icon}
Authors: Raja Kushalnagar, Han Zhang, Rotem Shalev-Arkushin, V. Baltatzis, Connor Gillis, Gierad Laput, Lorna C. Quandt, Leah Findlater, Abdelkareem Bedri, Colin Lea
Venue: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI 2025) (2025). doi:10.1145/3706598.3713855
Sign languages are essential for the Deaf and Hard-of-Hearing (DHH) community. Sign language generation systems have the potential to support communication by translating from written languages, such as English, into signed videos. However, current systems often fail to meet user needs due to poor translation of grammatical structures, the absence of facial cues and body language, and insufficient visual and motion fidelity. We address these challenges by building on recent advances in LLMs and video generation models to translate English sentences into natural-looking AI ASL signers. The text component of our model extracts information for manual and non-manual components of ASL, which are used to synthesize skeletal pose sequences and corresponding video frames. Our findings from a user study with 30 DHH participants and thorough technical evaluations demonstrate significant progress and identify critical areas necessary to meet user needs.
In this paper, we proposed a prototype ASL generation system aimed at improving the naturalness, comprehensiveness, and overall quality of generated signs, addressing key limitations in existing approaches. Our technical evaluations indicate that our proposed approaches improve these aspects, enhancing the quality of generated ASL content. Feedback from DHH participants was mixed; while there was general interest in the system, concerns regarding visual quality and naturalness were noted. Reflecting on our design process and study findings, we discuss key insights and identify key areas for future improvement. While further work is needed, our study takes an initial step toward developing sign language generation systems that better meet the needs of the DHH and signing communities, offering real-world value.
