Text-to-Sign Language Video Generation Using GANs, BERT, and Sora

This paper explores the generation of American Sign Language (ASL) videos using Generative Adversarial Networks (GANs), BERT-based text embeddings, and a dataset comprising authentic and synthetic SL clips. The original Kaggle dataset was enriched by creating a manually crafted collection. OpenAI's Sora video generator was then employed to augment the dataset by producing synthetic videos using multimodal prompts. Researchers implemented and compared several GAN architectures, including unimodal and Feature-wise Linear Modulation (FiLM) 3D convolutional generators, which integrate text embeddings for modal fusion. While preliminary training was conducted, quantitative evaluation revealed significant challenges in generating realistic and coherent ASL videos. Current results highlight the complexities of ASL video synthesis and underscore the need for advanced ASL generative applications.