Artificial Intelligence and accessible theater for the deaf and blind
UC3M/DICYT This new technology from UC3M’s SoftLab research group has been implemented to give accessibility to the Broadway musical comedy “The Addams Family” at the Teatro Calderón of Madrid. The creative agency C&W, COMUNICAdos and Escena Global also collaborated on the project, which received support from the Consejería de Educación e Innovación (Council for Education and Innovation) of the Autonomous Community of Madrid. Thanks to this project, individuals with visual and auditory impairment have been able to enjoy any of the performances of this musical comedy since the 3rd of December, the International Day of Persons with Disabilities. This means that over one million deaf people in Spain can, for the first time in their lives, attend an important, high-quality musical such as this.
This innovation, developed by this research group from UC3M’s Instituto de Desarrollo Tecnológico y Promoción de la Innovación Pedro Juan de Lastanosa (Pedro Juan de Lastanosa Institute for Technological Development and Innovation), is based on software that uses an augmented reality system to allow members of the audience to individually see adapted subtitles and a sign language interpreter. Moreover, the system is automatically synchronized with the play, with no human involvement, thanks to the artificial intelligence techniques that constantly detect what is happening on the stage. Then the spectator receives the contents by means of the application GoAll.
Stage-sync is a technology “learns” as more performances are held, and adapts to the changes in rhythm and actors that typically occur in the theater. Using deep learning techniques combined with audio processing the software achieves the perfect synchronization of the performance and accessibility elements. In addition, with the augmented reality visualization system, it is no longer necessary to include subtitles in the scenery because each spectator can see them integrated into the show.
According to the head of UC3M’s SoftLab, Professor Ángel García Crespo, until now theater performances with this target audience in mind were few and far between, and they did not include sign language. “Now”, he stated, “It is possible for ALL performances to be accessible.”
A project with multiple collaborators
This adaptation project at the Teatro Calderón included the collaboration of Samsung, as well as that of the art director Carlos Alcalde, through the production companies Entrecalles Producciones and DTF Transmedia, which worked closely with the creative teams from the agency C&W to integrate artistic and esthetic material with the accessibility elements, another unique, pioneering action in this sphere. “When LETSGO and Esteve started to adapt the play, we realized that Thing (the hand in “The Addams Family”) could not receive the same focus in the theater as it did in the film, and we thought – really—this was the perfect opportunity to give it an important part in the play, by converting Thing into our sign language interpreter; this seems like a natural role for the character, given that it expresses itself exclusively through its hands,” declared Carmelo Rodríguez, of C&W creative agency.
The project is financed with the support of the Consejería de Educación e Investigación (Council for Education and Research) of the Autonomous Community Madrid as part of the grants for the "Cheque Innovación" (Innovation Check) program, which is co-financed by the European Regional Development Fund to incentivize the use of Innovative services.