A speech-driven embodied entrainment character system with a delayed voice back-channel based on negative emotional expression utterances

The prior research includes the development of a speech-driven embodied entrainment computer-generated character called ”InterActor”, which automatically generates communicative motions and actions such as nods for entrained interaction from voice rhythm based on only speech input. However, the conventional InterActor generates only positive actions and back-channels without negative reactions, which may promote negative emotions of the user in the case of negative utterances such as self-denial. In this study, we developed an embodied character system with a delayed voice back-channel based on negative emotional expression utterances. Two experiments were performed to confirm the evaluation of back-channel feedback timing by sensory evaluation. In the first experiment, the timing of the voice back-channel to nodding motion was examined. In the second experiment, the timing of the voice back-channel to nodding motion in negative utterances was examined. As a result, it was shown that the timing of the voice back-channel delay of about 600ms was allowed from the start of the nodding motion estimated by InterActor. In negative utterances, the timing of allowance was about 900ms. Finally, we developed a prototype system based on the speaker’s emotional state using speech recognition.