Bayesian Fusion for Multimodal Human-Robot Interaction

To cope with perceptual uncertainty, humans integrate information from multiple sources, such as visual, auditory, and haptic input. A large body of evidence shows that this multimodal integration is often statistically optimal and can be explained by Bayes’ rule. In an inherently uncertain world, however, combining information from multiple sources is crucial not only for humans but also for robots, which are increasingly taking on tasks in everyday and professional settings. In this talk, I will present how Bayes-optimal fusion of information from different sources can improve human–robot interaction. In particular, I will show how a robot’s uncertainty about human intentions can be reduced by optimally combining multimodal signals such as speech, gestures, and gaze. I will further present how to improve robot learning through Bayesian fusion of multimodal human advice, and how robots can detect a person’s intention to initiate interaction from multimodal cues. Across all these use cases, the proposed methods reduce uncertainty in a principled way and improve interaction quality and decision support in uncertain environments.