A Review of Speech-Based Bimodal Recognition

Speech recognition and speaker recognition by
machine are crucial ingredients for many important applications
such as natural and flexible human-machine interfaces. Most
developments in speech-based automatic recognition have relied
on acoustic speech as the sole input signal, disregarding its visual
counterpart. However, recognition based on acoustic speech alone can be afflicted with deficiencies that preclude its use in many real-world applications, particularly under adverse conditions.
The combination of auditory and visual modalities promises
higher recognition accuracy and robustness than can be obtained with a single modality. Multimodal recognition is therefore acknowledged as a vital component of the next generation of spoken language systems. This paper reviews the components of bimodal recognizers, discusses the accuracy of bimodal recognition, and highlights some outstanding research issues as well as possible application domains.