TY - GEN
T1 - Deep audio-visual speech separation based on facial motion
AU - Rigal, Rémi
AU - Chodorowski, Jacques
AU - Zerr, Benoît
N1 - Publisher Copyright:
Copyright © 2021 ISCA.
PY - 2021/1/1
Y1 - 2021/1/1
N2 - We present a deep neural network that relies on facial motion and time-domain audio for isolating speech signals from a mixture of speeches and background noises. Recent studies in deep learning-based audio-visual speech separation and speech enhancement have proven that leveraging visual information in addition to audio can yield substantial improvement to the prediction quality and robustness. We propose to use facial motion, inferred from optical flow techniques, as a visual feature input for our model. Combined with state-of-the-art audio-only speech separation approaches, we demonstrate that facial motion significantly improves the speech quality as well as the versatility of the model. Our proposed method offers a signal-to-distortion improvement of up to 4.2 dB on two-speaker mixtures when compared to other audio-visual approaches.
AB - We present a deep neural network that relies on facial motion and time-domain audio for isolating speech signals from a mixture of speeches and background noises. Recent studies in deep learning-based audio-visual speech separation and speech enhancement have proven that leveraging visual information in addition to audio can yield substantial improvement to the prediction quality and robustness. We propose to use facial motion, inferred from optical flow techniques, as a visual feature input for our model. Combined with state-of-the-art audio-only speech separation approaches, we demonstrate that facial motion significantly improves the speech quality as well as the versatility of the model. Our proposed method offers a signal-to-distortion improvement of up to 4.2 dB on two-speaker mixtures when compared to other audio-visual approaches.
KW - Computer vision
KW - Deep learning
KW - speech separation
UR - https://www.scopus.com/pages/publications/85119180555
U2 - 10.21437/Interspeech.2021-1560
DO - 10.21437/Interspeech.2021-1560
M3 - Conference contribution
AN - SCOPUS:85119180555
T3 - Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
SP - 2358
EP - 2362
BT - 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021
PB - International Speech Communication Association
T2 - 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021
Y2 - 30 August 2021 through 3 September 2021
ER -