即時單通道語音增強技術之研究

Abstract

隨著深度學習的發展，語音增強 (Speech Enhancement) 技術更能在各種不同的噪音環境下發揮效果。本論文針對語音增強的兩個子議題進行研究，分別是對於語音辨識 (Speech Recognition) 系統的語音增強前端預處理 (Front-end Preprocessing)，以及線上 (On-line) 的串流式即時語音增強。語音增強研究首要目標是提升感知品質 (Perceptual Quality)，然而若將最大化感知品質的模型用於預處理，可能會影響下游任務所依賴的聲學特徵，這樣的失真會導致效果不如預期，甚至衰退。而除了維持聲學特徵外，本論文也探討如何更有效的使用頻譜資訊，以及針對即時語音增強模型提高運算效率。 本論文在兩個方法上各自提出了改進，第一項為 DCT-TENET，以時序反轉增強網路 (Time-reversal Enhancement NETwork, TENET) 為基礎，針對語音增強的訓練流程調整，在保有一定增強效果的前提上，作為語音辨識前端處理機制也能更有效的提升辨識率，並且減少額外訓練聲學模型的需求。第二項為可調適性全次頻帶融合網路 (Adaptive-FSN)，透過延伸「次頻帶能有效處理局部樣態 (Local Pattern)」的概念，提出一個可調適性次頻帶機制，壓縮大範圍相鄰頻帶之有效資訊來提高語音品質，並搭配其他改進以提高運算的效能。我們使用 VoiceBank-DEMAND 資料集對兩個方法進行實驗，改進後的 DCT-TENET相較 TENET 模型，能進一步的提升語音辨識系統於受噪語音的辨識率。使用乾淨情境聲學模型辨識 DEMAND 噪音之測試集降低相對約 7.9% 的詞錯誤率，使用多情境聲學模型於額外的未見噪音測試集也能降低相對約 10.6% 的詞錯誤率。另一部分，Adaptive-FSN 也相較基礎的 FullSubNet+，在語音品質指標上有更佳的表現，於 CPU 上運算則能有效的降低相對 44% 的實時率 (Real-time Factor)。With the development of deep learning, Speech Enhancement techniques can be more effective in various noise situations. This paper investigates two sub-topics of speech enhancement, namely, front-end pre-processing with speech enhancement for speech recognition, and online real-time speech enhancement. The primary goal of speech enhancement is to improve perceptual quality. However, if we use the model that training to maximize perceptual quality for pre-processing, it may affect the acoustic features that downstream tasks relied on, such distortion may lead to less than expected results or degradation. In addition to maintaining acoustic features, this paper also explores how to use spectrogram information more effectively and improve computational efficiency in a real-time speech enhancement model.This paper proposes improvements to each of the two models. The first one is DCT-TENET, which is based on Time-reversal Enhancement NETwork (TENET). The proposed model adjusts the training process for speech enhancement, which can improve the recognition rate more effectively as a front-end processing mechanism for speech recognition while maintaining desirable perceptual quality, and finally reduce the need for additional training in acoustic models of ASR. The second one is Adaptive-FSN, which extends the concept of"sub-band mechanism can effectively handle local pattern". We propose an adaptive sub-band mechanism to compress the rich information in a wide range of adjacent bands to improve speech quality, along with other improvements to speed up the computation. We use the VoiceBank-DEMAND dataset to experiment with the two methods: 1) the improved DCT-TENET can improve the recognition rate of the downstream ASR for noisy speech compared to the TENET model. The relative word error rate (WER) of using the clean context acoustic model on the DEMAND test set reduces by about 7.9%, and the relative WER of using the multi-context acoustic model on the supplemental unseen noise test set also reduces by about 10.6%. 2) Adaptive-FSN performs better than the basic FullSubNet+ in terms of speech quality metrics and effectively reduces the Real-time Factor by 44% on the CPU.