第二外語學習者之自動發音評測及錯誤發音偵測研究

Abstract

隨著全球化的趨勢，電腦輔助發音訓練(CAPT)系統越來越受歡迎，應用於 減輕教師工作量、發音評測線上課程和幫助學習者練習語言技能等場景。本 論文提出了一系列創新的 CAPT 建模技術，以應對各種教學和自學應用，展 示了其強大的潛力和實用價值。在自動語音評估 (Automatic Pronunciation Assessment, ASA) 方面，我們針對資料不平衡問題，採用了類平衡損失函數 和重新採樣方法，縮小了訓練集和測試集之間的差距，並在不平衡資料集 speechocean762 上顯示出顯著的性能提升。在錯誤發音偵測與診斷 (Mispronunciation Detection and Diagnosis, MDD) 方面，我們使用了一種新穎 的基於文本提示引導聽寫模型，通過音素依賴閾值有效平衡精度和召回率， 同時引入多視角音頻編碼器提供細粒度發音提示。這些創新方法能夠更精確 地識別並診斷 L2 學習者的發音錯誤，並提供即時反饋。在 L2-ARCTIC 基準 數據集上的綜合實驗結果表明，我們的方法在多個競爭基線中具有實際可行 性。然而未來的研究可以探索更多樣化的語言和發音情境，以進一步提升 CAPT 系統的適用性和實用性。同時，我們也希望未來可以探索 APA 和 MDD 的聯合模型，以充分利用兩者的優勢，提供給學習者在使用系統上得到更好 的回饋。

With the trend of globalization, computer-assisted pronunciation training (CAPT) systems are becoming increasingly popular, applied in scenarios such as reducing teachers' workload, pronunciation assessment in online courses, and helping learners practice language skills. This thesis proposes a series of innovative CAPT modeling techniques to address various teaching and self-study applications, demonstrating their strong potential and practical value. In the area of Automatic Pronunciation Assessment (APA), we tackled the issue of data imbalance by adopting a balanced loss function and resampling methods, narrowing the gap between training and test sets, and showing significant performance improvements on the imbalanced dataset Speechocean762. In the field of Mispronunciation Detection and Diagnosis (MDD), we employed a novel prompt-guided model, effectively balancing precision and recall through phone-dependent thresholds while introducing a multi-view audio encoder to provide fine-grained articulatory cues. These innovative methods enable more precise identification and diagnosis of pronunciation errors in L2 learners, offering timely feedback. Comprehensive experimental results on the L2-ARCTIC benchmark dataset indicate that our methods are practically feasible compared to multiple competitive baselines.Future research can explore more diverse language and pronunciation scenarios to further enhance the applicability and practicality of CAPT systems. We also hope to explore joint models of APA and MDD in the future, leveraging the advantages of both to provide better feedback for learners using the system. In summary, this study demonstrates the potential of using innovative technologies in CAPT systems, which not only improves the accuracy of pronunciation assessment but also better assists language learners in improving their pronunciation skills. This thesis explores the feasibility of increasing the practical application of CAPT and aims to have a positive impact on the field of language education, promoting the spread and development of language learning.