結合臉部表情及聲音之嬰兒情緒辨識系統

Abstract

嬰兒的情緒發展會影響未來的學習力、注意力甚至於成長後的個性及人際關係，而在人一生的情緒發展中以嬰兒時期的情緒發展最為重要。所以若是能得知嬰兒目前情緒及生理需求並予以滿足，對未來發展影響甚大，然而嬰兒在1歲之前只能使用臉部表情及不帶詞意的聲音來向父母表達自己目前的情緒及生理需求。所以本論文開發一套結合嬰兒臉部表情及聲音的監控系統，適時協助轉達嬰兒情緒，以減輕父母照顧嬰兒的負擔，更幫助父母妥善的照顧嬰兒。 本系統一開始分成兩部分執行，一部分為影像部分，另一部分為聲音部分。影像部分主要分為嬰兒臉部偵測及臉部特徵擷取，當系統讀入連續的嬰兒影像後，會從影像中擷取膚色區域並從這些膚色區域中找出嬰兒的臉部區域。接著採用local ternary pattern標示影像中嬰兒臉部輪廓，並進行差分影像累積，最後計算累積差分影像中0階至3階的Zernike moments值，當作嬰兒臉部特徵使用。而聲音方面利用常見的mel frequency cepstral coefficients與其差量倒頻譜係數當作嬰兒聲音特徵使用。最後利用support vector machine將影像及聲音特徵分別進行分類，並將兩者分類結果整合成嬰兒情緒類別。 實驗影片共有100段，其中每段影片僅包含單一情緒類別，合計影片長度為100分鐘，拍攝嬰兒之月齡為1個月至7個月，而嬰兒情緒辨識之平均正確率約為85.3%，由此可知，本系統的辨識結果具有一定的可信度。

The emotional development of infants will affect their learning ability, attention, personality and interpersonal in the future, thus it is very important in the life of person. However infants are difficult to use words to express their emotions or physiological needs, others can understand their emotion or physiological needs by their facial expressions, vocalization, and body movements. Therefore, the study presents an infant emotion recognition system using both facial expressions and vocalization to reduce the burden of parents to take care of the infants. The system can be divided into two parts: image processing part and speech processing part. Image processing part consists of two main stages: infant face detection and facial expression feature extraction. In the infant face detection stage, the system detects the skin color pixels from the input images and uses the connect component technology to find the biggest skin color region which is regarded as the face of infants. In the facial expression feature extraction stage, the system uses the local ternary pattern technology to label the face contour of the infants and calculates the values of 0 to 3 order Zernike moments in the cumulative difference image. In speech processing part, the system uses common mel frequency cepstral coefficients and its delta cepstrum coefficients as speech features. Finally the system uses support vector machine to classify the facial expression features and vocalization respectively. By combining two types of classification results, the system gets the emotion of the infants. The number of experimental sequence is 100 with total length 100 minutes and the infants in these sequences are 1-7 months old. Each sequence only contains one emotion, while the average rate of infant emotions is 85.3%. As a result, the proposed system is robust and efficient.