栩栩如生:動畫人物動作重演

Abstract

動作重演旨在將驅動影片中人物的動作轉換為來源影像中人物的動作，並依照轉換結果生成一段動作重演動畫。近期研究多探討真實人類至真實人類的動作重演，甚至已經能夠生成真假難辨的動畫。本研究認為動作重演的價值在於將真實人類的動作轉換為動畫人物的動作，因為可提升動畫人物的動作品質，豐富使用者的娛樂體驗。本研究期望動畫人物可重演真實人類談話時的面部表情、頭部轉動和肩膀移動等動作，於是分別從VoxCeleb資料集收集名人訪談影片和AnimeCeleb資料集收集動畫人物動作影片，並將這兩種類型的影片組成Celebrity450訓練集。藉由讓系統從Celebrity450訓練集中學習真實人類和動畫人物的動作轉換，預測真實人類至動畫人物的動作轉換。現有技術易受到真實人類和動畫人物幾何差異的影響，產生面部動作轉換能力不足的問題，本研究提出多解析度光流技術，讓系統分別從不同解析度的特徵圖學習相應解析度的光流。同時，AnimeCeleb資料集沒有提供動畫人物肩膀移動的動作資訊，造成系統不能學習動畫人物肩膀移動的動作轉換。本研究在動畫人物動作影片進行特定型態的資料增強，透過移動每張影格的像素座標，模擬動畫人物各類型的肩膀動作。此外，當前指標不能明確評估動作轉換的性能，本研究提出反轉評估技術，透過比對重建之來源影像與來源影像的差距，間接評估動作重演的性能。實驗表明本研究在動畫人物動作重演領域取得重大的突破，多解析度光流技術不僅改善現有技術的缺失，還能將真實人類面部動作的細節轉換至動畫人物；特定型態資料增強讓系統可從訓練集學習動畫人物肩膀移動的動作轉換，實現生動的動畫人物動作重演；反轉評估技術展現穩健的信效度，提供未來研究一個明確可行的評估指標。

Motion transfer aims to transfer motion of the character from driving video to source image, and generates animation based on the transferred results. Recent works have focused on motion transfer from real humans to real humans, even producing animations that are nearly indistinguishable from reality. We believe the value of motion transfer lies in transferring motion from real humans to animated characters, since it can improve the motion quality of animated characters and enhance the user’s entertainment experience.We expect animated characters to reenact real human facial expressions, head rotations, and shoulder motions. Thus, we collected some celebrity talking videos from VoxCeleb dataset and some animated character motion videos from AnimeCeleb dataset, then combined these two types of videos into the Celebrity450 training set. The system can predict motion transfer from real humans to animated characters by learning the motion transfer of both real humans and animated characters in the Celebrity450 training set. The current technology is influenced by geometric differences between real humans and animated characters, which leads to inadequate facial motion transfer capability. We propose multi-resolution optical flows technology, allowing the system to learn optical flow at different resolutions from the feature maps at each resolution." Meanwhile, The AnimeCeleb dataset does not provide any animated character shoulder motion information, making the system unable to learn the shoulder motion transfer of animated characters. We perform specific types of data augmentation on animated character motion videos, simulating various types of animated character shoulder motions by adjusting the pixel coordinates within each frame. In addition, the current metric cannot clearly evaluate the motion transfer performance. We propose inverse metric technology to indirectly evaluate motion transfer performance by measuring the difference between reconstructed source image and source image.Experiments demonstrate that our methods have achieved a very significant breakthrough in animated character motion transfer. Multi-resolution optical flows technology not only addresses the shortcomings of current technology, but also transfers fine-detail facial motion from real humans to animated characters. Specific types of data augmentation enable the system to learn the shoulder motion transfer of animated characters, achieving vivid animated character motion transfer. The inverse metric technology demonstrates steady reliability and validity, providing a clear metric for future work.