Sensor-Based Gesture Detection Using Bidirectional LSTM with Self-Attention and Conditional Random Field

Abstract

本論文的目的是為感測器產生的數據提供一種新穎的手勢檢演算法，在該演算法中，我們使用了self-attention，雙向LSTM（Bi-LSTM）和條件式隨機場（CRF）。self-attention的作用是使模型專注於輸入訊號重要的部分，Bi-LSTM則同時參考過去和未來時間點的訊息，最後，CRF則針對Bi-LSTM的輸出以預測結果應有的行為模式做修正，以產生最終的檢測結果。我們使用了實驗室原創的、配有感測器的智慧手套進行實驗來評估該演算法的效能。實驗結果表明，此演算法不但能對訊號中的手勢序列進行有效地檢測，還能與現有的手勢分類算法結合使用，從而達到以傳感器為基礎的手勢序列識別的目的。

The goal of this thesis is to present a novel hand gesture detection algorithm for the sensory data produced by flex sensors. In the algorithm, the self-attention operations, Bi-directional Long Short Term Memory (Bi-LSTM), and Continuous Random Field (CRF) are employed for the effective detection of hand gestures. The self-attention operations are adopted for hi-lighting the significant portions of input sensory data for detection. The Bi-LSTM further exploits the correlation among the input sequences in both directions. The correlation between input samples and output labels, and among the output labels, are then explored by the CRF to produce the final detection results. A prototype of smart glove equipped with flex sensors has been built for the evaluation of the proposed algorithm. Experimental results reveal that the proposed algorithm is able to carry out detection of gesture sequences in the sensory data. The proposed algorithm can also operate in conjunction with existing gesture classification algorithms for the accurate recognition of gesture sequences based on flex sensors.