Sensor-Based Gesture Detection Using Bidirectional LSTM with Self-Attention and Conditional Random Field

No Thumbnail Available



Journal Title

Journal ISSN

Volume Title



The goal of this thesis is to present a novel hand gesture detection algorithm for the sensory data produced by flex sensors. In the algorithm, the self-attention operations, Bi-directional Long Short Term Memory (Bi-LSTM), and Continuous Random Field (CRF) are employed for the effective detection of hand gestures. The self-attention operations are adopted for hi-lighting the significant portions of input sensory data for detection. The Bi-LSTM further exploits the correlation among the input sequences in both directions. The correlation between input samples and output labels, and among the output labels, are then explored by the CRF to produce the final detection results. A prototype of smart glove equipped with flex sensors has been built for the evaluation of the proposed algorithm. Experimental results reveal that the proposed algorithm is able to carry out detection of gesture sequences in the sensory data. The proposed algorithm can also operate in conjunction with existing gesture classification algorithms for the accurate recognition of gesture sequences based on flex sensors.



自注意力機制, 條件式隨機場, self-attention, conditional random field