Sequence Recognition of Indoor Tennis Actions Using Transfer Learning and Long Short-Term Memory

Abstract

Recognizing tennis actions during a practice session can be widely used as the coaching assistant. Accurate classification of actions is tricky, with a massive similarity among different actions. Hybrid deep neural networks composed of Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) such as Long Short-term Memory (LSTM) are widely employed while dealing with spatial and temporal features. We leverage transfer learning as the spatial feature extractor, allowing weights extracted by training over a massive dataset. Transfer learning from three pre–trained models such as InceptionResNetV2, ResNet152V2, and Xception are utilized in this work. Two approaches are exercised to determine the best one: (a) developing a single hybrid CNN–LSTM model and (b) passing extracted CNN features to a single LSTM-based model. Experimental results prove the effectiveness of the latter approach for recognizing indoor tennis actions. The publicly available THETIS dataset is employed for all the evaluations and approaches, which contains 12 different tennis actions performed in indoor practice sessions. The Xception–based model outperforms other models by attaining 75% accuracy.